jpatidar30 opened a new issue, #2712:
URL: https://github.com/apache/iggy/issues/2712
### **Context**
I am currently working on a data pipeline to sync data from **Iggy** to an
**Apache Iceberg** table via a REST catalog (Lakekeeper). During deployment on
AWS, I encountered two critical blockers in the `iggy-connectors` runtime that
prevent the sink from initializing.
---
### **Description of Blockers**
#### **1. Metadata Panic (`creation time is not available`)**
The `local_provider.rs` attempts to access file creation metadata via
`std::fs::Metadata::created()`. On many Linux environments (including AWS
EBS/XFS and Docker Overlay2), the filesystem driver does not support "birth
time." This causes a hard panic when the runtime scans the `conf.d` directory.
**Observed Error:**
`Error: IoError(Error { kind: Unsupported, message: "creation time is not
available on this platform currently" })`
**Temporary Workaround:** I had to patch the source to use `.modified()` as
a fallback when `.created()` returns `Unsupported`.
### **2. Forced `.so` Plugin Loading for Internal Sinks**
Even after addressing metadata issues, the runtime fails to load the
**internal** Iceberg sink when using a configuration file in `conf.d`.
The configuration parser requires the `path` field to be present in the
TOML. However, the runtime interprets the presence of this `path` as a
instruction to load an external shared library. It appends `.so` to the string
and panics because it cannot find the file, even though the Iceberg driver is
already compiled into the binary.
**I am using this configuration:**
```toml
version = 1
type = "sink"
key = "iceberg"
enabled = true
name = "iceberg-market-sink"
path = "iceberg_sink_state"
[[streams]]
stream = "1"
topics = ["1"]
group_id = "iggy_iceberg_group"
schema = "json"
[state]
path = "iceberg_sink_state"
[config]
catalog_type = "rest"
catalog_uri = ""
warehouse = ""
table_namespace = ""
table_name = ""
region = "ap-northeast-1"
s3_bucket = ""
commit_interval = "1m"
max_file_size_bytes = 134217728
[config.schema]
fields = [
{ name = "id", type = "long", nullable = false },
]
---
And I am getting this error:
thread 'main' (1) panicked at core/connectors/runtime/src/sink.rs:89:49:
Failed to load sink container: OpeningLibraryError(Custom { kind: Other, error:
"iceberg.so: cannot open shared object file: No such file or directory" })
### **System Information**
* **Iggy Version:** `v0.5.4` (and `main` as of Feb 2026)
* **OS:** Ubuntu 24.04 (AWS EC2)
* **Filesystem:** EBS (XFS/ext4)
* **Deployment:** Docker (privileged mode)
---
### **Steps to Reproduce**
1. Create a sink configuration in
`/etc/iggy/connectors/conf.d/iceberg_sink.toml`.
2. Include the required `path` field (e.g., `path = "iceberg"`).
3. Set `key = "iceberg"` and `type = "sink"`.
4. Start the `iggy-connectors` runtime.
5. Observe the panic as the loader attempts to find an external `iceberg.so`
file.
---
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
