seokjin0414 opened a new pull request, #3799: URL: https://github.com/apache/iggy/pull/3799
## What Adds a source connector that reads an [Apache Fluss](https://fluss.apache.org/) log table and publishes each row into an Apache Iggy stream as JSON. This is phase 1 of #3688: log tables and `Schema::Json`. ## How it works The connector resolves the table schema while opening, so a missing table, a primary-key table, a partitioned table or a column type with no JSON form all fail at startup rather than once per batch. The log scanner is created once in `open()` and reused, which works because it owns its client state rather than borrowing the connection. Offsets are tracked per bucket and returned in every `ProducedMessages`, including empty polls, so the runtime can persist them. On restart, buckets already present in the state keep their offset and only new ones fall back to `starting_offset`, which keeps a widened bucket count from rewinding buckets that were already consumed. Each message carries an id derived from its bucket and offset so Apache Iggy can deduplicate after an at-least-once replay. Column projection is pushed down to the server when `columns` is set. Temporal values keep their Fluss-native integer form, because Fluss carries no timezone that a formatted string could honour. ## The packaging question I asked in #3688 `fluss-rs` compiles its protocol definitions in a build script, so the workspace now needs a system `protoc`. I raised this in https://github.com/apache/iggy/issues/3688#issuecomment-5156005392 and said I leaned towards registering the crate as a regular workspace member, so that is what the last two commits do: `protobuf-compiler` goes into the shared Rust setup action, which every Rust workflow already uses, and CONTRIBUTING gets the prerequisite. If you would rather keep it out of the workspace build via `exclude`, only `build: register the Apache Fluss source in the workspace` and `ci: install protoc ...` need reworking. The connector and its test do not change. Happy to flip it. ## Testing Unit tests cover config validation, the state round-trip, offset resolution and the row-to-JSON mapping (including nulls, base64 for binary and non-finite floats). The integration test appends rows to a real Fluss log table and asserts on what comes back out of the Apache Iggy topic, offsets and bucket included. The cluster runs as a single container: the image ships `local-cluster.sh`, which starts the same embedded ZooKeeper, coordinator and tablet server, but it rewrites the tablet server's bind port to 0 so it cannot collide with the coordinator, and a random port inside the container cannot be published to the host. Giving the tablet server an explicit second port keeps both reachable and lets both advertise `localhost`, which then resolves the same way inside the container and from the test process. ## Notes on scope Three limits come from the released `fluss-rs` 0.1.0 rather than from the connector, and each is rejected at startup with the reason rather than silently ignored: - Primary-key changelog scanning is not in the release, so `table_type = "primary_key"` is refused. - `payload_format = "arrow_ipc"` is doable (0.1.0 does expose an Arrow `RecordBatch` scanner) but it is a separate scanner with its own offset-tracking path, so I left it for a follow-up. - `starting_offset = "latest"` needs `FlussAdmin::list_offsets`, whose offset spec argument has no public path in 0.1.0, so it cannot be resolved from outside the crate. Refs #3688 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
