numinnex opened a new issue, #4725: URL: https://github.com/apache/iggy/issues/4725
### Description The Iceberg sink makes one `fast_append` commit per `consume` call, so every poll batch adds one snapshot and one manifest. Nothing expires snapshots or merges manifests, so the table metadata grows for as long as the sink runs and every commit costs more than the one before. Measured with `batch_length = 100` at 2,000 rows/s on a single-node setup, with the duplicate-file check from the manifest issue already turned off: - Commit time rose from 58 ms to 167 ms over ~2,780 commits. - `metadata.json` reached 1.8 MB. Throughput is about batch size divided by commit latency, so at batch 100 the sink cannot keep up with 2,000 rows/s. An external test saw the same pattern: 1,200 snapshots per 120k rows, and the sink drained about 350 rows/s against 2,000 rows/s produced. ### Affected area / component Connectors: `iggy_connector_iceberg_sink` ### Proposed solution 1. Document that `batch_length`, `min_count` and `max_wait` (available since #4629) set the commit rate, and recommend values for Iceberg. 2. Decide between maintenance inside the sink and documented external maintenance. iceberg-rust 0.10 has `ExpireSnapshotsAction`, so the sink could offer opt-in `retain_last` or older-than settings. The alternative is to document that operators run snapshot expiry and manifest rewrites themselves. 3. Commit on a size or time threshold instead of per poll batch. This is only safe once sink offsets follow delivery (#4232). ### Alternatives considered Leaving all maintenance to an external engine with no guidance. Today the sink's docs don't mention it at all. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
