numinnex opened a new issue, #4721: URL: https://github.com/apache/iggy/issues/4721
### Bug description `write_data` in `core/connectors/sinks/iceberg_sink/src/router/mod.rs` commits with `fast_append()`, which keeps iceberg-rust's default `check_duplicate = true`. Before each commit, `validate_duplicate_files` (iceberg 0.10.1, `transaction/snapshot.rs`) loads every manifest of the current snapshot, each on its own spawned task, with no limit on concurrency. The sink writes one manifest per commit, so commit N issues N concurrent S3 GETs. This has two effects: - OpenDAL sends every request through one shared reqwest pool. The pool grows to the peak concurrency and keeps idle connections for its 90 s idle timeout, so the number of open S3 sockets follows the number of commits and never levels off under load. With the macOS default limit of 256 open files, the sink fails with "too many open files". - Commit time grows linearly with the table, so the drain rate falls the longer the sink runs. The check can never find a duplicate for this sink. Data file names come from `DefaultFileNameGenerator`, seeded with a fresh UUIDv4 per batch. ### Affected area / component Connectors: `iggy_connector_iceberg_sink` ### Versions master @ 3c265dd89, iceberg / iceberg-catalog-rest / iceberg-storage-opendal 0.10.1, opendal 0.57.0 ### Reproduction - Single-node server, one topic with 4 partitions. - `apache/iceberg-rest-fixture` with `floci/floci:2.1.0` as S3, the same setup as `core/integration/tests/connectors/fixtures/iceberg`. - Iceberg sink with `batch_length = 100`. - 2,000 JSON rows/s for 300 s. - Debug build, Linux. Sockets were counted with `ss` against the runtime pid. Cycle time per batch is taken from the gaps between snapshot timestamps. | | master | `fast_append().with_check_duplicate(false)` | |---|---|---| | S3 sockets held by the runtime | 1,815 at the end and still rising | 1 for the whole run | | Cycle time per batch | 72 ms to 365 ms over ~1,860 commits | 58 ms to 167 ms over ~2,780 commits | | Rows drained of 600,000 sent | 186,500 | 278,900 | At one sample in the master run, 664 sockets were open against 695 snapshots. An external test on macOS showed the same behaviour: - S3 connections rose from about 1,600 to 2,330 over 5 minutes. - The sink hit "too many open files" at the 256 limit. - Commit time rose from 206 ms to 360 ms. ### Proposed fix `table_commit.fast_append().with_check_duplicate(false)`. Commit time still grows after the fix, because the table metadata and the manifest list are rewritten on every commit. That part is tracked separately as snapshot expiry. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
