Hi folks, Short heads-up from the Apache Storm side.
We have a new external module, storm-iceberg, that ingests streaming tuples directly into an Iceberg table from inside a Storm topology, without routing through Kafka Connect, Flink or Spark. Everything is in the PR: https://github.com/apache/storm/pull/8950 In short: - Append-only, plain data file appends, no deletes or upserts. At-least-once: tuples are acked only after the file containing them is committed. - The commit policy is split across two orthogonal dimensions. Space: writers roll on a configured target file size based on estimated bytes written, which is our defense against the small-files problem. Time: a Storm tick tuple scans the write-ahead log and commits the files that are eligible, bounding visibility latency for low-traffic tables. File granularity and freshness are therefore tuned independently. - The WAL also drives recovery: on restart we replay it, drop orphan files and re-attempt pending commits. - Catalog properties go verbatim to CatalogUtil.buildIcebergCatalog, partitioned tables use the fanout writer, and periodic table refresh means schema and spec evolution needs no restart. Alternatives obviously already exist and work well. The difference here is granularity: mapping, filtering and partition routing happen per tuple inside the topology, in the same bolt that owns the writers, so the operator controls which tuple lands in which data file. The module is merged on master and targeted for Storm 3.1.0, so it can be built and run today from a master checkout. Two things would help us most before the release. First, benchmarks: we would rather compare against the existing sinks with someone else's methodology than publish our own numbers, so if you have a harness you already trust, we would like to use it. Second, bug reports: please open issues. Cheers, Gianluca
