Hi @Gianluca Graziadei ! I've never used Iceberg (nor Trident for that matter), so my input here is very limited.
Properties-only would still result in many small files on the storage, correct? And would slightly increase pressure on the transport layer. So does not seem quite viable. Default batching and the options with withCommitIntervalBytes, documented, seem a good first iteration. On Sun, 26 Jul 2026 at 11:45, Gianluca Graziadei <[email protected]> wrote: > > Hi folks, > I'd like to propose a new external module, *storm-iceberg*, and get the > community's thoughts before/while pushing it as a PR. > > Today the common way to land Storm data into a table format like Iceberg is > indirect: Kafka plus a separate ingestion job (Spark Structured Streaming, > Kafka Connect, etc.). That's a fine default for most streaming-to-lakehouse > pipelines, but it adds a hop and gives up something Storm is unusually good > at: tuple-at-a-time processing with per-record acking and backpressure. For > use cases that need real, sub-second control over what has and hasn't been > durably persisted, not "eventually, in the next micro-batch", Storm can > write straight to Iceberg, with nothing in between. > > What > - A Trident State (*IcebergState / IcebergStateFactory / > IcebergStateUpdater*) that appends TridentTuples to an Iceberg table. > - Exactly-once via the same txid-marker pattern HdfsState already uses: > each commit is a single Iceberg transaction that atomically appends the > batch's data files and records the transaction id as a table property. A > replayed batch is detected and skipped. > - Works with any Iceberg catalog (Hadoop, Hive, REST, Glue, Nessie, ...), > catalog properties are passed verbatim to CatalogUtil.buildIcebergCatalog. > - Partitioned tables via Iceberg's PartitionedFanoutWriter. > - Optional commit batching: by default every Trident batch is one Iceberg > commit and one snapshot, which doesn't scale for high-rate, small-batch > streams, thousands of snapshots, one tiny file per batch. > *withCommitIntervalBytes > / withCommitIntervalMillis* buffer several batches and commit them together > once a size or time threshold is crossed, so files grow towards the table's > target size and the snapshot count stays sane. > > Buffering weakens the delivery guarantee, and I don't think it can be made > not to. Trident considers a batch delivered the moment commit() returns, so > a batch buffered towards the next threshold is already acknowledged, if the > worker dies before the flush, those batches are lost, not duplicated. It's > opt-in and off by default, documented as such, but I'd welcome opinions on > whether that's the right trade-off to offer at all, or whether there's a > design that keeps exactly-once while still batching commits. (I looked at a > properties-only commit per batch plus a single deferred append, keeps > exactly-once, still costs one catalog round-trip per batch. Happy to > discuss both.) > > Draft PR: https://github.com/apache/storm/pull/8950 > > Feedback, questions, and contributors are all very welcome. > > Cheers, > Gianluca
