numinnex opened a new pull request, #4759: URL: https://github.com/apache/iggy/pull/4759
Relates to #4232, #4236, #4728 An external data-loss test of the Iceberg sink lost rows whenever a write did not complete: 500 rows over ten random kills of the sink, 400 during a 30 s S3 outage, and 4,700 across one catalog restart. The runtime stored the offset of a batch when it read it, so a batch the sink failed to commit was gone. #4734 gave the runtime at-least-once delivery and #4740 made the sink retry its catalog commits. This lets the Iceberg sink opt in: with `[delivery] guarantee = "at_least_once"` the runtime stores an offset only after the batch is committed to every destination table, so a crash, a failed write or a catalog or object store outage delivers the batch again instead of losing it. At most once stays the default. Delivering again must not append rows twice. Each append now records in its snapshot summary the partition it came from and its last message: offset, message ID and the time the server stored it. When a batch arrives again and carries the last message of any append on the table's current snapshot history, the sink appends only the messages after the newest append. That covers a crash between the commit and the offset store, a backlog of several unflushed batches, and a batch that committed to one table and failed on another, so each table of a fan-out is appended once (point 3 of #4728). The ID and timestamp keep a recreated topic that reuses offsets from being mistaken for the old one, and walking only the current snapshot history keeps rows removed by a rollback from counting. Markers are written only at least once, so a replay that recovers rows lost while delivering at most once is not skipped. A batch that carries no appended message, for example after `batch_length` changed, cannot be proven and is appended in full, which the README states with the other limits. At least once, nothing is dropped without failing the call. A declared table that did not load at startup is loaded again by each batch instead of being left out until a restart: the batch fails transiently while the catalog is unreachable and for good when the table does not exist, and a declared name without a namespace makes the sink refuse at-least-once. Dynamic routing treats only a missing table or route value as unroutable. Any other failure to load a table, such as a 403 from the catalog, fails the call even with `invalid_rows = "skip"`, which remains an explicit choice to drop a message with a logged offset; the SDK doc of `enable_at_least_once` now says a configuration may opt into such skips. A rejected message now stops the call after the messages before it are committed, and the error names its offset. This covers a row that does not match the table schema, a payload with no JSON document, and a message dynamic routing cannot route. Storing that offset, as the runtime README describes for moving past a message, then gives up that message alone. Before, the whole call failed and the documented step lost the messages ahead of it. Some of this also changes behaviour at most once, each time losing fewer rows than before: - the prefix of a call before a rejected row is committed instead of failing the whole call; - a declared table that did not load at startup is loaded again by later batches; - a payload without JSON is skipped with `invalid_rows = "skip"` instead of failing the call; - a commit the catalog refuses for good, such as a denied write, is permanent instead of retried forever with new orphan files each time, while exhausted commit conflicts stay transient; - writer and reader setup failures that repeat on every attempt are permanent. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
