numinnex opened a new pull request, #4740: URL: https://github.com/apache/iggy/pull/4740
Closes #4723 Relates to #4232, #4236 The Iceberg sink committed each append once. iceberg-rust retries only HTTP 409 conflicts, so a catalog that was unreachable, restarting or answering with a 5xx failed the commit at once, and the runtime had already stored the offset of the batch. A single catalog restart in an external test lost 4,700 rows. The sink now retries the commit when the catalog cannot be reached or answers with an unexpected status, with a capped exponential backoff set by three new plugin options: `max_retries` (total attempts, default 5), `retry_delay` (default `1s`) and `max_retry_delay` (default `30s`). The data files are written once, and only the commit is repeated. A commit that the catalog refuses for good, such as one to a missing table, fails at once as before. The default of 5 attempts is above the usual 3 because a retry only repeats one catalog request, and its waits of about 15 seconds cover a catalog restart. A failed commit may still have been applied, for example when a gateway answers 502 or 504 while the catalog keeps working. Each append therefore carries an `iggy.commit-id` snapshot summary property, and before every retry the sink reloads the table and stops if a snapshot with that id is already there. A request that the catalog applies only after that check can still be appended twice, which the README states. The same marker can identify a batch that the runtime delivers again once #4232 lands, for #4236. Fan-out to several tables is still not atomic. Unit tests drive the commit against a memory catalog that fails on demand: an unreachable catalog is retried until the commit lands, a lost response after an applied commit adds no second snapshot, a refused commit is not retried, and the attempts are bounded. An integration test kills the REST catalog container, sends a batch, waits until the sink has read it, and starts the catalog again, after which all 7 rows are in the table. The REST catalog fixture now maps a reserved host port, which a container restart keeps. Disabling the marker check, treating no error as transient, or treating every error as transient each fails a unit test, and the integration test fails with `max_retries = 1`. Stacked on #4739. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
