numinnex opened a new pull request, #4739: URL: https://github.com/apache/iggy/pull/4739
Closes #4724 Relates to #4232, #4235 The Iceberg sink decodes a batch with one arrow-json reader built from the table schema, so a single row with a wrong type or a missing required field fails the whole `consume`. The runtime stores the offset on read, so every good row of that batch is lost with the bad one. A new `invalid_rows` plugin option picks what happens to such a row. `fail`, the default, keeps today's behaviour. With `skip`, the sink decodes the rows one at a time after the batch decode fails, writes the rows that match, and logs each skipped row as a warning with its offset and the decode error. A batch in which no row matches writes no data file and commits nothing. A batch without bad rows is decoded once, as before. Sending skipped rows somewhere durable stays with #4235. Unit tests cover both modes and the batch with only bad rows, and an integration test sends one bad row among four good ones with `skip` and checks that the snapshot added four records. The integration test fails when the fixture sets `fail`, and the unit test fails when `skip` behaves like `fail`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
