numinnex opened a new issue, #4724: URL: https://github.com/apache/iggy/issues/4724
### Bug description A single row that does not fit the table schema fails its whole batch. `write_data_files` (`core/connectors/sinks/iceberg_sink/src/router/mod.rs`) decodes every JSON document of a batch with one arrow-json `Reader` built from the table schema. An error on any row, such as a type mismatch or a missing required field, fails the whole `consume` with `InvalidRecordValue` or `SchemaMismatch`. The runtime then drops the batch, because its offset was already stored on read (#4232). An external test sent 1 bad row per 1,000 and lost 11,880 good rows: 99 good rows per bad row at a batch size of 100. ### Affected area / component Connectors: `iggy_connector_iceberg_sink` ### Versions master @ 3c265dd89 ### Proposed fix When the batch decode fails, decode the rows one at a time. Write the rows that fit, and log the rejected ones with their offsets and the decode error. Make the behaviour a policy setting, for example `invalid_rows = "skip" | "fail"`, with `fail` keeping today's semantics. Sending rejected rows somewhere durable is part of #4235. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
