Raghuramchadalawada commented on PR #20336: URL: https://github.com/apache/hudi/pull/20336#issuecomment-6206885358
Reworked the approach based on the review here and on #18914, and squashed to one commit. Positional restore (zip) of `_corrupt_record` is dropped entirely. Any transformer that filters, reorders or repartitions can misalign it silently, and a misaligned or null-filled column makes `processErrorEvents` write error records to the target table. A row key doesn't help either, since a projecting transformer drops it too. So now: - dropped while all null → re-added as null (the case in #18915); - dropped while holding values → `HoodieValidationException` naming the transformer, same as before this PR. On the open review comments: - Null fallback sending error rows to the main table: there's no fallback any more; a dropped column with values fails loudly. - Misaligned zip after `orderBy`/`repartition`: the zip is removed. - Per-transformer `localCheckpoint` cost and executor-loss risk: the checkpoint is removed. The only extra work is one `isEmpty()` job, and only after a transformer drops the column following one that kept it. - Internal ticket reference in a test comment: removed. - Loop readability (latest review): the flag is now set in one place with no `continue`, and the dropped-column check is in `validateNoErrorRecordsDropped`, pushed as a separate commit. No behaviour change. - From #18914: the lazy-zip and stash-materialization comments no longer apply, and @wombatu-kun's point is covered by the existing mark-then-drop test, which still expects the exception. `TestErrorTableAwareChainedTransformer` passes (14/14); the 4 new tests fail on master. `TestStreamSync` passes. I also ran the chain's output through `SourceFormatAdapter.processErrorEvents` (as `StreamSync` does) with `SqlQueryBasedTransformer`: a `SELECT` without `_corrupt_record` now writes every row and sends nothing to the error table, rows marked before or after a projection go to the error table, and marking followed by `WHERE` plus a projection fails without writing anything. Could a committer please approve the CI workflows for this PR and close #18914 in favour of this one? Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
