yihua opened a new issue, #19942: URL: https://github.com/apache/hudi/issues/19942
### Problem `ErrorTableAwareChainedTransformer` injects the error-table corrupt-record column (`_corrupt_record`) once, before the transformer chain runs, and then calls `ErrorTableUtils.validate(dataset)` after every transformer. That validate throws `HoodieValidationException` when the column is gone. Any transformer that projects a subset of columns removes the column, and the whole sync dies: ``` HoodieValidationException: Invalid condition, columnName=_corrupt_record is not present in transformer output schema ``` This is not an exotic case. A custom JAR transformer that does `dataset.select(userColumns)` hits it, and so does a column filter configured with an include list. The user has done nothing wrong: they asked for a set of columns, and Hudi's own bookkeeping column is not one they know to ask for. The validate is also self-defeating. The chain already knows how to add the column (`ErrorTableUtils.addNullValueErrorTableCorruptRecordColumn`), so it can restore what a transformer dropped instead of aborting. The existing test `TestErrorTableAwareChainedTransformer.testForErrorRecordColumn` pins the abort as intended behaviour, which is what this issue proposes to change. ### Proposed fix Re-inject the column after each transformer rather than validating and throwing, and delete `ErrorTableUtils.validate` (it has no other caller). A transformer that keeps the column sees a no-op; one that drops it gets it back with null values, which is the same state the chain establishes at the start. ### Environment Reproduces on master with `hoodie.errortable.enable=true` and any column-projecting transformer in `hoodie.streamer.transformer.class`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
