raghusrhyme commented on code in PR #18914:
URL: https://github.com/apache/hudi/pull/18914#discussion_r3599467381
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/transform/ErrorTableAwareChainedTransformer.java:
##########
@@ -55,8 +55,9 @@ public Dataset<Row> apply(JavaSparkContext jsc, SparkSession
sparkSession, Datas
for (TransformerInfo transformerInfo : transformers) {
Transformer transformer = transformerInfo.getTransformer();
dataset = transformer.apply(jsc, sparkSession, dataset,
transformerInfo.getProperties(properties, transformers));
- // validate in every stage to ensure ErrorRecordColumn not dropped by
one of the transformer and added by next transformer.
- ErrorTableUtils.validate(dataset);
+ // Re-inject _corrupt_record if the transformer dropped it (e.g. custom
JAR transformers
+ // that do column projection like ColumnFilter with mode=include).
+ dataset =
ErrorTableUtils.addNullValueErrorTableCorruptRecordColumn(dataset);
Review Comment:
Made changes, Testing them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
