lokeshj1703 commented on code in PR #8574:
URL: https://github.com/apache/hudi/pull/8574#discussion_r1192120083
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/transform/ChainedTransformer.java:
##########
@@ -93,9 +103,13 @@ public List<String> getTransformersNames() {
@Override
public Dataset<Row> apply(JavaSparkContext jsc, SparkSession sparkSession,
Dataset<Row> rowDataset, TypedProperties properties) {
Dataset<Row> dataset = rowDataset;
+ Option<Schema> incomingSchemaOpt = sourceSchemaOpt;
for (TransformerInfo transformerInfo : transformers) {
Transformer transformer = transformerInfo.getTransformer();
dataset = transformer.apply(jsc, sparkSession, dataset,
transformerInfo.getProperties(properties));
+ if (enableSchemaValidation) {
+ incomingSchemaOpt = validateAndGetTransformedSchema(transformerInfo,
dataset, incomingSchemaOpt, jsc, sparkSession, properties);
Review Comment:
This is due to chained transformer. Transformed schema of 1st transformer is
input schema for 2nd transformer. Added a comment for it.
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/transform/ChainedTransformer.java:
##########
@@ -59,11 +65,15 @@ public ChainedTransformer(List<Transformer>
transformersList) {
* Creates a chained transformer using the input transformer class names.
Refer {@link HoodieDeltaStreamer.Config#transformerClassNames}
* for more information on how the transformers can be configured.
*
- * @param configuredTransformers List of configured transformer class names.
- * @param ignore Added for avoiding two methods with same erasure. Ignored.
+ * @param sourceSchemaOpt Source Schema
Review Comment:
Addressed
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/transform/ChainedTransformer.java:
##########
@@ -47,6 +51,8 @@ public class ChainedTransformer implements Transformer {
private static final String ID_TRANSFORMER_CLASS_NAME_DELIMITER = ":";
private final List<TransformerInfo> transformers;
+ private Option<Schema> sourceSchemaOpt = Option.empty();
Review Comment:
Addressed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
