vingov commented on code in PR #5294:
URL: https://github.com/apache/hudi/pull/5294#discussion_r847848179
##########
hudi-common/src/main/java/org/apache/hudi/avro/HoodieAvroUtils.java:
##########
@@ -441,6 +441,17 @@ public static List<GenericRecord>
rewriteRecords(List<GenericRecord> records, Sc
return records.stream().map(r -> rewriteRecord(r,
newSchema)).collect(Collectors.toList());
}
+ /**
+ * Given an Avro record and list of columns to remove, this method removes
the list of columns from
+ * the given avro record using rewriteRecord method.
+ * <p>
+ * To better understand how it removes please check {@link
#rewriteRecord(GenericRecord, Schema)}
+ */
+ public static GenericRecord removeFields(GenericRecord record, List<String>
columnsToRemove) {
Review Comment:
The reason for not invoking `rewriteRecord` method directly from
DeltaStreamer code is because I was getting serialization error
`org.apache.spark.SparkException: Task not serializable` when passing the
`targetSchema` to that method from the map lambda expression.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
