alexeykudinkin commented on code in PR #5376:
URL: https://github.com/apache/hudi/pull/5376#discussion_r854359454
##########
hudi-common/src/main/java/org/apache/hudi/avro/HoodieAvroUtils.java:
##########
@@ -743,7 +757,20 @@ private static Object rewriteRecordWithNewSchema(Object
oldRecord, Schema oldSch
Schema.Field field = fields.get(i);
if (oldSchema.getField(field.name()) != null) {
Schema.Field oldField = oldSchema.getField(field.name());
- helper.put(i,
rewriteRecordWithNewSchema(indexedRecord.get(oldField.pos()),
oldField.schema(), fields.get(i).schema()));
+ helper.put(i,
rewriteRecordWithNewSchema(indexedRecord.get(oldField.pos()),
oldField.schema(), fields.get(i).schema(), renameCols));
+ }
+ // deal with rename
+ if (!renameCols.isEmpty() && oldSchema.getField(field.name()) ==
null) {
+ String fieldName = field.name();
+ for (Map.Entry<String, String> entry : renameCols.entrySet()) {
+ List<String> nameParts =
Arrays.asList(entry.getKey().split("\\."));
Review Comment:
If we're accepting the dot-path specification, why are we looking at the
last element of the chain?
Since we're doing top-down traversal, we should look at the first and make
sure we're also need to either to keep an index at what level we are, or trim
the column names as we traverse
##########
hudi-common/src/main/java/org/apache/hudi/avro/HoodieAvroUtils.java:
##########
@@ -431,6 +431,18 @@ public static GenericRecord
rewriteRecordWithMetadata(GenericRecord genericRecor
return newRecord;
}
+ // TODO Unify the logical of rewriteRecordWithMetadata and
rewriteEvolutionRecordWithMetadata, and delete this function.
+ public static GenericRecord rewriteEvolutionRecordWithMetadata(GenericRecord
genericRecord, Schema newSchema, String fileName) {
+ GenericRecord newRecord =
HoodieAvroUtils.rewriteRecordWithNewSchema(genericRecord, newSchema, new
HashMap<>());
+ // do not preserve FILENAME_METADATA_FIELD
Review Comment:
Records should be immutable by default, with only limited scopes where
treating them as mutable is acceptable
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieWriteHandle.java:
##########
@@ -98,6 +101,8 @@
protected final String fileId;
protected final String writeToken;
protected final TaskContextSupplier taskContextSupplier;
+ // For full schema evolution
+ protected final boolean schemaOnReadEnable;
Review Comment:
nit: better to suffix it w/ `Enabled`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
