Balajee Nagasubramaniam created HUDI-1441:
---------------------------------------------
Summary: HoodieAvroUtils - rewrite() is not handling evolution of
a nested record field.
Key: HUDI-1441
URL: https://issues.apache.org/jira/browse/HUDI-1441
Project: Apache Hudi
Issue Type: Bug
Components: Common Core
Reporter: Balajee Nagasubramaniam
When a schema has nested record field and one of the fields of the nested
record evolves, then rewrite() results in SchemaCompatibilityException (or
ArrayIndexOutOfBoundsException).
{{/*
* OldRecord: NewRecord:
* field1 : String field1 : String
* field2 : record field2 : record
* field_21 : string field_21 : string
* field_22 : Integer field_22 : Integer
* field3: Integer field_23 : String
* field_24 : Integer
* field3: Integer
*
* When a nested record has changed/evolved, newRecord.put(field2,
oldRecord.get(field2)), is not sufficient.
* Requires a deep-copy/rewrite of the evolved field.
*/}}
Note 1: When reading the parquet file using the writer schema, this should not
be a problem, as new fields are substituted with null. When reading the
parquet using reader schema and writing to a new file using the writer schema,
this issue is manifested.
Note 2: Hudi test suite - upsertNode exercies this path. (fixed as a work
around in a separate task).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
