umehrot2 commented on a change in pull request #1427: [HUDI-727]: Copy default
values of fields if not present when rewriting incoming record with new schema
URL: https://github.com/apache/incubator-hudi/pull/1427#discussion_r397037244
##########
File path:
hudi-common/src/test/java/org/apache/hudi/common/util/TestHoodieAvroUtils.java
##########
@@ -57,4 +60,16 @@ public void testPropsPresent() {
}
Assert.assertTrue("column pii_col doesn't show up", piiPresent);
}
+
+ @Test
+ public void testDefaultValue() {
+ GenericRecord rec = new GenericData.Record(new
Schema.Parser().parse(EXAMPLE_SCHEMA));
+ rec.put("_row_key", "key1");
+ rec.put("non_pii_col", "val1");
+ rec.put("pii_col", "val2");
+ rec.put("timestamp", 3.5);
Review comment:
Can you help me understand how you are running into this issue with default
values ?
Based on my understanding, conversion to avro is internal to Hudi and a
custom avro schema (with default values) is not something that user can
themselves pass. And how `spark-avro` converts `struct schema to avro` there is
no special handling there from `default value` perspective. So I guess I am not
sure whether this is an issue in the first place.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
