yihua commented on code in PR #20354:
URL: https://github.com/apache/hudi/pull/20354#discussion_r4288710011
##########
hudi-sync/hudi-hive-sync/src/test/java/org/apache/hudi/hive/util/TestHiveSchemaUtil.java:
##########
@@ -198,4 +198,55 @@ void testSchemaTypeConversion() throws IOException {
"STRUCT< `title` : string, `variant_data` : STRUCT< `metadata` :
binary, `value` : binary>>");
assertEquals(expected, actual);
}
+
+ /**
+ * Fields of the Avro primitive type "null" (not a nullable union) are valid
Avro but carry no data.
+ * They are not written to parquet and have no Hive equivalent, so Hive sync
must skip them,
+ * at the top level as well as inside nested records.
+ */
+ @Test
+ void testNullTypeFieldsAreSkipped() throws IOException {
Review Comment:
non-blocking: It might be worth one `TestHiveSyncTool` case that writes a
table whose schema has a `"null"` type field and runs `syncHoodieTable` end to
end, since that's the path in the stack trace (create table, then
`syncProperties`). The util tests pin the converters, but an e2e case would
also cover the hand-off through `SparkDataSourceTableUtils`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
