prashantwason commented on PR #20483: URL: https://github.com/apache/hudi/pull/20483#issuecomment-6638905457
@voonhous thanks for the review. Both points are addressed in 6416e5011170. **E2E:** I added `TestHiveSyncTool#testSyncWithUnionFields`, parameterized over the sync modes. It writes real parquet with a nullable union, a union with a record member, and an array of unions, then checks the synced Hive columns and the `spark.sql.sources.schema` property. It then adds a commit with the same schema, resyncs, and asserts that both schemas are unchanged. As you suggested, it leaves out the single-branch union because the random data fixture can't generate one; `TestHiveSchemaUtil` still covers that case. **UUID:** you're right. `HiveSchemaUtil` has mapped `UUID` to `binary` since #14344. By default (`parquet.avro.write-parquet-uuid=false`) the parquet writer stores a UUID as `binary (STRING)`, which the parquet-schema-based sync registered as `string`. It now syncs as `string` again, which also matches `SparkSchemaUtils`. I updated the expectation in `TestHiveSchemaUtil#testSchemaTypeConversion` to match. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
