yihua opened a new issue, #20498: URL: https://github.com/apache/hudi/issues/20498
### Bug Description **What happened:** Found by code reading while reviewing #20483. Not reproduced yet. Three union shapes get a different type depending on which component converts the schema: | Union | Avro parquet writer | Spark schema converter | Flink schema converter | Meta sync (after #20483) | |---|---|---|---|---| | `["string"]` (single branch, no null) | group with `member0` | `string` | `string` | `struct<member0:string>` | | `["int","long"]` | group with `member0`, `member1` | `long` | RAW | `struct<member0:int,member1:bigint>` | | `["float","double"]` | group with `member0`, `member1` | `double` | RAW | `struct<member0:float,member1:double>` | - **Avro parquet writer**: [`AvroSchemaConverterWithTimestampNTZ#convertUnion`](https://github.com/apache/hudi/blob/c0ed52483fb22e57c57a9c7b211b9bcb4919cf5b/hudi-hadoop-common/src/main/java/org/apache/parquet/avro/AvroSchemaConverterWithTimestampNTZ.java#L304-L341) writes every union other than `[null, T]` as a group of optional `member<N>` fields, including a single-branch `[T]`. This matches parquet-avro. - **Spark**: [`HoodieSparkSchemaConverters`](https://github.com/apache/hudi/blob/c0ed52483fb22e57c57a9c7b211b9bcb4919cf5b/hudi-client/hudi-spark-client/src/main/scala/org/apache/spark/sql/avro/HoodieSparkSchemaConverters.scala#L380-L396) maps `[T]` to `T`, `[int,long]` to `long` and `[float,double]` to `double`, following spark-avro. The Spark record type writer builds its parquet schema from that `StructType` ([`HoodieRowParquetWriteSupport#convert`](https://github.com/apache/hudi/blob/c0ed52483fb22e57c57a9c7b211b9bcb4919cf5b/hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/io/storage/row/HoodieRowParquetWriteSupport.java#L763-L773)), so it writes these columns as plain primitives. - **Flink**: [`HoodieSchemaConverter#convertUnion`](https://github.com/apache/hudi/blob/c0ed52483fb22e57c57a9c7b211b9bcb4919cf5b/hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/util/HoodieSchemaConverter.java#L629-L632) maps `[T]` to `T`. So for these shapes the parquet layout depends on the record type used for the write, and the read schema from the engine converters matches only the Spark record type layout. A file written by the Avro path has a group where the Spark reader requests a primitive. Meta sync after #20483 follows the Avro writer layout, which is also what releases before #14344 registered. **What you expected:** One mapping for these union shapes across the parquet writers, the engine schema converters and meta sync, so a column has the same type no matter which writer produced the file. **Steps to reproduce (not verified yet):** 1. Write a table with the Avro record type (for example HoodieStreamer with an Avro schema provider) whose schema has a field of type `["int","long"]` or `["string"]`. 2. Read it with Spark through the Hudi datasource. 3. Expect the parquet read to fail, since the requested schema has a primitive where the file has a group. ### Environment **Hudi version:** master (1.3.0-SNAPSHOT, c0ed52483fb2) **Query engine:** Spark, Flink **Relevant configs:** record type used on write (Avro vs Spark) ### Logs and Stack Trace None yet; found by code reading. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
