rangareddy commented on issue #3558: URL: https://github.com/apache/hudi/issues/3558#issuecomment-5101243253
This issue was reviewed as part of the JIRA-migrated backlog triage. Findings: This is fixed. Type promotion from `int` to `double` across a subset of partitions now works, including for partitions that were never rewritten. Verified by re-running the exact scenario from this thread -- batch 1 writes an INT column into `part_0` and `part_3`, batch 2 upserts the *same* column as DOUBLE into `part_0` and `part_9` leaving `part_3` untouched, then the whole table is read back (COW, hive-style partitioning, `hoodie.schema.on.read.enable` **not** set, Spark 3.4.4): | Hudi | write INT | upsert DOUBLE | read (vectorized) | read (non-vectorized) | values | | --- | --- | --- | --- | --- | --- | | 0.14.0 | OK | OK | OK, dtype `double` | OK, dtype `double` | correct | | 1.2.0 | OK | OK | OK, dtype `double` | OK, dtype `double` | correct | `part_3` -- the partition that was never rewritten and whose base file is therefore still physically INT -- reads back correctly as `30.0` / `31.0` on both versions. That is the case that produced the original `UnsupportedOperationException: PlainIntegerDictionary ... decodeToDouble`, so it is the one that matters. The relevant fix is #7480 (`[HUDI-5400] Fix read issues when Hudi-FULL schema evolution is not enabled`, merged 2022-12-24, first released in **0.13.0**) -- note it targets exactly the default path, with full schema evolution *not* enabled, which is how the test above is configured. The mechanism is `HoodieParquetFileFormatHelper.buildImplicitSchemaChangeInfo` (`hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/HoodieParquetFileFormatHelper.scala:26` at `release-0.14.0`): for each parquet file it converts the file's *physical* schema, diffs it against the required schema, records the `(requiredType, fileType)` pair, and reads the column at its physical type so the reader can cast it up. That is what makes a physically-INT base file readable under a DOUBLE table schema. One note for anyone tracking this: JIRA `HUDI-3019` is still marked **Open** with `fixVersion 1.1.0`. Based on the runs above that is stale JIRA bookkeeping rather than a live bug, and the JIRA should be resolved. Closing as fixed. If you still hit a type-promotion failure on a recent version, please reopen with the Hudi version, the before/after schemas and which partitions were rewritten. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
