deepakpanda93 commented on issue #10309: URL: https://github.com/apache/hudi/issues/10309#issuecomment-4845921621
This is addressed in Hudi 1.x. There were two separate problems in this thread: - The write-side error (cannot update origin type double to long) reported by @lei-su-awx was fixed in 0.14.1 (confirmed above). - The OP's read-side failure — reading old parquet files physically stored as INT64 after evolving the column to double (SchemaColumnConvertNotSupportedException: Expected double, Found INT64) — happened because on 0.14/0.15 the read went through Spark's native vectorized parquet reader, which doesn't promote INT64→double. In 1.x this is handled by the engine-agnostic FileGroup reader (default since 1.0.0, hoodie.file.group.reader.enabled=true), which adds schema-evolution support and disables the vectorized reader for unsupported type cases, with numeric type-promotion fixes: - HUDI-7567 [#10957](https://github.com/apache/hudi/pull/10957) (schema evolution in the filegroup reader, 1.0.0) - HUDI-7068 [#10043](https://github.com/apache/hudi/pull/10043) (disable vectorized read when schema not supported, 1.0.0) - HUDI-8902 [#13188](https://github.com/apache/hudi/pull/13188) (float→double read correctness, 1.0.2) - HUDI-9705 [#13714](https://github.com/apache/hudi/pull/13714) (type-promotion/field-rename reader fixes, 1.1.0) long → double is a valid promotion, so reading the old long-typed files back as double works on 1.x. Your 0.15.0 test predates the FileGroup reader becoming the default, which is why it still failed there. Please retest on a recent 1.1.x release (with hoodie.schema.on.read.enable=true and the file group reader default). Closing as fixed in 1.x and if you still hit Expected double, Found INT64 on 1.1.x, please reopen with a minimal repro. Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
