deepakpanda93 commented on issue #11803: URL: https://github.com/apache/hudi/issues/11803#issuecomment-4856532908
@kiwi-chen Here's why: ALTER COLUMN … TYPE (with hoodie.schema.on.read.enable=true) commits an empty schema-only commit — it records the new type (long) in the timeline but does not rewrite the existing data files, which still physically store the old type (int). On select *, Hudi must promote those old int files to long, but on 0.15.0 that read uses Spark's vectorized parquet reader, which can't do that physical-type promotion and throws the NullPointerException at WritableColumnVector.arrayData. Running an INSERT afterward writes a data commit (files in the new schema) and re-triggers schema resolution, so the next read avoids the broken vectorized path — that's why it appears to "fix" it, but it's really a side effect. Two ways to avoid needing the insert: - On 0.15.0: set spark.sql.parquet.enableVectorizedReader=false for the read after a type change — this forces the non-vectorized reader, which applies the schema-on-read type promotion correctly. - Upgrade to 1.x: the engine-agnostic FileGroup reader (default since 1.0.0) automatically falls back from the vectorized reader when a type change isn't vectorization-supported and applies the promotion (HUDI-7068 [#10043](https://github.com/apache/hudi/pull/10043); vectorized-read schema-evolution fix HUDI-8803 [#12560](https://github.com/apache/hudi/pull/12560), 1.0.1). select * right after ALTER COLUMN works there without the insert. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
