nada-attia opened a new issue, #19654: URL: https://github.com/apache/hudi/issues/19654
### Describe the problem you faced When trying to update existing rows in a table to add new columns, the new columns don't get properly reconciled into the schema written to commit metadata without SCHEMA_EVOLUTION_ENABLE=true. ### To Reproduce This issue was exposed by a test case. **What the test case does:** Generates a sample DataFrame and writes it to a fresh Hudi table (SaveMode.Append), then runs a SQL query assertion expecting 2 rows. Reads back the table schema via TableSchemaResolver and asserts it has 11 fields (the original schema). Builds an "updated" DataFrame from the same input, adding a new columns with complex nested type: phone. Sets SET_NULL_FOR_MISSING_COLUMNS = false and writes the updated DataFrame, expecting this to fail (old records can't be reconciled against the new schema without nullable backfill) — the test asserts an exception is thrown. Sets SET_NULL_FOR_MISSING_COLUMNS = true and re-writes the same updated DataFrame, expecting this to succeed this time. Re-runs the SQL query assertion (still expecting 2 rows — same records, evolved schema). Reloads the table's active timeline and re-reads the schema, asserting it now has 12 fields (11 original + phone) **What happened:** The "add columns" step wrote the DataFrame with the new phone/address.country columns using only SET_NULL_FOR_MISSING_COLUMNS = true. SCHEMA_EVOLUTION_ENABLE (hoodie.schema.on.read.enable) was left at its default (false). Without InternalSchema reconciliation active, the merge handle fell back to plain Avro schema resolution for backfilling old records against the new schema, and the write didn't actually persist the new column into the table's Avro schema (the phone field never got added). The "verify evolved state" step then read back the schema and asserted 12 fields — but only got 11 (the original count, unchanged), causing the assertion failure above. ### Expected behavior Schema should get evolved implicitly ### Environment Description * Hudi version: * Spark version: * Flink version: * Hive version: * Hadoop version: * Storage (HDFS/S3/GCS..): * Running on Docker? (yes/no): ### Additional context _No response_ ### Stacktrace ```shell ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
