hudi-bot opened a new issue, #15441: URL: https://github.com/apache/hudi/issues/15441
I tried using MERGE INTO with UPDATE * and INSERT * statement with full schema evolution enabled. I noticed that during insert new columns from incoming batch (that do not exist in target table yet) are dropped and target schema is applied. No warnings nor failed writes. Therefore can we as users automatically evolve schema on MERGE INTO operations? I guess this should only be supported when we use update set * and insert * in merge operation. *Expected behavior* When incoming data is missing columns that already declared in target table these should be injected with default/null values. When incoming data has new columns that are not yet declared in the target table, these should be added to the target table. Case when incoming data has both missing columns and new columns, missing columns should be injected with null/ default values, new columns should be added to the target table. New columns should be reflected in metastore table schema. Should support complex types, and nested schemas. Currently similar thing is supported for dataframe writes if both schema reconciliation and schema evolution configs are enabled, see HUDI-4276. From user experience perspective it would be easier if I had _mergeSchema_ (as for parquet spark datasource) config to enable this feature for both spark sql and df write. Thread from dev mailing list as a reference: [https://lists.apache.org/thread/kr59hh7yqr2c1y33kzfv3n97h6ydbz9b] GH issue: [https://github.com/apache/hudi/issues/5899] ## JIRA info - Link: https://issues.apache.org/jira/browse/HUDI-4872 - Type: Improvement - Epic: https://issues.apache.org/jira/browse/HUDI-1297 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
