[
https://issues.apache.org/jira/browse/HUDI-2023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17370692#comment-17370692
]
Sagar Sumit commented on HUDI-2023:
-----------------------------------
Validated with delta streamer and the results are summarized as below:
|| ||COW||MOR||
|Adding a new nullable column at root level at the end|succeeds|succeeds|
|Adding a new nullable column to inner struct (at the end)|succeeds|succeeds|
|Adding a new non-nullable column at root level at the end|fails|fails|
|Adding a new non-nullable column to inner struct (at the end)|fails|fails |
The failure after adding a new non-nullable column in case of MOR is:
{code:java}
Caused by: org.apache.hudi.exception.HoodieUpsertException: Failed to merge old
record into new file for key impression_598 from old file
file:/tmp/hudi-deltastreamer-op/impressions_mor/user_86/08046c02-14e3-4629-899a-614518dfc545-0_53-6-148_20210628211956.parquet
to new file
file:/tmp/hudi-deltastreamer-op/impressions_mor/user_86/08046c02-14e3-4629-899a-614518dfc545-0_8-22-301_20210628212147.parquet
...
at org.apache.hudi.io.HoodieMergeHandle.write(HoodieMergeHandle.java:320) at
org.apache.hudi.table.action.commit.AbstractMergeHelper$UpdateHandler.consumeOneRecord(AbstractMergeHelper.java:122)
at
org.apache.hudi.table.action.commit.AbstractMergeHelper$UpdateHandler.consumeOneRecord(AbstractMergeHelper.java:112)
at
org.apache.hudi.common.util.queue.BoundedInMemoryQueueConsumer.consume(BoundedInMemoryQueueConsumer.java:37)
at
org.apache.hudi.common.util.queue.BoundedInMemoryExecutor.lambda$null$2(BoundedInMemoryExecutor.java:121)
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 more Caused
by: java.lang.RuntimeException: Null-value for required field: evolvedField at
org.apache.parquet.avro.AvroWriteSupport.writeRecordFields(AvroWriteSupport.java:194)
at org.apache.parquet.avro.AvroWriteSupport.write(AvroWriteSupport.java:165)
at
org.apache.parquet.hadoop.InternalParquetRecordWriter.write(InternalParquetRecordWriter.java:128)
at org.apache.parquet.hadoop.ParquetWriter.write(ParquetWriter.java:299) at
org.apache.hudi.io.storage.HoodieParquetWriter.writeAvro(HoodieParquetWriter.java:89)
at org.apache.hudi.io.HoodieMergeHandle.write(HoodieMergeHandle.java:315)
{code}
> Validate Schema evolution in hudi
> ---------------------------------
>
> Key: HUDI-2023
> URL: https://issues.apache.org/jira/browse/HUDI-2023
> Project: Apache Hudi
> Issue Type: Test
> Components: Testing
> Reporter: sivabalan narayanan
> Assignee: sivabalan narayanan
> Priority: Major
>
> Test schema evolution in hudi and document the same
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
