[ https://issues.apache.org/jira/browse/HUDI-774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17107473#comment-17107473 ]
Vinoth Chandar commented on HUDI-774: ------------------------------------- yes [~uditme] is going to look at it as well > Spark to Avro converter incorrectly generates optional fields > ------------------------------------------------------------- > > Key: HUDI-774 > URL: https://issues.apache.org/jira/browse/HUDI-774 > Project: Apache Hudi (incubating) > Issue Type: Bug > Reporter: Alexander Filipchik > Priority: Major > Labels: pull-request-available > Fix For: 0.6.0 > > Time Spent: 10m > Remaining Estimate: 0h > > I think https://issues.apache.org/jira/browse/SPARK-28008 is a good > descriptions of what is happening. > > It can cause a situation when schema in the MOR log files is incompatible > with the schema produced by RowBasedSchemaProvider, so compactions will stall. > > I have a fix which is a bit hacky -> postprocess schema produced by the > converter and > 1) Make sure unions with null types have those null types at position 0 > 2) They have default values set to null > I couldn't find a way to do a clean fix as some classes that are problematic > are from Hive and called from Spark. -- This message was sent by Atlassian Jira (v8.3.4#803005)