[ 
https://issues.apache.org/jira/browse/HUDI-774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17107473#comment-17107473
 ] 

Vinoth Chandar commented on HUDI-774:
-------------------------------------

yes [~uditme] is going to look at it as well 

> Spark to Avro converter incorrectly generates optional fields
> -------------------------------------------------------------
>
>                 Key: HUDI-774
>                 URL: https://issues.apache.org/jira/browse/HUDI-774
>             Project: Apache Hudi (incubating)
>          Issue Type: Bug
>            Reporter: Alexander Filipchik
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 0.6.0
>
>          Time Spent: 10m
>  Remaining Estimate: 0h
>
> I think https://issues.apache.org/jira/browse/SPARK-28008 is a good 
> descriptions of what is happening.
>  
> It can cause a situation when schema in the MOR log files is incompatible 
> with the schema produced by RowBasedSchemaProvider, so compactions will stall.
>  
> I have a fix which is a bit hacky -> postprocess schema produced by the 
> converter and
> 1) Make sure unions with null types have those null types at position 0
> 2) They have default values set to null
> I couldn't find a way to do a clean fix as some classes that are problematic 
> are from Hive and called from Spark.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Reply via email to