[
https://issues.apache.org/jira/browse/HUDI-774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gary Li updated HUDI-774:
-------------------------
Affects Version/s: 0.9.0
> Spark to Avro converter incorrectly generates optional fields
> -------------------------------------------------------------
>
> Key: HUDI-774
> URL: https://issues.apache.org/jira/browse/HUDI-774
> Project: Apache Hudi
> Issue Type: Bug
> Affects Versions: 0.9.0
> Reporter: Alexander Filipchik
> Priority: Major
> Labels: pull-request-available, sev:critical, user-support-issues
> Fix For: 0.8.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> I think https://issues.apache.org/jira/browse/SPARK-28008 is a good
> descriptions of what is happening.
>
> It can cause a situation when schema in the MOR log files is incompatible
> with the schema produced by RowBasedSchemaProvider, so compactions will stall.
>
> I have a fix which is a bit hacky -> postprocess schema produced by the
> converter and
> 1) Make sure unions with null types have those null types at position 0
> 2) They have default values set to null
> I couldn't find a way to do a clean fix as some classes that are problematic
> are from Hive and called from Spark.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
