[
https://issues.apache.org/jira/browse/HUDI-774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17411499#comment-17411499
]
ASF GitHub Bot commented on HUDI-774:
-------------------------------------
hudi-bot commented on pull request #1514:
URL: https://github.com/apache/hudi/pull/1514#issuecomment-914631306
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "3baebd032231974a7e7d9410b5bfeb879c9790b1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3baebd032231974a7e7d9410b5bfeb879c9790b1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3baebd032231974a7e7d9410b5bfeb879c9790b1 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
> Spark to Avro converter incorrectly generates optional fields
> -------------------------------------------------------------
>
> Key: HUDI-774
> URL: https://issues.apache.org/jira/browse/HUDI-774
> Project: Apache Hudi
> Issue Type: Bug
> Affects Versions: 0.9.0
> Reporter: Alexander Filipchik
> Priority: Major
> Labels: pull-request-available, sev:critical, user-support-issues
> Fix For: 0.9.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> I think https://issues.apache.org/jira/browse/SPARK-28008 is a good
> descriptions of what is happening.
>
> It can cause a situation when schema in the MOR log files is incompatible
> with the schema produced by RowBasedSchemaProvider, so compactions will stall.
>
> I have a fix which is a bit hacky -> postprocess schema produced by the
> converter and
> 1) Make sure unions with null types have those null types at position 0
> 2) They have default values set to null
> I couldn't find a way to do a clean fix as some classes that are problematic
> are from Hive and called from Spark.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
