[ https://issues.apache.org/jira/browse/SPARK-27838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Apache Spark reassigned SPARK-27838: ------------------------------------ Assignee: (was: Apache Spark) > Support user provided non-nullable avro schema for nullable catalyst schema > without any null record > --------------------------------------------------------------------------------------------------- > > Key: SPARK-27838 > URL: https://issues.apache.org/jira/browse/SPARK-27838 > Project: Spark > Issue Type: New Feature > Components: SQL > Affects Versions: 2.4.3 > Reporter: DB Tsai > Priority: Major > Fix For: 3.0.0 > > > When the data is read from the sources, the catalyst schema is always > nullable. Since Avro uses Union type to represent nullable, when any > non-nullable avro file is read and then written out, the schema will always > be changed. This PR provides a solution for users to keep the Avro schema > without being forced to use Union type. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org