[
https://issues.apache.org/jira/browse/SPARK-40507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17613787#comment-17613787
]
Anil Dasari commented on SPARK-40507:
-------------------------------------
does anyone have an thoughts on this ?
> Spark creates an optional columns in hive table for fields that are not null
> ----------------------------------------------------------------------------
>
> Key: SPARK-40507
> URL: https://issues.apache.org/jira/browse/SPARK-40507
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.3.0
> Reporter: Anil Dasari
> Priority: Major
>
> Dataframe saveAsTable sets all columns as optional/nullable while creating
> the table here
> [https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala#L531]
> (`outputColumns.toStructType.asNullable`)
> This makes source parquet schema and hive table schema doesn't match and is
> problematic when large dataframe(s) process uses hive as temporary storage to
> avoid the memory pressure.
> Hive 3.x supports non null constraints on table columns. Please add support
> for non null constraints on Spark sql hive table.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
