Ziya Mukhtarov created SPARK-54220:
--------------------------------------
Summary: NullType/VOID/UNKNOWN Type Support in Parquet
Key: SPARK-54220
URL: https://issues.apache.org/jira/browse/SPARK-54220
Project: Spark
Issue Type: New Feature
Components: SQL
Affects Versions: 4.0.1
Reporter: Ziya Mukhtarov
Currently, Spark does not support NullType columns in Parquet files. This is
blocked at the ParquetFileFormat level, which throws an exception if there's an
attempt to read or write NullType column. Parquet has [UNKNOWN logical type
annotation|https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#unknown-always-null],
which can be used for columns whose values are always null. This can be used
to write NullType columns into Parquet files, and when reading from Parquet
files that already has this annotation.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
