Thejdeep Gudivada created SPARK-38651:
-----------------------------------------
Summary: Writing out empty or nested empty schemas in Datasource
should be configurable
Key: SPARK-38651
URL: https://issues.apache.org/jira/browse/SPARK-38651
Project: Spark
Issue Type: Task
Components: SQL
Affects Versions: 2.4.0
Reporter: Thejdeep Gudivada
In SPARK-23372, we introduced a backwards incompatible change that would remove
support for writing out empty or nested empty schemas in file based
datasources. This introduces backward incompatibility for users who have been
using a schema that met the above condition since the datasource supported it.
Except for Parquet and text, other file based sources support this behavior.
We should either :
* Make it configurable to enable/disable writing out empty schemas
* Enable the validation check only for sources that do not support it -
Parquet / Text
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
