Michael Armbrust created SPARK-4523:
---------------------------------------
Summary: Improve handling of serialized schema information
Key: SPARK-4523
URL: https://issues.apache.org/jira/browse/SPARK-4523
Project: Spark
Issue Type: Bug
Components: SQL
Reporter: Michael Armbrust
Priority: Blocker
There are several issues with our current handling of metadata serialization,
which is especially troublesome since this is the only place that we persist
information directly using Spark SQL. Moving forward we should do the
following:
- Relax the parsing so that it does not fail when optional fields are missing
(i.e. containsNull or metadata)
- Include a regression suite that attempts to read old parquet files written
by previous versions of Spark SQL.
- Provide better warning messages when various forms of parsing fail (I think
that it is silent right now which makes tracking down bugs more difficult than
it needs to be).
- Deprecate (display a warning) when reading data with the old case class
schema representation and eventually remove it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
