Chris Wood created SPARK-3833:
---------------------------------
Summary: Allow Spark SQL SchemaRDDs to me berged
Key: SPARK-3833
URL: https://issues.apache.org/jira/browse/SPARK-3833
Project: Spark
Issue Type: Wish
Components: SQL
Reporter: Chris Wood
We have JSON flowing into Spark SQL.
I can successfully store them as parquet and read them with sqlContext.jsonRDD,
but the inferred schemas cannot be merged into a single table to do queries.
I'd like a way to allow for parquet file schemas to be merged, whether they
match or not, since we know the schema should be a union of the schemas from
the files.
This will allow us to have the data define the schema and new columns will just
appear.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
