GitHub user budde opened a pull request:
https://github.com/apache/spark/pull/5214
[SPARK-6538][SQL] Add missing nullable Metastore fields when merging a
Parquet schema
Opening to replace #5188.
When Spark SQL infers a schema for a DataFrame, it will take the union of
all field types present in the structured source data (e.g. an RDD of JSON
data). When the source data for a row doesn't define a particular field on the
DataFrame's schema, a null value will simply be assumed for this field. This
workflow makes it very easy to construct tables and query over a set of
structured data with a nonuniform schema. However, this behavior is not
consistent in some cases when dealing with Parquet files and an external table
managed by an external Hive metastore.
In our particular usecase, we use Spark Streaming to parse and transform
our input data and then apply a window function to save an arbitrary-sized
batch of data as a Parquet file, which itself will be added as a partition to
an external Hive table via an *"ALTER TABLE... ADD PARTITION..."* statement.
Since our input data is nonuniform, it is expected that not every partition
batch will contain every field present in the table's schema obtained from the
Hive metastore. As such, we expect that the schema of some of our Parquet files
may not contain the same set fields present in the full metastore schema.
In such cases, it seems natural that Spark SQL would simply assume null
values for any missing fields in the partition's Parquet file, assuming these
fields are specified as nullable by the metastore schema. This is not the case
in the current implementation of ParquetRelation2. The
**mergeMetastoreParquetSchema()** method used to reconcile differences between
a Parquet file's schema and a schema retrieved from the Hive metastore will
raise an exception if the Parquet file doesn't match the same set of fields
specified by the metastore.
This pull requests alters the behavior of **mergeMetastoreParquetSchema()**
by having it first add any nullable fields from the metastore schema to the
Parquet file schema if they aren't already present there.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/budde/spark nullable-fields
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/5214.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #5214
----
commit 9041bfa3de565e3bafeaf87fc33a44925a0a0320
Author: Adam Budde <[email protected]>
Date: 2015-03-25T19:59:34Z
Add missing nullable Metastore fields when merging a Parquet schema
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
