GitHub user liancheng opened a pull request:
https://github.com/apache/spark/pull/12866
[SPARK-14237][SQL] De-duplicate partition value appending logic in various
buildReader() implementations
## What changes were proposed in this pull request?
Currently, various `FileFormat` data sources share approximately the same
code for partition value appending. This PR tries to eliminate this duplication.
A new method `buildReaderWithPartitionValues()` is added to `FileFormat`
with a default implementation that appends partition values to `InternalRow`s
produced by the reader function returned by `buildReader`.
Special data sources like Parquet, which implements partition value
appending inside `buildReader` because of the vectorized reader, and the Text
data source, which doesn't support partitioning, override
`buildReaderWithPartitionValues()` and simply delegate to `buildReader`.
This PR brings two benefits:
1. Apparently, it de-duplicates partition value appending logic
2. The reader function returned by `FileFormat.buildReader()` is only
required to produce `InternalRow`s rather than `UnsafeRow`s now.
Because the safe-to-unsafe conversion is also performed while appending
partition values. This makes 3rd-party data sources (e.g. spark-avro) easier to
implement since they no longer need to access private APIs involving
`UnsafeRow`.
## How was this patch tested?
Existing tests should do the work.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/liancheng/spark
spark-14237-simplify-partition-values-appending
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/12866.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #12866
----
commit fa0d19d89ee01ba871c0f13f53fe92fb4927854c
Author: Cheng Lian <[email protected]>
Date: 2016-05-03T10:27:57Z
Abstracts away partition value appending
commit 4c946a91e390f902f31581df58fe4e43477a9ce8
Author: Cheng Lian <[email protected]>
Date: 2016-05-03T10:53:30Z
Fixes serialization
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
