GitHub user ueshin opened a pull request:
https://github.com/apache/spark/pull/18655
[SPARK-21440][SQL][PYSPARK] Refactor ArrowConverters and add DecimalType,
ArrayType and StructType support.
## What changes were proposed in this pull request?
This is a refactoring of `ArrowConverters` and related classes.
1. Introduce `ArrowColumnVector` to read Arrow data in Java.
2. Refactor `ColumnWriter` as `ArrowWriter`.
3. Add `DecimalType`, `ArrayType` and `StructType` support.
4. Refactor `ArrowConverters` to skip intermediate `ArrowRecordBatch`
creation.
## How was this patch tested?
Added some tests and existing tests.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ueshin/apache-spark issues/SPARK-21440
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/18655.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #18655
----
commit 1439fe6d320208ccb565d18fc4b7485210068330
Author: Takuya UESHIN <[email protected]>
Date: 2017-07-13T08:45:33Z
Introduce ArrowWriter and ArrowColumnVector.
commit 6fcf700d381a18704b3ee485083ee51c82bbd2f8
Author: Takuya UESHIN <[email protected]>
Date: 2017-07-14T05:48:33Z
Use ArrowWriter for ArrowConverters.
commit 579def2db0a0f015760a458032d3bd916669201c
Author: Takuya UESHIN <[email protected]>
Date: 2017-07-14T09:42:32Z
Refactor ArrowConverters.
commit 58cd46506b02800269380f7c8acb5f9825664cad
Author: Takuya UESHIN <[email protected]>
Date: 2017-07-17T06:30:17Z
Move releasing memory into task completion listener.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
