GitHub user kiszk opened a pull request:
https://github.com/apache/spark/pull/18014
[SPARK-20783][SQL] Enhance ColumnVector to keep UnsafeArrayData for array
## What changes were proposed in this pull request?
This PR enhances `ColumnVector` to keep `UnsafeArrayData` for array to use
`ColumnVector` for table cache (e.g. CACHE table, DataFrame.cache).
Current `ColumnVector` accepts only primitive-type Java array as an input
for array. It is good to keep data from Parquet.
This PR changed or added the following APIs:
`ColumnVector ColumnVector.allocate(int capacity, DataType type, MemoryMode
mode, boolean useUnsafeArrayData)`
* When the last is true, the `ColumnVector` can keep `UnsafeArrayData`. If
it is false, the `ColumnVector` cannot keep `UnsafeArrayData`.
`int ColumnVector.putArray(int rowId, ArrayData array)`
* When this `ColumnVector` was generated with `useUnsafeArrayData=true`,
this method stores `UnsafeArrayData` into `ColumnVector`. Otherwise, throw an
exception.
`ArrayData ColumnVector.getArray(int rowId)`
* When this `ColumnVector` was generated with `useUnsafeArrayData=true`,
this method returns `UnsafeArrayData`.
## How was this patch tested?
Update existing testsuite
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/kiszk/spark OnHeapColumnVector
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/18014.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #18014
----
commit b939a0819ac9feb4ff779f50a19ceb101cada999
Author: Kazuaki Ishizaki <[email protected]>
Date: 2017-04-24T16:56:48Z
Keep UnsafeArrayData for Array in ColumnVector
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
