[
https://issues.apache.org/jira/browse/ARROW-4713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yurui Zhou updated ARROW-4713:
------------------------------
Description:
Currently the Arrow C++ provide a naive adapter implementation that allow user
to read orc file to Arrow RecordBatch. However, this implementation have
several drawbacks:
* Inefficient conversion that incurs huge memcpy overhead
** currently the ORC adapter are performing byte to byte memcpy to move data
to ORC VectorBatch to Arrow RecordBatch regardless of the fact that ORC
VectorBatch shares the same memory layout with Arrow in most of the Data Types
* Huge memory footprint because the lack of TableReader implementation
** The ORC adapter currently only allow user to read data with the unit of
stripe. However, as a columnar format with high compression ration, data read
from a ORC stripe can potential takes over gigabytes of memory, which makes the
ORC adapter not quite usable in production environment.
Here we propose a new ORC adapter implementation to fix the issues mentioned
above:
* To reduce conversion overhead, instead of performing naive data copy, the
new adapter would be able to fully taking advantage of the memory layout
similarity between ORC VectorBatch and Arrow RecordBatch. Namely the new
adapter will perform pointer manipulation to transfer the memory ownership from
VectorBatch to Arrow RecordBatch whenever possible.
* The new ORC Adapter would be able to provide user a row level granularity
when reading data from Orc File. The user should be able to specify how many
rows should be expected on output RecordBatch and the ORC Adapter should make
sure no more the requested number of rows would be returned.
was:
Currently the Arrow C++ provide a naive adapter implementation that allow user
to read orc file to Arrow RecordBatch. However, this implementation have
several drawbacks:
* Inefficient conversion that incurs huge memcpy overhead
** currently the ORC adapter are performing byte to byte memcpy to move data
to ORC VectorBatch to Arrow RecordBatch regardless of the fact that ORC
VectorBatch shares the same memory layout with Arrow in most of the Data Types
* Huge memory footprint because the lack of TableReader implementation
** The ORC adapter currently only allow user to read data with the unit of
stripe. However, as a columnar format with high compression ration, data read
from a ORC stripe can potential takes over gigabytes of memory, which makes the
ORC adapter not quite usable in production environment.
Here we propose a new ORC adapter implementation to fix the issue
> Improve C++ Orc Adapter performance and memory footprint
> --------------------------------------------------------
>
> Key: ARROW-4713
> URL: https://issues.apache.org/jira/browse/ARROW-4713
> Project: Apache Arrow
> Issue Type: Improvement
> Components: C++
> Reporter: Yurui Zhou
> Priority: Major
>
> Currently the Arrow C++ provide a naive adapter implementation that allow
> user to read orc file to Arrow RecordBatch. However, this implementation have
> several drawbacks:
> * Inefficient conversion that incurs huge memcpy overhead
> ** currently the ORC adapter are performing byte to byte memcpy to move data
> to ORC VectorBatch to Arrow RecordBatch regardless of the fact that ORC
> VectorBatch shares the same memory layout with Arrow in most of the Data Types
> * Huge memory footprint because the lack of TableReader implementation
> ** The ORC adapter currently only allow user to read data with the unit of
> stripe. However, as a columnar format with high compression ration, data read
> from a ORC stripe can potential takes over gigabytes of memory, which makes
> the ORC adapter not quite usable in production environment.
> Here we propose a new ORC adapter implementation to fix the issues mentioned
> above:
> * To reduce conversion overhead, instead of performing naive data copy, the
> new adapter would be able to fully taking advantage of the memory layout
> similarity between ORC VectorBatch and Arrow RecordBatch. Namely the new
> adapter will perform pointer manipulation to transfer the memory ownership
> from VectorBatch to Arrow RecordBatch whenever possible.
> * The new ORC Adapter would be able to provide user a row level granularity
> when reading data from Orc File. The user should be able to specify how many
> rows should be expected on output RecordBatch and the ORC Adapter should make
> sure no more the requested number of rows would be returned.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
