[jira] [Updated] (ARROW-7800) [Python] Expose GetRecordBatchReader API in PyArrow
[
https://issues.apache.org/jira/browse/ARROW-7800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rollo Konig-brock updated ARROW-7800:
-
Description:
The GetRecordBatchReader API is really useful for streaming ParquetFiles with
lots of RLE.
I propose exposing this API in PyArrow in the following manner:
{code}
file_ = ParquetFile('file/path.parquet')
for batch in file_.get_batches(batch_size=100):
pass
{code}
(If anyone has any better ideas hit me up, I'm not 100% sold on exposing it
this way.)
was:
The GetRecordBatchReader API is really useful for streaming ParquetFiles with
lots of RLE.
I propose exposing this API in PyArrow in the following manner:
{code}
file_ = ParquetFile('file/path.parquet', batch_size=100)
for batch in file_.get_batches():
pass
{code}
(If anyone has any better ideas hit me up, I'm not 100% sold on exposing it
this way.)
> [Python] Expose GetRecordBatchReader API in PyArrow
> ---
>
> Key: ARROW-7800
> URL: https://issues.apache.org/jira/browse/ARROW-7800
> Project: Apache Arrow
> Issue Type: Improvement
> Components: Python
>Reporter: Rollo Konig-brock
>Priority: Minor
> Labels: features, pull-request-available
> Fix For: 1.0.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> The GetRecordBatchReader API is really useful for streaming ParquetFiles with
> lots of RLE.
> I propose exposing this API in PyArrow in the following manner:
> {code}
> file_ = ParquetFile('file/path.parquet')
> for batch in file_.get_batches(batch_size=100):
> pass
> {code}
> (If anyone has any better ideas hit me up, I'm not 100% sold on exposing it
> this way.)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (ARROW-7800) [Python] Expose GetRecordBatchReader API in PyArrow
[
https://issues.apache.org/jira/browse/ARROW-7800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rollo Konig-brock updated ARROW-7800:
-
Issue Type: Improvement (was: Bug)
> [Python] Expose GetRecordBatchReader API in PyArrow
> ---
>
> Key: ARROW-7800
> URL: https://issues.apache.org/jira/browse/ARROW-7800
> Project: Apache Arrow
> Issue Type: Improvement
> Components: Python
>Reporter: Rollo Konig-brock
>Priority: Minor
> Labels: features, pull-request-available
> Fix For: 1.0.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> The GetRecordBatchReader API is really useful for streaming ParquetFiles with
> lots of RLE.
> I propose exposing this API in PyArrow in the following manner:
> {code}
> file_ = ParquetFile('file/path.parquet', batch_size=100)
> for batch in file_.get_batches():
> pass
> {code}
> (If anyone has any better ideas hit me up, I'm not 100% sold on exposing it
> this way.)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (ARROW-7800) [Python] Expose GetRecordBatchReader API in PyArrow
[
https://issues.apache.org/jira/browse/ARROW-7800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rollo Konig-brock updated ARROW-7800:
-
Priority: Minor (was: Major)
> [Python] Expose GetRecordBatchReader API in PyArrow
> ---
>
> Key: ARROW-7800
> URL: https://issues.apache.org/jira/browse/ARROW-7800
> Project: Apache Arrow
> Issue Type: Bug
> Components: Python
>Reporter: Rollo Konig-brock
>Priority: Minor
> Labels: features, pull-request-available
> Fix For: 1.0.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> The GetRecordBatchReader API is really useful for streaming ParquetFiles with
> lots of RLE.
> I propose exposing this API in PyArrow in the following manner:
> {code}
> file_ = ParquetFile('file/path.parquet', batch_size=100)
> for batch in file_.get_batches():
> pass
> {code}
> (If anyone has any better ideas hit me up, I'm not 100% sold on exposing it
> this way.)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (ARROW-7800) [Python] Expose GetRecordBatchReader API in PyArrow
[
https://issues.apache.org/jira/browse/ARROW-7800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated ARROW-7800:
--
Labels: features pull-request-available (was: features)
> [Python] Expose GetRecordBatchReader API in PyArrow
> ---
>
> Key: ARROW-7800
> URL: https://issues.apache.org/jira/browse/ARROW-7800
> Project: Apache Arrow
> Issue Type: Bug
> Components: Python
>Reporter: Rollo Konig-brock
>Priority: Major
> Labels: features, pull-request-available
> Fix For: 1.0.0
>
>
> The GetRecordBatchReader API is really useful for streaming ParquetFiles with
> lots of RLE.
> I propose exposing this API in PyArrow in the following manner:
> {code}
> file_ = ParquetFile('file/path.parquet', batch_size=100)
> for batch in file_.get_batches():
> pass
> {code}
> (If anyone has any better ideas hit me up, I'm not 100% sold on exposing it
> this way.)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
