[
https://issues.apache.org/jira/browse/ARROW-17380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Apache Arrow JIRA Bot reassigned ARROW-17380:
---------------------------------------------
Assignee: (was: Ziheng Wang)
> [C++] [Python] Tag record batches with start_byte and end_byte infromation
> --------------------------------------------------------------------------
>
> Key: ARROW-17380
> URL: https://issues.apache.org/jira/browse/ARROW-17380
> Project: Apache Arrow
> Issue Type: Improvement
> Components: C++, Python
> Reporter: Ziheng Wang
> Priority: Major
>
> It might be desirable for a record batch to have information of where it came
> from in the source dataset. This can be used for a few purposes:
> * Rereading a particular record batch without rereading the entire fragment
> * Easily tracking progress of how much a particular (file) dataset has been
> consumed.
> It could also be useful for debugging if a record batch resulted in an error
> downstream.
> The plan is to add some attribute like this here:
> [https://github.com/apache/arrow/blob/master/cpp/src/arrow/dataset/scanner.cc#L923]
> that will be tagged on to the record batch by the Scanner as it is being
> generated.
> This is useful for file based formats like CSV. In Parquet this is less
> necessary since record batches (usually) correspond to row groups and row
> group ids can be used to serve this function.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
