[
https://issues.apache.org/jira/browse/ARROW-17380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17631185#comment-17631185
]
Apache Arrow JIRA Bot commented on ARROW-17380:
-----------------------------------------------
This issue was last updated over 90 days ago, which may be an indication it is
no longer being actively worked. To better reflect the current state, the issue
is being unassigned per [project
policy|https://arrow.apache.org/docs/dev/developers/bug_reports.html#issue-assignment].
Please feel free to re-take assignment of the issue if it is being actively
worked, or if you plan to start that work soon.
> [C++] [Python] Tag record batches with start_byte and end_byte infromation
> --------------------------------------------------------------------------
>
> Key: ARROW-17380
> URL: https://issues.apache.org/jira/browse/ARROW-17380
> Project: Apache Arrow
> Issue Type: Improvement
> Components: C++, Python
> Reporter: Ziheng Wang
> Assignee: Ziheng Wang
> Priority: Major
>
> It might be desirable for a record batch to have information of where it came
> from in the source dataset. This can be used for a few purposes:
> * Rereading a particular record batch without rereading the entire fragment
> * Easily tracking progress of how much a particular (file) dataset has been
> consumed.
> It could also be useful for debugging if a record batch resulted in an error
> downstream.
> The plan is to add some attribute like this here:
> [https://github.com/apache/arrow/blob/master/cpp/src/arrow/dataset/scanner.cc#L923]
> that will be tagged on to the record batch by the Scanner as it is being
> generated.
> This is useful for file based formats like CSV. In Parquet this is less
> necessary since record batches (usually) correspond to row groups and row
> group ids can be used to serve this function.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
