[
https://issues.apache.org/jira/browse/ARROW-17313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17576154#comment-17576154
]
Antoine Pitrou commented on ARROW-17313:
----------------------------------------
Ok, so perhaps byte ranges should actually be provided by the partitioning
scheme ? Most partitioning schemes would only emit trivial byte ranges (i.e.
the entire file), but the Substrait partitioning would emit whatever is in the
Substrait plan.
Of course, it's up to the Substrait producer to ensure that these byte ranges
actually make sense for the given data format. We shouldn't ask the file format
implementations (for example CSV, Parquet or Orc) to accept innacurate byte
ranges.
> [C++] Add Byte Range to CSV Reader ReadOptions
> ----------------------------------------------
>
> Key: ARROW-17313
> URL: https://issues.apache.org/jira/browse/ARROW-17313
> Project: Apache Arrow
> Issue Type: Improvement
> Components: C++, Python
> Reporter: Ziheng Wang
> Assignee: Ziheng Wang
> Priority: Major
>
> Sometimes it's desirable to just read a portion of a CSV. The best way to do
> that is to pass in a list of byte ranges to CSV read options that specify
> where in the CSV you want to read. These byte ranges don't necessarily have
> to be aligned on line break boundaries, the CSV reader should just read until
> the end of the line, and skip anything before the first line break in a byte
> range.
> Based on discussion, the scope is going to be reduced here. The first
> implementation will support a single byte range that is already assumed to be
> aligned on byte boundaries.
> Will not handle quotes/returns and other edge cases.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
