marsupialtail opened a new pull request, #13830: URL: https://github.com/apache/arrow/pull/13830
This is going to be a draft PR based on discussions in https://issues.apache.org/jira/browse/ARROW-17313. This PR addresses the CSV format. The Parquet format I believe is covered by already implemented API "subset". The main theme is we want to be able to specify what portion of a FileFragment we want to read when user does .to_batches() in Python or ScanBatchesAsync() in C++. The implementation here adds two fields start_byte_ and end_byte_ to the FileFragment class which can get set by using .slice() in Python or calling the setters in C++. I make sure that the slice() and set_bounds() methods in the Python ParquetFileFormat and C++ respectively do the right thing due to inheritance. This is meant to be a draft. Happy to update the interface if something else makes more sense. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
