brancz opened a new pull request, #10354: URL: https://github.com/apache/arrow-rs/pull/10354
Remove ParquetObjectReader/ParquetObjectWriter in favor of implementing AsyncFileReader directly (with an example on the trait docs) and passing an AsyncWrite such as object_store's BufWriter to AsyncArrowWriter. To reduce the code needed by implementors I also added SpawnedReader and ParquetMetaDataReader::with_arrow_reader_options. Still keeping object_store as a dev-dependency so we can ensure compatibility and use it in benchmarking. # Which issue does this PR close? - Partially Closes #10308 If we agree on the approach on this, I basically already have the same thing ready for avro, but wanted to keep the PR smaller than necessary so we can align on the direction before doing it all at once. # What changes are included in this PR? Remove the object_store dependency and add some extra helpers to allow users to replace it with very few lines of code. # Are these changes tested? Yes # Are there any user-facing changes? Yes, it both removes the object_store feature from the parquet crate and all functionality that was directly using it. # Additional notes Full disclaimer: Since we don't use parquet, I don't know this code super well, so I used Claude Code a bit to help me in creating this patch. That said, I've fully read the AI-generated parts and understand what they do, and they are how I would have written it, and how I interpreted the prior discussion in #10308. @alamb -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
