alamb commented on code in PR #24030:
URL: https://github.com/apache/datafusion/pull/24030#discussion_r3693191873
##########
benchmarks/src/cancellation.rs:
##########
@@ -300,7 +300,7 @@ async fn generate_data(
});
let to_write = RecordBatch::try_from_iter(data).unwrap();
let path =
object_store::path::Path::from(format!("{file_num}.parquet").as_str());
- let object_store_writer = ParquetObjectWriter::new(Arc::clone(&store)
as _, path);
+ let object_store_writer = BufWriter::new(Arc::clone(&store) as _,
path);
Review Comment:
ParquetObjectWriter wsas deprecated:
https://github.com/apache/arrow-rs/issues/10308
ParquetObjectWriter is a thin wrapper around BufWriter:
https://docs.rs/parquet/latest/src/parquet/arrow/async_writer/store.rs.html#72-74
##########
datafusion/datasource-parquet/src/reader.rs:
##########
@@ -119,14 +125,46 @@ impl AsyncFileReader for ParquetFileReader {
{
let total: u64 = ranges.iter().map(|r| r.end - r.start).sum();
self.file_metrics.bytes_scanned.add(total as usize);
- self.inner.get_byte_ranges(ranges)
+ async move {
+ self.store
+ .get_ranges(&self.partitioned_file.object_meta.location,
&ranges)
+ .await
+ .map_err(|e| ParquetError::External(Box::new(e)))
+ }
+ .boxed()
}
fn get_metadata<'a>(
&'a mut self,
options: Option<&'a ArrowReaderOptions>,
) -> BoxFuture<'a, parquet::errors::Result<Arc<ParquetMetaData>>> {
- self.inner.get_metadata(options)
+ let object_meta = self.partitioned_file.object_meta.clone();
Review Comment:
THe need for this replication is removed by unifying the structures, as I
propose to do do in
- https://github.com/apache/datafusion/pull/24036
##########
datafusion/common/src/nested_struct.rs:
##########
@@ -1123,8 +1123,8 @@ mod tests {
Arc::new(non_null_field(
"entries",
struct_type(vec for details.
The Minimum Supported Rust Version (MSRV) has been updated to [`1.94.0`].
[`1.94.0`]: https://releases.rs/docs/1.94.0/
+
+### `ParquetObjectReader` / `ParquetObjectWriter` deprecated upstream
+
+The [`parquet` crate] deprecated [`ParquetObjectReader`]
+and [`ParquetObjectWriter`] in favor of implementing
+[`AsyncFileReader`] directly (see the example on the [`AsyncFileReader`] trait
and
+[`parquet/examples/object_store.rs`] in `arrow-rs`) or passing an
+[`BufWriter`] straight to [`AsyncArrowWriter`].
+
+**Who is affected:**
+
+- Custom [`ParquetFileReaderFactory`] implementations that construct a
+ [`ParquetObjectReader`] directly and now see a deprecation warning after
+ upgrading the `parquet` dependency.
+
+**Migration guide:**
+
+If your [`AsyncFileReader`] implementation exists mainly to read from an
Review Comment:
This will need to get updated if we merge this one first
- https://github.com/apache/datafusion/pull/24036
##########
datafusion-examples/examples/data_io/parquet_advanced_index.rs:
##########
@@ -601,7 +605,15 @@ impl AsyncFileReader for ParquetReaderWithCache {
range: Range<u64>,
) -> BoxFuture<'_, datafusion::parquet::errors::Result<Bytes>> {
println!("get_bytes: {} Reading range {:?}", self.filename, range);
- self.inner.get_bytes(range)
+ let object_store = Arc::clone(&self.object_store);
Review Comment:
This is due to inlining the logic for ParquetObjectReader
It turns out that DataFusion already has its own copy of the logic of the
ParquetObjectReader between ParquetFileReader and CachedParquetFileReader. I
have a PR that will unify them:
- https://github.com/apache/datafusion/pull/24036
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
