This is an automated email from the ASF dual-hosted git repository.
tustvold pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/arrow-rs.git
The following commit(s) were added to refs/heads/master by this push:
new 2fe71ca52d Allow custom tree builder for parquet::record::RowIter
(#4783)
2fe71ca52d is described below
commit 2fe71ca52da53586282f6a24d9290ffca776025f
Author: Yuri Kotov <[email protected]>
AuthorDate: Thu Sep 7 20:47:11 2023 +0700
Allow custom tree builder for parquet::record::RowIter (#4783)
* Allow custom tree builder for parquet::record::RowIter
It will allow to read parquet with custom batch_size.
Currently the only possible batch_size for parquet::record::RowIter is 1024
* Change with_tree_builder to with_batch_size to be more future-proof
---------
Co-authored-by: Yuri Kotov <[email protected]>
---
parquet/src/record/reader.rs | 6 ++++++
1 file changed, 6 insertions(+)
diff --git a/parquet/src/record/reader.rs b/parquet/src/record/reader.rs
index 1069eab15f..2a9b6dbb0b 100644
--- a/parquet/src/record/reader.rs
+++ b/parquet/src/record/reader.rs
@@ -747,6 +747,12 @@ impl<'a> RowIter<'a> {
}
}
+ /// Sets batch size for this row iter.
+ pub fn with_batch_size(mut self, batch_size: usize) -> Self {
+ self.tree_builder = self.tree_builder.with_batch_size(batch_size);
+ self
+ }
+
/// Returns common tree builder, so the same settings are applied to both
iterators
/// from file reader and row group.
#[inline]
