etseidl commented on code in PR #10554:
URL: https://github.com/apache/arrow-rs/pull/10554#discussion_r3905977638
##########
parquet/src/column/writer/byte_budget_chunker.rs:
##########
@@ -23,6 +23,52 @@ use crate::column::writer::encoder::ColumnValueEncoder;
use crate::file::properties::WriterProperties;
use crate::schema::types::ColumnDescriptor;
+/// How [`write_granular_chunk`] should cut mini-batch windows in one chunk.
+///
+/// Cutting on exact value counts is the precise option and the expensive one:
+/// it roughly doubles the number of mini-batches on a nullable column, because
+/// the level:value ratio no longer rounds a window up to cover a second value.
+/// It is used only where that precision buys something.
+///
+/// [`write_granular_chunk`]: super::GenericColumnWriter::write_granular_chunk
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub(crate) enum SubBatch {
Review Comment:
Maybe something like `SubBatchStrategy` would be a better name 🤷
##########
parquet/src/column/writer/byte_budget_chunker.rs:
##########
@@ -23,6 +23,52 @@ use crate::column::writer::encoder::ColumnValueEncoder;
use crate::file::properties::WriterProperties;
use crate::schema::types::ColumnDescriptor;
+/// How [`write_granular_chunk`] should cut mini-batch windows in one chunk.
+///
+/// Cutting on exact value counts is the precise option and the expensive one:
+/// it roughly doubles the number of mini-batches on a nullable column, because
+/// the level:value ratio no longer rounds a window up to cover a second value.
+/// It is used only where that precision buys something.
+///
+/// [`write_granular_chunk`]: super::GenericColumnWriter::write_granular_chunk
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub(crate) enum SubBatch {
+ /// Cut after exactly this many values, walking definition levels to find
Review Comment:
Maybe summarize a bit more here, it's getting pretty long. Maybe some can be
replaced with a link to the issue and PRs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
