zeevm commented on a change in pull request #7586:
URL: https://github.com/apache/arrow/pull/7586#discussion_r448745197
##########
File path: rust/parquet/src/column/writer.rs
##########
@@ -276,12 +372,60 @@ impl<T: DataType> ColumnWriterImpl<T> {
&values[values_offset..],
def_levels.map(|lv| &lv[levels_offset..]),
rep_levels.map(|lv| &lv[levels_offset..]),
+ calculate_page_stats,
)?;
// Return total number of values processed.
Ok(values_offset)
}
+ /// Writes batch of values, definition levels and repetition levels.
+ /// Returns number of values processed (written).
+ ///
+ /// If definition and repetition levels are provided, we write fully those
levels and
+ /// select how many values to write (this number will be returned), since
number of
+ /// actual written values may be smaller than provided values.
+ ///
+ /// If only values are provided, then all values are written and the
length of
+ /// of the values buffer is returned.
+ ///
+ /// Definition and/or repetition levels can be omitted, if values are
+ /// non-nullable and/or non-repeated.
+ pub fn write_batch(
+ &mut self,
+ values: &[T::T],
+ def_levels: Option<&[i16]>,
+ rep_levels: Option<&[i16]>,
+ ) -> Result<usize> {
+ self.write_batch_internal(
+ values, def_levels, rep_levels, &None, &None, None, None,
+ )
+ }
+
+ /// Writer may optionally provide pre-calculated statistics for this
batch, in which case we do
+ /// not calculate page level statistics as this will defeat the purpose of
speeding up the write
+ /// process with pre-calculated statistics.
+ pub fn write_batch_with_statistics(
+ &mut self,
+ values: &[T::T],
+ def_levels: Option<&[i16]>,
+ rep_levels: Option<&[i16]>,
+ min: &Option<T::T>,
Review comment:
IIUC the format specifies that the various stats are optional so it
seems reasonable to allow the caller to specify only some of the values isn't
it?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
