leaves12138 commented on code in PR #477:
URL: https://github.com/apache/paimon-rust/pull/477#discussion_r3556692453
##########
crates/paimon/src/arrow/format/parquet.rs:
##########
@@ -87,11 +108,23 @@ impl ParquetFormatWriter {
schema: arrow_schema::SchemaRef,
compression: &str,
zstd_level: i32,
+ write_fields: Option<&[DataField]>,
+ format_options: &HashMap<String, String>,
) -> crate::Result<Self> {
let async_write = output.async_writer().await?;
let codec = parse_compression(compression, zstd_level);
let inner = create_parquet_arrow_writer(async_write, schema, codec)?;
- Ok(Self { inner })
+ let stats_modes = write_fields
+ .map(|fields| {
+ CoreOptions::new(format_options)
Review Comment:
Parsing the per-column stats modes is not sufficient here because Java also
passes `metadata.stats-dense-store` (default `true`) into
`SimpleStatsConverter`. Rust currently ignores that option and
`extract_value_stats` always returns dense, filtered stats with
`Some(column_names)`. With `metadata.stats-dense-store=false`, Java requires
full-width stats and `valueStatsCols = null`; with dense storage enabled, Java
also uses `null` when all write columns are present. Please thread this flag
through and add compatibility tests. A local one-column `counts` regression
with dense store disabled currently returns `Some(["id"])` instead of `None`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
