mbutrovich commented on code in PR #3590:
URL: https://github.com/apache/iceberg-rust/pull/3590#discussion_r4605822154
##########
crates/iceberg/src/writer/file_writer/parquet_writer.rs:
##########
@@ -170,8 +171,21 @@ impl FileWriterBuilder for ParquetWriterBuilder {
.encryption_manager
.as_ref()
.map(|em| em.generate_key_metadata());
- let writer_properties =
- resolve_writer_properties(self.props.clone(),
key_metadata.as_ref())?;
+ let mut props = self.props.clone();
+ // Like Java's `MetricsConfig.forPositionDelete`, keep full statistics
for a position
+ // delete file. A truncated `file_path` gives unequal bounds, so the
file can no
+ // longer be matched to the one data file it references.
Review Comment:
At this commit a truncated `file_path` gets no bounds at all, not unequal
ones. `MinMaxColAggregator::update` skips any statistic that isn't exact
([L501-L515](https://github.com/apache/iceberg-rust/blob/20a67a89a28bdcf4193283c980cb0ad779319657/crates/iceberg/src/writer/file_writer/parquet_writer.rs#L501-L515)),
and with the `if` disabled the new test fails with `left: None` for the
70-byte path. Unequal bounds only show up once #3537 lands. Could we word the
comment so it holds either way?
```suggestion
// Like Java's `MetricsConfig.forPositionDelete`, keep full
statistics for a position
// delete file. A truncated `file_path` has no exact bounds, so the
file can no
// longer be matched to the one data file it references.
```
This also quietly overrides a caller's `WriterProperties`, and
`test_statistics_truncation_kept_off_for_position_deletes` checks that an
explicit `Some(16)` gets replaced. That's the right call, but someone who
passes their own truncate length to `ParquetWriterBuilder::new` or
`from_table_properties` can't find this out from the rustdoc
([L58](https://github.com/apache/iceberg-rust/blob/20a67a89a28bdcf4193283c980cb0ad779319657/crates/iceberg/src/writer/file_writer/parquet_writer.rs#L58),
[L67-L72](https://github.com/apache/iceberg-rust/blob/20a67a89a28bdcf4193283c980cb0ad779319657/crates/iceberg/src/writer/file_writer/parquet_writer.rs#L67-L72)).
Should we add a sentence to the `ParquetWriterBuilder` struct docs, since all
three constructors share this behavior? Something like:
```rust
/// If the schema has the position delete `file_path` field, statistics
truncation is
/// turned off regardless of `WriterProperties`, so the file keeps exact
`file_path` bounds.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]