dwsmith1983 commented on code in PR #7130:
URL: https://github.com/apache/datafusion-comet/pull/7130#discussion_r4391120964
##########
native/core/src/execution/operators/iceberg_scan.rs:
##########
@@ -576,21 +576,36 @@ impl fmt::Debug for RedactedProperties<'_> {
}
}
+/// Exact name matching: iceberg-rust batches and Spark's attributes carry the
table's own names.
+fn iceberg_parquet_options() -> SparkParquetOptions {
+ let mut options = SparkParquetOptions::new(EvalMode::Legacy, "UTC");
+ options.case_sensitive = true;
Review Comment:
> Please rebase onto #7028. Reading `s.X` as `s.x` loses timestamps.
I merged this branch with #7028 (793e32081) and could not reproduce it. At
iceberg-rust b154603a, `RecordBatchTransformer` matches a file's nested fields
to the table's by field id and builds the struct with the table's field names
([`PromotePlan::build`](https://github.com/apache/iceberg-rust/blob/b154603a710d730d2f43f8af4db257cc137237ef/crates/iceberg/src/arrow/record_batch_transformer.rs#L236-L255),
[`apply_struct`](https://github.com/apache/iceberg-rust/blob/b154603a710d730d2f43f8af4db257cc137237ef/crates/iceberg/src/arrow/record_batch_transformer.rs#L459)).
A file written with `s.X` reaches Comet as `s.x`, so the exact match finds it.
To check, I renamed `X` to `x` in structs holding a timestamp, a
timestamp_ntz and an int, with files written before and after the rename. I
read the whole struct, `s.x`, `s.X` with case sensitivity off, and `VERSION AS
OF` the old snapshot, with nested schema pruning on and off. On the merged tree
every read stayed native and matched Spark, on 4.1 and 3.5. On this branch
alone, those reads still fall back with the nested schema evolution reason.
The merge does affect two tests here. "nested struct field renamed by case
only" and "nested field renamed under A beside a" expect the fallback that
#7028 removes. Their results match Spark, and only the fallback assertion
fails; with a native scan check in its place, both pass on 4.1 and 3.5. So it
is the same arrangement as with #7085: whichever of the two lands second
switches those checks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
