mixermt commented on code in PR #5898:
URL: https://github.com/apache/datafusion-comet/pull/5898#discussion_r4394916715
##########
native/Cargo.toml:
##########
@@ -85,8 +85,8 @@ aws-sdk-sts = { version = "1.114.0", default-features = false
}
# Pinned to a revision rather than a release because no published iceberg-rust
# version is on our arrow: 0.10.1 requires arrow/parquet ^58. Move to a plain
# version requirement once a release ships on arrow 59. Bump policy: #5645.
-iceberg = { git = "https://github.com/apache/iceberg-rust";, rev =
"1f3bc34058e3f3e45336bc5428f37b92649a608b" }
-iceberg-storage-opendal = { git = "https://github.com/apache/iceberg-rust";,
rev = "1f3bc34058e3f3e45336bc5428f37b92649a608b", features = ["opendal-memory",
"opendal-fs", "opendal-s3", "opendal-gcs", "opendal-oss", "opendal-azdls"] }
+iceberg = { git = "https://github.com/apache/iceberg-rust";, rev =
"ca21f1587c6caaeb88dd3a9e1cc245747e1aba08" }
Review Comment:
The pin now sits past apache/iceberg-rust#3437 (`de0a6ac8`): a struct's
fields in a file with neither field ids nor a name mapping read as NULL, as
Spark reads them, whether the struct is read whole or pruned, and a required
one errors in both readers. The test asserts parity with Spark for `s`, `s.b`
and `count(*) WHERE s.a IS NULL` with nested pruning on and off. Two container
shapes still differ and are documented: Spark returns NULL for an optional
struct itself where the native scan returns a struct of NULL fields, and an
empty list for a list of structs where the native scan keeps the elements with
NULL fields.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
