ziting-openai commented on code in PR #5262:
URL: https://github.com/apache/datafusion-comet/pull/5262#discussion_r3930137552
##########
native/Cargo.toml:
##########
@@ -35,18 +35,18 @@ license = "Apache-2.0"
edition = "2021"
# Comet uses the same minimum Rust version as DataFusion
-rust-version = "1.88"
+rust-version = "1.94.0"
[workspace.dependencies]
-arrow = { version = "58.4.0", features = ["prettyprint", "ffi", "chrono-tz"] }
-arrow-select = { version = "58.4.0" }
+arrow = { version = "59.2.0", features = ["prettyprint", "ffi", "chrono-tz"] }
+arrow-select = { version = "59.2.0" }
async-trait = { version = "0.1" }
bytes = { version = "1.11.1" }
-parquet = { version = "58.4.0", default-features = false, features =
["experimental"] }
-datafusion = { version = "54.1.0", default-features = false, features =
["unicode_expressions", "crypto_expressions", "nested_expressions", "parquet"] }
-datafusion-datasource = { version = "54.1.0" }
-datafusion-physical-expr-adapter = { version = "54.1.0" }
-datafusion-spark = { version = "54.1.0", features = ["core"] }
+parquet = { version = "59.2.0", default-features = false, features =
["experimental"] }
+datafusion = { version = "55.0.0", default-features = false, features =
["unicode_expressions", "crypto_expressions", "nested_expressions", "parquet"] }
Review Comment:
Update for `3ee7fa65b2f5e7018b5c60f4088a57eba2b9934f`: the compiler
migrations and obsolete `collect_set` drift sentinel are fixed at source level.
The revised test correctly retains the `collect_list` negative check and
positive coerced-state checks for both aggregates.
The two nullable-map-key diagnostic assertions remain unchanged at
`CometCelebornShuffleReaderSuite.scala:1343`. Arrow 59.3 rejects the
nullable-key schema during IPC decoding with `Map key field must not be
nullable`, before Comet reaches its `type mismatch` check. Please accept the
appropriate upstream rejection while preserving the no-output and fetch-failure
checks.
This remains the existing DF/Arrow migration P2, not a new production
correctness finding. [Earlier shuffle
CI](https://github.com/apache/datafusion-comet/actions/runs/33824701130/job/100880184870)
reproduced this on synthetic merge `27cfb46bbf6309823fcfa3d90a56140fe15c898f`
for prior head `598f5e979bdaf82b929596dfcae6ec116bae265e`; the relevant test,
decoder and dependency blobs are unchanged in this update. Current-head CI was
still running when checked. No local tests were run.
##########
native/Cargo.toml:
##########
@@ -62,8 +62,8 @@ object_store = { version = "0.13.2", features = ["gcp",
"azure", "aws", "http"]
url = "2.2"
aws-config = "1.8.18"
aws-credential-types = "1.2.13"
-iceberg = { git = "https://github.com/apache/iceberg-rust";, rev =
"8adaa872f31549dd5ad8255848715758228038bc" }
-iceberg-storage-opendal = { git = "https://github.com/apache/iceberg-rust";,
rev = "8adaa872f31549dd5ad8255848715758228038bc", features = ["opendal-memory",
"opendal-fs", "opendal-s3", "opendal-gcs", "opendal-oss", "opendal-azdls"] }
+iceberg = { git = "https://github.com/apache/iceberg-rust";, rev =
"665c64e48e8d33797ecb1a421f327edd9b024879" }
Review Comment:
Update for `3ee7fa65b2f5e7018b5c60f4088a57eba2b9934f`: adding top-level
partition source fields addresses the reported projected-task-schema failure,
including the ordinary metadata scan used by partitioned `DELETE`. The
merged-partition-type fixture also now avoids the invalid spec-without-values
task while preserving its pool-order assertions. These fixes are verified at
source level; current-head CI was still running when checked.
The mixed live/dropped v1 partition case remains. A v1 table partitioned by
`[identity(region), identity(category)]` can remove the trailing category
partition, write under its retained `void` spec, then drop the category column.
`serializePartitionData` still filters the now-UNKNOWN category value from
`task.spec().partitionType()` but serializes the full real spec because region
remains. The new task-schema union does not rebind or change that Java task
spec or the serialized values. The builder therefore still rejects one value
versus two spec fields. Please keep the serialized spec and values consistent
for this supported evolution path, preserving field order/IDs and the spec ID.
This remains the existing Iceberg task-migration P2. The dropped-field case
is source-traced; no local execution was performed. The earlier [scan
CI](https://github.com/apache/datafusion-comet/actions/runs/33824701130/job/100880184873),
[DELETE
CI](https://github.com/apache/datafusion-comet/actions/runs/33824701130/job/100880832892),
and [Rust fixture
failure](https://github.com/apache/datafusion-comet/actions/runs/33824701130/job/100875289278)
were from prior head `598f5e979bdaf82b929596dfcae6ec116bae265e` through
synthetic merge `27cfb46bbf6309823fcfa3d90a56140fe15c898f`, not executions of
the new head.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
