sunchao commented on code in PR #6179:
URL: https://github.com/apache/datafusion-comet/pull/6179#discussion_r4367212296
##########
spark/src/main/scala/org/apache/comet/expressions/CometCast.scala:
##########
@@ -131,6 +131,10 @@ object CometCast
Compatible()
} else if (CometTimeZone.nativeId(cast.timeZoneId).isEmpty) {
CometTimeZone.supportLevel(cast.timeZoneId)
+ } else if (canProduceNullMapKey(cast)) {
Review Comment:
[P2] Preserve native execution for map-key conversions that always succeed.
For a `MAP<BOOLEAN,INT>` column, this guard rejects `TRY_CAST(b AS
MAP<INT,INT>)`, although both possible keys convert to `0` or `1`. It also
rejects default `STRING` to `BINARY` and `ARRAY<INT>` to `ARRAY<BIGINT>` keys.
All three ran natively on the base. In the Boolean case, the fallback moved
both aggregate stages to Spark and required converting every input row. Extend
`keyCastCannotFail` for these proven-safe cases and pin their routing with
`expect_native(try_cast)`.
Evidence: On Spark 4.1.3 with `local[1]`, AQE disabled, and one shuffle
partition, write `SELECT map(false, CAST(id + 1 AS INT), true, CAST(id AS INT))
AS b FROM range(262144)` to Parquet. Read it as `t` and run `SELECT
sum(element_at(try_cast(b AS map<int,int>), 1)) FROM t`. Both SHAs returned
`34359607296`. The base used `CometHashAggregate` and converted one result row.
The head used Spark `HashAggregate` and converted 262,144 input rows in 32
batches. Across three executions after two warmups, median executor task-thread
allocation rose from 4,261,536 to 58,994,104 bytes. The string and array cases
reproduced the same plan change. Scalar `INT`-to-`BIGINT` key widening
converted one result row on both SHAs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
