viirya commented on code in PR #5788:
URL: https://github.com/apache/datafusion-comet/pull/5788#discussion_r3971570487
##########
spark/src/test/scala/org/apache/spark/sql/benchmark/CometShuffleBenchmark.scala:
##########
@@ -492,6 +567,25 @@ object CometShuffleBenchmark extends CometBenchmarkBase {
}
}
+ runBenchmarkWithTable("Nested hash partitioning key", 1024 * 1024 * 1) { v
=>
+ // Shapes whose leaves are primitives take the vectorized element path;
the last two force
+ // the per-element path in the native kernel.
+ val shapes = Seq(
+ "struct<int, string>" -> "named_struct('a', c1, 'b', CAST(c1 AS
STRING))",
+ "array<int>" -> "ARRAY_REPEAT(c1, 10)",
+ "struct<array<int>, string>" ->
+ "named_struct('a', ARRAY_REPEAT(c1, 10), 'b', CAST(c1 AS STRING))",
+ "array<struct<int, string>>" ->
+ "ARRAY_REPEAT(named_struct('a', c1, 'b', CAST(c1 AS STRING)), 10)",
+ "struct<map<string, int>, int>" ->
+ "named_struct('m', MAP(CAST(c1 AS STRING), c1), 'i', c1)")
Review Comment:
Addressed in 8d9eede01. The benchmark now skips native cases with
map-containing keys on Spark versions below 4.0. For supported cases, it checks
for exactly one `CometNativeShuffle` exchange outside the timer and records the
physical plan.
Verified on Spark 3.5.9 at both 5 and 201 partitions: struct keys use native
shuffle, while map-containing keys explicitly skip the native arm. The full
Spark 4.1.3 run passed all 18 native exchange checks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
