sunchao commented on code in PR #6845:
URL: https://github.com/apache/datafusion-comet/pull/6845#discussion_r4241029222
##########
native/spark-expr/src/map_funcs/map_extract.rs:
##########
@@ -206,6 +208,28 @@ fn spark_map_extract(
// representation the format permits rather than one known to arrive here.
Guard it anyway:
// Spark returns NULL for a NULL map under both ANSI modes, for
`element_at` and `GetMapValue`
// alike, and only `element_at` has a nullable-input guard upstream of
this kernel.
+ if matches!(value_type, DataType::List(_) | DataType::LargeList(_)) {
Review Comment:
[P2] Preserve a fast path for nested string values. This branch sends every
list through range planning, including `array<string>`, whose children
ultimately use the same generic `MutableArrayData` copier as before. For 8,192
`map<int,array<string>>` rows with two one-element arrays per non-null map,
scalar lookup key `1`, and 75% null maps with empty entry ranges, the exact
base/head kernels measured 72.51 versus 83.48 microseconds per invocation, a
15.8% median paired slowdown. Non-null maps also slowed about 5% while
retaining identical output capacity. This regresses existing `m[1]` and
`element_at(m,1)` workloads, and the new nested benchmarks use integer children
so they miss it. Please retain Arrow `take` plus selective compaction for
unoptimized child types, or optimize/gate this path, and add nested-string
benchmark coverage.
Evidence: Disposable harness: `/tmp/review6845-final-epg2b0uo/kernel`. It
includes the unmodified base/head `map_extract.rs`, current `array_gather.rs`,
and current `AccountingAllocator<System>`. Build uses release optimization,
thin LTO, one codegen unit, Arrow 59.3.0 and DataFusion 55.1.0 adapters because
55.2.0 was unavailable locally. Construct 8,192 map rows with keys [1,2], each
value a one-element Utf8 list containing `string-value-{i}`. For rows where
`row % 100 < 75`, set map validity false and use an empty entry range. Look up
scalar Int32(1). After 100 warmups, measure 40 alternating base/head rounds of
100 calls using process CPU time. Outputs compare equal and validate
successfully. Median head/base ratio was 1.1577, with 10th–90th percentiles
1.0989–1.1830. The independent Arrow-only harness at
`/tmp/review6845-final-epg2b0uo/gather/src/bin/gather_perf.rs` reproduced a
1.1370 median ratio for non-null string-list gathering without DataFusion
adapters.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
