sunchao commented on code in PR #6845:
URL: https://github.com/apache/datafusion-comet/pull/6845#discussion_r4323356562
##########
native/spark-expr/src/map_funcs/map_extract.rs:
##########
@@ -224,11 +227,8 @@ fn spark_map_extract(
}
let indices = UInt32Array::new(indices.into(), nulls.finish());
- Ok(ColumnarValue::Array(take(
- map_array.values(),
- &indices,
- None,
- )?))
+ let result = take(map_array.values(), &indices, None)?;
+ Ok(ColumnarValue::Array(compact_nested_take_result(result)))
Review Comment:
[P2] Preserve a fast path for nested lookups that mostly miss. For 8,192
non-null `map<int,array<int>>` rows with two 128-element values, scalar key
`1`, and about 75% missing keys, this post-`take` compaction makes the
exact-head kernel 29–32% slower than the base. Outputs remain equal and
retained capacity falls from 4,228,100 to 1,079,300 bytes, but every qualifying
lookup pays the reallocation cost even when memory is plentiful. This affects
ordinary `m[1]` and `element_at(m,1)` workloads without requiring null maps
with retained entries. Could the gather avoid that extra cost, or could a
supported compaction option preserve the previous native path for CPU-bound
workloads?
Evidence: Reproduction:
`/tmp/review6845-c174-nb8sywtc/src/bin/sparse_keys_perf.rs`, with logs
`sparse-keys-1.log`, `sparse-keys-2.log` and `sparse-keys-3.log`. Uses
unmodified base/head map kernels, exact-head compaction and
`AccountingAllocator<System>`, Arrow 59.3.0 and DataFusion common/expr 55.2.0,
release optimization, thin LTO and one codegen unit. All map rows are valid.
Rows with `row % 100 < 75` have keys `[2,3]`, others `[1,2]`. Both values
contain 128 integers. Lookup uses scalar Int32(1). Results compare equal and
pass `validate_full()`. After 50 warmups, each run measures 40 alternating
pairs of 30 calls using process CPU time. Median head/base ratios were 1.3236,
1.3239 and 1.2873. Third-run means were 123.28 µs base versus 158.79 µs head.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
