comphead commented on code in PR #5300:
URL: https://github.com/apache/datafusion-comet/pull/5300#discussion_r4488344130
##########
native/spark-expr/benches/array_size.rs:
##########
@@ -145,11 +145,20 @@ fn criterion_benchmark(c: &mut Criterion) {
&create_list_array(rows, 5, false),
);
- // LargeList: exposes the Int64 length -> Int32 cast (extra allocation) on
top of
- // the length kernel.
+ // LargeList: builds Int32 lengths from i64 offsets (see
spark_size_large_list_from_offsets).
+ // Shapes mirror List coverage (short/long + 10% null, short + no nulls).
bench(
"spark_size: LargeList (10% null)",
- &create_large_list_array(rows, 5),
+ &create_large_list_array(rows, 5, true),
+ );
+ bench(
+ "spark_size: LargeList of long arrays",
+ &create_large_list_array(rows, 64, true),
+ );
+ // No-null LargeList: production path after CometSize's isnotnull filter.
Review Comment:
After #7473 the no-null shape is no longer the only production path: with
`spark.sql.legacy.sizeOfNull` on, `CometSize` skips the `isnotnull` guard and
nulls reach the kernel, which the 10% null rows above cover. Could this say
something like "No-null LargeList: what the kernel sees behind `CometSize`'s
`isnotnull` guard, or for a non-nullable child"?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
