ninglinLiu commented on code in PR #11925:
URL: https://github.com/apache/arrow-rs/pull/11925#discussion_r4390030871
##########
arrow/benches/merge_kernels.rs:
##########
@@ -225,6 +225,36 @@ fn bench_merge_input_on_all_masks(
}
}
+/// An Int8-keyed dictionary of `len` random keys into `distinct` string
values.
+/// Each call builds its own values array, so two calls share values but not
+/// the dictionary itself
+fn generate_int8_dictionary(seed: u64, len: usize, distinct: usize) ->
ArrayRef {
+ let mut rng = StdRng::seed_from_u64(seed);
+ let values = StringArray::from_iter_values((0..distinct).map(|i|
format!("value_{i}")));
+ let keys = Int8Array::from_iter_values((0..len).map(|_|
rng.random_range(0..distinct) as i8));
+ Arc::new(DictionaryArray::<Int8Type>::try_new(keys,
Arc::new(values)).unwrap())
+}
+
+fn bench_merge_dictionaries(c: &mut Criterion) {
+ const ARRAY_LEN: usize = 8192;
+
+ let mut group =
c.benchmark_group(format!("merge_{ARRAY_LEN}_from_int8_dictionaries"));
+ let masks = mask_cases(ARRAY_LEN);
+
+ // 2 x 50 values fit in Int8 keys when the dictionaries are concatenated,
+ // 2 x 100 values do not, so merging them has to merge the dictionaries
+ for (description, distinct) in [
+ ("array_vs_array_concatenated_values_fit", 50),
Review Comment:
There isn't one at the moment: before this PR, `merge_kernels.rs` only
benchmarked `i32`, strings and binary, so no `merge` benchmark used dictionary
inputs at all. This case is the only baseline for dictionaries whose
concatenated values still fit the key type (the `MutableArrayData` path), and
it lets the overflow case below be compared against it. I'd like to keep it for
that reason, but I'm happy to drop it if you'd rather keep only the overflow
case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
