kumarUjjawal commented on code in PR #21575:
URL: https://github.com/apache/datafusion/pull/21575#discussion_r3077345404
##########
datafusion/functions-aggregate/benches/count_distinct.rs:
##########
@@ -150,5 +174,101 @@ fn count_distinct_benchmark(c: &mut Criterion) {
});
}
-criterion_group!(benches, count_distinct_benchmark);
+/// Create group indices with uniform distribution
+fn create_uniform_groups(num_groups: usize) -> Vec<usize> {
+ let mut rng = StdRng::seed_from_u64(42);
+ (0..BATCH_SIZE)
+ .map(|_| rng.random_range(0..num_groups))
+ .collect()
+}
+
+/// Create group indices with skewed distribution (80% in 20% of groups)
+fn create_skewed_groups(num_groups: usize) -> Vec<usize> {
+ let mut rng = StdRng::seed_from_u64(42);
+ let hot_groups = (num_groups / 5).max(1);
+ (0..BATCH_SIZE)
+ .map(|_| {
+ if rng.random_range(0..100) < 80 {
+ rng.random_range(0..hot_groups)
+ } else {
+ rng.random_range(0..num_groups)
+ }
+ })
+ .collect()
+}
+
+fn count_distinct_groups_benchmark(c: &mut Criterion) {
+ let count_fn = Count::new();
+
+ // bench different scenarios
+ let scenarios = [
+ // (name, num_groups, distinct_pct, group_fn)
+ ("sparse_uniform", 10, 80, "uniform"),
+ ("moderate_uniform", 100, 80, "uniform"),
+ ("dense_uniform", 1000, 80, "uniform"),
+ ("sparse_skewed", 10, 80, "skewed"),
+ ("dense_skewed", 1000, 80, "skewed"),
+ ("sparse_high_cardinality", 10, 99, "uniform"),
+ ("dense_low_cardinality", 1000, 20, "uniform"),
+ ];
+
+ for (name, num_groups, distinct_pct, group_type) in scenarios {
+ let n_distinct = BATCH_SIZE * distinct_pct / 100;
+ let values = Arc::new(create_i64_array(n_distinct)) as ArrayRef;
+ let group_indices = if group_type == "uniform" {
+ create_uniform_groups(num_groups)
+ } else {
+ create_skewed_groups(num_groups)
+ };
+
+ let (_schema, args) = prepare_args(DataType::Int64);
+
+ if count_fn.groups_accumulator_supported(args.clone()) {
Review Comment:
I think the better approach is to benchmark the real grouped fallback
adapter here, instead of the manual per-row loop. That is the path DataFusion
uses today when COUNT(DISTINCT) has no native group accumulator, so it gives a
fair before/after baseline for the follow-up PR. I would keep the current if
supported { native } else { fallback adapter } shape. That way this benchmark
measures the real current path now, and will automatically measure the
optimized path once the follow-up lands.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
