yew1eb commented on code in PR #2406:
URL: https://github.com/apache/auron/pull/2406#discussion_r3627522828
##########
native-engine/datafusion-ext-plans/src/sort_exec.rs:
##########
@@ -1598,6 +1698,92 @@ mod fuzztest {
use crate::sort_exec::SortExec;
+ /// Benchmark helper: build a single Int64 column where each value is
+ /// repeated `repeat` times, shuffled, to simulate TPC-H
+ /// lineitem.l_orderkey distribution.
+ fn build_repeated_i64_batch(num_rows: usize, repeat: usize, seed: u64) ->
RecordBatch {
+ use rand::{Rng, SeedableRng};
+ let unique_keys = (num_rows + repeat - 1) / repeat;
+ let mut values: Vec<i64> = (0..unique_keys)
+ .flat_map(|v| std::iter::repeat(v as i64).take(repeat))
+ .take(num_rows)
+ .collect();
+ let mut rng = rand::rngs::StdRng::seed_from_u64(seed);
+ rand::seq::SliceRandom::shuffle(values.as_mut_slice(), &mut rng);
+ let schema = Arc::new(arrow::datatypes::Schema::new(vec![
+ arrow::datatypes::Field::new("l_orderkey",
arrow::datatypes::DataType::Int64, false),
+ ]));
+ RecordBatch::try_new(schema, vec![Arc::new(Int64Array::from(values))
as ArrayRef])
+ .expect("failed to create benchmark batch")
+ }
+
+ async fn bench_sort_repeat(repeat: usize, mem: usize, use_auron: bool) ->
Result<(usize, f64)> {
+ MemManager::init(mem);
Review Comment:
You are right — that was an apples-to-oranges comparison.
`MemManager::init(2 MB)` is Auron-specific; DataFusion's native `SortExec` does
not honor it, so the external group had Auron spilling to disk while DataFusion
sorted fully in memory. I fixed the benchmark: the external/spill group now
reports Auron spill cost on its own (no DataFusion column), and the in-mem
group (1 GB, both stay in memory) is the fair Auron-vs-DataFusion comparison. I
also called this out in a comment on the benchmark.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
