dianaclarke commented on pull request #9971: URL: https://github.com/apache/arrow/pull/9971#issuecomment-820534777
I'm still playing with these numbers. So far, I know: All of the benchmarks that either improved or regressed with mimalloc version 2 were C++ benchmarks. None were Python or R benchmarks. I wrote a little script that first created a skip list of any benchmark that regressed or improved in the placebo run on machine `ursa-thinkcentre-m75q` (which was 4% of the benchmarks). Then I generated a list of benchmarks that got better or worse under version 2 of mimalloc, minus that skip list. With the skips in mind (at a threshold of 5%) 180 benchmarks got worse with version 2 of mimalloc, and 42 got better. Here are the top 10 benchmarks that got worse. ```[114% worse, 'arrow-compute-scalar-cast-benchmark.CastInt64ToDoubleUnsafe, 524288/1000'] [114% worse, 'arrow-compute-scalar-cast-benchmark.CastInt64ToDoubleUnsafe, 524288/10'] [114% worse, 'arrow-compute-scalar-cast-benchmark.CastInt64ToDoubleUnsafe, 524288/1'] [115% worse, 'arrow-compute-scalar-cast-benchmark.CastInt64ToDoubleUnsafe, 524288/2'] [115% worse, 'arrow-compute-scalar-cast-benchmark.CastInt64ToDoubleUnsafe, 524288/0'] [124% worse, 'arrow-compute-scalar-string-benchmark.AsciiUpper'] [124% worse, 'arrow-compute-scalar-string-benchmark.AsciiLower'] [127% worse, 'arrow-io-memory-benchmark.BufferOutputStreamLargeWrites, real_time'] [137% worse, 'arrow-io-memory-benchmark.BufferOutputStreamSmallWrites, real_time'] [316% worse, 'arrow-compute-vector-selection-benchmark.TakeInt64RandomIndicesWithNulls, 524288/1'] ``` And here are the top 10 that got better. ```[55% better, 'arrow-compute-scalar-arithmetic-benchmark.ArrayScalarKernel, <Add, Int32Type>/524288/0'] [55% better, 'arrow-compute-scalar-arithmetic-benchmark.ArrayScalarKernel, <Add, Int32Type>/524288/100'] [54% better, 'arrow-compute-scalar-arithmetic-benchmark.ArrayScalarKernel, <AddChecked, Int8Type>/524288/0'] [54% better, 'arrow-compute-scalar-arithmetic-benchmark.ArrayScalarKernel, <AddChecked, UInt8Type>/524288/0'] [42% better, 'arrow-compute-scalar-arithmetic-benchmark.ArrayScalarKernel, <AddChecked, UInt8Type>/524288/100'] [41% better, 'arrow-compute-scalar-arithmetic-benchmark.ArrayScalarKernel, <MultiplyChecked, Int8Type>/524288/100'] [30% better, 'arrow-compute-vector-sort-benchmark.ArraySortIndicesInt64Narrow, 32768/10000'] [25% better, 'arrow-compute-scalar-arithmetic-benchmark.ArrayScalarKernel, <Add, UInt32Type>/524288/100'] [25% better, 'arrow-compute-scalar-arithmetic-benchmark.ArrayScalarKernel, <Add, UInt32Type>/524288/0'] [25% better, 'arrow-compute-scalar-arithmetic-benchmark.ArrayScalarKernel, <Subtract, Int32Type>/524288/0'] ``` So, if we want to try and track down the mimalloc bug in version 2, I would start with this benchmark since it regressed 316%. `arrow-compute-vector-selection-benchmark.TakeInt64RandomIndicesWithNulls/524288/1` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
