On Tue, May 27, 2025 at 4:27 PM Yury Norov <[email protected]> wrote:
> So, 95% CI means z=1.96, isn't? And to me it should be, for example for > the first line: 5.18 +- 1.96*0.32/sqrt(32) = 5.18 +- 0.11 = [5.07, 5.29]. > Can you check your math please? Facepalm... I used scipy to get the CI, and passed alpha (5%) instead of 1-alpha (95%) for confidence level. I am very sorry... > > Results for sparse bitmap: > > +------------+------+-----------+--------------+-----------+-----------+ > > | Benchmark | Code | Mean (ns) | Std Dev (ns) | 95% CI Lo | 95% CI Hi | > > +------------+------+-----------+--------------+-----------+-----------+ > > | find_bit/ | C | 22.51 | 12.34 | 22.38 | 22.65 | > > | next_bit | Rust | 30.53 | 20.44 | 30.30 | 30.75 | > > +------------+------+-----------+--------------+-----------+-----------+ > > | find_zero/ | C | 5.69 | 0.22 | 5.68 | 5.69 | > > | next_zero | Rust | 5.68 | 0.29 | 5.68 | 5.68 | > > +------------+------+-----------+--------------+-----------+-----------+ > > Your numbers look pretty weird. I wrote the test such that on a typical > x86 system it takes milliseconds for each subtest to pass. Here you > report nanoseconds-scaled times. Did you divide by the number of > iterations? If so, please mention it. Yes, I had divided by number of iterations. Now that you are questioning this, I have second thoughts. I will repeat the benchmark, without dividing by iterations. > Please print raw output of your test in patch #4 which adds the test. > Because the test is tightly coupled to it's C version, we need to make > sure it hast the same format - fields alignment, etc. Will do. In v10, I have removed the module name from the output and the lines match exactly now. > I would prefer to have detailed performance discussion in the > corresponding patch (#4), and here in cover letter you'd just mention > the overall performance difference - 2%, as I can see. > ok. cheers, Burak
