Re: [Numpy-discussion] Looking for a difference between Numpy 0.19.5 and 0.20 explaining a perf regression with Pythran
On Sun, 2021-03-14 at 17:15 +1100, Juan Nunez-Iglesias wrote:
> Hi Pierre,
>
> If you’re able to compile NumPy locally and you have reliable
> benchmarks, you can write a script that tests the runtime of your
> benchmark and reports it as a test pass/fail. You can then use “git
> bisect run” to automatically find the commit that caused the issue.
> That will help narrow down the discussion before it gets completely
> derailed a second time.
>
> https://lwn.net/Articles/317154/
Let me share this partial benchmark result for branch I just worked on
in NumPy:
before after ratio
[c5de5b5c] [2d9e11ea]
+ 2.12±0.01μs 3.69±0.02μs 1.74
bench_io.Copy.time_cont_assign('float32')
+ 22.6±0.08μs 36.0±0.2μs 1.59 bench_io.CopyTo.time_copyto_sparse
+ 49.4±0.8μs 55.2±0.1μs 1.12
bench_io.CopyTo.time_copyto_8_sparse
- 7.40±0.06μs 4.11±0.01μs 0.56 bench_io.CopyTo.time_copyto_dense
- 6.99±0.05μs 3.77±0μs 0.54
bench_io.Copy.time_cont_assign('float64')
- 6.94±0.02μs 3.73±0.01μs 0.54
bench_io.Copy.time_cont_assign('complex64')
That looks weird! The benchmark sometimes speeds up by a factor of
almost 2, and sometimes the (de-facto) same code slows down by just as
much? (Focus on the `time_cont_assign` with float64 vs. float32).
Even better: I know 100% that no related code is touched! The core of
that benchmark is just:
arrya[...] = 1
and I did not even come close to any code related to that operation.
I have, as I did before, tried quite a few things (not as much as in
Victor Stinner's blog when it comes to compiler flags).
Such as enabling/disabling huge-pages, disabling address-space-
randomization (and disabling the NumPy small-array cache).
Note that the results are *stable*, as in: On this branch, I get
extremely reliable results for the benchmark [1]!
As you noticed, I have also seen these (or similar) changes "toggle"
e.g. when copying the array multiple times. And I have dug down into
profiling one instance on the instruction level with `perf` so I know
for a fact that it is memory access speed. (Which is a no-brainer
here, the operations are obviously memory or even cache speed bound.)
The point I was hoping to make is: Its complicated, and I am not
holding my breath that you can find an answer without digging much
deeper.
The blog post from Victor Stinner gave me the thought that profile-
guided-optimization *might* be a way to avoid some random fluctuations,
but I have not checked that the inner-loop for the code actually
compiles to different byte-code.
I would hope that someone comes along and "just knows" what is going
on. But, I don't know where to ask or what to google for.
My best bets right now (they may be terrible!) are:
* Profiler guided optimization might help (as in stabilize compiler
output due to *random* changes in code). Which probably is involved in
some way or another. But Victor Stinner's timed Python and that may
not have any massively memory bound operations (which are the "big"
things here).
* Maybe try to make the NumPy allocator align all its allocation to
much larger boundaries, such as the CPU cache-line size. But I think I
tried to check whether alignment seems to matter, and it didn't. Also,
the arrays feel large enough that it shouldn't matter?
* CPU caching L1/L2 uses a lot of fancy heuristics these days. Maybe to
really understand whats going on, you would have to drill into what the
CPU caches are doing here?
The only thing I do know for sure currently, is that it is a rabbit
hole that I would love to understand, but don't really want to spend
days just to get nowhere.
Cheers,
Sebastian
[1] That run above is without address space randomization, it feels
even more stable than the others. But that doesn't matter, since we
average in any case, so ASR is probably useless and maybe even
detrimental.
>
> Juan.
>
> > On 13 Mar 2021, at 10:34 am, PIERRE AUGIER <
> > pierre.aug...@univ-grenoble-alpes.fr> wrote:
> >
> > Hi,
> >
> > I tried to compile Numpy with `pip install numpy==1.20.1 --no-
> > binary numpy --force-reinstall` and I can reproduce the regression.
> >
> > Good news, I was able to reproduce the difference with only Numpy
> > 1.20.1.
> >
> > Arrays prepared with (`df` is a Pandas dataframe)
> >
> > arr = df.values.copy()
> >
> > or
> >
> > arr = np.ascontiguousarray(df.values)
> >
> > lead to "slow" execution while arrays prepared with
> >
> > arr = np.copy(df.values)
> >
> > lead to faster execution.
> >
> > arr.copy() or np.copy(arr) do not give the same result, with arr
> > obtained from a Pandas dataframe with arr = df.values. It's strange
> > because type(df.values) gives so I would
> > expect arr.copy() and np.copy(arr) to give exactly the same result.
> >
> > Note that I think I'm doing quite serious and reproducible
> > benchmarks. I also checked that this
Re: [Numpy-discussion] Looking for a difference between Numpy 0.19.5 and 0.20 explaining a perf regression with Pythran
Hi Pierre, If you’re able to compile NumPy locally and you have reliable benchmarks, you can write a script that tests the runtime of your benchmark and reports it as a test pass/fail. You can then use “git bisect run” to automatically find the commit that caused the issue. That will help narrow down the discussion before it gets completely derailed a second time. https://lwn.net/Articles/317154/ Juan. > On 13 Mar 2021, at 10:34 am, PIERRE AUGIER > wrote: > > Hi, > > I tried to compile Numpy with `pip install numpy==1.20.1 --no-binary numpy > --force-reinstall` and I can reproduce the regression. > > Good news, I was able to reproduce the difference with only Numpy 1.20.1. > > Arrays prepared with (`df` is a Pandas dataframe) > > arr = df.values.copy() > > or > > arr = np.ascontiguousarray(df.values) > > lead to "slow" execution while arrays prepared with > > arr = np.copy(df.values) > > lead to faster execution. > > arr.copy() or np.copy(arr) do not give the same result, with arr obtained > from a Pandas dataframe with arr = df.values. It's strange because > type(df.values) gives so I would expect arr.copy() > and np.copy(arr) to give exactly the same result. > > Note that I think I'm doing quite serious and reproducible benchmarks. I also > checked that this regression is reproducible on another computer. > > Cheers, > > Pierre > > - Mail original - >> De: "Sebastian Berg" >> À: "numpy-discussion" >> Envoyé: Vendredi 12 Mars 2021 22:50:24 >> Objet: Re: [Numpy-discussion] Looking for a difference between Numpy 0.19.5 >> and 0.20 explaining a perf regression with >> Pythran > >>> On Fri, 2021-03-12 at 21:36 +0100, PIERRE AUGIER wrote: >>> Hi, >>> >>> I'm looking for a difference between Numpy 0.19.5 and 0.20 which >>> could explain a performance regression (~15 %) with Pythran. >>> >>> I observe this regression with the script >>> https://github.com/paugier/nbabel/blob/master/py/bench.py >>> >>> Pythran reimplements Numpy so it is not about Numpy code for >>> computation. However, Pythran of course uses the native array >>> contained in a Numpy array. I'm quite sure that something has changed >>> between Numpy 0.19.5 and 0.20 (or between the corresponding wheels?) >>> since I don't get the same performance with Numpy 0.20. I checked >>> that the values in the arrays are the same and that the flags >>> characterizing the arrays are also the same. >>> >>> Good news, I'm now able to obtain the performance difference just >>> with Numpy 0.19.5. In this code, I load the data with Pandas and need >>> to prepare contiguous Numpy arrays to give them to Pythran. With >>> Numpy 0.19.5, if I use np.copy I get better performance that with >>> np.ascontiguousarray. With Numpy 0.20, both functions create array >>> giving the same performance with Pythran (again, less good that with >>> Numpy 0.19.5). >>> >>> Note that this code is very efficient (more that 100 times faster >>> than using Numpy), so I guess that things like alignment or memory >>> location can lead to such difference. >>> >>> More details in this issue >>> https://github.com/serge-sans-paille/pythran/issues/1735 >>> >>> Any help to understand what has changed would be greatly appreciated! >>> >> >> If you want to really dig into this, it would be good to do profiling >> to find out at where the differences are. >> >> Without that, I don't have much appetite to investigate personally. The >> reason is that fluctuations of ~30% (or even much more) when running >> the NumPy benchmarks are very common. >> >> I am not aware of an immediate change in NumPy, especially since you >> are talking pythran, and only the memory space or the interface code >> should matter. >> As to the interface code... I would expect it to be quite a bit faster, >> not slower. >> There was no change around data allocation, so at best what you are >> seeing is a different pattern in how the "small array cache" ends up >> being used. >> >> >> Unfortunately, getting stable benchmarks that reflect code changes >> exactly is tough... Here is a nice blog post from Victor Stinner where >> he had to go as far as using "profile guided compilation" to avoid >> fluctuations: >> >> https://vstinner.github.io/journey-to-stable-benchmark-deadcode.html >> >> I somewhat hope that this is also the reason for the huge fluctuations >> we see in the NumPy benchmarks due to absolutely unrelated code >> changes. >> But I did not have the energy to try it (and a probably fixed bug in >> gcc makes it a bit harder right now). >> >> Cheers, >> >> Sebastian >> >> >> >> >>> Cheers, >>> Pierre >>> ___ >>> NumPy-Discussion mailing list >>> NumPy-Discussion@python.org >>> https://mail.python.org/mailman/listinfo/numpy-discussion >>> >> >> >> ___ >> NumPy-Discussion mailing list >> NumPy-Discussion@python.org >>
Re: [Numpy-discussion] Looking for a difference between Numpy 0.19.5 and 0.20 explaining a perf regression with Pythran
On Sat, 2021-03-13 at 00:33 +0100, PIERRE AUGIER wrote: > Hi, > > I tried to compile Numpy with `pip install numpy==1.20.1 --no-binary > numpy --force-reinstall` and I can reproduce the regression. > > Good news, I was able to reproduce the difference with only Numpy > 1.20.1. > > Arrays prepared with (`df` is a Pandas dataframe) > > arr = df.values.copy() > > or > > arr = np.ascontiguousarray(df.values) > > lead to "slow" execution while arrays prepared with > > arr = np.copy(df.values) > > lead to faster execution. > > arr.copy() or np.copy(arr) do not give the same result, with arr > obtained from a Pandas dataframe with arr = df.values. It's strange > because type(df.values) gives so I would > expect arr.copy() and np.copy(arr) to give exactly the same result. The only thing that can change would be the arrays flags and `arr.strides`, but they should not have cahnged. And there is no change in NumPy that I can even remotely think of. Array data is just allocated with `malloc`. That is: as I understand it, you are *not* timing `np.copy` or `np.ascontiguouscopy` itself, but just operating on the array returned. NumPy only ever uses `malloc` for allocating array content. > > Note that I think I'm doing quite serious and reproducible > benchmarks. I also checked that this regression is reproducible on > another computer. I absolutely trust the benchmark results. I was hoping you might also be running a profiler (as in analyze the running program) to find out where the difference originate on the C side. That would allow to say with certainty either what changed or that there was no actual related code change. E.g. I have seen huge speed differences in the same `memcpy` or similar calls, due to whatever reasons (maybe due to compiler changes, or due to address space changes... or maybe the former causing the latter, I don't know.). Cheers, Sebastian > > Cheers, > > Pierre > > - Mail original - > > De: "Sebastian Berg" > > À: "numpy-discussion" > > Envoyé: Vendredi 12 Mars 2021 22:50:24 > > Objet: Re: [Numpy-discussion] Looking for a difference between > > Numpy 0.19.5 and 0.20 explaining a perf regression with > > Pythran > > > On Fri, 2021-03-12 at 21:36 +0100, PIERRE AUGIER wrote: > > > Hi, > > > > > > I'm looking for a difference between Numpy 0.19.5 and 0.20 which > > > could explain a performance regression (~15 %) with Pythran. > > > > > > I observe this regression with the script > > > https://github.com/paugier/nbabel/blob/master/py/bench.py > > > > > > Pythran reimplements Numpy so it is not about Numpy code for > > > computation. However, Pythran of course uses the native array > > > contained in a Numpy array. I'm quite sure that something has > > > changed > > > between Numpy 0.19.5 and 0.20 (or between the corresponding > > > wheels?) > > > since I don't get the same performance with Numpy 0.20. I checked > > > that the values in the arrays are the same and that the flags > > > characterizing the arrays are also the same. > > > > > > Good news, I'm now able to obtain the performance difference just > > > with Numpy 0.19.5. In this code, I load the data with Pandas and > > > need > > > to prepare contiguous Numpy arrays to give them to Pythran. With > > > Numpy 0.19.5, if I use np.copy I get better performance that with > > > np.ascontiguousarray. With Numpy 0.20, both functions create > > > array > > > giving the same performance with Pythran (again, less good that > > > with > > > Numpy 0.19.5). > > > > > > Note that this code is very efficient (more that 100 times faster > > > than using Numpy), so I guess that things like alignment or > > > memory > > > location can lead to such difference. > > > > > > More details in this issue > > > https://github.com/serge-sans-paille/pythran/issues/1735 > > > > > > Any help to understand what has changed would be greatly > > > appreciated! > > > > > > > If you want to really dig into this, it would be good to do > > profiling > > to find out at where the differences are. > > > > Without that, I don't have much appetite to investigate personally. > > The > > reason is that fluctuations of ~30% (or even much more) when > > running > > the NumPy benchmarks are very common. > > > > I am not aware of an immediate change in NumPy, especially since > > you > > are talking pythran, and only the memory space or the interface > > code > > should matter. > > As to the interface code... I would expect it to be quite a bit > > faster, > > not slower. > > There was no change around data allocation, so at best what you are > > seeing is a different pattern in how the "small array cache" ends > > up > > being used. > > > > > > Unfortunately, getting stable benchmarks that reflect code changes > > exactly is tough... Here is a nice blog post from Victor Stinner > > where > > he had to go as far as using "profile guided compilation" to avoid > > fluctuations: > > > > > >
Re: [Numpy-discussion] Looking for a difference between Numpy 0.19.5 and 0.20 explaining a perf regression with Pythran
On 2021/03/12 1:33 PM, PIERRE AUGIER wrote: arr.copy() or np.copy(arr) do not give the same result, with arr obtained from a Pandas dataframe with arr = df.values. It's strange because type(df.values) gives so I would expect arr.copy() and np.copy(arr) to give exactly the same result. According to the docstrings for numpy.copy and arr.copy, the function and the method have different defaults for the memory layout. np.copy() tries to maintain the order of the original while arr.copy() defaults to C order. Eric ___ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Looking for a difference between Numpy 0.19.5 and 0.20 explaining a perf regression with Pythran
Hi, I tried to compile Numpy with `pip install numpy==1.20.1 --no-binary numpy --force-reinstall` and I can reproduce the regression. Good news, I was able to reproduce the difference with only Numpy 1.20.1. Arrays prepared with (`df` is a Pandas dataframe) arr = df.values.copy() or arr = np.ascontiguousarray(df.values) lead to "slow" execution while arrays prepared with arr = np.copy(df.values) lead to faster execution. arr.copy() or np.copy(arr) do not give the same result, with arr obtained from a Pandas dataframe with arr = df.values. It's strange because type(df.values) gives so I would expect arr.copy() and np.copy(arr) to give exactly the same result. Note that I think I'm doing quite serious and reproducible benchmarks. I also checked that this regression is reproducible on another computer. Cheers, Pierre - Mail original - > De: "Sebastian Berg" > À: "numpy-discussion" > Envoyé: Vendredi 12 Mars 2021 22:50:24 > Objet: Re: [Numpy-discussion] Looking for a difference between Numpy 0.19.5 > and 0.20 explaining a perf regression with > Pythran > On Fri, 2021-03-12 at 21:36 +0100, PIERRE AUGIER wrote: >> Hi, >> >> I'm looking for a difference between Numpy 0.19.5 and 0.20 which >> could explain a performance regression (~15 %) with Pythran. >> >> I observe this regression with the script >> https://github.com/paugier/nbabel/blob/master/py/bench.py >> >> Pythran reimplements Numpy so it is not about Numpy code for >> computation. However, Pythran of course uses the native array >> contained in a Numpy array. I'm quite sure that something has changed >> between Numpy 0.19.5 and 0.20 (or between the corresponding wheels?) >> since I don't get the same performance with Numpy 0.20. I checked >> that the values in the arrays are the same and that the flags >> characterizing the arrays are also the same. >> >> Good news, I'm now able to obtain the performance difference just >> with Numpy 0.19.5. In this code, I load the data with Pandas and need >> to prepare contiguous Numpy arrays to give them to Pythran. With >> Numpy 0.19.5, if I use np.copy I get better performance that with >> np.ascontiguousarray. With Numpy 0.20, both functions create array >> giving the same performance with Pythran (again, less good that with >> Numpy 0.19.5). >> >> Note that this code is very efficient (more that 100 times faster >> than using Numpy), so I guess that things like alignment or memory >> location can lead to such difference. >> >> More details in this issue >> https://github.com/serge-sans-paille/pythran/issues/1735 >> >> Any help to understand what has changed would be greatly appreciated! >> > > If you want to really dig into this, it would be good to do profiling > to find out at where the differences are. > > Without that, I don't have much appetite to investigate personally. The > reason is that fluctuations of ~30% (or even much more) when running > the NumPy benchmarks are very common. > > I am not aware of an immediate change in NumPy, especially since you > are talking pythran, and only the memory space or the interface code > should matter. > As to the interface code... I would expect it to be quite a bit faster, > not slower. > There was no change around data allocation, so at best what you are > seeing is a different pattern in how the "small array cache" ends up > being used. > > > Unfortunately, getting stable benchmarks that reflect code changes > exactly is tough... Here is a nice blog post from Victor Stinner where > he had to go as far as using "profile guided compilation" to avoid > fluctuations: > > https://vstinner.github.io/journey-to-stable-benchmark-deadcode.html > > I somewhat hope that this is also the reason for the huge fluctuations > we see in the NumPy benchmarks due to absolutely unrelated code > changes. > But I did not have the energy to try it (and a probably fixed bug in > gcc makes it a bit harder right now). > > Cheers, > > Sebastian > > > > >> Cheers, >> Pierre >> ___ >> NumPy-Discussion mailing list >> NumPy-Discussion@python.org >> https://mail.python.org/mailman/listinfo/numpy-discussion >> > > > ___ > NumPy-Discussion mailing list > NumPy-Discussion@python.org > https://mail.python.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Looking for a difference between Numpy 0.19.5 and 0.20 explaining a perf regression with Pythran
On Fri, 2021-03-12 at 21:36 +0100, PIERRE AUGIER wrote: > Hi, > > I'm looking for a difference between Numpy 0.19.5 and 0.20 which > could explain a performance regression (~15 %) with Pythran. > > I observe this regression with the script > https://github.com/paugier/nbabel/blob/master/py/bench.py > > Pythran reimplements Numpy so it is not about Numpy code for > computation. However, Pythran of course uses the native array > contained in a Numpy array. I'm quite sure that something has changed > between Numpy 0.19.5 and 0.20 (or between the corresponding wheels?) > since I don't get the same performance with Numpy 0.20. I checked > that the values in the arrays are the same and that the flags > characterizing the arrays are also the same. > > Good news, I'm now able to obtain the performance difference just > with Numpy 0.19.5. In this code, I load the data with Pandas and need > to prepare contiguous Numpy arrays to give them to Pythran. With > Numpy 0.19.5, if I use np.copy I get better performance that with > np.ascontiguousarray. With Numpy 0.20, both functions create array > giving the same performance with Pythran (again, less good that with > Numpy 0.19.5). > > Note that this code is very efficient (more that 100 times faster > than using Numpy), so I guess that things like alignment or memory > location can lead to such difference. > > More details in this issue > https://github.com/serge-sans-paille/pythran/issues/1735 > > Any help to understand what has changed would be greatly appreciated! > If you want to really dig into this, it would be good to do profiling to find out at where the differences are. Without that, I don't have much appetite to investigate personally. The reason is that fluctuations of ~30% (or even much more) when running the NumPy benchmarks are very common. I am not aware of an immediate change in NumPy, especially since you are talking pythran, and only the memory space or the interface code should matter. As to the interface code... I would expect it to be quite a bit faster, not slower. There was no change around data allocation, so at best what you are seeing is a different pattern in how the "small array cache" ends up being used. Unfortunately, getting stable benchmarks that reflect code changes exactly is tough... Here is a nice blog post from Victor Stinner where he had to go as far as using "profile guided compilation" to avoid fluctuations: https://vstinner.github.io/journey-to-stable-benchmark-deadcode.html I somewhat hope that this is also the reason for the huge fluctuations we see in the NumPy benchmarks due to absolutely unrelated code changes. But I did not have the energy to try it (and a probably fixed bug in gcc makes it a bit harder right now). Cheers, Sebastian > Cheers, > Pierre > ___ > NumPy-Discussion mailing list > NumPy-Discussion@python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > signature.asc Description: This is a digitally signed message part ___ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
