On Wed, Oct 26, 2016 at 9:10 AM, Julian Taylor <jtaylor.deb...@googlemail.com> wrote: > On 10/26/2016 06:00 PM, Julian Taylor wrote: >> >> On 10/26/2016 10:59 AM, Ralf Gommers wrote: >>> >>> >>> >>> On Wed, Oct 26, 2016 at 8:33 PM, Julian Taylor >>> <jtaylor.deb...@googlemail.com <mailto:jtaylor.deb...@googlemail.com>> >>> wrote: >>> >>> On 26.10.2016 06:34, Charles R Harris wrote: >>> > Hi All, >>> > >>> > There is a proposed random number package PR now up on github: >>> > https://github.com/numpy/numpy/pull/8209 >>> <https://github.com/numpy/numpy/pull/8209>. It is from >>> > oleksandr-pavlyk <https://github.com/oleksandr-pavlyk >>> <https://github.com/oleksandr-pavlyk>> and implements >>> > the number random number package using MKL for increased speed. >>> I think >>> > we are definitely interested in the improved speed, but I'm not >>> sure >>> > numpy is the best place to put the package. I'd welcome any >>> comments on >>> > the PR itself, as well as any thoughts on the best way organize >>> or use >>> > of this work. Maybe scikit-random >>> >>> >>> Note that this thread is a continuation of >>> https://mail.scipy.org/pipermail/numpy-discussion/2016-July/075822.html >>> >>> >>> >>> I'm not a fan of putting code depending on a proprietary library >>> into numpy. >>> This should be a standalone package which may provide the same >>> interface >>> as numpy. >>> >>> >>> I don't really see a problem with that in principle. Numpy can use Intel >>> MKL (and Accelerate) as well if it's available. It needs some thought >>> put into the API though - a ``numpy.random_intel`` module is certainly >>> not what we want. >>> >> >> For me there is a difference between being able to optionally use a >> proprietary library as an alternative to free software libraries if the >> user wishes to do so and offering functionality that only works with >> non-free software. >> We are providing a form of advertisement for them by allowing it (hey if >> you buy this black box that you cannot modify or use freely you get this >> neat numpy feature!). >> >> I prefer for the full functionality of numpy to stay available with a >> stack of community owned software, even if it may be less powerful that >> way. > > But then if this is really just the same random numbers numpy already > provides just faster, it is probably acceptable in principle. I haven't > actually looked at the PR yet.

The RNG stream is totally different, so yeah, it can't just be a silent drop-in replacement like BLAS/LAPACK. The patch also adds ~10,000 lines of code; here's an example of what some of it looks like: https://github.com/oleksandr-pavlyk/numpy/blob/b53880432c19356f4e54b520958272516bf391a2/numpy/random_intel/mklrand/mkl_distributions.cpp#L1724-L1833 I don't see how we can realistically commit to maintaining this. I'm also not really seeing how shipping it as part of numpy provides extra benefits to maintainers or users? AFAICT right now it's basically structured as a standalone library that's been dropped into the numpy source tree, and it would be just as easy to ship separately (or am I wrong?). And since the public API is that all the functionality comes from importing this specific new module ('numpy.random_intel'), it'd be a one-line change for users to import from a non-numpy namespace, like 'mkl.random' or whatever. If it were more integrated with the rest of numpy then the trade-offs would be more complicated, but in its present form this seems like an easy call. The other question is whether it could/should change to *become* more integrated... that's more tricky. There's been some work towards supporting swappable backends inside np.random; but the focus has mostly been on allowing new core generators, though, and this code seems to want to take over the whole thing (core generator + distributions), so even once the swappable backends stuff is working I'm not sure it would be relevant here. The one case I can think of that does seem promising is that if we get an API for users to say "I don't care about stream compatibility, just give me un-reproducible variates as fast as you can", then it might make sense for that to silently use MKL if available -- this would be pretty analogous to the use of MKL in np.linalg. But we don't have that API yet, I'm not sure how the MKL fallback could be maintainably implemented given that it would require somehow swapping the entire RandomState implementation, and it's entirely possible that once we figure out solutions to those then it'd still make sense for the actual MKL wrappers to live in a third-party library that numpy imports. -n -- Nathaniel J. Smith -- https://vorpus.org _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion