Re: [Numpy-discussion] Multi-distribution Linux wheels - please test
On Sat, Feb 6, 2016 at 9:28 PM, Nadav Horesh wrote: > Test platform: python 3.4.1 on archlinux x86_64 > > scipy test: OK > > OK (KNOWNFAIL=97, SKIP=1626) > > > numpy tests: Failed on long double and int128 tests, and got one error: > > Traceback (most recent call last): > File "/usr/lib/python3.5/site-packages/nose/case.py", line 198, in runTest > self.test(*self.arg) > File > "/usr/lib/python3.5/site-packages/numpy/core/tests/test_longdouble.py", line > 108, in test_fromstring_missing > np.array([1])) > File "/usr/lib/python3.5/site-packages/numpy/testing/utils.py", line 296, > in assert_equal > return assert_array_equal(actual, desired, err_msg, verbose) > File "/usr/lib/python3.5/site-packages/numpy/testing/utils.py", line 787, > in assert_array_equal > verbose=verbose, header='Arrays are not equal') > File "/usr/lib/python3.5/site-packages/numpy/testing/utils.py", line 668, > in assert_array_compare > raise AssertionError(msg) > AssertionError: > Arrays are not equal > > (shapes (6,), (1,) mismatch) > x: array([ 1., -1., 3., 4., 5., 6.]) > y: array([1]) > > -- > Ran 6019 tests in 28.029s > > FAILED (KNOWNFAIL=13, SKIP=12, errors=1, failures=18 Great - thanks so much for doing this. Do you get a different error if you compile from source? If you compile from source, do you link to OpenBLAS? Thanks again, Matthew ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Multi-distribution Linux wheels - please test
Test platform: python 3.4.1 on archlinux x86_64 scipy test: OK OK (KNOWNFAIL=97, SKIP=1626) numpy tests: Failed on long double and int128 tests, and got one error: Traceback (most recent call last): File "/usr/lib/python3.5/site-packages/nose/case.py", line 198, in runTest self.test(*self.arg) File "/usr/lib/python3.5/site-packages/numpy/core/tests/test_longdouble.py", line 108, in test_fromstring_missing np.array([1])) File "/usr/lib/python3.5/site-packages/numpy/testing/utils.py", line 296, in assert_equal return assert_array_equal(actual, desired, err_msg, verbose) File "/usr/lib/python3.5/site-packages/numpy/testing/utils.py", line 787, in assert_array_equal verbose=verbose, header='Arrays are not equal') File "/usr/lib/python3.5/site-packages/numpy/testing/utils.py", line 668, in assert_array_compare raise AssertionError(msg) AssertionError: Arrays are not equal (shapes (6,), (1,) mismatch) x: array([ 1., -1., 3., 4., 5., 6.]) y: array([1]) -- Ran 6019 tests in 28.029s FAILED (KNOWNFAIL=13, SKIP=12, errors=1, failures=18 From: NumPy-Discussion on behalf of Matthew Brett Sent: 06 February 2016 22:26 To: Discussion of Numerical Python; SciPy Developers List Subject: [Numpy-discussion] Multi-distribution Linux wheels - please test Hi, As some of you may have seen, Robert McGibbon and Nathaniel have just guided a PEP for multi-distribution Linux wheels past the approval process over on distutils-sig: https://www.python.org/dev/peps/pep-0513/ The PEP includes a docker image on which y'all can build wheels which match the PEP: https://quay.io/repository/manylinux/manylinux Now we're at the stage where we need stress-testing of the built wheels to find any problems we hadn't thought of. I've built numpy and scipy wheels here: https://nipy.bic.berkeley.edu/manylinux/ So, if you have a Linux distribution handy, we would love to hear from you about the results of testing these guys, maybe on the lines of: pip install -f https://nipy.bic.berkeley.edu/manylinux numpy scipy python -c 'import numpy; numpy.test()' python -c 'import scipy; scipy.test()' These manylinux wheels should soon be available on pypi, and soon after, installable with latest pip, so we would like to fix as many problems as possible before going live. Cheers, Matthew ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] resizeable arrays using shared memory?
On Sa, 2016-02-06 at 16:56 -0600, Elliot Hallmark wrote: > Hi all, > > I have a program that uses resize-able arrays. I already over > -provision the arrays and use slices, but every now and then the data > outgrows that array and it needs to be resized. > > Now, I would like to have these arrays shared between processes > spawned via multiprocessing (for fast interprocess communication > purposes, not for parallelizing work on an array). I don't care > about mapping to a file on disk, and I don't want disk I/O happening. > I don't care (really) about data being copied in memory on resize. > I *do* want the array to be resized "in place", so that the child > processes can still access the arrays from the object they were > initialized with. > > > I can share arrays easily using arrays that are backed by memmap. > Ie: > > ``` > #Source: http://github.com/rainwoodman/sharedmem > > > class anonymousmemmap(numpy.memmap): > def __new__(subtype, shape, dtype=numpy.uint8, order='C'): > > descr = numpy.dtype(dtype) > _dbytes = descr.itemsize > > shape = numpy.atleast_1d(shape) > size = 1 > for k in shape: > size *= k > > bytes = int(size*_dbytes) > > if bytes > 0: > mm = mmap.mmap(-1,bytes) > else: > mm = numpy.empty(0, dtype=descr) > self = numpy.ndarray.__new__(subtype, shape, dtype=descr, > buffer=mm, order=order) > self._mmap = mm > return self > > def __array_wrap__(self, outarr, context=None): > return > numpy.ndarray.__array_wrap__(self.view(numpy.ndarray), outarr, > context) > ``` > > This cannot be resized because it does not own it's own data > (ValueError: cannot resize this array: it does not own its data). > (numpy.memmap has this same issue [0], even if I set refcheck to > False and even though the docs say otherwise [1]). > > arr._mmap.resize(x) fails because it is annonymous (error: [Errno 9] > Bad file descriptor). If I create a file and use that fileno to > create the memmap, then I can resize `arr._mmap` but the array itself > is not resized. > > Is there a way to accomplish what I want? Or, do I just need to > figure out a way to communicate new arrays to the child processes? > I guess the answer is no, but the first question should be whether you can create a new array viewing the same data that is just larger? Since you have the mmap, that would be creating a new view into it. I.e. your "array" would be the memmap, and to use it, you always rewrap it into a new numpy array. Other then that, you would have to mess with the internal ndarray structure, since these kind of operations appear rather unsafe. - Sebastian > Thanks, > Elliot > > [0] https://github.com/numpy/numpy/issues/4198. > > [1] http://docs.scipy.org/doc/numpy/reference/generated/numpy.memmap. > resize.html > > > ___ > NumPy-Discussion mailing list > NumPy-Discussion@scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion signature.asc Description: This is a digitally signed message part ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Numpy 1.11.0b2 released
Robert, Thanks for pointing out auditwheel. We're experimenting with a GCC 5.2 toolchain, and this tool will be invaluable. Chris, Both conda-build-all and obvious-ci are excellent projects, and we'll leverage them where we can (particularly conda-build-all). Obvious CI and conda-smithy are in a slightly different space, as we want to use our own anaconda.org build service, rather than write scripts to run on other CI services. With more control, we can do cool things like splitting up build jobs and further parallelizing them on more workers, which I see as very important if we're going to be building downstream stuff. As I see it, the single, massive recipe repo that is conda-recipes has been a disadvantage for a while in terms of complexity, but now may be an advantage in terms of building downstream packages (how else would dependency get resolved?) It remains to be seen whether git submodules might replace individual folders in conda-recipes - I think this might give project maintainers more direct control over their packages. The goal, much like ObviousCI, is to enable project maintainers to get their latest releases available in conda sooner, and to simplify the whole CI setup process. We hope we can help each other rather than compete. Best, Michael On Sat, Feb 6, 2016 at 5:53 PM Chris Barker wrote: > On Sat, Feb 6, 2016 at 3:42 PM, Michael Sarahan > wrote: > >> FWIW, we (Continuum) are working on a CI system that builds conda >> recipes. >> > > great, could be handy. I hope you've looked at the open-source systems > that do this: obvious-ci and conda-build-all. And conda-smithy to help set > it all up.. > > Chris, it may still be useful to use docker here (perhaps on the build >> worker, or elsewhere), also, as the distinction between build machines and >> user machines is important to make. Docker would be great for making sure >> that all dependency requirements are met on end-user systems >> > > yes -- veryhandy, I have certainly accidentally brough in other system > libs in a build > > Too bad it's Linux only. Though very useful for manylinux. > > > -Chris > > > > -- > > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R(206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > chris.bar...@noaa.gov > ___ > NumPy-Discussion mailing list > NumPy-Discussion@scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Numpy 1.11.0b2 released
On Sat, Feb 6, 2016 at 3:42 PM, Michael Sarahan wrote: > FWIW, we (Continuum) are working on a CI system that builds conda recipes. > great, could be handy. I hope you've looked at the open-source systems that do this: obvious-ci and conda-build-all. And conda-smithy to help set it all up.. Chris, it may still be useful to use docker here (perhaps on the build > worker, or elsewhere), also, as the distinction between build machines and > user machines is important to make. Docker would be great for making sure > that all dependency requirements are met on end-user systems > yes -- veryhandy, I have certainly accidentally brough in other system libs in a build Too bad it's Linux only. Though very useful for manylinux. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Numpy 1.11.0b2 released
> (we've had a few recent issues with libgfortran accidentally missing as a requirement of scipy). On this topic, you may be able to get some milage out of adapting pypa/auditwheel, which can load up extension module `.so` files inside a wheel (or conda package) and walk the shared library dependency tree like the runtime linker (using pyelftools), and check whether things are going to resolve properly and where shared libraries are loaded from. Something like that should be able to, with minimal adaptation to use the conda dependency resolver, check that a conda package properly declares all of the shared library dependencies it actually needs. -Robert On Sat, Feb 6, 2016 at 3:42 PM, Michael Sarahan wrote: > FWIW, we (Continuum) are working on a CI system that builds conda > recipes. Part of this is testing not only individual packages that change, > but also any downstream packages that are also in the repository of > recipes. The configuration for this is in > https://github.com/conda/conda-recipes/blob/master/.binstar.yml and the > project doing the dependency detection is in > https://github.com/ContinuumIO/ProtoCI/ > > This is still being established (particularly, provisioning build > workers), but please talk with us if you're interested. > > Chris, it may still be useful to use docker here (perhaps on the build > worker, or elsewhere), also, as the distinction between build machines and > user machines is important to make. Docker would be great for making sure > that all dependency requirements are met on end-user systems (we've had a > few recent issues with libgfortran accidentally missing as a requirement of > scipy). > > Best, > Michael > > On Sat, Feb 6, 2016 at 5:22 PM Chris Barker wrote: > >> On Fri, Feb 5, 2016 at 3:24 PM, Nathaniel Smith wrote: >> >>> On Fri, Feb 5, 2016 at 1:16 PM, Chris Barker >>> wrote: >>> >> >> >>> >> > If we set up a numpy-testing conda channel, it could be used to >>> cache >>> >> > binary builds for all he versions of everything we want to test >>> >> > against. >>> >> Anaconda doesn't always have the >>> > latest builds of everything. >> >> >> OK, this may be more or less helpful, depending on what we want to built >> against. But a conda environment (maybe tied to a custom channel) really >> does make a nice contained space for testing that can be set up fast on a >> CI server. >> >> If whoever is setting up a test system/matrix thinks this would be >> useful, I'd be glad to help set it up. >> >> -Chris >> >> >> >> >> >> -- >> >> Christopher Barker, Ph.D. >> Oceanographer >> >> Emergency Response Division >> NOAA/NOS/OR&R(206) 526-6959 voice >> 7600 Sand Point Way NE (206) 526-6329 fax >> Seattle, WA 98115 (206) 526-6317 main reception >> >> chris.bar...@noaa.gov >> ___ >> NumPy-Discussion mailing list >> NumPy-Discussion@scipy.org >> https://mail.scipy.org/mailman/listinfo/numpy-discussion >> > > ___ > NumPy-Discussion mailing list > NumPy-Discussion@scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > > -- -Robert ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Numpy 1.11.0b2 released
FWIW, we (Continuum) are working on a CI system that builds conda recipes. Part of this is testing not only individual packages that change, but also any downstream packages that are also in the repository of recipes. The configuration for this is in https://github.com/conda/conda-recipes/blob/master/.binstar.yml and the project doing the dependency detection is in https://github.com/ContinuumIO/ProtoCI/ This is still being established (particularly, provisioning build workers), but please talk with us if you're interested. Chris, it may still be useful to use docker here (perhaps on the build worker, or elsewhere), also, as the distinction between build machines and user machines is important to make. Docker would be great for making sure that all dependency requirements are met on end-user systems (we've had a few recent issues with libgfortran accidentally missing as a requirement of scipy). Best, Michael On Sat, Feb 6, 2016 at 5:22 PM Chris Barker wrote: > On Fri, Feb 5, 2016 at 3:24 PM, Nathaniel Smith wrote: > >> On Fri, Feb 5, 2016 at 1:16 PM, Chris Barker >> wrote: >> > > >> >> > If we set up a numpy-testing conda channel, it could be used to cache >> >> > binary builds for all he versions of everything we want to test >> >> > against. >> > Anaconda doesn't always have the >> > latest builds of everything. > > > OK, this may be more or less helpful, depending on what we want to built > against. But a conda environment (maybe tied to a custom channel) really > does make a nice contained space for testing that can be set up fast on a > CI server. > > If whoever is setting up a test system/matrix thinks this would be useful, > I'd be glad to help set it up. > > -Chris > > > > > > -- > > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R(206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > chris.bar...@noaa.gov > ___ > NumPy-Discussion mailing list > NumPy-Discussion@scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Numpy 1.11.0b2 released
On Fri, Feb 5, 2016 at 3:24 PM, Nathaniel Smith wrote: > On Fri, Feb 5, 2016 at 1:16 PM, Chris Barker > wrote: > > >> > If we set up a numpy-testing conda channel, it could be used to cache > >> > binary builds for all he versions of everything we want to test > >> > against. > Anaconda doesn't always have the > > latest builds of everything. OK, this may be more or less helpful, depending on what we want to built against. But a conda environment (maybe tied to a custom channel) really does make a nice contained space for testing that can be set up fast on a CI server. If whoever is setting up a test system/matrix thinks this would be useful, I'd be glad to help set it up. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] resizeable arrays using shared memory?
Hi all, I have a program that uses resize-able arrays. I already over-provision the arrays and use slices, but every now and then the data outgrows that array and it needs to be resized. Now, I would like to have these arrays shared between processes spawned via multiprocessing (for fast interprocess communication purposes, not for parallelizing work on an array). I don't care about mapping to a file on disk, and I don't want disk I/O happening. I don't care (really) about data being copied in memory on resize. I *do* want the array to be resized "in place", so that the child processes can still access the arrays from the object they were initialized with. I can share arrays easily using arrays that are backed by memmap. Ie: ``` #Source: http://github.com/rainwoodman/sharedmem class anonymousmemmap(numpy.memmap): def __new__(subtype, shape, dtype=numpy.uint8, order='C'): descr = numpy.dtype(dtype) _dbytes = descr.itemsize shape = numpy.atleast_1d(shape) size = 1 for k in shape: size *= k bytes = int(size*_dbytes) if bytes > 0: mm = mmap.mmap(-1,bytes) else: mm = numpy.empty(0, dtype=descr) self = numpy.ndarray.__new__(subtype, shape, dtype=descr, buffer=mm, order=order) self._mmap = mm return self def __array_wrap__(self, outarr, context=None): return numpy.ndarray.__array_wrap__(self.view(numpy.ndarray), outarr, context) ``` This cannot be resized because it does not own it's own data (ValueError: cannot resize this array: it does not own its data). (numpy.memmap has this same issue [0], even if I set refcheck to False and even though the docs say otherwise [1]). arr._mmap.resize(x) fails because it is annonymous (error: [Errno 9] Bad file descriptor). If I create a file and use that fileno to create the memmap, then I can resize `arr._mmap` but the array itself is not resized. Is there a way to accomplish what I want? Or, do I just need to figure out a way to communicate new arrays to the child processes? Thanks, Elliot [0] https://github.com/numpy/numpy/issues/4198. [1] http://docs.scipy.org/doc/numpy/reference/generated/numpy.memmap.resize.html ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] Multi-distribution Linux wheels - please test
Hi, As some of you may have seen, Robert McGibbon and Nathaniel have just guided a PEP for multi-distribution Linux wheels past the approval process over on distutils-sig: https://www.python.org/dev/peps/pep-0513/ The PEP includes a docker image on which y'all can build wheels which match the PEP: https://quay.io/repository/manylinux/manylinux Now we're at the stage where we need stress-testing of the built wheels to find any problems we hadn't thought of. I've built numpy and scipy wheels here: https://nipy.bic.berkeley.edu/manylinux/ So, if you have a Linux distribution handy, we would love to hear from you about the results of testing these guys, maybe on the lines of: pip install -f https://nipy.bic.berkeley.edu/manylinux numpy scipy python -c 'import numpy; numpy.test()' python -c 'import scipy; scipy.test()' These manylinux wheels should soon be available on pypi, and soon after, installable with latest pip, so we would like to fix as many problems as possible before going live. Cheers, Matthew ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] ANN: numexpr 2.5
= Announcing Numexpr 2.5 = Numexpr is a fast numerical expression evaluator for NumPy. With it, expressions that operate on arrays (like "3*a+4*b") are accelerated and use less memory than doing the same calculation in Python. It wears multi-threaded capabilities, as well as support for Intel's MKL (Math Kernel Library), which allows an extremely fast evaluation of transcendental functions (sin, cos, tan, exp, log...) while squeezing the last drop of performance out of your multi-core processors. Look here for a some benchmarks of numexpr using MKL: https://github.com/pydata/numexpr/wiki/NumexprMKL Its only dependency is NumPy (MKL is optional), so it works well as an easy-to-deploy, easy-to-use, computational engine for projects that don't want to adopt other solutions requiring more heavy dependencies. What's new == In this version, a lock has been added so that numexpr can be called from multithreaded apps. Mind that this does not prevent numexpr to use multiple cores internally. Also, a new min() and max() functions have been added. Thanks to contributors! In case you want to know more in detail what has changed in this version, see: https://github.com/pydata/numexpr/blob/master/RELEASE_NOTES.rst Where I can find Numexpr? = The project is hosted at GitHub in: https://github.com/pydata/numexpr You can get the packages from PyPI as well (but not for RC releases): http://pypi.python.org/pypi/numexpr Share your experience = Let us know of any bugs, suggestions, gripes, kudos, etc. you may have. Enjoy data! -- Francesc Alted ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
