Re: [Numpy-discussion] numpy 1.8.0b1 mkl test_xerbla failure
Charles R Harris wrote: On Thu, Sep 5, 2013 at 5:34 AM, Neal Becker ndbeck...@gmail.com wrote: Just want to make sure this post had been noted: Neal Becker wrote: Built on fedora linux 19 x86_64 using mkl: build OK using: env ATLAS=/usr/lib64 FFTW=/usr/lib64 BLAS=/usr/lib64 LAPACK=/usr/lib64 CFLAGS=-mtune=native -march=native -O3 LDFLAGS=-Wl,- rpath=/opt/intel/mkl/lib/intel64 python setup.py build and attached site.cfg: == FAIL: test_linalg.test_xerbla -- Traceback (most recent call last): File /usr/lib/python2.7/site-packages/nose/case.py, line 197, in runTest self.test(*self.arg) File /home/nbecker/.local/lib/python2.7/site- packages/numpy/testing/decorators.py, line 146, in skipper_func return f(*args, **kwargs) File /home/nbecker/.local/lib/python2.7/site- packages/numpy/linalg/tests/test_linalg.py, line 925, in test_xerbla assert_(False) File /home/nbecker/.local/lib/python2.7/site- packages/numpy/testing/utils.py, line 44, in assert_ raise AssertionError(msg) AssertionError -- Ran 5271 tests in 57.567s FAILED (KNOWNFAIL=5, SKIP=13, failures=1) nose.result.TextTestResult run=5271 errors=0 failures=1 What version of MKL is this? The bug doesn't show in Christolph's compiles with MKL on windows, so it might be an MKL bug. Is it repeatable? seems to be 2013.3.163 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Weird behavior of gufuncs
On Fri, Sep 6, 2013 at 12:09 AM, Jaime Fernández del Río jaime.f...@gmail.com wrote: Hi all, I am seeing some very weird behavior on a gufunc I coded. It has a pretty complicated signature: '(r,c,p),(i,j,k,n),(u,v),(d),(n,q)-(q,r,c)' And a single registered loop function, for types: uint8, uint16, uint16, uintp, uint8-uint8. In general it performs beautifully well, returning the expected results about x10 faster than a MATLAB version using a 12 core CPU (!) But today I found out that for certain sizes of the inputs, it doesn't even call the loop function and returns a zeroed output array immediately. I am using Python 2.7.5 and Numpy 1.7.1 under Windows 7, and have seen the behavior in both x64 and x86 versions. The thresholds vary, and I don't fully understand the logic behind it, but there is a clear threshold of sizes: keeping all other inputs the same, one byte more in the other argument and it returns all zeros, one byte less and everything works fine. The thresholds are different for 64 and 32 bit versions. I have a strong suspicion that this may be a NumPy 1.7 bug, because a colleague running MacOS and NumPy 1.8 compiled from the development branch isn't seeing any of this issues I have. It could of course be a Windows thing, but somehow that seems less likely, especially since there are other bugs in gufuncs that have been fixed in 1.8 but haven't been backported to 1.7. Tomorrow I am going to try and get access to a Linux box with Numpy 1.7 to try to reproduce it there. Does this sound familiar to anyone? Any known (and hopefully solved) issue that could trigger this behavior? A search for ufunc in github's issue tracker didn't give me any meaningful leads... Doesn't ring any bell directly, but as you note there are definitely some major gufunc bug fixes in 1.8 and master, so your best bet on tracking it down might be to (a) check whether it occurs in master, (b) if not, bisect it. -n ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] Networkx graph - numpy array - networkx.has_path() function
Hi experts! I wanna use networkx.has_path(). This is applies, necesary, to a networkx graph. I have a adjacency matrix of a undirected graph (M, wich is a numpy matrix (array of N x N elements)). How can I do for use M in networkx.has_path()? If I must transform M into a networkx graph: how can I do that? Waiting for your answers. Thanks a lot!! José Luis___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Funny business with 'is' operator?:
The .data attribute is generated on the fly when accessed. So it returns an anonymous temporary that's deallocated as soon as it's no longer needed. a.data is b.data needs both objects, so both get allocated and then compared. In the second one though, each object gets allocated one at a time and then immediately released. So the a.data and b.data objects are different... but since they have non-overlapping lifetimes, they happen to be out at the same memory location. id() is only guaranteed to be unique while the object is alive. Notice also: a.data is a.data - False -n On 6 Sep 2013 17:21, James Bergstra bergs...@iro.umontreal.ca wrote: Hi, could someone help me understand why this assertion fails? def test_is(self): a = np.empty(1) b = np.empty(1) if a.data is not b.data: assert id(a.data) != id(b.data) # -- fail I'm trying to write an alternate may_share_memory function. Thanks, - James ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Funny business with 'is' operator?
On Fri, Sep 6, 2013 at 9:19 AM, James Bergstra bergs...@iro.umontreal.cawrote: def test_is(self): a = np.empty(1) b = np.empty(1) if a.data is not b.data: assert id(a.data) != id(b.data) # -- fail I'm not familiar with the internals, but: In [27]: a = np.empty(1) In [28]: a.data Out[28]: read-write buffer for 0x122df30, size 8, offset 0 at 0x3501860 so .data is a buffer object, wrapped around a particular pointer. In [29]: id(a.data) Out[29]: 55581376 In [30]: id(a.data) Out[30]: 55581728 but it looks like the buffer object itself is created on the fly -- so you are getting a different pyton object, even though it wraps the same data pointer. So you need to compare the value of the pointer compared, not the identity of the buffer object. Not sure how to get that value though, other that parsing __repr__ In [44]: a.data.__repr__().split()[-1][:-1] Out[44]: '0x3512100' ugly hack -- there must be a better way! But: In [38]: a = np.zeros((100,)) In [39]: b = a[50:] In [40]: a.data Out[40]: read-write buffer for 0x1443990, size 800, offset 0 at 0x3501da0 In [41]: b.data Out[41]: read-write buffer for 0x1617130, size 400, offset 0 at 0x3501c20 a and b share memory, but don't have the same data pointer as b's is offset. HTH, -Chris I'm trying to write an alternate may_share_memory function. Thanks, - James ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/ORR(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Funny business with 'is' operator?
On Fri, Sep 6, 2013 at 10:19 AM, James Bergstra bergs...@iro.umontreal.cawrote: Hi, could someone help me understand why this assertion fails? def test_is(self): a = np.empty(1) b = np.empty(1) if a.data is not b.data: assert id(a.data) != id(b.data) # -- fail I'm trying to write an alternate may_share_memory function. id seems not as advertised: In [22]: for i in range(10): print id(a.data) 66094640 66095792 66095792 66095792 66095792 66095792 66094640 66094640 66094640 66094640 Not sure why. Chuck ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Funny business with 'is' operator?
On Fri, Sep 6, 2013 at 5:58 PM, James Bergstra bergs...@iro.umontreal.ca wrote: I'm stumped. I can't figure out how to extract from e.g. view = A[:, 3] that the view starts at element 3 of A. I was planning to make a may_share_memory implementation based on the idea of swapping in a buffer of 0s, and using the shapes, strides, itemsize etc. to increment just the parts of the 0s buffer touched by the two ndarrays. If there are any 2s in the incremented buffer, it's because the two views overlapped. It's not the best algorithm for comparing tiny views of huge arrays, I was wondering if in my case it would have been quicker than the built-in method (I don't know how it works). It certainly won't be faster. may_share_memory() is very simplistic. It just checks if the outermost bounds of the each array overlap. Thus, it can give false positives for arrays that are interleaved but do not actually intersect. This is why it is named may_share_memory() instead of does_share_memory(). I actually have another data structure around to pull out that shape and stride info, but it's a shame to use it, because then I can't use the algorithm to compare ndarrays in general (or at least ones that have been created by typical construction methods and slicing). All of the information you need is stored in the __array_interface__ attribute of an ndarray. The right way to solve this is to solve a bounded, linear Diophantine equation. That's where you should be looking if you want to crack this problem. -- Robert Kern ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Funny business with 'is' operator?
Thanks for the tips! FWIW my guess is that since '.data' is dynamically generated property rather than an attribute, it is being freed and re-allocated in the loop, and once for each of my id() expressions. On Fri, Sep 6, 2013 at 12:32 PM, Charles R Harris charlesr.har...@gmail.com wrote: On Fri, Sep 6, 2013 at 10:19 AM, James Bergstra bergs...@iro.umontreal.ca wrote: Hi, could someone help me understand why this assertion fails? def test_is(self): a = np.empty(1) b = np.empty(1) if a.data is not b.data: assert id(a.data) != id(b.data) # -- fail I'm trying to write an alternate may_share_memory function. id seems not as advertised: In [22]: for i in range(10): print id(a.data) 66094640 66095792 66095792 66095792 66095792 66095792 66094640 66094640 66094640 66094640 Not sure why. Chuck ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion -- http://www-etud.iro.umontreal.ca/~bergstrj ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] Funny business with 'is' operator?
Hi, could someone help me understand why this assertion fails? def test_is(self): a = np.empty(1) b = np.empty(1) if a.data is not b.data: assert id(a.data) != id(b.data) # -- fail I'm trying to write an alternate may_share_memory function. Thanks, - James ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Really cruel draft of vbench setup for NumPy (.add.reduce benchmarks since 2011)
On 6 September 2013 21:21, Yaroslav Halchenko li...@onerussian.com wrote: some old ones are still there, some might be specific to my CPU here How long does one run take? Maybe I can run it in my machine (Intel i5) for comparison. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Really cruel draft of vbench setup for NumPy (.add.reduce benchmarks since 2011)
On Fri, Sep 6, 2013 at 3:21 PM, Yaroslav Halchenko li...@onerussian.com wrote: FWIW -- updated runs of the benchmarks are available at http://yarikoptic.github.io/numpy-vbench which now include also maintenance/1.8.x branch (no divergences were detected yet). There are only recent improvements as I see and no new (but some old ones are still there, some might be specific to my CPU here) performance regressions. You would have enough data to add some quality control bands to the charts (like cusum chart for example). Then it would be possible to send a congratulation note or ring an alarm bell without looking at all the plots. Josef Cheers, -- Yaroslav O. Halchenko, Ph.D. http://neuro.debian.net http://www.pymvpa.org http://www.fail2ban.org Senior Research Associate, Psychological and Brain Sciences Dept. Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755 Phone: +1 (603) 646-9834 Fax: +1 (603) 646-1419 WWW: http://www.linkedin.com/in/yarik ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Really cruel draft of vbench setup for NumPy (.add.reduce benchmarks since 2011)
On Fri, Sep 6, 2013 at 1:21 PM, Yaroslav Halchenko li...@onerussian.comwrote: FWIW -- updated runs of the benchmarks are available at http://yarikoptic.github.io/numpy-vbench which now include also maintenance/1.8.x branch (no divergences were detected yet). There are only recent improvements as I see and no new (but some old ones are still there, some might be specific to my CPU here) performance regressions. This work is really nice. Thank you Yaroslav. Chuck ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Really cruel draft of vbench setup for NumPy (.add.reduce benchmarks since 2011)
FWIW -- updated runs of the benchmarks are available at http://yarikoptic.github.io/numpy-vbench which now include also maintenance/1.8.x branch (no divergences were detected yet). There are only recent improvements as I see and no new (but some old ones are still there, some might be specific to my CPU here) performance regressions. Cheers, -- Yaroslav O. Halchenko, Ph.D. http://neuro.debian.net http://www.pymvpa.org http://www.fail2ban.org Senior Research Associate, Psychological and Brain Sciences Dept. Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755 Phone: +1 (603) 646-9834 Fax: +1 (603) 646-1419 WWW: http://www.linkedin.com/in/yarik ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] Join and stacking for structured arrays
For the astropy Table class (which wraps numpy structured arrays), I wrote functions that perform table joins and concatenate tables along rows or columns. These are reasonably full-featured and handle most of the common needs for these operations. The join function here addresses some limitations in the lib.recfunctions.join_by() function. You can see the documentation for the Table interface to these functions at [1]. The reason I'm writing is that the implementation is split into a pure numpy-based module (no astropy dependence [5]) and the astropy bit. Essentially all of the real work is done in the numpy-based module, so I'm wondering if the numpy project is interested in this code for inclusion into numpy. If so, I'd be happy to put together a pull request. I would just need direction on: - Where to put it - What function names to use - How to setup for included Cython code The Python and Cython code and tests can be see at [2], [3], [4]. The tests are currently based on astropy Tables, so these would be modified to use plain ndarray or masked arrays. Cheers, Tom [1]: http://astropy.readthedocs.org/en/latest/table/operations.html [2]: https://github.com/astropy/astropy/blob/master/astropy/table/np_utils.py [3]: https://github.com/astropy/astropy/blob/master/astropy/table/_np_utils.pyx [4]: https://github.com/astropy/astropy/blob/master/astropy/table/tests/test_operations.py [5]: There is a requirement for an OrderedDict which is currently provided by astropy for python 2.6. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Funny business with 'is' operator?
Thanks, this is exactly what I was looking for. I'll look into what this Diophantine equation is. Also, relatedly, a few months ago Julian Taylor at least wrote what was there in C, which made it faster, if not better. - James On Fri, Sep 6, 2013 at 1:27 PM, Robert Kern robert.k...@gmail.com wrote: On Fri, Sep 6, 2013 at 5:58 PM, James Bergstra bergs...@iro.umontreal.ca wrote: I'm stumped. I can't figure out how to extract from e.g. view = A[:, 3] that the view starts at element 3 of A. I was planning to make a may_share_memory implementation based on the idea of swapping in a buffer of 0s, and using the shapes, strides, itemsize etc. to increment just the parts of the 0s buffer touched by the two ndarrays. If there are any 2s in the incremented buffer, it's because the two views overlapped. It's not the best algorithm for comparing tiny views of huge arrays, I was wondering if in my case it would have been quicker than the built-in method (I don't know how it works). It certainly won't be faster. may_share_memory() is very simplistic. It just checks if the outermost bounds of the each array overlap. Thus, it can give false positives for arrays that are interleaved but do not actually intersect. This is why it is named may_share_memory() instead of does_share_memory(). I actually have another data structure around to pull out that shape and stride info, but it's a shame to use it, because then I can't use the algorithm to compare ndarrays in general (or at least ones that have been created by typical construction methods and slicing). All of the information you need is stored in the __array_interface__ attribute of an ndarray. The right way to solve this is to solve a bounded, linear Diophantine equation. That's where you should be looking if you want to crack this problem. -- Robert Kern ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion -- http://www-etud.iro.umontreal.ca/~bergstrj ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Funny business with 'is' operator?
I'm stumped. I can't figure out how to extract from e.g. view = A[:, 3] that the view starts at element 3 of A. I was planning to make a may_share_memory implementation based on the idea of swapping in a buffer of 0s, and using the shapes, strides, itemsize etc. to increment just the parts of the 0s buffer touched by the two ndarrays. If there are any 2s in the incremented buffer, it's because the two views overlapped. It's not the best algorithm for comparing tiny views of huge arrays, I was wondering if in my case it would have been quicker than the built-in method (I don't know how it works). I actually have another data structure around to pull out that shape and stride info, but it's a shame to use it, because then I can't use the algorithm to compare ndarrays in general (or at least ones that have been created by typical construction methods and slicing). On Fri, Sep 6, 2013 at 12:48 PM, James Bergstra bergs...@iro.umontreal.cawrote: Thanks for the tips! FWIW my guess is that since '.data' is dynamically generated property rather than an attribute, it is being freed and re-allocated in the loop, and once for each of my id() expressions. On Fri, Sep 6, 2013 at 12:32 PM, Charles R Harris charlesr.har...@gmail.com wrote: On Fri, Sep 6, 2013 at 10:19 AM, James Bergstra bergs...@iro.umontreal.ca wrote: Hi, could someone help me understand why this assertion fails? def test_is(self): a = np.empty(1) b = np.empty(1) if a.data is not b.data: assert id(a.data) != id(b.data) # -- fail I'm trying to write an alternate may_share_memory function. id seems not as advertised: In [22]: for i in range(10): print id(a.data) 66094640 66095792 66095792 66095792 66095792 66095792 66094640 66094640 66094640 66094640 Not sure why. Chuck ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion -- http://www-etud.iro.umontreal.ca/~bergstrj -- http://www-etud.iro.umontreal.ca/~bergstrj ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
