Re: [Numpy-discussion] [numfocus] Growing the contributor base of Numpy
On Mon, Mar 25, 2013 at 7:56 PM, Jonathan Rocher jroc...@enthought.comwrote: Dear all, One recurring question is how to *grow the contributor base* to NumPy and provide help and relief to core developers and maintainers. One way to do this would be to *leverage the upcoming SciPy conference*in 2 ways: 1. Provide an intermediate or advanced level tutorial on NumPy focusing on teaching the C-API and the architecture of the package to help people navigate the source code, and find answers to precise deep questions. I think that many users would be interested in being better able to understand the underlayers to become powerful users (and contributors if they want to). 2. Organize a Numpy sprint to leverage all this freshly graduated students apply what they learned to tackle some of the work under the guidance of core developers. This would be a great occasion to share and grow knowledge that is fundamental to our community. And the fact that the underlayers are in C is fine IMHO: SciPy is about scientific programming in Python and that is done with a lot of C. *Thoughts? Anyone interested in leading a tutorial (can be a team of people)? Anyone willing to coordinate the sprint? Who would be willing to be present and help during the sprint? * First thought: excellent initiative. I'm not going to be at SciPy, but I'm happy to coordinate a numpy/scipy sprint at EuroScipy. Going to email the organizers right now. Ralf Note that there is less than 1 week left until the tutorial submission deadline. I am happy to help brainstorm on this to make it happen. Thanks, Jonathan and Andy, for the SciPy2013 organizers -- Jonathan Rocher, PhD Scientific software developer SciPy2013 conference co-chair Enthought, Inc. jroc...@enthought.com 1-512-536-1057 http://www.enthought.com -- You received this message because you are subscribed to the Google Groups NumFOCUS group. To unsubscribe from this group and stop receiving emails from it, send an email to numfocus+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] variables not defined in numpy.random__init.py__ ?
@ Ralf. I missed info.py at the top and it is a valid statement. @ Brad. My project is using Numpy and Scipy and falls over at this point when using PyInstaller. One of the project source files has an import random from the Standard Library. As you say, at this point in tempfile.py, it is attempting to import random from the Standard Library but instead is importing the one from Numpy (numpy.random). How can this be fixed? Or, is it something for PyInstaller to fix? Thx. From: Bradley M. Froehle Sent: Monday, March 25, 2013 1:26 PM To: Discussion of Numerical Python Subject: Re: [Numpy-discussion] variables not defined in numpy.random__init.py__ ? On Mon, Mar 25, 2013 at 12:51 PM, Ralf Gommers ralf.gomm...@gmail.com wrote: On Mon, Mar 25, 2013 at 4:23 PM, Dinesh B Vadhia dineshbvad...@hotmail.com wrote: Using PyInstaller, the following error occurs: Traceback (most recent call last): File string, line 9, in module File //usr/lib/python2.7/dist-packages/PIL/Image.py, line 355, in init __import__(f, globals(), locals(), []) File //usr/lib/python2.7/dist-packages/PIL/IptcImagePlugin.py, line 23, in module import os, tempfile File /usr/lib/python2.7/tempfile.py, line 34, in module from random import Random as _Random File //usr/lib/python2.7/dist-packages/numpy/random/__init__.py, line 90, in module ranf = random = sample = random_sample NameError: name 'random_sample' is not defined Is line 90 in __init.py__ valid? It is. In my reading of this the main problem is that `tempfile` is trying to import `random` from the Python standard library but instead is importing the one from within NumPy (i.e., `numpy.random`). I suspect that somehow `sys.path` is being set incorrectly --- perhaps because of the `PYTHONPATH` environment variable. -Brad___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Implementing a find first style function
Bump. I'd be interested to know if this is a desirable feature for numpy? (specifically the 1D find functionality rather than the any/all also discussed) If so, I'd be more than happy to submit a PR, but I don't want to put in the effort if the principle isn't desirable in the core of numpy. Cheers, On 8 March 2013 17:38, Phil Elson pelson@gmail.com wrote: Interesting. I hadn't thought of those. I've implemented (very roughly without a sound logic check) and benchmarked: def my_any(a, predicate, chunk_size=2048): try: next(find(a, predicate, chunk_size)) return True except StopIteration: return False def my_all(a, predicate, chunk_size=2048): return not my_any(a, lambda a: ~predicate(a), chunk_size) With the following setup: import numpy as np import numpy.random np.random.seed(1) a = np.random.randn(1e8) For a low frequency *any*: In [12]: %timeit (np.abs(a) 6).any() 1 loops, best of 3: 1.29 s per loop In [13]: %timeit my_any(a, lambda a: np.abs(a) 6) 1 loops, best of 3: 792 ms per loop In [14]: %timeit my_any(a, lambda a: np.abs(a) 6, chunk_size=1) 1 loops, best of 3: 654 ms per loop For a False *any*: In [16]: %timeit (np.abs(a) 7).any() 1 loops, best of 3: 1.22 s per loop In [17]: %timeit my_any(a, lambda a: np.abs(a) 7) 1 loops, best of 3: 2.4 s per loop For a high probability *any*: In [28]: %timeit (np.abs(a) 1).any() 1 loops, best of 3: 972 ms per loop In [27]: %timeit my_any(a, lambda a: np.abs(a) 1) 1 loops, best of 3: 67 us per loop --- For a low probability *all*: In [18]: %timeit (np.abs(a) 6).all() 1 loops, best of 3: 1.16 s per loop In [19]: %timeit my_all(a, lambda a: np.abs(a) 6) 1 loops, best of 3: 880 ms per loop In [20]: %timeit my_all(a, lambda a: np.abs(a) 6, chunk_size=1) 1 loops, best of 3: 706 ms per loop For a True *all*: In [22]: %timeit (np.abs(a) 7).all() 1 loops, best of 3: 1.47 s per loop In [23]: %timeit my_all(a, lambda a: np.abs(a) 7) 1 loops, best of 3: 2.65 s per loop For a high probability *all*: In [25]: %timeit (np.abs(a) 1).all() 1 loops, best of 3: 978 ms per loop In [26]: %timeit my_all(a, lambda a: np.abs(a) 1) 1 loops, best of 3: 73.6 us per loop On 6 March 2013 21:16, Benjamin Root ben.r...@ou.edu wrote: On Tue, Mar 5, 2013 at 9:15 AM, Phil Elson pelson@gmail.com wrote: The ticket https://github.com/numpy/numpy/issues/2269 discusses the possibility of implementing a find first style function which can optimise the process of finding the first value(s) which match a predicate in a given 1D array. For example: a = np.sin(np.linspace(0, np.pi, 200)) print find_first(a, lambda a: a 0.9) ((71, ), 0.900479032457) This has been discussed in several locations: https://github.com/numpy/numpy/issues/2269 https://github.com/numpy/numpy/issues/2333 http://stackoverflow.com/questions/7632963/numpy-array-how-to-find-index-of-first-occurrence-of-item *Rationale* For small arrays there is no real reason to avoid doing: a = np.sin(np.linspace(0, np.pi, 200)) ind = (a 0.9).nonzero()[0][0] print (ind, ), a[ind] (71,) 0.900479032457 But for larger arrays, this can lead to massive amounts of work even if the result is one of the first to be computed. Example: a = np.arange(1e8) print (a == 5).nonzero()[0][0] 5 So a function which terminates when the first matching value is found is desirable. As mentioned in #2269, it is possible to define a consistent ordering which allows this functionality for 1D arrays, but IMHO it overcomplicates the problem and was not a case that I personally needed, so I've limited the scope to 1D arrays only. *Implementation* My initial assumption was that to get any kind of performance I would need to write the *find* function in C, however after prototyping with some array chunking it became apparent that a trivial python function would be quick enough for my needs. The approach I've implemented in the code found in #2269 simply breaks the array into sub-arrays of maximum length *chunk_size* (2048 by default, though there is no real science to this number), applies the given predicating function, and yields the results from *nonzero()*. The given function should be a python function which operates on the whole of the sub-array element-wise (i.e. the function should be vectorized). Returning a generator also has the benefit of allowing users to get the first *n* matching values/indices. *Results* I timed the implementation of *find* found in my comment at https://github.com/numpy/numpy/issues/2269#issuecomment-14436725 with an obvious test: In [1]: from np_utils import find In [2]: import numpy as np In [3]: import numpy.random In [4]: np.random.seed(1) In [5]: a = np.random.randn(1e8) In [6]: a.min(), a.max() Out[6]: (-6.1194900990552776, 5.9632246301166321) In [7]: next(find(a,
[Numpy-discussion] howto reduce along arbitrary axis
In the following code, the function maxstar is applied along the last axis. Can anyone suggest how to modify this to apply reduction along a user-specified axis? def maxstar2 (a, b): return max (a, b) + log1p (exp (-abs (a - b))) def maxstar (u): s = u.shape[-1] if s == 1: return u[...,0] elif s == 2: return maxstar2 (u[...,0], u[...,1]) else: return maxstar2 ( maxstar (u[...,:s/2]), maxstar (u[...,s/2:])) ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] howto reduce along arbitrary axis
Hi Neal, I forward you this mail which I think might be of help to your question. Chao -- Forwarded message -- From: Chao YUE chaoyue...@gmail.com Date: Sat, Mar 16, 2013 at 5:40 PM Subject: indexing of arbitrary axis and arbitrary slice? To: Discussion of Numerical Python numpy-discussion@scipy.org Dear all, Is there some way to index the numpy array by specifying arbitrary axis and arbitrary slice, while not knowing the actual shape of the data? For example, I have a 3-dim data, data.shape = (3,4,5) Is there a way to retrieve data[:,0,:] by using something like np.retrieve_data(data,axis=2,slice=0), by this way you don't have to know the actual shape of the array. for for 4-dim data, np.retrieve_data(data,axis=2,slice=0) will actually be data[:,0,:,:] thanks in advance, Chao -- *** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 -- *** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] howto reduce along arbitrary axis
Oh sorry, my fault... here is the answer by Nathaniel Smith: def retrieve_data(a, ax, idx): full_idx = [slice(None)] * a.ndim full_idx[ax] = idx return a[tuple(full_idx)] Or for the specific case where you do know the axis in advance, you just don't know how many trailing axes there are, use a[:, :, 0, ...] and the ... will expand to represent the appropriate number of :'s. probably you can sue something simlaer. Chao On Tue, Mar 26, 2013 at 3:33 PM, Neal Becker ndbeck...@gmail.com wrote: Thank you, but do you also have an answer to this question? I only see the question. On Tue, Mar 26, 2013 at 10:23 AM, Chao YUE chaoyue...@gmail.com wrote: Hi Neal, I forward you this mail which I think might be of help to your question. Chao -- Forwarded message -- From: Chao YUE chaoyue...@gmail.com Date: Sat, Mar 16, 2013 at 5:40 PM Subject: indexing of arbitrary axis and arbitrary slice? To: Discussion of Numerical Python numpy-discussion@scipy.org Dear all, Is there some way to index the numpy array by specifying arbitrary axis and arbitrary slice, while not knowing the actual shape of the data? For example, I have a 3-dim data, data.shape = (3,4,5) Is there a way to retrieve data[:,0,:] by using something like np.retrieve_data(data,axis=2,slice=0), by this way you don't have to know the actual shape of the array. for for 4-dim data, np.retrieve_data(data,axis=2,slice=0) will actually be data[:,0,:,:] thanks in advance, Chao -- *** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 -- *** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 -- *** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] [numfocus] Growing the contributor base of Numpy
On Mon, Mar 25, 2013 at 6:56 PM, Jonathan Rocher jroc...@enthought.com wrote: Dear all, One recurring question is how to grow the contributor base to NumPy and provide help and relief to core developers and maintainers. One way to do this would be to leverage the upcoming SciPy conference in 2 ways: Provide an intermediate or advanced level tutorial on NumPy focusing on teaching the C-API and the architecture of the package to help people navigate the source code, and find answers to precise deep questions. I think that many users would be interested in being better able to understand the underlayers to become powerful users (and contributors if they want to). Organize a Numpy sprint to leverage all this freshly graduated students apply what they learned to tackle some of the work under the guidance of core developers. This would be a great occasion to share and grow knowledge that is fundamental to our community. And the fact that the underlayers are in C is fine IMHO: SciPy is about scientific programming in Python and that is done with a lot of C. Thoughts? Anyone interested in leading a tutorial (can be a team of people)? Anyone willing to coordinate the sprint? Who would be willing to be present and help during the sprint? I would be happy to be part of the team doing it, David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] ANN: NumPy 1.7.1rc1 release
On Sun, Mar 24, 2013 at 8:00 PM, Charles R Harris charlesr.har...@gmail.com wrote: On Sun, Mar 24, 2013 at 3:02 PM, Ondřej Čertík ondrej.cer...@gmail.com wrote: Hi, I'm pleased to announce the availability of the first release candidate of NumPy 1.7.1rc1. Sources and binary installers can be found at https://sourceforge.net/projects/numpy/files/NumPy/1.7.1rc1/ Please test it and report any bugs. It fixes a few bugs, listed below. I would like to thank everybody who contributed patches to this release: Nathaniel J. Smith, Sebastian Berg, Charles Harris, Bradley M. Froehle, Ralf Gommers, Christoph Gohlke, Mark Wiebe and Maximilian Albert. Cheers, Ondrej = NumPy 1.7.1 Release Notes = This is a bugfix only release in the 1.7.x series. Issues fixed gh-2973 Fix `1` is printed during numpy.test() gh-2983 BUG: gh-2969: Backport memory leak fix 80b3a34. gh-3007 Backport gh-3006 gh-2984 Backport fix complex polynomial fit gh-2982 BUG: Make nansum work with booleans. gh-2985 Backport large sort fixes gh-3039 Backport object take gh-3105 Backport nditer fix op axes initialization gh-3108 BUG: npy-pkg-config ini files were missing after Bento build. gh-3124 BUG: PyArray_LexSort allocates too much temporary memory. gh-3131 BUG: Exported f2py_size symbol prevents linking multiple f2py modules. gh-3117 Backport gh-2992 gh-3135 DOC: Add mention of PyArray_SetBaseObject stealing a reference gh-3134 DOC: Fix typo in fft docs (the indexing variable is 'm', not 'n'). gh-3136 Backport #3128 Checksums = 28c3f3e71b5eaa6bfab6e8340dbd35e7 release/installers/numpy-1.7.1rc1.tar.gz 436f416dee10d157314bd9da7ab95c9c release/installers/numpy-1.7.1rc1-win32-superpack-python2.7.exe a543c8cf69f66ff2b4c9565646105863 release/installers/numpy-1.7.1rc1-win32-superpack-python2.6.exe 6dfcbbd449b7fe4e841c5fd1bfa7af7c release/installers/numpy-1.7.1rc1-win32-superpack-python2.5.exe 22912792a1b6155ae2bdbc30bee8fadc release/installers/numpy-1.7.1rc1-win32-superpack-python3.2.exe 95bc5a5fcce9fcbc2717a774dccae31b release/installers/numpy-1.7.1rc1-win32-superpack-python3.3.exe 33cf283765a148846b49b89fb96d67d5 release/installers/numpy-1.7.1rc1-win32-superpack-python3.1.exe 9761de4b35493fed38c5d177da9c3b37 release/installers/numpy-1.7.1rc1.zip __ Great. The fix for the memory leak should make some folks happy. Yes. I created an issue here for them to test it: https://github.com/scikit-learn/scikit-learn/issues/1809 Just to make sure. Ondrej ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Any plans for windows 64-bit installer for 1.7?
Hi Ondrej, On Thu, Feb 7, 2013 at 3:18 PM, Ondřej Čertík ondrej.cer...@gmail.com wrote: On Thu, Feb 7, 2013 at 12:29 PM, Chris Barker - NOAA Federal chris.bar...@noaa.gov wrote: On Thu, Feb 7, 2013 at 11:38 AM, Matthew Brett matthew.br...@gmail.com wrote: a) If we cannot build Scipy now, it may or may not be acceptable to release numpy now. I think it is, you (Ralf) think it isn't, we haven't discussed that. It may not come up. Is anyone suggesting we hold the whole release for this? I fnot, then Just to make it clear, I do not plan to hold the whole release because of this. Previous releases also didn't have this official 64bit Windows binary, so there is no regression. Once we figure out how to create 64bit binaries, then we'll start uploading them. Did you make any progress with this? Worth making some notes? Anything we can do to help? Cheers, Matthew ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
