Re: [Numpy-discussion] Is there a pure numpy recipe for this?
Chris Barker - NOAA Federal wrote note that numpy arrays are not re-sizable, so np.append() and np.insert() have to make a new array, and copy all the old data over. If you are appending one at a time, this can be pretty darn slow. I wrote a grow_array class once, it was a wrapper around a numpy array that pre-allocated extra data to make appending more efficient. It's kind of half-baked code now, but let me know if you are interested. Hi Chris, Yes, it is a good point and I am aware of it. For some of these functions it would have been nice if i could have parsed a preallocated, properly sliced array to the functions, which i could then reuse in each iteration step. It is indeed the memory allocation which appear to take more time than the actual calculations. Still it is much faster to create a few arrays than to loop through a thousand individual elements in pure Python. Interesting with the grow_array class. I think that what I have for now is sufficient, but i will keep your offer in mind:) --Slaunger -- View this message in context: http://numpy-discussion.10968.n7.nabble.com/Is-there-a-pure-numpy-recipe-for-this-tp37077p37102.html Sent from the Numpy-discussion mailing list archive at Nabble.com. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Windows wheels using MKL?
On Thu, Mar 27, 2014 at 12:29 AM, Matthew Brett matthew.br...@gmail.com wrote: Hi, On Wed, Mar 26, 2014 at 4:48 PM, Matthew Brett matthew.br...@gmail.com wrote: Hi, Can I check what is stopping us building official numpy binary wheels for Windows using the Intel Math Kernel Library? * We'd need developer licenses, but those sound like they would be easy to come by * We'd have to add something to the license for the wheel on the lines of the Canopy license [1], derived from the MKL license [2] - is that a problem? Are there other problems for numpy? Talking with Fernando, we identified these as being the key problem clauses in the MKL license [1]: start quote D. DISTRIBUTION: Distribution of the Redistributables is also subject to the following limitations: [snipped clauses] (iv) shall use a license agreement that prohibits disassembly and reverse engineering of the Redistributables, (v) shall indemnify, hold harmless, and defend Intel and its suppliers from and against any claims or lawsuits, including attorney's fees, that arise or result from your distribution of any product. /end quote The first is a problem that might conceivably be adequately solved by adding a paragraph to the Pypi page for numpy (If you download and install the windows binaries, you also agree... ) and copying a new clause into the license in the installed tree. Maybe. The second looks like it would be very hard to deal with for open source project like us It would be confusing to distribute these non-BSD wheels on the same PyPI page that declares most prominently that numpy is BSD-licensed. Adding some text elsewhere on the PyPI page is not going to help very much: people look at the License: BSD first and foremost. Nothing stops anyone else from building and distributing MKL-built binaries, a la C. Gohlke, but I don't think it is wise to do so on the PyPI page. -- Robert Kern ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Default builds of OpenBLAS development branch are now fork safe
2014-03-26 16:27 GMT+01:00 Olivier Grisel olivier.gri...@ensta.org: Hi Carl, I installed Python 2.7.6 64 bits on a windows server instance from rackspace cloud and then ran get-pip.py and then could successfully install the numpy and scipy wheel packages from your google drive folder. I tested dot products and scipy.linalg.svd and they work as expected. Then I uncompressed your mingw toolchain in c:\mingw, put c:\mingw\bin in my PATH and tried to build the scikit-learn git master with it, however it fails with: building 'sklearn.__check_build._check_build' extension compiling C sources C compiler: gcc -DMS_WIN64 -O2 -msse -msse2 -Wall -Wstrict-prototypes compile options: '-D__MSVCRT_VERSION__=0x0900 -Ic:\Python27\lib\site-packages\numpy\core\include -Ic:\Python27\lib\site-packages\numpy\core\include -Ic:\Python2 7\include -Ic:\Python27\PC -c' gcc -DMS_WIN64 -O2 -msse -msse2 -Wall -Wstrict-prototypes -D__MSVCRT_VERSION__=0x0900 -Ic:\Python27\lib\site-packages\numpy\core\include -Ic:\Python27\lib\site- packages\numpy\core\include -Ic:\Python27\include -Ic:\Python27\PC -c sklearn\__check_build\_check_build.c -o build\temp.win-amd64-2.7\Release\sklearn\__check_b uild\_check_build.o Found executable c:\mingw\bin\gcc.exe gcc -shared -Wl,-gc-sections -Wl,-s build\temp.win-amd64-2.7\Release\sklearn\__check_build\_check_build.o -Lc:\Python27\libs -Lc:\Python27\PCbuild\amd64 -Lbuild \temp.win-amd64-2.7 -lpython27 -lmsvcr90 -o build\lib.win-amd64-2.7\sklearn\__check_build\_check_build.pyd build\temp.win-amd64-2.7\Release\sklearn\__check_build\_check_build.o:_check_build.c:(.text+0x3): undefined reference to `__imp__Py_NoneStruct' build\temp.win-amd64-2.7\Release\sklearn\__check_build\_check_build.o:_check_build.c:(.text+0x1ca): undefined reference to `__imp__PyThreadState_Current' build\temp.win-amd64-2.7\Release\sklearn\__check_build\_check_build.o:_check_build.c:(.text+0x405): undefined reference to `__imp_PyExc_ImportError' c:/mingw/bin/../lib/gcc/x86_64-w64-mingw32/4.8.2/../../../../x86_64-w64-mingw32/bin/ld.exe: build\temp.win-amd64-2.7\Release\sklearn\__check_build\_check_build. o: bad reloc address 0x0 in section `.data' collect2.exe: error: ld returned 1 exit status error: Command gcc -shared -Wl,-gc-sections -Wl,-s build\temp.win-amd64-2.7\Release\sklearn\__check_build\_check_build.o -Lc:\Python27\libs -Lc:\Python27\PCbui ld\amd64 -Lbuild\temp.win-amd64-2.7 -lpython27 -lmsvcr90 -o build\lib.win-amd64-2.7\sklearn\__check_build\_check_build.pyd failed with exit status 1 Ignore that, I had forgotten to copy the libpython17.a file in c:\Python27\libs on that instance. Building scikit-learn works with the static toolchain. I have failing tests but those are probably not related to the toolchain. -- Olivier http://twitter.com/ogrisel - http://github.com/ogrisel ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Default builds of OpenBLAS development branch are now fork safe
On Wed, Mar 26, 2014 at 5:17 PM, Olivier Grisel olivier.gri...@ensta.org wrote: My understanding of Carl's effort is that the long term goal is to have official windows whl packages for both numpy and scipy published on PyPI with a builtin BLAS / LAPACK implementation so that users can do `pip install scipy` under windows and get something that just works without have to install any compiler (fortran or C) nor any additional library manually. Most windows users are beginners and you cannot really expect them to understand how to build the whole scipy stack from source. The current solution (executable setup installers) is not optimal as it requires Administrator rights to run, does not resolve dependencies as pip does and cannot be installed in virtualenvs. as small related point: The official installers can be used to install in virtualenv The way I do it: Run the superpack, official installer, wait until it extracts the correct (SSE) install exe, then cancel Then easy_install the install exe file that has been extracted to the temp folder into the virtualenv. I don't remember if the extraction already requires admin rights, but I think not. easy_install doesn't require any, IIRC. Josef If we can build numpy / scipy whl packages for windows with the Atlas dlls then fine embedded in the numpy package then good. It does not need to be the fastest BLAS / LAPACK lib in my opinion. Just something that works. The problem with ATLAS is that you need to select the number of thread at build time AFAIK. But we could set it to a reasonable default (e.g. 4 threads) for the default windows package. -- Olivier ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Default builds of OpenBLAS development branch are now fork safe
2014-03-27 14:55 GMT+01:00 josef.p...@gmail.com: On Wed, Mar 26, 2014 at 5:17 PM, Olivier Grisel olivier.gri...@ensta.org wrote: My understanding of Carl's effort is that the long term goal is to have official windows whl packages for both numpy and scipy published on PyPI with a builtin BLAS / LAPACK implementation so that users can do `pip install scipy` under windows and get something that just works without have to install any compiler (fortran or C) nor any additional library manually. Most windows users are beginners and you cannot really expect them to understand how to build the whole scipy stack from source. The current solution (executable setup installers) is not optimal as it requires Administrator rights to run, does not resolve dependencies as pip does and cannot be installed in virtualenvs. as small related point: The official installers can be used to install in virtualenv The way I do it: Run the superpack, official installer, wait until it extracts the correct (SSE) install exe, then cancel Then easy_install the install exe file that has been extracted to the temp folder into the virtualenv. I don't remember if the extraction already requires admin rights, but I think not. easy_install doesn't require any, IIRC. Hackish but interesting. Maybe the extraction can be done with generic tools like winzip? -- Olivier http://twitter.com/ogrisel - http://github.com/ogrisel ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Default builds of OpenBLAS development branch are now fork safe
On Thu, Mar 27, 2014 at 9:59 AM, Olivier Grisel olivier.gri...@ensta.org wrote: 2014-03-27 14:55 GMT+01:00 josef.p...@gmail.com: On Wed, Mar 26, 2014 at 5:17 PM, Olivier Grisel olivier.gri...@ensta.org wrote: My understanding of Carl's effort is that the long term goal is to have official windows whl packages for both numpy and scipy published on PyPI with a builtin BLAS / LAPACK implementation so that users can do `pip install scipy` under windows and get something that just works without have to install any compiler (fortran or C) nor any additional library manually. Most windows users are beginners and you cannot really expect them to understand how to build the whole scipy stack from source. The current solution (executable setup installers) is not optimal as it requires Administrator rights to run, does not resolve dependencies as pip does and cannot be installed in virtualenvs. as small related point: The official installers can be used to install in virtualenv The way I do it: Run the superpack, official installer, wait until it extracts the correct (SSE) install exe, then cancel Then easy_install the install exe file that has been extracted to the temp folder into the virtualenv. I don't remember if the extraction already requires admin rights, but I think not. easy_install doesn't require any, IIRC. Hackish but interesting. Maybe the extraction can be done with generic tools like winzip? I tried to open and unzip with WinRAR but couldn't make sense of the content. BTW: easy_install for other installers like matplotlib also works nicely for virtualenv However, the official installers are only for 32-bit python, and I appreciate all the efforts to modernize the numpy and scipy builds. Josef -- Olivier http://twitter.com/ogrisel - http://github.com/ogrisel ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Is there a pure numpy recipe for this?
I find this interesting, since I work with medical data sets of 100s of MB, and regularly run into memory allocation problems when doing a lot of Fourrier analysis, waterfalls etc. The per-process limit seems to be about 1.3GB on this 6GB quad-i7 with Win7. For live data collection routines I simply creates zeros() of say 300MB and trim the array when saving to disk. memmaps are also limited to RAM, and take a long time to create (seconds). So, I've been investigating Pandas and segmentaxis - just a bit so far. - Ray Schumacher At 12:02 AM 3/27/2014, you wrote: Chris Barker - NOAA Federal wrote note that numpy arrays are not re-sizable, so np.append() and np.insert() have to make a new array, and copy all the old data over. If you are appending one at a time, this can be pretty darn slow. I wrote a grow_array class once, it was a wrapper around a numpy array that pre-allocated extra data to make appending more efficient. It's kind of half-baked code now, but let me know if you are interested. Hi Chris, Yes, it is a good point and I am aware of it. For some of these functions it would have been nice if i could have parsed a preallocated, properly sliced array to the functions, which i could then reuse in each iteration step. It is indeed the memory allocation which appear to take more time than the actual calculations. Still it is much faster to create a few arrays than to loop through a thousand individual elements in pure Python. Interesting with the grow_array class. I think that what I have for now is sufficient, but i will keep your offer in mind:) --Slaunger ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Is there a pure numpy recipe for this?
You might want to look at hdf5 if you're routinely running out of ram. I'm using h5py with multi gigabyte data on an ssd right now. It is very fast. You still have to be careful with your computations and try to avoid creating copies though. hypy: www.h5py.org aaron On Thu 27 Mar, RayS wrote: I find this interesting, since I work with medical data sets of 100s of MB, and regularly run into memory allocation problems when doing a lot of Fourrier analysis, waterfalls etc. The per-process limit seems to be about 1.3GB on this 6GB quad-i7 with Win7. For live data collection routines I simply creates zeros() of say 300MB and trim the array when saving to disk. memmaps are also limited to RAM, and take a long time to create (seconds). So, I've been investigating Pandas and segmentaxis - just a bit so far. - Ray Schumacher At 12:02 AM 3/27/2014, you wrote: Chris Barker - NOAA Federal wrote note that numpy arrays are not re-sizable, so np.append() and np.insert() have to make a new array, and copy all the old data over. If you are appending one at a time, this can be pretty darn slow. I wrote a grow_array class once, it was a wrapper around a numpy array that pre-allocated extra data to make appending more efficient. It's kind of half-baked code now, but let me know if you are interested. Hi Chris, Yes, it is a good point and I am aware of it. For some of these functions it would have been nice if i could have parsed a preallocated, properly sliced array to the functions, which i could then reuse in each iteration step. It is indeed the memory allocation which appear to take more time than the actual calculations. Still it is much faster to create a few arrays than to loop through a thousand individual elements in pure Python. Interesting with the grow_array class. I think that what I have for now is sufficient, but i will keep your offer in mind:) --Slaunger ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Is there a pure numpy recipe for this?
On Thu, 27 Mar 2014 16:19:54 + Aaron O'Leary aaron.ole...@gmail.com wrote: You might want to look at hdf5 if you're routinely running out of ram. I'm using h5py with multi gigabyte data on an ssd right now. It is very fast. You still have to be careful with your computations and try to avoid creating copies though. Both for h5py and for memmapped files ... switching from windows to linux are likely to help ... -- Jérôme Kieffer tel +33 476 882 445 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Windows wheels using MKL?
Hi, On Thu, Mar 27, 2014 at 3:18 AM, Robert Kern robert.k...@gmail.com wrote: On Thu, Mar 27, 2014 at 12:29 AM, Matthew Brett matthew.br...@gmail.com wrote: Hi, On Wed, Mar 26, 2014 at 4:48 PM, Matthew Brett matthew.br...@gmail.com wrote: Hi, Can I check what is stopping us building official numpy binary wheels for Windows using the Intel Math Kernel Library? * We'd need developer licenses, but those sound like they would be easy to come by * We'd have to add something to the license for the wheel on the lines of the Canopy license [1], derived from the MKL license [2] - is that a problem? Are there other problems for numpy? Talking with Fernando, we identified these as being the key problem clauses in the MKL license [1]: start quote D. DISTRIBUTION: Distribution of the Redistributables is also subject to the following limitations: [snipped clauses] (iv) shall use a license agreement that prohibits disassembly and reverse engineering of the Redistributables, (v) shall indemnify, hold harmless, and defend Intel and its suppliers from and against any claims or lawsuits, including attorney's fees, that arise or result from your distribution of any product. /end quote The first is a problem that might conceivably be adequately solved by adding a paragraph to the Pypi page for numpy (If you download and install the windows binaries, you also agree... ) and copying a new clause into the license in the installed tree. Maybe. The second looks like it would be very hard to deal with for open source project like us It would be confusing to distribute these non-BSD wheels on the same PyPI page that declares most prominently that numpy is BSD-licensed. Adding some text elsewhere on the PyPI page is not going to help very much: people look at the License: BSD first and foremost. Nothing stops anyone else from building and distributing MKL-built binaries, a la C. Gohlke, but I don't think it is wise to do so on the PyPI page. Can you see any circumstances in which we could use the MKL binaries from pypi? Cheers, Matthew ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Windows wheels using MKL?
Hi, On Thu, Mar 27, 2014 at 12:10 PM, Matthew Brett matthew.br...@gmail.com wrote: Hi, On Thu, Mar 27, 2014 at 3:18 AM, Robert Kern robert.k...@gmail.com wrote: On Thu, Mar 27, 2014 at 12:29 AM, Matthew Brett matthew.br...@gmail.com wrote: Hi, On Wed, Mar 26, 2014 at 4:48 PM, Matthew Brett matthew.br...@gmail.com wrote: Hi, Can I check what is stopping us building official numpy binary wheels for Windows using the Intel Math Kernel Library? * We'd need developer licenses, but those sound like they would be easy to come by * We'd have to add something to the license for the wheel on the lines of the Canopy license [1], derived from the MKL license [2] - is that a problem? Are there other problems for numpy? Talking with Fernando, we identified these as being the key problem clauses in the MKL license [1]: start quote D. DISTRIBUTION: Distribution of the Redistributables is also subject to the following limitations: [snipped clauses] (iv) shall use a license agreement that prohibits disassembly and reverse engineering of the Redistributables, (v) shall indemnify, hold harmless, and defend Intel and its suppliers from and against any claims or lawsuits, including attorney's fees, that arise or result from your distribution of any product. /end quote The first is a problem that might conceivably be adequately solved by adding a paragraph to the Pypi page for numpy (If you download and install the windows binaries, you also agree... ) and copying a new clause into the license in the installed tree. Maybe. The second looks like it would be very hard to deal with for open source project like us It would be confusing to distribute these non-BSD wheels on the same PyPI page that declares most prominently that numpy is BSD-licensed. Adding some text elsewhere on the PyPI page is not going to help very much: people look at the License: BSD first and foremost. Nothing stops anyone else from building and distributing MKL-built binaries, a la C. Gohlke, but I don't think it is wise to do so on the PyPI page. Can you see any circumstances in which we could use the MKL binaries from pypi? Christoph - have you considered building binary wheels for the projects you support? If not, is there any help I / we can give? Cheers, Matthew ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Is there a pure numpy recipe for this?
On Thu, Mar 27, 2014 at 7:42 AM, RayS r...@blue-cove.com wrote: I find this interesting, since I work with medical data sets of 100s of MB, and regularly run into memory allocation problems when doing a lot of Fourrier analysis, waterfalls etc. The per-process limit seems to be about 1.3GB on this 6GB quad-i7 with Win7. This sounds like 32 bit -- have you tried a 64 bit Python_numpy? Nt that you wont have issues anyway, but you should be abel to do better than 1.3GB... memmaps are also limited to RAM, I don't think so, no -- but are limited to 2GB (I think) if you're using a 32 bit process There is also a compressed array package out there -- I can't remember what it's called -- but if you have large compressible arrays -- that might help. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/ORR(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Is there a pure numpy recipe for this?
Thanks for all of the suggestions; we are migrating to 64bit Python soon as well. The environments are Win7 and Mac Maverics. carray sounds like what you said Chris - more I just found at http://kmike.ru/python-data-structures/ - Ray Schumacher At 12:31 PM 3/27/2014, you wrote: On Thu, Mar 27, 2014 at 7:42 AM, RayS mailto:r...@blue-cove.comr...@blue-cove.com wrote: I find this interesting, since I work with medical data sets of 100s of MB, and regularly run into memory allocation problems when doing a lot of Fourrier analysis, waterfalls etc. The per-process limit seems to be about 1.3GB on this 6GB quad-i7 with Win7. This sounds like 32 bit -- have you tried a 64 bit Python_numpy? Nt that you wont have issues anyway, but you should be abel to do better than 1.3GB...   memmaps are also limited to RAM, I don't think so, no -- but are limited to 2GB (I think)  if you're using a 32 bit process There is also a compressed array package out there -- I can't remember what it's called -- but if you have large compressible arrays -- that might help.  -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/ORR       (206) 526-6959  voice 7600 Sand Point Way NE   (206) 526-6329  fax Seattle, WA  98115     (206) 526-6317  main reception mailto:chris.bar...@noaa.govchris.bar...@noaa.gov ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] f2py links extensions to incorrect python installation on OSX / Anaconda
Hi all, I have used f2py in the past on a Linux machine with virtually no issues. However on my Mac, I get the following error when importing an f2py generated extension: Fatal Python error: PyThreadState_Get: no current thread Abort trap: 6 After doing some research I found out that the extension is linked to the wrong python installation: otool -L add.so add.so: ./add.so (compatibility version 0.0.0, current version 0.0.0) /System/Library/Frameworks/Python.framework/Versions/2.7/Python (compatibility version 2.7.0, current version 2.7.2) /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 169.3.0) This seems odd because I am using the f2py executable included in Anaconda 1.9.1. I can easily fix this problem by manually using install_name_tool -change on the extension to link the correct library location, but this is really cumbersome. Is there an alternative solution, such as an additional command-line argument when invoking f2py? For what it is worth, I am also using Version 14.0.2 of the Intel Fortran Compiler. Thanks, Alex -- Alex Goodman Graduate Research Assistant Department of Atmospheric Science Colorado State University ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] f2py links extensions to incorrect python installation on OSX / Anaconda
On Thu, Mar 27, 2014 at 8:30 PM, Alex Goodman alex.good...@colostate.eduwrote: Hi all, I have used f2py in the past on a Linux machine with virtually no issues. However on my Mac, I get the following error when importing an f2py generated extension: Fatal Python error: PyThreadState_Get: no current thread Abort trap: 6 After doing some research I found out that the extension is linked to the wrong python installation: otool -L add.so add.so: ./add.so (compatibility version 0.0.0, current version 0.0.0) /System/Library/Frameworks/Python.framework/Versions/2.7/Python (compatibility version 2.7.0, current version 2.7.2) /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 169.3.0) This seems odd because I am using the f2py executable included in Anaconda 1.9.1. I can easily fix this problem by manually using install_name_tool -change on the extension to link the correct library location, but this is really cumbersome. Is there an alternative solution, such as an additional command-line argument when invoking f2py? This sounds like an issue specific to Anaconda, and you may get better support on the Anaconda support ML. David For what it is worth, I am also using Version 14.0.2 of the Intel Fortran Compiler. Thanks, Alex -- Alex Goodman Graduate Research Assistant Department of Atmospheric Science Colorado State University ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] f2py links extensions to incorrect python installation on OSX / Anaconda
On Thu, Mar 27, 2014 at 8:50 PM, David Cournapeau courn...@gmail.com wrote: On Thu, Mar 27, 2014 at 8:30 PM, Alex Goodman alex.good...@colostate.edu wrote: Hi all, I have used f2py in the past on a Linux machine with virtually no issues. However on my Mac, I get the following error when importing an f2py generated extension: Fatal Python error: PyThreadState_Get: no current thread Abort trap: 6 After doing some research I found out that the extension is linked to the wrong python installation: otool -L add.so add.so: ./add.so (compatibility version 0.0.0, current version 0.0.0) /System/Library/Frameworks/Python.framework/Versions/2.7/Python (compatibility version 2.7.0, current version 2.7.2) /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 169.3.0) This seems odd because I am using the f2py executable included in Anaconda 1.9.1. I can easily fix this problem by manually using install_name_tool -change on the extension to link the correct library location, but this is really cumbersome. Is there an alternative solution, such as an additional command-line argument when invoking f2py? This sounds like an issue specific to Anaconda, and you may get better support on the Anaconda support ML. I think it's our bug. numpy.distutils adds an explicit `-framework Python` in the Intel Fortran link line. We should be just be using `-undefined dynamic_lookup`. https://github.com/numpy/numpy/blob/master/numpy/distutils/fcompiler/intel.py#L71 Alex, can you edit that file to remove the '-Wl,-framework,Python' from that list and try building again? -- Robert Kern ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] [SciPy-Dev] Windows wheels using MKL?
On Thu, Mar 27, 2014 at 7:10 PM, Matthew Brett matthew.br...@gmail.com wrote: Hi, On Thu, Mar 27, 2014 at 3:18 AM, Robert Kern robert.k...@gmail.com wrote: It would be confusing to distribute these non-BSD wheels on the same PyPI page that declares most prominently that numpy is BSD-licensed. Adding some text elsewhere on the PyPI page is not going to help very much: people look at the License: BSD first and foremost. Nothing stops anyone else from building and distributing MKL-built binaries, a la C. Gohlke, but I don't think it is wise to do so on the PyPI page. Can you see any circumstances in which we could use the MKL binaries from pypi? No. Most of the point of adding binary wheels to PyPI would be to make `pip install numpy` work. That gives users *no* chance to see any documentation about the proprietary license of those binaries. -- Robert Kern ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] f2py links extensions to incorrect python installation on OSX / Anaconda
Hi Robert, That did the trick, thanks! Alex On Thu, Mar 27, 2014 at 3:02 PM, Robert Kern robert.k...@gmail.com wrote: On Thu, Mar 27, 2014 at 8:50 PM, David Cournapeau courn...@gmail.com wrote: On Thu, Mar 27, 2014 at 8:30 PM, Alex Goodman alex.good...@colostate.edu wrote: Hi all, I have used f2py in the past on a Linux machine with virtually no issues. However on my Mac, I get the following error when importing an f2py generated extension: Fatal Python error: PyThreadState_Get: no current thread Abort trap: 6 After doing some research I found out that the extension is linked to the wrong python installation: otool -L add.so add.so: ./add.so (compatibility version 0.0.0, current version 0.0.0) /System/Library/Frameworks/Python.framework/Versions/2.7/Python (compatibility version 2.7.0, current version 2.7.2) /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 169.3.0) This seems odd because I am using the f2py executable included in Anaconda 1.9.1. I can easily fix this problem by manually using install_name_tool -change on the extension to link the correct library location, but this is really cumbersome. Is there an alternative solution, such as an additional command-line argument when invoking f2py? This sounds like an issue specific to Anaconda, and you may get better support on the Anaconda support ML. I think it's our bug. numpy.distutils adds an explicit `-framework Python` in the Intel Fortran link line. We should be just be using `-undefined dynamic_lookup`. https://github.com/numpy/numpy/blob/master/numpy/distutils/fcompiler/intel.py#L71 Alex, can you edit that file to remove the '-Wl,-framework,Python' from that list and try building again? -- Robert Kern ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion -- Alex Goodman Graduate Research Assistant Department of Atmospheric Science Colorado State University ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Is there a pure numpy recipe for this?
Id recommend taking a look at pytables as well. It has support for out-of-core array computations on large arrays. On Thu, Mar 27, 2014 at 9:00 PM, RayS r...@blue-cove.com wrote: Thanks for all of the suggestions; we are migrating to 64bit Python soon as well. The environments are Win7 and Mac Maverics. carray sounds like what you said Chris - more I just found at http://kmike.ru/python-data-structures/ - Ray Schumacher At 12:31 PM 3/27/2014, you wrote: On Thu, Mar 27, 2014 at 7:42 AM, RayS r...@blue-cove.com wrote: I find this interesting, since I work with medical data sets of 100s of MB, and regularly run into memory allocation problems when doing a lot of Fourrier analysis, waterfalls etc. The per-process limit seems to be about 1.3GB on this 6GB quad-i7 with Win7. This sounds like 32 bit -- have you tried a 64 bit Python_numpy? Nt that you wont have issues anyway, but you should be abel to do better than 1.3GB...   memmaps are also limited to RAM, I don't think so, no -- but are limited to 2GB (I think)  if you're using a 32 bit process There is also a compressed array package out there -- I can't remember what it's called -- but if you have large compressible arrays -- that might help.  -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/ORR       (206) 526-6959  voice 7600 Sand Point Way NE   (206) 526-6329  fax Seattle, WA  98115     (206) 526-6317  main reception chris.bar...@noaa.gov ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] [SciPy-Dev] Windows wheels using MKL?
On Thu, Mar 27, 2014 at 2:04 PM, Robert Kern robert.k...@gmail.com wrote: On Thu, Mar 27, 2014 at 7:10 PM, Matthew Brett matthew.br...@gmail.com wrote: Hi, On Thu, Mar 27, 2014 at 3:18 AM, Robert Kern robert.k...@gmail.com wrote: It would be confusing to distribute these non-BSD wheels on the same PyPI page that declares most prominently that numpy is BSD-licensed. Adding some text elsewhere on the PyPI page is not going to help very much: people look at the License: BSD first and foremost. Nothing stops anyone else from building and distributing MKL-built binaries, a la C. Gohlke, but I don't think it is wise to do so on the PyPI page. Can you see any circumstances in which we could use the MKL binaries from pypi? No. Most of the point of adding binary wheels to PyPI would be to make `pip install numpy` work. That gives users *no* chance to see any documentation about the proprietary license of those binaries. OK - fair enough. Does anyone disagree? If not, I suggest we remove MKL from the options we consider in the future. Cheers, Matthew ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Dates and times and Datetime64 (again)
Hi all,Apologies for the delay in following up, here is an expanded version of the proposal, which hopefully clears up most of the details. I have not included specific implementation details for the code, such as which functions to modify etc. since I think those are not traditionally included in NEPs?Please find attached the expanded proposal, and the rendered version is available here:https://github.com/Sankarshan-Mudkavi/numpy/blob/Enhance-datetime64/doc/neps/datetime-improvement-proposal.rst datetime-improvement-proposal.rst Description: Binary data I look forward to comments, agreements/disagreements with this (and clarification if this needs even further expansion).Please find attached theOn Mar 24, 2014, at 12:39 AM, Chris Barker chris.bar...@noaa.gov wrote:On Fri, Mar 21, 2014 at 3:43 PM, Nathaniel Smith n...@pobox.com wrote: On Thu, Mar 20, 2014 at 11:27 PM, Chris Barker chris.bar...@noaa.gov wrote: * I think there are more or less three options: 1) a) don't have any timezone handling at all -- all datetime64s are UTC. Always b) don't have any timezone handling at all -- all datetime64s are naive (the only difference between these two is I/O of strings, and maybe I/O of datetime objects with a time zone) 2) Have a time zone associated with the array -- defaulting to either UTC or None, but don't provide any implementation other than the tagging, with the ability to add in TZ handler if you want (can this be done efficiently?) 3) Full on proper TZ handling. I think (3) is off the table for now. I think the first goal is to define what a plain vanilla datetime64 does, without any extra attributes. This is for two practical reasons: First, our overriding #1 goal is to fix the nasty I/O problems that default datetime64's show, so until that's done any other bells and whistles are a distraction. And second, adding parameters to dtypes right now is technically messy. This rules out (2) and (3).yup -- though I'm not sure I agree that we need to do this, if we are going to do something more later anyway. But you have a key point - maybe the dtype system simply isn't ready to do it right, and then it may be better not to try. In which case, we are down to naive or always UTC -- and again, those really aren't very different. Though I prefer naive -- always UTC adds some complication if you don't actually want UTC, and I'm not sure it actually buys us anything. And maybe it's jsut me, but all my code would need to use naive, so I"d be doing a bit of working around to use a UTC-always system. If we additionally want to keep the option of adding a timezone parameter later, and have the result end up looking like stdlib datetime, then I think 1(b) is the obvious choice. My guess is that this is also what's most compatible with pandas, which is currently keeping its own timezone object outside of the dtype.Good point, all else being equal, compatability with Pandas would be a good thing. Any downsides? I guess this would mean that we start raising an error on ISO 8601's with offsets attached, which might annoy some people?yes, but errors are better than incorrect values... Writing this made me think of a third option -- tracking, but no real manipulation, of TZ. This would be analogous to the ISO 8601 does -- all it does is note an offset. A given DateTime64 array would have a given offset assigned to it, and the appropriate addition and subtraction would happen at I/O. Offset of 0.00 would be UTC, and there would be a None option for naive. Please no! An integer offset is a terrible way to represent timezones,well, it would solve the being able to read ISO strings problem, and being able to perform operations with datetimes in multiple time zones. though I guess you could get most of that with UTC-always. and hardcoding this would just get in the way of a proper solution.well, that's a point -- if we think there is any hope of a proper solution down the road, then yes, it would be better not to make that harder. -Chris-- Christopher Barker, Ph.D.OceanographerEmergency Response DivisionNOAA/NOS/ORR (206) 526-6959 voice7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main receptionchris.bar...@noaa.gov ___NumPy-Discussion mailing listNumPy-Discussion@scipy.orghttp://mail.scipy.org/mailman/listinfo/numpy-discussion --Sankarshan MudkaviUndergraduate in Physics, University of Waterloowww.smudkavi.com signature.asc Description: Message signed with OpenPGP using GPGMail ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
