[Numpy-discussion] Hang in numpy.zeros
Hi, I have a multithreaded multiprocessing Python 2.7 project with Theano. In my main proc, I recognized a hang in numpy.zeros. It deadlocks with the Python GIL and other native threads will lock on the Python GIL. I found a somewhat related problem described here about such a deadlock in numpy.dot: http://stackoverflow.com/questions/23963997/python-child-process-crashes-on-numpy-dot-if-pyside-is-imported I have Numpy 1.9.1 on Ubuntu 12.04. How can I fix this? Is this a known problem? Kind Regards, Albert ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] EuroScipy 2015: Call for talks, posters tutorials
Dear all, EuroScipy 2015, the annual conference on Python in science will take place in Cambridge, UK on 26-30 August 2015. The conference features two days of tutorials followed by two days of scientific talks posters and an extra day dedicated to developer sprints. It is the major event in Europe in the field of technical/scientific computing within the Python ecosystem. Data scientists, analysts, quants, PhD's, scientists and students from more than 20 countries attended the conference last year. The topics presented at EuroSciPy are very diverse, with a focus on advanced software engineering and original uses of Python and its scientific libraries, either in theoretical or experimental research, from both academia and the industry. Submissions for posters, talks tutorials (beginner and advanced) are welcome on our website at http://www.euroscipy.org/2015/ http://www.euroscipy.org/2015/ Sprint proposals should be addressed directly to the organisation at euroscipy-...@python.org mailto:euroscipy-...@python.org?subject=Sprint%20proposal Important dates Mar 24, 2015Call for talks, posters tutorials Apr 30, 2015Talk and tutorials submission deadline May 1, 2015 Registration opens May 30, 2015Final program announced Jun 15, 2015Early-bird registration ends Aug 26-27, 2015 Tutorials Aug 28-29, 2015 Main conference Aug 30, 2015Sprints We look forward to an exciting conference and hope to see you in Cambridge The EuroSciPy 2015 Team - http://www.euroscipy.org/2015/ http://www.euroscipy.org/2015/___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] netcdf lat lon to coord - ValueError: need more than 1 value to unpack
I would like to find the nearest coord in a netcdf from a given latitude and longitude. I found some fantastic code that does this - http://nbviewer.ipython.org/github/Unidata/unidata-python-workshop/blob/master/netcdf-by-coordinates.ipynb but I keep receiving this error - I am receiving a ValueError: need more than 1 value to unpack I have pasted the code and full error below. Any help will be greatly appreciated. import numpy as np import netCDF4 def naive_fast(latvar,lonvar,lat0,lon0): # Read latitude and longitude from file into numpy arrays latvals = latvar[:] lonvals = lonvar[:] ny,nx = latvals.shape dist_sq = (latvals-lat0)**2 + (lonvals-lon0)**2 minindex_flattened = dist_sq.argmin() # 1D index of min element iy_min,ix_min = np.unravel_index(minindex_flattened, latvals.shape) return iy_min,ix_min filename = /Users/T_SFC.nc ncfile = netCDF4.Dataset(filename, 'r') latvar = ncfile.variables['latitude'] lonvar = ncfile.variables['longitude'] iy,ix = naive_fast(latvar, lonvar, -38.009, 146.438) print 'Closest lat lon:', latvar[iy,ix], lonvar[iy,ix] ncfile.close() --- ValueErrorTraceback (most recent call last) /Applications/Canopy.app/appdata/canopy-1.3.0.1715.macosx-x86_64/Canopy.app/Contents/lib/python2.7/site-packages/IPython/utils/py3compat.pyc in execfile(fname, *where) 202 else: 203 filename = fname -- 204 __builtin__.execfile(filename, *where) /Users/latlon_to_closestgrid.py in module() 22 lonvar = ncfile.variables['longitude'] 23 --- 24 iy,ix = naive_fast(latvar, lonvar, -38.009, 146.438) 25 print 'Closest lat lon:', latvar[iy,ix], lonvar[iy,ix] 26 ncfile.close() /Users/latlon_to_closestgrid.py in naive_fast(latvar, lonvar, lat0, lon0) 12 latvals = latvar[:] 13 lonvals = lonvar[:] --- 14 ny,nx = latvals.shape 15 dist_sq = (latvals-lat0)**2 + (lonvals-lon0)**2 16 minindex_flattened = dist_sq.argmin() # 1D index of min element ValueError: need more than 1 value to unpack ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Introductory mail and GSoc Project Vector math library integration
Am 12.03.2015 um 13:48 schrieb Julian Taylor jtaylor.deb...@googlemail.com: On 03/12/2015 10:15 AM, Gregor Thalhammer wrote: Another note, numpy makes it easy to provide new ufuncs, see http://docs.scipy.org/doc/numpy-dev/user/c-info.ufunc-tutorial.html from a C function that operates on 1D arrays, but this function needs to support arbitrary spacing (stride) between the items. Unfortunately, to achieve good performance, vector math libraries often expect that the items are laid out contiguously in memory. MKL/VML is a notable exception. So for non contiguous in- or output arrays you might need to copy the data to a buffer, which likely kills large amounts of the performance gain. The elementary functions are very slow even compared to memory access, they take in the orders of hundreds to tens of thousand cycles to complete (depending on range and required accuracy). Even in the case of strided access that gives the hardware prefetchers plenty of time to load the data before the previous computation is done. That might apply to the mathematical functions from the standard libraries, but that is not true for the optimized libraries. Typical numbers are 4-10 CPU cycles per operation, see e.g. https://software.intel.com/sites/products/documentation/doclib/mkl_sa/112/vml/functions/_performanceall.html The benchmarks at https://github.com/geggo/uvml https://github.com/geggo/uvml show that memory access to main memory limits the performance for the calculation of exp for large array sizes . This test was done quite some time ago, memory bandwidth now typically is higher, but also computational power. This also removes the requirement from the library to provide a strided api, we can copy the strided data into a contiguous buffer and pass it to the library without losing much performance. It may not be optimal (e.g. a library can fine tune the prefetching better for the case where the hardware is not ideal) but most likely sufficient. Copying the data to a small enough buffer so it fits into cache might add a few cycles, this already impacts performance significantly. Curious to see how much. Gregor Figuring out how to best do it to get the best performance and still being flexible in what implementation is used is part of the challenge the student will face for this project. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] netcdf lat lon to coord - ValueError: need more than 1 value to unpack
2015-03-24 11:02 GMT+01:00 questions anon questions.a...@gmail.com: I would like to find the nearest coord in a netcdf from a given latitude and longitude. I found some fantastic code that does this - http://nbviewer.ipython.org/github/Unidata/unidata-python-workshop/blob/master/netcdf-by-coordinates.ipynb but I keep receiving this error - I am receiving a ValueError: need more than 1 value to unpack I have pasted the code and full error below. Any help will be greatly appreciated. import numpy as np import netCDF4 def naive_fast(latvar,lonvar,lat0,lon0): # Read latitude and longitude from file into numpy arrays latvals = latvar[:] lonvals = lonvar[:] ny,nx = latvals.shape dist_sq = (latvals-lat0)**2 + (lonvals-lon0)**2 minindex_flattened = dist_sq.argmin() # 1D index of min element iy_min,ix_min = np.unravel_index(minindex_flattened, latvals.shape) return iy_min,ix_min filename = /Users/T_SFC.nc ncfile = netCDF4.Dataset(filename, 'r') latvar = ncfile.variables['latitude'] lonvar = ncfile.variables['longitude'] iy,ix = naive_fast(latvar, lonvar, -38.009, 146.438) print 'Closest lat lon:', latvar[iy,ix], lonvar[iy,ix] ncfile.close() --- ValueErrorTraceback (most recent call last) /Applications/Canopy.app/appdata/canopy-1.3.0.1715.macosx-x86_64/Canopy.app/Contents/lib/python2.7/site-packages/IPython/utils/py3compat.pyc in execfile(fname, *where) 202 else: 203 filename = fname -- 204 __builtin__.execfile(filename, *where) /Users/latlon_to_closestgrid.py in module() 22 lonvar = ncfile.variables['longitude'] 23 --- 24 iy,ix = naive_fast(latvar, lonvar, -38.009, 146.438) 25 print 'Closest lat lon:', latvar[iy,ix], lonvar[iy,ix] 26 ncfile.close() /Users/latlon_to_closestgrid.py in naive_fast(latvar, lonvar, lat0, lon0) 12 latvals = latvar[:] 13 lonvals = lonvar[:] --- 14 ny,nx = latvals.shape 15 dist_sq = (latvals-lat0)**2 + (lonvals-lon0)**2 16 minindex_flattened = dist_sq.argmin() # 1D index of min element ValueError: need more than 1 value to unpack It seems that latvals and lonvals should be a 2D array and you are providing just a 1D array. Maybe you could use numpy.meshgrid [1] to get 2D inputs from 1D arrays. [1] http://docs.scipy.org/doc/numpy/reference/generated/numpy.meshgrid.html ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Asking proposal review/feedback for GSOC 15
Hi Nikolay, Thanks for pointing out that! It really helped. I think it looks better and easier to review now. I appreciate any comment/feedback. My proposal is at https://gist.github.com/oguzhanunlu/1f8bf3ffc6ac5c420dd1 Thanks in advance, Oguzhan Hi, Oguzhan. I suggest to add .md extension to the gist file, now it is displayed as raw text. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Improve Numpy Datetime Functionality for Gsoc
Hi Saprative and Smruti, Sorry for the slow reply, I overlooked this thread. http://thread.gmane.org/gmane.comp.python.numeric.general/53805 and the discussion that followed (also linked from the ideas page) should give you some idea of what is required. If you want to start working on a patch I recommend to start small: https://github.com/numpy/numpy/issues?q=is%3Aopen+is%3Aissue+label%3A%22Easy+Fix%22 There are also a number of related issues that you could look at: https://github.com/numpy/numpy/issues?utf8=%E2%9C%93q=is%3Aissue+is%3Aopen+datetime. Trying to tackly one of those should give you an idea of the level of difficulty of this project (it's one of the harder ones on our list). Cheers, Ralf On Thu, Mar 19, 2015 at 11:15 AM, SMRUTI RANJAN SAHOO c99.smr...@gmail.com wrote: i am also student developer. if i will get anything i will tell you. On Thu, Mar 19, 2015 at 11:55 AM, Saprative Jana saprativej...@gmail.com wrote: hi, I am Saprative .I am new to numpy devlopment. I want to work on the project of improving datetime functionality numpy project .I want to solve some related bugs and get started with the basics. As there is no irc channel for numpy so i am facing a problem of contacting with the mentors moreover there is no mentors mentioned for this project. So anybody who can help me out please contact with me. from, Saprative Jana (Mob: +919477325233) ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Asking proposal review/feedback for GSOC 15
On Tue, Mar 24, 2015 at 1:12 PM, Oğuzhan Ünlü cengoguzhanu...@gmail.com wrote: Hi Nikolay, Thanks for pointing out that! It really helped. I think it looks better and easier to review now. I appreciate any comment/feedback. My proposal is at https://gist.github.com/oguzhanunlu/1f8bf3ffc6ac5c420dd1 Regarding your schedule: - I would remove the parts related to benchmarks. There's no nice benchmark infrastructure in numpy itself at the moment (that's a separate GSoC idea), so the two times 1 week that you have are likely not enough to get something off the ground there. - The implement a flexible interface part will need some discussion, probably it makes sense to first draft a document (call it a NEP - Numpy Enhancement Proposal) that lays out the options and makes a proposal. - I wouldn't put investigate accuracy differences at the end. What if you find out there that you've been working on something for the whole summer that's not accurate enough? - The researching possible options I would do in the community bonding period - when the coding period starts you should have a fairly well-defined plan. - 3 weeks for implementing the interface looks optimistic. Cheers, Ralf ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Improve Numpy Datetime Functionality for Gsoc
you are saying that if i will find out this bugs ,then i will selected for gsoc 2015 ?? and where i will find my mentor?? On Wed, Mar 25, 2015 at 3:33 AM, Ralf Gommers ralf.gomm...@gmail.com wrote: Hi Saprative and Smruti, Sorry for the slow reply, I overlooked this thread. http://thread.gmane.org/gmane.comp.python.numeric.general/53805 and the discussion that followed (also linked from the ideas page) should give you some idea of what is required. If you want to start working on a patch I recommend to start small: https://github.com/numpy/numpy/issues?q=is%3Aopen+is%3Aissue+label%3A%22Easy+Fix%22 There are also a number of related issues that you could look at: https://github.com/numpy/numpy/issues?utf8=%E2%9C%93q=is%3Aissue+is%3Aopen+datetime. Trying to tackly one of those should give you an idea of the level of difficulty of this project (it's one of the harder ones on our list). Cheers, Ralf On Thu, Mar 19, 2015 at 11:15 AM, SMRUTI RANJAN SAHOO c99.smr...@gmail.com wrote: i am also student developer. if i will get anything i will tell you. On Thu, Mar 19, 2015 at 11:55 AM, Saprative Jana saprativej...@gmail.com wrote: hi, I am Saprative .I am new to numpy devlopment. I want to work on the project of improving datetime functionality numpy project .I want to solve some related bugs and get started with the basics. As there is no irc channel for numpy so i am facing a problem of contacting with the mentors moreover there is no mentors mentioned for this project. So anybody who can help me out please contact with me. from, Saprative Jana (Mob: +919477325233) ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Improve Numpy Datetime Functionality for Gsoc
The most recent discussion about datetime64 was back in March and April of last year: http://mail.scipy.org/pipermail/numpy-discussion/2014-March/thread.html#69554 http://mail.scipy.org/pipermail/numpy-discussion/2014-April/thread.html#69774 In addition to unfortunate timezone handling, datetime64 has a lot of bugs -- so many that I don't bother reporting them. But if anyone ever plans on working on them, I can certainly help to assemble a long list of the issues (many of these are mentioned in the above threads). Unfortunately, though I would love to see datetime64 fixed, I'm not really a suitable mentor for this role (I don't know C), ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Asking proposal review/feedback for GSOC 15
Hi, Oguzhan. I suggest to add .md extension to the gist file, now it is displayed as raw text. Date: Tue, 24 Mar 2015 01:16:40 +0200 From: cengoguzhanu...@gmail.com To: numpy-discussion@scipy.org Subject: Re: [Numpy-discussion] Asking proposal review/feedback for GSOC 15 Hi again, Thanks Ralf, I understand that. Then, I would like to share a public link to my proposal and I appreciate anybody who take time and leave comment/give feedback. It is on Github Gist. URL: https://gist.github.com/oguzhanunlu/1f8bf3ffc6ac5c420dd1 Thanks in advance, Oguzhan Hi, My name is O?uzhan(You may use 'Oguzhan'). I submitted a proposal on the system with the title 'NumPy - Vector math library integration'. Ralf commented on my proposal and advised to ask for a feedback on mailing list and here I am. I would appreciate any feedback from community. I think community members are able to view my proposal, its visibility is set to 'Organization members'. I preferred my name in its original form, if any mentor would like to search, I provide my name on system below. Name: O?uzhan ?nl? Hi O?uzhan, There are only a handful of potential mentors signed up in Melange, and this list is read by hundreds of people. So it would be good to post your proposal in a publicly accessible place and post the link here. Good options are on Github or on StackEdit. Cheers, Ralf P.S. for those who do have access to Melange: http://www.google-melange.com/gsoc/proposal/review/org/google/gsoc2015/blacksimit/5741031244955648 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] netcdf lat lon to coord - ValueError: need more than 1 value to unpack
perfect, thank you! On Tue, Mar 24, 2015 at 9:14 PM, Kiko kikocorre...@gmail.com wrote: 2015-03-24 11:02 GMT+01:00 questions anon questions.a...@gmail.com: I would like to find the nearest coord in a netcdf from a given latitude and longitude. I found some fantastic code that does this - http://nbviewer.ipython.org/github/Unidata/unidata-python-workshop/blob/master/netcdf-by-coordinates.ipynb but I keep receiving this error - I am receiving a ValueError: need more than 1 value to unpack I have pasted the code and full error below. Any help will be greatly appreciated. import numpy as np import netCDF4 def naive_fast(latvar,lonvar,lat0,lon0): # Read latitude and longitude from file into numpy arrays latvals = latvar[:] lonvals = lonvar[:] ny,nx = latvals.shape dist_sq = (latvals-lat0)**2 + (lonvals-lon0)**2 minindex_flattened = dist_sq.argmin() # 1D index of min element iy_min,ix_min = np.unravel_index(minindex_flattened, latvals.shape) return iy_min,ix_min filename = /Users/T_SFC.nc ncfile = netCDF4.Dataset(filename, 'r') latvar = ncfile.variables['latitude'] lonvar = ncfile.variables['longitude'] iy,ix = naive_fast(latvar, lonvar, -38.009, 146.438) print 'Closest lat lon:', latvar[iy,ix], lonvar[iy,ix] ncfile.close() --- ValueErrorTraceback (most recent call last) /Applications/Canopy.app/appdata/canopy-1.3.0.1715.macosx-x86_64/Canopy.app/Contents/lib/python2.7/site-packages/IPython/utils/py3compat.pyc in execfile(fname, *where) 202 else: 203 filename = fname -- 204 __builtin__.execfile(filename, *where) /Users/latlon_to_closestgrid.py in module() 22 lonvar = ncfile.variables['longitude'] 23 --- 24 iy,ix = naive_fast(latvar, lonvar, -38.009, 146.438) 25 print 'Closest lat lon:', latvar[iy,ix], lonvar[iy,ix] 26 ncfile.close() /Users/latlon_to_closestgrid.py in naive_fast(latvar, lonvar, lat0, lon0) 12 latvals = latvar[:] 13 lonvals = lonvar[:] --- 14 ny,nx = latvals.shape 15 dist_sq = (latvals-lat0)**2 + (lonvals-lon0)**2 16 minindex_flattened = dist_sq.argmin() # 1D index of min element ValueError: need more than 1 value to unpack It seems that latvals and lonvals should be a 2D array and you are providing just a 1D array. Maybe you could use numpy.meshgrid [1] to get 2D inputs from 1D arrays. [1] http://docs.scipy.org/doc/numpy/reference/generated/numpy.meshgrid.html ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
