Re: [Numpy-discussion] [OT] any image io module that works with python3?
pygame On Sat, Mar 12, 2011 at 9:08 AM, Nadav Horesh nad...@visionsense.comwrote: Having numpy, scipy, and matplotlib working reasonably with python3, a major piece of code I miss for a major python3 migration is an image IO. I found that pylab's imread works fine for png image, but I need to read all the other image format as well as png and jpeg output. Any hints (including advices how easyly construct my own module) are appreciated. Nadav. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] speed of numpy.ndarray compared to Numeric.array
Hi, Spatial hashes are the common solution. Another common optimization is using the distance squared for collision detection. Since you do not need the expensive sqrt for this calc. cu. On Mon, Jan 10, 2011 at 3:25 PM, Pascal pascal...@parois.net wrote: Hi, On 01/10/2011 09:09 AM, EMMEL Thomas wrote: No I didn't, due to the fact that these values are coordinates in 3D (x,y,z). In fact I work with a list/array/tuple of arrays with 10 to 1M of elements or more. What I need to do is to calculate the distance of each of these elements (coordinates) to a given coordinate and filter for the nearest. The brute force method would look like this: #~ def bruteForceSearch(points, point): minpt = min([(vec2Norm(pt, point), pt, i) for i, pt in enumerate(points)], key=itemgetter(0)) return sqrt(minpt[0]), minpt[1], minpt[2] #~~ def vec2Norm(pt1,pt2): xDis = pt1[0]-pt2[0] yDis = pt1[1]-pt2[1] zDis = pt1[2]-pt2[2] return xDis*xDis+yDis*yDis+zDis*zDis I am not sure I understood the problem properly but here what I would use to calculate a distance from horizontally stacked vectors (big): ref=numpy.array([0.1,0.2,0.3]) big=numpy.random.randn(100, 3) big=numpy.add(big,-ref) distsquared=numpy.sum(big**2, axis=1) Pascal ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] ANN: NumPy 1.5.1 release candidate 1
hi,
this is another instance of a bug caused by the 'no old file handling'
problem with distutils/numpy.
On Sun, Oct 24, 2010 at 4:29 PM, Charles R Harris charlesr.har...@gmail.com
wrote:
On Sun, Oct 24, 2010 at 9:22 AM, Darren Dale dsdal...@gmail.com wrote:
On Sun, Oct 17, 2010 at 7:35 AM, Ralf Gommers
ralf.gomm...@googlemail.com wrote:
Hi,
I am pleased to announce the availability of the first release
candidate of NumPy 1.5.1. This is a bug-fix release with no new
features compared to 1.5.0.
[...]
Please report any other issues on the Numpy-discussion mailing list.
Just installed on kubuntu-10.10, python-2.7 and python-3.1.2. Tests
look fine for py2.7, but I see datetime errors with py3k:
.
==
ERROR: test_creation (test_datetime.TestDateTime)
--
Traceback (most recent call last):
File
/home/darren/.local/lib/python3.1/site-packages/numpy/core/tests/test_datetime.py,
line 10, in test_creation
dt1 = np.dtype('M8[750%s]'%unit)
TypeError: data type not understood
==
ERROR: test_creation_overflow (test_datetime.TestDateTime)
--
Traceback (most recent call last):
File
/home/darren/.local/lib/python3.1/site-packages/numpy/core/tests/test_datetime.py,
line 62, in test_creation_overflow
timesteps = np.array([date], dtype='datetime64[s]')[0].astype(np.int64)
TypeError: data type not understood
==
ERROR: test_divisor_conversion_as (test_datetime.TestDateTime)
--
Traceback (most recent call last):
File
/home/darren/.local/lib/python3.1/site-packages/numpy/core/tests/test_datetime.py,
line 58, in test_divisor_conversion_as
self.assertRaises(ValueError, lambda : np.dtype('M8[as/10]'))
File /usr/lib/python3.1/unittest.py, line 589, in assertRaises
callableObj(*args, **kwargs)
File
/home/darren/.local/lib/python3.1/site-packages/numpy/core/tests/test_datetime.py,
line 58, in lambda
self.assertRaises(ValueError, lambda : np.dtype('M8[as/10]'))
TypeError: data type not understood
==
ERROR: test_divisor_conversion_bday (test_datetime.TestDateTime)
--
Traceback (most recent call last):
File
/home/darren/.local/lib/python3.1/site-packages/numpy/core/tests/test_datetime.py,
line 32, in test_divisor_conversion_bday
assert np.dtype('M8[B/12]') == np.dtype('M8[2h]')
TypeError: data type not understood
==
ERROR: test_divisor_conversion_day (test_datetime.TestDateTime)
--
Traceback (most recent call last):
File
/home/darren/.local/lib/python3.1/site-packages/numpy/core/tests/test_datetime.py,
line 37, in test_divisor_conversion_day
assert np.dtype('M8[D/12]') == np.dtype('M8[2h]')
TypeError: data type not understood
==
ERROR: test_divisor_conversion_fs (test_datetime.TestDateTime)
--
Traceback (most recent call last):
File
/home/darren/.local/lib/python3.1/site-packages/numpy/core/tests/test_datetime.py,
line 54, in test_divisor_conversion_fs
assert np.dtype('M8[fs/100]') == np.dtype('M8[10as]')
TypeError: data type not understood
==
ERROR: test_divisor_conversion_hour (test_datetime.TestDateTime)
--
Traceback (most recent call last):
File
/home/darren/.local/lib/python3.1/site-packages/numpy/core/tests/test_datetime.py,
line 42, in test_divisor_conversion_hour
assert np.dtype('m8[h/30]') == np.dtype('m8[2m]')
TypeError: data type not understood
==
ERROR: test_divisor_conversion_minute (test_datetime.TestDateTime)
--
Traceback (most recent call last):
File
/home/darren/.local/lib/python3.1/site-packages/numpy/core/tests/test_datetime.py,
line 46, in test_divisor_conversion_minute
assert np.dtype('m8[m/30]') == np.dtype('m8[2s]')
TypeError: data type not understood
==
ERROR: test_divisor_conversion_month (test_datetime.TestDateTime)
--
Traceback (most recent call last):
Re: [Numpy-discussion] Memory profiling NumPy code?
numpy could implement __sizeof__ On Mon, Apr 26, 2010 at 12:16 PM, David Cournapeau courn...@gmail.comwrote: On Mon, Apr 26, 2010 at 7:57 PM, Dag Sverre Seljebotn da...@student.matnat.uio.no wrote: I'd like to profile the memory usage of my application using tools like e.g. Heapy. However since NumPy arrays are allocated in C they are not tracked by Python memory profiling. Does anybody have a workaround to share? I really just want to track a few arrays in a friendly way from within Python (I am aware of the existance of C-level profilers). I think heapy has some hooks so that you can add support for extensions. Maybe we could provide a C API in numpy to make this easy, David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] My GSoC Proposal to Implement a Subset of NumPy for PyPy
On Wed, Apr 21, 2010 at 1:21 PM, Dag Sverre Seljebotn da...@student.matnat.uio.no wrote: Dan Roberts wrote: Thanks for the reply. You're certainly right that your work is extremely beneficial to mine. At present I'm afraid a great deal of NumPy C code isn't easily reusable and it's great you're addressing that. I may not have been thinking in line with Maciej, but I was thinking ufuncs would be written in pure Python and jit compiled to an efficient form. (We can make lots of nice assumptions about them) That said, I think being able to write generic ufuncs is a very good idea, and absolutely doable. This might be relevant? http://conference.scipy.org/proceedings/SciPy2008/paper_16/ -- Dag Sverre ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion Also this: http://numcorepy.blogspot.com/ ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] multiprocessing shared arrays and numpy
On Fri, Mar 5, 2010 at 7:29 PM, Brian Granger ellisonbg@gmail.com wrote: Francesc, Yeah, 10% of improvement by using multi-cores is an expected figure for memory bound problems. This is something people must know: if their computations are memory bound (and this is much more common that one may initially think), then they should not expect significant speed-ups on their parallel codes. +1 Thanks for emphasizing this. This is definitely a big issue with multicore. Cheers, Brian Hi, here's a few notes... A) cache B) multiple cores/cpu multiplies other optimisations. A) Understanding cache is also very useful. Cache at two levels: 1. disk cache. 2. cpu/core cache. 1. Mmap'd files are useful since you can reuse disk cache as program memory. So large files don't waste ram on the disk cache. For example, processing a 1 gig file can use 1GB of memory with mmap, but 2GB without. ps, learn about madvise for extra goodness :) mmap behaviour is very different on windows/linux/mac osx. The best mmap implementation is on linux. Note, that on some OS's the disk cache has separate reserved areas of memory which processes can not use... so mmap is the easiest way to access it. mmaping on SSDs is also quite fast :) 2. cpu cache is what can give you a speedup when you use extra cpus/cores. There are a number of different cpu architectures these days... but generally you will get a speed up if your cpus access different areas of memory. So don't get cpu1 to process one part of data, then cpu2 - otherwise the cache can get invalidated. Especially if you have a 8MB cache per cpu :) This is why the Xeons, and other high end cpus will give you numpy speedups more easily. Also consider processing in chunks less than the size of your cache (especially for multi pass arguments). There's a lot to caching, but I think the above gives enough useful hints :) B) Also, multiple processes can multiply the effects of your other optimisations. A 2x speed up via SSE or other SIMD can be multiplied over each cpu/core. So if you code gets 8x faster with multiple processes, then the 2x optimisation is likely a 16x speed up. The following is a common with optimisation pattern with python code. From python to numpy you can get a 20x speedup. From numpy to C/C++ you can get up to 5 times speed up (or 50x over python). Then an asm optimisation is 2-4x faster again. So up to 200x faster compared to pure python... then multiply that by 8x, and you have up to 1600x faster code :) Also small optimisations add up... a small 0.2 times speedup can turn into a 1.6 times speed up easily when you have multiple cpus. So as you can see... multiple cores makes it EASIER to optimise programs, since your optimisations are often multiplied. cu, ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Python 3 porting
AWESOME :) On Sun, Feb 21, 2010 at 1:43 PM, Pauli Virtanen p...@iki.fi wrote: Hi, The test suite passes now on Pythons 2.4 - 3.1. Further testing is very welcome -- also on Python 2.x. Please check that your favourite software still builds and works with SVN trunk Numpy. Currently, Scipy has some known failures because of (i) removed new= keyword in numpy.histogram (ii) Cython supports only native size/alignment PEP 3118 buffers, and Numpy arrays are most naturally expressed in the standardized sizes. Supporting the full struct module alignment stuff appears to be a slight PITA. I'll try to take a look at how to address this. But everything else seems to work on Python 2.6. *** Python version 2.4.6 (#2, Jan 21 2010, 23:27:36) [GCC 4.4.1] Ran 2509 tests in 18.892s OK (KNOWNFAIL=4, SKIP=2) Python version 2.5.4 (r254:67916, Jan 20 2010, 21:44:03) [GCC 4.4.1] Ran 2512 tests in 18.531s OK (KNOWNFAIL=4) Python version 2.6.4 (r264:75706, Dec 7 2009, 18:45:15) [GCC 4.4.1] Ran 2519 tests in 19.367s OK (KNOWNFAIL=4) Python version 3.1.1+ (r311:74480, Nov 2 2009, 14:49:22) [GCC 4.4.1] Ran 2518 tests in 23.239s OK (KNOWNFAIL=5) Cheers, Pauli ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Multithreading support
hi, see: http://numcorepy.blogspot.com/ They see a benefit when working with large arrays. Otherwise you are limited by memory - and the extra cores don't help with memory bandwidth. cheers, On Sat, Feb 13, 2010 at 2:20 PM, David Cournapeau courn...@gmail.comwrote: On Sat, Feb 13, 2010 at 6:20 PM, Wolfgang Kerzendorf wkerzend...@googlemail.com wrote: Dear all, I don't know much about parallel programming so I don't know how easy it is to do that: When doing simple arrray operations like adding two arrays or adding a number to the array, is numpy able to put this on multiple cores? I have tried it but it doesnt seem to do that. Is there a special multithread implementation of numpy. Depending on your definition of simple operations, Numpy supports multithreaded execution or not. For ufuncs (which is used for things like adding two arrays together, etc...), there is no multithread support. IDL has this feature where it checks how many cores available and uses them. This feature in numpy would make an already amazing package even better. AFAIK, using multi-thread at the core level of NumPy has been tried only once a few years ago, without much success (no significant performance improvement). Maybe the approach was flawed in some ways. Some people have suggested using OpenMP, but nobody has every produced something significant AFAIK: http://mail.scipy.org/pipermail/numpy-discussion/2008-March/031897.html Note that Linear algebra operations can run in // depending on your libraries. In particular, the dot function runs in // if your blas/lapack does. cheers, David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] numpy release process, adding a compatibility test step.
Hi, may I suggest an addition to the release process... 'Tests against popular libraries that rely on numpy at the RC stage. Test at least these libraries pass their numpy related tests: matplotlib, scipy, pygame, (insert others here?). The release manage should ask the mailing list for people to test the RC against these libraries to make sure they work ok.' - This will catch problems like the ABI one with the recent release. - It is an explicit request for testing, with concrete tasks that people can do. Rather than a more abstract request for testing. - not much extra work for the release manager. testing work can be distributed to anyone who can try an app, or library with the binaries supplied in the RC stage of a release. - binary installer users can test as well, not just people who build from source. - tests libraries the numpy developers may not be using all the time themselves. On another project, adding this simple step helped prevent a number of bugs which affected other programs and libraries using the project. Before this was added we had a couple of releases which broke a few popular libraries/apps with very simple regressions or bugs. cu, ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Audio signal capture and processing
hi, pyaudio is pretty good for recording audio. It is based on portaudio and has binaries available for win/mac - and is included in many linux distros too (so is pygame). You can load, and play audio with pygame. You can use the pygame.sndarray module for converting the pygame.Sound objects into numpy arrays. apt-get install python-pygame import pygame, pygame.sndarray, sys fname = sys.argv[1] pygame.init() sound = pygame.mixer.Sound(fname) an_array = pygame.sndarray.array(sound) Also see the sndarray demo: pygame.examples.sound_array_demos. `python -m pygame.examples.sound_array_demos` Other sndarray using examples can be found on pygame.org with the search function. Also audiolab uses bindings to libsndfile - so you can open a number of formats. However it is pretty new, so isn't packaged by distros(yet), and there are no mac binaries(yet). It's probably the best way to go if you can handle compiling it yourself and the dependency. cheers, ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Announcing toydist, improving distribution and packaging situation
On Tue, Dec 29, 2009 at 11:20 PM, David Cournapeau courn...@gmail.com wrote: On Wed, Dec 30, 2009 at 3:36 AM, René Dudfield ren...@gmail.com wrote: On Tue, Dec 29, 2009 at 2:34 PM, David Cournapeau courn...@gmail.com wrote: On Tue, Dec 29, 2009 at 10:27 PM, René Dudfield ren...@gmail.com wrote: Buildout is what a lot of the python community are using now. I would like to note that buildout is a solution to a problem that I don't care to solve. This issue is particularly difficult to explain to people accustomed with buildout in my experience - I have not found a way to explain it very well yet. Hello, The main problem buildout solves is getting developers up to speed very quickly on a project. They should be able to call one command and get dozens of packages, and everything else needed ready to go, completely isolated from the rest of the system. If a project does not want to upgrade to the latest versions of packages, they do not have to. This reduces the dependency problem a lot. As one package does not have to block on waiting for 20 other packages. It makes iterating packages daily, or even hourly to not be a problem - even with dozens of different packages used. This is not theoretical, many projects iterate this quickly, and do not have problems. Backwards compatibility is of course a great thing to keep up... but harder to do with dozens of packages, some of which are third party ones. For example, some people are running pygame applications written 8 years ago that are still running today on the latest versions of pygame. I don't think people in the python world understand API, and ABI compatibility as much as those in the C world. However buildout is a solution to their problem, and allows them to iterate quickly with many participants, on many different projects. Many of these people work on maybe 20-100 different projects at once, and some machines may be running that many applications at once too. So using the system pythons packages is completely out of the question for them. This is all great, but I don't care about solving this issue, this is a *developer* issue. I don't mean this is not an important issue, it is just totally out of scope. The developer issues I care about are much more fine-grained (corrent dependency handling between target, toolchain customization, etc...). Note however that hopefully, by simplifying the packaging tools, the problems you see with numpy on 2.6 would be less common. The whole distutils/setuptools/distribute stack is hopelessly intractable, given how messy the code is. The numpy issue is because of the change in package handling for 2.6, which numpy 1.3 was not developed for. It is very easy to include a dozen packages in a buildout, so that you have all the packages required. I think there is a confusion - I mostly care about *end users*. People who may not have compilers, who want to be able to easily upgrade one package, etc... I was just describing the problems that buildout solves(for others). If I have a project that depends on numpy and 12 other packages, I can send it to other people who can get their project up and running fairly quickly (assuming everything installs ok). btw, numpy 1.4 works with buildout! (at least on my ubuntu box) sweet :) cd /tmp/ bzr branch lp:numpybuildout cd numpybuildout/trunk/ python bootstrap.py -d ./bin/buildout ./bin/py import numpy numpy.__file__ '/tmp/numpybuildout/trunk/eggs/numpy-1.4.0-py2.6-linux-i686.egg/numpy/__init__.pyc' ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Announcing toydist, improving distribution and packaging situation
hello again, On Tue, Dec 29, 2009 at 2:22 PM, David Cournapeau courn...@gmail.com wrote: On Tue, Dec 29, 2009 at 10:27 PM, René Dudfield ren...@gmail.com wrote: Hi, In the toydist proposal/release notes, I would address 'what does toydist do better' more explicitly. A big problem for science users is that numpy does not work with pypi + (easy_install, buildout or pip) and python 2.6. Working with the rest of the python community as much as possible is likely a good goal. Yes, but it is hopeless. Most of what is being discussed on distutils-sig is useless for us, and what matters is ignored at best. I think most people on distutils-sig are misguided, and I don't think the community is representative of people concerned with packaging anyway - most of the participants seem to be around web development, and are mostly dismissive of other's concerns (OS packagers, etc...). Sitting down with Tarek(who is one of the current distutils maintainers) in Berlin we had a little discussion about packaging over pizza and beer... and he was quite mindful of OS packagers problems and issues. He was also interested to hear about game developers issues with packaging (which are different again to scientific users... but similar in many ways). However these systems were developed by the zope/plone/web crowd, so they are naturally going to be thinking a lot about zope/plone/web issues. Debian, and ubuntu packages for them are mostly useless because of the age. Waiting a couple of years for your package to be released is just not an option (waiting even an hour for bug fixes is sometimes not an option). Also isolation of packages is needed for machines that have 100s of different applications running, written by different people, each with dozens of packages used by each application. Tools like checkinstall and stdeb ( http://pypi.python.org/pypi/stdeb/ ) can help with older style packaging systems like deb/rpm. I think perhaps if toydist included something like stdeb as not an extension to distutils, but a standalone tool (like toydist) there would be less problems with it. One thing the various zope related communities do is make sure all the relevant and needed packages are built/tested by their compile farms. This makes pypi work for them a lot better than a non-coordinated effort does. There are also lots of people trying out new versions all of the time. I want to note that I am not starting this out of thin air - I know most of distutils code very well, I have been the mostly sole maintainer of numpy.distutils for 2 years now. I have written extensive distutils extensions, in particular numscons which is able to fully build numpy, scipy and matplotlib on every platform that matters. Simply put, distutils code is horrible (this is an objective fact) and flawed beyond repair (this is more controversial). IMHO, it has almost no useful feature, except being standard. yes, I have also battled with distutils over the years. However it is simpler than autotools (for me... maybe distutils has perverted my fragile mind), and works on more platforms for python than any other current system. It is much worse for C/C++ modules though. It needs dependency, and configuration tools for it to work better (like what many C/C++ projects hack into distutils themselves). Monkey patching, and extensions are especially a problem... as is the horrible code quality of distutils by modern standards. However distutils has had more tests and testing systems added, so that refactoring/cleaning up of distutils can happen more so. If you want a more detailed explanation of why I think distutils and all tools on top are deeply flawed, you can look here: http://cournape.wordpress.com/2009/04/01/python-packaging-a-few-observations-cabal-for-a-solution/ I agree with many things in that post. Except your conclusion on multiple versions of packages in isolation. Package isolation is like processes, and package sharing is like threads - and threads are evil! Leave my python site-packages directory alone I say... especially don't let setuptools infect it :) Many people currently find the multi versions of packages in isolation approach works well for them - so for some use cases the tools are working wonderfully. numpy used to work with buildout in python2.5, but not with 2.6. buildout lets other team members get up to speed with a project by running one command. It installs things in the local directory, not system wide. So you can have different dependencies per project. I don't think it is a very useful feature, honestly. It seems to me that they created a huge infrastructure to split packages into tiny pieces, and then try to get them back together, imaganing that multiple installed versions is a replacement for backward compatibility. Anyone with extensive packaging experience knows that's a deeply flawed model in general. Science is supposed to allow
Re: [Numpy-discussion] Announcing toydist, improving distribution and packaging situation
On Wed, Dec 30, 2009 at 2:26 PM, Ravi lists_r...@lavabit.com wrote: On Wednesday 30 December 2009 06:15:45 René Dudfield wrote: I agree with many things in that post. Except your conclusion on multiple versions of packages in isolation. Package isolation is like processes, and package sharing is like threads - and threads are evil! You have stated this several times, but is there any evidence that this is the desire of the majority of users? In the scientific community, interactive experimentation is critical and users are typically not seasoned systems administrators. For such users, almost all packages installed after installing python itself are packages they use. In particular, all I want to do is to use apt/yum to get the packages (or ask my sysadmin, who rightfully has no interest in learning the intricacies of python package installation, to do so) and continue with my work. Packages-in-isolation is for people whose job is to run server farms, not interactive experimenters. 500+ packages on pypi. Provide a counter point, otherwise the evidence is against your position - overwhelmingly. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Announcing toydist, improving distribution and packaging situation
On Wed, Dec 30, 2009 at 7:08 PM, Robert Kern robert.k...@gmail.com wrote: On Wed, Dec 30, 2009 at 12:47, René Dudfield ren...@gmail.com wrote: On Wed, Dec 30, 2009 at 2:26 PM, Ravi lists_r...@lavabit.com wrote: On Wednesday 30 December 2009 06:15:45 René Dudfield wrote: I agree with many things in that post. Except your conclusion on multiple versions of packages in isolation. Package isolation is like processes, and package sharing is like threads - and threads are evil! You have stated this several times, but is there any evidence that this is the desire of the majority of users? In the scientific community, interactive experimentation is critical and users are typically not seasoned systems administrators. For such users, almost all packages installed after installing python itself are packages they use. In particular, all I want to do is to use apt/yum to get the packages (or ask my sysadmin, who rightfully has no interest in learning the intricacies of python package installation, to do so) and continue with my work. Packages-in-isolation is for people whose job is to run server farms, not interactive experimenters. 500+ packages on pypi. Provide a counter point, otherwise the evidence is against your position - overwhelmingly. Linux distributions, which are much, much more popular than any collection of packages on PyPI you might care to name. Isolated environments have their uses, but they are the exception, not the rule. wrong. pypi has way more python packages than any linux distribution. 8500+ listed, compared to how many in debian? ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Announcing toydist, improving distribution and packaging situation
Hi, In the toydist proposal/release notes, I would address 'what does toydist do better' more explicitly. A big problem for science users is that numpy does not work with pypi + (easy_install, buildout or pip) and python 2.6. Working with the rest of the python community as much as possible is likely a good goal. At least getting numpy to work with the latest tools would be great. An interesting read is the history of python packaging here: http://faassen.n--tree.net/blog/view/weblog/2009/11/09/0 Buildout is what a lot of the python community are using now. Getting numpy to work nicely with buildout and pip would be a good start. numpy used to work with buildout in python2.5, but not with 2.6. buildout lets other team members get up to speed with a project by running one command. It installs things in the local directory, not system wide. So you can have different dependencies per project. Plenty of good work is going on with python packaging. Lots of the python community are not using compiled packages however, so the requirements are different. There are a lot of people (thousands) working with the python packaging system, improving it, and building tools around it. Distribute for example has many committers, as do buildout/pip. eg, there are fifty or so buildout plugins, which people use to customise their builds( see the buildout recipe list on pypi at http://pypi.python.org/pypi?:action=browseshow=allc=512 ). There are build farms for windows packages and OSX uploaded to pypi. Start uploading pre releases to pypi, and you get these for free (once you make numpy compile out of the box on those compile farms). There are compile farms for other OSes too... like ubuntu/debian, macports etc. Some distributions even automatically download, compile and package new releases once they spot a new file on your ftp/web site. Speeding up the release cycle to be continuous can let people take advantage of these tools built together. If you get your tests running after the build step, all of these distributions also turn into test farms :) pypm: http://pypm.activestate.com/list-n.html#numpy ubuntu PPA: https://launchpad.net/ubuntu/+ppas the snakebite project : http://www.snakebite.org/ (seems mostly dead... but they have a lot of hardware) suse build service: https://build.opensuse.org/ pony-build: http://wiki.github.com/ctb/pony-build zope, and pygame also have their own build/test farms. They are two other compiled python packages projects. As do a number of other python projects(eg twisted...). Projects like pony-build should hopefully make it easier for people to run their own build farms, independently of the main projects. You just really need a script to: (download, build, test, post results), and then post a link to your mailing list... and someone will be able to run a build farm. Documentation projects are being worked on to document, give tutorials and make python packaging be easier all round. As witnessed by 20 or so releases on pypi every day(and growing), lots of people are using the python packaging tools successfully. Documenting how people can make numpy addon libraries(plugins) would encourage people to do so. Currently there is no documentation from the numpy community, or encouragement to do so. This combined with numpy being broken with python2.6+pypi will result in less science related packages. There is still a whole magnitude of people not releasing on pypi though, there are thousands of projects on the pygame.org website that are not on the pypi website for example. There are likely many hundreds or thousands of scientific projects not listed on their either. Given all of these projects not on pypi, obviously things could be improved. The pygame.org website also shows that community specific websites are very helpful. A science view of pypi would make it much more useful - so people don't have to look through web/game/database etc packages. Here is a view of 535 science/engineering related packages on pypi now: http://pypi.python.org/pypi?:action=browsec=385 458 science/research packages on pypi: http://pypi.python.org/pypi?:action=browseshow=allc=40 So there are already hundreds of science related packages and hundreds of people making those science related packages for pypi. Not too bad. Distribution of Applications is another issue that needs improving. That is so that people can share applications without needing to install a whole bunch of things. Think about sending applications to your grandma. Do you ask her to download python, grab these libraries, do this... do that. It would be much better if you could give her a url, and away you go! Bug tracking, and diff tracking between distributions is an area where many projects can improve. Searching through the distributions bug trackers, and diffs to apply to the core dramatically helps packages getting updated. So does maintaining good communication with different
Re: [Numpy-discussion] Announcing toydist, improving distribution and packaging situation
On Tue, Dec 29, 2009 at 2:34 PM, David Cournapeau courn...@gmail.com wrote: On Tue, Dec 29, 2009 at 10:27 PM, René Dudfield ren...@gmail.com wrote: Buildout is what a lot of the python community are using now. I would like to note that buildout is a solution to a problem that I don't care to solve. This issue is particularly difficult to explain to people accustomed with buildout in my experience - I have not found a way to explain it very well yet. Hello, The main problem buildout solves is getting developers up to speed very quickly on a project. They should be able to call one command and get dozens of packages, and everything else needed ready to go, completely isolated from the rest of the system. If a project does not want to upgrade to the latest versions of packages, they do not have to. This reduces the dependency problem a lot. As one package does not have to block on waiting for 20 other packages. It makes iterating packages daily, or even hourly to not be a problem - even with dozens of different packages used. This is not theoretical, many projects iterate this quickly, and do not have problems. Backwards compatibility is of course a great thing to keep up... but harder to do with dozens of packages, some of which are third party ones. For example, some people are running pygame applications written 8 years ago that are still running today on the latest versions of pygame. I don't think people in the python world understand API, and ABI compatibility as much as those in the C world. However buildout is a solution to their problem, and allows them to iterate quickly with many participants, on many different projects. Many of these people work on maybe 20-100 different projects at once, and some machines may be running that many applications at once too. So using the system pythons packages is completely out of the question for them. A scientist who installs numpy, scipy, etc... to try things out want to have everything available in one python interpreter, and does not want to jump to different virtualenvs and whatnot to try different packages. It is very easy to include a dozen packages in a buildout, so that you have all the packages required. Anyway... here is a skeleton buildout project that uses numpy if anyone wants to have a play. http://renesd.blogspot.com/2009/12/buildout-project-that-uses-numpy.html cheers, ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Python 3K merge
On Thu, Dec 3, 2009 at 10:39 AM, Pauli Virtanen pav...@iki.fipav%2...@iki.fi wrote: Tue, 01 Dec 2009 17:31:10 +0900, David Cournapeau wrote: On Tue, Dec 1, 2009 at 5:04 AM, Charles R Harris charlesr.har...@gmail.com wrote: It looks like you doing great stuff with the py3k transition. Do you and David have any sort of merge schedule in mind? I have updated my py3k branch for numpy.distutils, and it is ready to merge: http://github.com/cournape/numpy/tree/py3k_bootstrap_take3 I have not thoroughly tested it, but it can run on both 2.4 and 3.1 on Linux at least. The patch is much smaller than my previous attempts as well, so I would just push it to the trunk, and deal with the issues as they come. I think I should rebase my branch on this, or vice versa, to avoid further duplicated work. I think most of my changes would be ready for SVN, after rebasing and regrouping via rebase -i -- they do not affect behavior on Py2, and for the most part the changes required in C code are quite obvious. The largest changes are probably related to the 'S' data type. In other news, we cannot support Py2 pickles in Py3 -- this is because Py2 str is unpickled as Py3 str, resulting to encoding failures even before the data is passed on to Numpy. Is this just for the type codes? Or is there other string data that needs to be pickle loaded? If it is just for the type codes, they are all within the ansi character set and unpickle fine without errors. I'm guessing numpy uses strings to pickle arrays? Note that the pickle module is extensible. So we might be able to get it to special case things. You can subclass Unpickler to make extensions... and there are other techniques. Or it's even possible to submit patches to python if we have a need for something it doesn't support. It is even possible to change the pickle code for py2, so that py3 compatible pickles are saved. In this case it would just require people to load, and resave their pickles with the latest numpy version. Using the python array module to store data might be the way to go(rather than strings), since that is available in both py2 and py3. The pickling/unpickling situtation should be marked as a todo, and documented anyway. As we should start a numpy specific 'porting your code to py3k' document. A set of pickles saved from python2 would be useful for testing. Forwards compatibility is also a useful thing to test. That is py3.1 pickles saved to be loaded with python2 numpy. cheers! ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] numpy 1.3.0 eggs with python2.6 seem broken on osx, and linux
On Sat, Nov 28, 2009 at 2:56 PM, David Cournapeau courn...@gmail.comwrote: On Sat, Nov 28, 2009 at 10:42 PM, René Dudfield ren...@gmail.com wrote: yeah, I completely understand the unfortunate packaging situation (eg, some of my packages do not work with this install method). Here is a simple package requiring numpy. It uses buildout (http://www.buildout.org/). To help easily reproduce the problem, here are the commands to reproduce. I have not tried it, but from your explanation, I would suspect that buildout does some monkeypatching which is not exactly the same as setuptools itself, and that would break numpy.distutils. The distutils architecture unfortunately requires to take into account and special case any distutils extensions, because of the extension as command inheritance approach. I know I prefer spending my time on new packaging solution, but I would take patches. David Package/module semantics changed during 2.6/3.0. I think it might have something to do with that. Hopefully some of the py3 work will fix these problems too. Actually, I think it might also be a buildout bug. It seems buildout uses the sys.path variable to add paths to the python interpreter. Whereas the 'proper' way is to use the PYTHONPATH environment imho. Since modifying sys.path does not affect sub processes paths. It seems numpy uses a sub process in the install? Or maybe buildout/python2.6 use a subprocess. Or with 2.5 already having a numpy installed (on osx), then perhaps it was using the old numpy.distutils already installed to do the install. Anyway, that's my findings so far. Not sure if I'll look into it more. cu, ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Numpy 1.4.0 rc1 released
On Wed, Dec 2, 2009 at 9:49 AM, David Cournapeau da...@ar.media.kyoto-u.ac.jp wrote: Nadav Horesh wrote: I got the following errors with a clean installation of numpy (previous installations deleted): Actually, there are still some leftover: the file numpy/core/test_defmatrix.py does not exist in the tarball. cheers, David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion hi, this is caused by distutils not having a mechanism for removing old files. See this bug: http://bugs.python.org/issue5342 We put in some hacks into pygame distutils for removing old files (with msi installer, and for setup.py installer). However I think the long term solution they are thinking of is to allow distutils to uninstall correctly. However that will only work for new pythons if it is implemented. The pygame distutils mod has a list of old files to remove, which are then acted on by the msi code, and the setup.py install code. I could submit a patch for numpy if you are interested? It would require a list of old files from previous versions of numpy (like numpy/core/test_defmatrix.py). cheers, ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Numpy 1.4.0 rc1 released
On Wed, Dec 2, 2009 at 11:51 AM, David Cournapeau da...@ar.media.kyoto-u.ac.jp wrote: René Dudfield wrote: We put in some hacks into pygame distutils for removing old files (with msi installer, and for setup.py installer). However I think the long term solution they are thinking of is to allow distutils to uninstall correctly. I think this is just wishful thinking from people who try to improve distutils - it is near impossible to implement this correctly with distutils, because of how install works (i.e. just dumping whatever happens to be in build directory, hope that the list of files is correctly handled - which is almost never the case in my experience). Ugly, but safe hacks to distutils trump breaking installs for me. There's another bug report about how moving the new install directory in is better. Then you have very little chance of having a broken install, and you do not need to worry about over writing files/new files old files. 'distutils race condition': http://bugs.python.org/issue7412 That's one of the major flaw of distutils which I hope to fix with my project toydist, by being very explicit at each step. In particular, the install steps are described from a file generated at build time, and all binary installers and eggs will be based on this generated file. Agreed. Separate steps are needed. I could submit a patch for numpy if you are interested? It would require a list of old files from previous versions of numpy (like numpy/core/test_defmatrix.py). Do you mean that the files to remove have to be tracked manually ? You can always submit a patch to trac, though, yeah, tracking files between releases manually. If there isn't already a list of files installed for each numpy release that is. If there is a file list, then it would be simple to figure out which files have been removed from each release. There is a new pep being worked on for making this file list in a specified place I believe. Actually, we have three sets of files... py_to_remove and ext_to_remove. Since oldfile.dll is different to oldfile.so etc. Also there are different .py files... eg, pyc, pyo and others. The third set is an explicit file, eg 'data/bla.png' I'll attach a patch to Trac with just with the 'numpy/core/test_defmatrix.py*' to be removed. Or you can view the pygame setup.py here: look for 'def remove_old_files', and also msilib.add_data(self.db, RemoveFile, records). http://www.seul.org/viewcvs/viewcvs.cgi/trunk/setup.py?rev=2653root=PyGamesortby=dateview=markup cheers, ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] numpy 1.3.0 eggs with python2.6 seem broken on osx, and linux
Hi, the other day I tried installing numpy with easy_install, and did not have much luck with python2.6. To reproduce, try installing with easy_install-2.6, or with buildout. The work around is to just install it in another way (eg, dmg or with setup.py install). cheers, ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] numpy 1.3.0 eggs with python2.6 seem broken on osx, and linux
On Sat, Nov 28, 2009 at 11:16 AM, David Cournapeau courn...@gmail.comwrote: On Sat, Nov 28, 2009 at 6:34 PM, René Dudfield ren...@gmail.com wrote: Hi, the other day I tried installing numpy with easy_install, and did not have much luck with python2.6. What problem exactly do you have ? Since 1.3.0, the release process for numpy uses virtualenv internally to build the installers, so at least python setupegg.py install works as far as setuptools is concerned. The work around is to just install it in another way (eg, dmg or with setup.py install). That's the recommended way anyway. I am trying to make sure we work under as many cases as possible for packaging, but the underlying python architecture for packaging is so fragile and flawed that things will break from time to time. David Hello David, yeah, I completely understand the unfortunate packaging situation (eg, some of my packages do not work with this install method). Here is a simple package requiring numpy. It uses buildout ( http://www.buildout.org/). To help easily reproduce the problem, here are the commands to reproduce. curl -O http://rene.f0o.com/~rene/numpy_buildout.tar.gz #wget http://rene.f0o.com/~rene/numpy_buildout.tar.gz tar xf numpy_buildout.tar.gz cd numpy_buildout python2.6 bootstrap.py ./bin/buildout It prints out heaps of stuff... then finally fails with: /var/folders/d1/d1p0zeCxF7Kz3G5FZJsh6E+++TM/-Tmp-/easy_install-2WsZVp/numpy-1.3.0/numpy/distutils/misc_util.py:219: RuntimeWarning: Parent module 'numpy.distutils' not found while handling absolute import Error in atexit._run_exitfuncs: Traceback (most recent call last): File /Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/atexit.py, line 24, in _run_exitfuncs func(*targs, **kargs) File /var/folders/d1/d1p0zeCxF7Kz3G5FZJsh6E+++TM/-Tmp-/easy_install-2WsZVp/numpy-1.3.0/numpy/distutils/misc_util.py, line 219, in clean_up_temporary_directory ImportError: No module named numpy.distutils Error in sys.exitfunc: Traceback (most recent call last): File /Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/atexit.py, line 24, in _run_exitfuncs func(*targs, **kargs) File /var/folders/d1/d1p0zeCxF7Kz3G5FZJsh6E+++TM/-Tmp-/easy_install-2WsZVp/numpy-1.3.0/numpy/distutils/misc_util.py, line 219, in clean_up_temporary_directory I hope that helps if someone wants to reproduce the problem very easily some time. notes: * you should not have numpy installed already to reproduce the problem, otherwise the buildout uses the installed version and works. * use `python bootstrap.py -d` if you want to use distribute instead of setuptools (which has python3.1 support). * numpy with buildout works ok with python2.5 cheers, ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Bytes vs. Unicode in Python3
On Fri, Nov 27, 2009 at 11:50 AM, Francesc Alted fal...@pytables.org wrote:
A Friday 27 November 2009 11:27:00 Pauli Virtanen escrigué:
Yes. But now I wonder, should
array(['foo'], str)
array(['foo'])
be of dtype 'S' or 'U' in Python 3? I think I'm leaning towards 'U',
which will mean unavoidable code breakage -- there's probably no
avoiding it.
Mmh, you are right. Yes, this seems to be difficult to solve. Well, I'm
changing my mind and think that both 'str' and 'S' should stand for Unicode in
NumPy for Python 3. If people is aware of the change for Python 3, they
should be expecting the same change happening in NumPy too, I guess. Then, I
suppose that a new dtype bytes that replaces the existing string would be
absolutely necessary.
Also, I suppose that there will be issues with the current Unicode
support in NumPy:
In [5]: u = np.array(['asa'], dtype=U10)
In [6]: u[0]
Out[6]: u'asa' # will become 'asa' in Python 3
In [7]: u.dtype.itemsize
Out[7]: 40 # not sure about the size in Python 3
I suspect the Unicode stuff will keep working without major changes,
except maybe dropping the u in repr. It is difficult to believe the
CPython guys would have significantly changed the current Unicode
implementation, if they didn't bother changing the names of the
functions :)
For example, if it is true that internal strings in Python 3 and Unicode
UTF-8 (as René seems to suggest), I suppose that the internal conversions
from 2- bytes or 4-bytes (depending on how the Python interpreter has
been compiled) in NumPy Unicode dtype to the new Python string should
have to be reworked (perhaps you have dealt with that already).
I don't think they are internally UTF-8:
http://docs.python.org/3.1/c-api/unicode.html
Python’s default builds use a 16-bit type for Py_UNICODE and store
Unicode values internally as UCS2.
Ah! No changes for that matter. Much better then.
Hello,
in py3...
'Hello\u0020World !'.encode()
b'Hello World !'
Äpfel.encode('utf-8')
b'\xc3\x84pfel'
Äpfel.encode()
b'\xc3\x84pfel'
The default encoding does appear to be utf-8 in py3.
Although it is compiled with something different, and stores it as
something different, that is UCS2 or UCS4.
I imagine dtype 'S' and 'U' need more clarification. As it misses the
concept of encodings it seems? Currently, S appears to mean 8bit
characters no encoding, and U appears to mean 16bit characters no
encoding? Or are some sort of default encodings assumed?
2to3/3to2 fixers will probably have to be written for users code
here... whatever is decided. At least warnings should be generated
I'm guessing.
btw, in my numpy tree there is a unicode_() alias to str in py3, and
to unicode in py2 (inside the compat.py file). This helped us in many
cases with compatible string code in the pygame port. This allows you
to create unicode strings on both platforms with the same code.
cheers,
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Bytes vs. Unicode in Python3
On Fri, Nov 27, 2009 at 1:41 PM, Pauli Virtanen p...@iki.fi wrote: 2to3/3to2 fixers will probably have to be written for users code here... whatever is decided. At least warnings should be generated I'm guessing. Possibly. Does 2to3 support plugins? If yes, it could be possible to write one. You can put them in here: [lib_dir]lib2to3/fixes/fix_*.py I'm not sure about how to use custom ones without just copying them in... need to research that. There's no documentation about how to write custom ones here: http://docs.python.org/library/2to3.html You can pass lib2to3 a package to try import fixers from. However I'm not sure how to make that appear from the command line, other than copying the fixer into place. I guess the numpy setup script could copy the fixer into place. btw, in my numpy tree there is a unicode_() alias to str in py3, and to unicode in py2 (inside the compat.py file). This helped us in many cases with compatible string code in the pygame port. This allows you to create unicode strings on both platforms with the same code. Yes, I saw that. The name unicode_ is however already taken by the Numpy scalar type, so we need to think of a different name for it. asstring, maybe. something like numpy.compat.unicode_ ? Btw, do you want to rebase your distutils changes on top of my tree? I tried yours out quickly, but there were some issues there that prevented distutils from working. (Also, you can use absolute imports both for Python 2 and 3 -- there's probably no need to use relative imports.) Pauli hey, yeah I definitely would :) I don't have much time for the next week or so though. cu, ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Bytes vs. Unicode in Python3
On Fri, Nov 27, 2009 at 1:49 PM, Francesc Alted fal...@pytables.org wrote: Correct. But, in addition, we are going to need a new 'bytes' dtype for NumPy for Python 3, right? I think so. However, I think S is probably closest to bytes... and maybe S can be reused for bytes... I'm not sure though. Also, what will a bytes dtype mean within a py2 program context? Does it matter if the bytes dtype just fails somehow if used in a py2 program? cheers, ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Bytes vs. Unicode in Python3
On Fri, Nov 27, 2009 at 3:07 PM, René Dudfield ren...@gmail.com wrote: hey, yeah I definitely would :) I don't have much time for the next week or so though. btw, feel free to just copy whatever you like from there into your tree. cheers, ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Bytes vs. Unicode in Python3
On Fri, Nov 27, 2009 at 1:37 AM, Charles R Harris
charlesr.har...@gmail.com wrote:
Hi Pauli,
On Thu, Nov 26, 2009 at 4:08 PM, Pauli Virtanen p...@iki.fi wrote:
Hi,
The Python 3 porting needs some decisions on what is Bytes and
what is Unicode.
I'm currently taking the following approach. Comments?
***
dtype field names
Either Bytes or Unicode.
But 'a' and b'a' are *different* fields.
The issue is that:
Python 2: {'a': 2}[u'a'] == 2, {u'a': 2}['a'] == 2
Python 3: {'a': 2}[b'a'], {b'a': 2}['a'] raise exceptions
so the current assumptions in the C code of u'a' == b'a'
cease to hold.
dtype titles
If Bytes or Unicode, work similarly as field names.
dtype format strings, datetime tuple, and any other protocol strings
Bytes. User can pass in Unicode, but it's converted using
UTF8 codec.
This will likely change repr() of various objects. Acceptable?
I'm not clear on your recommendation here, is it that we should use bytes,
with unicode converted to UTF8? Will that support arrays that have been
pickled and such? Or will we just have a minimum of code to fix up? And
could you expand on the changes that repr() might undergo?
Mind, I think using bytes sounds best, but I haven't looked into the whole
strings part of the transition and don't have an informed opinion on the
matter.
Chuck
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
To help clarify for people who are not familiar with python 3...
To put it simply... in py3,
str == unicode utf-8, there is no 'unicode' anymore.
bytes == raw data... but kind of like a str type with less methods.
bytes + encoding is how you do non utf-8 strings.
'array' exists in both py2 and py3 with a very similar interface on both.
There's a more precise description of strings in python3 on these pages:
http://diveintopython3.org/strings.html
http://diveintopython3.org/porting-code-to-python-3-with-2to3.html
It depends on the use cases for each thing which will depend on how it
should work imho. Mostly if you are using the str type, then keep
using the str type.
Many functions take both bytes and strings. Since it is sane to work
on both bytes and strings from a users perspective. There have been
some methods in the stdlib that have not consumed both, and they have
been treated as bugs, and are being fixed (eg, some urllib methods).
For dtype, using the python 'str' by default seems ok. Since all of
those characters come out in the same manner on both pythons for the
data used by numpy.
eg. 'float32' is shown the same as a py3 string as a py2 string.
Internally it is unicode data however.
Within py2, we save a pickle with the str:
import pickle
pickle.dump(s, open('/tmp/p.pickle', 'wb'))
pickle.dump(s, open('/tmp/p.pickle', 'wb'))
pickle.dump('float32', open('/tmp/p.pickle', 'wb'))
Within py3 we open the pickle with the str:
import pickle
pickle.load(open('/tmp/p.pickle', 'rb'))
'float32'
cheers,
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] boolean arrays
On Thu, Nov 26, 2009 at 7:35 PM, Nils Wagner nwag...@iam.uni-stuttgart.de wrote: Sorry, I mixed up '+' and '' a = array(([True,True],[True,True])) b = array(([False,False],[False,False])) a b array([[False, False], [False, False]], dtype=bool) Cheers, Nils hey, this is a classical problem with + (sometimes pronounced 'and') and (also pronounced 'and'). Happens to all of us sometimes. cu, ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Numpy on Python3
On Mon, Nov 23, 2009 at 7:19 AM, David Cournapeau courn...@gmail.com wrote: On Mon, Nov 23, 2009 at 2:35 PM, Pauli Virtanen p...@iki.fi wrote: It might be nice to have this merged in at some point after 1.4.0 (after the most obvious glaring bugs have been fixed), so that we could perhaps start aiming for Python3 compatibility in Numpy 1.5.0. One thing I have on my end is a numpy.distutils which runs under both python 2 and 3, so that you don't have to run 2to3 everytime you make a change. I did not put it in the trunk because I did not want to modify numpy.distutils for 1.4.0 at this point, but I will include the changes as soon as I branch the trunk into 1.4.0, hi, my tree has this fairly far along (2 and 3 compatible). cheers, ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Numpy on Python3
awesome work Pauli! ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Numpy on Python3
On Mon, Nov 23, 2009 at 10:08 AM, Pauli Virtanen p...@iki.fi wrote: Setup.py runs 2to3 automatically for all changed files. Of course, if it's possible to cater for24 and 3 at the same time,that's good. How do you work around the relative imports andthe changed exception catching syntax? hi, see my tree for how I did these things. http://github.com/illume/numpy3k I'm not sure I like how I handled the changed relative imports. However the exception handling is fairly easy/clean to do. cheers, ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] 2to3c: 2to3 for C extensions
On Sun, Nov 22, 2009 at 7:16 PM, Pauli Virtanen p...@iki.fi wrote: la, 2009-11-21 kello 20:00 -0500, David Warde-Farley kirjoitti: Guido posted a link to this on Twitter, it might be of interest to people interested in the NumPy Python 3 transition: http://dmalcolm.livejournal.com/3935.html It's doubtful this is a magic bullet (at least, not yet) but it may still help reduce the amount of busywork involved. Interesting -- though at the moment it doesn't seem to solve very many of our problems. I did some cleanup of numpy.testing, distutils, and core so that using 2to3 the build_ext in numpy.core can at least be started: http://github.com/pv/numpy-work/tree/py3k python3 tools/py3test testing distutils core Of course, the build currently fails with zillion errors in multiarraymodule. Pauli hi, I've been working on this too... see the previous py3k thread for details of my plan. See http://github.com/illume/numpy3k/tree/work for my (not very much) progress. I don't have time for about a week to do any more work on my tree. Maybe your tree is ahead anyway... I am taking the approach we used with pygame so that the tree can work with both python 2.x and 3.x. I got up to getting a fair amount of the setup code working... but there are still some issues. The 2to3 script for C code does look useful. However it will not work in a backwards compatible manner. However maybe it can be modified to work in such a way. anyway... I'll look at your tree again next time I have time to work on this in a week or so. cheers! ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] 2to3c: 2to3 for C extensions
On Sun, Nov 22, 2009 at 7:28 PM, René Dudfield ren...@gmail.com wrote: On Sun, Nov 22, 2009 at 7:16 PM, Pauli Virtanen p...@iki.fi wrote: la, 2009-11-21 kello 20:00 -0500, David Warde-Farley kirjoitti: Guido posted a link to this on Twitter, it might be of interest to people interested in the NumPy Python 3 transition: http://dmalcolm.livejournal.com/3935.html It's doubtful this is a magic bullet (at least, not yet) but it may still help reduce the amount of busywork involved. Interesting -- though at the moment it doesn't seem to solve very many of our problems. I did some cleanup of numpy.testing, distutils, and core so that using 2to3 the build_ext in numpy.core can at least be started: http://github.com/pv/numpy-work/tree/py3k python3 tools/py3test testing distutils core Of course, the build currently fails with zillion errors in multiarraymodule. Pauli hi, I've been working on this too... see the previous py3k thread for details of my plan. See http://github.com/illume/numpy3k/tree/work for my (not very much) progress. I don't have time for about a week to do any more work on my tree. Maybe your tree is ahead anyway... I am taking the approach we used with pygame so that the tree can work with both python 2.x and 3.x. I got up to getting a fair amount of the setup code working... but there are still some issues. The 2to3 script for C code does look useful. However it will not work in a backwards compatible manner. However maybe it can be modified to work in such a way. hi again, I was wrong about it not generating 2 and 3 compatible source... the tool does do things like this: #if PY_MAJOR_VERSION = 3 foo(); #else bar(); #endif Very cool!!! cheers, ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Solaris Sparc build broken
On Fri, Nov 6, 2009 at 7:46 PM, David Cournapeau courn...@gmail.com wrote: On Sat, Nov 7, 2009 at 3:41 AM, David Cournapeau courn...@gmail.com wrote: I don't think LGPL has much meaning for python code, especially pure python code (and m4 for that matter) This is funny - besides pyqt, the only LGPL reference with python I found was twisted, and they claim that: ... It would be a halfway accurate statement that I selected the LGPL exactly because it doesn't make any sense. http://twistedmatrix.com/pipermail/twisted-python/2004-May/007946.html David pygame is also LGPL... as are a number of other libraries. (pyqt is GPL btw). LGPL basically means you can link to the library source, but if you make changes to the library you should give them back. Users should also be able to change the library if they want... either through source or relinking. That's the spirit of the LGPL in two sentences. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Objected-oriented SIMD API for Numpy
On Wed, Oct 21, 2009 at 2:14 PM, Pauli Virtanen pav...@iki.fipav%2...@iki.fi wrote: Wed, 21 Oct 2009 14:47:02 +0200, Francesc Alted wrote: [clip] Do you have any interest in adding SIMD to some core numpy (transcendental functions). If so, I would try to go back to the problem of runtime SSE detection and loading of optimized shared library in a cross-platform way - that's something which should be done at some point in numpy, and people requiring it would be a good incentive. I don't personally have a lot of interest implementing this for numpy. But in case anyone does, I find the next library: http://gruntthepeon.free.fr/ssemath/ very interesting. Perhaps there could be other (free) implementations... Optimized transcendental functions could be interesting. For example for tanh, call overhead is overcome already for ~30-element arrays. Since these are ufuncs, I suppose the SSE implementations could just be put in a separate module, which is always compiled. Before importing the module, we could simply check from Python side that the CPU supports the necessary instructions. If everything is OK, the accelerated implementations would then just replace the Numpy routines. This type of project could probably also be started outside Numpy, and just monkey-patch the Numpy routines on import. -- Pauli Virtanen ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion Anyone seen the corepy numpy gsoc project? http://numcorepy.blogspot.com/ It implements a number of functions with the corepy runtime assembler. The project showed nice simd speedups for numpy. I've been following the liborc project... which is a runtime assembler that uses a generic assembly language and supports many different simd assembly languages (eg SSE, MMX, ARM, Altivec). It's the replacement for the liboil library (used in gstreamer etc). http://code.entropywave.com/projects/orc/ cu! ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Best way to insert C code in numpy code
On Mon, Sep 21, 2009 at 8:12 PM, David Warde-Farley d...@cs.toronto.edu wrote: On 21-Sep-09, at 2:55 PM, Xavier Gnata wrote: Should I read that to learn you cython and numpy interact? Or is there another best documentation (with examples...)? You should have a look at the Bresenham algorithm thread you posted. I went to the trouble of converting some Python code for Bresenham's algorithm to Cython, and a pointer to the Cython+NumPy tutorial: http://wiki.cython.org/tutorials/numpy David I don't know about the best way... but here are two approaches I like... Another way is to make your C function then load it with ctypes(or wrap it with something else) and pass it pointers with array.ctype.data. You can find the shape of the array in python, and pass it to your C function. The benefit is it's just C code, and you can avoid the GIL too if you want. Then if you keep your C code separate from python stuff other people can use your C code in other languages more easily. cinpy is another one(similar to weave) which can be good for fast prototyping... http://www.cs.tut.fi/~ask/cinpy/ or for changing the code to fit your data. cheers, ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] I want to help with a numpy python 3.1.x port
On Sun, Sep 20, 2009 at 7:15 PM, Robert Kern robert.k...@gmail.com wrote: On Sun, Sep 20, 2009 at 13:13, René Dudfield ren...@gmail.com wrote: Hi again, I noticed numpy includes a copy of distutils. I guess because it's been modified in some way? numpy.distutils is a set of extensions to distutils; it is not a copy of distutils. cool thanks. btw, my work is in the 'work' branch here: http://github.com/illume/numpy3k/tree/work I probably could have called it something other than 'work'... but I just copy/pasted from the guide... so there you go. Only done a bit more on it so far... will try next weekend to get at least the setup.py build stuff running. I think that will be a good first step for me to post a diff, and then try and get it merged into svn. cu! ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Best way to insert C code in numpy code
On Tue, Sep 22, 2009 at 3:45 PM, Sturla Molden stu...@molden.no wrote:
Xavier Gnata skrev:
I have a large 2D numpy array as input and a 1D array as output.
In between, I would like to use C code.
C is requirement because it has to be fast and because the algorithm
cannot be written in a numpy oriented way :( (no way...really).
There are certain algorithms that cannot be vectorized, particularly
those that are recursive/iterative.
Hi,
one thing you can do is guess(predict) what the previous answer is and
continue on a path from that guess. Say you have 1000 processing
units, you could get the other 999 working on guesses for the answers
and go from there. If one of your guess paths is right, you might be
able to skip a bunch of steps. That's what cpus do with their
'speculative execution and branch prediction'. even more OT, you can
take advantage of this sometimes by getting the cpu to work out
multiple things for you at once by putting in well placed if/elses,
but that's fairly cpu specific. It's also used with servers... you
ask say 5 servers to give you a result, and wait for the first one to
give you the answer.
That cython pure module looks cool, thanks for pointing it out.
I wonder if anyone has tried using that with traces? So common paths
in your code record the types, and then can be used by the cython pure
module to try and generate the types for you? It could generate a
file containing a {callable : @cython.locals} mapping to be used on
compilation. Similar to how psyco works, but with a separate run
program/compilation step.
cheers,
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Numpy question: Best hardware for Numpy?
hi, Definitely memory speed is probably the biggest thing to consider. Also using 64bit if you need to do lots of calculations, and cache things. ACML is another alternative... but I've never tried linking it with numpy http://developer.amd.com/cpu/Libraries/acml/Pages/default.aspx AMD Phenom II is their latest chip, but I haven't tried that either. The chips in the latest mac pro run really quick :) Dual 4 core... with lower ghz, but faster memory speed makes my numpy stuff go faster than the higher ghz previous gen mac pro cpus. cu, ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Simple pattern recognition
On Mon, Sep 21, 2009 at 6:45 PM, Gökhan Sever gokhanse...@gmail.com wrote: I asked this question at http://stackoverflow.com/questions/1449139/simple-object-recognition and get lots of nice feedback, and finally I have managed to implement what I wanted. What I was looking for is named connected component labelling or analysis for my connected component extraction I have put the code (lab2.py) and the image (particles.png) under: http://code.google.com/p/ccnworks/source/browse/#svn/trunk/AtSc450/labs What do you think of improving that code and adding into scipy's ndimage library (like connected_components()) ? Comments and suggestions are welcome :) On Wed, Sep 16, 2009 at 7:22 PM, Gökhan Sever gokhanse...@gmail.com wrote: Hello all, I want to be able to count predefined simple rectangle shapes on an image as shown like in this one: http://img7.imageshack.us/img7/2327/particles.png Which is in my case to count all the blue pixels (they are ice-snow flake shadows in reality) in one of the column. What is the way to automate this task, which library or technique should I study to tackle it. Thanks. -- Gökhan Hi, cool! I didn't even know there was an ndimage... :) Something similar(connected components) can be found in pygame.transform and pygame.mask. However that code is in C. cheers, ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] I want to help with a numpy python 3.1.x port
Hi again, I noticed numpy includes a copy of distutils. I guess because it's been modified in some way? cheers, ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] I want to help with a numpy python 3.1.x port
On Sat, Sep 19, 2009 at 6:41 AM, Charles R Harris charlesr.har...@gmail.com wrote: Hi, I'll be uploading stuff to github at http://github.com/illume/numpy3k. git clone git://github.com/illume/numpy3k.git Hmm, that doesn't work for me, git finds no commits in common if I pull and, as far as I can tell, essentially clones the whole thing. I quess that's because I'm using git-svn and started in a different place for the initial co and every commit to svn rebases. Diffs might be the easiest way to go. Chuck ah, oh well. I'll just upload diffs as I go. Which versions of python is numpy to support? 2.5 and greater? Or 2.4 and greater? I can't see if it's written anywhere which versions are supported. Personally, I think at least python2.5 should be supported... as that's still very widely used... but older versions should be able to be supported with my changes if needed. cheers, ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] I want to help with a numpy python 3.1.x port
On Sat, Sep 19, 2009 at 10:31 AM, Pauli Virtanen p...@iki.fi wrote: pe, 2009-09-18 kello 23:41 -0600, Charles R Harris kirjoitti: [clip] Hmm, that doesn't work for me, git finds no commits in common if I pull and, as far as I can tell, essentially clones the whole thing. I quess that's because I'm using git-svn and started in a different place for the initial co and every commit to svn rebases. Diffs might be the easiest way to go. Use our Git mirror if you need to collaborate: http://projects.scipy.org/numpy/wiki/GitMirror It's there precisely to allow a central tree around which to base your changes, and avoids precisely these problems. Also, avoid touching git-svn as much as possible. If you want to preserve interoperability, git-svn dcommit to SVN always from a separate commit branch, to which you cherry-pick changes from the branch you use to collaborate. This is because git-svn rewrites the commits you dcommit to SVN. Cool, I'll follow that guide and set up a new repo... avoiding the git-svn parts of the guide. thanks. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] I want to help with a numpy python 3.1.x port
Hi again, I deleted the old git repo there, and I followed the scipy git guide and put the changes here: http://github.com/illume/numpy3k. git clone git://github.com/illume/numpy3k.git So hopefully you'll be able to take those changes from git into svn. If it's still not working... please let me know and I'll still use that git repo, but just upload diffs for you. cheers, ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] I want to help with a numpy python 3.1.x port
Hello, as a big numpy user, and someone wanting to help with the python 3 migration, I'd like to help with a python 3.1 port of numpy. We(at the pygame project) have mostly completed our port of pygame to python 3.0 and 3.1 so can offer some insight into what it takes with a CPython extension. pygame supports python 2.3 through to 3.1, so it should be possible to also keep backwards compatibility with the port for numpy. We can also use some of the helper code we used for the pygame port. I haven't asked the other pygame developers if they are interested in helping too... but maybe they will be interested in helping too(since they use numpy too). I'd also be interested in other people helping with the effort. Once I have put some of the ground work in place, and got a few parts done it should be easier for other people to see what needs changing. If the python 3 port is something you'd want included with numpy, then I'd like to begin this weekend. I'm not super familiar with the numpy code base yet, so I'd like to do it in small changes making small parts compatible with py3k and then having them reviewed/tested. I can either start a branch myself using mecurial, or perhaps you'd like me to do the work somewhere else ( like in a branch in the numpy svn?). Which code should I base the port off? trunk ? Or maybe I missed something and a port has started already? In which case, is there a branch somewhere I can help with? cheers, ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] I want to help with a numpy python 3.1.x port
On Fri, Sep 18, 2009 at 3:10 PM, Charles R Harris charlesr.har...@gmail.com wrote: Hi René, On Fri, Sep 18, 2009 at 6:01 AM, René Dudfield ren...@gmail.com wrote: Hello, as a big numpy user, and someone wanting to help with the python 3 migration, I'd like to help with a python 3.1 port of numpy. We(at the pygame project) have mostly completed our port of pygame to python 3.0 and 3.1 so can offer some insight into what it takes with a CPython extension. pygame supports python 2.3 through to 3.1, so it should be possible to also keep backwards compatibility with the port for numpy. We can also use some of the helper code we used for the pygame port. I haven't asked the other pygame developers if they are interested in helping too... but maybe they will be interested in helping too(since they use numpy too). I'd also be interested in other people helping with the effort. Once I have put some of the ground work in place, and got a few parts done it should be easier for other people to see what needs changing. If the python 3 port is something you'd want included with numpy, then I'd like to begin this weekend. I'm not super familiar with the numpy code base yet, so I'd like to do it in small changes making small parts compatible with py3k and then having them reviewed/tested. I can either start a branch myself using mecurial, or perhaps you'd like me to do the work somewhere else ( like in a branch in the numpy svn?). Which code should I base the port off? trunk ? Darren Dale and I are just getting started on a port and welcome any help you can offer. Because of the difficulty of maintaining two branches the only route that looks good at this point is to get the python parts of numpy in a state that will allow 2to3 to work and use #ifdef's in the c code. What was your experience with pygames? ah cool! That's great to hear. Where is your work going to be done? Well, we managed to make the code base runnable in python 2.x and 3 without using the 2to3 tool. We didn't need, or want to support different versions of source code. In lots of cases it's possible to support versions of python at once... if you aren't using any new python3 features that the older pythons don't have. For the C code there's a header which helps with a lot of the backward compat stuff. Had to implement a few things because of missing things... like simple slicing is gone from py3k (sq_slice) etc. Here's a header file we used with some compatibility macros... http://www.seul.org/viewcvs/viewcvs.cgi/trunk/src/pgcompat.h?rev=2049root=PyGamesortby=dateview=markup but we also have some other compat stuff in other headers... but I think most of the py3k stuff is in there. You could probably just copy that header file to use it. The module initialisation works slightly differently so we have #ifdef PY3 for those differences. See the bottom of this file for an example module: http://www.seul.org/viewcvs/viewcvs.cgi/trunk/src/rect.c?rev=2617root=PyGamesortby=dateview=markup For things like the missing sq_slice, rather than rewriting things we just wrote code to support stuff that py3k removed. Because the numpy c code is difficult to come to grips with the easiest part for inexperienced c coders and newbies is to start on is probably the python code. There is a fair amount of that so some plan of attack and a check list probably needs to be set up. Any such list should be kept in svn along with any helpful notes about problems/solutions encountered along the way. Well plenty of tests are something that's needed for sure... so add that to your checklist. It can be a good idea to get the setup.py, and testing framework ported early, so it's easier to check if code still works. I'm happy to help with the C stuff, and the .py stuff too. We also need David C. to commit his build changes for py3k so we can actually build the whole thing when the time comes. (Hint, hint). Also, I'm thinking of skipping 3.0 and starting with 3.1 because of the fixes, particularly the disappearance of cmp, that went in between the versions. Chuck yeah, skipping 3.0 is a good idea... just go straight to 3.1. We started our port during 3.0 before 3.1... but there weren't all that many differences for us between the two versions. cheers, ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] I want to help with a numpy python 3.1.x port
one more thing... there's also notes about porting to py3k here: http://wiki.python.org/moin/cporting and here: http://wiki.python.org/moin/PortingExtensionModulesToPy3k Which are better than the python.org docs for cporting. That's probably a pretty good page to store notes about porting as we go. cu, ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] I want to help with a numpy python 3.1.x port
On Fri, Sep 18, 2009 at 4:05 PM, Charles R Harris charlesr.har...@gmail.com wrote: On Fri, Sep 18, 2009 at 8:52 AM, René Dudfield ren...@gmail.com wrote: one more thing... there's also notes about porting to py3k here: http://wiki.python.org/moin/cporting and here: http://wiki.python.org/moin/PortingExtensionModulesToPy3k Which are better than the python.org docs for cporting. That's probably a pretty good page to store notes about porting as we go. Thanks! Numpy defines a lot of extension python types, so that is what I got started on, using NPY_PY3K as the flag. Numpy also exports the numeric operations (don't ask) and I think that needs to be changed so it looks like a reqular c++ in c class with getters and setters, which will make things a bit easier with the major changes that happened there. IIRC, there is an include file that provides the old conversions between python numeric types and c types. Did you use that? no, I don't know about that. We could give you commit privileges for this work, or we could work offline with git, i.e., you could use git svn and I would pull from you to make the commits. if that works for you, that sounds good. Should I clone from trunk, or is it going to be in a separate branch? cheers, ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] I want to help with a numpy python 3.1.x port
On Fri, Sep 18, 2009 at 4:29 PM, Charles R Harris charlesr.har...@gmail.com wrote: On Fri, Sep 18, 2009 at 9:09 AM, René Dudfield ren...@gmail.com wrote: On Fri, Sep 18, 2009 at 4:05 PM, Charles R Harris charlesr.har...@gmail.com wrote: On Fri, Sep 18, 2009 at 8:52 AM, René Dudfield ren...@gmail.com wrote: one more thing... there's also notes about porting to py3k here: http://wiki.python.org/moin/cporting and here: http://wiki.python.org/moin/PortingExtensionModulesToPy3k Which are better than the python.org docs for cporting. That's probably a pretty good page to store notes about porting as we go. Thanks! Numpy defines a lot of extension python types, so that is what I got started on, using NPY_PY3K as the flag. Numpy also exports the numeric operations (don't ask) and I think that needs to be changed so it looks like a reqular c++ in c class with getters and setters, which will make things a bit easier with the major changes that happened there. IIRC, there is an include file that provides the old conversions between python numeric types and c types. Did you use that? no, I don't know about that. We could give you commit privileges for this work, or we could work offline with git, i.e., you could use git svn and I would pull from you to make the commits. if that works for you, that sounds good. Should I clone from trunk, or is it going to be in a separate branch? If things work like we want, the changes will have to end up in trunk. That's not to say you can't work in a git branch. One concern I have is pulling between git repos that have been rebased to svn, as I understand it this can lead to problems. OTOH, I think things should work if both git repos are rebased to the same svn revision. Hmm, it might be best to work in a branch that is kept rebased to the master which is kept in sync with svn. Maybe someone who knows a bit more about git can weigh in here. Chuck Well, for now I can just send patches with a svn diff... if you'd be so kind as to apply them after a review :) Integrating working changes into trunk incrementally seems like a good idea (not know numpy dev process very well). cu, ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] I want to help with a numpy python 3.1.x port
On Fri, Sep 18, 2009 at 5:36 PM, René Dudfield ren...@gmail.com wrote: On Fri, Sep 18, 2009 at 5:33 PM, Charles R Harris charlesr.har...@gmail.com wrote: Numpy relies on nose for testing. I know that there is a py3k branch for nose but it doesn't look very active and I don't know its current state. Do you know anything about that? Chuck ah, bugger. No I don't. I can find out though... I'll write back with what I find. cu, ok... so this is the py3k branch here... svn checkout http://python-nose.googlecode.com/svn/branches/py3k nose3 It does look old, as nose has since moved to mecurial from svn. Apparently it works though... but I haven't tried it. Here is the mailing list post from april which says it's working: http://groups.google.com/group/nose-users/browse_thread/thread/3463fc48bad31ee8# btw, for python code you can run the test suite with python -3 to get it to print warnings about python3 incompatible things used. I think we could use this to at least start getting python3 incompatible stuff running on python2.6. In fact, that's where I'll start with. cheers, ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] I want to help with a numpy python 3.1.x port
On Fri, Sep 18, 2009 at 5:49 PM, René Dudfield ren...@gmail.com wrote: On Fri, Sep 18, 2009 at 5:36 PM, René Dudfield ren...@gmail.com wrote: On Fri, Sep 18, 2009 at 5:33 PM, Charles R Harris charlesr.har...@gmail.com wrote: Numpy relies on nose for testing. I know that there is a py3k branch for nose but it doesn't look very active and I don't know its current state. Do you know anything about that? Chuck ah, bugger. No I don't. I can find out though... I'll write back with what I find. cu, ok... so this is the py3k branch here... svn checkout http://python-nose.googlecode.com/svn/branches/py3k nose3 It does look old, as nose has since moved to mecurial from svn. Apparently it works though... but I haven't tried it. Here is the mailing list post from april which says it's working: http://groups.google.com/group/nose-users/browse_thread/thread/3463fc48bad31ee8# btw, for python code you can run the test suite with python -3 to get it to print warnings about python3 incompatible things used. I think we could use this to at least start getting python3 incompatible stuff running on python2.6. In fact, that's where I'll start with. cheers, Hi, I'll be uploading stuff to github at http://github.com/illume/numpy3k. git clone git://github.com/illume/numpy3k.git It's from numpy trunk... I hope that is ok. Not much there yet of course. Only added a PY3K.txt file with notes from this thread. This will be the file you talked about with the porting notes, and todos/status/plans etc. Will email when there's other stuff to look at. Maybe you could try pulling from there and committing that file to svn as a test that my setup is working? cheers, ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] I want to help with a numpy python 3.1.x port
Hi again, I found your numpy/core/src/py3k_notes.txt file. I left the PY3K.txt I made in there for now, so its easy for people to find... but will use the other file for more specific information like is in there now. Or I can move the information across from PY3K.txt into py3k_notes.txt if you prefer. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] I want to help with a numpy python 3.1.x port
hi, Added a numpy/compat.py file from pygame. This defines these things for compatibility: __all__ = ['geterror', 'long_', 'xrange_', 'ord_', 'unichr_', 'unicode_', 'raw_input_'] geterror() is useful for exceptions compatible between py3k and pyv2... As in py3k you can't do this: except ImportError, e: So with geterror() it's like this: except ImportError: e = compat.geterror() The other ones are just compatibility versions of stuff that's missing. If numpy/compat.py is the wrong place please let me know. cheers, ps. Also... I've asked Lenard, and Marcus from the pygame project if they mind giving their py3k related code from the LGPL pygame to numpy and Marcus has said yes. Marcus doesn't have the time for (yet another) project... but is willing to answer questions if we have any issues. Haven't heard back from Lenard yet... so better wait off before importing the pygame code into numpy until he gives the all clear. pps. I was wrong about the -3 flag for python2.6... that only warns about stuff it knows 2to3 can not handle. 2to3 comes with python2.6 as well as python3.0/3.1. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
