Re: [Numpy-discussion] Python 3 porting
Hi, Do you plan to make some noise about that when numpy2.0 will be release? IMHO you should. Do you for instance plan to have a clear announcement on the scipy web site? Xavier > Hi, > > The test suite passes now on Pythons 2.4 - 3.1. Further testing is very > welcome -- also on Python 2.x. Please check that your favourite software > still builds and works with SVN trunk Numpy. > > Currently, Scipy has some known failures because of > > (i) removed new= keyword in numpy.histogram > (ii) Cython supports only native size/alignment PEP 3118 buffers, and > Numpy arrays are most naturally expressed in the standardized > sizes. Supporting the full struct module alignment stuff appears > to be a slight PITA. I'll try to take a look at how to address > this. > > But everything else seems to work on Python 2.6. > > *** > > Python version 2.4.6 (#2, Jan 21 2010, 23:27:36) [GCC 4.4.1] > Ran 2509 tests in 18.892s > OK (KNOWNFAIL=4, SKIP=2) > > Python version 2.5.4 (r254:67916, Jan 20 2010, 21:44:03) [GCC 4.4.1] > Ran 2512 tests in 18.531s > OK (KNOWNFAIL=4) > > Python version 2.6.4 (r264:75706, Dec 7 2009, 18:45:15) [GCC 4.4.1] > Ran 2519 tests in 19.367s > OK (KNOWNFAIL=4) > > Python version 3.1.1+ (r311:74480, Nov 2 2009, 14:49:22) [GCC 4.4.1] > Ran 2518 tests in 23.239s > OK (KNOWNFAIL=5) > > > Cheers, > Pauli > > ___ > NumPy-Discussion mailing list > NumPy-Discussion@scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Python 3 porting
su, 2010-02-21 kello 20:45 +0100, Dag Sverre Seljebotn kirjoitti:
> Pauli Virtanen wrote:
[clip]
> > Currently, Scipy has some known failures because of
> >
> > (i) removed new= keyword in numpy.histogram
> > (ii) Cython supports only native size/alignment PEP 3118 buffers, and
> > Numpy arrays are most naturally expressed in the standardized
> > sizes. Supporting the full struct module alignment stuff appears
> > to be a slight PITA. I'll try
>
> Hmm. How much would it help if Cython dealt with standardized sizes when
> possible? Is Cython the only reason to have NumPy export native
> size/alignment?
Possibly. Anyway, I managed to implement this so that the format string
is in the native+aligned '@' form when possible, and falls back to the
unaligned alternatives when not.
Now the question is: should it prefer the standard unaligned types ('=')
or the native types ('^')? For non-native byte orders of course there is
only the standard alternative. This also means that long doubles or
64-bit long longs in non-native byte order cannot be exported.
Btw, do you know if the '@' format should include the padding xxx or
not? And if not, does the implicit padding also pad the end of the
structure to even alignment? Or is alignment <= itemsize always?
> Also, wouldn't it be a pain to export align=True dtypes in standard
> size/alignment?
Not really, as the padding needed for alignment is computed at the time
the dtype is constructed, so the necessary info is readily available.
It's actually exporting '@' dtypes properly that's painful, since this
requires thinking about what information must be omitted, and checking
when it is possible to do.
Notes to self:
- I think I possibly forgot the possible padding at the end of the
dtype in the provider.
- '@' format strings should probably not include padding that is
included in the alignment. I assumed this in the consumer interface,
but forgot about it in the provider. Roundtrips probably work all
right, though, since explicit padding causes zero implicit padding.
Pauli
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Python 3 porting
Pauli Virtanen wrote: > Hi, > > The test suite passes now on Pythons 2.4 - 3.1. Further testing is very > welcome -- also on Python 2.x. Please check that your favourite software > still builds and works with SVN trunk Numpy. > > Currently, Scipy has some known failures because of > > (i) removed new= keyword in numpy.histogram > (ii) Cython supports only native size/alignment PEP 3118 buffers, and > Numpy arrays are most naturally expressed in the standardized > sizes. Supporting the full struct module alignment stuff appears > to be a slight PITA. I'll try Hmm. How much would it help if Cython dealt with standardized sizes when possible? Is Cython the only reason to have NumPy export native size/alignment? Also, wouldn't it be a pain to export align=True dtypes in standard size/alignment? (As a quick hack in SciPy, there's always np.ndarray[int, cast=True] to skip the format string checking (size is still checked).) Dag Sverre ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Python 3 porting
AWESOME :) On Sun, Feb 21, 2010 at 1:43 PM, Pauli Virtanen wrote: > Hi, > > The test suite passes now on Pythons 2.4 - 3.1. Further testing is very > welcome -- also on Python 2.x. Please check that your favourite software > still builds and works with SVN trunk Numpy. > > Currently, Scipy has some known failures because of > > (i) removed new= keyword in numpy.histogram > (ii) Cython supports only native size/alignment PEP 3118 buffers, and > Numpy arrays are most naturally expressed in the standardized > sizes. Supporting the full struct module alignment stuff appears > to be a slight PITA. I'll try to take a look at how to address > this. > > But everything else seems to work on Python 2.6. > >*** > > Python version 2.4.6 (#2, Jan 21 2010, 23:27:36) [GCC 4.4.1] > Ran 2509 tests in 18.892s > OK (KNOWNFAIL=4, SKIP=2) > > Python version 2.5.4 (r254:67916, Jan 20 2010, 21:44:03) [GCC 4.4.1] > Ran 2512 tests in 18.531s > OK (KNOWNFAIL=4) > > Python version 2.6.4 (r264:75706, Dec 7 2009, 18:45:15) [GCC 4.4.1] > Ran 2519 tests in 19.367s > OK (KNOWNFAIL=4) > > Python version 3.1.1+ (r311:74480, Nov 2 2009, 14:49:22) [GCC 4.4.1] > Ran 2518 tests in 23.239s > OK (KNOWNFAIL=5) > > > Cheers, > Pauli > > ___ > NumPy-Discussion mailing list > NumPy-Discussion@scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Python 3 porting
Hi, The test suite passes now on Pythons 2.4 - 3.1. Further testing is very welcome -- also on Python 2.x. Please check that your favourite software still builds and works with SVN trunk Numpy. Currently, Scipy has some known failures because of (i) removed new= keyword in numpy.histogram (ii) Cython supports only native size/alignment PEP 3118 buffers, and Numpy arrays are most naturally expressed in the standardized sizes. Supporting the full struct module alignment stuff appears to be a slight PITA. I'll try to take a look at how to address this. But everything else seems to work on Python 2.6. *** Python version 2.4.6 (#2, Jan 21 2010, 23:27:36) [GCC 4.4.1] Ran 2509 tests in 18.892s OK (KNOWNFAIL=4, SKIP=2) Python version 2.5.4 (r254:67916, Jan 20 2010, 21:44:03) [GCC 4.4.1] Ran 2512 tests in 18.531s OK (KNOWNFAIL=4) Python version 2.6.4 (r264:75706, Dec 7 2009, 18:45:15) [GCC 4.4.1] Ran 2519 tests in 19.367s OK (KNOWNFAIL=4) Python version 3.1.1+ (r311:74480, Nov 2 2009, 14:49:22) [GCC 4.4.1] Ran 2518 tests in 23.239s OK (KNOWNFAIL=5) Cheers, Pauli ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Python 3 porting
ti, 2010-02-16 kello 12:11 -0600, Bruce Southey kirjoitti: [clip] > I managed to get 2to3 (I think from Python 3.1) to crash and isolated > it to the file numpy-work/numpy/lib/arrayterator.py > > So I might hitting this ' assertion error in 2to3' bug: > http://bugs.python.org/issue7824 > > I try to get the latest version of 2to3 and try again. Try the latest head of the py3k branch, this should be worked around there so that also earlier 2to3 work. -- Pauli Virtanen ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Python 3 porting
New try new error: gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions build/temp.linux-x86_64-3.1/numpy/core/src/multiarray/multiarraymodule_onefile.o -Lbuild/temp.linux-x86_64-3.1 -lnpymath -lm -o build/lib.linux-x86_64-3.1/numpy/core/multiarray.so /usr/bin/ld: build/temp.linux-x86_64-3.1/numpy/core/src/multiarray/multiarraymodule_onefile.o: relocation R_X86_64_PC32 against undefined symbol `_numpymemoryview_init' can not be used when making a shared object; recompile with -fPIC /usr/bin/ld: final link failed: Bad value collect2: ld returned 1 exit status /usr/bin/ld: build/temp.linux-x86_64-3.1/numpy/core/src/multiarray/multiarraymodule_onefile.o: relocation R_X86_64_PC32 against undefined symbol `_numpymemoryview_init' can not be used when making a shared object; recompile with -fPIC /usr/bin/ld: final link failed: Bad value collect2: ld returned 1 exit status error: Command "gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions build/temp.linux-x86_64-3.1/numpy/core/src/multiarray/multiarraymodule_onefile.o -Lbuild/temp.linux-x86_64-3.1 -lnpymath -lm -o build/lib.linux-x86_64-3.1/numpy/core/multiarray.so" failed with exit status 1 No clue why :( Xavier >> Ok! >> git clone git://github.com/pv/numpy-work.git >> git checkout origin/py3k >> NPY_SEPARATE_BUILD=1 python3.1 setup.py build >> >> but now it fails during the build: >> >> In file included from numpy/core/src/multiarray/buffer.c:14, >> from numpy/core/src/multiarray/multiarraymodule_onefile.c:36: >> numpy/core/src/multiarray/buffer.h: At top level: >> numpy/core/src/multiarray/buffer.h:14: error: conflicting types for >> ‘_descriptor_from_pep3118_format’ >> numpy/core/src/multiarray/common.c:220: note: previous implicit >> declaration of ‘_descriptor_from_pep3118_format’ was here >> In file included from >> numpy/core/src/multiarray/multiarraymodule_onefile.c:36: >> numpy/core/src/multiarray/buffer.c: In function ‘_buffer_format_string’: >> numpy/core/src/multiarray/buffer.c:151: warning: unused variable ‘repr’ >> > Hmm, I probably tested only the separate compilation properly as it > seems the single-file build is failing. The environment variable is > actually NPY_SEPARATE_COMPILATION=1, not *_BUILD. > > ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Python 3 porting
On 02/15/2010 10:55 AM, Pauli Virtanen wrote:
ma, 2010-02-15 kello 10:18 -0600, Bruce Southey kirjoitti:
[clip]
Is there a correct way to get Python3.1 to find the relative path on Linux?
I can change the import statement to work but I do not think that is viable.
You need to use relative imports. 2to3 should be able to take care of
this.
[clip]
File "/usr/local/lib/python3.1/site-packages/numpy/lib/__init__.py",
line 1, in
from info import __doc__
ImportError: No module named info
That statement should read
from .info import __doc__
and indeed, it reads like that for me. Check how it is in
build/py3k/numpy/lib/__init__.py
Most likely you interrupted the build by Ctrl+C and 2to3 did not finish
the conversion of the files to Python3 format. Try removing the build/
directory and trying again -- if you interrupt it, 2to3 may not have
finished running.
Of course, it should be more robust, but at the moment, it isn't
(patches welcome).
Hi,
I managed to get 2to3 (I think from Python 3.1) to crash and isolated it
to the file numpy-work/numpy/lib/arrayterator.py
So I might hitting this ' assertion error in 2to3' bug:
http://bugs.python.org/issue7824
I try to get the latest version of 2to3 and try again.
Bruce
$ 2to3 -w arrayterator.py
RefactoringTool: Skipping implicit fixer: buffer
RefactoringTool: Skipping implicit fixer: idioms
RefactoringTool: Skipping implicit fixer: set_literal
RefactoringTool: Skipping implicit fixer: ws_comma
Traceback (most recent call last):
File "/usr/local/bin/2to3", line 6, in
sys.exit(main("lib2to3.fixes"))
File "/usr/local/lib/python3.1/lib2to3/main.py", line 159, in main
options.processes)
File "/usr/local/lib/python3.1/lib2to3/refactor.py", line 616, in
refactor
items, write, doctests_only)
File "/usr/local/lib/python3.1/lib2to3/refactor.py", line 276, in
refactor
self.refactor_file(dir_or_file, write, doctests_only)
File "/usr/local/lib/python3.1/lib2to3/refactor.py", line 656, in
refactor_file
*args, **kwargs)
File "/usr/local/lib/python3.1/lib2to3/refactor.py", line 328, in
refactor_file
tree = self.refactor_string(input, filename)
File "/usr/local/lib/python3.1/lib2to3/refactor.py", line 358, in
refactor_string
self.refactor_tree(tree, name)
File "/usr/local/lib/python3.1/lib2to3/refactor.py", line 392, in
refactor_tree
self.traverse_by(self.post_order_heads, tree.post_order())
File "/usr/local/lib/python3.1/lib2to3/refactor.py", line 418, in
traverse_by
node.replace(new)
File "/usr/local/lib/python3.1/lib2to3/pytree.py", line 133, in replace
assert self.parent is not None, str(self)
AssertionError:
def __init__(self, var, buf_size=None):
self.var = var
self.buf_size = buf_size
self.start = [0 for dim in var.shape]
self.stop = [dim for dim in var.shape]
self.step = [1 for dim in var.shape]
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Python 3 porting
ma, 2010-02-15 kello 15:51 -0700, Charles R Harris kirjoitti: [clip] > A lot of the remaining failures are of this sort: > > x: array([b'pi', b'pi', b'pi', b'four', b'five'], > dtype='|S8') > y: array(['pi', 'pi', 'pi', 'four', 'five'], > dtype=' > > This looks fixable by specifying the dtype Specifying the dtype in the test changes the meaning of the test. Rather, the expected results should be made bytes on Py3. This is what I've done so far. There are asbytes() and asbytes_nested() macros available in numpy.compat that can be used to portably get bytes literals. > >>> np.array([b'pi']) > array([b'pi'], > dtype='|S2') > >>> np.array(['pi']) > array(['pi'], > dtype=' >>> np.array(['pi'], dtype='|S2') > array([b'pi'], > dtype='|S2') > > I expect we will break a lot of code if b'pi' can't somehow be made > the default. I don't think we should make the unicode str map to bytes_ dtype, it's just too magical. Any Python code being ported to Py3 will anyway need to go the str vs. bytes transition so there will be breakage in any case. > Hmm. The 'b' prefix is an undocumented feature of python 2.6 but > doesn't work for earlier versions. But these tests can be fixed by > being a bit more explicit about the type. I think the doctests can be partly fixed by using asstr() from numpy.compat. Probably not completely, though -- I've seen some complaints that doctests are a lot of work to convert to Py3. Pauli ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Python 3 porting
On Mon, Feb 15, 2010 at 3:19 PM, Charles R Harris wrote: > > > On Mon, Feb 15, 2010 at 3:05 PM, Pauli Virtanen wrote: > >> ma, 2010-02-15 kello 21:58 +0200, Pauli Virtanen kirjoitti: >> [clip] >> > > Program received signal SIGABRT, Aborted. >> > > 0x7fff84e1efe6 in __kill () >> > > (gdb) bt >> > > #0 0x7fff84e1efe6 in __kill () >> > > #1 0x7fff84ebfe32 in abort () >> > > #2 0x0001000cd715 in Py_FatalError () >> > > #3 0x000100069508 in unicode_dealloc () >> > > #4 0x00010065cd1f in UNICODE_to_STRING (ip=0x101f06fd0 "a", >> > > op=0x101f01470 "abc", n=4, aip=0x101ce5de8, aop=0x101901a28) at >> > > arraytypes.c.src:1501 >> > > #5 0x0001006771ca in PyArray_CastTo (out=0x101901a28, >> > > mp=0x101ce5de8) at convert_datatype.c:336 >> > > #6 0x0001006772e1 in PyArray_CastToType (mp=0x101ce5de8, >> > > at=, fortran_=0) >> > > at convert_datatype.c:73 >> > > #7 0x000100681569 in array_cast (self=0x101ce5de8, args=> > > temporarily unavailable, due to optimizations>) at methods.c:760 >> > >> > The platform is OSX -- 32 or 64 bits? Is your Python unicode narrow or >> > wide? Which test triggers the issue? >> >> Ok, I think I managed to nail that and all other remaining 64-bit >> specific issues. Also Python 3.1.1's 2to3 should now work. >> >> > Much better: > > FAILED (KNOWNFAIL=4, SKIP=4, errors=35, failures=51) > > A lot of the remaining failures are of this sort: x: array([b'pi', b'pi', b'pi', b'four', b'five'], dtype='|S8') y: array(['pi', 'pi', 'pi', 'four', 'five'], dtype='>> np.array([b'pi']) array([b'pi'], dtype='|S2') >>> np.array(['pi']) array(['pi'], dtype='>> np.array(['pi'], dtype='|S2') array([b'pi'], dtype='|S2') I expect we will break a lot of code if b'pi' can't somehow be made the default. Hmm. The 'b' prefix is an undocumented feature of python 2.6 but doesn't work for earlier versions. But these tests can be fixed by being a bit more explicit about the type. More problematic are failing doctests, mostly because of unconverted print statements. Chuck ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Python 3 porting
On Mon, Feb 15, 2010 at 3:05 PM, Pauli Virtanen wrote: > ma, 2010-02-15 kello 21:58 +0200, Pauli Virtanen kirjoitti: > [clip] > > > Program received signal SIGABRT, Aborted. > > > 0x7fff84e1efe6 in __kill () > > > (gdb) bt > > > #0 0x7fff84e1efe6 in __kill () > > > #1 0x7fff84ebfe32 in abort () > > > #2 0x0001000cd715 in Py_FatalError () > > > #3 0x000100069508 in unicode_dealloc () > > > #4 0x00010065cd1f in UNICODE_to_STRING (ip=0x101f06fd0 "a", > > > op=0x101f01470 "abc", n=4, aip=0x101ce5de8, aop=0x101901a28) at > > > arraytypes.c.src:1501 > > > #5 0x0001006771ca in PyArray_CastTo (out=0x101901a28, > > > mp=0x101ce5de8) at convert_datatype.c:336 > > > #6 0x0001006772e1 in PyArray_CastToType (mp=0x101ce5de8, > > > at=, fortran_=0) > > > at convert_datatype.c:73 > > > #7 0x000100681569 in array_cast (self=0x101ce5de8, args= > > temporarily unavailable, due to optimizations>) at methods.c:760 > > > > The platform is OSX -- 32 or 64 bits? Is your Python unicode narrow or > > wide? Which test triggers the issue? > > Ok, I think I managed to nail that and all other remaining 64-bit > specific issues. Also Python 3.1.1's 2to3 should now work. > > Much better: FAILED (KNOWNFAIL=4, SKIP=4, errors=35, failures=51) Chuck ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Python 3 porting
ma, 2010-02-15 kello 21:58 +0200, Pauli Virtanen kirjoitti: [clip] > > Program received signal SIGABRT, Aborted. > > 0x7fff84e1efe6 in __kill () > > (gdb) bt > > #0 0x7fff84e1efe6 in __kill () > > #1 0x7fff84ebfe32 in abort () > > #2 0x0001000cd715 in Py_FatalError () > > #3 0x000100069508 in unicode_dealloc () > > #4 0x00010065cd1f in UNICODE_to_STRING (ip=0x101f06fd0 "a", > > op=0x101f01470 "abc", n=4, aip=0x101ce5de8, aop=0x101901a28) at > > arraytypes.c.src:1501 > > #5 0x0001006771ca in PyArray_CastTo (out=0x101901a28, > > mp=0x101ce5de8) at convert_datatype.c:336 > > #6 0x0001006772e1 in PyArray_CastToType (mp=0x101ce5de8, > > at=, fortran_=0) > > at convert_datatype.c:73 > > #7 0x000100681569 in array_cast (self=0x101ce5de8, args= > temporarily unavailable, due to optimizations>) at methods.c:760 > > The platform is OSX -- 32 or 64 bits? Is your Python unicode narrow or > wide? Which test triggers the issue? Ok, I think I managed to nail that and all other remaining 64-bit specific issues. Also Python 3.1.1's 2to3 should now work. Cheers, Pauli ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Python 3 porting
On 15 February 2010 23:11, Charles R Harris wrote: >> OSX, 64-bit with (I'm assuming) UCS2 (since I didn't specify the UCS4 >> variant when building). Is there an easy way to check the unicode >> width? >> > I found it in the pyconfig.h file: > > pyconfig.h:#define Py_UNICODE_SIZE 4' Aha! #define Py_UNICODE_SIZE 2 Stéfan ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Python 3 porting
2010/2/15 Stéfan van der Walt > On 15 February 2010 21:58, Pauli Virtanen wrote: > > ma, 2010-02-15 kello 21:41 +0200, Stéfan van der Walt kirjoitti: > > [clip] > >> .Fatal Python error: Inconsistent interned string > >> state. > >> > > The platform is OSX -- 32 or 64 bits? Is your Python unicode narrow or > > wide? Which test triggers the issue? > > OSX, 64-bit with (I'm assuming) UCS2 (since I didn't specify the UCS4 > variant when building). Is there an easy way to check the unicode > width? > > I found it in the pyconfig.h file: pyconfig.h:#define Py_UNICODE_SIZE 4' Chuck ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Python 3 porting
On 15 February 2010 21:58, Pauli Virtanen wrote: > ma, 2010-02-15 kello 21:41 +0200, Stéfan van der Walt kirjoitti: > [clip] >> .Fatal Python error: Inconsistent interned string >> state. >> > The platform is OSX -- 32 or 64 bits? Is your Python unicode narrow or > wide? Which test triggers the issue? OSX, 64-bit with (I'm assuming) UCS2 (since I didn't specify the UCS4 variant when building). Is there an easy way to check the unicode width? The test that failed was "test_from_unicode_array (test_defchararray.TestBasic)". Cheers Stéfan ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Python 3 porting
On Mon, Feb 15, 2010 at 12:58 PM, Pauli Virtanen wrote: > ma, 2010-02-15 kello 21:41 +0200, Stéfan van der Walt kirjoitti: > [clip] > > > On Python 3.1, the numpymemoryview_init is a stub function that does > > > nothing. I guess this is another single-file compilation issue -- the > > > new file should be included in multiarraymodule_onefile.c. Should be > > > fixed now. > > > > Thanks, it works now. > > > > I wonder if 2to3 is doing its job, though. I had to make the attached > > changes before I could import. > > This seems to be the same problem Bruce had. > > Doesn't seem like 2to3 is doing what it's supposed to do. In fact, the > 2to3 shipped with Python 2.6 does those changes by itself, but the one > with Python 3.1.1 does not. Obviously, I was all the time using Python > 2.6 shipped 2to3. I'll try to find a workaround. > > > > Progress: the unit test suite starts to run, but fails soon after. > > > > .Fatal Python error: Inconsistent interned string > > state. > > > > Program received signal SIGABRT, Aborted. > > 0x7fff84e1efe6 in __kill () > > (gdb) bt > > #0 0x7fff84e1efe6 in __kill () > > #1 0x7fff84ebfe32 in abort () > > #2 0x0001000cd715 in Py_FatalError () > > #3 0x000100069508 in unicode_dealloc () > > #4 0x00010065cd1f in UNICODE_to_STRING (ip=0x101f06fd0 "a", > > op=0x101f01470 "abc", n=4, aip=0x101ce5de8, aop=0x101901a28) at > > arraytypes.c.src:1501 > > #5 0x0001006771ca in PyArray_CastTo (out=0x101901a28, > > mp=0x101ce5de8) at convert_datatype.c:336 > > #6 0x0001006772e1 in PyArray_CastToType (mp=0x101ce5de8, > > at=, fortran_=0) > > at convert_datatype.c:73 > > #7 0x000100681569 in array_cast (self=0x101ce5de8, args= > temporarily unavailable, due to optimizations>) at methods.c:760 > > The platform is OSX -- 32 or 64 bits? Is your Python unicode narrow or > wide? Which test triggers the issue? > > Is there an easy way to discover what the unicode size is? Chuck ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Python 3 porting
ma, 2010-02-15 kello 21:41 +0200, Stéfan van der Walt kirjoitti: [clip] > > On Python 3.1, the numpymemoryview_init is a stub function that does > > nothing. I guess this is another single-file compilation issue -- the > > new file should be included in multiarraymodule_onefile.c. Should be > > fixed now. > > Thanks, it works now. > > I wonder if 2to3 is doing its job, though. I had to make the attached > changes before I could import. This seems to be the same problem Bruce had. Doesn't seem like 2to3 is doing what it's supposed to do. In fact, the 2to3 shipped with Python 2.6 does those changes by itself, but the one with Python 3.1.1 does not. Obviously, I was all the time using Python 2.6 shipped 2to3. I'll try to find a workaround. > Progress: the unit test suite starts to run, but fails soon after. > > .Fatal Python error: Inconsistent interned string > state. > > Program received signal SIGABRT, Aborted. > 0x7fff84e1efe6 in __kill () > (gdb) bt > #0 0x7fff84e1efe6 in __kill () > #1 0x7fff84ebfe32 in abort () > #2 0x0001000cd715 in Py_FatalError () > #3 0x000100069508 in unicode_dealloc () > #4 0x00010065cd1f in UNICODE_to_STRING (ip=0x101f06fd0 "a", > op=0x101f01470 "abc", n=4, aip=0x101ce5de8, aop=0x101901a28) at > arraytypes.c.src:1501 > #5 0x0001006771ca in PyArray_CastTo (out=0x101901a28, > mp=0x101ce5de8) at convert_datatype.c:336 > #6 0x0001006772e1 in PyArray_CastToType (mp=0x101ce5de8, > at=, fortran_=0) > at convert_datatype.c:73 > #7 0x000100681569 in array_cast (self=0x101ce5de8, args= temporarily unavailable, due to optimizations>) at methods.c:760 The platform is OSX -- 32 or 64 bits? Is your Python unicode narrow or wide? Which test triggers the issue? Thanks, Pauli ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Python 3 porting
2010/2/15 Stéfan van der Walt : > Thanks, it works now. Progress: the unit test suite starts to run, but fails soon after. .Fatal Python error: Inconsistent interned string state. Program received signal SIGABRT, Aborted. 0x7fff84e1efe6 in __kill () (gdb) bt #0 0x7fff84e1efe6 in __kill () #1 0x7fff84ebfe32 in abort () #2 0x0001000cd715 in Py_FatalError () #3 0x000100069508 in unicode_dealloc () #4 0x00010065cd1f in UNICODE_to_STRING (ip=0x101f06fd0 "a", op=0x101f01470 "abc", n=4, aip=0x101ce5de8, aop=0x101901a28) at arraytypes.c.src:1501 #5 0x0001006771ca in PyArray_CastTo (out=0x101901a28, mp=0x101ce5de8) at convert_datatype.c:336 #6 0x0001006772e1 in PyArray_CastToType (mp=0x101ce5de8, at=, fortran_=0) at convert_datatype.c:73 #7 0x000100681569 in array_cast (self=0x101ce5de8, args=) at methods.c:760 Cheers Stéfan ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Python 3 porting
On 15 February 2010 21:19, Pauli Virtanen wrote: > Oh crap, nothing seems to work for anyone else ;) Don't speak too soon: we have import! > On Python 3.1, the numpymemoryview_init is a stub function that does > nothing. I guess this is another single-file compilation issue -- the > new file should be included in multiarraymodule_onefile.c. Should be > fixed now. Thanks, it works now. I wonder if 2to3 is doing its job, though. I had to make the attached changes before I could import. Stéfan py3k_imports.patch Description: Binary data ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Python 3 porting
ma, 2010-02-15 kello 21:07 +0200, Stéfan van der Walt kirjoitti: [clip] > After your latest changes, numpy builds on OSX, although importing is > still broken: > > from . import multiarray > ImportError: > dlopen(/Users/stefan/lib/python3.1/site-packages/numpy/core/multiarray.so, > 2): Symbol not found: __numpymemoryview_init > Referenced from: > /Users/stefan/lib/python3.1/site-packages/numpy/core/multiarray.so > Expected in: flat namespace > in /Users/stefan/lib/python3.1/site-packages/numpy/core/multiarray.so Oh crap, nothing seems to work for anyone else ;) > Have you seen this before? Looks like something isn't linked > properly, but I'm not sure where memoryview would be defined. Is this > part of the new PEP implementation? Yep, it's a part of that, but it's only necessary on Python 2.6. The Memoryview object is a part of Python proper starting from Python 2.7. It was a huge convenience for the implementation to be able to keep track of buffers via refcounting, so I backported that bit. On Python 3.1, the numpymemoryview_init is a stub function that does nothing. I guess this is another single-file compilation issue -- the new file should be included in multiarraymodule_onefile.c. Should be fixed now. Pauli ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Python 3 porting
Hi Pauli Well done! You and Charles have made huge strides since last I looked at the problem. After your latest changes, numpy builds on OSX, although importing is still broken: from . import multiarray ImportError: dlopen(/Users/stefan/lib/python3.1/site-packages/numpy/core/multiarray.so, 2): Symbol not found: __numpymemoryview_init Referenced from: /Users/stefan/lib/python3.1/site-packages/numpy/core/multiarray.so Expected in: flat namespace in /Users/stefan/lib/python3.1/site-packages/numpy/core/multiarray.so Have you seen this before? Looks like something isn't linked properly, but I'm not sure where memoryview would be defined. Is this part of the new PEP implementation? Regards Stéfan On 15 February 2010 19:55, Pauli Virtanen wrote: > ma, 2010-02-15 kello 10:23 -0700, Charles R Harris kirjoitti: > [clip] >> Segfaults: >> >> test_multiarray.TestNewBufferProtocol.test_export_simple_1d ... FAIL >> test_multiarray.TestNewBufferProtocol.test_export_simple_nd ... ok >> test_multiarray.TestNewBufferProtocol.test_export_subarray ... FAIL >> test_multiarray.TestNewBufferProtocol.test_roundtrip ... Segmentation >> fault > > Worksforme, and the tests that FAIL above also pass for me... No idea > what could be different. > >> Are there changes you haven't pushed to github? > > No. The current is commit 2132bdf550d12af5c2198027182778a47d5d19ab > >> I don't want to be making fixes that already exist. It would also be >> easier to work on this if the current state was in the main repository >> so that the rest of us could push changes. > > I will push the changes to SVN once I clean up some parts of the commit > history. I'll try to do this ASAP. > > Anyway, I will not rebase the py3k branch so it's safe to work on, and > I'll push any new stuff immediately there. > > Pauli > > > > > ___ > NumPy-Discussion mailing list > NumPy-Discussion@scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Python 3 porting
On Mon, Feb 15, 2010 at 10:55 AM, Pauli Virtanen wrote: > ma, 2010-02-15 kello 10:18 -0600, Bruce Southey kirjoitti: > [clip] >> Is there a correct way to get Python3.1 to find the relative path on Linux? >> I can change the import statement to work but I do not think that is viable. > > You need to use relative imports. 2to3 should be able to take care of > this. > > [clip] >> File "/usr/local/lib/python3.1/site-packages/numpy/lib/__init__.py", >> line 1, in >> from info import __doc__ >> ImportError: No module named info > > That statement should read > > from .info import __doc__ > > and indeed, it reads like that for me. Check how it is in > build/py3k/numpy/lib/__init__.py Not for me > > Most likely you interrupted the build by Ctrl+C and 2to3 did not finish > the conversion of the files to Python3 format. Try removing the build/ > directory and trying again -- if you interrupt it, 2to3 may not have > finished running. Nope. I'll go through the log and see if anything looks weird. > Of course, it should be more robust, but at the moment, it isn't > (patches welcome). > Well that it is the whole point of trying this as I would like to move to Python 3. Bruce ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Python 3 porting
On Mon, Feb 15, 2010 at 10:55 AM, Pauli Virtanen wrote: > ma, 2010-02-15 kello 10:23 -0700, Charles R Harris kirjoitti: > [clip] > > Segfaults: > > > > test_multiarray.TestNewBufferProtocol.test_export_simple_1d ... FAIL > > test_multiarray.TestNewBufferProtocol.test_export_simple_nd ... ok > > test_multiarray.TestNewBufferProtocol.test_export_subarray ... FAIL > > test_multiarray.TestNewBufferProtocol.test_roundtrip ... Segmentation > > fault > > Worksforme, and the tests that FAIL above also pass for me... No idea > what could be different. > > Metadata: this is on ubuntu karmic 64 bit with the distro version of python3.1. Chuck > > Are there changes you haven't pushed to github? > > No. The current is commit 2132bdf550d12af5c2198027182778a47d5d19ab > > > I don't want to be making fixes that already exist. It would also be > > easier to work on this if the current state was in the main repository > > so that the rest of us could push changes. > > I will push the changes to SVN once I clean up some parts of the commit > history. I'll try to do this ASAP. > > Great. Chuck ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Python 3 porting
ma, 2010-02-15 kello 10:23 -0700, Charles R Harris kirjoitti: [clip] > Segfaults: > > test_multiarray.TestNewBufferProtocol.test_export_simple_1d ... FAIL > test_multiarray.TestNewBufferProtocol.test_export_simple_nd ... ok > test_multiarray.TestNewBufferProtocol.test_export_subarray ... FAIL > test_multiarray.TestNewBufferProtocol.test_roundtrip ... Segmentation > fault Worksforme, and the tests that FAIL above also pass for me... No idea what could be different. > Are there changes you haven't pushed to github? No. The current is commit 2132bdf550d12af5c2198027182778a47d5d19ab > I don't want to be making fixes that already exist. It would also be > easier to work on this if the current state was in the main repository > so that the rest of us could push changes. I will push the changes to SVN once I clean up some parts of the commit history. I'll try to do this ASAP. Anyway, I will not rebase the py3k branch so it's safe to work on, and I'll push any new stuff immediately there. Pauli ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Python 3 porting
On Mon, Feb 15, 2010 at 9:55 AM, Pauli Virtanen wrote: > ma, 2010-02-15 kello 10:18 -0600, Bruce Southey kirjoitti: > [clip] > > Is there a correct way to get Python3.1 to find the relative path on > Linux? > > I can change the import statement to work but I do not think that is > viable. > > You need to use relative imports. 2to3 should be able to take care of > this. > > [clip] > >File "/usr/local/lib/python3.1/site-packages/numpy/lib/__init__.py", > > line 1, in > > from info import __doc__ > > ImportError: No module named info > > That statement should read > >from .info import __doc__ > > and indeed, it reads like that for me. Check how it is in > build/py3k/numpy/lib/__init__.py > > Most likely you interrupted the build by Ctrl+C and 2to3 did not finish > the conversion of the files to Python3 format. Try removing the build/ > directory and trying again -- if you interrupt it, 2to3 may not have > finished running. > > Of course, it should be more robust, but at the moment, it isn't > (patches welcome). > > Segfaults: test_multiarray.TestNewBufferProtocol.test_export_simple_1d ... FAIL test_multiarray.TestNewBufferProtocol.test_export_simple_nd ... ok test_multiarray.TestNewBufferProtocol.test_export_subarray ... FAIL test_multiarray.TestNewBufferProtocol.test_roundtrip ... Segmentation fault Are there changes you haven't pushed to github? I don't want to be making fixes that already exist. It would also be easier to work on this if the current state was in the main repository so that the rest of us could push changes. Chuck ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Python 3 porting
ma, 2010-02-15 kello 10:18 -0600, Bruce Southey kirjoitti: [clip] > Is there a correct way to get Python3.1 to find the relative path on Linux? > I can change the import statement to work but I do not think that is viable. You need to use relative imports. 2to3 should be able to take care of this. [clip] >File "/usr/local/lib/python3.1/site-packages/numpy/lib/__init__.py", > line 1, in > from info import __doc__ > ImportError: No module named info That statement should read from .info import __doc__ and indeed, it reads like that for me. Check how it is in build/py3k/numpy/lib/__init__.py Most likely you interrupted the build by Ctrl+C and 2to3 did not finish the conversion of the files to Python3 format. Try removing the build/ directory and trying again -- if you interrupt it, 2to3 may not have finished running. Of course, it should be more robust, but at the moment, it isn't (patches welcome). -- Pauli Virtanen ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Python 3 porting
On 02/15/2010 12:55 AM, Pauli Virtanen wrote: > ma, 2010-02-15 kello 00:38 +0100, Xavier Gnata kirjoitti: > >> Ok! >> git clone git://github.com/pv/numpy-work.git >> git checkout origin/py3k >> NPY_SEPARATE_BUILD=1 python3.1 setup.py build >> >> but now it fails during the build: >> >> In file included from numpy/core/src/multiarray/buffer.c:14, >> from numpy/core/src/multiarray/multiarraymodule_onefile.c:36: >> numpy/core/src/multiarray/buffer.h: At top level: >> numpy/core/src/multiarray/buffer.h:14: error: conflicting types for >> ‘_descriptor_from_pep3118_format’ >> numpy/core/src/multiarray/common.c:220: note: previous implicit >> declaration of ‘_descriptor_from_pep3118_format’ was here >> In file included from >> numpy/core/src/multiarray/multiarraymodule_onefile.c:36: >> numpy/core/src/multiarray/buffer.c: In function ‘_buffer_format_string’: >> numpy/core/src/multiarray/buffer.c:151: warning: unused variable ‘repr’ >> > Hmm, I probably tested only the separate compilation properly as it > seems the single-file build is failing. The environment variable is > actually NPY_SEPARATE_COMPILATION=1, not *_BUILD. > > Hi, Is there a correct way to get Python3.1 to find the relative path on Linux? I can change the import statement to work but I do not think that is viable. I tried appending the directory with sys.path but that did not work. Python 3.1.1 (r311:74480, Feb 15 2010, 09:08:21) [GCC 4.4.1 20090725 (Red Hat 4.4.1-2)] on linux2 >>> import numpy Traceback (most recent call last): File "", line 1, in File "/usr/local/lib/python3.1/site-packages/numpy/__init__.py", line 136, in from . import add_newdocs File "/usr/local/lib/python3.1/site-packages/numpy/add_newdocs.py", line 9, in from numpy.lib import add_newdoc File "/usr/local/lib/python3.1/site-packages/numpy/lib/__init__.py", line 1, in from info import __doc__ ImportError: No module named info Thanks Bruce ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
