Re: [Zope-dev] Linux x86_64 [Was: Zope Tests: 3 OK, 5 Failed]
On Sat, Aug 8, 2009 at 13:19, Hanno Schlichting wrote: > One thing to note here is that the above change is indeed only of of > probably many that need to be made to support 64-bit platforms > properly. The other thing to note is that the change now introduces a > hard requirement on Python 2.5 or later. > > Neither Martijn nor me could figure out a way to make the function > calls to Build have different format identifier arguments dependent on > the Python version. On Python 2.4 there is only "i" for integer and > 2.5 introduces "n" for Py_ssize_t. Since this is an argument to a > function showing up in various combinations of "nn", "nO" and others, > there seems to be no easy way to make this work. > > Someone with more knowledge about pre-processor tricks might come up > with a solution to this, otherwise we will have to choose between > dropping 64-bit support or dropping the unofficial Python 2.4 support. It turned out to be really simple as consecutive string literals are concatenated into one after macros are expanded. The following changeset should make Acquisition python 2.4 compatible again and also properly 64-bit capable: http://svn.zope.org/Acquisition/trunk/src/Acquisition/_Acquisition.c?rev=102577&view=rev To summarize: when passing Py_BuildValue a Py_ssize_t value to build a python int, you need to use the 'n' format string instead of 'i'. But because 'n' and Py_ssize_t are both defined as of python 2.5, for C code to work in python 2.4 you need to redefine these two back to 'i' and int. -- Martijn Pieters ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Linux x86_64 [Was: Zope Tests: 3 OK, 5 Failed]
On 08.08.09 14:19, Hanno Schlichting wrote: > On Fri, Aug 7, 2009 at 3:26 PM, Martijn Pieters wrote: > >> The following checkin fixed this particular problem: >> >> >> http://svn.zope.org/Acquisition/trunk/src/Acquisition/_Acquisition.c?rev=102564&view=rev >> >> The acquisition slice wrapper accepted Py_ssize_t arguments, but then >> passed these of as C ints (format string 'i'). Changing the format >> string to 'n' (Py_ssize_t) fixed *this particular case*. Likely more >> such fixes must be made. >> > One thing to note here is that the above change is indeed only of of > probably many that need to be made to support 64-bit platforms > properly. The other thing to note is that the change now introduces a > hard requirement on Python 2.5 or later. > > Neither Martijn nor me could figure out a way to make the function > calls to Build have different format identifier arguments dependent on > the Python version. On Python 2.4 there is only "i" for integer and > 2.5 introduces "n" for Py_ssize_t. Since this is an argument to a > function showing up in various combinations of "nn", "nO" and others, > there seems to be no easy way to make this work. > > Someone with more knowledge about pre-processor tricks might come up > with a solution to this, otherwise we will have to choose between > dropping 64-bit support or dropping the unofficial Python 2.4 support. > Dropping 64-bit support is no option, dropping Python 2.4 support is :) Adreas begin:vcard fn:Andreas Jung n:Jung;Andreas org:ZOPYX Ltd. & Co. KG adr;quoted-printable:;;Charlottenstr. 37/1;T=C3=BCbingen;;72070;Germany email;internet:i...@zopyx.com title:CEO tel;work:+49-7071-793376 tel;fax:+49-7071-7936840 tel;home:+49-7071-793257 x-mozilla-html:FALSE url:www.zopyx.com version:2.1 end:vcard ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Linux x86_64 [Was: Zope Tests: 3 OK, 5 Failed]
On Fri, Aug 7, 2009 at 3:26 PM, Martijn Pieters wrote: > The following checkin fixed this particular problem: > > http://svn.zope.org/Acquisition/trunk/src/Acquisition/_Acquisition.c?rev=102564&view=rev > > The acquisition slice wrapper accepted Py_ssize_t arguments, but then > passed these of as C ints (format string 'i'). Changing the format > string to 'n' (Py_ssize_t) fixed *this particular case*. Likely more > such fixes must be made. One thing to note here is that the above change is indeed only of of probably many that need to be made to support 64-bit platforms properly. The other thing to note is that the change now introduces a hard requirement on Python 2.5 or later. Neither Martijn nor me could figure out a way to make the function calls to Build have different format identifier arguments dependent on the Python version. On Python 2.4 there is only "i" for integer and 2.5 introduces "n" for Py_ssize_t. Since this is an argument to a function showing up in various combinations of "nn", "nO" and others, there seems to be no easy way to make this work. Someone with more knowledge about pre-processor tricks might come up with a solution to this, otherwise we will have to choose between dropping 64-bit support or dropping the unofficial Python 2.4 support. Hanno ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Linux x86_64 [Was: Zope Tests: 3 OK, 5 Failed]
On Fri, Aug 7, 2009 at 14:55, Martijn Pieters wrote: > The problem is caused by an acquisition wrapped object with a > __getslice__ method get the wrong indices passed in when the end > parameter is ommitted: > > data[start:] > > Image.Pdata classes have a __getslice__ method, and when data is > acquisition wrapped, that method gets (start, -1) passed in on 64-bit > platforms, while on my Macbook (start, sys.maxint) is passed in > instead. Obviously this means that the slice returns not everything > from 'start' to the end, but from 'start' until the one-but-last byte. > > However, as far as I can tell cAcquisition.c doesn't touch the start > and end values. Anyone want to investigate why this goes wrong? The following checkin fixed this particular problem: http://svn.zope.org/Acquisition/trunk/src/Acquisition/_Acquisition.c?rev=102564&view=rev The acquisition slice wrapper accepted Py_ssize_t arguments, but then passed these of as C ints (format string 'i'). Changing the format string to 'n' (Py_ssize_t) fixed *this particular case*. Likely more such fixes must be made. In any case, the HTTP range test no longer fails with this fix. -- Martijn Pieters ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Linux x86_64 [Was: Zope Tests: 3 OK, 5 Failed]
On Fri, Aug 7, 2009 at 14:55, Martijn Pieters wrote: > Here is the simple test case: > > from Acquisition import Implicit > > class Root(Implicit): > pass > > class Slicer(Implicit): > def __getslice__(self, start, end): > return [start, end] > > root = Root() > slicer = Slicer().__of__(root) > print slicer[1:] > > Run this as bin/zopepy acquisition.py and verify that [1, -1] is > printed on 64-bit platforms as opposed to [1, 2147483647] on 32-bit > platforms. Further datapoints, the following extra lines for the above script give more context to compare: import sys print slicer[sys.maxint:] print slicer[sys.maxint-1:] print Slicer()[1:] So we pass in sys.maxint and sys.maxint-1 as start values, and also run the Slicer without acquisition wrapping to demonstrate that the wrapper is the culprit, not python itself. Python passes in sys.maxint for the end value for the slice if you omit it, and these high values overflow somewhere in the wraper. -- Martijn Pieters ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Linux x86_64 [Was: Zope Tests: 3 OK, 5 Failed]
On Mon, Aug 3, 2009 at 08:04, Martijn Pieters wrote: >> AssertionError: Incorrect Content-Length is set! Expected 20425, >> got 20426. >> >> I don't grok the range support code at all: probably Martijn Pieters is >> the only person in the world who does. The tests are quite obscure to >> me, so I can't diagnose even whether the failure is a testing artifact >> or a real bug. > > Here's my response to Tres; for some reason my mail reader didn't > include zope-dev. I can add now that I'll be at a sprint this weekend > where I'll probably have time to take a look myself: I have found the problem, but do not understand where exactly the error occurs. The problem is caused by an acquisition wrapped object with a __getslice__ method get the wrong indices passed in when the end parameter is ommitted: data[start:] Image.Pdata classes have a __getslice__ method, and when data is acquisition wrapped, that method gets (start, -1) passed in on 64-bit platforms, while on my Macbook (start, sys.maxint) is passed in instead. Obviously this means that the slice returns not everything from 'start' to the end, but from 'start' until the one-but-last byte. However, as far as I can tell cAcquisition.c doesn't touch the start and end values. Anyone want to investigate why this goes wrong? Here is the simple test case: from Acquisition import Implicit class Root(Implicit): pass class Slicer(Implicit): def __getslice__(self, start, end): return [start, end] root = Root() slicer = Slicer().__of__(root) print slicer[1:] Run this as bin/zopepy acquisition.py and verify that [1, -1] is printed on 64-bit platforms as opposed to [1, 2147483647] on 32-bit platforms. -- Martijn Pieters ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Linux x86_64 [Was: Zope Tests: 3 OK, 5 Failed]
On 03.08.09 09:04, Martijn Pieters wrote:
> On Tue, Jul 21, 2009 at 18:40, Tres Seaver wrote:
>
>> File "/home/tseaver/projects/Zope-trunk/src/OFS/tests/testRanges.py",
>> line 332, in testMultipleRangesBigFileOutOfOrder
>> (7, 80001)])
>> File "/home/tseaver/projects/Zope-trunk/src/OFS/tests/testRanges.py",
>> line 209, in expectMultipleRanges
>> str(len(body)), rsp.getHeader('content-length')))
>> File "/opt/Python-2.6.2/lib/python2.6/unittest.py", line 321, in
>> failIf
>> if expr: raise self.failureException, msg
>> AssertionError: Incorrect Content-Length is set! Expected 20425,
>> got 20426.
>>
>> I don't grok the range support code at all: probably Martijn Pieters is
>> the only person in the world who does. The tests are quite obscure to
>> me, so I can't diagnose even whether the failure is a testing artifact
>> or a real bug.
>>
> Here's my response to Tres; for some reason my mail reader didn't
> include zope-dev. I can add now that I'll be at a sprint this weekend
> where I'll probably have time to take a look myself:
Oki.
Tnx,
Andreas
begin:vcard
fn:Andreas Jung
n:Jung;Andreas
org:ZOPYX Ltd. & Co. KG
adr;quoted-printable:;;Charlottenstr. 37/1;T=C3=BCbingen;;72070;Germany
email;internet:i...@zopyx.com
title:CEO
tel;work:+49-7071-793376
tel;fax:+49-7071-7936840
tel;home:+49-7071-793257
x-mozilla-html:FALSE
url:www.zopyx.com
version:2.1
end:vcard
___
Zope-Dev maillist - Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Linux x86_64 [Was: Zope Tests: 3 OK, 5 Failed]
On Tue, Jul 21, 2009 at 18:40, Tres Seaver wrote:
> File "/home/tseaver/projects/Zope-trunk/src/OFS/tests/testRanges.py",
> line 332, in testMultipleRangesBigFileOutOfOrder
> (7, 80001)])
> File "/home/tseaver/projects/Zope-trunk/src/OFS/tests/testRanges.py",
> line 209, in expectMultipleRanges
> str(len(body)), rsp.getHeader('content-length')))
> File "/opt/Python-2.6.2/lib/python2.6/unittest.py", line 321, in
> failIf
> if expr: raise self.failureException, msg
> AssertionError: Incorrect Content-Length is set! Expected 20425,
> got 20426.
>
> I don't grok the range support code at all: probably Martijn Pieters is
> the only person in the world who does. The tests are quite obscure to
> me, so I can't diagnose even whether the failure is a testing artifact
> or a real bug.
Here's my response to Tres; for some reason my mail reader didn't
include zope-dev. I can add now that I'll be at a sprint this weekend
where I'll probably have time to take a look myself:
8<--
Sorry you find them obscure; I followed unit test expectations as best
as I could.
The test that fails does the following:
1) Create a big file (uploadBigFile, enough random data to be more
than one pdata chunk)
2) Request for multiple HTTP ranges (slices) of the file to be sent.
These slices should be returned a) in the same order, b) of the
correct length, and of course c) the correct slice of the big file
created in 1.
In this case line 330 specifies that bytes 10-15 (inclusive), the last
1 bytes, and bytes 7-8 (inclusive) are are to be returned.
The call to expectMultipleRanges includes the expected slices (python
syntax, so end value is exclusive).
3) On line 209 of expectMultipleRanges it then tests if the resulting
response has the correct content length header for the whole response
set. The failure reported is that the Content-Length header reports 1
byte more than what was actually sent, so the body was 20425 bytes
long, while the content-header claims it would be 20426 bytes.
Because this is a multi-part range request, Image.py line 277 and
onwards handle this request. The response ends up looking something
like this:
Content-length: size we pre-calculate
Content-type: multipart/byteranges; boundary="boundarystring"
More headers
--boundarystring
Content-type: image/whatever
Content-range: bytes start-end/total-file-size
data ...
--boundarystring
... Another section
... etc.
--boundarystring--
There are 2 ways the failure could be caused.
Something could go wrong with the size calculation on lines 280-287;
the total body length is the length of the range data plus the length
of the headers per multi-part plus the length of the MIME boundaries
between them. There is thus a length per part plus the length of the
last boundary.
Or something goes wrong when writing the range data in lines 309-358.
Because there is only one byte difference between what was calculated
on lines 280-287 and what is produced here this is probably a newline
somewhere.
I do find it puzzling this only fails on 64 bit platforms. It could be
that the pdata chunk handling is borken and has a off-by-one error
when running on 64-bit platforms.
Hope this clarifies things a little. I have little time right now to
chase this myself right now, sorry!
8<--
--
Martijn Pieters
___
Zope-Dev maillist - Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Linux x86_64 [Was: Zope Tests: 3 OK, 5 Failed]
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hanno Schlichting wrote: > On Sun, Aug 2, 2009 at 8:30 PM, Tres Seaver wrote: >> I think I have fixed the ExtensionClass and Acquisition problems for >> 64-bit machines: I released new eggs to PyPI for them: > > Awesome! > > You wouldn't be interested to look at the C code in Persistence and > Zope2 itself? ;-) I have updated the cAccessControl and Record extensions for conformance with the PEP. I don't think the Persistence code needs anything: AFAICT, it is already good to go. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFKdgWp+gerLs4ltQ4RApPJAJ4neMSFrdUaLomc5fnImtkuDQVkwgCgsZNL 1iBevGRfBSM5m1OFJILnqp0= =oXK6 -END PGP SIGNATURE- ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Linux x86_64 [Was: Zope Tests: 3 OK, 5 Failed]
On 02.08.09 20:30, Tres Seaver wrote: > Andreas Jung wrote: > > On 25.07.09 13:57, Hanno Schlichting wrote: > >> At this point I think we need to declare 64-bit platforms as > >> unsupported. Hopefully people with the right kind of knowledge would > >> like to step up and start working on these issues. > > This is really not an option. Zope 2.12 must run on 32 and 64 bit > > systems without issues. Since this is a major blocker I will defer > > the next release (scheduled for this week) until we have fixed the > issue. > > I think I have fixed the ExtensionClass and Acquisition problems for > 64-bit machines: I released new eggs to PyPI for them: > > - http://pypi.python.org/pypi/ExtensionClass/2.11.3 > > - http://pypi.python.org/pypi/Acquisition/2.12.2 > Thanks Tres - the issue seems to be resolved. Right now there is only one failure with the http ranges tests (Martijn, can you check this?) and a complete failure of the zope.index tests (see other mail). Andreas begin:vcard fn:Andreas Jung n:Jung;Andreas org:ZOPYX Ltd. & Co. KG adr;quoted-printable:;;Charlottenstr. 37/1;T=C3=BCbingen;;72070;Germany email;internet:i...@zopyx.com title:CEO tel;work:+49-7071-793376 tel;fax:+49-7071-7936840 tel;home:+49-7071-793257 x-mozilla-html:FALSE url:www.zopyx.com version:2.1 end:vcard ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Linux x86_64 [Was: Zope Tests: 3 OK, 5 Failed]
On Sun, Aug 2, 2009 at 8:30 PM, Tres Seaver wrote: > I think I have fixed the ExtensionClass and Acquisition problems for > 64-bit machines: I released new eggs to PyPI for them: Awesome! You wouldn't be interested to look at the C code in Persistence and Zope2 itself? ;-) Hanno ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Linux x86_64 [Was: Zope Tests: 3 OK, 5 Failed]
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Andreas Jung wrote: > On 25.07.09 13:57, Hanno Schlichting wrote: >> At this point I think we need to declare 64-bit platforms as >> unsupported. Hopefully people with the right kind of knowledge would >> like to step up and start working on these issues. > This is really not an option. Zope 2.12 must run on 32 and 64 bit > systems without issues. Since this is a major blocker I will defer > the next release (scheduled for this week) until we have fixed the issue. I think I have fixed the ExtensionClass and Acquisition problems for 64-bit machines: I released new eggs to PyPI for them: - - http://pypi.python.org/pypi/ExtensionClass/2.11.3 - - http://pypi.python.org/pypi/Acquisition/2.12.2 Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFKddsu+gerLs4ltQ4RArPiAJ0USyr/xHsBthWCdG4GAfpan1tirQCgh711 eu6hqMsep/VtglUXSshIgoY= =6amd -END PGP SIGNATURE- ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Linux x86_64 [Was: Zope Tests: 3 OK, 5 Failed]
On 25.07.09 13:57, Hanno Schlichting wrote: > > At this point I think we need to declare 64-bit platforms as > unsupported. Hopefully people with the right kind of knowledge would > like to step up and start working on these issues. This is really not an option. Zope 2.12 must run on 32 and 64 bit systems without issues. Since this is a major blocker I will defer the next release (scheduled for this week) until we have fixed the issue. Andreas begin:vcard fn:Andreas Jung n:Jung;Andreas org:ZOPYX Ltd. & Co. KG adr;quoted-printable:;;Charlottenstr. 37/1;T=C3=BCbingen;;72070;Germany email;internet:i...@zopyx.com title:CEO tel;work:+49-7071-793376 tel;fax:+49-7071-7936840 tel;home:+49-7071-793257 x-mozilla-html:FALSE url:www.zopyx.com version:2.1 end:vcard ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Linux x86_64 [Was: Zope Tests: 3 OK, 5 Failed]
On Sun, Jul 26, 2009 at 02:55:58AM +0200, Hanno Schlichting wrote:
> On Sat, Jul 25, 2009 at 8:01 PM, Shane Hathaway wrote:
> > Hanno Schlichting wrote:
> >>
> >> I kind of suspect that we are seeing the results of
> >> http://www.python.org/dev/peps/pep-0353 though.
> >
> > That is very likely. BTW, that PEP links to a handy tool that reveals most
> > 64 bit portability issues in Python-oriented C code.
>
> Right, I've given that a shot and it reports plenty of problems for
> Acquisition.c for example. In addition to the general compiler
> warnings it spits out, which don't tell me much.
>
> >> From what I understand we need to change code in C extension modules
> >> following the instructions in that PEP, to be able to run code
> >> reliably on 64-bit platforms. To my knowledge nobody changed any C
> >> code in Zope yet to do so.
> >
> > We fixed the C portability problems in Zope 3 a while back, but it seems we
> > didn't fix the extensions that are specific to Zope 2.
>
> How thorough has that been?
I'm not familiar with the source code of the extension modules, and I
haven't reviewed them for possible 64-bit issues. But we've been
running a Zope 3 app in production on 64-bit Linux for almost two years
now. No 64-bit-related problems bit us so far.
There's a buildbot for Zope 3.4 packages that runs their tests on 32 and
64 bit platforms on Python 2.4 and 2.5: http://zope3.pov.lt/buildbot/
Currently ZODB tests are disabled on it because they used to have
irritating spurious failures that seem to be related to the speed of the
machine rather than its platform.
I believe there's a similar buildbot for Zope 3.5 KGS out there
somewhere.
> I can see the Py_ssize_t in some of the c
> files in zope.* packages, but looking for example at:
>
> http://svn.zope.org/zope.proxy/trunk/src/zope/proxy/_zope_proxy_proxy.c?rev=95468&view=markup
>
> I see a stuff like this:
>
> PyObject *
> WrapperType_Lookup(PyTypeObject *type, PyObject *name)
> {
> int i, n;
>
> [...]
>
> n = PyTuple_GET_SIZE(mro) - 1;
>
> [...]
> }
>
> where PyTuple_GET_SIZE is defined to return a Py_ssize_t and not an int.
It shouldn't be a problem until you start seeing tuples with 2 billion
items in them, no?
> Either I misunderstand something here or I was just "lucky" to find
> something like this as my first test.
Marius Gedminas
--
http://pov.lt/ -- Zope 3 consulting and development
signature.asc
Description: Digital signature
___
Zope-Dev maillist - Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Linux x86_64 [Was: Zope Tests: 3 OK, 5 Failed]
On Sat, Jul 25, 2009 at 8:01 PM, Shane Hathaway wrote:
> Hanno Schlichting wrote:
>>
>> I kind of suspect that we are seeing the results of
>> http://www.python.org/dev/peps/pep-0353 though.
>
> That is very likely. BTW, that PEP links to a handy tool that reveals most
> 64 bit portability issues in Python-oriented C code.
Right, I've given that a shot and it reports plenty of problems for
Acquisition.c for example. In addition to the general compiler
warnings it spits out, which don't tell me much.
>> From what I understand we need to change code in C extension modules
>> following the instructions in that PEP, to be able to run code
>> reliably on 64-bit platforms. To my knowledge nobody changed any C
>> code in Zope yet to do so.
>
> We fixed the C portability problems in Zope 3 a while back, but it seems we
> didn't fix the extensions that are specific to Zope 2.
How thorough has that been? I can see the Py_ssize_t in some of the c
files in zope.* packages, but looking for example at:
http://svn.zope.org/zope.proxy/trunk/src/zope/proxy/_zope_proxy_proxy.c?rev=95468&view=markup
I see a stuff like this:
PyObject *
WrapperType_Lookup(PyTypeObject *type, PyObject *name)
{
int i, n;
[...]
n = PyTuple_GET_SIZE(mro) - 1;
[...]
}
where PyTuple_GET_SIZE is defined to return a Py_ssize_t and not an int.
Either I misunderstand something here or I was just "lucky" to find
something like this as my first test.
>> The other interesting bit to this is, that it seems doable but hard to
>> do these changes while preserving compatibility with Python 2.4. These
>> changes along with the definition of "Py_ssize_t" were introduced into
>> Python 2.5.
>
> Preprocessor macros solve that fairly cleanly, actually:
>
> #ifndef Py_ssize_t
> #define Py_ssize_t int
> #endif
Ah ok. I was more concerned about the function pointer changes.
intargfunc to ssizeargfunc and so on. I wasn't sure if these are as
easy to redefine as simple data types.
>> At this point I think we need to declare 64-bit platforms as
>> unsupported. Hopefully people with the right kind of knowledge would
>> like to step up and start working on these issues.
>
> I just checked in some code that solves the portability problem in
> zope.index, although I haven't created a release. It looks like the
> Acquisition module needs some 64 bit TLC.
Cool! I kinda suspect that specific code is also somewhere in Zope2 in
the PluginIndexes where it has been ported from recently.
Hanno
___
Zope-Dev maillist - Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Linux x86_64 [Was: Zope Tests: 3 OK, 5 Failed]
Hanno Schlichting wrote: > I kind of suspect that we are seeing the results of > http://www.python.org/dev/peps/pep-0353 though. That is very likely. BTW, that PEP links to a handy tool that reveals most 64 bit portability issues in Python-oriented C code. >>From what I understand we need to change code in C extension modules > following the instructions in that PEP, to be able to run code > reliably on 64-bit platforms. To my knowledge nobody changed any C > code in Zope yet to do so. We fixed the C portability problems in Zope 3 a while back, but it seems we didn't fix the extensions that are specific to Zope 2. > The other interesting bit to this is, that it seems doable but hard to > do these changes while preserving compatibility with Python 2.4. These > changes along with the definition of "Py_ssize_t" were introduced into > Python 2.5. Preprocessor macros solve that fairly cleanly, actually: #ifndef Py_ssize_t #define Py_ssize_t int #endif > At this point I think we need to declare 64-bit platforms as > unsupported. Hopefully people with the right kind of knowledge would > like to step up and start working on these issues. I just checked in some code that solves the portability problem in zope.index, although I haven't created a release. It looks like the Acquisition module needs some 64 bit TLC. Shane ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Linux x86_64 [Was: Zope Tests: 3 OK, 5 Failed]
On Sat, Jul 18, 2009 at 2:08 PM, Stefan H. Holek wrote: > All failures appear to be due to the new box running Linux x86_64. > Tests still pass fine on the Mac. > > Any experience with this? MemoryError in Acquisition? WTH? I don't have any experience with either 64-bit nor can I write any C. And it seems obvious that these kind of crashes are caused by C extension code. I kind of suspect that we are seeing the results of http://www.python.org/dev/peps/pep-0353 though. >From what I understand we need to change code in C extension modules following the instructions in that PEP, to be able to run code reliably on 64-bit platforms. To my knowledge nobody changed any C code in Zope yet to do so. The other interesting bit to this is, that it seems doable but hard to do these changes while preserving compatibility with Python 2.4. These changes along with the definition of "Py_ssize_t" were introduced into Python 2.5. At this point I think we need to declare 64-bit platforms as unsupported. Hopefully people with the right kind of knowledge would like to step up and start working on these issues. Hanno ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Linux x86_64 [Was: Zope Tests: 3 OK, 5 Failed]
I can reproduce the failures in ExtensionClass and also get some weird errors while trying to buildout the Acquisition package: aj...@blackmoon:/develop/repository/svn.zope.org/Acquisition/trunk> bin/buildout Develop: '/data/develop/repository/svn.zope.org/Acquisition/trunk/.' Unused options for buildout: 'download-directory'. Installing interpreter. While: Installing interpreter. An internal error occured due to a bug in either zc.buildout or in a recipe being used: Traceback (most recent call last): File "/tmp/tmpt2EP2J/zc.buildout-1.3.0-py2.6.egg/zc/buildout/buildout.py", line 1519, in main File "/tmp/tmpt2EP2J/zc.buildout-1.3.0-py2.6.egg/zc/buildout/buildout.py", line 473, in install File "/tmp/tmpt2EP2J/zc.buildout-1.3.0-py2.6.egg/zc/buildout/buildout.py", line 1101, in _call File "/home/ajung/.buildout/eggs/zc.recipe.egg-1.2.2-py2.6.egg/zc/recipe/egg/egg.py", line 169, in install relative_paths=self._relative_paths, File "/tmp/tmpt2EP2J/zc.buildout-1.3.0-py2.6.egg/zc/buildout/easy_install.py", line 964, in scripts File "/tmp/tmpt2EP2J/zc.buildout-1.3.0-py2.6.egg/zc/buildout/easy_install.py", line , in _pyscript IOError: [Errno 26] Text file busy: '/data/develop/repository/svn.zope.org/Acquisition/trunk/bin/python' Both issues are severe and are a blocker for the next release. Andreas On 18.07.09 14:08, Stefan H. Holek wrote: > All failures appear to be due to the new box running Linux x86_64. > Tests still pass fine on the Mac. > > Any experience with this? MemoryError in Acquisition? WTH? > > Stefan > > > On 18.07.2009, at 14:00, Zope Tests Summarizer wrote: > > >> Summary of messages to the zope-tests list. >> Period Fri Jul 17 12:00:00 2009 UTC to Sat Jul 18 12:00:00 2009 UTC. >> There were 8 messages: 8 from Zope Tests. >> >> >> Test failures >> - >> >> Subject: FAILED (failures=1) : Zope-2.12 Python-2.6.2 : Linux >> From: Zope Tests >> Date: Fri Jul 17 20:50:00 EDT 2009 >> URL: http://mail.zope.org/pipermail/zope-tests/2009-July/012082.html >> >> Subject: FAILED (failures=1) : Zope-2.12-alltests Python-2.4.6 : Linux >> From: Zope Tests >> Date: Fri Jul 17 20:52:00 EDT 2009 >> URL: http://mail.zope.org/pipermail/zope-tests/2009-July/012083.html >> >> Subject: FAILED (failures=7) : Zope-2.12-alltests Python-2.6.2 : Linux >> From: Zope Tests >> Date: Fri Jul 17 20:54:01 EDT 2009 >> URL: http://mail.zope.org/pipermail/zope-tests/2009-July/012084.html >> >> Subject: FAILED (failures=1) : Zope-trunk Python-2.6.2 : Linux >> From: Zope Tests >> Date: Fri Jul 17 20:56:01 EDT 2009 >> URL: http://mail.zope.org/pipermail/zope-tests/2009-July/012085.html >> >> Subject: FAILED (failures=7) : Zope-trunk-alltests Python-2.6.2 : >> Linux >> From: Zope Tests >> Date: Fri Jul 17 20:58:01 EDT 2009 >> URL: http://mail.zope.org/pipermail/zope-tests/2009-July/012086.html >> > > -- > Stefan H. Holek > ste...@epy.co.at > > > > > ___ > Zope-Dev maillist - Zope-Dev@zope.org > http://mail.zope.org/mailman/listinfo/zope-dev > ** No cross posts or HTML encoding! ** > (Related lists - > http://mail.zope.org/mailman/listinfo/zope-announce > http://mail.zope.org/mailman/listinfo/zope ) > -- ZOPYX Ltd. & Co KG \ ZOPYX & Friends Charlottenstr. 37/1 \ The experts for your Python, Zope and D-72070 Tübingen \ Plone projects www.zopyx.com, i...@zopyx.com \ www.zopyx.de/friends, frie...@zopyx.de E-Publishing, Python, Zope & Plone development, Consulting begin:vcard fn:Andreas Jung n:Jung;Andreas org:ZOPYX Ltd. & Co. KG adr;quoted-printable:;;Charlottenstr. 37/1;T=C3=BCbingen;;72070;Germany email;internet:i...@zopyx.com title:CEO tel;work:+49-7071-793376 tel;fax:+49-7071-7936840 tel;home:+49-7071-793257 x-mozilla-html:FALSE url:www.zopyx.com version:2.1 end:vcard ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Linux x86_64 [Was: Zope Tests: 3 OK, 5 Failed]
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
Stefan H. Holek wrote:
> All failures appear to be due to the new box running Linux x86_64.
> Tests still pass fine on the Mac.
>
> Any experience with this? MemoryError in Acquisition? WTH?
I can reproduce the OFS.tests.testRange failures on my x86_64 box:
$ uname -a
Linux mred 2.6.27-14-generic #1 SMP Wed Apr 15 19:29:46 UTC \
2009 x86_64 GNU/Linux
$ cd /tmp
$ svn co svn+ssh://svn.zope.org/repos/main/Zope/trunk Zope-trunk
$ cd Zope-trunk
$ /opt/Python-2.6.2/bin/python bootstrap.py
...
$ bin/buildout
...
$ bin/test --all
...
Failure in test testMultipleRangesBigFileOutOfOrder
(OFS.tests.testRanges.TestRequestRange)
Traceback (most recent call last):
File "/opt/Python-2.6.2/lib/python2.6/unittest.py", line 279, in run
testMethod()
File "/home/tseaver/projects/Zope-trunk/src/OFS/tests/testRanges.py",
line 332, in testMultipleRangesBigFileOutOfOrder
(7, 80001)])
File "/home/tseaver/projects/Zope-trunk/src/OFS/tests/testRanges.py",
line 209, in expectMultipleRanges
str(len(body)), rsp.getHeader('content-length')))
File "/opt/Python-2.6.2/lib/python2.6/unittest.py", line 321, in
failIf
if expr: raise self.failureException, msg
AssertionError: Incorrect Content-Length is set! Expected 20425,
got 20426.
I don't grok the range support code at all: probably Martijn Pieters is
the only person in the world who does. The tests are quite obscure to
me, so I can't diagnose even whether the failure is a testing artifact
or a real bug.
I see another failure in a test Hanno recently added:
Failure in test test_datetime
(Products.PluginIndexes.DateRangeIndex.tests.test_DateRangeIndex.DRI_Tests)
Traceback (most recent call last):
File "/opt/Python-2.6.2/lib/python2.6/unittest.py", line 279, in run
testMethod()
File
"/home/tseaver/projects/Zope-trunk/src/Products/PluginIndexes/DateRangeIndex/tests/test_DateRangeIndex.py",
line 155, in test_datetime
self.assertEqual(work.getEntryForObject(0), (20790915, 20790990))
File "/opt/Python-2.6.2/lib/python2.6/unittest.py", line 350, in
failUnlessEqual
(msg or '%r != %r' % (first, second))
AssertionError: (20791275, 20791350) != (20790915, 20790990)
This one looks to be the result of using "naive" datetime objects while
expecting them to be converted to hard-wired integers, so that tests
which pass on Hanno's machine fail on mine, because of different timezones.
Tres.
- --
===
Tres Seaver +1 540-429-0999 tsea...@palladion.com
Palladion Software "Excellence by Design"http://palladion.com
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iD8DBQFKZe9j+gerLs4ltQ4RAl4nAKDJQyWHdnlEEQv1zbQJeJiR7gGuTACfQxDi
n5GO/tP+jpRhqUddn2CbaKg=
=RdUR
-END PGP SIGNATURE-
___
Zope-Dev maillist - Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Linux x86_64 [Was: Zope Tests: 3 OK, 5 Failed]
All failures appear to be due to the new box running Linux x86_64. Tests still pass fine on the Mac. Any experience with this? MemoryError in Acquisition? WTH? Stefan On 18.07.2009, at 14:00, Zope Tests Summarizer wrote: > Summary of messages to the zope-tests list. > Period Fri Jul 17 12:00:00 2009 UTC to Sat Jul 18 12:00:00 2009 UTC. > There were 8 messages: 8 from Zope Tests. > > > Test failures > - > > Subject: FAILED (failures=1) : Zope-2.12 Python-2.6.2 : Linux > From: Zope Tests > Date: Fri Jul 17 20:50:00 EDT 2009 > URL: http://mail.zope.org/pipermail/zope-tests/2009-July/012082.html > > Subject: FAILED (failures=1) : Zope-2.12-alltests Python-2.4.6 : Linux > From: Zope Tests > Date: Fri Jul 17 20:52:00 EDT 2009 > URL: http://mail.zope.org/pipermail/zope-tests/2009-July/012083.html > > Subject: FAILED (failures=7) : Zope-2.12-alltests Python-2.6.2 : Linux > From: Zope Tests > Date: Fri Jul 17 20:54:01 EDT 2009 > URL: http://mail.zope.org/pipermail/zope-tests/2009-July/012084.html > > Subject: FAILED (failures=1) : Zope-trunk Python-2.6.2 : Linux > From: Zope Tests > Date: Fri Jul 17 20:56:01 EDT 2009 > URL: http://mail.zope.org/pipermail/zope-tests/2009-July/012085.html > > Subject: FAILED (failures=7) : Zope-trunk-alltests Python-2.6.2 : > Linux > From: Zope Tests > Date: Fri Jul 17 20:58:01 EDT 2009 > URL: http://mail.zope.org/pipermail/zope-tests/2009-July/012086.html -- Stefan H. Holek ste...@epy.co.at ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
