Re: [Zope-dev] zope.index 3.5.2 broken
Hey, Andreas Jung wrote: [snip] The diff between 3.5.1 and 3.5.2 is pretty long and substantial. I doubt that such a major change us ok as a bugfix release. I should have become a new major release. From what you say, agreed. Regards, Martijn ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] zope.index 3.5.2 broken
On Sun, Aug 02, 2009 at 08:48:24PM +0200, Andreas Jung wrote: Hi, the doctests for zope.index 3.5.2 - as used in Zope 2.12 - fail badly: File /home/ajung/.buildout/eggs/zope.index-3.5.2-py2.6-linux-x86_64.egg/zope/index/text/tests/../textindex.txt, line 143, in textindex.txt Failed example: [(k, %.4f % v) for (k, v) in index2.apply(Zorro).items()] == result I'm assuming items() returns a plain Python dictionary with string keys. Python's string hashes return different valuesfor half of all the strings on 64-bit machines. This influences the ordering of dictionary keys and some other things too (such as the sequence of random numbers you get if you use a string as the seed). Add a sorted() on both sides and the test should pass. Marius Gedminas -- http://pov.lt/ -- Zope 3 consulting and development signature.asc Description: Digital signature ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] zope.index 3.5.2 broken
Marius Gedminas wrote: On Sun, Aug 02, 2009 at 08:48:24PM +0200, Andreas Jung wrote: Hi, the doctests for zope.index 3.5.2 - as used in Zope 2.12 - fail badly: File /home/ajung/.buildout/eggs/zope.index-3.5.2-py2.6-linux-x86_64.egg/zope/index/text/tests/../textindex.txt, line 143, in textindex.txt Failed example: [(k, %.4f % v) for (k, v) in index2.apply(Zorro).items()] == result I'm assuming items() returns a plain Python dictionary with string keys. Python's string hashes return different valuesfor half of all the strings on 64-bit machines. This influences the ordering of dictionary keys and some other things too (such as the sequence of random numbers you get if you use a string as the seed). Add a sorted() on both sides and the test should pass. Actually, those tests were plain insane and I've fixed them on the trunk. I intend to make a new zope.index release today. How insane were these tests? Well, the author of the tests noticed that the C optimization produces different scores than the Python version, and compensated for that in a way that dramatically reduced readability. Furthermore, the C code was producing incorrect results due to an unsafe optimization that accidentally took 32 bits of a Python float and pretended it was an integer. The breakage only became obvious when we started taking 64 bits instead, leading to completely different results. Shane ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] zope.index 3.5.2 broken
On 8/3/09 1:07 PM, Shane Hathaway wrote: Marius Gedminas wrote: On Sun, Aug 02, 2009 at 08:48:24PM +0200, Andreas Jung wrote: Hi, the doctests for zope.index 3.5.2 - as used in Zope 2.12 - fail badly: File /home/ajung/.buildout/eggs/zope.index-3.5.2-py2.6-linux-x86_64.egg/zope/index/text/tests/../textindex.txt, line 143, in textindex.txt Failed example: [(k, %.4f % v) for (k, v) in index2.apply(Zorro).items()] == result I'm assuming items() returns a plain Python dictionary with string keys. Python's string hashes return different valuesfor half of all the strings on 64-bit machines. This influences the ordering of dictionary keys and some other things too (such as the sequence of random numbers you get if you use a string as the seed). Add a sorted() on both sides and the test should pass. Actually, those tests were plain insane and I've fixed them on the trunk. I intend to make a new zope.index release today. How insane were these tests? Well, the author of the tests noticed that the C optimization produces different scores than the Python version, and compensated for that in a way that dramatically reduced readability. /me hangs head in shame. - C ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] zope.index 3.5.2 broken
On 03.08.09 20:15, Chris McDonough wrote: On 8/3/09 1:07 PM, Shane Hathaway wrote: Marius Gedminas wrote: On Sun, Aug 02, 2009 at 08:48:24PM +0200, Andreas Jung wrote: Hi, the doctests for zope.index 3.5.2 - as used in Zope 2.12 - fail badly: File /home/ajung/.buildout/eggs/zope.index-3.5.2-py2.6-linux-x86_64.egg/zope/index/text/tests/../textindex.txt, line 143, in textindex.txt Failed example: [(k, %.4f % v) for (k, v) in index2.apply(Zorro).items()] == result I'm assuming items() returns a plain Python dictionary with string keys. Python's string hashes return different valuesfor half of all the strings on 64-bit machines. This influences the ordering of dictionary keys and some other things too (such as the sequence of random numbers you get if you use a string as the seed). Add a sorted() on both sides and the test should pass. Actually, those tests were plain insane and I've fixed them on the trunk. I intend to make a new zope.index release today. How insane were these tests? Well, the author of the tests noticed that the C optimization produces different scores than the Python version, and compensated for that in a way that dramatically reduced readability. Is there a buildbot for the zope.* or ZTK packages testing them under Linux 32bit and 64 bit? Andreas begin:vcard fn:Andreas Jung n:Jung;Andreas org:ZOPYX Ltd. Co. KG adr;quoted-printable:;;Charlottenstr. 37/1;T=C3=BCbingen;;72070;Germany email;internet:i...@zopyx.com title:CEO tel;work:+49-7071-793376 tel;fax:+49-7071-7936840 tel;home:+49-7071-793257 x-mozilla-html:FALSE url:www.zopyx.com version:2.1 end:vcard ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] zope.index 3.5.2 broken
Chris McDonough wrote: On 8/3/09 1:07 PM, Shane Hathaway wrote: How insane were these tests? Well, the author of the tests noticed that the C optimization produces different scores than the Python version, and compensated for that in a way that dramatically reduced readability. /me hangs head in shame. That was you? Well, it could have been worse. :-) Shane ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] zope.index 3.5.2 broken
On 8/3/09 2:34 PM, Shane Hathaway wrote: Chris McDonough wrote: On 8/3/09 1:07 PM, Shane Hathaway wrote: How insane were these tests? Well, the author of the tests noticed that the C optimization produces different scores than the Python version, and compensated for that in a way that dramatically reduced readability. /me hangs head in shame. That was you? Well, it could have been worse. :-) I didn't write the original tests but I did paper over the okascore result differences by making the tests compare true or false vs. the expected output. Ten lashes. - C ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] zope.index 3.5.2 broken
* Andreas Jung li...@zopyx.com [2009-08-03 20:21]: On 03.08.09 20:15, Chris McDonough wrote: On 8/3/09 1:07 PM, Shane Hathaway wrote: Marius Gedminas wrote: On Sun, Aug 02, 2009 at 08:48:24PM +0200, Andreas Jung wrote: the doctests for zope.index 3.5.2 - as used in Zope 2.12 - fail badly: Python's string hashes return different valuesfor half of all the strings on 64-bit machines. This influences the ordering of dictionary keys and some other things too (such as the sequence of random numbers you get if you use a string as the seed). Is there a buildbot for the zope.* or ZTK packages testing them under Linux 32bit and 64 bit? AFAICS all three buildbots listed on http://docs.zope.org/zopetoolkit/process/tools.html do. Wolfgang ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] zope.index 3.5.2 broken
Hi, the doctests for zope.index 3.5.2 - as used in Zope 2.12 - fail badly: File /home/ajung/.buildout/eggs/zope.index-3.5.2-py2.6-linux-x86_64.egg/zope/index/text/tests/../textindex.txt, line 143, in textindex.txt Failed example: [(k, %.4f % v) for (k, v) in index2.apply(Zorro).items()] == result Expected: True Got: False Thoughts? Andreas begin:vcard fn:Andreas Jung n:Jung;Andreas org:ZOPYX Ltd. Co. KG adr;quoted-printable:;;Charlottenstr. 37/1;T=C3=BCbingen;;72070;Germany email;internet:i...@zopyx.com title:CEO tel;work:+49-7071-793376 tel;fax:+49-7071-7936840 tel;home:+49-7071-793257 x-mozilla-html:FALSE url:www.zopyx.com version:2.1 end:vcard ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] zope.index 3.5.2 broken
On 02.08.09 20:48, Andreas Jung wrote: Hi, the doctests for zope.index 3.5.2 - as used in Zope 2.12 - fail badly: File /home/ajung/.buildout/eggs/zope.index-3.5.2-py2.6-linux-x86_64.egg/zope/index/text/tests/../textindex.txt, line 143, in textindex.txt Failed example: [(k, %.4f % v) for (k, v) in index2.apply(Zorro).items()] == result Expected: True Got: False Obviously another Python 2.5/.6 incompatiblity. zope.index 3.5.1 works fine with Python 2.4-2.6 - obviously changes to okascore.c introduced with zope.index 3.5.2 are to blame?! Andreas begin:vcard fn:Andreas Jung n:Jung;Andreas org:ZOPYX Ltd. Co. KG adr;quoted-printable:;;Charlottenstr. 37/1;T=C3=BCbingen;;72070;Germany email;internet:i...@zopyx.com title:CEO tel;work:+49-7071-793376 tel;fax:+49-7071-7936840 tel;home:+49-7071-793257 x-mozilla-html:FALSE url:www.zopyx.com version:2.1 end:vcard ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] zope.index 3.5.2 broken
On 02.08.09 21:00, Andreas Jung wrote: On 02.08.09 20:48, Andreas Jung wrote: Hi, the doctests for zope.index 3.5.2 - as used in Zope 2.12 - fail badly: File /home/ajung/.buildout/eggs/zope.index-3.5.2-py2.6-linux-x86_64.egg/zope/index/text/tests/../textindex.txt, line 143, in textindex.txt Failed example: [(k, %.4f % v) for (k, v) in index2.apply(Zorro).items()] == result Expected: True Got: False Obviously another Python 2.5/.6 incompatiblity. zope.index 3.5.1 works fine with Python 2.4-2.6 - obviously changes to okascore.c introduced with zope.index 3.5.2 are to blame?! The diff between 3.5.1 and 3.5.2 is pretty long and substantial. I doubt that such a major change us ok as a bugfix release. I should have become a new major release. I suggest that the current trunk should be released as 3.6.0 and the current 3.5.1 released should be copied to tags/3.5.3 and re-released. Andreas begin:vcard fn:Andreas Jung n:Jung;Andreas org:ZOPYX Ltd. Co. KG adr;quoted-printable:;;Charlottenstr. 37/1;T=C3=BCbingen;;72070;Germany email;internet:i...@zopyx.com title:CEO tel;work:+49-7071-793376 tel;fax:+49-7071-7936840 tel;home:+49-7071-793257 x-mozilla-html:FALSE url:www.zopyx.com version:2.1 end:vcard ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] zope.index 3.5.2 broken
On 8/2/09 3:07 PM, Andreas Jung wrote: On 02.08.09 21:00, Andreas Jung wrote: On 02.08.09 20:48, Andreas Jung wrote: Hi, the doctests for zope.index 3.5.2 - as used in Zope 2.12 - fail badly: File /home/ajung/.buildout/eggs/zope.index-3.5.2-py2.6-linux-x86_64.egg/zope/index/text/tests/../textindex.txt, line 143, in textindex.txt Failed example: [(k, %.4f % v) for (k, v) in index2.apply(Zorro).items()] == result Expected: True Got: False Obviously another Python 2.5/.6 incompatiblity. zope.index 3.5.1 works fine with Python 2.4-2.6 - obviously changes to okascore.c introduced with zope.index 3.5.2 are to blame?! The diff between 3.5.1 and 3.5.2 is pretty long and substantial. I doubt that such a major change us ok as a bugfix release. I should have become a new major release. I suggest that the current trunk should be released as 3.6.0 and the current 3.5.1 released should be copied to tags/3.5.3 and re-released. I think your test failure is related to a 64-bit issue. Shane has some fixes on the trunk that may solve this. That said, why does Zope 2.1.2 use the zope.index textindex? The Zope 2 implementation is still very superior. Before we started fixing it, there were places that the zope.index text index didn't actually work at all Where it did work, it worked very, very slowly. We've been trying to improve it incrementally (mostly by backporting all the fixes that have accreted to the Zope 2 implementation). I'll try to pin this down and make a 3.6.0 release, but Zope 2 shouldn't be using this index. - C ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] zope.index 3.5.2 broken
On 8/2/09 4:41 PM, Chris McDonough wrote: On 8/2/09 3:07 PM, Andreas Jung wrote: On 02.08.09 21:00, Andreas Jung wrote: On 02.08.09 20:48, Andreas Jung wrote: Hi, the doctests for zope.index 3.5.2 - as used in Zope 2.12 - fail badly: File /home/ajung/.buildout/eggs/zope.index-3.5.2-py2.6-linux-x86_64.egg/zope/index/text/tests/../textindex.txt, line 143, in textindex.txt Failed example: [(k, %.4f % v) for (k, v) in index2.apply(Zorro).items()] == result Expected: True Got: False Obviously another Python 2.5/.6 incompatiblity. zope.index 3.5.1 works fine with Python 2.4-2.6 - obviously changes to okascore.c introduced with zope.index 3.5.2 are to blame?! The diff between 3.5.1 and 3.5.2 is pretty long and substantial. I doubt that such a major change us ok as a bugfix release. I should have become a new major release. I suggest that the current trunk should be released as 3.6.0 and the current 3.5.1 released should be copied to tags/3.5.3 and re-released. I think your test failure is related to a 64-bit issue. Shane has some fixes on the trunk that may solve this. That said, why does Zope 2.1.2 use the zope.index textindex? The Zope 2 implementation is still very superior. Before we started fixing it, there were places that the zope.index text index didn't actually work at all Where it did work, it worked very, very slowly. We've been trying to improve it incrementally (mostly by backporting all the fixes that have accreted to the Zope 2 implementation). I'll try to pin this down and make a 3.6.0 release, but Zope 2 shouldn't be using this index. The tests now pass on my 32-bit machine. Can someone with a 64-bit system try them? - C ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] zope.index 3.5.2 broken
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Andreas Jung wrote: Hi, the doctests for zope.index 3.5.2 - as used in Zope 2.12 - fail badly: File /home/ajung/.buildout/eggs/zope.index-3.5.2-py2.6-linux-x86_64.egg/zope/index/text/tests/../textindex.txt, line 143, in textindex.txt Failed example: [(k, %.4f % v) for (k, v) in index2.apply(Zorro).items()] == result Expected: True Got: False Thoughts? Shoot that test: it sucks both as a test and as documentation. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFKdgL4+gerLs4ltQ4RAntBAJ0eC3YcZxxDh8qw2hIRSFK2UA1Q1wCghbn+ +eMTXYnbaDt0cmHAbTyYgmY= =uVOK -END PGP SIGNATURE- ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] zope.index 3.5.2 broken
On Sun, Aug 2, 2009 at 10:41 PM, Chris McDonoughchr...@plope.com wrote: That said, why does Zope 2.1.2 use the zope.index textindex? The Zope 2 implementation is still very superior. The current policy for the test runs is to test both all direct dependencies and all transitive dependencies including all test dependencies on all levels. We have spent quite some time reducing the real dependencies already, but the test dependencies are still in a somewhat worse state. So in this case zope.app.zptpage has a code dependency on zope.index. And zope.app.zptpage is a test dependency of a number of packages, including zope.traversing, zope.app.exception and zope.app.publication. I think it might be time to remove the test extras again for reusable zope.* packages and declare them as real dependencies. This would allow us to actually see them more easily and get truly embarrassed - a good way of spurring motivation to fix the mess ;-) Hanno ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] zope.index 3.5.2 broken
On 8/2/09 5:19 PM, Tres Seaver wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Andreas Jung wrote: Hi, the doctests for zope.index 3.5.2 - as used in Zope 2.12 - fail badly: File /home/ajung/.buildout/eggs/zope.index-3.5.2-py2.6-linux-x86_64.egg/zope/index/text/tests/../textindex.txt, line 143, in textindex.txt Failed example: [(k, %.4f % v) for (k, v) in index2.apply(Zorro).items()] == result Expected: True Got: False Thoughts? Shoot that test: it sucks both as a test and as documentation. Already shot on the trunk (by Shane). - C ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] zope.index 3.5.2 broken
On 8/2/09 5:23 PM, Hanno Schlichting wrote: On Sun, Aug 2, 2009 at 10:41 PM, Chris McDonoughchr...@plope.com wrote: That said, why does Zope 2.1.2 use the zope.index textindex? The Zope 2 implementation is still very superior. The current policy for the test runs is to test both all direct dependencies and all transitive dependencies including all test dependencies on all levels. We have spent quite some time reducing the real dependencies already, but the test dependencies are still in a somewhat worse state. So in this case zope.app.zptpage has a code dependency on zope.index. And zope.app.zptpage is a test dependency of a number of packages, including zope.traversing, zope.app.exception and zope.app.publication. OK. There's no way any release of Zope 2 should depend on zope.index, at least until its index implementations are as good as the ones in Zope 2 itself. I don't know if in the short term (for a release) that means that the Zope 2 test regime needs to change, or if the dependencies need to be fixed, or what. In the medium term, we should fix the dependencies obviously. Another immediate thing to do would just be to pin the zope.index version to 3.5.1, given that it doesn't matter one way or another whether these tests pass for purposes of a Zope 2 release. I think it might be time to remove the test extras again for reusable zope.* packages and declare them as real dependencies. This would allow us to actually see them more easily and get truly embarrassed - a good way of spurring motivation to fix the mess ;-) Oh good, I might win that beer from Martijn after all. ;-) - C ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] zope.index 3.5.2 broken
On Sun, Aug 2, 2009 at 11:35 PM, Chris McDonoughchr...@plope.com wrote: There's no way any release of Zope 2 should depend on zope.index, at least until its index implementations are as good as the ones in Zope 2 itself. I don't know if in the short term (for a release) that means that the Zope 2 test regime needs to change, or if the dependencies need to be fixed, or what. In the medium term, we should fix the dependencies obviously. Note that we only depend on zope.index via a test dependency of one of our dependencies, so anyone downloading and installing the Zope2 distribution won't actually pull in zope.index. Even running the tests for Zope2 itself won't need it. Only if you want to run the tests for all dependencies of Zope2, like we do in our core development buildout, you actually get it. Hanno ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
