Re: [Python-Dev] Summary of 2 years of Python fuzzing
On Mon, Jan 25, 2010 at 2:34 PM, Victor Stinner victor.stin...@haypocalc.com wrote: Interaction with the Python developers == I open an issue for each bug found in CPython. I describe how to reproduce it and try to write a patch. I have learn to always write an unit test, useful to reproduce the bug, and it makes Python commiters happy :-) Yes! :-) The reaction depends on the impacted component, the severity of the bug, the complexity of the code reproducing the bug, and the quality of my bug report :-) The answer was always quick for core components. But some modules are maintained by everyone, which means nobody, like imageop, audioop or cProfile/hotshot. Having a module maitainer, like Guilherme Polo aka gpolo for Tkiner, does really help! It looks like fuzzing bugs are not always appreciated by developers, maybe because they are always borderline cases (not realist). As many others have said, I greatly appreciate your work in this area. Even if obscure, all of the crashes are important to fix. Stability has been a big selling point of Python and will likely remain so in the future. While it may not be important in some environments, it's critical in others. We should strive to do the best job we can and Fusil makes it easier to be more robust. Sometimes, even if I write a patch, an unit test, explain the problem and the solution, I don't get any comment. It doesn't motivate me to continue fuzzing :-/ Yeah, I know this can be frustrating. As Terry mentioned later in this post, many of your patches have been committed which is very impressive. The outstanding ones are at most about a month old. I would expect most of those to move along as people find the time to address them. If you keep creating more high quality patches (e.g., with unit tests), you could be on your way to becoming a Python committer which would help solve this problem. I definitely hope you continue to find and fix problems in Python. It helps everyone who uses Python even those who will never know to thank you. Who knows, someone might even write a book about Fusil someday about a topic as obscure as Beautiful Testing. :-) n ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Summary of 2 years of Python fuzzing
On Wed, Jan 27, 2010 at 11:58 AM, Ben Finney ben+pyt...@benfinney.id.au wrote: Chris Bergstresser ch...@subtlety.com writes: On Wed, Jan 27, 2010 at 2:54 AM, Ben Finney ben+pyt...@benfinney.id.au wrote: Neal Norwitz nnorw...@gmail.com writes: Who knows, someone might even write a book about Fusil someday about a topic as obscure as Beautiful Testing. :-) Your suggested title is already taken, though, for exactly this purpose. The book “Beautiful Testing”, published by O'Reilly, might help URL:http://oreilly.com/catalog/9780596159825. I suspect Neal already knows that, since he cowrote chapter 9 Beautiful is Better than Ugly. Ah, good. I did suspect that might be the case, but couldn't easily find Neal's name associated with the book. I decided to post anyway, preferring the information to be out there at the risk of seeming to miss the joke. Maybe one day I'll get to read the book :- Ben, thanks. Your comment gave me a great laugh and I really appreciated it. :-) The chapter is about the general Python development process, including testing, static analysis, dynamic analysis, including Fusil among other things. One of my points to Victor and everyone else like him is that even though it may seem no one is listening to you or cares, you might be surprised to find out how many people really are paying attention and do care. If you have something you want to do, make it happen. One person really can do amazing things. You may never get recognized for much of what you do, that doesn't make it any less important. n ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of the Buildbot fleet and related bugs
On Fri, Nov 6, 2009 at 12:27 AM, Mark Dickinson dicki...@gmail.com wrote: On Fri, Nov 6, 2009 at 3:53 AM, R. David Murray rdmur...@bitdance.com wrote: (2) issue 4970: consistent signal 32 error on the norwitz-x86 Gentoo buildslave in 3.1 and 3.x. This may be due to the box running an old threading library, but it does make one wonder what changed in 3.x that exposed it. This error has been happening since well before 3.0 was released. Asking for access to Neal's machine is probably the only sensible way to diagnose it. (A less invasive but slower way to debug would be to create a branch especially for this bug and do repeated runs to figure out which part of test_os is causing the failure.) IIRC, I spent quite a bit of time trying to nail this down. I don't remember finding any useful information on the cause (beyond narrowing it to some tests). As Mark said, this has been happening for a long time. I'm reticent to provide access to the machine, as it's not really mine. I'm not even sure I have access, I haven't logged in for a long time. I'd just like to say thanks again to everyone for making the buildbots more green and also improving the general testing infrastructure for Python. n ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] thanks to everyone cleaning up the tests
On Mon, Nov 2, 2009 at 2:18 AM, Mark Dickinson dicki...@gmail.com wrote: On Sat, Oct 31, 2009 at 9:47 PM, Brett Cannon br...@python.org wrote: Just wanted to publicly thank everyone who has been causing all the checkins to fix and stabilize the test suite (I think it's mostly Antoine and Mark, but I could be missing somebody; I'm under a deadline so only have marginal higher brain functionality). Not guilty, your honour. Blame Antoine and David. :) David's new buildslave seems to be making a difference, too. It's really been awesome to see all the work improving the test suite. This will make it easier to implement change without fear of breaking everything. These changes will reap benefits for a long time to come. Kudos for picking up this work and to Brett for calling you out. :-) Thanks, n ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] nonlocal keyword in 2.x?
On Wed, Oct 21, 2009 at 6:56 PM, Mike Krell mbk.li...@gmail.com wrote: Is there any possibility of backporting support for the nonlocal keyword into a 2.x release? I see it's not in 2.6, but I don't know if that was an intentional design choice or due to a lack of demand / round tuits. I'm also not sure if this would fall under the scope of the proposed moratorium on new language features (although my first impression was that it could be allowed since it already exists in python 3. IIRC, Jeremy Hylton tried to backport it during a Pycon sprint a few years back. I think it was difficult and he dropped it. I don't know if there were any showstoppers or if Jeremy saved his work...or if my memory is even correct. :-) n ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] cleanup before 3.1 is released
Has anyone run valgrind/purify and pychecker/pylint on the 3.1 code recently? Both sets of tools should be used before the final release so we can fix any obvious problems. n ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Adding support to curses library
On Tue, Feb 24, 2009 at 2:18 PM, Heracles
st...@integrityintegrators.net wrote:
Hello,
I am working on a patch to add to the _cursesmodule.c file of the Python
core libraries. I figured I would take on one of the implemented functions
to try to get my feet wet contributing to the project. At any rate, I have
the following function defined in the 2.7.a version updated from SVN this
morning:
I'm glad you are interested in developing Python. I'm not sure if
this is the best forum. OTOH, I'm not sure if comp.lang.python would
be appropriate either.
I'd suggest making a proper patch and submitting it to http://bugs.python.org
- Snippet ---
// Insert new method color_set Steve Owens 2/24/2009
// The curses library color_set function has the following signature:
// int color_set(short color_pair_number, void* opts);
static PyObject *
PyCurses_color_set(PyObject *self, PyObject *args)
{
short color_pair_number;
void * opts;
int erg;
// These macros ought to be documented in the API docs
// but they aren't yet.
PyCursesInitialised
PyCursesInitialisedColor
// Per ncurses Man Page:
// The routine color_set sets the current color of the given window to
// the foreground/background combination described by the
color_pair_number.
// The parameter opts is reserved for future use, applications must
supply a
// null pointer.
switch(PyTuple_Size(args))
{
case 1:
// Dont make them pass a useless null pointer.
if (!PyArg_ParseTuple(args, h, color_pair_number)) return NULL;
break;
case 2:
// Allow them to pass the opts pointer so that when ncurses is later
updated.
// This method will still work.
if (!PyArg_ParseTuple(args, hO, color_pair_number, opts))
return
NULL;
break;
default:
PyErr_SetString(PyExc_TypeError, color_set requires 1 or 2 arguments
(color_pair_number[, opts]?));
return NULL;
}
erg = color_set(color_pair_number, opts); // Debating on forcing null
here.
if (erg == ERR)
return PyCursesCheckERR(erg, color_set);
else
PyInt_FromLong((long) 1L);
I did a cursory review of the patch and if this is the exact code,
this is a problem. You are missing a return statement. The compiler
should have issued a warning for this too.
}
-End Snippet ---
I also have the following added in (see last line of the snippet):
- Snippet ---
static PyMethodDef PyCurses_methods[] = {
{baudrate, (PyCFunction)PyCurses_baudrate, METH_NOARGS},
{beep, (PyCFunction)PyCurses_beep, METH_NOARGS},
{can_change_color, (PyCFunction)PyCurses_can_change_color,
METH_NOARGS},
{cbreak, (PyCFunction)PyCurses_cbreak, METH_VARARGS},
{color_content, (PyCFunction)PyCurses_Color_Content,
METH_VARARGS},
{color_pair, (PyCFunction)PyCurses_color_pair, METH_VARARGS},
{color_set, (PyCFunction)PyCurses_color_set, METH_VARARGS},
-End Snippet ---
The code compiles and installs fine, but when I run the following unit test,
I get a segmentation fault:
- Snippet ---
import unittest, curses
from test import test_support
def testCursesColorSet(stdscrn):
curses.init_pair(1, curses.COLOR_RED, curses.COLOR_WHITE)
curses.init_pair(2, curses.COLOR_WHITE, curses.COLOR_BLUE);
i = curses.color_set(1, NULL);
stdscrn.addstr(RED/BLACK (%0)\n.format(i))
i = curses.color_set(2, NULL);
stdscrn.print(WHITE/BLUE (%0)\n.format(i))
i = curses.color_set(0, NULL);
stdscrn.print(Default (%0)\n.format(i))
def test_main(stdscrn):
curses.savetty()
if curses.has_color():
testCursesColorSet(stdscrn)
else
stdscr.addstr( Test Aborted: Color not supported on this terminal.)
if __name__ == '__main__':
curses.wrapper(test_main)
-End Snippet ---
It turns out that by commenting out this line in the _cursesmodule.c code,
allows the unit test to run
obviously reporting the error as expected:
- Snippet ---
//erg = color_set(color_pair_number, opts); // Debating on forcing null
here.
-End Snippet ---
At any rate I am stuck. I am still trying to build just a plain C file
which will test the color_set function
outside of python, but that is another task.
Any suggestions?
Beyond what I said above, typically you need to go the next step.
Fire up a debugger and determine exactly where and why it's crashing.
Good luck!
n
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
Re: [Python-Dev] Daily documentation builds
On Wed, Feb 11, 2009 at 6:24 AM, A.M. Kuchling a...@amk.ca wrote: On Tue, Feb 10, 2009 at 09:16:48PM -0800, Neal Norwitz wrote: I ran 2.6, 3.0, and 3.1 manually. 2.7 should get picked up on the next run. The problem is that regrtest.py -R hangs from time to time which caused the machine to run out of memory. Does anyone else have regrtest.py -R hang for them? Some tests were disabled to try to prevent the problem, but it still happens from time to time. It's also possible that tools/sphinx needs a manual 'svn update'. A recent change to sphinxext/pyspecific.py imports a new package, sphinx.builders. I had to do this to keep my source tree building the docs. Ok, I'll take a look later. Misc/build.sh should be doing a make update which should fix most problems. There was one problem a while back I had to fix manually though. If you see ways to make Misc/build more robust, feel free to check in changes. The doc section is at the bottom of the file. n ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Daily documentation builds
I ran 2.6, 3.0, and 3.1 manually. 2.7 should get picked up on the next run. The problem is that regrtest.py -R hangs from time to time which caused the machine to run out of memory. Does anyone else have regrtest.py -R hang for them? Some tests were disabled to try to prevent the problem, but it still happens from time to time. n On Tue, Feb 10, 2009 at 11:33 AM, Raymond Hettinger pyt...@rcn.com wrote: The online development docs stopped their nightly rebuilds four days ago. Raymond ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/nnorwitz%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] python breakpoint opcode
On Thu, Jan 29, 2009 at 2:38 AM, Dr Andrew Perella a...@eutechnyx.com wrote: Hi, I was thinking of adding a breakpoint opcode to python to enable less invasive debugging. I came across posts from 1999 by Vladimir Marangozov and Christian Tismer discussing this issue but the links to the code are all out of date. Can you provide the links? n ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] enabling a configure option
If you only care about this running on a single machine to get some coverage and don't care about all architectures, you can change Misc/build.sh to add the configure option. n On Mon, Jan 26, 2009 at 2:31 PM, Antoine Pitrou solip...@pitrou.net wrote: Martin v. Löwis martin at v.loewis.de writes: Me. Does it have to be a configure option? It is difficult to invoke different commands in different branches; better if the configures in all branches get the same options. Well, after a quick test, it seems that configure doesn't complain if you pass it an unknown option (at least one that begins with '--with'). So we can still use the same options on all branches. (as for the need for it to be a configure option, it was the consensus which emerged after discussion in the tracker entry, both to provide some flexibility and for fear that enabling it by default could trigger some compiler bugs -- although the latter is of course unlikely) Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/nnorwitz%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] status of 2.5
Should we plan to put out a final 2.5 release? If so, should we continue to backport fixes (like Martin's removal of Alpha in setup.py)? My preference is that we do put out a final 2.5 that has all accumulated bug fixes. Then close the branch. That way if we put out a security release for 2.5, it will be clean and easy. n ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Doc nits question
On Thu, Oct 2, 2008 at 6:21 AM, Georg Brandl [EMAIL PROTECTED] wrote: Jesse Noller schrieb: So, we just released and there are a few doc typo bugs being filed - my question is if all doc-fixes have to wait for 2.6.1/2.7 or if we can hotfix the 2.6 docs? I intend to set things up so that the docs at docs.python.org are continually rebuilt, just like the /dev docs were until now. The 2.6 docs are now updated similar to how 2.5 was (hourly). 2.5 docs are no longer updated. In case you can't guess the url, it's: http://docs.python.org/dev/2.6/ 3.0 should continue to work. Let me know if you have any problems. n Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/nnorwitz%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of MS Windows CE port
On Tue, Sep 23, 2008 at 1:52 AM, Ulrich Eckhardt [EMAIL PROTECTED] wrote: Greetings! I'm currently trying to assess the effort required for a CE port. I'm already aware of the project at http://pythonce.sourceforge.net, but I find it a waste of time to have two separate projects which then require syncing changes back and forth. Questions: - What are the feelings towards a port in the mainline? I'm aware that you are trying to get 2.6 and 3.0 out of the door and are probably not willing to risk such changes to the release code right now, but in general? In general, we try to ensure that there is an active maintainer, preferably more than one. As long as it doesn't make the code that much harder to maintain, it is desirable to support more platforms. - Is anyone already working on this? I know that Brad Clements made an attempt some years ago but didn't get it integrated, are there any others? Maybe even any ongoing work? I'm not sure if anyone here knows. I haven't seen anything on this list, but that doesn't mean there isn't another list either on python.org or elsewhere that does have the information. Try searching if you haven't. - I see that you are using Subversion and Bazaar, would it be possible to get a feature branch for the CE port? I'm thinking about the effort going on at pythonce.sf.net and some further porting that I need for my specific platform. IMHO both would be better off in the same repository as your upstream sources, since merging changes back and forth is much easier that way. It's possible. The best way to get the ball rolling is to produce some patches against the mainline. Demonstrate what you can do and privileges will follow. It's a similar process to becoming a python-committer. Perhaps the best way to start is using Bazaar or some distributed version control system for your patches. Once you've demonstrated your ability and intent, we can make a branch. Cheers, n ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] Python Regression Test Failures basics (1)
On Sun, Sep 14, 2008 at 5:24 AM, Benjamin Peterson [EMAIL PROTECTED] wrote: On Sun, Sep 14, 2008 at 4:07 AM, Nick Coghlan [EMAIL PROTECTED] wrote: Neal Norwitz wrote: test_epoll skipped -- kernel doesn't support epoll() ... test_ioctl skipped -- Unable to open /dev/tty ... test_multiprocessing skipped -- OSError raises on RLock creation, see issue 3111! ... test test_normalization failed -- Traceback (most recent call last): File /tmp/python-test/local/lib/python2.6/test/test_normalization.py, line 90, in test_main self.failUnless(X == NFC(X) == NFD(X) == NFKC(X) == NFKD(X), c) AssertionError: 6918 ... 326 tests OK. 1 test failed: test_normalization ... 3 skips unexpected on linux2: test_epoll test_multiprocessing test_ioctl Neal, What environment are you using to run the debug-mode regression tests? The above four tests run without any problems for me, but I'm just running them in a normal Kubuntu desktop shell. Something in Neal's build which has made a difference before is that the tests are run after a make install. Benjamin is correct. See Misc/build.sh for the script that generates this. See http://docs.python.org/dev/results/ for details about the most recent run. n ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] ?spurious? Timeout in BSDDB under MS Windows
On Mon, Sep 8, 2008 at 3:24 AM, Trent Nelson [EMAIL PROTECTED] wrote: On Fri, Sep 05, 2008 at 05:55:13PM +0200, Jesus Cea wrote: Trent, are you available to look at the ?spurious? timeout failures in bsddb replication code in the Windows buildbot?. Ten seconds timeout should be plenty enough. I can't debug any MS Windows issue myself; this is a Microsoft-free environment. I think I added in 10 seconds 'cause the tests kept failing when it was at 2 seconds ;-) I remember digging around the code a bit when I bumped bsddb to 4.7 on Windows to try and figure out what was going on. As far as I could tell it wasn't anything obvious caused by the Python code; is it possible this could be an issue with the underlying bsddb code? Are the buildbot(s) running under vmware? n ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] performance
Thanks Antoine! On Sun, Aug 24, 2008 at 5:58 AM, Antoine Pitrou [EMAIL PROTECTED] wrote: Neal Norwitz nnorwitz at gmail.com writes: Can someone (else) compare performance of 2.5, 2.6, and 3.0? Tests done on a 32-bit Linux installation on an Athlon 3600+ X2. Compiled with gcc in UCS2 mode. pystone --- - 2.5: 43859.6 pystones/second - 2.6: 42016.8 pystones/second - 3.0: 38759.7 pystones/second So 3.0 is about 10% slower than 2.x. Given all the changes, that doesn't seem too bad. Though, see below. It looks like at least class attribute lookup is much, much slower. Can you investigate some of these and file bugs as appropriate? richards.py --- (roughly, richards is an object-oriented method-calling benchmark) - 2.5: 770.54 ms per iteration - 2.6: 572.84 ms per iteration - 3.0: 566.69 ms per iteration I'm a little concerned about why the big change here. Though if I'm reading this right it's a speed up...or am I just being optimistic? The only things I remember that changed between 2.5 and 2.6 that might affect this (without looking at any code) were: 1) Armin's method caching, and 2) the slowdown to isinstance/issubclass which has a release blocker: http://bugs.python.org/issue2534 Can you dig into this and see what caused the slowdown? stringbench --- - 2.5: unicode: 265.84 s / bytes: 180.50 s - 2.6: unicode: 256.22 s / bytes: 184.45 s - 3.0: unicode: 248.07 s / bytes: not tested pybench: 2.6 vs. 2.5 this is 2.6, other is 2.5. Hopefully anything 10% is a real change and not just noise. Test minimum run-timeaverage run-time thisother diffthisother diff --- BuiltinFunctionCalls: 177ms 181ms -2.2% 178ms 182ms -2.1% BuiltinMethodLookup: 157ms 177ms -11.1% 158ms 177ms -11.0% Maybe explained by Armin's patch. CompareFloats: 171ms 171ms -0.3% 171ms 171ms -0.3% CompareFloatsIntegers: 158ms 162ms -2.1% 160ms 167ms -4.8% CompareIntegers: 208ms 208ms -0.0% 209ms 209ms -0.3% CompareInternedStrings: 189ms 183ms +3.4% 189ms 184ms +2.7% CompareLongs: 154ms 154ms -0.2% 154ms 154ms -0.2% CompareStrings: 162ms 159ms +1.7% 163ms 159ms +2.5% CompareUnicode: 144ms 149ms -3.6% 144ms 152ms -5.0% ComplexPythonFunctionCalls: 173ms 242ms -28.5% 176ms 243ms -27.6% Maybe explained by Armin's patch. ConcatStrings: 196ms 198ms -0.9% 203ms 200ms +1.4% ConcatUnicode: 153ms 151ms +1.5% 155ms 157ms -1.5% CreateInstances: 172ms 169ms +1.5% 173ms 170ms +1.4% CreateNewInstances: 131ms 148ms -11.6% 132ms 151ms -12.2% Maybe explained by Armin's patch. CreateStringsWithConcat: 209ms 206ms +1.4% 209ms 208ms +0.8% CreateUnicodeWithConcat: 128ms 124ms +3.7% 129ms 124ms +3.4% DictCreation: 115ms 149ms -22.8% 116ms 150ms -22.8% Why? What changed? DictWithFloatKeys: 208ms 207ms +0.4% 208ms 208ms +0.0% DictWithIntegerKeys: 173ms 173ms -0.0% 174ms 173ms +0.5% DictWithStringKeys: 162ms 162ms -0.1% 162ms 162ms +0.1% ForLoops: 181ms 181ms -0.2% 181ms 182ms -0.2% IfThenElse: 169ms 168ms +0.2% 169ms 169ms -0.2% ListSlicing: 109ms 108ms +0.2% 109ms 109ms +0.1% NestedForLoops: 198ms 197ms +0.2% 198ms 197ms +0.1% NormalClassAttribute: 176ms 172ms +2.0% 176ms 173ms +1.7% NormalInstanceAttribute: 162ms 161ms +1.0% 163ms 161ms +1.1% PythonFunctionCalls: 161ms 151ms +6.6% 162ms 153ms +6.0% PythonMethodCalls: 188ms 189ms -0.7% 189ms 193ms -2.0% Recursion: 235ms 230ms +2.2% 236ms 233ms +1.2% SecondImport: 113ms 114ms -1.0% 115ms 115ms +0.0% SecondPackageImport: 120ms 116ms +3.6% 120ms 117ms +2.9% SecondSubmoduleImport: 154ms 146ms +6.0% 156ms 148ms +5.7% SimpleComplexArithmetic: 163ms 151ms +8.3% 163ms 151ms +8.6% SimpleDictManipulation: 177ms 173ms +2.6% 180ms 174ms +3.2% SimpleFloatArithmetic: 169ms 164ms +3.0% 169ms 166ms +2.2% SimpleIntFloatArithmetic: 153ms 152ms +1.2% 156ms 153ms +1.8% SimpleIntegerArithmetic: 156ms 152ms +2.5% 156ms 153ms +2.0% SimpleListManipulation: 156ms 158ms -1.0% 157ms 158ms
[Python-Dev] what version of bsddb to use/accept for 2.6/3.0
Many buildbots are running bsddb 4.7, particularly the debian/ubuntu ones (4.7.25 which seems to be the latest). Some of them are crashing, others are not. The max version we support in both 2.6 and 3.0 is 4.7. Should we allow this version or should we use a lower maximum that is more likely to work (ie, not crash)? It looks like the WIndows buildbots use 4.4.20. Unfortunately, the Windows bots aren't in great shape either. Additionally, there are reference leaks in both 2.6 and 3.0: test_bsddb3 leaked [80, 80] references, sum=160 (2.6) test_bsddb3 leaked [63, 63] references, sum=126 (3.0) It would be nice to get as many of these things fixed up before release. Jesus, Greg, Trent, can you make some time over the next week to stabilize bsddb support? Thanks, n PS. To change the max version of bsddb supported in Unix, modify max_db_ver in setup.py. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] current problems that should be fixed before release (2.6)
Below are the problems I found that have not been fixed at r65995 on trunk (2.6). There will be a separate mail for the 3.0 problems. I've done the following: * built in debug and opt mode (gcc 4.1.2) fixing the important warnings * run all the tests in both modes * run all the tests (except test_logging and test_ssl) under valgrind 3.3.1 fixing the problems * run failmalloc (1.0) on startup for the first 4000 allocations fixing most problems * run pychecker 0.8.18 over the stdlib (Lib/*.py, Lib/[bcdelhjmswx]*/*.py) Can someone run purify on windows over 2.6? Can someone (else) compare performance of 2.5, 2.6, and 3.0? pychecker reported problems: Lib/gzip.py:114: Using property (filename) in classic class GzipFile may not work Lib/io.py:1394: No module attribute (device_encoding) found - device_encoding access is protected, but I don't see where it can be set Lib/sched.py:122: Using property (queue) in classic class scheduler may not work Lib/tempfile.py:535: Using property (closed) in classic class SpooledTemporaryFile may not work Lib/tempfile.py:539: Using property (encoding) in classic class SpooledTemporaryFile may not work Lib/tempfile.py:553: Using property (mode) in classic class SpooledTemporaryFile may not work Lib/tempfile.py:557: Using property (name) in classic class SpooledTemporaryFile may not work Lib/tempfile.py:561: Using property (newlines) in classic class SpooledTemporaryFile may not work Lib/tempfile.py:580: Using property (softspace) in classic class SpooledTemporaryFile may not work Lib/multiprocessing/connection.py:95: No global (PipeListener) found Lib/multiprocessing/connection.py:132: No global (PipeClient) found - both of the Pipe classes are defined only for windows. I don't see a similar def for Unix. Lib/multiprocessing/connection.py:378: No global (AuthenticationError) found I haven't looked into the property complaint to see how bad it is. test_bsddb3 is failing on several 2.6 buildbots. test_bsddb3 leaks 80 references. Results of valgrind 3.3.1 on Ubuntu 6.10 amd64, gcc 4.1.2 (prerelease): Uninitialized memory reads: No problems in python code. Though it's possible there are errors in ctypes. I'm ignoring those reports due to a problem in dlopen. Wild memory writes: No problems in python code. Memory leaks: No major problems. However, it seems that there are a bunch of little leaks when forking a process. I haven't determined if these are real problems or not. There is one memory leak in ctypes (sorry no real useful information): 44 bytes in 1 blocks are definitely lost in loss record 43 of 191 malloc (vg_replace_malloc.c:207) resize (callproc.c:1733) It seems to always leak 44 bytes n ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Think a dead import finding script would be handy?
On Sun, Aug 17, 2008 at 3:04 PM, Brett Cannon [EMAIL PROTECTED] wrote: On Sun, Aug 17, 2008 at 1:40 PM, Georg Brandl [EMAIL PROTECTED] wrote: Brett Cannon schrieb: After Christian mentioned how we could speed up interpreter start-up by removing some dead imports he found, I decided to write up a quick script that generates the AST for a source file and (very roughly) tries to find imports that are never used. People think it's worth tossing into Tools, even if it is kind of rough? Otherwise I might toss it into the sandbox or make a quick Google code project out of it. Regardless, one interesting side-effect of the script is that beyond finding some extraneous imports in various places, it also found some holes in __all__. I have the script look for the definition of __all__ and consider an import used if it is listed there. pylint already finds unused imports. It finds tons of other, relatively useless, stuff in the default configuration, but I'm sure it can be coaxed into only showing unused imports too. Does anyone ever run pylint over the stdlib on occasion? I usually run pychecker (which also finds unused imports and a whole lot more) before releases. I typically wait for things to settle down (ie, well into beta). n ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Interesting blog post by Ben Sussman-Collins
On Thu, Jun 12, 2008 at 8:41 PM, Guido van Rossum [EMAIL PROTECTED] wrote: My colleague and SVN developer Ben Sussman-Collins occasionally blogs about the social side of (mostly open source) software development. He just posted a new one that struck a chord: http://blog.red-bean.com/sussman/?p=96 The story's main moral: submit your code for review early and often; work in a branch if you need to, but don't hide your code from review in a local repository until it's perfect. Let's all remember this and make sure not to drop code bombs on each other. :-) Ben mentions this in the post, but it's a good reminder: comments on python-checkins are *not* personal. The goal is to make the code better and/or gain better understanding. We all make mistakes, better to correct them early before they become big problems.. n ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Proposed unittest changes
[Michael working on cleaning up the unittest module] it seems like most of the good ideas have been captured already. I'll through two more (low priority) ideas out there. 1) Randomized test runner/option that runs tests in a random order (like regrtest.py -r, but for methods) 2) decorator to verify a test method is supposed to fail #2 is useful for getting test cases into the code sooner rather than later. I'm pretty sure I have a patch that implements this (http://bugs.python.org/issue1399935). It didn't fit in well with the old unittest structure, but seems closer to the direction you are headed. One other idea that probably ought not be done just yet: add a way of failing with the test continuing. We use this at work (not in Python though) and when used appropriately, it works quite well. It provides more information about the failure. It looks something like this: def testMethod(self): # setup self.assertTrue(precondition) self.expectTrue(value) self.expectEqual(expected_result, other_value) All the expect methods duplicate the assert methods. Asserts can the test to fail immediately, expects don't fail immediately allowing the test to continue. All the expect failures are collected and printed at the end of the method run. I was a little skeptical about assert vs expect at first, but it has proven useful in the long run. As I said, I don't think this should be done now, maybe later. n ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] r62342 - python/branches/py3k/Objects/bytesobject.c
On Tue, Apr 15, 2008 at 2:21 AM, Stefan Behnel [EMAIL PROTECTED] wrote: Neal Norwitz wrote: Iteration with the dict methods (e.g., keys - iterkeys()), map/zip/filter returning iterator rather than list. That's only an optimisation, it's not functionally required. A list behaves like an iterator in most use cases, so it's rather unlikely that Py3 code will break in the backport because of this (and it's very unlikely that static analysis can catch the cases where it breaks). There should be a rule to optimise list(map(...)) into map(...) and list(x.keys()) into plain x.keys() etc., but I don't think there is much more to do here. It's not just an optimization if a copy won't fit in memory. I'd like the solution to be closer to 100% than 95%. int - (int, long) Is there any case where this might be required? I don't see any reason why back-converted Py3 code should break here. What would long() be needed for in working Py3 code that int() doesn't provide in Py2? Although you might have been referring to isinstance(x, int) in Py3? Yes, sorry, I wasn't explicit. isinstance is specifically what I was referring to (or other type checks such as type(x) in (int, long)). str - basestring or (str, unicode) This is an issue, although I think that Py3 is explicit enough here to make this easy to handle by static renaming (and maybe optimising isinstance(s, (str, bytes)) into ..., basestring))). __bool__ - __nonzero__ exec/execfile input - rawinput Also valid issues that can be handled through renaming and static syntactic adjustments. Most things that have a fixer in 2to3 would also require one in 3to2. I think the more explicit syntax in Py3 will actually make it easier to back-convert the code statically from 3to2 than to migrate it from 2to3. Sure, that's the idea. I haven't seen any action on 3to2 (although I'm very behind on email). Stefan, could you try to implement some of these and report back how it works? n ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python 2.4.4/2.4.5 test_pty failure on Solaris 10
On Sat, Apr 12, 2008 at 11:02 AM, [EMAIL PROTECTED] wrote: I know this is old stuff, but... I want to update our Python 2.4 installation at work from 2.4.2 to 2.4.5 (the latest 2.4 source release). I get a test failure for test_pty, an extra ^M at the end of one line. I don't get a failure in the 2.4.2 installation, but the 2.4.4 and 2.4.5 both fail this test. Looking at the code in test_pty.py, it appears to me that r43570 fixed things for OSF/1 and IRIX which both do weird things with output while breaking things for any other platform by suppressing the \r\n - \n mapping which used to be performed for all platforms. So, for Solaris, that mapping doesn't happen and the actual and expected outputs don't agree. This was probably me. Perhaps a fix wasn't backported? I notice the 2.5 version of the test changed from the 2.4 version and does a str.replace rather than changing the last chars of the string. You can try using the 2.5 version and my guess is it will work (ie, the test will pass). The change is in normalize_output. n ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] r62342 - python/branches/py3k/Objects/bytesobject.c
On Mon, Apr 14, 2008 at 2:56 PM, Christian Heimes [EMAIL PROTECTED] wrote: Good idea! A 3to2 project is going to make backporting io.py and other new stuff less painful and much faster. For the io.py backport I spent most time on removing annotations and replacing with u. What needs to be done? * remove funtion annotation * Add object to all empty class definition * replace class Egg(metaclass=Spam) with class Egg(object):\n__metaclass__ = Spam * Add __future__ imports for print_function and unicode literals Anything else? Iteration with the dict methods (e.g., keys - iterkeys()), map/zip/filter returning iterator rather than list. int - (int, long) str - basestring or (str, unicode) __bool__ - __nonzero__ exec/execfile input - rawinput Most things that have a fixer in 2to3 would also require one in 3to2. Only things that work in backwards compatible ways like apply/has_key removal, etc don't need a 3to2 fixer. Although most of these 3to2 fixers are probably pretty low priority as they are not real likely to be used in the python code base. They are still needed for general user code. n ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Next monthly sprint/bugfix day?
On Wed, Apr 9, 2008 at 7:12 AM, Trent Nelson [EMAIL PROTECTED] wrote: Hi, Is there another online sprint/bugfix day in the pipeline? If not, can there be? ;-) Trent, I think you just volunteered to lead it. :-) We should either do it this weekend Apr 19-20 or wait until after the release. The first available date should be May 10. The schedule http://www.python.org/dev/peps/pep-0361/ has the upcoming May 7 release as the last alpha. That means we are getting closer to an API freeze. Anything that might change an API for 2.6/3.0 should be addressed sooner rather than later. If we have to change an API before release, please update the bug tracker priority to release blocker. n ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Reserving an arg space for Jython
I was also going to suggest a platform independent option. I like -Xwhat-follows-is-impl-dependent. n On Fri, Apr 11, 2008 at 12:39 PM, Dino Viehland [EMAIL PROTECTED] wrote: IronPython already uses -X:OptionName for special IronPython only options so +1 for -X. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Frank Wierzbicki Sent: Friday, April 11, 2008 12:14 PM To: Brett Cannon Cc: Guido van Rossum; python-dev@python.org Subject: Re: [Python-Dev] Reserving an arg space for Jython On Fri, Apr 11, 2008 at 2:51 PM, Brett Cannon [EMAIL PROTECTED] wrote: On Fri, Apr 11, 2008 at 7:40 PM, Guido van Rossum [EMAIL PROTECTED] wrote: Works for me. We should have a patch to CPython that looks for -J and rejects it with -J is reserved for Jython. Do we want it to be Jython-specific, or should it be available to any alternative VM? I don't know if the IronPython folks need anything for .NET, but maybe they would like one? -X was suggested on Jython's irc. I kind of like -J, but -X would work for us too. -Frank ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/dinov%40microsoft.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/nnorwitz%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] generated NEWS files
On Fri, Apr 11, 2008 at 2:53 PM, Martin v. Löwis [EMAIL PROTECTED] wrote: I just tried it, and although it works, I get this output: svn: 'post-revprop-change' hook failed; no error output available Significant? It's the mailer.py hook failing. I'm not quite sure why it fails, and had no time to look it up. It reads #!/bin/sh REPOS=$1 REV=$2 USER=$3 PROPNAME=$4 /data/repos/projects/hooks/mailer.py propchange $@ The script is the same mailer.py that is also invoked in post-commit. Can anybody see a problem with that? No, but how many parameters are passed to mailer.py? if cmd == 'commit': if len(sys.argv) 5: usage() if len(sys.argv) 4: config_fname = sys.argv[4] elif cmd == 'propchange': if len(sys.argv) 6 or len(sys.argv) 7: usage() # ... For commit, sys.argv must be = 5, but for propchange sys.argv must == 6, unless my tired eyes are reading that wrong. I don't know if the # of args passed are the same in both cases or not. n ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] osx x86 buildbot offer
On Fri, Apr 4, 2008 at 11:52 AM, Thomas Heller [EMAIL PROTECTED] wrote: I can offer an OS X x86 machine to run a buildbot on. This is a physical machine, running OS X 10.5 Leopard. Thanks Thomas! Martin and I will coordinate with you off-list. n ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Annotations support in python
The release schedule for 2.6/3.0 is http://www.python.org/dev/peps/pep-0361/ 3.0 will have the feature, 2.6 may or may not. n On Thu, Apr 3, 2008 at 4:47 AM, Heshan Suriyaarachchi [EMAIL PROTECTED] wrote: Hi, I need to work with annotations as it is said in [1]. Currently I am using python 2.5.1. I would like to know whether the next release of python will support this feature. If the next version support this feature I would like to know when are you planning to release it. I used the __future__ module but I could not get the annotations with it. [1] - http://www.python.org/dev/peps/pep-3107/ Thanx Heshan Suriyaarachchi ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/nnorwitz%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] fixing tests on windows
The Windows buildbots are still failing because some tests keep files opened. This causes subsequent tests which use the same file to fail. Here is a recent run which had a failure early on: http://www.python.org/dev/buildbot/stable/x86%20XP-3%20trunk/builds/1209/step-test/0 I'm assuming the first failure (test_bufio) was due to an open file. (Can't tell, no error msg.) That means the problem was in that test or an earlier test. The only earlier tests are: test_import test_sys test_descr test_xdrlib test_urllibnet test_binhex test_strptime test_importhooks test_copy test_hmac test_genericpath test_complex test_timeout test_quopri test_marshal test_zipfile test_mutants test_csv test_ucn test_hash test_wsgiref test_mmap test_ftplib test_pickletools test_codecmaps_cn Of those tests, only some of them reference TESTFN which is the probable file: test_import 22 test_descr 3 test_urllibnet 2 test_binhex 2 test_genericpath 44 test_complex 3 test_marshal 33 test_zipfile 136 test_mutants 7 test_mmap 32 I inspected test_mmap which had been modified somewhat recently and tried to fix a few things. I'm not sure I helped and there are probably more problems lurking. This issue may not be noticed in the default test run. It definitely occurs when the tests are run in a random order as they are on the buildbots. Try running: ./python.exe ./Lib/test/regrtest.py -r That should help provoke more errors. As always, patches are greatly appreciated. Cheers, n ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] refleaks on trunk
test_io is the only leaky test on trunk that I know of. I narrowed
down the leaks to the code below.
It's possible there are other leaks in test_io.
n
--
import sys, gc
import _fileio, io
class FileIO(_fileio._FileIO, io.RawIOBase):
def close(self):
io.RawIOBase.close(self)
def main():
class MyFileIO(FileIO): pass
f = MyFileIO('tt-test', w)
f.close()
for i in range(10):
main()
print(gc.collect())
print(sys.gettotalrefcount())
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] No time for svn merge
On Sun, Mar 30, 2008 at 7:41 PM, Benjamin Peterson [EMAIL PROTECTED] wrote: On Sun, Mar 30, 2008 at 4:54 PM, Martin v. Löwis [EMAIL PROTECTED] wrote: If you'd like, I can merge the rest. If you have the time to figure it all out, sure. I found that quite a tedious task, and had to spent on some patches quite a long time to figure out what they do, and what the 3.x equivalent should be. Ok. I merged some more of the low hanging fruit. Somebody who understands AST better than me should probably do the merges with that. Are you done for today/tonight? If so, I can merge the rest. The last checkin to regrtest I saw looked like it doesn't work. I thought it had print foo without parens. Did I miss something? n ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-3000] the release gods are angry at python
On Fri, Mar 28, 2008 at 3:31 AM, [EMAIL PROTECTED] wrote: Neal Anything that connects to a remote host is definitely flaky. Would it maybe help to set up a dedicated host (or virtual host) to serve as the sole target of all network tests? It would help, but not fix the problem. There will still be transient network issues that come up. n ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-3000] the release gods are angry at python
On Wed, Mar 26, 2008 at 5:52 PM, [EMAIL PROTECTED] wrote: The next releases of 2.6/3.0 are planned for April 2, just over a week from now. There is much work that needs to be done. The buildbots are in a pretty sad state and the gods are seeing too much red. http://www.python.org/dev/buildbot/stable/ http://www.python.org/dev/buildbot/all/ Is there some reason the g4 osx.4 trunk buildbot isn't running? When I checked it this morning it was idle with two pending. I forced a run. I checked later today and it showed five pending. Now it shows 10 pending. It looks like its last run was about 18 hours ago. You would think by now it would have been able to make another run. It pings fine. I logged in to the machine an the buildbot is running. I think what happens is that sometimes the master loses track of the slaves. The fix requires restarting the master, but I didn't want to do that last night while there were still builds going on. I'll try to find a quiet time tonight and restart the master. I expect that will fix the problem. n ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] the release gods are angry at python
The next releases of 2.6/3.0 are planned for April 2, just over a week from now. There is much work that needs to be done. The buildbots are in a pretty sad state and the gods are seeing too much red. http://www.python.org/dev/buildbot/stable/ http://www.python.org/dev/buildbot/all/ See my other mail that discusses the stable buildbots. The criteria for release is that all the stable buildbots are passing all the tests. So we really gotta get these green before Barry notices. You don't want to see Barry angry. You wouldn't like him when he's angry. I propose a code chill for new features. Changes to doc and tests can continue as usual. However, only submit a new feature *after* you fix a broken test first. If we have to get draconian, we can start breaking fingers when you break a test just like we do at work. :-) Specifically tests that need some TLC are: * test_winsound - http://www.python.org/dev/buildbot/stable/x86%20XP-3%20trunk/builds/1166/step-test/0 * test_threading - test_no_refcycle_through_target - http://www.python.org/dev/buildbot/stable/x86%20XP-3%20trunk/builds/1166/step-test/0 * test_socket deadlocks and times out - http://www.python.org/dev/buildbot/stable/x86%20W2k8%20trunk/builds/210/step-test/0 * test_ssl deadlocks and times out - http://www.python.org/dev/buildbot/all/S-390%20Debian%20trunk/builds/255/step-test/0 * test_xmlrpc transient socket errors - http://www.python.org/dev/buildbot/stable/g4%20osx.4%20trunk/builds/3101/step-test/0 * test_mailbox - http://www.python.org/dev/buildbot/stable/x86%20XP-3%203.0/builds/723/step-test/0 * test_asynchat - http://www.python.org/dev/buildbot/all/alpha%20Tru64%205.1%20trunk/builds/2756/step-test/0 Hopefully test_timeout is fixed, but that might be flaky too. There have been other tests that have also been flaky like test_asynchat, test_smtplib, test_ssl, test_urllib2net, test_urllibnet, test_xmlrpc_net and some of the tests that use networking. These all need to be fixed so the tests are 100% reliable and only fail when there is a real error. There are currently no release blocker issues: http://bugs.python.org/issue?%40columns=title%40columns=id%40sort=activitypriority=1%40group=prioritystatus=1%40action=search There are 48 critical issues: http://bugs.python.org/issue?%40columns=title%40columns=id%40sort=activitypriority=2%40group=prioritystatus=1%40action=search If you believe any issue should block the release, set the priority to release blocker and assign it to me (nnorwitz). Many of the critical issues are those that require 2to3 fixers. These can be checked in as they are written. Be sure to test them thoroughly and try to think of all the conditions that could possibly cause the fixer to fail or do the wrong thing. Right now, I don't know of any reason to hold up the release other than the failing tests. n ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] stable buildbots
We need to get the tests for Python to be more stable so we can push out solid releases. In order to achieve this result, we need tests that are *100% reliable* and fail _only when there is a problem with Python_. While we aren't nearly as close to that goal as we need to be, we have to work towards it. The buildbots that have been more reliable are separated onto their own page: http://www.python.org/dev/buildbot/stable/ This is the page that should be tracked by most people. This is the page we will use to determine if Python is ready for release. All green means the release is ready to ship. This page should *always* show all green. Any change that causes any buildbot to fail, should be immediately reverted. When you commit code, make sure to refresh the stable buildbot page to ensure you haven't broken anything. The stable buildbots need to represent all major platforms, at least Windows, Mac OSX, and Linux. All of those are currently represented on the page along with several others. Unfortunately some of the tests, particularly on Windows, are not passing. We need to get these in better shape. Please help out. See my other mail for details or use the link above to find the current set of problems. As we fix more tests and more platforms become stable, I will add them to the stable page. This requires restarting the server, so I plan to do this during quiet times when all the buildbot slaves are idle. n ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] stable buildbots
On Wed, Mar 26, 2008 at 9:04 AM, Barry Warsaw [EMAIL PROTECTED] wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Mar 26, 2008, at 8:14 AM, Facundo Batista wrote: 2008/3/26, Neal Norwitz [EMAIL PROTECTED]: We need to get the tests for Python to be more stable so we can push out solid releases. In order to achieve this result, we need tests that are *100% reliable* and fail _only when there is a problem with +1 Python_. While we aren't nearly as close to that goal as we need to be, we have to work towards it. The buildbots that have been more reliable are separated onto their own page: http://www.python.org/dev/buildbot/stable/ This is for trunk or 3k? The page has stable buildbots for trunk, 2.5, and 3.0. I'm thinking that we should remove the 2.5 buildbots from the stable page. Neal, if you agree, can you do that? I'm fine with removing 2.5, but I'm not sure if it's easy. HINT HINT. If everyone else fixes the broken and flaky tests, I'll have more than enough time to fix this. n ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Py3k and asyncore/asynchat
On Thu, Feb 14, 2008 at 10:09 AM, Giampaolo Rodola' [EMAIL PROTECTED] wrote: On 14 Feb, 16:36, Giampaolo Rodola' [EMAIL PROTECTED] wrote: Ok, I'll try to take a look at all asyncore/chat reports and try to summarize them by splitting patches which solve bugs and patches which add enhancements or functionalities. === Patches for existing issues === - 1736190 which includes fixes for the following issues among other improvements: - 1063924 (asyncore should handle ECONNRESET in send()) - 1736101 (asyncore should handle ECONNABORTED in recv()) - 760475 (handle_error() should call handle_close() instead of close()) - 1740572 (refill_buffer() should call handle_close() rather than close()) - 777588 (wrong connection refused behavior on Windows) - 889153 (wrong connect() behavior) - 953599 (asyncore misses socket closes when poll is used) - 1025525 (asyncore.file_dispatcher should not take fd as argument) - 1519 (async_chat.__init__() and asyncore.dispatcher.__init__ parameters inconsistency) - 1541 (Bad OOB data management when using asyncore with select.poll()) - 2073 (asynchat push always sends 512 bytes (ignoring ac_out_buffer_size)) === Open issues with no patches (need review) === - 658749 (asyncore connect() and winsock errors) - 1161031 (neverending warnings from asyncore) === Enhancements new features === - 1641 (add delayed calls feature) - 1563 (conversion to py3k and some other changes) That's a lot of patches. My immediate concern is that test_asynchat is very flaky and fails often. Sometimes it causes other tests to fail. Is there a patch that addresses this? If you need examples, just look through the buildbot mails that mention test_asynchat in: http://mail.python.org/pipermail/python-checkins/ Some platforms have more problems than others, but almost all platforms have failed test_asynchat at one point or another. Please break up the patches into 2 sets and prioritize the patches with the set. Set #1: Patches that have a test and doc updates if required Set #2: Patches that don't have a test or doc updates as required For the patches that fall into Set #1, list them in priority order. Top priority would be a problem that fixes failures seen in the buildbots. Next priority would go to the patches that solve more serious problems. Post the results here. I will try to look at them. For every patch you list, make sure that it conforms to the proper style (e.g, PEP 8) and is essentially perfect and ready for inclusion. This means that there is a single file to download that contains all the modifications. The changes are appropriately commented, lines are less than 80 characters, etc. One of the modifications should be an entry in Misc/NEWS. n ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] State of N-dimensional array interface
Hi Mike. Travis is the best person to discuss the status of the buffer APIs. Cheers, n On Sat, Mar 22, 2008 at 2:50 PM, Mike Sullivan [EMAIL PROTECTED] wrote: What is the current state of the N-D Array Interface proposal (http://numpy.scipy.org/array_interface.shtml). Some work was done on this in an earlier Summer of Code, but it seems to never have been included. Does anybody know what state that work is in and what prevented it's inclusion? (I am interested in completing this as a Summer of Code project. I posted about this on the SOC list, and received a suggestion to try asking python-dev.) -- Mike Sullivan ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/nnorwitz%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] First green Windows x64 buildbots!
Great work Trent! You'll need to take a picture of Martin buying you the beer once you get the rest green. :-) n On Wed, Mar 19, 2008 at 5:57 PM, Trent Nelson [EMAIL PROTECTED] wrote: We've just experienced our first 2.6 green x64 Windows builds on the build slaves! Well, almost green. Thomas's 'amd64 XP trunk' ran out of disk: 304 tests OK. 1 test failed: test_largefile == ERROR: test_seek (test.test_largefile.TestCase) -- Traceback (most recent call last): File C:\buildbot\trunk.heller-windows-amd64\build\lib\test\test_largefile.py, line 42, in test_seek f.flush() IOError: [Errno 28] No space left on device Sorry about that Thomas ;-) Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/nnorwitz%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] trunk buildbot status
Unfortunately, I don't have ssh access from my hotel room. This means I won't be able to help much until I get home. On Wed, Mar 19, 2008 at 7:26 PM, Trent Nelson [EMAIL PROTECTED] wrote: Quick update on the status of the trunk buildbots: Failing: [x86 gentoo trunk (Neal Norwitz)] This has been failing at the same point for the past couple of days now: test_sqlite command timed out: 1800 seconds without output, killing pid 15168 process killed by signal 9 program finished with exit code -1 None of the other buildbots seem to be encountering the same problem. Neal, got any idea what's going on with this one? Last status was here: http://mail.python.org/pipermail/python-checkins/2008-March/066824.html I haven't logged in to check what's going on. Gerhard had some ideas in the same thread: http://mail.python.org/pipermail/python-checkins/2008-March/066863.html I just need to have some time on the machine and look into the problem. If I determine the problem is with the underlying sqlite, I'll try to upgrade it. [alpha True64 5.1 trunk (Neal Norwitz)] [hppa Ubuntu trunk (Matthias Klose)] I can probably diagnose both of these too when I get back from Chicago. Neal/Martin, I'd like to promote the following slaves to the stable list: [g4 osx.4] [x86 W2k8] [AMD64 W2k8] [ppc Debian unstable] [sparc Ubuntu] [sparc Debian] [PPC64 Debian] [S-390 Debian] [x86 XP-3] [amd64 XP] [x86 FreeBSD] [x86 FreeBSD 3] The trunk builds of these slaves have been the most reliable since I've been tracking. Most of these have already been promoted to stable. I just didn't restart the buildbot master since making the change. It requires a restart, not just a reconfig. I was waiting for a quiet time when the bots weren't busy, but that hasn't happened yet. :-) I can make more changes and restart when I get home. n ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] adding json to stdlib (was: Re: PEP 361: Python 2.6/3.0 release schedule)
On Tue, Mar 18, 2008 at 11:17 AM, John Millikin [EMAIL PROTECTED] wrote: Possible features for 2.6 New modules in the standard library: - JSON implementation Have there been any plans made for which one? All of the No. This was something I added as a nice to have for 2.6. implementations I'm aware of (cjson, simplejson, jsonlib, demjson, python-json) have serious issues that in my opinion should be fixed before inclusion in the standard library[1]. I am the author of jsonlib, and am willing to write patches for whichever implementation becomes the standard, but it would be very nice to know what to focus on. I'm not familiar enough with any of the libraries to comment. If it's premature to add a particular implementation, that's fine, we can wait. Apologies if this has been discussed already, but there was no link to a PEP in 361 and a search of python-dev and c.l.p returned no relevant results. I don't believe it has been discussed before. I've changed the subject and would like to discuss this now. It would be great if you could pull together and get the community behind a single solution that is robust enough to include in the stdlib. If that an be finished for 2.6, great. If it waits for 2.7, that would still be fine. n [1] * cjson and python-json have almost complete absense of Unicode support. * simplejson is slow and writes incorrect unicode escapes. * demjson is far too forgiving when parsing and accepts all sorts of invalid input. * jsonlib is not PEP 8-compliant and has terrible error handling. * python-json is abandoned, slow, and lacks Unicode support. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/nnorwitz%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Change in priority fields
On Tue, Mar 18, 2008 at 10:51 AM, Guido van Rossum [EMAIL PROTECTED] wrote: The key goal here (well, mine in any case :-) is to manage developers, not to get releases out at all cost. Even though the release schedule is set in stone, I think we should send out a variety of reminders ahead of each scheduled release, and these reminders must come from a human, not from a bot. ... Maybe the reminder mail could include an exhortation to create new showstopper issues for anything that a developer feels should really be addressed before the release, even if it's not thought of a bug. The release reminder emails act as a kind of wake-up call to many developers. I think I did this for 2.5 and plan to do improve communications for 2.6. I'll need to work the details out with Barry, but it is my intention to communicate as much as possible. The next release (in two weeks) will probably be a little haphazard as I try to get up to date after pycon. I'll try to get more organized so all subsequent releases go smoothly. Suggestions to python-dev on how to improve the process are always encouraged. n ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] changing regrtest to handle import errors
[changing to: and subject: ]
On Tue, Mar 18, 2008 at 4:09 PM, Guido van Rossum [EMAIL PROTECTED] wrote:
On Tue, Mar 18, 2008 at 3:58 PM, neal.norwitz
[EMAIL PROTECTED] wrote:
Get this test to pass (UserList/UserDict no longer exist and caused a
skip).
I think the automatic skip on ImportError is harmful.
We should add a helper function to test_support so that you can write
foobar = test_support.import_optional('foobar')
and it will skip the test if foobar cannot be imported; all other
failing imports should cause the test to *fail*.
Any takers? This should be an easy two-part task.
This would be a great starter project for a new developer.
http://bugs.python.org/issue2409
Let me know if you could use some help. Feel free to contact me on or off list.
n
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pre-checkin test suggestion for python repository
On Tue, Mar 18, 2008 at 4:43 PM, Gregory P. Smith [EMAIL PROTECTED] wrote: The tabs/spaces checker that is run before doing a svn ci on the python repository spits out an error message about which files have problems. Could someone please update this error message to say something to the effect of run Tools/scripts/reindent.py on every file listed above and rerun your tests to fix this before checking in Done. n ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python 3000: Special type for object attributes map keys
On Wed, Mar 5, 2008 at 4:27 PM, Henrik Vendelbo [EMAIL PROTECTED] wrote:
It appears to me that if you can make mapping mechanisms faster in
Python you can make significant
overall speed improvements. I also think the proposed concept could
add flexibility to persistence formats
and RMI interfaces.
My basic idea is to have a constant string type with an interpreter
globally unique hash. If the original constant
is created in a manner different from string constants, it can be
tracked and handled differently by the interpreter.
Part of this is done, but very differently in that all strings used in
code objects are interned (stored in a dictionary so we don't increase
memory by storing multiple string objects which contain the same
string) . So there is partially a mechanism to do what you suggest.
But there would be many places that would need to be modified.
I think we briefly discussed this in the past.
P.S. I originally thought of this in a JavaScript context so forgive
me if this would make little difference in Python.
Your message was a little confusing at first because the terminology
is a little different, but the idea makes sense. It's not clear how
much this would speed up the interpreter. The best way to test your
theory would be to create a patch and measure the performance
difference.
First, you should measure the current speed difference. Something like:
$ ./python.exe -m timeit -s 'd = {1: None}' 'd[1]'
100 loops, best of 3: 0.793 usec per loop
$ ./python.exe -m timeit -s 'd = {1: None}' 'd[1]'
100 loops, best of 3: 0.728 usec per loop
My python is a debug version, so a release version might be faster for
ints. If not, the first task would be to speed up int lookups. :-)
(You should verify more with real world dict sizes.) It is possible
to optimize dicts with int keys since string keys are specialized in
dicts, but ints are not. You would need to look in
Objects/dictobject.c. See http://python.org/dev/faq/ for general
hints on how to get started.
If ints were faster, some of the next steps would be:
* keep the globally unique number (very easy)
* update the source that generates byte codes to use the globally unique number
* store ints in dicts and update all the places for how they use attributes
* update byte code when a module is imported to use the globally unique number
Feel free to ask questions.
Cheers,
n
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PyErr_Warn or PyErr_WarnEx
On Tue, Mar 18, 2008 at 10:07 PM, Benjamin Peterson [EMAIL PROTECTED] wrote: Now that we're aggressively adding Py3k warnings to the trunk, I think it's a good time to get this straightened out. The docs [1] say PyErr_Warn is deprecated in favor of PyErr_WarnEx. However, I have seen both in recent checkins. What is preferred? PyErr_WarnEx should be used. PyErr_Warn is just (see from Include/pyerrors.h): #define PyErr_Warn(category, msg) PyErr_WarnEx(category, msg, 1) If someone wants to remove the macro in 3k, go for it. There are many of these compatibility macros and stub functions left around for binary compatibility. We should try to get rid of those in 3k and shrink the API. n ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Change in priority fields
On Mon, Mar 17, 2008 at 9:10 PM, Brett Cannon [EMAIL PROTECTED] wrote: Barry, Neal, and myself had a conversation and changed the priority fields in the tracker. You can click on 'priority' to see an explanation, but the new fields are: - release blocker - critical - high - normal - low So release blocker blocks a release. Critical could very easily block a release, but not the current one. High issues should be addressed, but won't block anything. Normal is normal. And low is for spelling errors and such. Primarily everyone should use normal for issues that are, uh, normal. Critical should be used for bugs that are things like: crashing the interpreter, serious memory/reference leaks. High could be used for large problems with resource usage (too much memory) or something that is otherwise, important. n ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] 3.0 buildbots all red
On Sun, Mar 16, 2008 at 12:07 PM, Martin v. Löwis [EMAIL PROTECTED] wrote:
New sprint idea: getting all (inc. trunk) the buildbots green by
Thursday. Anyone interested?
I think the chance to achieve that is close to zero.
Sounds like a challenge if ever I've heard one -- care to wager a
beer on it? (Only applies to buildbots that are connected/online.)
Even with that restriction: I'll happily buy you a beer if you
manage to get all the online ones pass all tests consistently.
I think this is possible, though considerable work. Probably the
biggest win will be creating a mock for socket and using mock sockets
in the tests for asyn{core,chat}, smtplib, xmlrpc, etc. That will fix
about 75% of the problems on 2.6. The remaining problems are:
* test_asyn{chat,core} might not be meaningful with mock sockets and are flaky
* the alpha fails test_signal/socket for weird alarm conditions.
this might be hard to debug/fix (I have access to this box though)
* test_sqlite is broken on x86 with an old sqlite (I have access to this box)
* test_bsddb may be flaky, I'm not sure
* probably a few platform specific problems
3.0 will get most of the improvements as the fixes are ported. 3.0
has a different problem on the x86 box dealing with signals. There
are probably some other 3.0 specific problems. Although, using a mock
socket could address this too. I can help on fixing these issues
during the sprints. It will be happy to work with Trent to try to fix
the problems. I'm sure we can greatly improve the situation.
n
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] 3.0 buildbots all red
On Sun, Mar 16, 2008 at 5:13 PM, Mark Dickinson [EMAIL PROTECTED] wrote:
On Sun, Mar 16, 2008 at 1:32 PM, Neal Norwitz [EMAIL PROTECTED] wrote:
I think this is possible, though considerable work. Probably the
biggest win will be creating a mock for socket and using mock sockets
in the tests for asyn{core,chat}, smtplib, xmlrpc, etc. That will fix
about 75% of the problems on 2.6. The remaining problems are:
* test_asyn{chat,core} might not be meaningful with mock sockets and are
flaky
* the alpha fails test_signal/socket for weird alarm conditions.
this might be hard to debug/fix (I have access to this box though)
* test_sqlite is broken on x86 with an old sqlite (I have access to this
box)
* test_bsddb may be flaky, I'm not sure
* probably a few platform specific problems
test_tokenize is also currently (sometimes) failing on many of the bots.
I've been looking into it, but I'm struggling to find the problem. The
traceback e.g. for the amd64 gentoo buildbot ends with
File /home/buildbot/slave/py-build/3.0.norwitz-amd64/build/Lib/io.py,
line 1081, in decode
output = self.decoder.decode(input, final=final)
File /home/buildbot/slave/py-build/3.0.norwitz-amd64/build/Lib/codecs.py,
line 291, in decode
(result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf8' codec can't decode bytes in position
12-15: invalid data
On my own machine (SuSE 9.3/i686) I'm seeing this test pass about 80%
of the time and fail the other 20% with something like the above,
the position of the reported invalid data changing from run to run. It
looks like data are getting corrupted somewhere along the line.
Anyone have any ideas?
Yeah, sounds like a memory issue. Did you try running with valgrind
or purify? I haven't done so for a long time, perhaps never on 3k
branch. It would be a good thing to run a tool soon.
n
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] difference between diff string implementations
This inconsistency goes back to 2.3 at least and probably to the
initial unicode implementation.
set(dir(u'')) - set(dir(''))
['isnumeric', 'isdecimal']
UserString contains these two methods even though 8-bit strings do
not. I'm not sure what we should do for 2.6 or 3.0. My preference
would be to remove these methods on unicode/UserString if they aren't
useful to a large audience. However, removing for 2.6 without a
deprecation seems bad.
Suggestions?
n
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Patch for trunk test_winsound.py (fixes my buildbot)
On Wed, Mar 5, 2008 at 2:18 AM, Nick Coghlan [EMAIL PROTECTED] wrote: Nick Coghlan wrote: While the patches are appreciated, please submit them to the tracker at bugs.python.org rather than mailing them directly to this list. This comment doesn't apply to your recent posts - looks like those have all been checked in already ;) Just to follow up on what Nick said, you should submit patches to the tracker. While it's usually pretty easy to submit really small things like this to the mailing list, it has two negative effects: 1) clutters the mailing list with trivial things many people may not care about, 2) it can easily be lost/forgotten. Typically the only reason we post an actual patch on python-dev is to spawn wider discussion. Generally, patches should be posted as links to the tracker rather than inline. But keep the patches coming! Cheers, n ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Windows buildbot test_bsddb3 problems (was RE: Buildbots for trunk are all red)
Trent, thanks for working on the buildbot. I fixed the first case you mentioned in r61233 wrt removing the directory before closing the file. It would be great if you could submit a patch when you are able to fix the remaining problems. Cheers, n On Tue, Mar 4, 2008 at 6:58 AM, Trent Nelson [EMAIL PROTECTED] wrote: Spent some time on my buildbot (x86 2k8 trunk) this morning trying to track down why test_bsddb3 is failing (trunk with db-4.4.20). The first test that fails is this: test01_GetsAndPuts (bsddb.test.test_basics.BasicBTreeWithEnvTestCase) ... ERROR That's slightly misleading though as the test runs fine -- the actual exception is being thrown in test_basics.BasicTestCase.tearDown() when os.remove() is called against the first db file (i.e. '__db.001'): WindowsError: [Error 5] Access is denied: 'c:\users\trent~1.nel\appdata\local\temp\2\db_home2808\__db.001 This isn't surprising, given that the python_d.exe process still seems to have __db.001, __db.002 and __db.003 open at the time os.remove() is called. The aforementioned tearDown() method looks like this: def tearDown(self): self.d.close() if self.env is not None: test_support.rmtree(self.homeDir) self.env.close() ## Make a new DBEnv to remove the env files from the home dir. ## (It can't be done while the env is open, nor after it has been ## closed, so we make a new one to do it.) #e = db.DBEnv() #e.remove(self.homeDir) #os.remove(os.path.join(self.homeDir, self.filename)) else: os.remove(self.filename) If I switch the order of statements such that self.env.close() is called before test_suppot.rmtree(self.homeDir), this particular test and a host of others that were also failing now pass (a runtime abort is no longer raised by the CRT half way into the tests either). (Note that the order was switched to that shown above by Neal in r61052 on Feb 24th, which is when these issues started occurring.) That said, there are still a lot more test failures down the track once this change has been made, either via the access denied WindowsError, or a DBInvalidArgError, e.g.: ERROR: test02_WithSource (bsddb.test.test_recno.SimpleRecnoTestCase) -- Traceback (most recent call last): File S:\src\svn\svn.python.org\projects\python\trunk\lib\bsddb\test\test_recno.py, line 33, in tearDown test_support.rmtree(self.homeDir) File S:\src\svn\svn.python.org\projects\python\trunk\lib\test\test_support.py, line 70, in rmtree shutil.rmtree(path) File S:\src\svn\svn.python.org\projects\python\trunk\lib\shutil.py, line 184, in rmtree onerror(os.remove, fullname, sys.exc_info()) File S:\src\svn\svn.python.org\projects\python\trunk\lib\shutil.py, line 182, in rmtree os.remove(fullname) WindowsError: [Error 5] Access is denied: 'c:\\users\\trent~1.nel\\appdata\\local\\temp\\2\\db_home4656\\tmp04_knk' == ERROR: test01_1WriterMultiReaders (bsddb.test.test_thread.BTreeConcurrentDataStore) -- Traceback (most recent call last): File S:\src\svn\svn.python.org\projects\python\trunk\lib\bsddb\test\test_thread.py, line 62, in setUp self.env.open(homeDir, self.envflags | db.DB_CREATE) DBInvalidArgError: (22, 'Invalid argument -- configured environment flags incompatible with existing environment') The DBInvalidArgError exception is only raised after a previous WindowsError is encountered, so I assume it's a side-effect of the tearDown() method not cleaning the environment correctly. It seems this comment in tearDown() is quite pertinent to our situation: ## Make a new DBEnv to remove the env files from the home dir. ## (It can't be done while the env is open, nor after it has been ## closed, so we make a new one to do it.) #e = db.DBEnv() #e.remove(self.homeDir) #os.remove(os.path.join(self.homeDir, self.filename)) Not sure why this is commented out -- quick search of svn history indicates it's been like that for at least the last year and a half. Will have some more time this evening to spend on this, however, work calls at the moment. Regards, Trent. From: [EMAIL PROTECTED] [EMAIL PROTECTED] On Behalf Of Facundo Batista [EMAIL PROTECTED] Sent: 26 February 2008 06:22 To: Thomas Hervé Cc: python-dev@python.org Subject: Re: [Python-Dev] Buildbots for trunk are all red 2008/2/25, Thomas Hervé [EMAIL PROTECTED]: I've worked on that problem during the bug day. I've open a ticket with a patch at http://bugs.python.org/issue2168.
Re: [Python-Dev] BSDDB3 (was: Re: Buildbots for trunk are all red)
On Tue, Mar 4, 2008 at 3:46 PM, Jesus Cea [EMAIL PROTECTED] wrote: That said, it is my aim to keep bsddb in stdlib, providing a stable and featureful module. I think keeping bsddb development inside python svn is not appropiate. Currently (I could change idea), my approach will be keeping pybssdb as a separate project and sync with python SVN from time to time. Mainly to take advantage of buildbot architecture and, of course, to be able to release python with current bindings. Since I have no python commit access, this seems a sensible approach. And I could do frequent pybssdb releases (let say, every couple of months) without waiting for a full python release (current approach). I think that approach is fine. Hopefully you can keep the changes reasonably small (preferably less than 500 lines per change). That will ensure more people will review your changes. Cheers, n ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Buildbots for trunk are all red
On Tue, Feb 26, 2008 at 11:47 PM, Brett Cannon [EMAIL PROTECTED] wrote: Well, in my opinion batteries included is great, but not when one of the batteries consistently acts up and requires a good shake to get working again. The bsddb module has consistent reliability issues when it comes to testing (and I suspect it has more to do with Sleepycat than the bindings). I know I am tired of getting buildbot errors saying that the bsddb tests died more consistently than most tests over their history. I agree that bsddb has been a pain. It's about 1 of 10 tests that fill that category. I've been working on reducing these problems (recently: test_bsddb3, test_smptlib, test_xmlrpclib, and I'm sure there are others I forgot). Rather than remove modules, it would be more productive if we fixed the flaky tests. Then we wouldn't have to ignore failures, we could trust the buildbots. test_urllib*net tests still fail regularly, I think because some hosts aren't available from time to time. Can someone look into making test_urllib*net more robust? We also need to make the tests more robust. By fixing test_smtplib, I sped it up by over 99% while making it more robust. Any test that uses threads and sleeps (really just sleeps) needs to be fixed similarly. Can someone find which tests still use sleep? n ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] February bug day outcome
On Tue, Feb 26, 2008 at 12:07 AM, Jeroen Ruigrok van der Werven [EMAIL PROTECTED] wrote: -On [20080226 08:09], Neal Norwitz ([EMAIL PROTECTED]) wrote: I agree with the theory. However, we have only a single BSD box currently working and it has been flaky. Primarily test_smtplib has been failing sporadically on it. For example: What are the requirements for a build box? I have both a a 6.3-STABLE AMD No requirements in particular. See http://wiki.python.org/moin/BuildBot Pretty much it should have good network connectivity and someone we can contact who can administer the box. For example, in the past we have had to have ports opened up so the tests can pass. Possibly we might want to get different version of libraries installed to test with (e.g., berkeley db v 4.x). Athlon XP 2200+ and an FreeBSD 7-PRERELEASE (STABLE once released) amd64 box I can let compile. The 6.3 box can do it (almost) continuously. I can probably add an Intel Pentium 4 6.3-STABLE box as well. These boxes are fine and faster than half the current machines. I can probably get NetBSD build bots up as well, just need to ask some people for appropriate access. It would be best if we get configurations we don't already have. We have no NetBSD boxes. We have one FreeBSD (6.2 supposedly), although it's 32-bit. n ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python 2.6 and 3.0 ...and applink.c?
On Mon, Feb 25, 2008 at 2:20 PM, Bugbee, Larry [EMAIL PROTECTED] wrote: Hi Barry, A question Do you know if OpenSSL's applink.c will be included in the Windows builds? If so, and I hope it is, great! If not, I'd like to encourage its inclusion. The best way to encourage its inclusion is with a patch. :-) n ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] February bug day outcome
On Mon, Feb 25, 2008 at 12:53 PM, Barry Warsaw [EMAIL PROTECTED] wrote: That's a really good idea. At least for the alpha's I would like to have a policy that the monthly release goes out unless 1) There are critical bugs open for 2.6 and/or 3.0 2) The important buildbots are red (maybe) In either case, it should probably be a priority for bug day to repair those failures. On #2, clearly we can't wait for greens across the board, so which platforms are important enough to hold up a monthly release? I'd say something representative of the source release and each of the binary releases we make, so maybe: 2.6: source = x86 gentoo trunk, windows = x86 xp-4 trunk, mac = g4 osx. 4 trunk 3.0: source = x86 gentoo 3.0, windows = x86 xp-4 3.0, mac g4 osx.4 3.0 Unfortunately, basing release status on buildbot health doesn't look very encouraging. :( It's been pretty bad the last month or so. Although it's getting better now. I would recommend these are the golden bots based on what have traditionally been fairly stable help expose errors: sparc solaris10 amd64 gentoo (this is really an ubuntu 6.10 box) x86 gentoo (*) g4 os x (this one has svn problems from time to time which is odd given that it is the only box colocated with the svn server) some win xp box (#4 wfm, i think that has been the most stable recently) ia64 ubuntu ppc debian (this may be ubuntu also) ppc64 debian (ditto) The biggest challenge will be having svn work on all the machines. The tests are getting more stable. I worked on many of them. There are still issues from time to time, but at this point I think more are caused by bad checkins. Sometimes these machines go away, if they are unavailable at time of release, so be it. If we can get more people watching the buildbots and ping those responsible for a failure, we can keep the red to a minimum. If we can fix the ~5 flaky tests, we will be in good shape. If we can fix the svn problems, we'll be in great shape. Nearly all of the flaky tests are due to networking problems. Sometimes transient conditions like the host is unavailable, others due to races. (*) x86 gentoo should not be used for 3.0. There is a problem with signal 32 that causes it to rarely work. I don't know the cause or how to fix this. n ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] February bug day outcome
On Mon, Feb 25, 2008 at 10:35 PM, Jeroen Ruigrok van der Werven [EMAIL PROTECTED] wrote: -On [20080226 06:05], Neal Norwitz ([EMAIL PROTECTED]) wrote: sparc solaris10 amd64 gentoo (this is really an ubuntu 6.10 box) x86 gentoo (*) g4 os x (this one has svn problems from time to time which is odd given that it is the only box colocated with the svn server) some win xp box (#4 wfm, i think that has been the most stable recently) ia64 ubuntu ppc debian (this may be ubuntu also) ppc64 debian (ditto) Might make sense to add at least one BSD box to the mix, generally when 1 BSD build works it should be quite similar on the rest. The above is a bit heavy on the Linux (glibc) side and will not really expose problems on that front aside from the commercial Solaris and OS X. I agree with the theory. However, we have only a single BSD box currently working and it has been flaky. Primarily test_smtplib has been failing sporadically on it. For example: test test_smtplib failed -- Traceback (most recent call last): File /usr/home/db3l/buildarea/trunk.bolen-freebsd/build/Lib/test/test_smtplib.py, line 375, in testBasic smtp = smtplib.SMTP(HOST, PORT, local_hostname='localhost', timeout=3) File /usr/home/db3l/buildarea/trunk.bolen-freebsd/build/Lib/smtplib.py, line 237, in __init__ (code, msg) = self.connect(host, port) File /usr/home/db3l/buildarea/trunk.bolen-freebsd/build/Lib/smtplib.py, line 294, in connect (code, msg) = self.getreply() File /usr/home/db3l/buildarea/trunk.bolen-freebsd/build/Lib/smtplib.py, line 335, in getreply line = self.file.readline() File /usr/home/db3l/buildarea/trunk.bolen-freebsd/build/Lib/socket.py, line 369, in readline data = self._sock.recv(self._rbufsize) timeout: timed out Ugh. I just looked at the test and see all the sleeps. I'll speed this test up by using events which will also hopefully reduce the flakiness. It currently takes 28 seconds. That should be decreased significantly. n ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] optimizing out local variables
Short description (see http://bugs.python.org/issue2181 for the patch and more details): Optimize code like: x = any_expression return x to: return any_expression The local variable x is no longer set before returning. Is this appropriate for .pyc generation or should it only be done for .pyo files? n ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] future_builtins (was: Backporting PEP 3127 to trunk)
On Sun, Feb 24, 2008 at 6:57 PM, Guido van Rossum [EMAIL PROTECTED] wrote: On Sat, Feb 23, 2008 at 5:59 PM, Neal Norwitz [EMAIL PROTECTED] wrote: On Sat, Feb 23, 2008 at 8:06 AM, Guido van Rossum [EMAIL PROTECTED] wrote: I do think map() and filter() should issue a warning under -3 when the first arg is None. (Or does 2to3 detect this now?) What's wrong with filter(None, seq)? That currently works in 3k: filter(None, range(5)) itertools.ifilter object at 0x2b5be60da450 [x for x in _] [1, 2, 3, 4] But that's a bug -- it's been spec'ed that this will stop working. (Can't remember where, perhaps PEP 3100?) I looked in 3100 and didn't see it. (Side note, shouldn't we change the names for filter/map?) Huh? What? Why? The function name returned by repr: itertools.ifilter. n ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] February bug day outcome
On Sun, Feb 24, 2008 at 6:52 PM, A.M. Kuchling [EMAIL PROTECTED] wrote: Yesterday's bug day was another success, closing 48 issues. Several committers were there: Facundo Bastista, Georg Brandl, and Christian Heimes. Facundo organized a local group of participants, and we committed a number of contributions from new people. Should we have one next month? The PyCon sprint will fall on Monday through Thursday, and few people not at PyCon will be available during the work week. OTOH, if we scheduled a bug day for the 29th, that's two weeks after the conference, and we may have recovered from our PyCon burnout at that point. What do people think? I'd rather push it out to mid-month assuming Barry starts releasing alphas at the end of each month. That should provide some time to stabalize. Any one see the buildbots recently? :-( http://www.python.org/dev/buildbot/all/ n ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-3000] Python 2.6 and 3.0
+1 to everything -- n On Fri, Feb 22, 2008 at 3:45 PM, Barry Warsaw [EMAIL PROTECTED] wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hi everyone, I've volunteered to be the release manager for Python 2.6 and 3.0. It's been several years since I've RM'd a Python release, and I'm happy to do it again (he says while the medication is still working :). I would like to get the next alpha releases of both versions out before Pycon, so I propose next Friday, February 29 for both. Guido reminded me that we released Python 1.6 and 2.0 together and it makes sense to both of us to do the same for Python 2.6 and 3.0. I don't think it will be that much more work (for me at least :) to release them in lockstep, so I think we should try it. I won't try to sync their pre-release version numbers except at the milestones (e.g. first beta, release candidates, final releases). I propose to change PEP 361 to outline the release schedule for both Python 2.6 and 3.0. I'm hoping we can work out a more definite schedule at Pycon, but for now I want to at least describe the lockstep release schedule and the Feb 29 date. I'd also like for us to consider doing regular monthly releases. Several other FLOSS projects I'm involved with are doing this to very good success. The nice thing is that everyone knows well in advance when the next release is going to happen, and so all developers and users know what to expect and what is needed from them. I'd like to propose that we do a joint release the last Friday of every month. For the alphas, it's basically what's in svn. This gives us some time to experiment with the process out and see if we like it enough to keep it going through the betas and final releases. Comments? - -Barry -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.8 (Darwin) iQCVAwUBR79ek3EjvBPtnXfVAQJrmQP+KAzy0lSzake2BZsVxErD0kpFQJM+Iij0 qN86wjH0NoqARMfYKVA6eUzEY8vZPu7sJl+SjCOmhnE/KP+l/ArOdis5smiSKwQI klL4XKd/qdorTMqQ9mWgA0DeBb0Asvln9y64nxzNqgve+36q9j6ytPuRey1GjSI5 nVWoJDb3WsM= =4SRa -END PGP SIGNATURE- ___ Python-3000 mailing list [EMAIL PROTECTED] http://mail.python.org/mailman/listinfo/python-3000 Unsubscribe: http://mail.python.org/mailman/options/python-3000/nnorwitz%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] future_builtins (was: Backporting PEP 3127 to trunk)
On Sat, Feb 23, 2008 at 8:06 AM, Guido van Rossum [EMAIL PROTECTED] wrote: I do think map() and filter() should issue a warning under -3 when the first arg is None. (Or does 2to3 detect this now?) What's wrong with filter(None, seq)? That currently works in 3k: filter(None, range(5)) itertools.ifilter object at 0x2b5be60da450 [x for x in _] [1, 2, 3, 4] (Side note, shouldn't we change the names for filter/map?) n ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] boilerplate.tex
On Fri, Feb 22, 2008 at 11:15 AM, Georg Brandl [EMAIL PROTECTED] wrote: [EMAIL PROTECTED] schrieb: This message has been popping up in the buildbot mails for several days: Conflict detected in commontex/boilerplate.tex. Doc build skipped. I have no idea what it means. I don't see it within the core distribution. Can someone take a look at this? Neal will have to fix it locally in his checkout. The message is generated by his automatic doc building script. Yeah, I fixed it. I have an outstanding change that is set to today so the docs show the correct date. But when I release is made, a conflict arises, so I have to fix manually each time the file is modified. n ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Backporting PEP 3127 to trunk
On Thu, Feb 21, 2008 at 2:26 PM, Eric Smith [EMAIL PROTECTED] wrote: I'm going to work on backporting PEP 3127, specifically the hex, oct(), and bin() builtins. I have bin() completed, and I'll check it in shortly. oct() will require a future import. Does anyone have any pointers for implementing this? I understand (and have completed) adding the future import, but what I don't understand is how to modify the behavior of oct() for only the module where the future import is execute. Any rough ideas or pointers to existing code that does something similar would be helpful. See co_flags in PyCodeObject in Include/code.h. When you are compiling the code objects, you have access to the future flags. These (can) get baked into the code objects when they are created. You will need to make a new CO_* macro (0x1 is the next available after CO_FUTURE_WITH_STATEMENT). In the past future imports have typically affected the parser or semantics of the VM (e.g., future division). In your case, you need something different. Thomas Wouters had a somewhat similar problem when changing dict methods. In his case though, he output different op codes for the interpreter to execute to call the proper methods (*). You could use a similar trick here. However, if you were doing that, it begs the question of why not do as Guido suggests and just replace the builtins. If you only stored the info in the co_flags of code objects, the only way I know of would be to access the callers frame and get its co_flags. This is yucky. For example, what if oct() was called from C code? I think Guido's suggestion makes the most sense. My description above is just so people know what needs to be done, not a recommendation that it ought to be done. n (*) - e.g. STORE_VIEWATTR in http://svn.python.org/projects/python/branches/twouters-dictviews-backport/Python/ceval.c http://svn.python.org/projects/python/branches/twouters-dictviews-backport/Python/compile.c ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] int/float freelists vs pymalloc
On Feb 7, 2008 5:09 AM, Andrew MacIntyre [EMAIL PROTECTED] wrote: So I've been testing with the freelists ripped out and ints and floats reverted to fairly standard PyObject_New allocation (retaining the small int interning) and thus relying on PyMalloc to do any compaction. The results have been somewhat surprising... The short version is that: - for ints, the freelist is ahead of PyMalloc by a very small margin (4%) - for floats, the freelist is a long way behind PyMalloc (35% slower) Martin did some profiling of ints in py3k 1.5 years ago. The results of his profiling are here: http://svn.python.org/projects/python/branches/py3k/INTBENCH I think Martin wrote a mail to describe his work in more detail. But the only mails I could find are not what I remember: http://mail.python.org/pipermail/python-3000/2006-August/003185.html http://mail.python.org/pipermail/python-3000/2006-August/003064.html I don't remember if he did any work on head or if he remembers any more that might be relevant here. n ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python on non IEEE-754 platforms: plea for information.
On Feb 1, 2008 2:52 PM, Mark Dickinson [EMAIL PROTECTED] wrote: The IBM format is particularly troublesome because it's base 16 instead of base 2 (so e.g. multiplying a float by 2 can lose bits), but it appears that recent IBM machines do both IBM format and IEEE format floating-point. I assume that the S-390 buildbots are using the IEEE side---is this true? I don't know and suspect the only way to figure it out would be to write a test that would expose which is being used. It's using gcc, so we probably get whatever the compiler defaults to. Sometimes we have to specify flags for certain platforms. For example -mieee on the Alpha. It's fine to check in something so that you can get an answer on a buildbot. n ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] refleaks and caches
On Jan 27, 2008 3:37 PM, Christian Heimes [EMAIL PROTECTED] wrote: Phillip J. Eby wrote: Expose an API to clear the cache, and clear it at shutdown? It should probably be part of interpreter shutdown anyway. Good point. I've implemented PyType_ClearCache and exposed it via sys._cleartypecache(). The function is called during finalization, too. I'm not sure we should expose an API to clear the cache, but I don't have strong opinions either way. If we keep the ability to clear the cache, should we also consider some control over the int/float freelist? These are worse than the tuple/frame free lists since int/floats are unbounded. I suspect the method free lists in Objects/methodobject.c and Objects/classobject.c don't have that many entries that could be removed. There may not be a lot we can do for the int/float caches and I'm not sure if it's worth it. We can't move objects once created, although we could scan the blocks and if there are no live objects in a block, free it. That would presumably help with this case: list(range(some_big_number)) I don't know how important these things are in practice. Can somebody please double check the change? The results are promising and I'm sure I've implemented it correctly but you never know ;) The biggest problem I have with the patch is the attribute name. I would prefer underscores. ie _clear_type_cache instead of _cleartypecache. Attributes in sys are currently inconsistent, but it seems that most of the newer names have underscores. (Aside: if we are going to move attrs out of sys for py3k, we should consider renaming them to be consistent too. Regardless of moving them, should we rename them.) n ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] refleaks and caches
Around Jan 13, the refleak hunting test that is reported on python-checkins started to report refleaks on virtually every run. I suspect this is due to r59944 (at 2008-01-13 16:29:41) which was from patch #1700288 to cache methods. With this patch it makes it much harder to spot refleaks. Does anyone have ideas how to fix it? The only one I have is to disable the cache with a special flag, env't variable, sys variable/function or the like. We could make this available only in debug mode. The cache should normally be enabled, but could be disabled solely on the refleak runs. Suggestions? n ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] 2.5.2 release coming up
We need to fix the Windows buildbots. 2 tests are currently failing
for 2.5.2: test_mailbox test_winreg
From:
http://www.python.org/dev/buildbot/all/x86%20XP-4%202.5/builds/107/step-test/0
The errors are:
File
E:\cygwin\home\db3l\buildarea\2.5.bolen-windows\build\lib\test\test_mailbox.py,
line 522, in test_consistent_factory
msg2 = box.get_message(key)
File E:\cygwin\home\db3l\buildarea\2.5.bolen-windows\build\lib\mailbox.py,
line 315, in get_message
subpath = self._lookup(key)
File E:\cygwin\home\db3l\buildarea\2.5.bolen-windows\build\lib\mailbox.py,
line 484, in _lookup
raise KeyError('No message with key: %s' % key)
KeyError: 'No message with key: 1201127998.M194999P232Q203.buildbot'
and: test test_winreg failed -- Not the correct number of sub keys
There are several warnings from the tests that we should try to clean up:
lib\test\test_generators.py:1: DeprecationWarning: raising string
exceptions is deprecated
tutorial_tests =
lib\zipfile.py:714: DeprecationWarning: struct integer overflow
masking is deprecated
0, 0, count, count, pos2 - pos1, -1, 0)
lib\zipfile.py:691: DeprecationWarning: struct integer overflow
masking is deprecated
header_offset)
lib\test\test_unicode_file.py:103: UnicodeWarning: Unicode equal
comparison failed to convert both arguments to Unicode - interpreting
them as being unequal
filename1==filename2
lib\shutil.py:36: UnicodeWarning: Unicode equal comparison failed to
convert both arguments to Unicode - interpreting them as being unequal
os.path.normcase(os.path.abspath(dst)))
n
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Coverity Scan, Python upgraded to rung 2
On Jan 10, 2008 8:01 AM, Joseph Armbruster [EMAIL PROTECTED] wrote: I am not a developer but i'm interested in browsing it. Is it possible to be added? Yes, I've added you to the list. I'll probably send the list off tomorrow, so let me know if you would like to be added. n -- On Jan 10, 2008 10:57 AM, Christian Heimes [EMAIL PROTECTED] wrote: Neal Norwitz wrote: I think only Coverity can add people. You can send them a message if you would like to be added: [EMAIL PROTECTED] Or you can send mail to me and I can forward along all the people that would like to be added. I'll wait a few days to collect names so I can batch up the request. Count me in! Christian ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/josepharmbruster%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Coverity Scan, Python upgraded to rung 2
On Jan 9, 2008 1:12 PM, Christian Heimes [EMAIL PROTECTED] wrote: Thomas Heller wrote: Seems they are referring to the results of the rung 1 run (what ever 'rung' means ;-). With the account Neal made me some months ago, I can login on this page: http://scan.coverity.com:7475/ and see the scan results for Python. Last run at 2007-12-27: 11 Outstanding Defects, 6 of them marked uninspected, 3 marked pending, and 2 marked bug. My dict says: rung (of a ladder)- Leitersprossen Python has climbed up one step (or rung) of the ladder. They botched the link where it says Sign in. Use the link Thomas posted, ie: http://scan.coverity.com:7475/ That will show you the results from the latest coverity checker. Do you have the required permission to add more users to the site? I think only Coverity can add people. You can send them a message if you would like to be added: [EMAIL PROTECTED] Or you can send mail to me and I can forward along all the people that would like to be added. I'll wait a few days to collect names so I can batch up the request. n ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Coverity Scan, Python upgraded to rung 2
On Jan 9, 2008 9:08 AM, [EMAIL PROTECTED] wrote: Christian I read the announcement of the Python Users list and figured Christian out that some of the other core developers might be Christian interested in the news, too. Christian Among other projects Python was upgraded to Rung 2 on the Christian Coverity Scan list: http://scan.coverity.com/ I went to the run2 page: http://scan.coverity.com/rung2.html I shows 6 uninspected defects for Python. How do we see what they are? What is an uninspected defect? Any idea how the Coverity folks compute Defects/KLOC? For example, how does tcl manage to get a 0.0 score? The 6 have been inspected by me and I never came to a conclusion of whether they were a problem or not. There are 3 things which should be fixed and I haven't gotten around to them. They are not a big deal: Python/traceback.c line 177 Objects/codeobject.cline 322 Modules/mmapmodule.cline 1080 For traceback.c, namebuf defined on line 155 should be moved out one block since filename is an alias to namebuf and it is used outside the current scope. I think this is unlikely to be a problem in practice, but is technically wrong and should be fixed. For codeobject.c, line 327 should not be reachable. I kinda like the code as it is even though it is currently dead. I never decided if I wanted to change that or suppress the warning. For mmapmodule.c, fd should be checked for -1 before calling stat on line 1064. The rest were not obvious problems to me, and I never returned to them. n ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Coverity Scan, Python upgraded to rung 2
On Jan 9, 2008 9:08 AM, [EMAIL PROTECTED] wrote: I went to the run2 page: http://scan.coverity.com/rung2.html I shows 6 uninspected defects for Python. How do we see what they are? What is an uninspected defect? Any idea how the Coverity folks compute Defects/KLOC? For example, how does tcl manage to get a 0.0 score? Sorry, I forgot to answer the second part of your question. I have no idea how they compute Defects/KLOC. But the data is very old so I wouldn't worry about what that says. The most recent run has 286622 lines in 602 files. I already mentioned the 3 defects that should be fixed. Not sure what to do about the rest. n ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Problems with GeneratorExit deriving from Exception
On Dec 2, 2007 11:29 AM, Chad Austin [EMAIL PROTECTED] wrote: Hi Antoine, Antoine Pitrou wrote: Hi, (I was asked to forward this from the bug tracker) We have also run into problems where a task tries to return (yield Return()) from within a try: except Exception: block. Since returning from a coroutine is roughly equivalent to raise GeneratorExit, the exception can be caught and ignored, with the same consequences as above. I may be missing something but why don't you just catch StandardError instead? If I believe Python 2.5's exception hierarchy it would catch anything under Exception except for GeneratorExit, StopIteration and the various Warnings. If socket.error, xmlrpclib.Fault, httplib.HTTPException, etc. all extended StandardError, then we would probably be fine with that approach. But I think the majority of exceptions, both in the standard library and our code, extend Exception directly. Note that StandardError was removed from py3k. n ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] -O2 faster than -O3?
On Nov 30, 2007 7:16 PM, Brett Cannon [EMAIL PROTECTED] wrote: On Nov 30, 2007 12:02 PM, Neil Toronto [EMAIL PROTECTED] wrote: On both of my systems, using -O2 reduces execution time in pystone by 9% and in pybench by 8%. It's function inlining: -O3 -fno-inline-functions works just as well as -O2. Removing -g has little effect on the result. Systems: - AMD Athlon 64 X2 Dual Core 4600+, 512 KB cache (desktop) - Intel T2300 Dual Core 1.66GHz, 512 KB cache (laptop) Both are Ubuntu 7.04, GCC 4.1.2. Does anybody else see this? It may be GCC being stupid (which has happened before) or not enough cache on my systems (definitely possible). If it's not one of those, I'd say it's because CPython core functions are already very large, and almost everything that ought to be inlined is already a macro. That's quite possible. Previous benchmarks by AMK have shown that perhaps -0m (or whatever the flag is to optimize for size) sometimes is the best solution. It has always been believed that the eval loop is already large and manages to hit some cache sweet spot. The flag is -Os. I suspect you will do better to limit the size of inlining rather disabling it completely. The option is -finline-limit=number. I don't know the default value or what you should try. I would be interested to hear more results though. n ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] New python developer
On Nov 13, 2007 1:05 PM, Amaury Forgeot d'Arc [EMAIL PROTECTED] wrote: I can also help on win32 specific development. As an example, I find that the distutils module don't work very well with the new compilers. I won't be of much help on Unix, though. We need more help on Windows! Most of us are Unix only, so it will be great to have some help here. There are many windows bugs in the bug tracker if you'd like to take a look there. Welcome and good work so far! It's great seeing some of the memory issues in py3k get addressed. n ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Request for inclusion in 2.5.2 (5-for-1)
Mike, thanks for reviewing the bugs and for creating a patch. Good work! On Nov 2, 2007 2:23 AM, Nick Coghlan [EMAIL PROTECTED] wrote: Mike Klaas wrote: http://bugs.python.org/issue1705170: reproduced. Conjecture as to why it is occurring, but I don't know the guts well enough to propose a decent fix. I've fixed this on the trunk (I'm afraid I have no opinion on the patch you're interested in though. Neal - where does the 2.5 branch stand at the moment? This would be a simple fix to slip into 2.5.2 if there's still time. I'm not sure if this was before my last status mail. We are hoping to get a release candidate out in a week or two. I agree with Guido that this isn't a bugfix and therefore shouldn't be backported to 2.5. n ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Windows amd64 warnings
There are many warnings from the trunk (and 2.5) builds on Win64. I suspect quite a few of these are real problems. Win64 is the most unusual platform wrt 64-bits due to long being 32-bit. If we need 64-bits, we need to use Py_ssize_t. IIRC, we've had some bug reports for Win64. I know I've fixed some problems found in py3k. It would be great if someone could create a patch to fix the warnings. The link below shows the warnings from a recent checkin. http://python.org/dev/buildbot/all/amd64%20XP%20trunk/builds/297/step-compile/0 n ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] 2.5.2 is coming
On Oct 24, 2007 7:22 AM, Facundo Batista [EMAIL PROTECTED] wrote: 2007/10/12, Neal Norwitz [EMAIL PROTECTED]: The plan is cut the release candidate around Tuesday/Wednesday next week (Oct 16/17). If all goes well, 2.5.2 final will follow a week later. Hi Neal! Do you have any update of this schedule? Sorry, the various schedules of everyone involved didn't allow us to get a release out as planned. We are hoping to get out a 2.5.2 release candidate in a couple of weeks. n ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] 2.5.2 is coming
On Oct 11, 2007 11:36 PM, Thomas Heller [EMAIL PROTECTED] wrote: Neal, I have one patch for ctypes still sitting in some of my sandboxes. ctypes doesn't work when configured with --with-system-ffi on machines where 'sizeof(long long) != sizeof(long)', in other words on 32-bit boxes. Is is ok to apply this patch? The patch makes sense to me. If you are comfortable that this will fix more problems than it's likely to create, go for it. n ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] 2.5.2 is coming
We are planning the release of 2.5.2 for next week. Unless there are serious bugs, please hold off making big changes to the 2.5 branch until after 2.5.2 final is released. Anthony gets cranky when things break and he scares me...a lot. :-) Doc/test fixes as always are still fine. The plan is cut the release candidate around Tuesday/Wednesday next week (Oct 16/17). If all goes well, 2.5.2 final will follow a week later. Please test the 2.5 branch now and let us know if there are any serious problems. If there are serious bugs that you think should be addressed, let us know. Bugs are here: http://bugs.python.org/ I see the bugs below as possibly needing attention. Any comments on these? http://bugs.python.org/issue1692335 Fix exception pickling: Move initial args assignment to BaseException.__new__ http://bugs.python.org/issue815646 thread unsafe file objects cause crash http://bugs.python.org/issue1179 Integer overflow in imageop module I think Anthony might have a patch for 1179. Cheers, n ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Other SSL issues in the tracker have been marked
Bill, Please mail your ssh key as an attachment to [EMAIL PROTECTED] n -- On 8/28/07, Guido van Rossum [EMAIL PROTECTED] wrote: Hi Bill, I think it's time you get to do your own checkins. There's a protocol but I forget how to do it -- I think you mail your ssh key to Martin. --Guido On 8/28/07, Bill Janssen [EMAIL PROTECTED] wrote: So, the patch is attached to issue 1052. I sent it out to python-dev, but it got blocked (too big). http://bugs.python.org/file8352/ssl-patch-5 This contains a number of things: 1) Improve the documentation of the SSL module, with a fuller explanation of certificate usage, another reference, proper formatting of this and that. 2) Fix Windows bug in ssl.py, and general bug in sslsocket.close(). Remove some unused code from ssl.py. Allow accept() to be called on sslsocket sockets. 3) Use try-except-else in import of ssl in socket.py. Deprecate use of socket.ssl(). 4) Remove use of socket.ssl() in every library module, except for test_socket_ssl.py and test_ssl.py. Bill -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Other SSL issues in the tracker have been marked
On 8/26/07, Bill Janssen [EMAIL PROTECTED] wrote: This occurs on at least 3 of the buildbots (ubuntu and debian on ia64, ppc, and hppa). Here's one example: http://python.org/dev/buildbot/all/ia64%20Ubuntu%20trunk%20trunk/builds/832/step-test/0 If I'm reading this right, it's passing tests on amd64 gentoo trunk, x86 gentoo trunk, g4 osx.4 trunk (no surprise there). And looking at the community buildbots, it works on x86 Redhat 9, x86 Debian unstable, amd64 Ubuntu gutsy, G5 OS X, and so on. I've tested it myself on FC 7 and it works. And looking at the ppc Debian unstable case, test_socket is also failing there, so the test_ssl failure is not a big surprise. I'm not familiar with what's in Debian trunk or Ubuntu trunk; any idea what version of OpenSSL they have in them? For ia64 http://python.org/dev/buildbot/all/ia64%20Ubuntu%20trunk%20trunk/builds/833/step-test/0 [EMAIL PROTECTED]:~$ openssl version OpenSSL 0.9.8b 04 May 2006 I have access to some of the machines but not all of them. All of these run inside a chroot which might be causing a problem. I remember some issues with getting socket stuff to work right on them. But that was over a year ago and I don't remember the details now. :-( svn/google probably knows if you want to trawl through checkins. I'm not sure that will help much though. On this machines I was able to successfully make a key. That's why I'm thinking it might be some strange socket issue. But I think this exposes a more generic bug in test_ssl.py, which is that the server thread doesn't die when one of these failures occurs. It probably should. I'll make a patch -- but I don't have a system that this fails on, how will I test it? Yeah, I know this is difficult. Hopefully someone with WIndows will step up to help. We can at least make the test more robust and verify the files exist and are non-zero in size. I will do that now. At least the it shouldn't cause the test to time out. n ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Other SSL issues in the tracker have been marked
Thanks. I applied it. -- n
On 8/26/07, Bill Janssen [EMAIL PROTECTED] wrote:
Well, as long as you have your ears on, here's another patch to test_ssl.
1) Fixes the bug that two class names are initial-lower-case.
2) Replaces the poll waiting for the server to become ready with
a threading.Event signal.
Bill
Index: Lib/test/test_ssl.py
===
--- Lib/test/test_ssl.py(revision 57521)
+++ Lib/test/test_ssl.py(working copy)
@@ -153,9 +153,9 @@
c2.close()
-class threadedEchoServer(threading.Thread):
+class ThreadedEchoServer(threading.Thread):
-class connectionHandler(threading.Thread):
+class ConnectionHandler(threading.Thread):
def __init__(self, server, connsock):
self.server = server
@@ -219,6 +219,7 @@
self.certreqs = certreqs
self.cacerts = cacerts
self.sock = socket.socket()
+self.flag = None
if hasattr(socket, 'SO_REUSEADDR'):
self.sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
if hasattr(socket, 'SO_REUSEPORT'):
@@ -228,15 +229,22 @@
threading.Thread.__init__(self)
self.setDaemon(False)
+def start (self, flag=None):
+self.flag = flag
+threading.Thread.start(self)
+
def run (self):
self.sock.settimeout(0.5)
self.sock.listen(5)
self.active = True
+if self.flag:
+# signal an event
+self.flag.set()
while self.active:
try:
newconn, connaddr = self.sock.accept()
#sys.stdout.write('\nserver: new connection from ' +
str(connaddr) + '\n')
-handler = self.connectionHandler(self, newconn)
+handler = self.ConnectionHandler(self, newconn)
handler.start()
except socket.timeout:
pass
@@ -337,9 +345,11 @@
server = None
if test_support.is_resource_enabled('network'):
-server = threadedEchoServer(10024, CERTFILE)
-server.start()
-time.sleep(1)
+server = ThreadedEchoServer(10024, CERTFILE)
+flag = threading.Event()
+server.start(flag)
+# wait for it to start
+flag.wait()
tests.append(ConnectedTests)
thread_info = test_support.threading_setup()
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/nnorwitz%40gmail.com
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] new bug tracker broken?
It looks like you found the meta tracker rather than the python one. To reset your password for the main python tracker: http://bugs.python.org/[EMAIL PROTECTED] To report bugs, go here: http://bugs.python.org/ To register a new account, go here: http://bugs.python.org/[EMAIL PROTECTED] Note, I'm not sure if you have an account, you might need to register. To report problems about the bug tracker (ie, not python problems), go here: http://psf.upfronthosting.co.za/roundup/meta/ n -- On 8/25/07, Bill Janssen [EMAIL PROTECTED] wrote: I've been trying to reset my password on the new RoundUp tracker, and it isn't working. Then I tried to register for a new account, and I got mail from [EMAIL PROTECTED] to confirm. When I did, I got access to a small 27-issue tracker, which seems to be the bug-tracker for the bug-tracker. Still can't get into the Python bug-tracker. Bill ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/nnorwitz%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Other SSL issues in the tracker have been marked
Bill, Could you also look into this problem: Traceback (most recent call last): File /home/pybot/buildarea/trunk.klose-debian-ia64/build/Lib/threading.py, line 486, in __bootstrap_inner self.run() File /home/pybot/buildarea/trunk.klose-debian-ia64/build/Lib/test/test_ssl.py, line 144, in run cert_reqs=self.server.certreqs) File /home/pybot/buildarea/trunk.klose-debian-ia64/build/Lib/ssl.py, line 98, in __init__ cert_reqs, ssl_version, ca_certs) sslerror: _ssl.c:271: SSL_CTX_use_PrivateKey_file error This occurs on at least 3 of the buildbots (ubuntu and debian on ia64, ppc, and hppa). Here's one example: http://python.org/dev/buildbot/all/ia64%20Ubuntu%20trunk%20trunk/builds/832/step-test/0 This also looks like it's not working on windows, but there is no info from here: http://python.org/dev/buildbot/all/x86%20XP-3%20trunk/builds/164/step-test/0 Other than: test_ssl The system cannot find the path specified. Which happens after it hangs for 1200 seconds. Thanks, n -- On 8/25/07, Bill Janssen [EMAIL PROTECTED] wrote: I've gone through the other open SSL issues. Looks like some can be closed with the adoption of 1018 and 1024: 1027394 4 months agosocket.ssl should explain that it is a 2/3 connection 889813 4 months agomaking the version of SSL configurable when creating sockets 1583946 9 months agoSSL issuer and server names cannot be parsed 783188 46 months ago support for server side transactions in _ssl Others are about various standard libraries that interact with SSL in various ways. I'm working on another patch that converts all the standard library modules over to use the new ssl module, and I'll look at the rest of the SSL-related bugs as part of that work. Bill ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/nnorwitz%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Segfault
On 8/24/07, Hrvoje Nikšić [EMAIL PROTECTED] wrote: On Thu, 2007-08-23 at 11:57 -0700, Neal Norwitz wrote: allow threads section that runs with the GIL released, file_close might acquire the GIL and be running in parallel to this code. If file_close sets f_fp to NULL after the if condition evaluates, but before the call to _portable_fseek completes, we still get a segfault. However, the setting of f_fp to NULL happens with the GIL set, ie while only one thread is running. I thought other IO-bound threads (the sections that allow threads) could also run in parallel. Otherwise what is the point of BEGIN_ALLOW_THREADS? Right. I looked at this with Jeffrey Yasskin and agree that a lock is needed for f_fp. The fix is ugly because it's needed around all accesses to f_fp and there are a ton of them. Some are with the GIL held and others when it isn't. I would be fine with the much simpler approach in the patch I sent (assuming the patch is finished). It doesn't completely address the problem, but does make the race condition much, much smaller. Part of the reason I'm ok with this is that in 3.0, all this code has already been removed. n ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Segfault
On 8/23/07, Hrvoje Nikšić [EMAIL PROTECTED] wrote: On Wed, 2007-08-22 at 21:32 -0700, Neal Norwitz wrote: Py_BEGIN_ALLOW_THREADS errno = 0; - ret = _portable_fseek(f-f_fp, offset, whence); + if (f-f_fp != NULL) + ret = _portable_fseek(f-f_fp, offset, whence); Doesn't this kind of code retain a race condition? Since this is an Exactly what I'm thinking. That's why I said: I'm not convinced the attached patch is good enough though. It definitely reduces the race and might fix the problem. allow threads section that runs with the GIL released, file_close might acquire the GIL and be running in parallel to this code. If file_close sets f_fp to NULL after the if condition evaluates, but before the call to _portable_fseek completes, we still get a segfault. However, the setting of f_fp to NULL happens with the GIL set, ie while only one thread is running. So I *think* (but I'm not sure) the condition you mention is not possible. Either way the fix seems brittle. (Also, the patch I attached was not complete.) Also all this code has gone away in 3.0. n ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Segfault
On 8/22/07, Hrvoje Nikšić [EMAIL PROTECTED] wrote:
On Tue, 2007-08-21 at 09:14 -0700, Neal Norwitz wrote:
The patch is insufficient to prevent all types of crashes that occur
when accessing a file from 2 threads (closing in one and doing
whatever in another).
You are right. I wouldn't go so far to say the file object
thread-unsafe, but it certainly has lurking bugs with calling close
while other threads are running. BTW your mail doesn't seem to contain
the actual patch.
Sorry about that. I thought something was up when gmail didn't
automatically upload it while I was writing the message. It looks
like it took this time. (The attached patch isn't complete, but
catches most cases.)
n
Index: Objects/fileobject.c
===
--- Objects/fileobject.c (revision 57244)
+++ Objects/fileobject.c (working copy)
@@ -439,13 +439,16 @@
{
int sts = 0;
if (f-f_fp != NULL) {
+ /* Invalidate f-f_fp *before* closing, so we don't attempt
+ to close it twice in separate threads. */
+ FILE *tmp_fp = f-f_fp;
+ f-f_fp = NULL;
if (f-f_close != NULL) {
Py_BEGIN_ALLOW_THREADS
errno = 0;
- sts = (*f-f_close)(f-f_fp);
+ sts = (*f-f_close)(tmp_fp);
Py_END_ALLOW_THREADS
}
- f-f_fp = NULL;
}
PyMem_Free(f-f_setbuf);
f-f_setbuf = NULL;
@@ -539,7 +542,7 @@
file_seek(PyFileObject *f, PyObject *args)
{
int whence;
- int ret;
+ int ret = 0;
Py_off_t offset;
PyObject *offobj;
@@ -560,8 +563,11 @@
Py_BEGIN_ALLOW_THREADS
errno = 0;
- ret = _portable_fseek(f-f_fp, offset, whence);
+ if (f-f_fp != NULL)
+ ret = _portable_fseek(f-f_fp, offset, whence);
Py_END_ALLOW_THREADS
+ if (f-f_fp == NULL)
+ return err_closed();
if (ret != 0) {
PyErr_SetFromErrno(PyExc_IOError);
@@ -580,8 +586,8 @@
{
Py_off_t newsize;
PyObject *newsizeobj = NULL;
- Py_off_t initialpos;
- int ret;
+ Py_off_t initialpos = 0;
+ int ret = 0;
if (f-f_fp == NULL)
return err_closed();
@@ -597,8 +603,11 @@
*/
Py_BEGIN_ALLOW_THREADS
errno = 0;
- initialpos = _portable_ftell(f-f_fp);
+ if (f-f_fp != NULL)
+ initialpos = _portable_ftell(f-f_fp);
Py_END_ALLOW_THREADS
+ if (f-f_fp == NULL)
+ return err_closed();
if (initialpos == -1)
goto onioerror;
@@ -625,8 +634,11 @@
*/
Py_BEGIN_ALLOW_THREADS
errno = 0;
- ret = fflush(f-f_fp);
+ if (f-f_fp != NULL)
+ ret = fflush(f-f_fp);
Py_END_ALLOW_THREADS
+ if (f-f_fp == NULL)
+ return err_closed();
if (ret != 0)
goto onioerror;
@@ -639,30 +651,40 @@
/* Have to move current pos to desired endpoint on Windows. */
Py_BEGIN_ALLOW_THREADS
errno = 0;
- ret = _portable_fseek(f-f_fp, newsize, SEEK_SET) != 0;
+ if (f-f_fp != NULL)
+ ret = _portable_fseek(f-f_fp, newsize, SEEK_SET) != 0;
Py_END_ALLOW_THREADS
+ if (f-f_fp == NULL)
+ return err_closed();
if (ret)
goto onioerror;
/* Truncate. Note that this may grow the file! */
Py_BEGIN_ALLOW_THREADS
errno = 0;
- hFile = (HANDLE)_get_osfhandle(fileno(f-f_fp));
- ret = hFile == (HANDLE)-1;
- if (ret == 0) {
- ret = SetEndOfFile(hFile) == 0;
- if (ret)
-errno = EACCES;
+ if (f-f_fp != NULL) {
+ hFile = (HANDLE)_get_osfhandle(fileno(f-f_fp));
+ ret = hFile == (HANDLE)-1;
+ if (ret == 0) {
+ret = SetEndOfFile(hFile) == 0;
+if (ret)
+ errno = EACCES;
+ }
}
Py_END_ALLOW_THREADS
+ if (f-f_fp == NULL)
+ return err_closed();
if (ret)
goto onioerror;
}
#else
Py_BEGIN_ALLOW_THREADS
errno = 0;
- ret = ftruncate(fileno(f-f_fp), newsize);
+ if (f-f_fp != NULL)
+ ret = ftruncate(fileno(f-f_fp), newsize);
Py_END_ALLOW_THREADS
+ if (f-f_fp == NULL)
+ return err_closed();
if (ret != 0)
goto onioerror;
#endif /* !MS_WINDOWS */
@@ -670,8 +692,11 @@
/* Restore original file position. */
Py_BEGIN_ALLOW_THREADS
errno = 0;
- ret = _portable_fseek(f-f_fp, initialpos, SEEK_SET) != 0;
+ if (f-f_fp != NULL)
+ ret = _portable_fseek(f-f_fp, initialpos, SEEK_SET) != 0;
Py_END_ALLOW_THREADS
+ if (f-f_fp == NULL)
+ return err_closed();
if (ret)
goto onioerror;
@@ -688,14 +713,17 @@
static PyObject *
file_tell(PyFileObject *f)
{
- Py_off_t pos;
+ Py_off_t pos = 0;
if (f-f_fp == NULL)
return err_closed();
Py_BEGIN_ALLOW_THREADS
errno = 0;
- pos = _portable_ftell(f-f_fp);
+ if (f-f_fp != NULL)
+ pos = _portable_ftell(f-f_fp);
Py_END_ALLOW_THREADS
+ if (f-f_fp == NULL)
+ return err_closed();
if (pos == -1) {
PyErr_SetFromErrno(PyExc_IOError);
clearerr(f-f_fp);
@@ -727,14 +755,17 @@
static PyObject *
file_flush(PyFileObject *f)
{
- int res;
+ int res = 0;
if (f-f_fp == NULL)
return err_closed();
Py_BEGIN_ALLOW_THREADS
errno = 0;
- res = fflush(f-f_fp);
+ if (f-f_fp != NULL)
+ res = fflush(f-f_fp);
Py_END_ALLOW_THREADS
+ if (f-f_fp == NULL)
+ return err_closed();
if (res != 0) {
PyErr_SetFromErrno(PyExc_IOError);
clearerr(f-f_fp);
@@ -747,12 +778,15 @@
static PyObject *
file_isatty
Re: [Python-Dev] Segfault
On 8/21/07, Hrvoje Nikšić [EMAIL PROTECTED] wrote:
On Mon, 2007-08-20 at 20:27 +0200, Maciej Fijalkowski wrote:
import thread
while 1:
f = open(/tmp/dupa, w)
thread.start_new_thread(f.close, ())
f.close()
file_close inadvertently allows fclose to be called twice on the same
stdio file. This patch should fix the problem:
The patch is insufficient to prevent all types of crashes that occur
when accessing a file from 2 threads (closing in one and doing
whatever in another). Almost every place that accesses f_fp is a
problem. For example, try changing one of the f.close to f.tell. Or
in the main thread do a write. With all of these changes to the test
above, I was able to crash python with the patch (until I fixed the
uses of f_fp).
For example:
import thread
while 1:
f = open(/tmp/dupa, w)
thread.start_new_thread(f.close, ())
try:
f.write('a')
f.close()
except: pass
I've attached a patch (against 2.5) that fixes most of the problems,
but there are still a few more. (Search for Py_BEGIN_ALLOW_THREADS
and check for usage of f_fp within the block.) I'm not convinced the
attached patch is good enough though.
I think there might already be a bug report about file not being
thread-safe. (It could have also been socket, but I think Martin
fixed a problem in socket a while ago.)
n
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Source file markers for Tru64?
On 8/17/07, Guido van Rossum [EMAIL PROTECTED] wrote: Neal added it in rev 43214, when it was still PEP 3000. I'm sure he didn't read PEP 11 and beyond that I doubt he remembers from where he pulled this info. :-) When I added it in 43214, it was in an outstanding questions section. It wasn't meant to become gospel. :-) We have pretty crappy support for many versions of Unix. There are various problems with Solaris, AIX, HP-UX, etc. Not all the tests work on many of these platforms. When people report problems, they are almost always left unanswered. Since it's difficult to get access to boxes with the problems, our support isn't likely to improve. That was part of the motivation to consider removing the code. Another motivating factor was to eliminate code and make python easier to maintain. It may be reasonable that little used platforms are maintained as patches outside the core. VMS did this for a long time. I'm not sure if stock python has full VMS support or if there are still some patches necessary. Below are more details in case anyone cares. The checkin spawned this thread: http://mail.python.org/pipermail/python-dev/2006-March/062741.html One result was that we got a Tru64 buildbot out of it (which I have access to and help maintain). I'm not against keeping lesser platforms. But if they aren't used, they should be removed. Here is HP's support for Tru64: http://h30097.www3.hp.com/support.html The supported versions show the last released version was V5.1B-4 Dec 2006. HP still supports B-3 and B-4 for a minimum of 42 months from release (Jan 2005 for B-3). The end of support is still TBD for B-3 and B-4. n -- On 8/17/07, [EMAIL PROTECTED] [EMAIL PROTECTED] wrote: I didn't make it up. I'm pretty sure I saw it somewhere in either the wiki, the python-3000 mailing list or in the source code itself. I'll dig around some and figure out where I found it. In the meantime, if someone else stumbles upon the smoking gun let me know. PEP 3100: Remove support for old systems, including: BeOS, RISCOS, (SGI) Irix, Tru64 I'll hold off on doing anything more until I can update PEP 11. Skip -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-3000] Python 3000 Sprint @ Google
On 8/15/07, Bill Janssen [EMAIL PROTECTED] wrote: Sounds like a good plan. I'm not a great coach though since I didn't write _ssl.c and I've never used openssl directly. But I can help you with the Python stuff of course! Thanks (though I think I can handle the Python end of it, too :-). It's been a while since I wrote any Python C code, though -- are there better tools these days for debugging reference counting? Anyone know? The way I typically do it is to configure --with-pydebug. That shows the ref count in the interpreter and allows running tests with the -R flag to regrtest. When regrtest reports leaks, narrow down the (Python) code which causes a leak using bisection, find the C code which corresponds, and visually inspect the C code. Most leaks are pretty obvious this way. With good tests, this doesn't take much time. For pure memory leaks, valgrind works pretty well. n ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-3000] Documentation switch imminent
On 8/15/07, Georg Brandl [EMAIL PROTECTED] wrote:
Georg Brandl schrieb:
Neal will change his build scripts, so that the 2.6 and 3.0 devel
documentation pages at docs.python.org will be built from these new
trees soon.
Okay, I made the switch. I tagged the state of both Python branches
before the switch as tags/py{26,3k}-before-rstdocs/.
http://docs.python.org/dev/
http://docs.python.org/dev/3.0/
The upgrade went smoothly. Below are all the issues I noticed. I had
to install a version of python 2.5 since that is a minimum
requirement. I had to change from a plain 'make' in the Doc directory
to 'make html'. The output is in build/html rather than html/ now.
2.6 output:
trying to load pickled env... failed: [Errno 2] No such file or
directory: 'build/doctrees/environment.pickle'
writing output...
... library/contextlib.rststring:3: Warning: 'with' will become a
reserved keyword in Python 2.6
tutorial/errors.rststring:1: Warning: 'with' will become a reserved
keyword in Python
3.0 output:
Traceback (most recent call last):
File tools/sphinx-build.py, line 13, in module
from sphinx import main
File /home/neal/python/py3k/Doc/tools/sphinx/__init__.py, line 16,
in module
from .builder import builders
File /home/neal/python/py3k/Doc/tools/sphinx/builder.py, line 35,
in module
from .environment import BuildEnvironment
File /home/neal/python/py3k/Doc/tools/sphinx/environment.py, line
34, in module
from docutils.parsers.rst.states import Body
File /home/neal/python/py3k/Doc/tools/docutils/parsers/rst/__init__.py,
line 77, in module
from docutils.parsers.rst import states
File /home/neal/python/py3k/Doc/tools/docutils/parsers/rst/states.py,
line 110, in module
import roman
ImportError: No module named roman
After this error, I just linked my tools directory to the one in 2.6
(trunk) and that worked. I'm not sure if this will create problems in
the future.
trying to load pickled env... failed: [Errno 2] No such file or
directory: 'build/doctrees/environment.pickle'
writing output...
... library/contextlib.rststring:3: Warning: 'with' will become a
reserved keyword in Python 2.6
library/shutil.rststring:17: Warning: 'as' will become a reserved
keyword in Python 2.6
library/subprocess.rststring:7: Warning: 'as' will become a reserved
keyword in Python 2.6
tutorial/errors.rststring:1: Warning: 'with' will become a reserved
keyword in Python 2.6
I realize none of these are a big deal. However, it would be nice if
it was cleaned up so that people unfamiliar with building the docs
aren't surprised.
n
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] T_PYSSIZET in Include/structmember.h can be hidden
Martin, Do you know why T_PYSSIZET is inside a #ifdef HAVE_LONG_LONG? That seems like a mistake. Here's the code: #ifdef HAVE_LONG_LONG #define T_LONGLONG 17 #define T_ULONGLONG 18 #define T_PYSSIZET 19 /* Py_ssize_t */ #endif /* HAVE_LONG_LONG */ ISTM, that T_PYSSIZET should be after the #endif. Was this a mistake or intentional? n ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Python sprint at Google
It's about that time of year again. We are starting to organize Python sprints again this year hosted at Google (ie, there was a 5 minute conversation and a couple of emails). Nothing has solidified yet, but here are our ideas: * Hosted at Google's headquarters in Mountain View, CA USA * From Wed, Aug 22 to Sat Aug 25 (4 days) * Cost: free, but need to code for food and internet :-) How does that work for people? You can attend for as little or as much time as you want. I expect most people will focus on py3k--the core language changes as well as tools to support the transition. We'd like to start figuring out how many people would be attending. If you are interested, please mail me privately. You don't need to commit to anything. Just how likely you are to attend. Could someone update the wiki we used for last years sprints: http://wiki.python.org/moin/GoogleSprint If you would be interested in other locations/dates, let me know. I don't know that we can accommodate anything else, but we can try. Cheers, n ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Python developers at Google (was Re: Add a -z interpreter flag to execute a zip file)
On 7/12/07, [EMAIL PROTECTED] [EMAIL PROTECTED] wrote: ... there is a heavy Google bias in the existing survey sample ... Not just this survey... There are many python developers at Google and that is not likely to change anytime soon. This means that it's even more important to hear differing points of view. We need to hear more from people outside of Google to ensure we are doing the best possible job. It would also be great to have more active committers that don't work at Google. n ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Add a -z interpreter flag to execute a zip file
On 7/12/07, Fred L. Drake, Jr. [EMAIL PROTECTED] wrote: Phillip Eby: Testing your package before you zip it, would be one. :) My personal main interest was in being able to add an item to sys.path without having to set $PYTHONPATH on Windows. That's why I'd like it to be possible to use -z more than once (or whatever the option ends up as). What happens if multiple entries contain __main__.py entries? I don't like this one so much. I don't know what Java does if you specify -jar more than once; that might suggest something. You can't with: java version 1.5.0_11 Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_11-b03) Java HotSpot(TM) 64-Bit Server VM (build 1.5.0_11-b03, mixed mode) -help says: or java [-options] -jar jarfile [args...] (to execute a jar file) args are passed to the jarfile being run. $ java -jar xalan2.jar -jar xalan2.jar Invalid option: -jar Invalid option: xalan2.jar n ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
