Re: [Python-Dev] Any reason that any()/all() do not take a predicate argument?
Andrew Koenig acm.org> writes: > How about this? > > if any(x==5 for x in seq): Cool! Thank you. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Any reason that any()/all() do not take a predicateargument?
>> seq = [1,2,3,4,5] >> if any(seq, lambda x: x==5): >> ... >> >> which is clearly more readable than >> >> reduce(seq, lambda x,y: x or y==5, False) > > How about this? > > if any(x==5 for x in seq): Aren't all of these equivalent to: if 5 in seq: ... ? Cheers, Brian ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Any reason that any()/all() do not take a predicateargument?
Brian Quinlan wrote: >>> if any(seq, lambda x: x==5): >> if any(x==5 for x in seq): > > Aren't all of these equivalent to: > > if 5 in seq: > ... There should be one-- and preferably only one --obvious way to do it. Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Checkin 45232: Patch #1429775
Hi Simon, On Thu, Apr 13, 2006 at 06:43:09PM +0200, Simon Percivall wrote: > Building SVN trunk with --enable-shared has been broken on Mac OS X > Intel > since rev. 45232 a couple of days ago. I can't say if this is the case > anywhere else as well. What happens is simply that ld can't find the > file > to link the shared mods against. For what it's worth, it still works on Linux (Gentoo/i386), insofar as it always worked -- which is that we need either to "make install" or to tweak /etc/ld.so.conf to let the executable find libpython2.5.so. A bientot, Armin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Checkin 45232: Patch #1429775
Armin Rigo wrote: > For what it's worth, it still works on Linux (Gentoo/i386), insofar as > it always worked -- which is that we need either to "make install" or to > tweak /etc/ld.so.conf to let the executable find libpython2.5.so. I usually set LD_LIBRARY_PATH in the shell where I want to use an --enable-share'd binary. Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Building Python with the free MS Toolkit compiler
Paul Moore wrote: > I've just added some instructions on how to build Python on Windows > with the free MS Toolkit C++ compiler. They are at > http://wiki.python.org/moin/Building_Python_with_the_free_MS_C_Toolkit. Cool! If you think that adding/changing files in Python would simplify the process, don't hesitate to produce patches. Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Checkin 45232: Patch #1429775
Hi Martin, On Sat, Apr 15, 2006 at 11:30:07AM +0200, "Martin v. L?wis" wrote: > Armin Rigo wrote: > > For what it's worth, it still works on Linux (Gentoo/i386), insofar as > > it always worked -- which is that we need either to "make install" or to > > tweak /etc/ld.so.conf to let the executable find libpython2.5.so. > > I usually set LD_LIBRARY_PATH in the shell where I want to use an > --enable-share'd binary. Thanks for reminding me of that trick! A bientot, Armin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Py_Finalize does not release all memory, not even closely
Tim Peters wrote: > Well, there may well be a bug (or multiple bugs) underlying that one > too. It's one thing for Py_Finalize() not to release all memory (it > doesn't and probably never will), but it's not necessarily the same > thing if running Py_Initialize() ... Py_Finalize() repeatedly keeps > leaking more and more memory. Running Py_Initialize/Py_Finalize once leaves 2150 objects behind (on Linux). The second run adds 180 additional objects; each subsequent run appears to add 156 more. > Not unless the module has a finalization function called by > Py_Finalize() that frees such things (like PyString_Fini and > PyInt_Fini). How should the module install such a function? PyString_Fini and PyInt_Fini are invoked explicitly in pythonrun.c. That doesn't scale to extension modules. > I'm not clear on whether, e.g., init_socket() may get called more than > once if socket-slinging code appears in a Py_Initialize() ... > Py_Finalize(). Module initialization functions are called each time. Py_Finalize "forgets" which modules had been loaded, and reloads them all. Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Py_Finalize does not release all memory, not even closely
Martin v. Löwis wrote: > Tim Peters wrote: >> Well, there may well be a bug (or multiple bugs) underlying that one >> too. It's one thing for Py_Finalize() not to release all memory (it >> doesn't and probably never will), but it's not necessarily the same >> thing if running Py_Initialize() ... Py_Finalize() repeatedly keeps >> leaking more and more memory. > > Running Py_Initialize/Py_Finalize once leaves 2150 objects behind (on > Linux). The second run adds 180 additional objects; each subsequent > run appears to add 156 more. With COUNT_ALLOCS, I get the following results: Ignoring the two initial rounds of init/fini, each subsequent init/fini pair puts this number of objects into garbage: builtin_function_or_method 9 cell 1 code 12 dict 23 function 12 getset_descriptor 9 instancemethod 7 int 9 list 6 member_descriptor 23 method_descriptor 2 staticmethod 1 str 86 tuple 78 type 14 weakref 38 wrapper_descriptor 30 This totals to 360, which is for some reason higher than the numbers I get when counting the objects on the global list of objects. Is it not right to obtain the number of live object by computing tp->tp_allocs-tp->tp_frees? Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Building Python with the free MS Toolkit compiler
On 4/15/06, "Martin v. Löwis" <[EMAIL PROTECTED]> wrote: > Paul Moore wrote: > > I've just added some instructions on how to build Python on Windows > > with the free MS Toolkit C++ compiler. They are at > > http://wiki.python.org/moin/Building_Python_with_the_free_MS_C_Toolkit. > > Cool! If you think that adding/changing files in Python would simplify > the process, don't hesitate to produce patches. I've just submitted http://www.python.org/sf/1470875 and assigned it to you. I hope that's OK. It's basically a documentation patch plus 2 supporting build files. Paul ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Any reason that any()/all() do not take a predicateargument?
> >> seq = [1,2,3,4,5] > >> if any(seq, lambda x: x==5): > >> ... > >> > >> which is clearly more readable than > >> > >> reduce(seq, lambda x,y: x or y==5, False) > > > > How about this? > > > > if any(x==5 for x in seq): > > Aren't all of these equivalent to: > > if 5 in seq: > ... Yeah, but you can't do more complicated expressions that way, like any(lambda x: x[3] == "thiskey") I think it makes a lot of sense for any and all to take optional predicate function arguments. But perhaps the syntax should be: X in SEQ If X is a predicate function, it gets called to determine "equals"; if an expression or other object, the normal rules apply. Of course, then you couldn't look for a function in a set of functions... I suppose (len([x for x in SEQ if PRED(x)]) > 0) will suffice for now. Obvious enough, Martin? Bill ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Any reason that any()/all() do not take a predicateargument?
Bill Janssen wrote: > Yeah, but you can't do more complicated expressions that way, like > > any(lambda x: x[3] == "thiskey") Not /quite/ sure what this is intended to mean, but most likely, you meant any(x[3]=="thiskey" for x in seq) > I think it makes a lot of sense for any and all to take optional > predicate function arguments. I don't believe that adds expressiveness: you can always formulate this with a generator expression - apparently, those are of the "read-only" nature, i.e. difficult to formulate (assuming you have no difficulties to read above term). > I suppose > > (len([x for x in SEQ if PRED(x)]) > 0) > > will suffice for now. Obvious enough, Martin? It's simpler written as any(PRED(x) for x in SEQ) or any(True for x in SEQ if PRED(x)) if you want Using any() has the advantage over len() that the any() code stops at the first value that becomes true, whereas the len code ill compute them all. Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] r45321 - in python/trunk: Lib/test/test_traceback.py Lib/traceback.py Misc/NEWS
On Sat, 15 Apr 2006, Tim Peters wrote: [...] > [also John] >> Sorry, please ignore the post of mine I'm replying to here. >> >> I missed part of the thread, and Tim has already answered my question... > > That's news to Tim ;-) You mentioned use of '...' / ELLIPSIS, IIRC, so I assumed that would work -- but it seems not, from your latest post (that I'm replying to here). > It's not possible to add a new doctest option > in 2.5 that would allow a doctest to work under both 2.4 and 2.5: the > test would blow up under 2.4, because 2.4's doctest wouldn't recognize > the (new in 2.5) option, and would raise a ValueError of its own > griping about the unknown option name. [...] > """ import decimal try: 1 / decimal.Decimal(0) except decimal.DivisionByZero: print "good" > good > """ Yes, that works, I see. Kind of annoying of course, but can't be helped. Hmm, will 2.5's doctest work under Python 2.4? I guess that's not guaranteed, since I don't see any comment in doctest.py implying it needs to be compatible with old Pythons. > Oddly enough, ELLIPSIS doesn't actually work for this purpose: [...] John ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] r45321 - in python/trunk: Lib/test/test_traceback.py Lib/traceback.py Misc/NEWS
[John J Lee] ... > You mentioned use of '...' / ELLIPSIS, IIRC, so I assumed that would work > -- but it seems not, from your latest post (that I'm replying to here). Different context -- answering why IGNORE_EXCEPTION_DETAIL exists given that ELLIPSIS can do everything it does (provided you don't care about Pythons before 2.4). That sub-thread was a red herring (a distraction from the real topic here). ... > Hmm, will 2.5's doctest work under Python 2.4? I guess that's not > guaranteed, since I don't see any comment in doctest.py implying it needs > to be compatible with old Pythons. doctest compatibility with 2.4 is neither a goal nor a non-goal for 2.5. I'm not sure why it's being asked, since the incompatible change projected for 2.5 is in how the trackback module spells some exception names; and doctest (every version of doctest) gets its idea of the name of an exception from the traceback module. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Py_Finalize does not release all memory, not even closely
[Martin] > Running Py_Initialize/Py_Finalize once leaves 2150 objects behind (on > Linux). The second run adds 180 additional objects; each subsequent > run appears to add 156 more. One thing I notice is that they're all > 0 :-) I believe that, at one time, the second and subsequent numbers were 0, but maybe that's just old age pining for the days when kids listened to good music. Because new-style classes create cycles that Py_Finalize() doesn't clean up, it may make analysis easier to stick a PyGC_Collect() call (or two! repeat until it returns 0) inside the loop now. ... >> Not unless the module has a finalization function called by >> Py_Finalize() that frees such things (like PyString_Fini and >> PyInt_Fini). > How should the module install such a function? There is no way at present, short of editing the source for Py_Finalize and recompiling. Presumably this is something that should be addressed in the module initialization/finalization PEP, right? I suppose people may want a way for Python modules to provide finalization functions too. >> I'm not clear on whether, e.g., init_socket() may get called more than >> once if socket-slinging code appears in a Py_Initialize() ... >> Py_Finalize(). > Module initialization functions are called each time. Py_Finalize > "forgets" which modules had been loaded, and reloads them all. OK, so things like the previous example (socketmodule.c's unconditional socket_gaierror = PyErr_NewException(...); in init_socket()) are guaranteed to leak "the old" socket_gaierror object across multiple socket module initializations. ... [later msg] ... > With COUNT_ALLOCS, I get the following results: Ignoring the two initial > rounds of init/fini, each subsequent init/fini pair puts this number > of objects into garbage: > > builtin_function_or_method 9 > cell 1 > code 12 > dict 23 > function 12 > getset_descriptor 9 > instancemethod 7 > int 9 > list 6 > member_descriptor 23 > method_descriptor 2 > staticmethod 1 > str 86 > tuple 78 > type 14 > weakref 38 > wrapper_descriptor 30 FYI, from debugging Zope C-code leaks, staring at leftover strings, dict keys, and tuple contents often helps identify the sources. types too. > This totals to 360, which is for some reason higher than the numbers > I get when counting the objects on the global list of objects. How much higher? Last time I looked at this stuff (2.3b1, I think), the "all" in "the global list of all objects" wasn't true, and I made many changes to the core at the time to make it "more true". For example, all static C singleton objects were missing from that list (from Py_None and Py_True to all builtin static type objects). As SpecialBuilds.txt says, it became true in 2.3 that "a static type object T does appear in this list if at least one object of type T has been created" when COUNT_ALLOCS is defined. I expect that for _most_ static type objects, just doing initialize/finalize will not create objects of that type, so they'll be missing from the global list of objects. It used to be a good check to sum ob_refcnt over all objects in the global list, and compare that to _Py_RefTotal. The difference was a good clue about how many objects were missing from the global list, and became a better clue after I added code to inc_count() to call _Py_AddToAllObjects() whenever an incoming type object has tp_allocs == 0. Because of the latter, every type object with a non-zero tp_allocs count is in the list of all objects now (but only when COUNT_ALLOCS is defined). It's possible that other static singletons (like Py_None) have been defined since then that didn't grow code to add them to the global list, although by construction _PyBuiltin_Init() does add all available from __builtin__. > Is it not right to obtain the number of live object by computing > tp->tp_allocs-tp->tp_frees? That's the theory ;-) This code is never tested, though, and I bet is rarely used. Every time I've looked at it (most recently for 2.3b1, about 3 years ago), it was easy to find bugs. They creep in over time. Common historical weak points are "fancy" object-creation code, and the possibility of resurrection in a destructor. The special builds need special stuff at such times, and exactly what's needed isn't obvious. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] valgrind reports
This was run on linux amd64. It would be great to run purify on windows and other platforms. If you do, please report back here, even if nothing was found. That would be a good data point. Summary of tests with problems: test_codecencodings_jp (1 invalid read) test_coding (1 invalid read) test_ctypes (crashes valgrind) test_socket_ssl (many invalid reads) test_sqlite (1 invalid read, 1 memory leak) Details: New one: test_codecencodings_jp ==19749== ==19749== Invalid read of size 1 ==19749==at 0xE4F7BB3: shift_jis_2004_decode (_codecs_jp.c:642) ==19749==by 0xE1AD8D5: decoder_feed_buffer (multibytecodec.c:839) ==19749==by 0xE1ADCEB: mbidecoder_decode (multibytecodec.c:1028) ==19749==by 0x46875C: call_function (ceval.c:3637) ==19749== Address 0x5E0DD63 is 0 bytes after a block of size 3 alloc'd ==19749==at 0x4A19AC6: malloc (vg_replace_malloc.c:149) ==19749==by 0xE1ADD7F: mbidecoder_decode (multibytecodec.c:1017) ==19749==by 0x46875C: call_function (ceval.c:3637) I think this is the same as: http://python.org/sf/1357836 test_coding ==19749== ==19749== Invalid read of size 1 ==19749==at 0x4135DF: tok_nextc (tokenizer.c:901) ==19749==by 0x413BE5: tok_get (tokenizer.c:1124) ==19749==by 0x4145D8: PyTokenizer_Get (tokenizer.c:1515) ==19749==by 0x411F12: parsetok (parsetok.c:135) ==19749==by 0x481051: PyParser_ASTFromFile (pythonrun.c:1318) ==19749== Address 0xC6E1EE6 is 2 bytes before a block of size 8,192 free'd ==19749==at 0x4A1A61D: free (vg_replace_malloc.c:235) ==19749==by 0x4125BC: error_ret (tokenizer.c:167) ==19749==by 0x412D34: decoding_fgets (tokenizer.c:513) ==19749==by 0x4134B0: tok_nextc (tokenizer.c:841) ==19749==by 0x413BE5: tok_get (tokenizer.c:1124) ==19749==by 0x4145D8: PyTokenizer_Get (tokenizer.c:1515) ==19749==by 0x411F12: parsetok (parsetok.c:135) test_ctypes crashes valgrind on amd64: http://bugs.kde.org/show_bug.cgi?id=125651 test_socket_ssl causes all kinds of havoc due to libssl. I don't know of any problems specific to Python. It would be good for others to test on different architectures. test_sqlite ==22745== Invalid read of size 4 ==22745==at 0x6B94D01: sqlite3VdbeFinalize (vdbeaux.c:1457) ==22745==by 0x6B7B24F: sqlite3_finalize (main.c:1304) ==22745==by 0x6858646: statement_dealloc (statement.c:300) ==22745==by 0x6854EAE: connection_call (connection.c:759) ==22745==by 0x419287: call_function_tail (abstract.c:1799) ==22745==by 0x4193AF: PyObject_CallFunction (abstract.c:1854) ==22745==by 0x685364C: cache_get (cache.c:170) ==22745==by 0x6856181: _query_execute (cursor.c:545) ==22745== Address 0x53FE38C is 148 bytes inside a block of size 696 free'd ==22745==at 0x4A1A61D: free (vg_replace_malloc.c:235) ==22745==by 0x6B8C39D: sqlite3FreeX (util.c:287) ==22745==by 0x6B94EB7: sqlite3VdbeDelete (vdbeaux.c:1524) ==22745==by 0x6B94D2D: sqlite3VdbeFinalize (vdbeaux.c:1462) ==22745==by 0x6B7B24F: sqlite3_finalize (main.c:1304) ==22745==by 0x6857F8B: statement_create (statement.c:82) ==22745==by 0x6854E67: connection_call (connection.c:748) ==22796== 8 bytes in 1 blocks are definitely lost in loss record 21 of 545 ==22796==at 0x4A19AC6: malloc (vg_replace_malloc.c:149) ==22796==by 0x6854A21: connection_set_isolation_level (connection.c:721) ==22796==by 0x6854C40: connection_init (connection.c:81) n ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Preserving the blamelist
[Tim] >> All the trunk buildbots started failing about 5 hours ago, in >> test_parser. There have been enough checkins since then that the >> boundary between passing and failing is about to scroll off forever. [Martin] > It's not lost, though; it's just not displayed anymore. It would be > possible to lengthen the waterfall from 12 hours to a larger period > of time. Since we're spread across time zones, I think 24 hours is a good minimum. If something is set to 12 hours now, doesn't look like it's working: when I wrote my msg, it showed (as I said) about 5 hours of history. Right now it shows only about 3 hrs, from Sat 15 Apr 2006 21:47:13 GMT to now (about 00:50:00 GMT Sunday). This is under Firefox on Windows, so nobody can blame it on an IE bug :-) > You can go back in time yourself, by editing the build number in > > http://www.python.org/dev/buildbot/trunk/g4%20osx.4%20trunk/builds/361 I don't care enough about osx.4 enough to bother ;-) Seriously, clicking on the waterfall display is much more efficient. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 359: The "make" Statement
Steven Bethard wrote: >make : > I don't like the position of the name being defined. It should be straight after the opening keyword, as with 'def' and 'class'. This makes it much easier to search for definitions of things, both by eyeball and editor search functions, etc. -- Greg ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Any reason that any()/all() do not take apredicateargument?
[Bill Janssen] > Yeah, but you can't do more complicated expressions that way, like > > any(lambda x: x[3] == "thiskey") You're not making any sense. The sequence argument is not listed and the lambda is unnecessary. Try this instead: any(x[3] == 'thiskey' for x in seq) > I think it makes a lot of sense for any and all to take optional > predicate function arguments. I think you don't understand what you already have to work with in Py2.5a -- a boolean expression in a genexp should always suffice -- no separate lambda based predicate function is ever required. Raymond ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 359: The "make" Statement
Greg Ewing wrote: > I don't like the position of the name being defined. > It should be straight after the opening keyword, as > with 'def' and 'class'. This makes it much easier > to search for definitions of things, both by eyeball > and editor search functions, etc. > > Also, all other definitions have a keyword or punctuation separating each token, probably for good reason. This might not be desirable though, as it would require yet another reserved word. (I personally think that make is too ubiquitous a name for it to be made a reserved word at this point, though.) The word 'of' might be appropriate as in: make class of type. However, 'from' seems to work better in most instances, and may be better or worse as it is already a reserved word. (make tag from Element) -- Ian D. Bollinger ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Py_Finalize does not release all memory, not even closely
Tim Peters wrote: > Because new-style classes create cycles that Py_Finalize() doesn't > clean up, it may make analysis easier to stick a PyGC_Collect() call > (or two! repeat until it returns 0) inside the loop now. I'm shy to do this: the comment in Py_Finalize suggests that things will break if there is a "late" garbage collection. > There is no way at present, short of editing the source for > Py_Finalize and recompiling. Presumably this is something that should > be addressed in the module initialization/finalization PEP, right? Indeed. >> This totals to 360, which is for some reason higher than the numbers >> I get when counting the objects on the global list of objects. > > How much higher? Well, I counted an increase of 156 objects on the "all objects" list, and an increase of 360 according to the COUNT_ALLOCS numbers. The first number was without COUNT_ALLOCS being defined, though. Anyway, thanks for your comments. I'll try to look at this from time to time, maybe I can resolve some of the leaks. Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
