Re: [Python-Dev] default of returning None hurts performance?
Hi Antoine, Wpython already addressed this with two new opcodes. RETURN_CONST constant_index which is an equivalent of: LOAD_CONST constant_index RETURN_VALUE and with: CALL_PROC_RETURN_CONST function_arguments, constant_index which is an equivalent of: CALL_FUNCTION function_arguments POP_TOP LOAD_CONST constant_index RETURN_VALUE The new opcodes handle common patterns that I found with a stats tool I made, and helps both on space size and execution speed. Regards, Cesare 2009/9/1 Antoine Pitrou solip...@pitrou.net Gregory P. Smith greg at krypto.org writes: I was just wondering if a bytecode for a superinstruction of the common sequence: 6 POP_TOP 7 LOAD_CONST 0 (None) 10 RETURN_VALUE might be worth it. I think superinstructions in general would be a good thing to experiment, as wpython showed. Direct addressing (via a pseudo register file combining locals and constants) would eliminate many bookkeeping-related opcodes in common bytecode. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/cesare.dimauro%40a-tono.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] scripts, applets, and applications (Was Re: PyCon Keynote)
David Lyon wrote: Eric Smith wrote: This discussion probably belongs on the distutils list. Yes, the discussion should be. Except that distutils doesn't have very much support for doing applets or applications. So it's logical to see why most posts which ask the same question, often go unanswered there. Probably better to try on comp.lang.python which is more general. There's perhaps a lot more real world experience available there. I really wasn't asking for myself. The questions were meant to be suggestions of what I (and others) may find interesting as a PyCon KeyNote. I was also hoping for a broader overview (as a subject at pycon) rather than focusing on just zip files, or just pipi, or just distutils, etc. For example there seems to be a fair amount of confusion in terminology to describe modules, packages, applets, and applications. Some people interchange those with each other. Sometimes the differences are blurred because some applications are installed as a python library modules or packages. Full applications may need more or better batteries than smaller applets do. So there may be some improvements to the library that can be made. I'd like to see separate tools for handling library modules, applets, and applications. For example a tool to list and install or uninstall library modules directly from pipi would be great. A tool to optionally include non-core library modules into a applet zip file might be useful. Then of course to continue improving distutils for the purpose of making things ready for platform installers would be good. I don't think it is helpful to try to do to much with one tool. Yes some of this is already possible. But I think these things can also be improved, clarified, and documented better. Regards, Ron ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] patch to make list.pop(0) work in O(1) time
Am 28.01.2010 05:30, schrieb Steve Howell: If you want tools that are easy to use correctly, make them bug-free and document their behavior. If you want tools that are easy to use well, then make them perform better. I am not sure how my patch contradicts either of these goals. You keep making the argument that deque is a better alternative to list in many situations. I actually agree with you. Most programming problems are best modelled by a queue. I am not sure why Python lists get all the syntax sugar and promotion over deque, when in reality, Python lists implement a pretty useless data structure. Python lists are a glorification of a C array built on top of a memory-upward-biased memory allocator. As such, they optimize list appends (good) but fail awfully on list prepops (bad). They are much better as stacks than queues, even though queues are more useful for the most common programming known to man--work through a work queue and delete tasks when they are done. It is not surprising that Python lists are starting to show their lack of versatility in 2010. They're based on 1970's technology. Python lists are really just a thin encapsulation of C arrays built on top of an asymmetrical memory manager. In 2010 you could improve Python lists by releasing from the constraints of 1970s semantics. But I am starting to think a more modern approach would be to take more useful data structures like deques and give them more sugar. This post made me laugh, and skip the rest of the thread. Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython
Cesare I think that wpython as a proof-of-concept have done its work, Cesare showing its potentials. If you haven't alreayd is there any chance you can run the Unladen Swallow performance test suite and post the results? The code is separate from U-S and should work with wpython: http://unladen-swallow.googlecode.com/svn/tests -- Skip Montanaro - s...@pobox.com - http://www.smontanaro.net/ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython
2010/1/29 s...@pobox.com Cesare I think that wpython as a proof-of-concept have done its work, Cesare showing its potentials. If you haven't alreayd is there any chance you can run the Unladen Swallow performance test suite and post the results? The code is separate from U-S and should work with wpython: http://unladen-swallow.googlecode.com/svn/tests -- Skip Montanaro - s...@pobox.com - http://www.smontanaro.net/ I work on a Windows machine, so I don't know if I can run the U-S test suite on it (the first time I tried, it failed since U-S used a module available on Unix machines only). If it works now, I can provide results with wpython 1.0 final and the current 1.1 I'm working on (which has additional optimizations; I've also moved all peephole optimizer code on compile.c). Anyway, Mart Sõmermaa provided some resultshttp://www.mail-archive.com/python-dev@python.org/msg43294.htmlbased on wpython 1.0 alpha (you can find the wpython 1.0 final here http://code.google.com/p/wpython2/downloads/list). Cesare ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] car, cdr
Steve Howell showell30 at yahoo.com writes: What do you think of LISP, and car in particular (apart from the stupidly cryptic name)? The question is not directed to me but I would say it's a brain-dead data structure. (pointed pairs did allow me to create C++ template classes with a variable number of parameters, though :-)) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython
Cesare ... (you can find the wpython 1.0 final here Cesare http://code.google.com/p/wpython2/downloads/list). I tried downloading it. Something about wpython10.7z and wpython10_fix.7z. What's a 7z file? What tool on my Mac will unpack that? Can I build and run wpython on my Mac or is it Windows only? Thx, Skip ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython
On 29/01/2010 13:24, s...@pobox.com wrote: Cesare ... (you can find the wpython 1.0 final here Cesare http://code.google.com/p/wpython2/downloads/list). I tried downloading it. Something about wpython10.7z and wpython10_fix.7z. What's a 7z file? What tool on my Mac will unpack that? Can I build and run wpython on my Mac or is it Windows only? 7z (7zip) is a semi-popular compression format, and yes there are cross-platform tools to decompress them. A quick google should reveal them. Michael Thx, Skip ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (”BOGUS AGREEMENTS”) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython
2010/1/29 s...@pobox.com Cesare ... (you can find the wpython 1.0 final here Cesare http://code.google.com/p/wpython2/downloads/list). I tried downloading it. Something about wpython10.7z and wpython10_fix.7z. What's a 7z file? What tool on my Mac will unpack that? Can I build and run wpython on my Mac or is it Windows only? Thx, Skip You can find 7-Zip tools here http://www.7-zip.org/download.html. If you use Mercurial, you can grab a local copy this way: hg clone https://wpython10.wpython2.googlecode.com/hg/ wpython2-wpython10 Wpython is intended to run on any platform where CPython 2.6.4 runs. Cesare ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython
Cesare Di Mauro cesare.di.mauro at gmail.com writes: If python dev community is interested, I can work on a 3.x branch, porting all optimizations I made (and many others that I've planned to implement) one step at the time, in order to carefully check and validate any change with expert people monitoring it. We are certainly more interested in a 3.x branch than in a 2.x one ;-) You can start by cloning http://code.python.org/hg/branches/py3k/ Or you could submit patches piecewise on http://bugs.python.org I think the first step would be to switch to 16-bit bytecodes. It would be uncontroversial (the increase in code size probably has no negative effect) and would provide the foundation for all of your optimizations. Are you going to PyCon? Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython
Cesare You can find 7-Zip tools here Cesare http://www.7-zip.org/download.html. Thanks. Found a tool named 7za in MacPorts which I was able to install. One strong suggestion for future releases: Please put a top-level directory in your archives. It is annoying to expect that only to have an archive expand into the current directory without creating a directory of its own. I've been burned often enough that I always check before expanding source archives from new (to me) sources, so no harm, no foul in this case. Skip ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython
2010/1/29 Antoine Pitrou solip...@pitrou.net Cesare Di Mauro cesare.di.mauro at gmail.com writes: If python dev community is interested, I can work on a 3.x branch, porting all optimizations I made (and many others that I've planned to implement) one step at the time, in order to carefully check and validate any change with expert people monitoring it. We are certainly more interested in a 3.x branch than in a 2.x one ;-) You can start by cloning http://code.python.org/hg/branches/py3k/ Or you could submit patches piecewise on http://bugs.python.org I prefer to make a branch with Mercurial, which I found a comfortable tool. :) I think the first step would be to switch to 16-bit bytecodes. It would be uncontroversial (the increase in code size probably has no negative effect) and would provide the foundation for all of your optimizations. I agree. At the beginning I need to disable the peepholer, so performances are better to be compared when all peephole optimizations will be ported to the wordcode model. I'll make the branch after I release wpython 1.1, which I'll do ASAP. Are you going to PyCon? Antoine. No, I don't. But if there's a python-dev meeting, I can make a (long) jump. May be it can be easier to talk about the superinstructions model, and I can show and comment all optimizations that I made. Cesare ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython
2010/1/29 s...@pobox.com One strong suggestion for future releases: Please put a top-level directory in your archives. It is annoying to expect that only to have an archive expand into the current directory without creating a directory of its own. I've been burned often enough that I always check before expanding source archives from new (to me) sources, so no harm, no foul in this case. Skip You're right. Excuse me. I'll do it next time. Cesare ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython
Antoine Pitrou wrote: Or you could submit patches piecewise on http://bugs.python.org I think the first step would be to switch to 16-bit bytecodes. It would be uncontroversial (the increase in code size probably has no negative effect) and would provide the foundation for all of your optimizations. I wouldn't consider changing from bytecode to wordcode uncontroversial - the potential to have an effect on cache hit ratios means it needs to be benchmarked (the U-S performance tests should be helpful there). It's the same basic problem where any changes to the ceval loop can have surprising performance effects due to the way they affect the compiled switch statements ability to fit into the cache and other low level processor weirdness. If there was an old style map of the CPython code base, the whole area would have 'ware, here be dragons written over the top of it ;) Cheers, Nick. P.S. Note that I'm not saying I'm fundamentally opposed to a change to wordcode - just that it needs to be benchmarked fairly thoroughly first. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia --- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython
Nick Coghlan ncoghlan at gmail.com writes: Antoine Pitrou wrote: Or you could submit patches piecewise on http://bugs.python.org I think the first step would be to switch to 16-bit bytecodes. It would be uncontroversial (the increase in code size probably has no negative effect) and would provide the foundation for all of your optimizations. I wouldn't consider changing from bytecode to wordcode uncontroversial - the potential to have an effect on cache hit ratios means it needs to be benchmarked (the U-S performance tests should be helpful there). Well I said /probably/ has no negative effect :-) Actually, wordcode could allow accesses in the eval loop to be done on aligned words, so as to fetch operands in one step on little-endian CPUs (instead of recombining bytes manually). The change would be uncontroversial, however, in that it wouldn't modify code complexity or increase the maintenance burden. Of course, performance checks have to be part of the review. If there was an old style map of the CPython code base, the whole area would have 'ware, here be dragons written over the top of it ;) Just FYI: weakrefs and memoryviews are guarded by a mantichore, and only Benjamin can tame it. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Summary of Python tracker Issues
ACTIVITY SUMMARY (01/22/10 - 01/29/10) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue number. Do NOT respond to this message. 2579 open (+34) / 17049 closed (+15) / 19628 total (+49) Open issues with patches: 1049 Average duration of open issues: 708 days. Median duration of open issues: 464 days. Open Issues Breakdown open 2544 (+34) pending34 ( +0) Issues Created Or Reopened (52) ___ Bad example in set tutorial 01/24/10 http://bugs.python.org/issue4570reopened flox Cannot use both read and readline method in same ZipExtFile obje 01/28/10 CLOSED http://bugs.python.org/issue7610reopened ezio.melotti patch test_lib2to3 fails if path contains space01/27/10 http://bugs.python.org/issue7666reopened flox cPickle.load() segfault on invalid data 01/22/10 CLOSED http://bugs.python.org/issue7758created dhelfman mhlib fails on Btrfs filesystem (test_mhlib failure) 01/22/10 http://bugs.python.org/issue7759created nascheme use_errno=True does not work 01/23/10 http://bugs.python.org/issue7760created Nikratio telnetlib Telnet.interact fails on Windows but not Linux 01/23/10 http://bugs.python.org/issue7761created mnewman (C API) PyUnicode_Tailmatch documentation01/23/10 CLOSED http://bugs.python.org/issue7762created july (C API) PyUnicode_Tailmatch documentation01/23/10 CLOSED http://bugs.python.org/issue7763created july Doc for itertools recipe consume is complicated and less efficie 01/24/10 http://bugs.python.org/issue7764reopened flox 'unittest' documentation misprints 01/23/10 CLOSED http://bugs.python.org/issue7765created july patch sys.getwindowsversion as PyStructSequence01/23/10 CLOSED http://bugs.python.org/issue7766created brian.curtin patch, patch, needs review Add PyLong_AsLongLongAndOverflow 01/24/10 http://bugs.python.org/issue7767created casevh patch, needs review raw_input should encode unicode prompt with std.stdout.encoding. 01/24/10 http://bugs.python.org/issue7768created naoki SimpleXMLRPCServer.SimpleXMLRPCServer.register_function as decor 01/24/10 http://bugs.python.org/issue7769created ahojnnes sin/cos function in decimal-docs 01/24/10 http://bugs.python.org/issue7770created ahojnnes patch dict view comparison methods are not documented 01/24/10 http://bugs.python.org/issue7771created zuo test_py3kwarn fails when running the whole test suite01/24/10 http://bugs.python.org/issue7772created flox patch platform._parse_release_file() can cause UnboundLocalError 01/24/10 CLOSED http://bugs.python.org/issue7773created Arfrever patch, needs review
Re: [Python-Dev] patch to make list.pop(0) work in O(1) time
On Thu, Jan 28, 2010 at 8:57 PM, Steve Howell showel...@yahoo.com wrote: --- On Thu, 1/28/10, Josiah Carlson josiah.carl...@gmail.com wrote: [...] in the decade+ that I've been using Python and needed an ordered sequence; lists were the right solution 99% of the time [...] What do you think of LISP, and car in particular (apart from the stupidly cryptic name)? Apples and oranges. Lisp lists are really stacks, and are analogous in Python to... lisp_list = (item1, (item2, (item3, (item4, () lisp_car, lisp_cdr = lisp_list In many typical uses of lisp lists, car/cdr are used as a replacement for the equivalent of iteration in other languages. - Josiah ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython
2010/1/29 Nick Coghlan ncogh...@gmail.com I wouldn't consider changing from bytecode to wordcode uncontroversial - the potential to have an effect on cache hit ratios means it needs to be benchmarked (the U-S performance tests should be helpful there). It's quite strange, but from the tests made it seems that wpython perform better with old architectures (such as my Athlon64 socket 754), which have less resources like caches. It'll be interesting to check how it works on more limited ISAs. I'm especially curious about ARMs. It's the same basic problem where any changes to the ceval loop can have surprising performance effects due to the way they affect the compiled switch statements ability to fit into the cache and other low level processor weirdness. Cheers, Nick. Sure, but consider that with wpython wordcodes require less space on average. Also, less instructions are executed inside the ceval loop, thanks to some natural instruction grouping. For example, I recently introduced in wpython 1.1 a new opcode to handle more efficiently expression generators. It's mapped as a unary operator, so it exposes interesting properties which I'll show you with an example. def f(a): return sum(x for x in a) With CPython 2.6.4 it generates: 0 LOAD_GLOBAL 0 (sum) 3 LOAD_CONST 1 (code object genexpr at 00512EC8, file stdin, line 1) 6 MAKE_FUNCTION 0 9 LOAD_FAST 0 (a) 12 GET_ITER 13 CALL_FUNCTION 1 16 CALL_FUNCTION 1 19 RETURN_VALUE With wpython 1.1: 0 LOAD_GLOBAL 0 (sum) 1 LOAD_CONST 1 (code object genexpr at 01F13208, file stdin, line 1) 2 MAKE_FUNCTION 0 3 FAST_BINOP get_generator a 5 QUICK_CALL_FUNCTION 1 6 RETURN_VALUE The new opcode is GET_GENERATOR, which is equivalent (but more efficient, using a faster internal function call) to: GET_ITER CALL_FUNCTION 1 The compiler initially generated the following opcodes: LOAD_FAST 0 (a) GET_GENERATOR then the peepholer recognized the pattern UNARY(FAST), and produced the single opcode: FAST_BINOP get_generator a In the end, the ceval loop executes a single instruction instead of three. The wordcode requires 14 bytes to be stored instead of 20, so it will use 1 data cache line instead of 2 on CPUs with 16 bytes lines data cache. The same grouping behavior happens with binary operators as well. Opcodes aggregation is a natural and useful concept with the new wordcode structure. Cheers, Cesare ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] patch to make list.pop(0) work in O(1) time
On Thu, Jan 28, 2010 at 9:48 PM, Terry Reedy tjre...@udel.edu wrote: On 1/28/2010 6:30 PM, Josiah Carlson wrote: I would also point out that the way these things are typically done is that programmers/engineers have use-cases that are not satisfied by existing structures, they explain the issues they have with existing structures, and they propose modifications. So far, Steve has not offered any use-cases for why his proposed change is necessary; merely Use of a list as a queue rather than as a stack, as in breadth-first search, where one only needs to pop off the front but never push to the front. That is not to say that this is common or that a deque or other options may no be pretty satisfactory. But it would certainly be easier, when presenting such algorithms, to just be able to use a list, which has already been taught, than to introduce another structure. Currently a deque is not a drop-in replacement for a list in that one cannot use all list methods with a deque. Being able to use a list and get good performance straight off would be *convenient*. But that's it. People use it as a queue, discover that it is slow, ask (or research), and discover the deque. The use-cases where having the full range of list methods *and* deque behavior are fairly slim (perhaps none in my experience), and I argue are better covered by structures that are neither a Python list (even with the modifications offered by Steve) nor a deque. - Josiah ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython
2010/1/29 Antoine Pitrou solip...@pitrou.net Actually, wordcode could allow accesses in the eval loop to be done on aligned words, so as to fetch operands in one step on little-endian CPUs (instead of recombining bytes manually). Regards Antoine. I think that big-endians CPUs can get benefits too, since a single word load operation is needed, followed by an instruction such as ROL #8 to adjust the result (supposing that the compiler is smart enough to recognize the pattern). Using bytecodes, two loads are needed to retrieve the two bytes, and some SHIFT OR instructions to combine them getting the correct word. Loads are generally more expensive / limited. All that not counting the operations needed to advance the instruction pointer. Regards, Cesare ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython
Collin Winter wrote: I added startup benchmarks for Mercurial and Bazaar yesterday (http://code.google.com/p/unladen-swallow/source/detail?r=1019) so we can use them as more macro-ish benchmarks, rather than merely starting the CPython binary over and over again. If you have ideas for better Mercurial/Bazaar startup scenarios, I'd love to hear them. The new hg_startup and bzr_startup benchmarks should give us some more data points for measuring improvements in startup time. One idea we had for improving startup time for apps like Mercurial was to allow the creation of hermetic Python binaries, with all necessary modules preloaded. This would be something like Smalltalk images. We haven't yet really fleshed out this idea, though. In Python you can do the same with the freeze.py utility. See http://www.egenix.com/www2002/python/mxCGIPython.html for an old project where we basically put the Python interpreter and stdlib into a single executable. We've recently revisited that project and created something we call pyrun. It fits Python 2.5 into a single executable and a set of shared modules (which for various reasons cannot be linked statically)... 12MB in total. If you load lots of modules from the stdlib this does provide a significant improvement over standard Python. Back to the PEP's proposal: Looking at the data you currently have, the negative results currently don't really look good in the light of the small performance improvements. Wouldn't it be possible to have the compiler approach work in three phases in order to reduce the memory footprint and startup time hit, ie. 1. run an instrumented Python interpreter to collect all the needed compiler information; write this information into a .pys file (Python stats) 2. create compiled versions of the code for various often used code paths and type combinations by reading the .pys file and generating an .so file as regular Python extension module 3. run an uninstrumented Python interpreter and let it use the .so files instead of the .py ones In production, you'd then only use step 3 and avoid the overhead of steps 1 and 2. Moreover, the .so file approach would only load the code for code paths and type combinations actually used in a particular run of the Python code into memory and allow multiple Python processes to share it. As side effect, you'd probably also avoid the need to have C++ code in the production Python runtime - that is unless LLVM requires some kind of runtime support which is written in C++. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 29 2010) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ ::: Try our new mxODBC.Connect Python Database Interface for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython
On Fri, Jan 29, 2010 at 7:22 AM, Nick Coghlan ncogh...@gmail.com wrote: Antoine Pitrou wrote: Or you could submit patches piecewise on http://bugs.python.org I think the first step would be to switch to 16-bit bytecodes. It would be uncontroversial (the increase in code size probably has no negative effect) and would provide the foundation for all of your optimizations. I wouldn't consider changing from bytecode to wordcode uncontroversial - the potential to have an effect on cache hit ratios means it needs to be benchmarked (the U-S performance tests should be helpful there). It's the same basic problem where any changes to the ceval loop can have surprising performance effects due to the way they affect the compiled switch statements ability to fit into the cache and other low level processor weirdness. Agreed. We originally switched Unladen Swallow to wordcode in our 2009Q1 release, and saw a performance improvement from this across the board. We switched back to bytecode for the JIT compiler to make upstream merger easier. The Unladen Swallow benchmark suite should provided a thorough assessment of the impact of the wordcode - bytecode switch. This would be complementary to a JIT compiler, rather than a replacement for it. I would note that the switch will introduce incompatibilities with libraries like Twisted. IIRC, Twisted has a traceback prettifier that removes its trampoline functions from the traceback, parsing CPython's bytecode in the process. If running under CPython, it assumes that the bytecode is as it expects. We broke this in Unladen's wordcode switch. I think parsing bytecode is a bad idea, but any switch to wordcode should be advertised widely. Collin Winter ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Forking and Multithreading - enemy brothers
Hello, I've just recently realized the huge problems surrounding the mix of multithreading and fork() - i.e that only the main thread actually survived the fork(), and that process data (in particular, synchronization primitives) could be left in a dangerously broken state because of such forks, if multithreaded programs. What bothers me most is that I've actually never seen, in python docs, any mention of that problems (linux docs are very discreet as well). It's as if multithreading and multiprocessing were orthogonal designs, whereas it can quickly happen that someone has a slightly multithreaded programs, and suddenly uses the multiprocessing module to perform a separate, performance-demanding task ; with disasters in store, since few people are blatantly aware of the underlying dangers... So here are a few propositions to improve this matter : * documenting the fork/multithreading danger, in fork(), multiprocessing and maybe subprocess (is it concerned, or is the fork+exec always safe ?) modules. If it's welcome, I might provide documentation patches of course. * providing means of taming the fork() beast : is there a possibility for the inclusion of python-atfork and similar projects into the stdlib (I mean, their semantic, not the monkey-patch way they currently use) ? It would also help a lot the proper management of file handle inheritance. * maybe the most important : providing means to get rid of fork() whenever wanted. I'm especially thinking about the multiprocessing module : it seems it always uses forking on *nix platforms. Wouldn't it be better to also offer a spawnl() semantic, to allow safe multiprocessing use even in applications crowded with threads ? Win32 already uses something like that, so all the infrastructure of data transfer is already there, and it would enforce cross-platform compatibility. Since multiprocessing theoretically means a low coupling, and little sharing of data, I guess this kind of spawnl() semantic would be highly sufficient for most situations, which don't require fork-based multiprocessing and its huge sharing of process data (in my opinion, inheriting file descriptors is all a child process can require from its parent. Does it make sense to you ? Regards, Pascal Chambon ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython
On 1/29/2010 4:19 PM, Collin Winter wrote: On Fri, Jan 29, 2010 at 7:22 AM, Nick Coghlanncogh...@gmail.com wrote: Agreed. We originally switched Unladen Swallow to wordcode in our 2009Q1 release, and saw a performance improvement from this across the board. We switched back to bytecode for the JIT compiler to make upstream merger easier. The Unladen Swallow benchmark suite should provided a thorough assessment of the impact of the wordcode - bytecode switch. This would be complementary to a JIT compiler, rather than a replacement for it. I would note that the switch will introduce incompatibilities with libraries like Twisted. IIRC, Twisted has a traceback prettifier that removes its trampoline functions from the traceback, parsing CPython's bytecode in the process. If running under CPython, it assumes that the bytecode is as it expects. We broke this in Unladen's wordcode switch. I think parsing bytecode is a bad idea, but any switch to wordcode should be advertised widely. Several years, there was serious consideration of switching to a registerbased vm, which would have been even more of a change. Since I learned 1.4, Guido has consistently insisted that the CPython vm is not part of the language definition and, as far as I know, he has rejected any byte-code hackery in the stdlib. While he is not one to, say, randomly permute the codes just to frustrate such hacks, I believe he has always considered vm details private and subject to change and any usage thereof 'at one's own risk'. tjr ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython
Hey Terry, On Fri, Jan 29, 2010 at 2:47 PM, Terry Reedy tjre...@udel.edu wrote: On 1/29/2010 4:19 PM, Collin Winter wrote: On Fri, Jan 29, 2010 at 7:22 AM, Nick Coghlanncogh...@gmail.com wrote: Agreed. We originally switched Unladen Swallow to wordcode in our 2009Q1 release, and saw a performance improvement from this across the board. We switched back to bytecode for the JIT compiler to make upstream merger easier. The Unladen Swallow benchmark suite should provided a thorough assessment of the impact of the wordcode - bytecode switch. This would be complementary to a JIT compiler, rather than a replacement for it. I would note that the switch will introduce incompatibilities with libraries like Twisted. IIRC, Twisted has a traceback prettifier that removes its trampoline functions from the traceback, parsing CPython's bytecode in the process. If running under CPython, it assumes that the bytecode is as it expects. We broke this in Unladen's wordcode switch. I think parsing bytecode is a bad idea, but any switch to wordcode should be advertised widely. Several years, there was serious consideration of switching to a registerbased vm, which would have been even more of a change. Since I learned 1.4, Guido has consistently insisted that the CPython vm is not part of the language definition and, as far as I know, he has rejected any byte-code hackery in the stdlib. While he is not one to, say, randomly permute the codes just to frustrate such hacks, I believe he has always considered vm details private and subject to change and any usage thereof 'at one's own risk'. No, I agree entirely: bytecode is an implementation detail that could be changed at any time. But like reference counting, it's an implementation detail that people have -- for better or worse -- come to rely on. My only point was that a switch to wordcode should be announced prominently in the release notes and not assumed to be without impact on user code. That people are directly munging CPython bytecode means that CPython should provide a better, more abstract way to do the same thing that's more resistant to these kinds of changes. Collin Winter ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython
On 10:47 pm, tjre...@udel.edu wrote: On 1/29/2010 4:19 PM, Collin Winter wrote: On Fri, Jan 29, 2010 at 7:22 AM, Nick Coghlanncogh...@gmail.com wrote: Agreed. We originally switched Unladen Swallow to wordcode in our 2009Q1 release, and saw a performance improvement from this across the board. We switched back to bytecode for the JIT compiler to make upstream merger easier. The Unladen Swallow benchmark suite should provided a thorough assessment of the impact of the wordcode - bytecode switch. This would be complementary to a JIT compiler, rather than a replacement for it. I would note that the switch will introduce incompatibilities with libraries like Twisted. IIRC, Twisted has a traceback prettifier that removes its trampoline functions from the traceback, parsing CPython's bytecode in the process. If running under CPython, it assumes that the bytecode is as it expects. We broke this in Unladen's wordcode switch. I think parsing bytecode is a bad idea, but any switch to wordcode should be advertised widely. Several years, there was serious consideration of switching to a registerbased vm, which would have been even more of a change. Since I learned 1.4, Guido has consistently insisted that the CPython vm is not part of the language definition and, as far as I know, he has rejected any byte- code hackery in the stdlib. While he is not one to, say, randomly permute the codes just to frustrate such hacks, I believe he has always considered vm details private and subject to change and any usage thereof 'at one's own risk'. Language to such effect might be a useful addition to this page (amongst others, perhaps): http://docs.python.org/library/dis.html which very clearly and helpfully lays out quite a number of APIs which can be used to get pretty deep into the bytecode. If all of this is subject to be discarded at the first sign that doing so might be beneficial for some reason, don't keep it a secret that people need to join python-dev to learn. Jean-Paul ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython
On 10:55 pm, collinwin...@google.com wrote: That people are directly munging CPython bytecode means that CPython should provide a better, more abstract way to do the same thing that's more resistant to these kinds of changes. Yes, definitely! Requesting a supported way to do the kind of introspection that you mentioned earlier (I think it's a little simpler than you recollect) has been on my todo list for a while. The hold-up is just figuring out exactly what kind of introspection API would make sense. It might be helpful to hear more about how the wordcode implementation differs from the bytecode implementation. It's challenging to abstract from a single data point. :) For what it's worth, I think this is the code in Twisted that Collin was originally referring to: # it is only really originating from # throwExceptionIntoGenerator if the bottom of the traceback # is a yield. # Pyrex and Cython extensions create traceback frames # with no co_code, but they can't yield so we know it's okay to just # return here. if ((not lastFrame.f_code.co_code) or lastFrame.f_code.co_code[lastTb.tb_lasti] != cls._yieldOpcode): return And it's meant to be used in code like this: def foo(): try: yield except: # raise a new exception def bar(): g = foo() g.next() try: g.throw(...) except: # Code above is invoked to determine if the exception raised # was the one that was thrown into the generator or a different # one. Jean-Paul ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython
On Windows, would a C extension author be able to distribute a single binary (bdist_wininst/bdist_msi) which would be compatible with with-LLVM and without-LLVM builds of Python? When PEP 384 gets implemented, you not only get that, but you will also be able to use the same extension module for 3.2, 3.3, 3.4, etc, with or without U-S. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython
On 1/29/2010 5:55 PM, Collin Winter wrote: Hey Terry, On Fri, Jan 29, 2010 at 2:47 PM, Terry Reedytjre...@udel.edu wrote: Several years, there was serious consideration of switching to a registerbased vm, which would have been even more of a change. Since I learned 1.4, Guido has consistently insisted that the CPython vm is not part of the language definition and, as far as I know, he has rejected any byte-code hackery in the stdlib. While he is not one to, say, randomly permute the codes just to frustrate such hacks, I believe he has always considered vm details private and subject to change and any usage thereof 'at one's own risk'. No, I agree entirely: bytecode is an implementation detail that could be changed at any time. But like reference counting, it's an implementation detail that people have -- for better or worse -- come to rely on. My only point was that a switch to wordcode should be announced prominently in the release notes and not assumed to be without impact on user code. Of course. If it does not already, What's New could routinely have a section Changes in the CPython Implementation that would include any such change that might impact people. It could also include a warning to not depend on certain details even if they are mentioned. tjr ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython
On 1/29/2010 6:45 PM, Martin v. Löwis wrote: On Windows, would a C extension author be able to distribute a single binary (bdist_wininst/bdist_msi) which would be compatible with with-LLVM and without-LLVM builds of Python? When PEP 384 gets implemented, you not only get that, but you will also be able to use the same extension module for 3.2, 3.3, 3.4, etc, with or without U-S. Even if CPython changes VC compiler version? In any case, greater usability of binaries would be really nice. tjr ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Improved Traceback Module
At 01:24 AM 1/30/2010 +0100, Ludvig Ericson wrote: On 28 jan 2010, at 22:47, P.J. Eby wrote: At 07:47 PM 1/28/2010 +0100, Benjamin Schweizer wrote: I like the idea of configuring the list of variables with using a convention like __trace__, though this requires me to specify what variables cause an exception and that would be hard? The idea is that you simply use __trace__ to format the local variables for that frame. In other words, the programmer specifies what they want to see at that point in the traceback, by setting a variable in their code. If there's a __trace__ variable in the frame locals at that point in the traceback, simply format the traceback locals+globals using that string's .safe_format() method. This seems very naïve to me. It makes the code ten times as hard to maintain; adding or removing a *local variable* suddenly means you have to check what __trace__ or whatever refers to. You lost me; the purpose of a __trace__ is to show *context* -- the primary *purpose* of the code at that point in the call stack. You don't put every variable in it, as that would go directly against the purpose. That is, it's intended to *reduce* information overload, not increase it. If you're putting more than 2 variables in a given __trace__ you're probably doing something wrong. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython
On 1/29/2010 8:43 AM, Cesare Di Mauro wrote: If you use Mercurial, you can grab a local copy this way: hg clone https://wpython10.wpython2.googlecode.com/hg/ wpython2-wpython10 Wpython is intended to run on any platform where CPython 2.6.4 runs. Cesare, just FYI, your Hg repository has lost the execute bits on some files (namely ./configure and ./Parser/asdl_c.py), so it does not quite build out-of-the-box. I took the liberty of cloning your repo into my laptop's VirtualBox instance of Ubuntu. I ran the default performance tests from the U-S repo, with VirtualBox at highest priority. As a sanity check, I ran it against the U-S trunk. I think the numbers speak for themselves. $ ./perf.py -r -b default \ ../python-2.6.4/python \ ../wpython2-wpython10/python Report on Linux narf-ubuntu 2.6.31-16-generic #53-Ubuntu SMP Tue Dec 8 04:01:29 UTC 2009 i686 Total CPU cores: 1 ### 2to3 ### Min: 21.629352 - 20.893306: 1.0352x faster Avg: 22.245390 - 21.061316: 1.0562x faster Significant (t=4.416683) Stddev: 0.58810 - 0.11618: 5.0620x smaller Timeline: http://tinyurl.com/yawzt5z ### django ### Min: 1.105662 - 1.115090: 1.0085x slower Avg: 1.117930 - 1.131781: 1.0124x slower Significant (t=-11.024923) Stddev: 0.00729 - 0.01023: 1.4027x larger Timeline: http://tinyurl.com/ydzn6e6 ### nbody ### Min: 0.535204 - 0.559320: 1.0451x slower Avg: 0.558861 - 0.572902: 1.0251x slower Significant (t=-7.484374) Stddev: 0.01309 - 0.01344: 1.0272x larger Timeline: http://tinyurl.com/ygjnh5x ### slowpickle ### Min: 0.788558 - 0.757067: 1.0416x faster Avg: 0.799407 - 0.774368: 1.0323x faster Significant (t=12.772246) Stddev: 0.00686 - 0.01836: 2.6759x larger Timeline: http://tinyurl.com/y8g3zjg ### slowspitfire ### Min: 1.200616 - 1.218915: 1.0152x slower Avg: 1.229028 - 1.255978: 1.0219x slower Significant (t=-8.165772) Stddev: 0.02133 - 0.02519: 1.1808x larger Timeline: http://tinyurl.com/y9gg2x5 ### slowunpickle ### Min: 0.355483 - 0.347013: 1.0244x faster Avg: 0.369828 - 0.359714: 1.0281x faster Significant (t=6.817449) Stddev: 0.01008 - 0.01089: 1.0804x larger Timeline: http://tinyurl.com/ybf3qg9 ### spambayes ### Min: 0.316724 - 0.314673: 1.0065x faster Avg: 0.327262 - 0.332370: 1.0156x slower Significant (t=-3.690136) Stddev: 0.00598 - 0.01248: 2.0876x larger Timeline: http://tinyurl.com/ydck59l $ ./perf.py -r -b default \ ../python-2.6.4/python \ ../unladen-swallow/python ### 2to3 ### Min: 24.833552 - 24.433527: 1.0164x faster Avg: 25.241577 - 24.878355: 1.0146x faster Not significant Stddev: 0.39099 - 0.28158: 1.3886x smaller Timeline: http://tinyurl.com/yc7nm79 ### django ### Min: 1.153900 - 0.892072: 1.2935x faster Avg: 1.198777 - 0.926776: 1.2935x faster Significant (t=61.465586) Stddev: 0.03914 - 0.02065: 1.8949x smaller Timeline: http://tinyurl.com/ykm6lnk ### nbody ### Min: 0.541474 - 0.307949: 1.7583x faster Avg: 0.564526 - 0.327311: 1.7247x faster Significant (t=57.615664) Stddev: 0.01784 - 0.03711: 2.0798x larger Timeline: http://tinyurl.com/ylmezw5 ### slowpickle ### Min: 0.832452 - 0.607266: 1.3708x faster Avg: 0.860438 - 0.651645: 1.3204x faster Significant (t=20.779110) Stddev: 0.01559 - 0.09926: 6.3665x larger Timeline: http://tinyurl.com/yaktykw ### slowspitfire ### Min: 1.204681 - 1.038169: 1.1604x faster Avg: 1.236843 - 1.085254: 1.1397x faster Significant (t=20.203736) Stddev: 0.02417 - 0.07103: 2.9391x larger Timeline: http://tinyurl.com/ykgmop5 ### slowunpickle ### Min: 0.374148 - 0.279743: 1.3375x faster Avg: 0.398137 - 0.315630: 1.2614x faster Significant (t=16.069155) Stddev: 0.01333 - 0.04958: 3.7203x larger Timeline: http://tinyurl.com/y9b5rza ### spambayes ### Min: 0.330391 - 0.302988: 1.0904x faster Avg: 0.349153 - 0.394819: 1.1308x slower Not significant Stddev: 0.01158 - 0.35049: 30.2739x larger Timeline: http://tinyurl.com/ylq8sef -- Scott Dial sc...@scottdial.com scod...@cs.indiana.edu ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Restore the warning about mktemp now that DeprecationWarnings are silenced
On 28/01/2010 8.39, Brett Cannon wrote: On Wed, Jan 27, 2010 at 15:54, Ezio Melottiezio.melo...@gmail.com wrote: I noticed that in the py3k doc 'mktemp' is marked as deprecated since Python 2.3 [1], but the function is still there and doesn't raise any warning. Looking at the source I found out that there is a warning, but it is commented out [2], the reason being because they are too annoying. There was already a discussion about this on python-dev [3], but now that the DeprecationWarnings are silenced by default [4] the warning could be restored. If the comment can be removed I will fix it in trunk/py3k and possibly in 2.6/3.1 too, if #7319 will be backported. (Brett, are you planning to backport it?) If you mean forward-port to Python 3.1, then yes I am planning to merge it. Since Python 3.2 is not due out for nearly a year I wanted to wait a little while to make sure that the wording in the warnings docs made sense. But if you and others want to start actively turning some warnings on that will simply be too annoying when running the test suite I can do it sooner rather than later. -Brett I mean port it to py3k first and then backport it from trunk to 2.6 and from py3k to 3.1. If you want to wait then it's probably enough to add a note to the ticket and I'll fix it later. Regards, Ezio ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Restore the warning about mktemp now that DeprecationWarnings are silenced
On Fri, Jan 29, 2010 at 17:37, Ezio Melotti ezio.melo...@gmail.com wrote: On 28/01/2010 8.39, Brett Cannon wrote: On Wed, Jan 27, 2010 at 15:54, Ezio Melottiezio.melo...@gmail.com wrote: I noticed that in the py3k doc 'mktemp' is marked as deprecated since Python 2.3 [1], but the function is still there and doesn't raise any warning. Looking at the source I found out that there is a warning, but it is commented out [2], the reason being because they are too annoying. There was already a discussion about this on python-dev [3], but now that the DeprecationWarnings are silenced by default [4] the warning could be restored. If the comment can be removed I will fix it in trunk/py3k and possibly in 2.6/3.1 too, if #7319 will be backported. (Brett, are you planning to backport it?) If you mean forward-port to Python 3.1, then yes I am planning to merge it. Since Python 3.2 is not due out for nearly a year I wanted to wait a little while to make sure that the wording in the warnings docs made sense. But if you and others want to start actively turning some warnings on that will simply be too annoying when running the test suite I can do it sooner rather than later. -Brett I mean port it to py3k first and then backport it from trunk to 2.6 and from py3k to 3.1. I am only porting from trunk to py3k. Since this is not backwards-compatible I am not backporting to 2.6 or 3.1. If you want to wait then it's probably enough to add a note to the ticket and I'll fix it later. Make sure to mark it as dependent on issue 7319 which I am using to track the fact I still need to port it. -Brett ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython
Terry Reedy wrote: On 1/29/2010 6:45 PM, Martin v. Löwis wrote: When PEP 384 gets implemented, you not only get that, but you will also be able to use the same extension module for 3.2, 3.3, 3.4, etc, with or without U-S. Even if CPython changes VC compiler version? In any case, greater usability of binaries would be really nice. Yep, one of the main goals of PEP 384 is to identify and cordon off all of the APIs that are dependent on the version of the C runtime and exclude them from the stable ABI. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia --- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython
2010/1/30 exar...@twistedmatrix.com On 10:55 pm, collinwin...@google.com wrote: That people are directly munging CPython bytecode means that CPython should provide a better, more abstract way to do the same thing that's more resistant to these kinds of changes. It might be helpful to hear more about how the wordcode implementation differs from the bytecode implementation. It's challenging to abstract from a single data point. :) Jean-Paul Wordcodes structure is simple. You can find information herehttp://wpython2.googlecode.com/files/Beyond%20Bytecode%20-%20A%20Wordcode-based%20Python.pdf. Slide 6 provide a description, 7 details about the structure and some examples. Slide 9 explains how I mapped most bytecodes grouping in 6 families. However, wordcodes internals can be complicated to pretty print, because they may carry many information. You can take a look at opcode.hhttp://code.google.com/p/wpython2/source/browse/Include/opcode.h?repo=wpython10and dis.pyhttp://code.google.com/p/wpython2/source/browse/Lib/dis.py?repo=wpython10(function common_disassemble) to understand why this happens. Cesare ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython
2010/1/30 Scott Dial scott+python-...@scottdial.comscott%2bpython-...@scottdial.com Cesare, just FYI, your Hg repository has lost the execute bits on some files (namely ./configure and ./Parser/asdl_c.py), so it does not quite build out-of-the-box. That's probably because I worked on Windows. I have to address this issue. Thanks. I took the liberty of cloning your repo into my laptop's VirtualBox instance of Ubuntu. I ran the default performance tests from the U-S repo, with VirtualBox at highest priority. As a sanity check, I ran it against the U-S trunk. I think the numbers speak for themselves. -- Scott Dial I see. I don't know why you got those numbers. Until now, what I saw are better performances on average with wpython. In a previous mail, Collin stated that when they implemented wordcode con U-S they got benefits from the new opcode structure. May be more tests on different hardware / platforms will help. Cesare ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Forking and Multithreading - enemy brothers
Pascal Chambon, 29.01.2010 22:58: I've just recently realized the huge problems surrounding the mix of multithreading and fork() - i.e that only the main thread actually survived the fork(), and that process data (in particular, synchronization primitives) could be left in a dangerously broken state because of such forks, if multithreaded programs. I would *never* have even tried that, but it doesn't surprise me that it works basically as expected. I found this as a quick intro: http://unix.derkeiler.com/Newsgroups/comp.unix.programmer/2003-09/0672.html That post suggests that the behaviour you describe is somewhat platform specific, which was to be expected as well. I liked this quote, attributed to David Butenhof: it is not polite to say to a thread: fork you! FWIW, I'm against getting rid of fork() whenever wanted, that would just make things a lot more complex than necessary all over the place. But please consider providing a doc patch to the bug tracker that mentions the potential issues when people try that. (OTOH, most people won't try, so it might be better not to spread bad ideas.) Stefan ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Forking and Multithreading - enemy brothers
Stefan Behnel, 30.01.2010 07:36: Pascal Chambon, 29.01.2010 22:58: I've just recently realized the huge problems surrounding the mix of multithreading and fork() - i.e that only the main thread actually survived the fork(), and that process data (in particular, synchronization primitives) could be left in a dangerously broken state because of such forks, if multithreaded programs. I would *never* have even tried that, but it doesn't surprise me that it works basically as expected. I found this as a quick intro: http://unix.derkeiler.com/Newsgroups/comp.unix.programmer/2003-09/0672.html ... and another interesting link that also describes exec() usage in this context. http://www.linuxprogrammingblog.com/threads-and-fork-think-twice-before-using-them Stefan ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Restore the warning about mktemp now that DeprecationWarnings are silenced
I mean port it to py3k first and then backport it from trunk to 2.6 and from py3k to 3.1. If you want to wait then it's probably enough to add a note to the ticket and I'll fix it later. I haven't followed this discussion, but I'd like to point out that the only changes allowed for 2.6 are bug fixes. That a change is backwards compatible (in some sense) is not sufficient - there must also be a bug that gets fixed by backporting this change. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] patch to make list.pop(0) work in O(1) time
Josiah Carlson writes: Lisp lists are really stacks No, they're really (ie, concretely) singly-linked lists. Now, stacks are an abstract data type, and singly-linked lists provide an efficient implementation of stacks. But that's not what linked lists really are. For example, singly-linked lists are also a reasonable way to implement inverted trees (ie, the node knows its parent, but not its children), which is surely not a stack. The Python use of list to denote what is concretely a dynamically extensible one-dimensional array confused me a bit. But what the heck, Guido needed a four-letter word to denote a concrete type used to implement a mutable sequence ADT, and he wasn't going to borrow one from that French guy on the ramparts, right? No big deal. Ahem... So the confusion here is that in Python, list denotes a particular concrete data type, while Steve H. is using a more abstract idea of list as mutable sequence to suggest there's a reason for optimizing certain mutations that Python's data type isn't good at. I don't think that's an effective way for him to make his point, unfortunately. But both usages are consistent with Python's usage; mutability is the usual way that lists are distinguished from tuples, for example, and the underlying dynamic array implementation is rarely mentioned. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] patch to make list.pop(0) work in O(1) time
Josiah Carlson writes: On Thu, Jan 28, 2010 at 8:57 PM, Steve Howell showel...@yahoo.com wrote: What do you think of LISP, and car in particular (apart from the stupidly cryptic name)? Apples and oranges. True, but speaking of Lisp lists, here's some possibly relevant experience. About 10 years ago, XEmacs converted its cons type from a special immediate representation (ie, cons == (car, cdr)) to a generic record representation (ie, cons == (pointer to type descriptor, car, cdr)). This resulted in a perceptible increase in VM usage and disk usage. A typical running XEmacs instance for me contains about 0.75 million conses and uses 200MB of VM, so with 32-bit pointers that's about 3MB extra, or 1.5%, and with 64-bit pointers it's 6MB extra, about 3%. However, I tend to have several big buffers (20-50MB) of pure character data; people who work with smaller buffers on 64-bit machines have reported as much as 10% extra overhead. On disk, the binary is typically about 9MB stripped. That contains about 50,000 conses, or an extra 200KB/400KB with the new structure, somewhat more than my experience (2% or 4%). Some people complained, but we considered this well worthwhile (moving one type bit from the car to the header allowed Lisp integers to cover the range -1G to +1G, and there are a surprising number of people who would like to use XEmacs on files 512MB). I suppose that Steve's proposal probably has similar impact on binaries and running instances of Python, but he hasn't given any use cases for list.pop(0) to compared to doubling the size of usable buffers. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
