Re: [Python-Dev] PEP 3147, __cached__, and PyImport_ExecCodeModuleEx()
Am 11.04.2010 00:25, schrieb Barry Warsaw: > On Apr 10, 2010, at 08:28 PM, Georg Brandl wrote: > >>Am 10.04.2010 18:12, schrieb Guido van Rossum: >>> On Sat, Apr 10, 2010 at 8:58 AM, Barry Warsaw wrote: On Apr 09, 2010, at 05:41 PM, Guido van Rossum wrote: >On Fri, Apr 9, 2010 at 3:54 PM, Paul Moore wrote: >> Would it be better to name this one _PyImport_ExecCodeModuleExEx (with >> an underscore) so that we *don't* need to create an ExExEx version in >> future? (Sorry, Barry :-)) > >I don't care about what name you pick, and my ExEx proposal was meant >to include half a wink, but http://docs.python.org/c-api/import.html >makes it clear that PyImport_ExecCodeModuleEx() is far from private! >(I don't know where Barry got that idea.) Note that it's the non-Ex version that's documented here. AFAICT, PyImport_ExecCodeModuleEx() is not documented. I'm happy to fix that in my branch as well. >>> >>> Ah, true. And yes, please. >> >>*cough* http://docs.python.org/dev/c-api/import.html#PyImport_ExecCodeModuleEx >>Not backported to stable yet, though. > > 'dev' is Python 2.7 (i.e. trunk) right? I was looking at the py3k url, which > it hasn't been ported to either yet I think. Yes. It was added in response to a ticket very recently, and therefore is only on trunk yet. Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python-Dev Digest, Vol 81, Issue 31
On Sun Apr 11 03:23:21 CEST 2010 Terry Reedy wrote: > I believe speculative proposals like this fit better on the python-list or >python-ideas list. I see. Thank you :) Denis Kolodin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [RELEASED] 2.7 beta 1
On Sat, Apr 10, 2010 at 20:52, Benjamin Peterson wrote: > On behalf of the Python development team, I'm merry > to announce the first beta release of Python 2.7. Cool! -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [RELEASED] 2.7 beta 1
On Sat, Apr 10, 2010 at 11:57 AM, Antoine Pitrou wrote: > Benjamin Peterson python.org> writes: >> >> On behalf of the Python development team, I'm merry to announce the first >> beta >> release of Python 2.7. > > Congratulations, and thanks for your patience :) Congratulations indeed! Let me use this occasion to point out what an awesome community python-dev is. Not only do we have an great release manager (kudos to Benjamin for taking on this thankless job of herding cats), but the whole developer community has been contributing to some excellent releases. This makes me very happy and confident that Python will continue to thrive for many, many years, with or without my direct involvement in the details. While rumors of my retirement are greatly exaggerated, I am very glad to be able to leave the details to the community. Keep them coming, folks! -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3147
On Apr 11, 2010, at 02:39 AM, Nick Coghlan wrote: >Barry Warsaw wrote: >> I don't have strong feelings about this. I thought it would be handy for >> future Python's to have access to this, but then, without access to previous >> version magic numbers, it probably doesn't help much. And as you say, >> CPython >> won't know about alternative implementation's tags. >> >> So I'm willing to call YAGNI on it and just expose the current Python's magic >> tag. While we're at it, how about making both the tag and the number >> attributes of the imp module, instead of functions like .get_magic()? Of >> course we'd keep the latter for backward compatibility. > >I think one of the virtues of the functions is making it bleedingly >obvious to all concerned that these are read only values. > >So +1 to only exposing the current version of the implementation tag and >magic number, and +0 to doing so via attributes rather than functions. > >(I'm still in favour of keeping the list of old tags and magic numbers >in a source comment though - commenting them out rather than deleting >them when updating them isn't a major hassle). Thinking about this more, I've decided to add imp.get_tag() and remove imp.magic_tags. This avoids changes to the existing API, and just adds to it in a more consistent way. -Barry signature.asc Description: PGP signature ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] "Fixing" the new GIL
Hello all, I would like to kick this discussion back to life with a simplified implementation of the BFS scheduler, designed by the Linux kernel hacker Con Kolivas: http://ck.kolivas.org/patches/bfs/sched-BFS.txt I submitted bfs.patch at http://bugs.python.org/issue7946. It is work in progress but is ready for some opinion. On my machine BFS gives comparable performance to gilinter, and seems to schedule threads more fairly, predictably, and with lower rate of context switching. Its basic design is very simple but nevertheless it was designed by an expert in this field, two characteristics which combine to make it attractive to this case. The problem addressed by the GIL has always been *scheduling* threads to the interpreter, not just controlling access to it, and therefore the GIL, a lock implemented as a simple semaphore was the wrong solution. The patches by Antoine and David essentially evolve the GIL into a scheduler, however both cause thread starvation or high rate of context switching in some scenarios: With Floren't write test (http://bugs.python.org/issue7946#msg101120): 2 bg threads, 2 cores set to performance, karmic, PyCon patch, context switching shoots up to 200K/s. 2 bg threads, 1 core, set to on-demand, karmic, idle machine, gilinter patch starves one of the bg threads. 4 bg threads, 4x1 core xeon, centos 5.3, gilinter patch, all bg threads starved, context switching shoots up to 250K/s. With UDP test (http://bugs.python.org/file16316/udp-iotest.py), add zlib.compress(b'GIL') to the workload: both gilinter and PyCon patches starve the IO thread. The BFS patch currently involves more overhead by reading the time stamp on each yield and schedule operations. In addition it still remains to address some issues related to timestamps such as getting different time stamp readings on different cores on some (older) multi-core systems. Any thoughts? Nir On Sun, Mar 14, 2010 at 12:46 AM, Antoine Pitrou wrote: > > Hello, > > As some of you may know, Dave Beazley recently exhibited a situation > where the new GIL shows quite a poor behaviour (the old GIL isn't very > good either, but still a little better). This issue is followed in > http://bugs.python.org/issue7946 > > This situation is when an IO-bound thread wants to process a lot of > incoming packets, while one (or several) CPU-bound thread is also > running. Each time the IO-bound thread releases the GIL, the CPU-bound > thread gets it and keeps holding it for at least 5 milliseconds > (default setting), which limits the number of individual packets which > can be recv()'ed and processed per second. > > I have proposed two mechanisms, based on the same idea: IO-bound > threads should be able to steal the GIL very quickly, rather than > having to wait for the whole "thread switching interval" (again, 5 ms > by default). They differ in how they detect an "IO-bound threads": > > - the first mechanism is actually the same mechanism which was > embodied in the original new GIL patch before being removed. In this > approach, IO methods (such as socket.read() in socketmodule.c) > releasing the GIL must use a separate C macro when trying to get the > GIL back again. > > - the second mechanism dynamically computes the "interactiveness" of a > thread and allows interactive threads to steal the GIL quickly. In > this approach, IO methods don't have to be modified at all. > > Both approaches show similar benchmark results (for the benchmarks > that I know of) and basically fix the issue put forward by Dave Beazley. > > Any thoughts? > > Regards > > Antoine. > > > ___ > Python-Dev mailing list > [email protected] > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/nir%40winpdb.org > ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] "Fixing" the new GIL
Nir, Per the POSIX standard, both pthread_cond_wait() and pthread_cond_timedwait() need to be performed in a loop. See the fourth paragraph of the description from: > http://www.opengroup.org/onlinepubs/95399/functions/pthread_cond_timedwait > .html For the Windows side, I think you have a similar problem. Condition variables are signaling mechanisms, and so they have a separate boolean predicate associated with them. If you release the mutex that protects the predicate, then after you reacquire the mutex, you have to reevaluate the predicate to ensure that the condition has actually been met. You might want to look at the following for a discussion (not sure how good it is, as I just google¹d it quickly) of how to implement POSIX semantics on Windows: > http://www.cs.wustl.edu/~schmidt/win32-cv-1.html Before you can evaluate the effectiveness of any of the proposed scheduling schemes, the fundamental uses of mutexes and condition variables, and their implementations, must be sound. -peter On 4/11/10 6:50 PM, "Nir Aides" wrote: > Hello all, > > I would like to kick this discussion back to life with a simplified > implementation of the BFS scheduler, designed by the Linux kernel hacker Con > Kolivas: http://ck.kolivas.org/patches/bfs/sched-BFS.txt > > I submitted bfs.patch at http://bugs.python.org/issue7946. It is work in > progress but is ready for some opinion. > > On my machine BFS gives comparable performance to gilinter, and seems to > schedule threads more fairly, predictably, and with lower rate of context > switching. Its basic design is very simple but nevertheless it was designed by > an expert in this field, two characteristics which combine to make it > attractive to this case. > > The problem addressed by the GIL has always been *scheduling* threads to the > interpreter, not just controlling access to it, and therefore the GIL, a lock > implemented as a simple semaphore was the wrong solution. > > The patches by Antoine and David essentially evolve the GIL into a scheduler, > however both cause thread starvation or high rate of context switching in some > scenarios: > > With Floren't write test (http://bugs.python.org/issue7946#msg101120): > 2 bg threads, 2 cores set to performance, karmic, PyCon patch, context > switching shoots up to 200K/s. > 2 bg threads, 1 core, set to on-demand, karmic, idle machine, gilinter patch > starves one of the bg threads. > 4 bg threads, 4x1 core xeon, centos 5.3, gilinter patch, all bg threads > starved, context switching shoots up to 250K/s. > > With UDP test (http://bugs.python.org/file16316/udp-iotest.py), add > zlib.compress(b'GIL') to the workload: > both gilinter and PyCon patches starve the IO thread. > > The BFS patch currently involves more overhead by reading the time stamp on > each yield and schedule operations. In addition it still remains to address > some issues related to timestamps such as getting different time stamp > readings on different cores on some (older) multi-core systems. > > Any thoughts? > > Nir > > > > On Sun, Mar 14, 2010 at 12:46 AM, Antoine Pitrou wrote: >> >> Hello, >> >> As some of you may know, Dave Beazley recently exhibited a situation >> where the new GIL shows quite a poor behaviour (the old GIL isn't very >> good either, but still a little better). This issue is followed in >> http://bugs.python.org/issue7946 >> >> This situation is when an IO-bound thread wants to process a lot of >> incoming packets, while one (or several) CPU-bound thread is also >> running. Each time the IO-bound thread releases the GIL, the CPU-bound >> thread gets it and keeps holding it for at least 5 milliseconds >> (default setting), which limits the number of individual packets which >> can be recv()'ed and processed per second. >> >> I have proposed two mechanisms, based on the same idea: IO-bound >> threads should be able to steal the GIL very quickly, rather than >> having to wait for the whole "thread switching interval" (again, 5 ms >> by default). They differ in how they detect an "IO-bound threads": >> >> - the first mechanism is actually the same mechanism which was >> embodied in the original new GIL patch before being removed. In this >> approach, IO methods (such as socket.read() in socketmodule.c) >> releasing the GIL must use a separate C macro when trying to get the >> GIL back again. >> >> - the second mechanism dynamically computes the "interactiveness" of a >> thread and allows interactive threads to steal the GIL quickly. In >> this approach, IO methods don't have to be modified at all. >> >> Both approaches show similar benchmark results (for the benchmarks >> that I know of) and basically fix the issue put forward by Dave Beazley. >> >> Any thoughts? >> >> Regards >> >> Antoine. >> >> >> ___ >> Python-Dev mailing list >> [email protected] >> http://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> http://mail.
[Python-Dev] Traceback object has no __class__?
I thought type-class unification was supposed to mean that all objects now have a __class__ attribute. But traceback objects don't seem to: import sys try: raise ValueError except ValueError: tb = sys.exc_info()[2] print tb print tb.__class__ results in: % python2.6 traceback_class.py Traceback (most recent call last): File "traceback_class.py", line 8, in print tb.__class__ AttributeError: __class__ -- Greg ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Traceback object has no __class__?
Greg Ewing wrote: > I thought type-class unification was supposed to mean > that all objects now have a __class__ attribute. But > traceback objects don't seem to: > > import sys > > try: > raise ValueError > except ValueError: > tb = sys.exc_info()[2] > print tb > print tb.__class__ I'm not sure what build you're getting that behaviour on, but my svn build of 2.6 has a __class__ attribute for traceback objects, as does the kubuntu 9.10 python 2.6.4 system install. >>> try: ... raise ValueError ... except: ... import sys ... tb = sys.exc_info()[2] ... print type(tb) ... print dir(tb) ... print tb.__class__ ... ['__class__', '__delattr__', '__doc__', '__format__', '__getattribute__', '__hash__', '__init__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', 'tb_frame', 'tb_lasti', 'tb_lineno', 'tb_next'] Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia --- ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
