Re: [Python-Dev] Removing the GIL (Me, not you!)

2007-09-13 Thread Justin Tulloss
On 9/14/07, Adam Olsen <[EMAIL PROTECTED]> wrote: > > Could be worth a try. A first step might be to just implement > > the atomic refcounting, and run that single-threaded to see > > if it has terribly bad effects on performance. > > I've done this experiment. It was about 12% on my box. Later,

Re: [Python-Dev] Removing the GIL (Me, not you!)

2007-09-13 Thread Steve Holden
Hrvoje Nikšić wrote: > On Thu, 2007-09-13 at 13:15 +0200, "Martin v. Löwis" wrote: >>> To put it another way, would it actually matter if the reference >>> counts for such objects became hopelessly wrong due to non-atomic >>> adjustments? >> If they drop to zero (which may happen due to non-atomic

Re: [Python-Dev] Removing the GIL (Me, not you!)

2007-09-13 Thread Adam Olsen
On 9/13/07, Greg Ewing <[EMAIL PROTECTED]> wrote: > [EMAIL PROTECTED] wrote: > > what if ... we use atomic test-and-set to > > handle reference counting (with a lock for those CPU architectures where we > > haven't written the necessary assembler fragment), then implement a lock for > > each mutabl

Re: [Python-Dev] Removing the GIL (Me, not you!)

2007-09-13 Thread Justin Tulloss
On 9/13/07, Greg Ewing <[EMAIL PROTECTED]> wrote: > > Jason Orendorff wrote: > > The clever bit is that SpiderMonkey's per-object > > locking does *not* require a context switch or even an atomic > > instruction, in the usual case where an object is *not* shared among > > threads. > > How does it t

Re: [Python-Dev] Removing the GIL (Me, not you!)

2007-09-13 Thread Tennessee Leeuwenburg
Pardon me for talking with no experience in such matters, but... Okay, incrementing a reference counter is atomic, therefore the cheapest possible operation. Is it possible to keep reference counting atomic in a multi-thread model? Could you do the following... let's consider two threads, "A" and

Re: [Python-Dev] Removing the GIL (Me, not you!)

2007-09-13 Thread Greg Ewing
Jason Orendorff wrote: > The clever bit is that SpiderMonkey's per-object > locking does *not* require a context switch or even an atomic > instruction, in the usual case where an object is *not* shared among > threads. How does it tell whether an object is shared between threads? That sounds like

Re: [Python-Dev] Removing the GIL (Me, not you!)

2007-09-13 Thread Greg Ewing
Prateek Sureka wrote: > Naturally, we need to make the locking more > fine-grained to resolve this. Hopefully we can do so in a way that > does not increase the lock overhead (hence my suggestion for a lock > free approach using an asynch queue and a core as dedicated server). What you don't

Re: [Python-Dev] Removing the GIL (Me, not you!)

2007-09-13 Thread Greg Ewing
[EMAIL PROTECTED] wrote: > what if ... we use atomic test-and-set to > handle reference counting (with a lock for those CPU architectures where we > haven't written the necessary assembler fragment), then implement a lock for > each mutable type and another for global state (thread state, interpret

Re: [Python-Dev] Removing the GIL (Me, not you!)

2007-09-13 Thread Greg Ewing
Jon Ribbens wrote: > To put it another way, would it actually matter if the reference > counts for such objects became hopelessly wrong due to non-atomic > adjustments? Again, it would cost time to check whether you could get away with doing non-atomic refcounting. If you're thinking that no chec

Re: [Python-Dev] Removing the GIL (Me, not you!)

2007-09-13 Thread Greg Ewing
Christian Heimes wrote: > Pardon my ignorance but why does Python do reference counting for truly > global and static objects Because it would cost more time to check whether the reference counting needed to be done than to just do it anyway. Remember that *most* refcount operations are on non-gl

Re: [Python-Dev] Removing the GIL (Me, not you!)

2007-09-13 Thread Jon Ribbens
On Thu, Sep 13, 2007 at 06:38:05PM -0500, [EMAIL PROTECTED] wrote: > Hrvoje> More precisely, Python will call the deallocator appropriate for > Hrvoje> the object type. If that deallocator does nothing, the object > Hrvoje> continues to live. Such objects could also start out with a >

Re: [Python-Dev] Removing the GIL (Me, not you!)

2007-09-13 Thread skip
Hrvoje> More precisely, Python will call the deallocator appropriate for Hrvoje> the object type. If that deallocator does nothing, the object Hrvoje> continues to live. Such objects could also start out with a Hrvoje> refcount of sys.maxint or so to ensure that calls to the no-o

Re: [Python-Dev] Windows package for new SSL package?

2007-09-13 Thread Mark Hammond
> You don't need VS and mingw binary installers, though - the mingw ones > will work for any Python (ignoring specialised custom builds, and > anyone doing one of them is probably capable of building the ssl > module!). Why I appreciate your points about building the extension with free tools, wou

Re: [Python-Dev] Removing the GIL (Me, not you!)

2007-09-13 Thread Greg Ewing
Martin v. Löwis wrote: > Now we are getting into details: you do NOT have to lock > an object to modify its reference count. An atomic > increment/decrement operation is enough. I stand corrected. But if it were as simple as that, I think it would have been done by now. I got the impression that

Re: [Python-Dev] Removing the GIL (Me, not you!)

2007-09-13 Thread Justin Tulloss
On 9/13/07, Adam Olsen <[EMAIL PROTECTED]> wrote: > > > Basically though, atomic incref/decref won't work. Once you've got > two threads modifying the same location the costs skyrocket. Even > without being properly atomic you'll get the same slowdown on x86 > (who's cache coherency is fairly str

Re: [Python-Dev] Windows package for new SSL package?

2007-09-13 Thread Paul Moore
On 13/09/2007, Bill Janssen <[EMAIL PROTECTED]> wrote: > > I could build some Windows installers if you want, but I'd need to > > download and install some extra versions of Python, so you'd have to > > tell me which you want doing (and I can't offer to commit to doing > > this on a regular basis..

Re: [Python-Dev] Windows package for new SSL package?

2007-09-13 Thread Bill Janssen
> I could build some Windows installers if you want, but I'd need to > download and install some extra versions of Python, so you'd have to > tell me which you want doing (and I can't offer to commit to doing > this on a regular basis...) Thanks, but let's wait till this settles down a bit (say, a

Re: [Python-Dev] Removing the GIL (Me, not you!)

2007-09-13 Thread Justin Tulloss
On 9/13/07, Jason Orendorff <[EMAIL PROTECTED]> wrote: > > On 9/13/07, Justin Tulloss <[EMAIL PROTECTED]> wrote: > > 1. Use message passing and transactions. [...] > > 2. Do it perl style. [...] > > 3. Come up with an elegant way of handling multiple python processes. > [...] > > 4. Remove the GIL

Re: [Python-Dev] Windows package for new SSL package?

2007-09-13 Thread Paul Moore
On 13/09/2007, Bill Janssen <[EMAIL PROTECTED]> wrote: > > Anyway, philosophy aside, I'll try to make some time in the next few > > days to get a working setup.py for the SSL package using mingw. > > Hopefully, Bill will then integrate this and we'll have mingw as a > > supported option. > > I'll b

Re: [Python-Dev] Removing the GIL (Me, not you!)

2007-09-13 Thread Adam Olsen
On 9/13/07, Justin Tulloss <[EMAIL PROTECTED]> wrote: > > > On 9/13/07, Adam Olsen <[EMAIL PROTECTED]> wrote: > > > > Basically though, atomic incref/decref won't work. Once you've got > > two threads modifying the same location the costs skyrocket. Even > > without being properly atomic you'll g

Re: [Python-Dev] [Tracker-discuss] Python tickets summary

2007-09-13 Thread Raghuram Devarakonda
On 9/13/07, Facundo Batista <[EMAIL PROTECTED]> wrote: > http://www.taniquetil.com.ar/facundo/py_tickets.html It looks like the column "Opened by" contains information for "Last update by" and vice versa. At least, that is the case with issue 1159. Thanks, Raghu ___

Re: [Python-Dev] Python tickets summary

2007-09-13 Thread Facundo Batista
2007/9/10, Facundo Batista <[EMAIL PROTECTED]>: > I modified my tool, whichs makes a summary of all the Python tickets > (I moved the source where the info is taken from SF to our Roundup). > > In result, the summary is now, again, updated daily: Taking an idea from Jeff Rush, now there're separa

Re: [Python-Dev] base64 -- should b64encode introduce line breaks?

2007-09-13 Thread Bill Janssen
> I see that base64.b64encode and base64.standard_b64encode no longer > introduce line breaks into the output strings, as base64.encodestring > does. Shouldn't there be an option on one of them to do this? See: http://mail.python.org/pipermail/python-bugs-list/2001-October/007856.html section 2

[Python-Dev] base64 -- should b64encode introduce line breaks?

2007-09-13 Thread Bill Janssen
I see that base64.b64encode and base64.standard_b64encode no longer introduce line breaks into the output strings, as base64.encodestring does. Shouldn't there be an option on one of them to do this? Bill ___ Python-Dev mailing list Python-Dev@python.or

Re: [Python-Dev] Removing the GIL (Me, not you!)

2007-09-13 Thread Jason Orendorff
On 9/13/07, Justin Tulloss <[EMAIL PROTECTED]> wrote: > 1. Use message passing and transactions. [...] > 2. Do it perl style. [...] > 3. Come up with an elegant way of handling multiple python processes. [...] > 4. Remove the GIL, use transactions for python objects, [...] The SpiderMonkey JavaSc

Re: [Python-Dev] SSL certs

2007-09-13 Thread Martin v. Löwis
>> However, there is an alternative to using multiple IP addresses: >> one could also use multiple "subject alternative names", and create >> a certificate that lists them all. > > Unfortunately, much of the client code that does the hostname > verification is wrapped up in gullible Web browsers o

Re: [Python-Dev] SSL certs

2007-09-13 Thread Bill Janssen
> However, there is an alternative to using multiple IP addresses: > one could also use multiple "subject alternative names", and create > a certificate that lists them all. Unfortunately, much of the client code that does the hostname verification is wrapped up in gullible Web browsers or Java HT

Re: [Python-Dev] Removing the GIL (Me, not you!)

2007-09-13 Thread Adam Olsen
On 9/13/07, Hrvoje Nikšić <[EMAIL PROTECTED]> wrote: > On Thu, 2007-09-13 at 13:15 +0200, "Martin v. Löwis" wrote: > > > To put it another way, would it actually matter if the reference > > > counts for such objects became hopelessly wrong due to non-atomic > > > adjustments? > > > > If they drop t

Re: [Python-Dev] Removing the GIL (Me, not you!)

2007-09-13 Thread Martin v. Löwis
> http://www.artima.com/weblogs/viewpost.jsp?thread=214235) that the > slowdown was 2x in a single threaded application (which couldn't be due > to lock contention), it must be due to lock overhead (unless the > programming was otherwise faulty or there is something else about locks > that I don't

Re: [Python-Dev] Removing the GIL (Me, not you!)

2007-09-13 Thread Prateek Sureka
On Sep 13, 2007, at 9:25 PM, Martin v. Löwis wrote: >>> "Since we are guaranteeing that synchronized code is running on a >>> single >>> core, it is the equivalent of a lock at the cost of a context >>> switch." >>> >>> This is precisely what a lock costs today: a context switch. >>> >> >> Re

Re: [Python-Dev] Windows package for new SSL package?

2007-09-13 Thread Bill Janssen
> Anyway, philosophy aside, I'll try to make some time in the next few > days to get a working setup.py for the SSL package using mingw. > Hopefully, Bill will then integrate this and we'll have mingw as a > supported option. I'll be happy to do that! Bill

Re: [Python-Dev] Windows package for new SSL package?

2007-09-13 Thread Bill Janssen
> In other words, both the standard and your extension module on Windows > bring along their own OpenSSL. I see -- thanks. Bill ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mai

Re: [Python-Dev] Removing the GIL (Me, not you!)

2007-09-13 Thread Martin v. Löwis
>> "Since we are guaranteeing that synchronized code is running on a single >> core, it is the equivalent of a lock at the cost of a context switch." >> >> This is precisely what a lock costs today: a context switch. >> > > Really? Wouldn't we save some memory allocation overhead (since in my > de

Re: [Python-Dev] Removing the GIL (Me, not you!)

2007-09-13 Thread Prateek Sureka
On Sep 13, 2007, at 10:12 AM, Martin v. Löwis wrote: >> What do you think? > > I think what you are describing is the situation of today, > except in a less-performant way. The kernel *already* > implements such a "synchronization server", except that > all CPUs can act as such. You write > > "Si

Re: [Python-Dev] Removing the GIL (Me, not you!)

2007-09-13 Thread Hrvoje Nikšić
On Thu, 2007-09-13 at 13:15 +0200, "Martin v. Löwis" wrote: > > To put it another way, would it actually matter if the reference > > counts for such objects became hopelessly wrong due to non-atomic > > adjustments? > > If they drop to zero (which may happen due to non-atomic adjustments), > Pytho

Re: [Python-Dev] Removing the GIL (Me, not you!)

2007-09-13 Thread skip
Jon> To put it another way, would it actually matter if the reference Jon> counts for such objects became hopelessly wrong due to non-atomic Jon> adjustments? I believe this was suggested and tried by someone (within the last few years). It wasn't any benefit. The costs of special-c

Re: [Python-Dev] Windows package for new SSL package?

2007-09-13 Thread Paul Moore
On 13/09/2007, David Bolen <[EMAIL PROTECTED]> wrote: > That's a fair point - my comments are all related to the standard > Python distribution and building extensions with the VS.NET compiler > (including the binary installer I had built for Bill). [...] > If we're talking about the construction o

Re: [Python-Dev] Removing the GIL (Me, not you!)

2007-09-13 Thread Jon Ribbens
On Thu, Sep 13, 2007 at 01:15:39PM +0200, "Martin v. Löwis" wrote: > > To put it another way, would it actually matter if the reference > > counts for such objects became hopelessly wrong due to non-atomic > > adjustments? > > If they drop to zero (which may happen due to non-atomic adjustments),

Re: [Python-Dev] Removing the GIL (Me, not you!)

2007-09-13 Thread Martin v. Löwis
> To put it another way, would it actually matter if the reference > counts for such objects became hopelessly wrong due to non-atomic > adjustments? If they drop to zero (which may happen due to non-atomic adjustments), Python will try to release the static memory, which will crash the malloc imp

Re: [Python-Dev] Windows package for new SSL package?

2007-09-13 Thread David Bolen
On 9/13/07, Paul Moore <[EMAIL PROTECTED]> wrote: > On 13/09/2007, David Bolen <[EMAIL PROTECTED]> wrote: > > I think where there's probably a small disconnect here is that, there > > really isn't an OpenSSL "installed" on the end user's machine. Well, > > there could be, but Python isn't using i

Re: [Python-Dev] Removing the GIL (Me, not you!)

2007-09-13 Thread Jon Ribbens
On Thu, Sep 13, 2007 at 12:19:21PM +0200, André Malo wrote: > > Pardon my ignorance but why does Python do reference counting for truly > > global and static objects like None, True, False, small and cached > > integers, sys and other builtins? If I understand it correctly these > > objects are nev

Re: [Python-Dev] Windows package for new SSL package?

2007-09-13 Thread Paul Moore
On 13/09/2007, David Bolen <[EMAIL PROTECTED]> wrote: > Bill Janssen <[EMAIL PROTECTED]> writes: > > > In that case, I think your idea of just hard-coding a path is probably > > the right thing to do. I'll add a note that this is how you need to do > > it if you are going to try "python setup.py b

Re: [Python-Dev] Windows package for new SSL package?

2007-09-13 Thread Paul Moore
On 13/09/2007, David Bolen <[EMAIL PROTECTED]> wrote: > "Mark Hammond" <[EMAIL PROTECTED]> writes: > > > It might be possible to try and use build_ssl.py to locate the openssl > > directory, but this will still require that someone building it has Python > > built from source - I'm fairly sure that

Re: [Python-Dev] Removing the GIL (Me, not you!)

2007-09-13 Thread André Malo
* Christian Heimes wrote: > Pardon my ignorance but why does Python do reference counting for truly > global and static objects like None, True, False, small and cached > integers, sys and other builtins? If I understand it correctly these > objects are never garbaged collected (at least they sho

Re: [Python-Dev] Removing the GIL (Me, not you!)

2007-09-13 Thread Christian Heimes
Phillip J. Eby wrote: > It's not just caches and counters. It's also every built-in type > structure, builtin module, builtin function... any Python object > that's a built-in, period. That includes things like None, True, and False. > > Caches would include such things as the pre-created int

Re: [Python-Dev] SSL certs

2007-09-13 Thread Martin v. Löwis
> My understanding is that the client tells the server which hostname it > wants to use; the server should then pass down that information. That's > how virtual hosting works in the first place. The only difference with > SSL is that the hostname must have a unique IP address, so that when the >

Re: [Python-Dev] Removing the GIL (Me, not you!)

2007-09-13 Thread Justin Tulloss
> What do you think? > I'm going to have to agree with Martin here, although I'm not sure I understand what you're saying entirely. Perhaps if you explained where the benefits of this approach come from, it would clear up what you're thinking. After a few days of thought, I'm starting to realize