Re: [Python-Dev] "Fixing" the new GIL

[Martin v. Löwis]

Cameron Simpson wrote:
> On 15Mar2010 09:28, Martin v. L�wis  wrote:
> | > As for the argument that an application with cpu intensive work being
> | > driven by the IO itself will work itself out...  No it won't, it can
> | > get into beat patterns where it is handling requests quite rapidly up
> | > until one that causes a long computation to start comes in.  At that
> | > point it'll stop performing well on other requests for as long (it
> | > could be a significant amount of time) as the cpu intensive request
> | > threads are running.  That is not a graceful degration in serving
> | > capacity / latency as one would normally expect.  It is a sudden drop
> | > off.
> | 
> | Why do you say that? The other threads continue to be served - and
> | Python couldn't use more than one CPU, anyway. Can you demonstrate that
> | in an example?
> 
> Real example:

... unfortunately without a demonstration. What's the throughput under
the old GIL? What's the throughput of this application under the new
GIL? How can I observe the "beat pattern"?

> The idea here is that one has a few threads receiving requests (eg a
> daemon watching a socket or monitoring a db queue table) which then use
> the FuncMultiQueue to manage how many actual requests are processed
> in parallel (yes, a semaphore can cover a lot of this, but not the
> asynchronous call modes).

Why do you say processing is in parallel? In Python, processing is
normally never in parallel, but always sequential (potentially
interleaving). Are you releasing the GIL for processing?

> So, suppose a couple of CPU-intensive callables get queued which work for a
> substantial time, and meanwhile a bunch of tiny tiny cheap requests arrive.
> Their timely response will be impacted by this issue.

By how much exactly? What operating system?

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] "Fixing" the new GIL

[Martin v. Löwis]

> If it's lurking behind a filesystem interface or in its daemon mode
> (remote archive store), multiple client processes can be using it at once,
> and it will be processing multiple tasks somewhat in parallel. Here one
> can get a compute bound thread answering one request, impacting quick
> response to other parallel-and-cheap requests.

However, "impacting" will always be the case. *Of course* any thread
that performs computation impacts everything else on the same processor.
There is nothing to prevent that, unless you schedule the long-running
activity at a point in time where you know the system will be idle for
the entire run of that task.

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] "Fixing" the new GIL

[Cameron Simpson]

On 16Mar2010 08:59, "Martin v. Löwis"  wrote:
| Cameron Simpson wrote:
| > On 15Mar2010 09:28, Martin v. L�wis  wrote:
| > | > As for the argument that an application with cpu intensive work being
| > | > driven by the IO itself will work itself out...  No it won't, it can
| > | > get into beat patterns where it is handling requests quite rapidly up
| > | > until one that causes a long computation to start comes in.  At that
| > | > point it'll stop performing well on other requests for as long (it
| > | > could be a significant amount of time) as the cpu intensive request
| > | > threads are running.  That is not a graceful degration in serving
| > | > capacity / latency as one would normally expect.  It is a sudden drop
| > | > off.
| > | 
| > | Why do you say that? The other threads continue to be served - and
| > | Python couldn't use more than one CPU, anyway. Can you demonstrate that
| > | in an example?
| > 
| > Real example:
| 
| ... unfortunately without a demonstration. What's the throughput under
| the old GIL? What's the throughput of this application under the new
| GIL? How can I observe the "beat pattern"?

You can't. This isn't an example where I am personally bitten by the GIL.
I may be, but there's plenty of other stuff in terms of tuning or
optimisation in my particular app before I get anywhere near the GIL.

My point is not that it's biting me right now but that earlier you
said the scenario was probably unrealistic, but I have an app which can
probably exhibit exactly the issue under discussion.

| > The idea here is that one has a few threads receiving requests (eg a
| > daemon watching a socket or monitoring a db queue table) which then use
| > the FuncMultiQueue to manage how many actual requests are processed
| > in parallel (yes, a semaphore can cover a lot of this, but not the
| > asynchronous call modes).
| 
| Why do you say processing is in parallel?
| In Python, processing is
| normally never in parallel, but always sequential (potentially
| interleaving). Are you releasing the GIL for processing?


I mean that I can have multiple actual requests being served, and they
are _not_ handled sequentially - the multi queue is to permit more than
one to be in progress at once so that an expensive request need not
inherently delay a following cheap request.

And in my it's-a-filesystem mode there will often be a bottleneck in the
pure-python data scanning part. And thus the CPU bound python active
while still accepting requests in I/O blocked threads.

| > So, suppose a couple of CPU-intensive callables get queued which work for a
| > substantial time, and meanwhile a bunch of tiny tiny cheap requests arrive.
| > Their timely response will be impacted by this issue.
| 
| By how much exactly? What operating system?

Can't tell you - I'm not claiming the current GIL behaviour is biting me
right now; I'm saying your claim that the scenario is unrealistic is a
bit unfair; my app wil probably be just the kind of app to be affected,
even only subtly. And it can hardly be alone in the "daemon" world.

Cheers,
-- 
Cameron Simpson  DoD#743
http://www.cskk.ezoshosting.com/cs/

There's two kinds of climbers...smart ones, and dead ones.  - Don Whillans
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] "Fixing" the new GIL

[Cameron Simpson]

On 16Mar2010 09:02, "Martin v. Löwis"  wrote:
| > If it's lurking behind a filesystem interface or in its daemon mode
| > (remote archive store), multiple client processes can be using it at once,
| > and it will be processing multiple tasks somewhat in parallel. Here one
| > can get a compute bound thread answering one request, impacting quick
| > response to other parallel-and-cheap requests.
| 
| However, "impacting" will always be the case. *Of course* any thread
| that performs computation impacts everything else on the same processor.
| There is nothing to prevent that, unless you schedule the long-running
| activity at a point in time where you know the system will be idle for
| the entire run of that task.

Sure, but one has only to dip one's toe in the long running Linux
interactivity scheduling wars to know that there's a _lot_ of attraction
in a scheduling approach that favours the usually-blocked task over the
CPU bound task, and various approaches like the early low-latency patches
to the later in-the-scheduler changes were greeted with much rejoicing.

Certainly there's only so much CPU to go around, but if the tasks blocked
waiting for something to happen get to run more immediately when an event
does occur, you tend to get better "responsiveness".

Cheers,
-- 
Cameron Simpson  DoD#743
http://www.cskk.ezoshosting.com/cs/

Be smart, be safe, be paranoid.
- Ryan Cousineau, cour...@compdyn.com DoD#863, KotRB, KotKWaWCRH
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] __file__ and bytecode-only

[Nick Coghlan]

Barry Warsaw wrote:
> On Mar 14, 2010, at 12:17 AM, Nick Coghlan wrote:
>> While it's probably OK if the import side-effects only create files
>> using the new scheme, the standard library modules will likely need to
>> support both schemes (although I'm not sure if "same as import system"
>> or "same as Python 3.1" make more sense as the default semantics -
>> probably the former).
> 
> I don't understand this point.
> 
> compileall probably /could/ be extended to understand bytecode-only
> (i.e. legacy or <3.2) layout.  I've added that to the PEP too.

In Python 3.1, *invoking* py_compile.compile() will create 2.x style
bytecode. Similarly, when force==False, compileall.compile_dir() and
compileall.compile_path() will check for 2.x style bytecode in order to
decide whether or not to compile the module.

The question for 3.2 is what bytecode layout py_compile.compile() should
generate. For the precompile-a-system-library use case it should clearly
generate a PEP 3147 layout and this probably makes sense as the default
behaviour in 3.2.

However, for production of bytecode-only packages, it would be
convenient to be able to explicitly invoke the 2.x style behaviour
without having to specify the target filename explicitly using the
'cfile' parameter (which isn't exposed at the compileall layer anyway).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
---
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] "Fixing" the new GIL

[Nick Coghlan]

Martin v. Löwis wrote:
> Cameron Simpson wrote:
>> The idea here is that one has a few threads receiving requests (eg a
>> daemon watching a socket or monitoring a db queue table) which then use
>> the FuncMultiQueue to manage how many actual requests are processed
>> in parallel (yes, a semaphore can cover a lot of this, but not the
>> asynchronous call modes).
> 
> Why do you say processing is in parallel? In Python, processing is
> normally never in parallel, but always sequential (potentially
> interleaving). Are you releasing the GIL for processing?

We know the GIL is timeslicing - that's the whole point. The real
question is *how often* is it time slicing and *which threads* are
getting to run.

The key issue is that the current new GIL implementation means that, in
the presence of a CPU bound GIL-holding thread, the *minimum* duration
of *any* C call that releases the GIL becomes sys.getcheckinterval().
The CPU bound thread *always* wants the GIL and once it gets it, it
won't give it back until sys.getcheckinterval() expires. This is in
contrast to the I/O bound (or non-GIL holding CPU bound) threads that
nearly always yield the GIL early, since they're waiting for something
else to happen (be it I/O or a native calculation).

Antoine's interactiveness patch resolves this by rewarding threads that
regularly release the GIL early: the interpreter quickly recognises them
as doing so and they are subsequently given priority over the threads
that only release the GIL when the check interval expires. When threads
that are flagged as interactive ask for the GIL back, the interpreter
gives it to them immediately instead of forcing them to wait until the
check interval expires (because it trusts those threads not to hang onto
the GIL for too long).

I like this significantly better than the explicit priority patch,
because it means threads are given priority based on their behaviour
(i.e. frequently holding the GIL for less than the check interval)
rather than relying on developers to correctly decide between using a
normal GIL request and a priority request.

Handling of thread pools that mix I/O bound and CPU bound GIL-holding
tasks isn't quite optimal with this approach (since the nature of any
given thread will change over time), but it isn't terrible either - the
interpreter doesn't need much time to reclassify a thread as interactive
or non-interactive (and there are a couple of parameters that can be
tuned to control how quickly these changes in status can happen).

Regards,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
---
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] "Fixing" the new GIL

[Kristján Valur Jónsson]

How about attacking the original problem, then?

The reason they thrash on pthreads implementation is that a pthreads mutex is 
assumed to be a short-held resource.  Therefore it will be optimized in the 
following ways for multicore machines:
1) There is a certain amount of spinning done, to try to acquire it before 
blocking
2) It will employ un-fair tactics to avoid lock-convoying, meaning that a 
thread coming in to acquire the mutex may get in before others that are queued. 
 This is why "ticking" the GIL works so badly:  The thread that releases the 
lock is usually the one that reaquires it even though others may be waiting.  
See e.g. 
http://www.bluebytesoftware.com/blog/PermaLink,guid,e40c2675-43a3-410f-8f85-616ef7b031aa.aspx
 for a discussion of this (albeit on windows.)

On Windows, this isn't a problem.  The reason is, that the GIL on windows is 
implemented using Event objects that don't cut these corners.  The Event 
provides you with a strict FIFO queue of objects waiting for the event.

If pthreads doesn't provide a synchronization primitive similar to that, 
someone that doesn't thrash and has a true FIFO queue, it is possible to 
construct such a thing using condition variables and critical sections.  
Perhaps the posix semaphore api is more appropriate in this case.

By the way, this also shows another problem with (old) python.  There is only 
one core locking primitive, the PyThread_type_lock.  It is being used both as a 
critical section in the traditional sense, and also as this sort-of-inverse 
lock that the GIL is.  In the modern world, where the intended behaviour of 
these is quite different, there is no one-size-fits all.  On windows in 
particular, the use of the Event object based lock is not ideal for other uses 
than the GIL.


In the new GIL, there appear to be several problems:
1) There is no FIFO queue of threads wanting the queue, thus thread scheduling 
becomes non-deterministic
2) The "ticking" of the GIL is now controled by a condition variable timeout.  
There appears to be no way to prevent many such timeouts to be in progress at 
the same time, thus you may have an unnecessarily high rate of ticking going on.
3) There isn't an immediate gil request made when an IO thread requests the gil 
back, only after an initial timeout.

What we are trying to write here is a thread scheduler, and that is complex 
business.
K



> -Original Message-
> From: python-dev-bounces+kristjan=ccpgames@python.org
> [mailto:python-dev-bounces+kristjan=ccpgames@python.org] On Behalf
> Of David Beazley
> Sent: 15. mars 2010 03:07
> To: python-dev@python.org
> Subject: Re: [Python-Dev] "Fixing" the new GIL
> 
> happen to be performing CPU intensive work at the same time, it would
> be nice if they didn't thrash on multiple cores (the problem with the
> old GIL) and if I/O is

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] "Fixing" the new GIL

[David Beazley]

Python doesn't use a pthreads mutex for the GIL.It has always used a binary 
semaphore implemented with condition variables (or just a pthreads semaphore if 
available).The reason the performance is so bad is precisely due to the 
fact that it is using this implementation and the fact that there *IS* a FIFO 
queue of threads (associated with the condition variable).   The I/O 
performance problem with the new GIL is gets much worse with many CPU-bound 
threads precisely because there is a FIFO queue involved.   This has been 
covered in my past GIL presentations.

-Dave



On Mar 16, 2010, at 5:52 AM, Kristján Valur Jónsson wrote:

> How about attacking the original problem, then?
> 
> The reason they thrash on pthreads implementation is that a pthreads mutex is 
> assumed to be a short-held resource.  Therefore it will be optimized in the 
> following ways for multicore machines:
> 1) There is a certain amount of spinning done, to try to acquire it before 
> blocking
> 2) It will employ un-fair tactics to avoid lock-convoying, meaning that a 
> thread coming in to acquire the mutex may get in before others that are 
> queued.  This is why "ticking" the GIL works so badly:  The thread that 
> releases the lock is usually the one that reaquires it even though others may 
> be waiting.  See e.g. 
> http://www.bluebytesoftware.com/blog/PermaLink,guid,e40c2675-43a3-410f-8f85-616ef7b031aa.aspx
>  for a discussion of this (albeit on windows.)
> 
> On Windows, this isn't a problem.  The reason is, that the GIL on windows is 
> implemented using Event objects that don't cut these corners.  The Event 
> provides you with a strict FIFO queue of objects waiting for the event.
> 
> If pthreads doesn't provide a synchronization primitive similar to that, 
> someone that doesn't thrash and has a true FIFO queue, it is possible to 
> construct such a thing using condition variables and critical sections.  
> Perhaps the posix semaphore api is more appropriate in this case.
> 
> By the way, this also shows another problem with (old) python.  There is only 
> one core locking primitive, the PyThread_type_lock.  It is being used both as 
> a critical section in the traditional sense, and also as this sort-of-inverse 
> lock that the GIL is.  In the modern world, where the intended behaviour of 
> these is quite different, there is no one-size-fits all.  On windows in 
> particular, the use of the Event object based lock is not ideal for other 
> uses than the GIL.
> 
> 
> In the new GIL, there appear to be several problems:
> 1) There is no FIFO queue of threads wanting the queue, thus thread 
> scheduling becomes non-deterministic
> 2) The "ticking" of the GIL is now controled by a condition variable timeout. 
>  There appears to be no way to prevent many such timeouts to be in progress 
> at the same time, thus you may have an unnecessarily high rate of ticking 
> going on.
> 3) There isn't an immediate gil request made when an IO thread requests the 
> gil back, only after an initial timeout.
> 
> What we are trying to write here is a thread scheduler, and that is complex 
> business.
> K
> 
> 
> 
>> -Original Message-
>> From: python-dev-bounces+kristjan=ccpgames@python.org
>> [mailto:python-dev-bounces+kristjan=ccpgames@python.org] On Behalf
>> Of David Beazley
>> Sent: 15. mars 2010 03:07
>> To: python-dev@python.org
>> Subject: Re: [Python-Dev] "Fixing" the new GIL
>> 
>> happen to be performing CPU intensive work at the same time, it would
>> be nice if they didn't thrash on multiple cores (the problem with the
>> old GIL) and if I/O is
> 

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] "Fixing" the new GIL

[Antoine Pitrou]

Hi Kristján,

> In the new GIL, there appear to be several problems:
> 1) There is no FIFO queue of threads wanting the queue, thus thread
> scheduling becomes non-deterministic

Thread scheduling has always been non-deterministic.

> 2) The "ticking" of the GIL is
> now controled by a condition variable timeout.  There appears to be
> no way to prevent many such timeouts to be in progress at the same
> time, thus you may have an unnecessarily high rate of ticking going
> on.

Unless there's a bug, no, there isn't.

> 3) There isn't an immediate gil request made when an IO thread
> requests the gil back, only after an initial timeout.

This is what we are looking to fix (perhaps).

> What we are trying to write here is a thread scheduler, and that is
> complex business. K

I would have rephrased:
"What we are trying to avoid here is a thread scheduler".

regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] "Fixing" the new GIL

[Kristján Valur Jónsson]

You are right, I was assuming a mutex. (why not then, use a semaphore?)
But the implementation isn't a simple binary semaphore.
It suffers from case 2) that I explained before:  Greedy aquisition of the 
"semaphore" even if someone is waiting.
Also, I should add, a condition variable doesn't specify that it employs a 
FIFO.  pythread_cond_signals() wakes up _a_ thread.  If you want particular 
behaviour then you have to add code to deal with that.  But it tries to be 
fair, and we can probably assum e FIFO behavioud, so let's ignore that part.  
So, in the following, let's assume that the condition variable is fair, and 
that it queues up the waiting threads in a FIFO manner:

Fixing 2) in this case is easy: Make sure you wait if others are waiting.  In 
python pseudocode:
def Aquire(gil):
  with gil.condition:
if gil.locked or gil.n_waiting:
  gil.n_waiting+=1
while gil.locked:
  gil.condition.wait()
  gil.n_waiting-=1
gil.locked = True

def Release(gil):
  with gil.condition:
gil.locked=False
if gil.n_waiting:
  gil.condition.notify()

This ought to fix problem 2) below.  Multicore CPUs ought to not behave worse 
than single core anymore.
(Btw, the C code is erroneous, one should not call pthread_cond_signal without 
having the lock held, but that has been brought up before.)

Now, if you want to improve IO performance, by ensuring that IO threads get 
woken up before (yielding) cpu threads, you can do something like this:
class GIL():
  def __init__(self, npri):
self.lock=RLock()
self.queues = [[Condition(self.lock), 0] for i in range(npri)]
self.locked = False

  def Acquire(self, pri):
"""Acquire the lock with priority "pri", where 0 is highest priority"""
with self.lock:
if self.locked or sum([q[1] for q in self.queues[:pri+1]]) #locked or 
someone with same or higher priority waiting
  self.queues[pri][1]+=1
  while self.locked:
self.queues[pri][0].wait()
  self.queues[pri][1]-=1
self.locked = True

  def Release(self):
with self.lock:
  self.locked = False
  for q in self.queues:
if q[1]:
  q[0].notify()
  break

  def Blip(self):
"""volountailily give up the lock and allow higher priority requests to 
run"""
self.Release()
self.Acquire(len(self.queues)-1)

Typically you would have only two priority classes. IO would then be done with:
gil.Release()
r = socket.recv(l)
gil.Acquire(0)

But here we are on dangerous grounds. Priority scheduling is full of pitfalls.
I should add here maybe, that I am a strong proponent of strict FIFO scheduling 
with perhaps few special cases.  We have been using FIFO tasklet scheduling for 
all tasklets, including those that are returing with IO, in EVE Online for 
three years now.  Before, we had somewhat chaotic scheduling (immediate 
switching to tasklets) and behavior was very erratic.  Switching to FIFO cured 
that and was very wholesome for the cluster in general..  I have been toying 
with the idea of adding a special IO tasklet priority class to stackless for a 
while, but haven't done it yet.  I am doubtful that it will on the whole change 
matters much.

Cheers,

Kristján


> -Original Message-
> From: David Beazley [mailto:d...@dabeaz.com]
> Sent: 16. mars 2010 11:03
> To: Kristján Valur Jónsson
> Cc: David Beazley; python-dev@python.org
> Subject: Re: [Python-Dev] "Fixing" the new GIL
> 
> Python doesn't use a pthreads mutex for the GIL.It has always used
> a binary semaphore implemented with condition variables (or just a
> pthreads semaphore if available).The reason the performance is so
> bad is precisely due to the fact that it is using this implementation
> and the fact that there *IS* a FIFO queue of threads (associated with
> the condition variable).   The I/O performance problem with the new GIL
> is gets much worse with many CPU-bound threads precisely because there
> is a FIFO queue involved.   This has been covered in my past GIL
> presentations.
> 
> -Dave
> 
> 
> 
> On Mar 16, 2010, at 5:52 AM, Kristján Valur Jónsson wrote:
> 
> > How about attacking the original problem, then?
> >
> > The reason they thrash on pthreads implementation is that a pthreads
> mutex is assumed to be a short-held resource.  Therefore it will be
> optimized in the following ways for multicore machines:
> > 1) There is a certain amount of spinning done, to try to acquire it
> before blocking
> > 2) It will employ un-fair tactics to avoid lock-convoying, meaning
> that a thread coming in to acquire the mutex may get in before others
> that are queued.  This is why "ticking" the GIL works so badly:  The
> thread that releases the lock is usually the one that reaquires it even
> though others may be waiting.  See e.g.
> http://www.bluebytesoftware.com/blog/PermaLink,guid,e40c2675-43a3-410f-
> 8f85-616ef7b031aa.aspx for a discussion of this (albeit on windows.)
> >
> > On Windows, this isn't a problem.  The reas

Re: [Python-Dev] "Fixing" the new GIL

[Kristján Valur Jónsson]

> -Original Message-
> From: python-dev-bounces+kristjan=ccpgames@python.org
> [mailto:python-dev-bounces+kristjan=ccpgames@python.org] On Behalf
> Of Antoine Pitrou
> Sent: 16. mars 2010 12:00
> To: python-dev@python.org
> Subject: Re: [Python-Dev] "Fixing" the new GIL
> 
> 
> Hi Kristján,
> 
> > In the new GIL, there appear to be several problems:
> > 1) There is no FIFO queue of threads wanting the queue, thus thread
> > scheduling becomes non-deterministic
> 
> Thread scheduling has always been non-deterministic.
Yes, but not the scheduling of "which thread has the GIL" which is what we are 
talking about.  With a gil, you have neutered the non-deterministic OS 
scheduler and forced the system to _effectively_ schedule threads as you with 
them to be scheduled.  The key to this is implementing your GIL in such a way 
that you (and not the system) chooses which threads runs next.  On Windows it 
works in a nice, determinitstic FIFO order becoause that's how the underlying 
Event object is supposed to work.  under pthreads, we rely on the behaviour of 
the condition variable which _probably_ is well behaved in this manner too.

> 
> > 2) The "ticking" of the GIL is
> > now controled by a condition variable timeout.  There appears to be
> > no way to prevent many such timeouts to be in progress at the same
> > time, thus you may have an unnecessarily high rate of ticking going
> > on.
> 
> Unless there's a bug, no, there isn't.
You're right, I had forgotten about "gil_switch_number".  Never mind.

> 
> > 3) There isn't an immediate gil request made when an IO thread
> > requests the gil back, only after an initial timeout.
> 
> This is what we are looking to fix (perhaps).
> 
> > What we are trying to write here is a thread scheduler, and that is
> > complex business. K
> 
> I would have rephrased:
> "What we are trying to avoid here is a thread scheduler".

I can't argue with that, but I'm afraid that in the end, you have to.  Python 
threads do behave very much in the same way as stackless python tasklets do.  
They own the cpu until they volounteeringly give it up.  You want to decide, 
then, which of the threads next gets its turn.  You are working on a completely 
different level than what the OS usually uses for its scheduling (timeslices, 
IO) so _you_ have to be in control.  By a happy coincidence, in the past on a 
single CPU, the old GIL did give us this, without us being explicit about it:  
FIFO execution of tasklets.  This fell out as a byproduct of the very simple 
way that the GIL was implemented, using a single Condition variable.  Now that 
we have a different mechanism, _where threads time out_ on the condition, this 
behaviour no longer falls out of the implementation on its own.  It may turn 
out, in this new, more complicated world, that you have to explicitly schedule 
the threads, by chaining them and giving each their own condition to wait on.

Btw, if you want to take up a private converstaion with me, I'd like to point 
out to you some problems with the Windows version of the Condition variable :)

Cheers,

Kristján

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] "Fixing" the new GIL

[Antoine Pitrou]

Le Tue, 16 Mar 2010 14:10:33 +,
Kristján Valur Jónsson  a écrit :
> 
> The key to this
> is implementing your GIL in such a way that you (and not the system)
> chooses which threads runs next.  On Windows it works in a nice,
> determinitstic FIFO order becoause that's how the underlying Event
> object is supposed to work.

Well, I don't think this has ever been by design, and it's not obvious
this is desirable either (see Dave Beazley's benchmark).

> Btw, if you want to take up a private converstaion with me, I'd like
> to point out to you some problems with the Windows version of the
> Condition variable :)

You can report a bug.

cheers

Antoine.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Decimal <-> float comparisons in py3k.

[Mark Dickinson]

Hello all,

Currently in Python 2.x, Decimal-to-float comparisons behave as follows:

>>> Decimal(1) < float(4)
False
>>> Decimal(4) < float(1)
False

That is, any Decimal sorts before any float (though this is arbitrary:
 it's possible that on some implementations any float sorts before any
Decimal).  This causes (a) confusion, and (b) bugs, especially when
floats and Decimals are accidentally combined in a program.  There
probably aren't too many legitimate reasons for deliberately mixing
floats and Decimals in the same calculation, so preventing accidents
is the main concern here.

In Python 3.x, however, such comparisons raise a TypeError
("unorderable types: Decimal() < float()").

http://bugs.python.org/issue2531 ('float compared to decimal is
silently incorrect') was opened for this a while ago.

I'm planning to commit a change to trunk that changes the behaviour so
that the comparison result is based on the respective values of the
arguments (so the results above would be True and False respectively).

Question for python-dev people (and the point of this email): should
this change be forward ported to py3k?

On the one hand there's something to be said for maintaining a clean
separation between the float and Decimal types, allowing only explicit
conversions from one to the other;  mixed-type arithmetic between
floats and Decimals was very deliberately not permitted in the
original PEP, and that's unlikely to change in a hurry.  On the other
hand, there's value in keeping 2.x and 3.x aligned where possible for
the sake of 2-to-3 porters, and the new behaviour may even be useful.
Even with the TypeError above, there are still some py3k surprises
arising from the ability to compare ints and Decimals, and ints and
floats, but not floats and Decimals.

A quick tour of some of these surprises, in trunk:

>>> from decimal import Decimal
>>> Decimal(1) < 2 < float(3) < Decimal(1)  # < is non-transitive
True
>>> Decimal(1) == 1 == float(1)   # so is equality
True
>>> Decimal(1) == float(1)
False
>>> d1, i1, f1 = Decimal(1), float(1), 1
>>> set([d1, i1, f1]) == set([f1, i1, d1])  # sets with the same elements are 
>>> different
False
>>> sorted([d1, i1, f1]) == sorted([f1, i1, d1])
False

and in py3k:

>>> from decimal import Decimal
>>> Decimal(1) < 2 < float(3) < Decimal(1)
Traceback (most recent call last):
  File "", line 1, in 
TypeError: unorderable types: float() < Decimal()
>>> Decimal(1) == 1 == float(1)
True
>>> Decimal(1) == float(1)
False
>>> d1, i1, f1 = Decimal(1), float(1), 1
>>> set([d1, i1, f1]) == set([f1, i1, d1])
False
>>> sorted([Decimal(1), 2, float(3)])
[Decimal('1'), 2, 3.0]
>>> sorted([2, Decimal(1), float(3)])
Traceback (most recent call last):
  File "", line 1, in 
TypeError: unorderable types: float() < Decimal()
>>> sorted([float(3), 2, Decimal(1)])
[Decimal('1'), 2, 3.0]

By the way, even with the patch there are still problems with other
numeric types: comparisons or set operations involving both Fraction
and Decimal instances are going to cause similar difficulties to those
above.  In practice I think this is much less of an issue than the
float/Decimal problem, since the chance of accidentally combining
Fraction and Decimal types in a calculation seems significantly
smaller than the chance of accidentally combining float and Decimal
types.

--
Mark
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Decimal <-> float comparisons in py3k.

[Facundo Batista]

On Tue, Mar 16, 2010 at 9:41 AM, Mark Dickinson  wrote:

> On the one hand there's something to be said for maintaining a clean
> separation between the float and Decimal types, allowing only explicit
> conversions from one to the other;  mixed-type arithmetic between
> floats and Decimals was very deliberately not permitted in the
> original PEP, and that's unlikely to change in a hurry.  On the other

But, to be fair, we didn't have "true value of the float at that time".

I'm +0 to allow these comparisons, being "Decimal(1) < .3" the same as
"Decimal(1) < Decimal.from_float(.3)"

-- 
.Facundo

Blog: http://www.taniquetil.com.ar/plog/
PyAr: http://www.python.org/ar/
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] "Fixing" the new GIL

[Bill Janssen]

Martin v. Löwis  wrote:

> > If it's lurking behind a filesystem interface or in its daemon mode
> > (remote archive store), multiple client processes can be using it at once,
> > and it will be processing multiple tasks somewhat in parallel. Here one
> > can get a compute bound thread answering one request, impacting quick
> > response to other parallel-and-cheap requests.
> 
> However, "impacting" will always be the case. *Of course* any thread
> that performs computation impacts everything else on the same processor.

That's not the problem, Martin.  The problem is that it also impacts
other threads on other cores, because of the interlock between the OS
thread scheduler and the internal Pythonic struggle to obtain the GIL.
And it shouldn't.

Bill
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Decimal <-> float comparisons in py3k.

[Mark Dickinson]

On Tue, Mar 16, 2010 at 2:58 PM, Facundo Batista
 wrote:
> On Tue, Mar 16, 2010 at 9:41 AM, Mark Dickinson  wrote:
>
>> On the one hand there's something to be said for maintaining a clean
>> separation between the float and Decimal types, allowing only explicit
>> conversions from one to the other;  mixed-type arithmetic between
>> floats and Decimals was very deliberately not permitted in the
>> original PEP, and that's unlikely to change in a hurry.  On the other
>
> But, to be fair, we didn't have "true value of the float at that time".

That's true.  I'd still be reluctant to start supporting operations
like Decimal('1.2') + 0.71, though.  At least comparisons have the
nice feature that the return type is unambiguous.

>
> I'm +0 to allow these comparisons, being "Decimal(1) < .3" the same as
> "Decimal(1) < Decimal.from_float(.3)"

Yes, I should have clarified that those are exactly the semantics I'm proposing.

Thanks!

Mark
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Decimal <-> float comparisons in py3k.

[P.J. Eby]

At 09:58 AM 3/16/2010 -0500, Facundo Batista wrote:

I'm +0 to allow these comparisons, being "Decimal(1) < .3" the same as
"Decimal(1) < Decimal.from_float(.3)"


Does Decimal.from_float() use the "shortest decimal representation" approach?

If not, it might be confusing if a number that prints as '.1' 
compares unequal to Decimal('.1').


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Decimal <-> float comparisons in py3k.

[Mark Dickinson]

On Tue, Mar 16, 2010 at 3:58 PM, P.J. Eby  wrote:
> At 09:58 AM 3/16/2010 -0500, Facundo Batista wrote:
>>
>> I'm +0 to allow these comparisons, being "Decimal(1) < .3" the same as
>> "Decimal(1) < Decimal.from_float(.3)"
>
> Does Decimal.from_float() use the "shortest decimal representation"
> approach?

No.  It does exact conversions:

>>> Decimal.from_float(1.1)
Decimal('1.100088817841970012523233890533447265625')
>>> Decimal.from_float(1.1) == 1.1
False
>>> Decimal('1.1') == float('1.1')  # returns False both pre- and post-patch
False

> If not, it might be confusing if a number that prints as '.1' compares
> unequal to Decimal('.1').

Agreed, but this is just your everyday floating-point confusion, to be
dealt with by social means (e.g., educating the programmer).  Any
technical solution that made "Decimal('1.1') == float('1.1')" evaluate
to True would, I suspect, be a cure worse than the original disease.

Mark
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Decimal <-> float comparisons in py3k.

[Victor Stinner]

Le mardi 16 mars 2010 16:58:22, P.J. Eby a écrit :
> At 09:58 AM 3/16/2010 -0500, Facundo Batista wrote:
> >I'm +0 to allow these comparisons, being "Decimal(1) < .3" the same as
> >"Decimal(1) < Decimal.from_float(.3)"
> 
> Does Decimal.from_float() use the "shortest decimal representation"
>  approach?
> 
> If not, it might be confusing if a number that prints as '.1'
> compares unequal to Decimal('.1').

In py3k, comparing bytes and str raise a TypeError("unorderable types: bytes() 
< str()"). I like this behaviour :-)

If comparaison of Decimal and float can have "unpredictable" result, I would 
suggest the same behaviour (raise an error).

-- 
Victor Stinner
http://www.haypocalc.com/
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Decimal <-> float comparisons in py3k.

[Mark Dickinson]

On Tue, Mar 16, 2010 at 4:11 PM, Mark Dickinson  wrote:
[...]

 Decimal.from_float(1.1) == 1.1
> False

Whoops.  To clarify, this is the pre-patch behaviour;  post-patch,
this gives True.

Mark
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Decimal <-> float comparisons in py3k.

[Mark Dickinson]

On Tue, Mar 16, 2010 at 4:18 PM, Victor Stinner
 wrote:
> If comparaison of Decimal and float can have "unpredictable" result, I would
> suggest the same behaviour (raise an error).

Well, it's not really `unpredictable':  the new behaviour is perfectly
predictable and sane, provided only that you remember the basic fact
that float("0.1") is not exactly 0.1.

Mark
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Decimal <-> float comparisons in py3k.

[Guido van Rossum]

I'd say if you're not going to forward-port this to Python 3, it
shouldn't go into Python 2 -- in that case it would make more sense to
me to back-port the exception-raising behavior.

Also supporting comparisons but not other mixed operations is going to
be confusing. If you are sticking to that behavior I think mixed
comparisons should also be ruled out.

--Guido

On Tue, Mar 16, 2010 at 7:41 AM, Mark Dickinson  wrote:
> Hello all,
>
> Currently in Python 2.x, Decimal-to-float comparisons behave as follows:
>
 Decimal(1) < float(4)
> False
 Decimal(4) < float(1)
> False
>
> That is, any Decimal sorts before any float (though this is arbitrary:
>  it's possible that on some implementations any float sorts before any
> Decimal).  This causes (a) confusion, and (b) bugs, especially when
> floats and Decimals are accidentally combined in a program.  There
> probably aren't too many legitimate reasons for deliberately mixing
> floats and Decimals in the same calculation, so preventing accidents
> is the main concern here.
>
> In Python 3.x, however, such comparisons raise a TypeError
> ("unorderable types: Decimal() < float()").
>
> http://bugs.python.org/issue2531 ('float compared to decimal is
> silently incorrect') was opened for this a while ago.
>
> I'm planning to commit a change to trunk that changes the behaviour so
> that the comparison result is based on the respective values of the
> arguments (so the results above would be True and False respectively).
>
> Question for python-dev people (and the point of this email): should
> this change be forward ported to py3k?
>
> On the one hand there's something to be said for maintaining a clean
> separation between the float and Decimal types, allowing only explicit
> conversions from one to the other;  mixed-type arithmetic between
> floats and Decimals was very deliberately not permitted in the
> original PEP, and that's unlikely to change in a hurry.  On the other
> hand, there's value in keeping 2.x and 3.x aligned where possible for
> the sake of 2-to-3 porters, and the new behaviour may even be useful.
> Even with the TypeError above, there are still some py3k surprises
> arising from the ability to compare ints and Decimals, and ints and
> floats, but not floats and Decimals.
>
> A quick tour of some of these surprises, in trunk:
>
 from decimal import Decimal
 Decimal(1) < 2 < float(3) < Decimal(1)  # < is non-transitive
> True
 Decimal(1) == 1 == float(1)   # so is equality
> True
 Decimal(1) == float(1)
> False
 d1, i1, f1 = Decimal(1), float(1), 1
 set([d1, i1, f1]) == set([f1, i1, d1])  # sets with the same elements are 
 different
> False
 sorted([d1, i1, f1]) == sorted([f1, i1, d1])
> False
>
> and in py3k:
>
 from decimal import Decimal
 Decimal(1) < 2 < float(3) < Decimal(1)
> Traceback (most recent call last):
>  File "", line 1, in 
> TypeError: unorderable types: float() < Decimal()
 Decimal(1) == 1 == float(1)
> True
 Decimal(1) == float(1)
> False
 d1, i1, f1 = Decimal(1), float(1), 1
 set([d1, i1, f1]) == set([f1, i1, d1])
> False
 sorted([Decimal(1), 2, float(3)])
> [Decimal('1'), 2, 3.0]
 sorted([2, Decimal(1), float(3)])
> Traceback (most recent call last):
>  File "", line 1, in 
> TypeError: unorderable types: float() < Decimal()
 sorted([float(3), 2, Decimal(1)])
> [Decimal('1'), 2, 3.0]
>
> By the way, even with the patch there are still problems with other
> numeric types: comparisons or set operations involving both Fraction
> and Decimal instances are going to cause similar difficulties to those
> above.  In practice I think this is much less of an issue than the
> float/Decimal problem, since the chance of accidentally combining
> Fraction and Decimal types in a calculation seems significantly
> smaller than the chance of accidentally combining float and Decimal
> types.
>
> --
> Mark
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> http://mail.python.org/mailman/options/python-dev/guido%40python.org
>



-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] "Fixing" the new GIL

[Kristján Valur Jónsson]

> -Original Message-
> From: python-dev-bounces+kristjan=ccpgames@python.org
> [mailto:python-dev-bounces+kristjan=ccpgames@python.org] On Behalf
> Of Antoine Pitrou
> > The key to this
> > is implementing your GIL in such a way that you (and not the system)
> > chooses which threads runs next.  On Windows it works in a nice,
> > determinitstic FIFO order becoause that's how the underlying Event
> > object is supposed to work.
> 
> Well, I don't think this has ever been by design, and it's not obvious
> this is desirable either (see Dave Beazley's benchmark).
>
Did Dave benchmark a straight FIFO system?
I can tell you out of experience, engineering a similar system (stacklessIO) 
that you _do_ want a deterministic scheduling algorithm.  The most 
straightforward is round-robin (or FIFO), and the only sane option really if 
you know nothing about your threads.  Upon this, you can start to build, by for 
example adding another priority layer, but you have to have a solid foundation. 
 We have never had high priority for IO threads in python (and its 
not-by-design round robin scheduler on single core cpus) and it is unclear why 
that should be required now as a fix.

 
> > Btw, if you want to take up a private converstaion with me, I'd like
> > to point out to you some problems with the Windows version of the
> > Condition variable :)
> 
> You can report a bug.
Or I could commit a fix.  I just thought You might be interested, is all.  I'll 
submit a patch.

K
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Decimal <-> float comparisons in py3k.

[Mark Dickinson]

On Tue, Mar 16, 2010 at 4:41 PM, Guido van Rossum  wrote:
> I'd say if you're not going to forward-port this to Python 3, it
> shouldn't go into Python 2 -- in that case it would make more sense to
> me to back-port the exception-raising behavior.

That's also a possible solution, and the one that I'd personally be
happiest with.  The main problem is that this has the potential to
break code:  lists containing both floats and Decimals are sortable in
2.6, but would no longer be sortable in 2.7.  If such breakage is
deemed acceptable then I'd happily backport the exception;  I really
don't have a good feeling for how much real-world code could break, if
any.

> Also supporting comparisons but not other mixed operations is going to
> be confusing. If you are sticking to that behavior I think mixed
> comparisons should also be ruled out.

Confusing, yes, but at least not bug-prone.  The current 2.x behaviour
has provoked complaints from a number of different people in various
different fora (I recently saw this come up on StackOverflow), and
after initially being skeptical I'm now convinced that it would be a
good idea to change it if at all possible.

Mark
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Decimal <-> float comparisons in py3k.

[Guido van Rossum]

On Tue, Mar 16, 2010 at 9:07 AM, Mark Dickinson  wrote:
> On Tue, Mar 16, 2010 at 4:41 PM, Guido van Rossum  wrote:
>> I'd say if you're not going to forward-port this to Python 3, it
>> shouldn't go into Python 2 -- in that case it would make more sense to
>> me to back-port the exception-raising behavior.
>
> That's also a possible solution, and the one that I'd personally be
> happiest with.  The main problem is that this has the potential to
> break code:  lists containing both floats and Decimals are sortable in
> 2.6, but would no longer be sortable in 2.7.  If such breakage is
> deemed acceptable then I'd happily backport the exception;  I really
> don't have a good feeling for how much real-world code could break, if
> any.

Definitely some. Stricter comparison rules are a frequent cause of
problems when code is first ported to 3.x. While you'd think that code
comparing a float and a Decimal is *already* broken, there's a
surprising number of situations where that's not necessary the case,
e.g. when an arbitrary but stable ordering is needed.

>> Also supporting comparisons but not other mixed operations is going to
>> be confusing. If you are sticking to that behavior I think mixed
>> comparisons should also be ruled out.
>
> Confusing, yes, but at least not bug-prone.  The current 2.x behaviour
> has provoked complaints from a number of different people in various
> different fora (I recently saw this come up on StackOverflow), and
> after initially being skeptical I'm now convinced that it would be a
> good idea to change it if at all possible.

Yeah, it should have raised an exception all along. But it's too late
for that now. I wonder if it should just become a py3k warning?

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] "Fixing" the new GIL

[Nir Aides]

Hello Dave,

The following documentation suggests ordering in Linux is not FIFO:
http://www.opengroup.org/onlinepubs/95399/functions/pthread_cond_timedwait.html#tag_03_518_08_06
"Threads waiting on mutexes and condition variables are selected to proceed
in an order dependent upon the scheduling policy rather than in some fixed
order (for example, FIFO or priority). Thus, the scheduling policy
determines which thread(s) are awakened and allowed to proceed."

Here is the code:
http://www.google.com/codesearch/p?hl=en#5ge3gHPB4K4/gnu/glibc/glibc-linuxthreads-2.1.1.tar.gz%7CaeB7Uqo7T9g/linuxthreads/queue.h&q=pthread_cond_timedwait&exact_package=http://ftp.gnu.org/gnu/glibc/glibc-linuxthreads-2.1.1.tar.gz

If this is so then it should affect the proposed fixes.
For example waiting with timeout should be avoided, no?

Nir

2010/3/16 David Beazley 

> Python doesn't use a pthreads mutex for the GIL.It has always used a
> binary semaphore implemented with condition variables (or just a pthreads
> semaphore if available).The reason the performance is so bad is
> precisely due to the fact that it is using this implementation and the fact
> that there *IS* a FIFO queue of threads (associated with the condition
> variable).   The I/O performance problem with the new GIL is gets much worse
> with many CPU-bound threads precisely because there is a FIFO queue
> involved.   This has been covered in my past GIL presentations.
>
> -Dave
>
>
>
> On Mar 16, 2010, at 5:52 AM, Kristján Valur Jónsson wrote:
>
> > How about attacking the original problem, then?
> >
> > The reason they thrash on pthreads implementation is that a pthreads
> mutex is assumed to be a short-held resource.  Therefore it will be
> optimized in the following ways for multicore machines:
> > 1) There is a certain amount of spinning done, to try to acquire it
> before blocking
> > 2) It will employ un-fair tactics to avoid lock-convoying, meaning that a
> thread coming in to acquire the mutex may get in before others that are
> queued.  This is why "ticking" the GIL works so badly:  The thread that
> releases the lock is usually the one that reaquires it even though others
> may be waiting.  See e.g.
> http://www.bluebytesoftware.com/blog/PermaLink,guid,e40c2675-43a3-410f-8f85-616ef7b031aa.aspxfor
>  a discussion of this (albeit on windows.)
> >
> > On Windows, this isn't a problem.  The reason is, that the GIL on windows
> is implemented using Event objects that don't cut these corners.  The Event
> provides you with a strict FIFO queue of objects waiting for the event.
> >
> > If pthreads doesn't provide a synchronization primitive similar to that,
> someone that doesn't thrash and has a true FIFO queue, it is possible to
> construct such a thing using condition variables and critical sections.
>  Perhaps the posix semaphore api is more appropriate in this case.
> >
> > By the way, this also shows another problem with (old) python.  There is
> only one core locking primitive, the PyThread_type_lock.  It is being used
> both as a critical section in the traditional sense, and also as this
> sort-of-inverse lock that the GIL is.  In the modern world, where the
> intended behaviour of these is quite different, there is no one-size-fits
> all.  On windows in particular, the use of the Event object based lock is
> not ideal for other uses than the GIL.
> >
> >
> > In the new GIL, there appear to be several problems:
> > 1) There is no FIFO queue of threads wanting the queue, thus thread
> scheduling becomes non-deterministic
> > 2) The "ticking" of the GIL is now controled by a condition variable
> timeout.  There appears to be no way to prevent many such timeouts to be in
> progress at the same time, thus you may have an unnecessarily high rate of
> ticking going on.
> > 3) There isn't an immediate gil request made when an IO thread requests
> the gil back, only after an initial timeout.
> >
> > What we are trying to write here is a thread scheduler, and that is
> complex business.
> > K
> >
> >
> >
> >> -Original Message-
> >> From: python-dev-bounces+kristjan=ccpgames@python.org
> >> [mailto:python-dev-bounces+kristjan =
> ccpgames@python.org] On Behalf
> >> Of David Beazley
> >> Sent: 15. mars 2010 03:07
> >> To: python-dev@python.org
> >> Subject: Re: [Python-Dev] "Fixing" the new GIL
> >>
> >> happen to be performing CPU intensive work at the same time, it would
> >> be nice if they didn't thrash on multiple cores (the problem with the
> >> old GIL) and if I/O is
> >
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/nir%40winpdb.org
>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-a

Re: [Python-Dev] "Fixing" the new GIL

[Peter Portante]

Yes, having another thread wait with a timeout is not good. CPU cycles
waisted doing things that don¹t help the thread holding the GIL release it.
They just note that they are present and then the GIL-holding-thread should
be responsible for handling the rest. -peter


On 3/16/10 1:11 PM, "Nir Aides"  wrote:

> Hello Dave, 
> 
> The following documentation suggests ordering in Linux is not FIFO:
> http://www.opengroup.org/onlinepubs/95399/functions/pthread_cond_timedwait
> .html#tag_03_518_08_06
> "Threads waiting on mutexes and condition variables are selected to proceed in
> an order dependent upon the scheduling policy rather than in some fixed order
> (for example, FIFO or priority). Thus, the scheduling policy determines which
> thread(s) are awakened and allowed to proceed."
> 
> Here is the code:
> http://www.google.com/codesearch/p?hl=en#5ge3gHPB4K4/gnu/glibc/glibc-linuxthre
> ads-2.1.1.tar.gz%7CaeB7Uqo7T9g/linuxthreads/queue.h&q=pthread_cond_timedwait&e
> xact_package=http://ftp.gnu.org/gnu/glibc/glibc-linuxthreads-2.1.1.tar.gz
> 
> If this is so then it should affect the proposed fixes. 
> For example waiting with timeout should be avoided, no?
> 
> Nir
> 
> 2010/3/16 David Beazley 
>> Python doesn't use a pthreads mutex for the GIL.    It has always used a
>> binary semaphore implemented with condition variables (or just a pthreads
>> semaphore if available).    The reason the performance is so bad is precisely
>> due to the fact that it is using this implementation and the fact that there
>> *IS* a FIFO queue of threads (associated with the condition variable).   The
>> I/O performance problem with the new GIL is gets much worse with many
>> CPU-bound threads precisely because there is a FIFO queue involved.   This
>> has been covered in my past GIL presentations.
>> 
>> -Dave
>> 
>> 
>> 
>> On Mar 16, 2010, at 5:52 AM, Kristján Valur Jónsson wrote:
>> 
>>> > How about attacking the original problem, then?
>>> >
>>> > The reason they thrash on pthreads implementation is that a pthreads mutex
>>> is assumed to be a short-held resource.  Therefore it will be optimized in
>>> the following ways for multicore machines:
>>> > 1) There is a certain amount of spinning done, to try to acquire it before
>>> blocking
>>> > 2) It will employ un-fair tactics to avoid lock-convoying, meaning that a
>>> thread coming in to acquire the mutex may get in before others that are
>>> queued.  This is why "ticking" the GIL works so badly:  The thread that
>>> releases the lock is usually the one that reaquires it even though others
>>> may be waiting.  See e.g.
>>> http://www.bluebytesoftware.com/blog/PermaLink,guid,e40c2675-43a3-410f-8f85-
>>> 616ef7b031aa.aspx for a discussion of this (albeit on windows.)
>>> >
>>> > On Windows, this isn't a problem.  The reason is, that the GIL on windows
>>> is implemented using Event objects that don't cut these corners.  The Event
>>> provides you with a strict FIFO queue of objects waiting for the event.
>>> >
>>> > If pthreads doesn't provide a synchronization primitive similar to that,
>>> someone that doesn't thrash and has a true FIFO queue, it is possible to
>>> construct such a thing using condition variables and critical sections.
>>>  Perhaps the posix semaphore api is more appropriate in this case.
>>> >
>>> > By the way, this also shows another problem with (old) python.  There is
>>> only one core locking primitive, the PyThread_type_lock.  It is being used
>>> both as a critical section in the traditional sense, and also as this
>>> sort-of-inverse lock that the GIL is.  In the modern world, where the
>>> intended behaviour of these is quite different, there is no one-size-fits
>>> all.  On windows in particular, the use of the Event object based lock is
>>> not ideal for other uses than the GIL.
>>> >
>>> >
>>> > In the new GIL, there appear to be several problems:
>>> > 1) There is no FIFO queue of threads wanting the queue, thus thread
>>> scheduling becomes non-deterministic
>>> > 2) The "ticking" of the GIL is now controled by a condition variable
>>> timeout.  There appears to be no way to prevent many such timeouts to be in
>>> progress at the same time, thus you may have an unnecessarily high rate of
>>> ticking going on.
>>> > 3) There isn't an immediate gil request made when an IO thread requests
>>> the gil back, only after an initial timeout.
>>> >
>>> > What we are trying to write here is a thread scheduler, and that is
>>> complex business.
>>> > K
>>> >
>>> >
>>> >
 >> -Original Message-
 >> From: python-dev-bounces+kristjan=ccpgames.com 
 @python.org 
 >> [mailto:python-dev-bounces+kristjan
  =ccpgames.com 
 @python.org  ] On Behalf
 >> Of David Beazley
 >> Sent: 15. mars 2010 03:07
 >> To: python-dev@python.org
 >> Subject: Re: [Python-Dev] "Fixing" the new GIL
 >>
 >> ha

Re: [Python-Dev] "Fixing" the new GIL

[Robert Hancock]

The Linux kernel scheduler deals with two types of ratings and has a
heuristic algorithm that rewards and penalizes on a rapid basis.  To
determine the next thread it inspects the priority array to find the highest
priority task that is runnable and then selects the first task in that
priority.  This is a gross simplification of an extremely complex system,
but it is not simply a simple queue.

To duplicate this exact type of scheduling in the interpreter would be a
daunting task, but Dave's example illustrates that some system of reward and
penalty can be beneficial.

Peter has brought up the point, both here and at the Open Space at Pycon,
that before attempting a downsized version of the a priority thread
scheduler that we should look at other programs that have had to deal with
similar problems.

Peter, since you have dealt with the nitty-gritty of concurrency before, can
you suggest some applications that could serve as models for research?

On Tue, Mar 16, 2010 at 3:22 PM, Peter Portante
wrote:

>  Yes, having another thread wait with a timeout is not good. CPU cycles
> waisted doing things that don’t help the thread holding the GIL release it.
> They just note that they are present and then the GIL-holding-thread should
> be responsible for handling the rest. -peter
>
>
>
> On 3/16/10 1:11 PM, "Nir Aides"  wrote:
>
> Hello Dave,
>
> The following documentation suggests ordering in Linux is not FIFO:
>
> http://www.opengroup.org/onlinepubs/95399/functions/pthread_cond_timedwait.html#tag_03_518_08_06
> "Threads waiting on mutexes and condition variables are selected to
> proceed in an order dependent upon the scheduling policy rather than in some
> fixed order (for example, FIFO or priority). Thus, the scheduling policy
> determines which thread(s) are awakened and allowed to proceed."
>
> Here is the code:
>
> http://www.google.com/codesearch/p?hl=en#5ge3gHPB4K4/gnu/glibc/glibc-linuxthreads-2.1.1.tar.gz%7CaeB7Uqo7T9g/linuxthreads/queue.h&q=pthread_cond_timedwait&exact_package=http://ftp.gnu.org/gnu/glibc/glibc-linuxthreads-2.1.1.tar.gz
>
> If this is so then it should affect the proposed fixes.
> For example waiting with timeout should be avoided, no?
>
> Nir
>
> 2010/3/16 David Beazley 
>
> Python doesn't use a pthreads mutex for the GIL.It has always used a
> binary semaphore implemented with condition variables (or just a pthreads
> semaphore if available).The reason the performance is so bad is
> precisely due to the fact that it is using this implementation and the fact
> that there *IS* a FIFO queue of threads (associated with the condition
> variable).   The I/O performance problem with the new GIL is gets much worse
> with many CPU-bound threads precisely because there is a FIFO queue
> involved.   This has been covered in my past GIL presentations.
>
> -Dave
>
>
>
> On Mar 16, 2010, at 5:52 AM, Kristján Valur Jónsson wrote:
>
> > How about attacking the original problem, then?
> >
> > The reason they thrash on pthreads implementation is that a pthreads
> mutex is assumed to be a short-held resource.  Therefore it will be
> optimized in the following ways for multicore machines:
> > 1) There is a certain amount of spinning done, to try to acquire it
> before blocking
> > 2) It will employ un-fair tactics to avoid lock-convoying, meaning that a
> thread coming in to acquire the mutex may get in before others that are
> queued.  This is why "ticking" the GIL works so badly:  The thread that
> releases the lock is usually the one that reaquires it even though others
> may be waiting.  See e.g.
> http://www.bluebytesoftware.com/blog/PermaLink,guid,e40c2675-43a3-410f-8f85-616ef7b031aa.aspxfor
>  a discussion of this (albeit on windows.)
> >
> > On Windows, this isn't a problem.  The reason is, that the GIL on windows
> is implemented using Event objects that don't cut these corners.  The Event
> provides you with a strict FIFO queue of objects waiting for the event.
> >
> > If pthreads doesn't provide a synchronization primitive similar to that,
> someone that doesn't thrash and has a true FIFO queue, it is possible to
> construct such a thing using condition variables and critical sections.
>  Perhaps the posix semaphore api is more appropriate in this case.
> >
> > By the way, this also shows another problem with (old) python.  There is
> only one core locking primitive, the PyThread_type_lock.  It is being used
> both as a critical section in the traditional sense, and also as this
> sort-of-inverse lock that the GIL is.  In the modern world, where the
> intended behaviour of these is quite different, there is no one-size-fits
> all.  On windows in particular, the use of the Event object based lock is
> not ideal for other uses than the GIL.
> >
> >
> > In the new GIL, there appear to be several problems:
> > 1) There is no FIFO queue of threads wanting the queue, thus thread
> scheduling becomes non-deterministic
> > 2) The "ticking" of the GIL is now controled by a condition varia

Re: [Python-Dev] Python 2.6.5 rc 2

[Sridhar Ratnakumar]

I just verified with our ActivePython build that 2.6.4rc2 builds fine on Linux, 
Windows, Mac, HP-UX, AIX and Solaris. 3.1.2rc1 builds fine except on AIX[1] and 
HP-UX[2] but those issues existed in 3.1.1 too, I believe.

-srid

[1] http://bugs.python.org/issue6645
[2] http://bugs.python.org/issue5999

On 2010-03-09, at 4:56 AM, Barry Warsaw wrote:

> Hi Python hackateers!
> 
> It looks like we finally have no more release blockers for 2.6.5rc2.  I would
> like to tag the tree tonight for rc2 so that Martin can build the Windows
> installer for a release tomorrow.  I am also moving the final release back to
> Friday March 19.
> 
> -Barry
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> http://mail.python.org/mailman/options/python-dev/sridharr%40activestate.com

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] tagging 3.1.2

[Benjamin Peterson]

My plan is to tag 3.1.2 sometime on Thursday, so binaries can be built
for the final release on Saturday. Agreeable?

-- 
Regards,
Benjamin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously

[Brian Quinlan]

Nick Coghlan wrote:
> You may want to consider providing global thread and process  
executors
> in the futures module itself. Code which just wants to say "do this  
in

> the background" without having to manage the lifecycle of its own
> executor instance is then free to do so. I've had a lot of experience
> with a framework that provides this and it is *very* convenient (it's
> also a good way to avoid deadlocks due to synchronous notification  
APIs).


This seems like a reasonable idea to me.

I take it that the thread/process pool should be unlimited in size.  
Should every thread/process exit when it finishes its job or should  
there be a smarter collection strategy?


Cheers,
Brian
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] tagging 2.6.5 (Re: tagging 3.1.2)

[Barry Warsaw]

On Mar 16, 2010, at 03:51 PM, Benjamin Peterson wrote:

>My plan is to tag 3.1.2 sometime on Thursday, so binaries can be built
>for the final release on Saturday. Agreeable?

I might as well do the same for 2.6.5, though I plan on releasing it Friday.

-Barry


signature.asc
Description: PGP signature
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Decimal <-> float comparisons in py3k.

[Greg Ewing]

Mark Dickinson wrote:


On the one hand there's something to be said for maintaining a clean
separation between the float and Decimal types, allowing only explicit
conversions from one to the other;  mixed-type arithmetic between
floats and Decimals was very deliberately not permitted in the
original PEP, and that's unlikely to change in a hurry.


I think that as long as you're disallowing arithmetic
between float and decimal, comparison should be disallowed
as well.

So if you're going to change anything, it would be better
to make such comparisons raise a TypeError in 2.7 to match
the current 3.x behaviour.

--
Greg
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Decimal <-> float comparisons in py3k.

[Greg Ewing]

Mark Dickinson wrote:

On Tue, Mar 16, 2010 at 3:58 PM, P.J. Eby  wrote:


If not, it might be confusing if a number that prints as '.1' compares
unequal to Decimal('.1').


Agreed, but this is just your everyday floating-point confusion, to be
dealt with by social means (e.g., educating the programmer).


Seems to me that this education would mostly consist of saying
"don't compare floats and decimals", which is why I think that
disallowing them in the first place would be better.

Then if a programmer truly needs to compare them for some
reason, he has to be explicit about how to do it.

--
Greg
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] "Fixing" the new GIL

[Nick Coghlan]

Kristján Valur Jónsson wrote:
>  We have never had high priority for IO threads in
> python (and its not-by-design round robin scheduler on single core
> cpus) and it is unclear why that should be required now as a fix.

David explained that in the issue tracker - 2.x typically doesn't do
that much work per bytecode instruction, so the interpreter releases and
reacquires the GIL far more frequently in a CPU-bound thread than it
does under the new time-based check interval.

The current settings mean we have less GIL overhead in the normal case,
but worse worst-case I/O latency.

There are two fairly straightforward ways to handle that:
- cut the check interval duration drastically (trading GIL overhead for
I/O responsiveness, as in 2.x)
- add some form of I/O thread prioritisation, such as Antoine's gilinter
patch (which automatically identifies and prioritises "interactive"
threads at the cost of a dozen or so additional lines of C code and a
small amount of overhead on GIL context switches to adjust thread
interactivity counters).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
---
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously

[Nick Coghlan]

Brian Quinlan wrote:
> I take it that the thread/process pool should be unlimited in size.
> Should every thread/process exit when it finishes its job or should
> there be a smarter collection strategy?

I'd be inclined to do something slightly smarter, similar to what we do
with memory overallocation for mutable containers.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
---
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] "Fixing" the new GIL

[Antoine Pitrou]

> David explained that in the issue tracker - 2.x typically doesn't do
> that much work per bytecode instruction,

Oh, but that's wrong in general.
Dave's *spinning loop* doesn't do much work per bytecode instruction,
however ;)

> The current settings mean we have less GIL overhead in the normal case,
> but worse worst-case I/O latency.

Actually, ccbench shows that worst case IO latency is much worse in 2.x
(when executing bytecodes which do a lot of work, e.g. matching a
regex).
What happens though is that best case IO latency is better in 2.x (e.g.
spinning loop, or short opcodes approaching the spinning loop case :-)).

cheers

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Decimal <-> float comparisons in py3k.

[Steven D'Aprano]

On Wed, 17 Mar 2010 03:23:30 am Mark Dickinson wrote:
> On Tue, Mar 16, 2010 at 4:11 PM, Mark Dickinson 
> wrote: [...]
>
>  Decimal.from_float(1.1) == 1.1
> >
> > False
>
> Whoops.  To clarify, this is the pre-patch behaviour;  post-patch,
> this gives True.

Whew! You had me worried there for a second. Just to clarify, you are 
proposing:

Decimal.from_float(1.1) == 1.1
Decimal.('1.1') != float('1.1')

+1 on this behaviour, even in the absence of supporting mixed Decimal 
and float arithmetic operations.

Both Decimals and floats are representations of real numbers, and not 
being able to compare two numbers is just weird: refusing to compare 
(say) Decimal(1) with float(1) makes as little sense to me as refusing 
to compare int(1) with float(1).

But mixed arithmetic runs into the problem, what do you want the result 
type to be? Given (say) decimal+float, returning either a Decimal or a 
float will be the wrong thing to do some of the time, so better to 
prohibit mixed arithmetic and let folks handle their own conversions. 
So +1 on continuing to prohibit mixed arithmetic.

But no such problems arise with comparisons, which will always return a 
bool, and will avoid the current ... interesting ... behaviour. In 3.1:

>>> Decimal(1) == 1 == 1.0
True
>>> Decimal(1) == 1.0
False
>>> Decimal.from_float(1.0) == 1 == 1.0
True
>>> Decimal.from_float(1.0) == 1.0
False

Replacing False with an exception doesn't make it any less bizarre.



-- 
Steven D'Aprano
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] "Fixing" the new GIL

[Antoine Pitrou]

Le Tue, 16 Mar 2010 17:02:13 +, Kristján Valur Jónsson a écrit :
>> 
>> Well, I don't think this has ever been by design, and it's not obvious
>> this is desirable either (see Dave Beazley's benchmark).
>>
> Did Dave benchmark a straight FIFO system?

Well, he presented benchmark figures with two threads, which is a FIFO 
system with the new GIL (because when a thread releases the GIL, it 
ensures that another thread takes it).

> We have never had high priority for IO threads in python
> (and its not-by-design round robin scheduler on single core cpus) and it
> is unclear why that should be required now as a fix.

I agree that Dave's situation doesn't seem new to me, although he often 
presents it as a deficiency of the new GIL only.

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] "Fixing" the new GIL

[Martin v. Löwis]

> You can't. This isn't an example where I am personally bitten by the GIL.
> I may be, but there's plenty of other stuff in terms of tuning or
> optimisation in my particular app before I get anywhere near the GIL.
> 
> My point is not that it's biting me right now but that earlier you
> said the scenario was probably unrealistic, but I have an app which can
> probably exhibit exactly the issue under discussion.

I don't understand. First you say that there is plenty of other stuff
 before you get anywhere near the GIL, and then you say that your
app can probably exhibit the issue under discussion. Which one is it?

I can easily believe that there is tons of other stuff that make the GIL
contention irrelevant, I can not believe that it will exhibit the issue
under discussion.

> | > The idea here is that one has a few threads receiving requests (eg a
> | > daemon watching a socket or monitoring a db queue table) which then use
> | > the FuncMultiQueue to manage how many actual requests are processed
> | > in parallel (yes, a semaphore can cover a lot of this, but not the
> | > asynchronous call modes).
> | 
> | Why do you say processing is in parallel?
> | In Python, processing is
> | normally never in parallel, but always sequential (potentially
> | interleaving). Are you releasing the GIL for processing?
> 
> 
> I mean that I can have multiple actual requests being served, and they
> are _not_ handled sequentially - the multi queue is to permit more than
> one to be in progress at once so that an expensive request need not
> inherently delay a following cheap request

But, in Python, due to the GIL, *everything* is sequential, *nothing* is
in parallel.

> Can't tell you - I'm not claiming the current GIL behaviour is biting me
> right now; I'm saying your claim that the scenario is unrealistic is a
> bit unfair; my app wil probably be just the kind of app to be affected,
> even only subtly. And it can hardly be alone in the "daemon" world.

Unfortunately, you are not able to demonstrate this (because the effect
would only be subtle).

My claim is that the scenario where the effect is *visible* is
unrealistic, i.e. that, in realistic scenarios, tons of other effects
actually lessen the problem.

Regards,
Martin

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Decimal <-> float comparisons in py3k.

[Nick Coghlan]

Steven D'Aprano wrote:
> But no such problems arise with comparisons, which will always return a 
> bool, and will avoid the current ... interesting ... behaviour. In 3.1:
> 
 Decimal(1) == 1 == 1.0
> True
 Decimal(1) == 1.0
> False
 Decimal.from_float(1.0) == 1 == 1.0
> True
 Decimal.from_float(1.0) == 1.0
> False
> 
> Replacing False with an exception doesn't make it any less bizarre.

Allowing the comparisons also doesn't introduce the potential for large
cumulative errors which are possible when actual implicit arithmetic
conversions are allowed.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
---
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Decimal <-> float comparisons in py3k.

[Steven D'Aprano]

On Wed, 17 Mar 2010 09:16:11 am Greg Ewing wrote:
> Mark Dickinson wrote:
> > On Tue, Mar 16, 2010 at 3:58 PM, P.J. Eby  
wrote:
> >>If not, it might be confusing if a number that prints as '.1'
> >> compares unequal to Decimal('.1').
> >
> > Agreed, but this is just your everyday floating-point confusion, to
> > be dealt with by social means (e.g., educating the programmer).
>
> Seems to me that this education would mostly consist of saying
> "don't compare floats and decimals", which is why I think that
> disallowing them in the first place would be better.

I'm sure you don't mean to suggest that the only (or even the main) 
source of floating point confusion comes from mixed Decimal/float 
operations.


> Then if a programmer truly needs to compare them for some
> reason, he has to be explicit about how to do it.

More explicit than someDecimal == someFloat? Seems pretty explicit to 
me.



-- 
Steven D'Aprano
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Decimal <-> float comparisons in py3k.

[Guido van Rossum]

On Tue, Mar 16, 2010 at 2:32 PM, Steven D'Aprano  wrote:
> But mixed arithmetic runs into the problem, what do you want the result
> type to be? Given (say) decimal+float, returning either a Decimal or a
> float will be the wrong thing to do some of the time, so better to
> prohibit mixed arithmetic and let folks handle their own conversions.
> So +1 on continuing to prohibit mixed arithmetic.

I'm not disagreeing, but I really wonder, what is the value of
supporting mixed comparisons then? Just because you *can* assign a
meaning to it doesn't mean you should.

OTOH I'm sure a lot of people would like to see mixed arithmetic
supported, the PEP be damned, and they would probably be happy with
any simple rule about the return type even if it's not always ideal. I
note that there are cases where converting a long to a float also is
the wrong thing to do, and yet mixed long/float operations always
return floats. If you are amenable to this argument, I would propose
to make the result of mixed operations return a Decimal, since in some
"intuitive complexity" sense an int is a simpler type than a float and
a float is a simpler type than a Decimal -- so results return the more
complex type. But my intuition on this isn't super strong and I could
live with always returning a float as well -- there are always casts
to force the issue.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] "Fixing" the new GIL

[Nir Aides]

Hi,

I posted a small patch to the GIL which demonstrates the scheduling policy
of pthreads conditions.

It seems that with pthreads a good policy is to allow (and help) the OS to
manage scheduling of threads via the condition queue without introducing
scheduling logic.
Some changes include removing the timeout from the new GIL wait, and
allowing CPU bound threads to run long enough to let the OS recognize them
as such.
The patch uses ticks countdown-to-switch but it should be changed to a time
based countdown (If taking a look at the clock is expensive maybe it can be
done once every N ticks during countdown).

Remains to explore what can be done on other platforms.

Nir



2010/3/16 Nir Aides 

> Hello Dave,
>
> The following documentation suggests ordering in Linux is not FIFO:
>
> http://www.opengroup.org/onlinepubs/95399/functions/pthread_cond_timedwait.html#tag_03_518_08_06
> "Threads waiting on mutexes and condition variables are selected to
> proceed in an order dependent upon the scheduling policy rather than in some
> fixed order (for example, FIFO or priority). Thus, the scheduling policy
> determines which thread(s) are awakened and allowed to proceed."
>
> Here is the code:
>
> http://www.google.com/codesearch/p?hl=en#5ge3gHPB4K4/gnu/glibc/glibc-linuxthreads-2.1.1.tar.gz%7CaeB7Uqo7T9g/linuxthreads/queue.h&q=pthread_cond_timedwait&exact_package=http://ftp.gnu.org/gnu/glibc/glibc-linuxthreads-2.1.1.tar.gz
>
> If this is so then it should affect the proposed fixes.
> For example waiting with timeout should be avoided, no?
>
> Nir
>
> 2010/3/16 David Beazley 
>
>> Python doesn't use a pthreads mutex for the GIL.It has always used a
>> binary semaphore implemented with condition variables (or just a pthreads
>> semaphore if available).The reason the performance is so bad is
>> precisely due to the fact that it is using this implementation and the fact
>> that there *IS* a FIFO queue of threads (associated with the condition
>> variable).   The I/O performance problem with the new GIL is gets much worse
>> with many CPU-bound threads precisely because there is a FIFO queue
>> involved.   This has been covered in my past GIL presentations.
>>
>>
>> -Dave
>>
>>
>>
>> On Mar 16, 2010, at 5:52 AM, Kristján Valur Jónsson wrote:
>>
>> > How about attacking the original problem, then?
>> >
>> > The reason they thrash on pthreads implementation is that a pthreads
>> mutex is assumed to be a short-held resource.  Therefore it will be
>> optimized in the following ways for multicore machines:
>> > 1) There is a certain amount of spinning done, to try to acquire it
>> before blocking
>> > 2) It will employ un-fair tactics to avoid lock-convoying, meaning that
>> a thread coming in to acquire the mutex may get in before others that are
>> queued.  This is why "ticking" the GIL works so badly:  The thread that
>> releases the lock is usually the one that reaquires it even though others
>> may be waiting.  See e.g.
>> http://www.bluebytesoftware.com/blog/PermaLink,guid,e40c2675-43a3-410f-8f85-616ef7b031aa.aspxfor
>>  a discussion of this (albeit on windows.)
>> >
>> > On Windows, this isn't a problem.  The reason is, that the GIL on
>> windows is implemented using Event objects that don't cut these corners.
>>  The Event provides you with a strict FIFO queue of objects waiting for the
>> event.
>> >
>> > If pthreads doesn't provide a synchronization primitive similar to that,
>> someone that doesn't thrash and has a true FIFO queue, it is possible to
>> construct such a thing using condition variables and critical sections.
>>  Perhaps the posix semaphore api is more appropriate in this case.
>> >
>> > By the way, this also shows another problem with (old) python.  There is
>> only one core locking primitive, the PyThread_type_lock.  It is being used
>> both as a critical section in the traditional sense, and also as this
>> sort-of-inverse lock that the GIL is.  In the modern world, where the
>> intended behaviour of these is quite different, there is no one-size-fits
>> all.  On windows in particular, the use of the Event object based lock is
>> not ideal for other uses than the GIL.
>> >
>> >
>> > In the new GIL, there appear to be several problems:
>> > 1) There is no FIFO queue of threads wanting the queue, thus thread
>> scheduling becomes non-deterministic
>> > 2) The "ticking" of the GIL is now controled by a condition variable
>> timeout.  There appears to be no way to prevent many such timeouts to be in
>> progress at the same time, thus you may have an unnecessarily high rate of
>> ticking going on.
>> > 3) There isn't an immediate gil request made when an IO thread requests
>> the gil back, only after an initial timeout.
>> >
>> > What we are trying to write here is a thread scheduler, and that is
>> complex business.
>> > K
>> >
>> >
>> >
>> >> -Original Message-
>> >> From: python-dev-bounces+kristjan=ccpgames@python.org
>> >> [mailto:python-dev-bounces+kristjan =
>> ccpgames

Re: [Python-Dev] tagging 3.1.2

[Martin v. Löwis]

Benjamin Peterson wrote:
> My plan is to tag 3.1.2 sometime on Thursday, so binaries can be built
> for the final release on Saturday. Agreeable?

Fine with me.

Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Decimal <-> float comparisons in py3k.

[Raymond Hettinger]

On Mar 16, 2010, at 9:41 AM, Guido van Rossum wrote:

> I'd say if you're not going to forward-port this to Python 3, it
> shouldn't go into Python 2 -- in that case it would make more sense to
> me to back-port the exception-raising behavior.

Python 3 doesn't need it because it is possible to not give a result
at all.  Python 2 does need it because we have to give *some*
result.

> 
> Also supporting comparisons but not other mixed operations is going to
> be confusing. If you are sticking to that behavior I think mixed
> comparisons should also be ruled out.

The difference is that mixed comparisons currently do give a result,
but one that is non-sensical.  The proposal is a make in give a
meaningful result, not as an extra feature, but in an effort to not
be wrong.

Since 2.x has to give a result, we should make it useful.
Since 3.x does not make the comparison, it is okay to punt.


Raymond



___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Decimal <-> float comparisons in py3k.

[Guido van Rossum]

On Tue, Mar 16, 2010 at 5:36 PM, Raymond Hettinger
 wrote:
>
>
>
> On Mar 16, 2010, at 9:41 AM, Guido van Rossum wrote:
>
>> I'd say if you're not going to forward-port this to Python 3, it
>> shouldn't go into Python 2 -- in that case it would make more sense to
>> me to back-port the exception-raising behavior.
>
> Python 3 doesn't need it because it is possible to not give a result
> at all.  Python 2 does need it because we have to give *some*
> result.
>
>>
>> Also supporting comparisons but not other mixed operations is going to
>> be confusing. If you are sticking to that behavior I think mixed
>> comparisons should also be ruled out.
>
> The difference is that mixed comparisons currently do give a result,
> but one that is non-sensical.  The proposal is a make in give a
> meaningful result, not as an extra feature, but in an effort to not
> be wrong.
>
> Since 2.x has to give a result, we should make it useful.
> Since 3.x does not make the comparison, it is okay to punt.

No. You are talking like there is no path from 2.7 to 3.x. Since the
result in 2.x was never useful, you should not add a dead-end feature.
And there is no reason (other than backwards compatibility) why the
comparison couldn't raise an exception in 2.x as well -- e.g.
str<->unicode comparisons do this already. (It used to be required
that comparisons had to always return a result, but that was done away
with long before Decimal was introduced.)

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Decimal <-> float comparisons in py3k.

[Raymond Hettinger]

On Mar 16, 2010, at 3:16 PM, Greg Ewing wrote:
> 
> Seems to me that this education would mostly consist of saying
> "don't compare floats and decimals", which is why I think that
> disallowing them in the first place would be better.

That makes sense.

I do worry that 2.x currently does make the comparison
and gives the wrong answer.  We have the ability to
make it a correct answer.  But, it seems like the mood
here is that wrong-is-better-than-right for an action
that someone shouldn't be doing in the first place.


Raymond___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com