Re: [Python-Dev] New calling convention to avoid temporarily tuples when calling functions

[Victor Stinner]

Oh, I found a nice pice of CPython history in Modules/_pickle.c.
Extract of Python 3.3:
-
/* A temporary cleaner API for fast single argument function call.

   XXX: Does caching the argument tuple provides any real performance benefits?

   A quick benchmark, on a 2.0GHz Athlon64 3200+ running Linux 2.6.24 with
   glibc 2.7, tells me that it takes roughly 20,000,000 PyTuple_New(1) calls
   when the tuple is retrieved from the freelist (i.e, call PyTuple_New() then
   immediately DECREF it) and 1,200,000 calls when allocating brand new tuples
   (i.e, call PyTuple_New() and store the returned value in an array), to save
   one second (wall clock time). Either ways, the loading time a pickle stream
   large enough to generate this number of calls would be massively
   overwhelmed by other factors, like I/O throughput, the GC traversal and
   object allocation overhead. So, I really doubt these functions provide any
   real benefits.

   On the other hand, oprofile reports that pickle spends a lot of time in
   these functions. But, that is probably more related to the function call
   overhead, than the argument tuple allocation.

   XXX: And, what is the reference behavior of these? Steal, borrow? At first
   glance, it seems to steal the reference of 'arg' and borrow the reference
   of 'func'. */
static PyObject *
_Pickler_FastCall(PicklerObject *self, PyObject *func, PyObject *arg)
-

Extract of Python 3.4 (same function):
-
/* Note: this function used to reuse the argument tuple. This used to give
   a slight performance boost with older pickle implementations where many
   unbuffered reads occurred (thus needing many function calls).

   However, this optimization was removed because it was too complicated
   to get right. It abused the C API for tuples to mutate them which led
   to subtle reference counting and concurrency bugs. Furthermore, the
   introduction of protocol 4 and the prefetching optimization via peek()
   significantly reduced the number of function calls we do. Thus, the
   benefits became marginal at best. */
-

It recalls me the story of property_descr_get() optimizations :-)

I hope that the new generic "fastcall" functions will provide a safe
and reliable optimization for the pickle module, property_descr_get()
and others optimized functions.

Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] New calling convention to avoid temporarily tuples when calling functions

[Victor Stinner]

2016-08-22 10:01 GMT+02:00 Victor Stinner :
> The next step is to support keyword parameters. In fact, it's already
> supported in all cases except of Python functions:
> https://bugs.python.org/issue27809

Serhiy Storchaka proposed to use a single C array for positional and
keyword arguments. Keyword arguments are passed as (key, value) pairs.
I just added this function:

 PyAPI_FUNC(PyObject *) _PyObject_FastCallKeywords(
PyObject *func,
PyObject **stack,
Py_ssize_t nargs,
Py_ssize_t nkwargs);

The function is not used yet. Serhiy proposed to enhance the functions
to parse arguments to support this format to pass arguments which
would allow to avoid the creation of a temporary dictionary in many
cases.

I proposed to use this format (instead of (PyObject **stack,
Py_ssize_t nargs, PyObject *kwargs)) for a new METH_FASTCALL calling
convention for C functions:
https://bugs.python.org/issue27810

Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 525

[Sven R. Kunze]

On 24.08.2016 21:05, Yury Selivanov wrote:
Sorry for making you irritated.  Please feel free to remind me about 
any concrete changes to the PEP that I promised to add on 
python-ideas.  I'll go and re-read that thread right now anyways.


No problem as it seems I wasn't the only one. So, it doesn't matter 
anymore. :)


Best,
Sven
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 525

[Sven R. Kunze]

On 24.08.2016 21:00, Yury Selivanov wrote:


For an async generator there are two cases: either it tries to yield 
another value (the first time this happens you can throw an error 
back into it) or it tries to await -- in that case you can also throw 
an error back into it, and if the error comes out unhandled you can 
print the error (in both cases actually).


It's probably to specify all this behavior using some kind of default 
finalizer (though you don't have to implement it that way).


Hopefully there will be other discussion as well, otherwise I'll have 
to accept the PEP once this issue is cleared up. :-)


Curious to hear your thoughts on two different approaches to 
finalization.  At this point, I'm inclined to change the PEP to use 
the second approach.  I think it gives much more power to event loops, 
and basically means that any kind of APIs to control AG (or to 
finalize the loop) is possible.


I think your alternative approach is the better one. It feels more 
integrated even though it's harder for event loop implementors (which 
are rarer than normal event loop users). Also AG finalization is 
something that's not really custom to each AG but makes more sense at 
the event loop level, I think.


Best,
Sven
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 525

[Yury Selivanov]

On 2016-08-24 3:01 PM, Sven R. Kunze wrote:


On 24.08.2016 18:35, Guido van Rossum wrote:
On Wed, Aug 24, 2016 at 8:17 AM, Yury Selivanov 
> wrote:


On 2016-08-23 10:38 PM, Rajiv Kumar wrote:

I was playing with your implementation to gain a better
understanding of the operation of asend() and friends. Since
I was explicitly trying to "manually" advance the generators,
I wasn't using asyncio or other event loop. This meant that
the first thing I ran into with my toy code was the
RuntimeError ("cannot iterate async generator without
finalizer set").

As you have argued elsewhere, in practice the finalizer is
likely to be set by the event loop. Since the authors of
event loops are likely to know that they should set the
finalizer, would it perhaps be acceptable to merely issue a
warning instead of an error if the finalizer is not set? That
way there isn't an extra hoop to jump through for simple
examples.

In my case, I just called
sys.set_asyncgen_finalizer(lambda g: 1)
to get around the error and continue playing :) (I realize
that's a bad thing to do but it didn't matter for the toy cases)


Yeah, maybe warning would be sufficient.  I just find it's highly
unlikely that a lot of people would use async generators without
a loop/coroutine runner, as it's a very tedious process.


Heh, I had the same reaction as Rajiv. I think the tediousness is 
actually a good argument that there's no reason to forbid this. I 
don't even think a warning is needed. People who don't use a 
coroutine runner are probably just playing around (maybe even in the 
REPL) and they shouldn't get advice unasked.


I also was irritated as Yury said there were absolutely no changes 
after python-ideas. He said he might consider a clearer warning for 
those examples at the beginning of the PEP to make them work for the 
reader.


Sorry for making you irritated.  Please feel free to remind me about any 
concrete changes to the PEP that I promised to add on python-ideas.  
I'll go and re-read that thread right now anyways.


Yury
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 525

[Sven R. Kunze]

On 24.08.2016 18:35, Guido van Rossum wrote:
On Wed, Aug 24, 2016 at 8:17 AM, Yury Selivanov 
> wrote:


On 2016-08-23 10:38 PM, Rajiv Kumar wrote:

I was playing with your implementation to gain a better
understanding of the operation of asend() and friends. Since I
was explicitly trying to "manually" advance the generators, I
wasn't using asyncio or other event loop. This meant that the
first thing I ran into with my toy code was the RuntimeError
("cannot iterate async generator without finalizer set").

As you have argued elsewhere, in practice the finalizer is
likely to be set by the event loop. Since the authors of event
loops are likely to know that they should set the finalizer,
would it perhaps be acceptable to merely issue a warning
instead of an error if the finalizer is not set? That way
there isn't an extra hoop to jump through for simple examples.

In my case, I just called
sys.set_asyncgen_finalizer(lambda g: 1)
to get around the error and continue playing :) (I realize
that's a bad thing to do but it didn't matter for the toy cases)


Yeah, maybe warning would be sufficient.  I just find it's highly
unlikely that a lot of people would use async generators without a
loop/coroutine runner, as it's a very tedious process.


Heh, I had the same reaction as Rajiv. I think the tediousness is 
actually a good argument that there's no reason to forbid this. I 
don't even think a warning is needed. People who don't use a coroutine 
runner are probably just playing around (maybe even in the REPL) and 
they shouldn't get advice unasked.


I also was irritated as Yury said there were absolutely no changes after 
python-ideas. He said he might consider a clearer warning for those 
examples at the beginning of the PEP to make them work for the reader.




Would it be possible to print a warning only when an async generator 
is being finalized and doesn't run straight to the end without 
suspending or yielding? For regular generators we have a similar 
exception (although I don't recall whether we actually warn) -- if you 
call close() and it tries to yield another value it is just GC'ed 
without giving the frame more control. For an async generator there 
are two cases: either it tries to yield another value (the first time 
this happens you can throw an error back into it) or it tries to await 
-- in that case you can also throw an error back into it, and if the 
error comes out unhandled you can print the error (in both cases 
actually).


It's probably to specify all this behavior using some kind of default 
finalizer (though you don't have to implement it that way).


Does a default finalizer solve the "event loop does not know its AGs" 
problem?


Hopefully there will be other discussion as well, otherwise I'll have 
to accept the PEP once this issue is cleared up. :-)


--
--Guido van Rossum (python.org/~guido )


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/srkunze%40mail.de


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 525

[Yury Selivanov]

On 2016-08-24 12:35 PM, Guido van Rossum wrote:

On Wed, Aug 24, 2016 at 8:17 AM, Yury Selivanov 
> wrote:


On 2016-08-23 10:38 PM, Rajiv Kumar wrote:

I was playing with your implementation to gain a better
understanding of the operation of asend() and friends. Since I
was explicitly trying to "manually" advance the generators, I
wasn't using asyncio or other event loop. This meant that the
first thing I ran into with my toy code was the RuntimeError
("cannot iterate async generator without finalizer set").

As you have argued elsewhere, in practice the finalizer is
likely to be set by the event loop. Since the authors of event
loops are likely to know that they should set the finalizer,
would it perhaps be acceptable to merely issue a warning
instead of an error if the finalizer is not set? That way
there isn't an extra hoop to jump through for simple examples.

In my case, I just called
sys.set_asyncgen_finalizer(lambda g: 1)
to get around the error and continue playing :) (I realize
that's a bad thing to do but it didn't matter for the toy cases)


Yeah, maybe warning would be sufficient.  I just find it's highly
unlikely that a lot of people would use async generators without a
loop/coroutine runner, as it's a very tedious process.


Heh, I had the same reaction as Rajiv. I think the tediousness is 
actually a good argument that there's no reason to forbid this. I 
don't even think a warning is needed. People who don't use a coroutine 
runner are probably just playing around (maybe even in the REPL) and 
they shouldn't get advice unasked.


Good point.



Would it be possible to print a warning only when an async generator 
is being finalized and doesn't run straight to the end without 
suspending or yielding? For regular generators we have a similar 
exception (although I don't recall whether we actually warn) -- if you 
call close() and it tries to yield another value it is just GC'ed 
without giving the frame more control.


Yes, we can implement the exact same semantics for AGs:

- A ResourceWarning will be issued if an AG is GCed and cannot be 
synchronously closed (that will happen if no finalizer is set and there 
are 'await' expressions in 'finally').


- A RuntimeError is issued when an AG is yielding (asynchronously) in 
its 'finally' block.


I think both of those things are already there in the reference 
implementation.  So we can just lift the requirement for asynchronous 
finalizer being set before you iterate an AG.


For an async generator there are two cases: either it tries to yield 
another value (the first time this happens you can throw an error back 
into it) or it tries to await -- in that case you can also throw an 
error back into it, and if the error comes out unhandled you can print 
the error (in both cases actually).


It's probably to specify all this behavior using some kind of default 
finalizer (though you don't have to implement it that way).


Hopefully there will be other discussion as well, otherwise I'll have 
to accept the PEP once this issue is cleared up. :-)


Curious to hear your thoughts on two different approaches to 
finalization.  At this point, I'm inclined to change the PEP to use the 
second approach.  I think it gives much more power to event loops, and 
basically means that any kind of APIs to control AG (or to finalize the 
loop) is possible.


Thank you,
Yury
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 525

[Guido van Rossum]

On Wed, Aug 24, 2016 at 8:17 AM, Yury Selivanov 
wrote:

> On 2016-08-23 10:38 PM, Rajiv Kumar wrote:
>
>> I was playing with your implementation to gain a better understanding of
>> the operation of asend() and friends. Since I was explicitly trying to
>> "manually" advance the generators, I wasn't using asyncio or other event
>> loop. This meant that the first thing I ran into with my toy code was the
>> RuntimeError ("cannot iterate async generator without finalizer set").
>>
>> As you have argued elsewhere, in practice the finalizer is likely to be
>> set by the event loop. Since the authors of event loops are likely to know
>> that they should set the finalizer, would it perhaps be acceptable to
>> merely issue a warning instead of an error if the finalizer is not set?
>> That way there isn't an extra hoop to jump through for simple examples.
>>
>> In my case, I just called
>> sys.set_asyncgen_finalizer(lambda g: 1)
>> to get around the error and continue playing :) (I realize that's a bad
>> thing to do but it didn't matter for the toy cases)
>>
>
> Yeah, maybe warning would be sufficient.  I just find it's highly unlikely
> that a lot of people would use async generators without a loop/coroutine
> runner, as it's a very tedious process.
>

Heh, I had the same reaction as Rajiv. I think the tediousness is actually
a good argument that there's no reason to forbid this. I don't even think a
warning is needed. People who don't use a coroutine runner are probably
just playing around (maybe even in the REPL) and they shouldn't get advice
unasked.

Would it be possible to print a warning only when an async generator is
being finalized and doesn't run straight to the end without suspending or
yielding? For regular generators we have a similar exception (although I
don't recall whether we actually warn) -- if you call close() and it tries
to yield another value it is just GC'ed without giving the frame more
control. For an async generator there are two cases: either it tries to
yield another value (the first time this happens you can throw an error
back into it) or it tries to await -- in that case you can also throw an
error back into it, and if the error comes out unhandled you can print the
error (in both cases actually).

It's probably to specify all this behavior using some kind of default
finalizer (though you don't have to implement it that way).

Hopefully there will be other discussion as well, otherwise I'll have to
accept the PEP once this issue is cleared up. :-)

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] File system path encoding on Windows

[Steve Dower]

On 23Aug2016 2150, Stephen J. Turnbull wrote:

Steve Dower writes:

 > * Stephen sees "no reason not to change locale.getpreferredencoding()"
 > (default encoding for open()) at the same time with the same switches,
 > while I'm not quite as confident. Do users generally specify an encoding
 > these days? I know I always put utf-8 there.

I was insufficiently specific.  "No reason not to" depends on separate
switches for file system encoding and preferred encoding.  That makes
things somewhat more complicated for implementation, and significantly
so for users.


Yes, it does, but it's about the only possible migration path.

I know Nick and Victor like the idea of a -X flag (or a direct -utf8 
flag), but I prefer more specific environment variables:


- PYTHONWINDOWSLEGACYSTDIO (for the console changes)
- PYTHONWINDOWSLEGACYPATHENCODING (assuming getfilesystemencoding() is utf8)
- PYTHONWINDOWSLEGACYLOCALEENCODING (assuming getpreferredencoding() is 
utf8)


I'm open to dropping "WINDOWS" from these if others don't think it's 
necessary. Swap "LEGACY" for "UNICODE" if we just offer the option to 
upgrade now rather than changing the default.


(I could also see the possibility of PYTHONWINDOWSSTRICT* options to use 
the default encoding but raise if decoding bytes fails - mbcs:strict 
rather than mbcs:replace. For utf-8 mode I'd want to use surrogatepass 
throughout, so it will always raise on invalid encoding, but there will 
be far fewer occurrences than for mbcs.)


I'll transform my earlier post into a PEP (or probably three PEPs - one 
for each change), since it seems like the paper trail is going to be 
more valuable than discussing it now. Without an actual build to test 
it's pretty hard to evaluate the impact.


Cheers,
Steve

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 525

[Yury Selivanov]

Hi Rajiv,

On 2016-08-23 10:38 PM, Rajiv Kumar wrote:

Hi Yury,

I was playing with your implementation to gain a better understanding 
of the operation of asend() and friends. Since I was explicitly trying 
to "manually" advance the generators, I wasn't using asyncio or other 
event loop. This meant that the first thing I ran into with my toy 
code was the RuntimeError ("cannot iterate async generator without 
finalizer set").


As you have argued elsewhere, in practice the finalizer is likely to 
be set by the event loop. Since the authors of event loops are likely 
to know that they should set the finalizer, would it perhaps be 
acceptable to merely issue a warning instead of an error if the 
finalizer is not set? That way there isn't an extra hoop to jump 
through for simple examples.


In my case, I just called
sys.set_asyncgen_finalizer(lambda g: 1)
to get around the error and continue playing :) (I realize that's a 
bad thing to do but it didn't matter for the toy cases)


Yeah, maybe warning would be sufficient.  I just find it's highly 
unlikely that a lot of people would use async generators without a 
loop/coroutine runner, as it's a very tedious process.


Thank you,
Yury
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com