date:20150424


Victor Stinner wrote:

Oh, I missed something in the PEP 3152: a obj__cocall__() method can
be an iterator/generator, it can be something different than a
cofunction.


In fact, it *can't* be cofunction. It's part of the
machinery for implementing cofunctions.


It's not easy to understand the whole puzzle. IMO the PEP 492 better
explains how pieces are put together ;-)


Yes, it's written in a rather minimal style, sorry
about that.

--
Greg
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3152 and yield from Future()


Yury Selivanov wrote:


It's a common pattern in asyncio when functions
return futures.  It's OK later to refactor those
functions to coroutines *and* vice-versa.  This
is a fundamental problem for PEP 3152 approach.


Hmmm. So you have an ordinary function that returns
a future, and you want to turn it into a coroutine
function, but still have it return a future in
order to keep the API the same, is that right?

Turning it into a coroutine means you're going
to have to change every site that calls it, so
its API has already changed. Given that, I'm not
sure what advantage there is in keeping the future-
returning part of the API.

However, if we use the await()-cofunction idea,
then a call to the initial version looks like

   cocall await(f(x))

and after the refactoring it becomes

   cocall await(cocall f(x))

That doesn't look so bad to me.

--
Greg
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] async/await in Python; v2


Stephen J. Turnbull wrote:

Yury Selivanov writes:

  I also read for async item in iter: as I'm iterating iter
  with async item.

I thought that was precisely the intended semantics: item is available
asynchronously.


The async-at-the-end idea could be used here as well.

   for item in iter async:
  ...

   with something as x async:
  ...

--
Greg
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] async/await in Python; v2


Barry Warsaw wrote:

Sure, tools can be updated but it is it *necessary*
to choose a syntax that breaks tools?

def async useful():

seems okay to me.


That will break any tool that assumes the word following
'def' is the name of the function being defined.

Putting it at the end would seem least likely to
cause breakage:

   def useful() async:

--
Greg
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] async/await in Python; v2


Yury Selivanov wrote:

In a PEP 3152 aware version of asyncio, it's just *not
possible to write*

cocall gather(coro1(1,2), coro(2,3))

you just have to use your 'costart' built-in:

cocall gather(costart(coro1, 1, 2), costart(coro, 2,3)).


Another way to write that would be

  cocall gather(Task(coro1, 1, 2), Task(coro, 2, 3))

I think that actually reads quite nicely, and makes it
very clear that parallel tasks are being spawned, rather
than invoked sequentially. With the current way, that's
not clear at all.

It's not quite as convenient, because you don't get
currying for free the way you do with generators. But
I feel that such implicit currying is detrimental to
readability. It looks like you're passing the results
returned by coro1 and coro2 to gather, rather than
coro1 and coro2 themselves.

Yes, it will require some code to be changed, but
if you're turning all your coroutines into cofunctions
or async defs, you're changing quite a lot of things
already.


PEP 3152 was created in pre-asyncio era, and it shows.


I would say that asyncio was created in a pre-PEP-3152
world. Or at least it was developed without allowing
for the possibility of adopting something like PEP 3152
in the future.

Asyncio was based on generators and yield-from because
it was the best thing we had at the time. I'll be
disappointed if we've raced so far ahead with those
ideas that it's now impossible to replace them with
anything better.

PEP 3152 is designed to present a *simpler* model of
coroutine programming, by having only one concept, the
suspendable function, instead of two -- generator
functions on the one hand, and iterators/futures/awaitables/
whatever you want to call them on the other.

PEP 492 doesn't do that. It adds some things and
changes some things, but it doesn't simplify anything.


Your idea of syntaticaly forcing to use 'cocall' with
parens is cute,


You say that as though forcing the use of parens were
a goal in itself. It's not -- it's a *consequence* of
what a cocall is.

--
Greg
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3152 and yield from Future()


Guido van Rossum wrote:

I think this is the nail in PEP 3152's coffin.


Seems more like a small tack to me. :-)
I've addressed all the issues raised there in
earlier posts.

--
Greg
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] async/await in Python; v2

2015-04-24 Thread Andrew Svetlov

On Fri, Apr 24, 2015 at 3:14 AM, Greg Ewing greg.ew...@canterbury.ac.nz wrote:
 Andrew Svetlov wrote:

 But we already have asyncio and code based on asyncio coroutines.
 To make it work I should always use costart() in places where asyncio
 requires coroutine.


 As I understand it, asyncio would require changes to
 make it work seamlessly with PEP 492 as well, since
 an object needs to have either a special flag or
 an __await__ method before it can have 'await'
 applied to it.

PEP 492 requires a change of asyncio.Future only.
PEP 3152 requires of change in any asyncio-based library, this is the
difference.

 --
 Greg
 ___
 Python-Dev mailing list
 Python-Dev@python.org
 https://mail.python.org/mailman/listinfo/python-dev
 Unsubscribe:
 https://mail.python.org/mailman/options/python-dev/andrew.svetlov%40gmail.com



-- 
Thanks,
Andrew Svetlov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] async/await in Python; v2


Yury Selivanov wrote:


If you have a future object 'fut', it's not intuitive
or pythonic to write 'cocall fut()'.


Another way to approach that would be to provide
a cofunction await() used like this:

   cocall await(fut)

That would read more naturally and wouldn't require
modifying fut at all.

--
Greg
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3152 and yield from Future()


Paul Moore wrote:

On 24 April 2015 at 09:34, Greg Ewing greg.ew...@canterbury.ac.nz wrote:


  cocall await(cocall f(x))

That doesn't look so bad to me.


I've not been following this discussion (and coroutines make my head
hurt) but this idiom looks like it's bound to result in people getting
the idea that you scatter cocall throughout an expression until you
get it to work.


They won't need to do that, because they'll get told
exactly where they've left one out, or put one in that
they shouldn't have.

Also, the places you need to put cocall are exactly
the same as the places you need yield-from currently,
or await under PEP 492.

--
Greg
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] typeshed for 3rd party packages

On Wed, Apr 22, 2015 at 11:26:14AM -0500, Ian Cordasco wrote:

 On a separate thread Cory provided an example of what the hints would look
 like for *part* of one function in the requests public functional API.
 While our API is outwardly simple, the values we accept in certain cases
 are actually non-trivially represented. Getting the hints *exactly* correct
 would be extraordinarily difficult.

I don't think you need to get them exactly correct. The type-checker 
does two things:

(1) catch type errors involving types which should not be allowed;

(2) allow code which involves types which should be allowed.

If the type hints are wrong, there are two errors: false positives, when 
code which should be allowed is flagged as a type error; and false 
negatives, when code which should be flagged as an error is not.
Ideally, there should be no false positives. But false negatives are not 
so important, since you will still be doing runtime checks. All that 
means is that the static type-checker will be a little less capable of 
picking up type errors at compile time.


-- 
Steve
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] async/await in Python; v2

2015-04-24 Thread Barry Warsaw

On Apr 24, 2015, at 08:03 PM, Greg Ewing wrote:

Putting it at the end would seem least likely to
cause breakage:

def useful() async:

That's not bad IMHO.  I wonder how crazy it is in the face of, ahem, function
annotations.

Cheers,
-Barry
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] async/await in Python; v2

On Thu, Apr 23, 2015 at 01:51:52PM -0400, Barry Warsaw wrote:

 Why async def and not def async?
 
 My concern is about existing tools that already know that def as the first
 non-whitespace on the line starts a function/method definition.  Think of a
 regexp in an IDE that searches backwards from the current line to find the
 function its defined on.  Sure, tools can be updated but it is it *necessary*
 to choose a syntax that breaks tools?

Surely its the other way? If I'm searching for the definition of a 
function manually, I search for def spam. `async def spam` will still 
be found, while `def async spam` will not.

It seems to me that tools that search for r^\s*def\s+spam\s*\( are 
going to break whichever choice is made, while a less pedantic search 
like rdef\s+spam\s*\( will work only if async comes first.


-- 
Steve
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] async/await in Python; v2

2015-04-24 Thread Barry Warsaw

On Apr 24, 2015, at 11:17 PM, Steven D'Aprano wrote:

It seems to me that tools that search for r^\s*def\s+spam\s*\( are

They would likely search for something like r^\s*def\s+[a-zA-Z0-9_]+ which
will hit def async spam but not async def.

Cheers,
-Barry
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3152 and yield from Future()


Greg,

On 2015-04-24 4:13 AM, Greg Ewing wrote:

Guido van Rossum wrote:

I think this is the nail in PEP 3152's coffin.


Seems more like a small tack to me. :-)
I've addressed all the issues raised there in
earlier posts.



I'm sorry, but this is ridiculous.

You haven't addressed the issues. We raise issues,
saying that your PEP isn't backwards compatible and
is breaking existing idioms.  You say - there is a
workaround for that; or that we can rewrite that and
make it like that; or something else that doesn't
make any sense for existing asyncio developers.

Your PEP isn't backwards compatible. Period.

It *will* be harder for people to get, as it *does*
introduce a new calling grammar that isn't obvious
for at least some people.

We, asyncio developers, who write asyncio code,
*don't* want to write 'cocall fut()'.  I don't
understand *why* I'm required to put parentheses
there (besides someone just requiring me to do so,
because they failed to solve some problem in
backwards compatible way).  You avoid confusion
in one place, but you introduce it in other places.

I'm sorry, but your current way of handling the
discussion isn't really productive.  You don't
listen to arguments by Victor Stinner, Andrew
Svetlov, and me.  At this point, this whole PEP
3152 related discussion isn't helping anyone.

Yury

P.S. I'm sorry if this sounded harsh, this wasn't
my intent.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3152 and yield from Future()

2015-04-24 Thread Andrew Svetlov

On Fri, Apr 24, 2015 at 11:34 AM, Greg Ewing
greg.ew...@canterbury.ac.nz wrote:
 Yury Selivanov wrote:

 It's a common pattern in asyncio when functions
 return futures.  It's OK later to refactor those
 functions to coroutines *and* vice-versa.  This
 is a fundamental problem for PEP 3152 approach.


 Hmmm. So you have an ordinary function that returns
 a future, and you want to turn it into a coroutine
 function, but still have it return a future in
 order to keep the API the same, is that right?

No. In asyncio there is no difference between coroutine and regular
function returning future.
From caller site next both are equal:

@asyncio.coroutine
def f():
return 1

def g():
fut = asyncio.Future()
fut.set_result(1)
return fut

Both may be called via `yield from`:
ret1 = yield from f()
ret2 = yield from g()


 Turning it into a coroutine means you're going
 to have to change every site that calls it, so
 its API has already changed. Given that, I'm not
 sure what advantage there is in keeping the future-
 returning part of the API.

 However, if we use the await()-cofunction idea,
 then a call to the initial version looks like

cocall await(f(x))

 and after the refactoring it becomes

cocall await(cocall f(x))

 That doesn't look so bad to me.

 --
 Greg

 ___
 Python-Dev mailing list
 Python-Dev@python.org
 https://mail.python.org/mailman/listinfo/python-dev
 Unsubscribe:
 https://mail.python.org/mailman/options/python-dev/andrew.svetlov%40gmail.com



-- 
Thanks,
Andrew Svetlov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] async/await in Python; v2

On Fri, Apr 24, 2015 at 09:32:51AM -0400, Barry Warsaw wrote:
 On Apr 24, 2015, at 11:17 PM, Steven D'Aprano wrote:
 
 It seems to me that tools that search for r^\s*def\s+spam\s*\( are
 
 They would likely search for something like r^\s*def\s+[a-zA-Z0-9_]+ which
 will hit def async spam but not async def.

Unless somebody wants to do a survey of editors and IDEs and other 
tools, arguments about what regex they may or may not use to search for 
function definitions is an exercise in futility. They may use regexes 
anchored to the start of the line. They may not. They may deal with def 
async better than async def, or the other way around. Either way, 
it's a pretty thin argument for breaking the invariant that the token 
following `def` is the name of the function.

Whatever new syntax is added, something is going to break.

-- 
Steve
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 492 vs. PEP 3152, new round

2015-04-24 Thread Guido van Rossum

On Fri, Apr 24, 2015 at 11:03 AM, Ethan Furman et...@stoneleaf.us wrote:

 On 04/24, Yury Selivanov wrote:
  On 2015-04-24 1:03 PM, Guido van Rossum wrote:

  Ditto for `__aiter__` and `__anext__`. I guess this means that the async
  equivalent to obtaining an iterator through `it = iter(xs)` followed by
  `for x over it` will have to look like `ait = await aiter(xs)` followed
 by
  `for x over ait`, where an iterator is required to have an `__aiter__`
  method that's an async function and returns self immediately. But what
 if
  you left out the `await` from the first call? I.e. can this work?
  ```
  ait = aiter(xs)
  async for x in ait:
   print(x)
 
  With the current semantics that PEP 492 proposes, await
  for aiter() is mandatory.
 
  You have to write
 
  ait = await aiter(xs)
  async for x in ait:
  print(c)

 As a new user to asyncio and this type of programming in general, 'await
 aiter'
 feels terribly redundant.


Yeah, but normally you would never do that. You'd just use `async for x in
xs`. I'm just bickering over the exact expansion of that.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 492 vs. PEP 3152, new round

2015-04-24 Thread Łukasz Langa


 On Apr 24, 2015, at 10:03 AM, Guido van Rossum gu...@python.org wrote:
 
 6. StopAsyncException
 
 What if we required `ait.__anext__()` to return a future?

On top of my previous response, one more thing to consider is that this idea 
brings a builtin Future back to the proposal, which has already been rejected 
in the No implicit wrapping in Futures” section of the PEP.

PEP 492 manages to solve all issues without introducing a built-in Future.

-- 
Best regards,
Łukasz Langa

WWW: http://lukasz.langa.pl/
Twitter: @llanga
IRC: ambv on #python-dev

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 492 vs. PEP 3152, new round

2015-04-24 Thread Guido van Rossum

Sorry, when I wrote future (lower-case 'f') I really meant what Yury
calls *awaitable*. That's either a coroutine or something with an __await__
emthod.

On Fri, Apr 24, 2015 at 3:17 PM, Łukasz Langa luk...@langa.pl wrote:


 On Apr 24, 2015, at 10:03 AM, Guido van Rossum gu...@python.org wrote:

 *6. StopAsyncException*

 What if we required `ait.__anext__()` to return a future?


 On top of my previous response, one more thing to consider is that this
 idea brings a builtin Future back to the proposal, which has already been
 rejected in the No implicit wrapping in Futures” section of the PEP.

 PEP 492 manages to solve all issues without introducing a built-in Future.

 --
 Best regards,
 Łukasz Langa

 WWW: http://lukasz.langa.pl/
 Twitter: @llanga
 IRC: ambv on #python-dev




-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Type hints -- a mediocre programmer's reaction

2015-04-24 Thread Ronan Lamy

Le 24/04/15 19:45, Paul Sokolovsky a écrit :

Hello,

On Fri, 24 Apr 2015 18:27:29 +0100
Ronan Lamy ronan.l...@gmail.com wrote:

PyPy's FAQ
has an explanation of why type hints are not for performance.
http://pypy.readthedocs.org/en/latest/faq.html#would-type-annotations-help-pypy-s-performance

You probably intended to write why type hints are not for *PyPy's*
performance. There're many other language implementations and
modules for which it may be useful, please don't limit your
imagination by a single case.

Those points apply to basically any compliant implementation of
Python relying on speculative optimisation. Python is simply too
dynamic for PEP484-style hints to provide any useful performance
improvement targets.

What's your point - saying that type annotations alone not enough to
achieve the best (C-like) performance, which is true, or saying that
if they are alone not enough, then they are not needed at all, which
is ... strange ?

My point is that the arguments in the PyPy FAQ aren't actually specific
to PyPy, and therefore that the conclusion, that hints are almost
entirely useless if you’re looking at performance, holds in general.
So let me restate these arguments in terms of a generic,
performance-minded implementation of the full Python language spec:

* Hints have no run-time effect. The interpreter cannot assume that they
are obeyed.
* PEP484 hints are too high-level. Replacing an 'int' object with a
single machine word would be useful, but an 'int' annotation gives no
guarantee that it's correct (because Python 3 ints can have arbitrary
size and because subclasses of 'int' can override any operation to
invoke arbitrary code).
* A lot more information is needed to produce good code (e.g. “this f()
called here really means this function there, and will never be
monkey-patched” – same with len() or list(), btw).

* Most of this information cannot easily be expressed as a type
* If the interpreter gathers all that information, it'll probably have
gathered a superset of what PEP484 can provide anyway.

And speaking of PyPy, it really should think how to improve its
performance - not of generated programs, but of generation itself.
If compilation of a trivial program on a pumpy hardware takes 5
minutes and gigabytes of RAM and diskspace, few people will use it
for other purposes beyond curiosity. There's something very
un-Pythonic in waiting 5 mins just to run 10-line script. Type
hints can help here too ;-) (by not wasting resources propagating
types thru the same old standard library for example).

Sorry, but that's nonsense. PyPy would be a seriously useless
interpreter if running a 10-line script required such a lengthy
compilation, so, obviously, that's not what happens.

You seem to misunderstand what PyPy is: it's an interpreter with a
just-in-time compiler, not a static compiler. It doesn't generate
programs in any meaningful sense. Instead, it interprets the program,
and when it detects a hot code path, it compiles it to machine code
based on the precise types it sees. No resources are wasted on code
that isn't actually executed.

Regardless of whether I understood that meta-meta stuff, I just
followed couple of tutorials, each of them warning of memory and disk
space issues, and both running long to get results. Everyone else
following tutorials will get the same message I did - PyPy is a
slow-to-work-with bloat.

Ah, I suppose you're talking about the RPython tool chain, which is used
to build PyPy. Though it's an interesting topic in itself (and is pretty
much comparable to Cython wrt. type hints), it has about as much
relevance to PyPy users as the inner workings of GCC have to CPython users.

Well, the thing is that people don't seem to want to write PyPy
tutorials, because it's boring. However, I can give you the definitive
3-line version:

1. Download and install PyPy [http://pypy.org/download.html]
2. Launch the 'pypy' executable.
3. Go read https://docs.python.org/2/tutorial/

As for uber-meta stuff PyPy offers - I'm glad that's all done in
my favorite language, leaving all other languages behind. I'm saddened
there's no mundane JIT or static compiler usable and accepted by all
community - many other languages have that.

This all goes pretty offtopic wrt to the original discussion, so again,
what's your point - you say that all these things can't be done in
Python, or there's no need for it to be done? That people should look
somewhere else? I submitted a bug to jinja2 project and posted message
on its mailing list - I didn't get reply for 3 months. Why? Because its
maintainer went hacking another language, how was it called, alGOl, or
something. You want me and other folks to go too? Sorry, I'm staying so
far, and keep dreaming of better Python's future (where for example if
I need to get more performance from existing app, I can gradually
optimize it based on need, not rewrite it in another language or be
hitting not implemented in

[Python-Dev] Summary of Python tracker Issues

2015-04-24 Thread Python tracker


ACTIVITY SUMMARY (2015-04-17 - 2015-04-24)
Python tracker at http://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.

Issues counts and deltas:
  open4814 (+22)
  closed 31000 (+43)
  total  35814 (+65)

Open issues with patches: 2249 


Issues opened (46)
==

#15582: Enhance inspect.getdoc to follow inheritance chains
http://bugs.python.org/issue15582  reopened by serhiy.storchaka

#23991: ZipFile sanity checks
http://bugs.python.org/issue23991  opened by Antony.Lee

#23992: multiprocessing: MapResult shouldn't fail fast upon exception
http://bugs.python.org/issue23992  opened by neologix

#23993: Use surrogateescape error handler by default in open() if the 
http://bugs.python.org/issue23993  opened by haypo

#23994: argparse fails to detect program name when there is a slash at
http://bugs.python.org/issue23994  opened by boramalper

#23995: msvcrt could not be imported
http://bugs.python.org/issue23995  opened by petrikas

#23996: _PyGen_FetchStopIterationValue() crashes on unnormalised excep
http://bugs.python.org/issue23996  opened by scoder

#23997: unicodedata_UCD_lookup() has theoretical buffer overflow
http://bugs.python.org/issue23997  opened by christian.heimes

#23999: Undefined behavior in dtoa.c (rshift 32 of 32bit data type)
http://bugs.python.org/issue23999  opened by christian.heimes

#24000: More fixes for the Clinic mapping of converters to format unit
http://bugs.python.org/issue24000  opened by larry

#24001: Clinic: use raw types in types= set
http://bugs.python.org/issue24001  opened by larry

#24004: avoid explicit generator type check in asyncio
http://bugs.python.org/issue24004  opened by scoder

#24009: Get rid of rare format units in PyArg_Parse*
http://bugs.python.org/issue24009  opened by serhiy.storchaka

#24010: Add error checks to PyInit__locale()
http://bugs.python.org/issue24010  opened by christian.heimes

#24011: Add error checks to PyInit_signal()
http://bugs.python.org/issue24011  opened by christian.heimes

#24012: Add error checks to PyInit_pyexpat()
http://bugs.python.org/issue24012  opened by christian.heimes

#24013: Improve os.scandir() and DirEntry documentation
http://bugs.python.org/issue24013  opened by benhoyt

#24015: timeit should start with 1 loop, not 10
http://bugs.python.org/issue24015  opened by nomeata

#24016: Add a Sprints organization/preparation section to devguide
http://bugs.python.org/issue24016  opened by willingc

#24017: Implemenation of the PEP 492 - Coroutines with async and await
http://bugs.python.org/issue24017  opened by haypo

#24018: add a Generator ABC
http://bugs.python.org/issue24018  opened by scoder

#24021: document urllib.urlretrieve
http://bugs.python.org/issue24021  opened by krichter

#24022: Python heap corruption issue
http://bugs.python.org/issue24022  opened by benjamin.peterson

#24024: str.__doc__ needs an update
http://bugs.python.org/issue24024  opened by lemburg

#24026: Python hangs forever in wait() of threading.py
http://bugs.python.org/issue24026  opened by appidman

#24027: IMAP library lacks documentation about expected parameter type
http://bugs.python.org/issue24027  opened by pmoleri

#24028: Idle: add doc subsection on calltips
http://bugs.python.org/issue24028  opened by terry.reedy

#24030: IMAP library encoding enhancement
http://bugs.python.org/issue24030  opened by pmoleri

#24032: urlparse.urljoin does not add query part
http://bugs.python.org/issue24032  opened by albertsmuktupavels

#24033: Update _test_multiprocessing.py to use script helpers
http://bugs.python.org/issue24033  opened by bobcatfish

#24034: Make fails Objects/typeslots.inc
http://bugs.python.org/issue24034  opened by masamoto

#24035: When Caps Locked, Shift + alpha-character still displayed as
http://bugs.python.org/issue24035  opened by principia1687

#24036: GB2312 codec is using a wrong covert table
http://bugs.python.org/issue24036  opened by Ma Lin

#24037: Argument Clinic: add the boolint converter
http://bugs.python.org/issue24037  opened by serhiy.storchaka

#24039: Minimize option doesn't work on Search Dialog box for idle
http://bugs.python.org/issue24039  opened by prince09cs

#24040: plistlib assumes dict_type is descendent of dict
http://bugs.python.org/issue24040  opened by Behdad.Esfahbod

#24041: Implement Mac East Asian encodings properly
http://bugs.python.org/issue24041  opened by Behdad.Esfahbod

#24042: Convert os._getfullpathname() and os._isdir() to Argument Clin
http://bugs.python.org/issue24042  opened by serhiy.storchaka

#24043: Implement mac_romanian and mac_croatian encodings
http://bugs.python.org/issue24043  opened by Behdad.Esfahbod

#24045: Behavior of large returncodes  (sys.exit(nn))
http://bugs.python.org/issue24045  opened by ethan.furman

#24046: Incomplete build on AIX
http://bugs.python.org/issue24046  opened by aixto...@gmail.com

#24048: remove_module() needs to save/restore exception state

Re: [Python-Dev] Type hints -- a mediocre programmer's reaction

2015-04-24 Thread Kevin Modzelewski

On Fri, Apr 24, 2015 at 6:05 PM, Ronan Lamy ronan.l...@gmail.com wrote:

Le 24/04/15 19:45, Paul Sokolovsky a écrit :

Hello,

On Fri, 24 Apr 2015 18:27:29 +0100
Ronan Lamy ronan.l...@gmail.com wrote:

PyPy's FAQ
has an explanation of why type hints are not for performance.

http://pypy.readthedocs.org/en/latest/faq.html#would-type-annotations-help-pypy-s-performance

My point is that the arguments in the PyPy FAQ aren't actually specific to
PyPy, and therefore that the conclusion, that hints are almost entirely
useless if you’re looking at performance, holds in general.
So let me restate these arguments in terms of a generic,
performance-minded implementation of the full Python language spec:

* Hints have no run-time effect. The interpreter cannot assume that they
are obeyed.
* PEP484 hints are too high-level. Replacing an 'int' object with a single
machine word would be useful, but an 'int' annotation gives no guarantee
that it's correct (because Python 3 ints can have arbitrary size and
because subclasses of 'int' can override any operation to invoke arbitrary
code).
* A lot more information is needed to produce good code (e.g. “this f()
called here really means this function there, and will never be
monkey-patched” – same with len() or list(), btw).
* Most of this information cannot easily be expressed as a type
* If the interpreter gathers all that information, it'll probably have
gathered a superset of what PEP484 can provide anyway.

I'm with the PyPy folks here -- I don't see any use for PEP 484 type hints
from a code generation perspective. Even if the hints were guaranteed to
be correct, the PEP 484 type system doesn't follow substitutability. I
don't mean that as a critique, I think it's a decision that makes it more
useful by keeping it in line with the majority of type usage in Python, but
it means that even if the hints are correct they don't really end up
providing any guarantees to the JIT.

Sorry, but that's nonsense. PyPy would be a seriously useless
interpreter if running a 10-line script required such a lengthy
compilation, so, obviously, that's not what happens.

Ah, I suppose you're talking about the RPython tool chain, which is used
to build PyPy. Though it's an interesting topic in itself (and is pretty
much comparable to Cython wrt. type hints), it has about as much relevance
to PyPy users as the inner workings of GCC have to CPython users.

Well, the thing is that people don't seem to want to write PyPy tutorials,
because it's boring. However, I can give you the definitive 3-line version:
1. Download and install PyPy [http://pypy.org/download.html]
2. Launch the 'pypy' executable.
3. Go read https://docs.python.org/2/tutorial/

This all goes pretty offtopic wrt to the original discussion, so again,
what's your point - you say that all these things can't

Re: [Python-Dev] Type hints -- a mediocre programmer's reaction

On Sat, Apr 25, 2015 at 02:05:15AM +0100, Ronan Lamy wrote:

 * Hints have no run-time effect. The interpreter cannot assume that they 
 are obeyed.

I know what you mean, but just for the record, annotations are runtime 
inspectable, so people can (and probably have already started) to write 
runtime argument checking decorators or frameworks which rely on the 
type hints.


 * PEP484 hints are too high-level. Replacing an 'int' object with a 
 single machine word would be useful, but an 'int' annotation gives no 
 guarantee that it's correct (because Python 3 ints can have arbitrary 
 size and because subclasses of 'int' can override any operation to 
 invoke arbitrary code).

Then create your own int16, uint64 etc types.


 * A lot more information is needed to produce good code (e.g. “this f() 
 called here really means this function there, and will never be 
 monkey-patched” – same with len() or list(), btw).
 * Most of this information cannot easily be expressed as a type
 * If the interpreter gathers all that information, it'll probably have 
 gathered a superset of what PEP484 can provide anyway.

All this is a red herring. If type hints are useful to PyPy, that's a 
bonus. Cython uses its own system of type hints, a future version may be 
able to use PEP 484 hints instead. But any performance benefit is a 
bonus. PEP 484 is for increasing correctness, not speed.



-- 
Steve
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 492 vs. PEP 3152, new round


Guido van Rossum wrote:
Yury, could you tweak the syntax for `await` so that we can write the 
most common usages without parentheses? In particular I'd like to be 
able to write

```
return await foo()
with await foo() as bar: ...
foo(await bar(), await bletch())
```


Making 'await' a prefix operator with the same precedence
as unary minus would allow most reasonable usages, I think.

The only reason yield from has such a constrained syntax
is that it starts with yield, which is similarly constrained.
Since 'await' is a brand new keyword isn't bound by those constraints.

--
Greg
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Type hints -- a mediocre programmer's reaction

On 22 April 2015 at 03:03, Carol Willing willi...@willingconsulting.com wrote:
 2. Clearly, great thought has been put into this PEP. If anyone has a good
 analysis of the potential impact on Python 3 adoption, please do pass along.
 I would be interested in reading the information.

I don't have hard data, but I do get to see quite a few of the
challenges that dynamically typed languages (including Python) have
scaling to large development teams. When you look at the way folks
actually *use* languages like Java and C# in practice, you find that
they're almost universally tool mavens (in Oliver Steele's sense:
http://blog.osteele.com/posts/2004/11/ides/) where automated tools are
taking care of most of the boilerplate for them. Find me a C#
developer, and I'll bet you they're a Visual Studio user, find me a
Java developer, and I'll bet you they're an Eclipse or IntelliJ user.
This approach actually offers a lot of benefits in putting a skill
floor under a development team - while you can't make people think,
you can at least automatically rule out broad categories of mundane
errors, and focus on the business logic of the problem you're aiming
to solve.

As a result, my main reaction to PEP 484 in a Python 3 adoption
context is that Python 3 offers all the agility and flexibility of
Python 2, with all the structural assurances of Java or C# is
actually a huge selling point for anyone in an institutional context
attempting to persuade their management chain to back a migration
effort from Python 2 to Python 3.

Another factor to consider here is that my understanding is that one
of the *reasons* folks want better structural analysis (by annotating
the Python 2 stdlib and key third part libraries in typeshed) is to
help automate Python 2 - Python 3 conversions in the absence of
comprehensive test coverage. While to some degree this works against
the previous point, there's a difference between having this as an
addon vs having it as a standard feature (and unlike the network
security changes and bundling pip, this is one where I'm entirely
happy leaving it as the kind of carrot that can take pride of place in
a corporate business case).

The legitimate concerns that arise are around what happens to
*community* code bases, including the standard library itself, as well
as what folks are likely to see if they run inspect.getsource() on
standard library components. For that, I think there's a lot of value
in continuing to have explicit type hints be the exception rather than
the rule in the upstream community, so the idea of the typeshed
project is enormously appealing to me. If anyone doesn't want to deal
with type hints themselves, but has a contributor that really wants to
annotate their library, then take it to typeshed will hopefully
become a recurring refrain :)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] How to behave regarding commiting

On 17 April 2015 at 11:41, Berker Peksağ berker.pek...@gmail.com wrote:
 On Fri, Apr 17, 2015 at 4:32 AM, Facundo Batista
 facundobati...@gmail.com wrote:
 Hola!

 I'm asking this because quite some time passed since I was active in
 the development of our beloved language.

 I'm trying to not break any new rule not known by me.

 I opened a bug recently [0], somebody else proposed a patch that I
 like. However, that patch has no test. I will do a test for that code,
 but then what?

 Shall I just commit and push? Or the whole branch should be proposed
 for further reviewing?

 Hi,

 Since writing a test for that patch is simple, I'd just commit it to
 the default branch.

(Catching up on several days of python-dev email)

The ideal case is getting pre-commit reviews, but as a matter of
practicality, we each get to decide whether or not we're OK with just
pushing a particular change and relying on the buildbots and
post-commit review.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Surely nullable is a reasonable name?

On 22 April 2015 at 03:31, Larry Hastings la...@hastings.org wrote:

 On 04/21/2015 04:50 AM, Tal Einat wrote:

 As for the default set of accepted types for various convertors, if we
 could choose any syntax we liked, something like accept=+{NoneType}
 would be much better IMO.


 In theory Argument Clinic could use any syntax it likes.  In practice, under
 the covers we tease out one or two bits of non-Python syntax, then run
 ast.parse over it.  Saved us a lot of work.

 s: accept={str,NoneType} is a legal Python parameter declaration; s:
 accept+={NoneType} is not.  If I could figure out a clean way to hack in
 support for += I'll support it.  Otherwise you'll be forced to spell it out.

Ellipsis seems potentially useful here to mean whatever the default
accepted types are: s: accept={...,NoneType}

My other question would be whether we can use None in preference to
NoneType, as PEP 484 does:
https://www.python.org/dev/peps/pep-0484/#using-none

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] async/await in Python; v2


Wild idea:

Let @ mean async when it's directly in front
of a keyword.

Then we would have:

  @def f():
 ...

  @for x in iter:
 ...

  @with context as thing:
 ...

--
Greg
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Surely nullable is a reasonable name?

On 25 April 2015 at 14:44, Nick Coghlan ncogh...@gmail.com wrote:
 On 22 April 2015 at 03:31, Larry Hastings la...@hastings.org wrote:

 On 04/21/2015 04:50 AM, Tal Einat wrote:

 As for the default set of accepted types for various convertors, if we
 could choose any syntax we liked, something like accept=+{NoneType}
 would be much better IMO.


 In theory Argument Clinic could use any syntax it likes.  In practice, under
 the covers we tease out one or two bits of non-Python syntax, then run
 ast.parse over it.  Saved us a lot of work.

 s: accept={str,NoneType} is a legal Python parameter declaration; s:
 accept+={NoneType} is not.  If I could figure out a clean way to hack in
 support for += I'll support it.  Otherwise you'll be forced to spell it out.

 Ellipsis seems potentially useful here to mean whatever the default
 accepted types are: s: accept={...,NoneType}

Ah, I misread Tal's suggestion. Using unary + is an even neater approach.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 492 vs. PEP 3152, new round


Victor Stinner wrote:


That's why I suggest to reconsider the idea of supporting an
*optional* from __future__ import async to get async and await as
keywords in the current file. This import would allow all crazy
syntax. The parser might suggest to use the import when it fails to
parse an async or await keyword :-)


To me, these new features *obviously* should require
a __future__ import. Anything else would be crazy.


I accept the compromise of creating a coroutine object without wait
for it (obvious and common bug when learning asyncio). Hopefully, we
keep the coroutine wrapper feature (ok, maybe I suggested this idea to
Yury because I suffered so much when I learnt how to use asyncio ;-)),
so it will still be easy to emit a warning in debug mode.


I'm disappointed that there will *still* be no direct and
reliable way to detect and clearly report this kind of
error, and that what there is will only be active in
a special debug mode.

--
Greg
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Changing PyModuleDef.m_reload to m_slots

On 18 April 2015 at 15:58, Stefan Behnel stefan...@behnel.de wrote:
 Petr Viktorin schrieb am 17.04.2015 um 15:52:
 As a background, the PyModuleDef structure [1] is currently:

 struct PyModuleDef{
   PyModuleDef_Base m_base;
   const char* m_name;
   const char* m_doc;
   Py_ssize_t m_size;
   PyMethodDef *m_methods;
   inquiry m_reload;
   traverseproc m_traverse;
   inquiry m_clear;
   freefunc m_free;
 };

 ... where the m_reload pointer is unused, and must be NULL.
 My proposal is to repurpose this pointer to hold an array of slots, in the
 style of PEP 384's PyType_Spec [2], which would allow adding extensions --
 both those needed for PEP 489 and future ones.

 FWIW, I'm +1 on this. It replaces a struct field that risks staying unused
 basically forever with an extensible interface that massively improves the
 current extension module protocol and allows future extensions (including
 getting back the pointer we replaced).

 The alternative of essentially duplicating all sorts of things to
 accommodate for a new metadata struct is way too cumbersome and ugly in
 comparison.

Sorry for the delayed reply (post-PyCon travel), but +1 from me as
well for the reasons Stefan gives.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] typeshed for 3rd party packages

On Fri, Apr 24, 2015 at 03:44:45PM +0100, Cory Benfield wrote:
 On 24 April 2015 at 15:21, Steven D'Aprano st...@pearwood.info wrote:
 
  If the type hints are wrong, there are two errors: false positives, when
  code which should be allowed is flagged as a type error; and false
  negatives, when code which should be flagged as an error is not.
  Ideally, there should be no false positives. But false negatives are not
  so important, since you will still be doing runtime checks. All that
  means is that the static type-checker will be a little less capable of
  picking up type errors at compile time.
 
 I think that's a rational view that will not be shared as widely as I'd like.

I can't tell if you are agreeing with me, or disagreeing. The above 
sentence seems to be agreeing with me, but you later end your message 
with do it properly or not at all which disagrees. So I'm confused.


 Given that the purpose of a type checker is to catch bugs caused by
 passing incorrectly typed objects to a function, it seems entirely
 reasonable to me to raise a bug against a type hint that allows code
 that was of an incorrect type where that incorrectness *could* have
 been caught by the type hint.

Of course it is reasonable for people to submit bug reports to do with 
the type hints. And it is also reasonable for the package maintainer to 
reject the bug report as Won't Fix if it makes the type hint too 
complex.

The beauty of gradual typing is that unlike Java or Haskell, you can 
choose to have as little or as much type checking as works for you. You 
don't have to satisfy the type checker over the entire program before 
the code will run, you only need check the parts you want to check.


 Extending from that into the general
 ratio of reports that are actually bugs versus reports that are
 errors on the part of the reporter, I can assume that plenty of
 people will raise bug reports for incorrect cases as well.

Okay. Do you get many false positive bug reports for your tests too?


 From the perspective of sustainable long-term maintenance, I think the
 only way to do type hints is to have them be sufficiently exhaustive
 that a user would have to actively *try* to hit an edge case false
 negative. I believe that requests' API is too dynamically-typed to fit
 into that category at this time.

I think we agree that, static type checks or no static type checks, 
requests is going to need to do runtime type checks. So why does it 
matter if it misses a few type errors at compile time?

I think we're all in agreement that for extremely dynamic code like 
requests, you may not get as much value from static type checks as some 
other libraries or applications. You might even decide that you get no 
value at all. Okay, that's fine. I'm just suggesting that you don't have 
just two choices, all or nothing. The whole point of gradual typing is 
to give developers more options.


 PS: I should mention that, as Gary Bernhardt pointed out at PyCon,
 people often believe (incorrectly) that types are a replacement for
 tests.

They *can* be a replacement for tests. You don't see Java or Haskell
programmers writing unit tests to check that their code never tries to 
add a string to a float. Even if they could write such as test, they 
don't bother because the type checker will catch that sort of error.

The situation in Python is a bit different, and as Antoine points out, 
libraries cannot rely on their callers obeying the type restrictions of 
the public API. (Private functions are different -- if you call my 
private function with the wrong type and blow up your computer, it's 
your own fault.) For libraries, I see type checks as complementing 
tests, not replacing them.

But for application code, type checks may replace unit tests, provided 
that nobody checks in production code until both the type checker and 
the unit tests pass. If you work under that rule, there's no point in 
having the unit tests check what the type checker already tested.


 For that reason I feel like underspecified type hints are
 something of an attractive nuisance. Again, I really think this is a
 case of do it properly or not at all.

In my opinion, underspecified type hints are no more of an attractive 
nuisance than a test suite which doesn't test enough. Full coverage is 
great, but 10% coverage is better than 5% coverage, which is better than 
nothing. That applies whether we are talking about tests, type checks, 
or documentation.


-- 
Steve
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Type hints -- a mediocre programmer's reaction

2015-04-24 Thread Ronan Lamy

Le 23/04/15 14:55, Paul Sokolovsky a écrit :

Hello,

On Thu, 23 Apr 2015 09:15:44 -0400
Daniel Holth dho...@gmail.com wrote:

[]

Also ask why no one used type specifier, they are possible since
Python 3.0 ?
Because it is the wrong way for Python.

That's an example of how perceptions differ. In my list, everyone(*)
uses them - MyPy, MicroPython, etc. Even more should use them (any
JIT module, which are many), but sit in the bushes, waiting for a
kick, like PEP484 provides.

It's OK that type hints are only to assist the programmer.

Yes, it's OK to have a situation where type hints assist only a
programmer. It's not OK to think that type hints may be useful only for
programmer, instead of bunch more purposes, several of which
were already shown in the long previous discussion.

PyPy's FAQ
has an explanation of why type hints are not for performance.
http://pypy.readthedocs.org/en/latest/faq.html#would-type-annotations-help-pypy-s-performance

You probably intended to write why type hints are not for *PyPy's*
performance. There're many other language implementations and modules
for which it may be useful, please don't limit your imagination by a
single case.

Those points apply to basically any compliant implementation of Python
relying on speculative optimisation. Python is simply too dynamic for
PEP484-style hints to provide any useful performance improvement targets.

And speaking of PyPy, it really should think how to improve its
performance - not of generated programs, but of generation itself. If
compilation of a trivial program on a pumpy hardware takes 5 minutes
and gigabytes of RAM and diskspace, few people will use it for other
purposes beyond curiosity. There's something very un-Pythonic in
waiting 5 mins just to run 10-line script. Type hints can help here
too ;-) (by not wasting resources propagating types thru the same old
standard library for example).

Sorry, but that's nonsense. PyPy would be a seriously useless
interpreter if running a 10-line script required such a lengthy
compilation, so, obviously, that's not what happens.

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] PEP 492 vs. PEP 3152, new round

2015-04-24 Thread Guido van Rossum

I've tried to catch up with the previous threads. A summary of issues
brought up:

1. precise syntax of `async def` (or do we need it at all)
2. do we need `async for` and `async with` (and how to spell them)
3. syntactic priority of `await`
4. `cocall` vs. `await`
5. do we really need `__aiter__` and friends
6. StopAsyncException
7. compatibility with asyncio and existing users of it

(I've added a few myself.)

I'll try to take them one by one.


*1. precise syntax of `async def`*

Of all the places to put `async` I still like *before* the `def` the best.
I often do imprecise search for e.g. /def foo/ and would be unhappy if
this didn't find async defs. Putting it towards the end (`def foo async()`
or `def foo() async`) makes it easier to miss. A decorator makes it hard to
make the syntactic distinctions required to reject `await` outside an async
function. So I still prefer *`async def`*.

*2. do we need `async for` and `async with`*

Yes we do. Most of you are too young to remember, but once upon a time you
couldn't loop over the lines of a file with a `for` loop like you do now.
The amount of code that was devoted to efficiently iterate over files was
tremendous. We're back in that stone age with the asyncio `StreamReader`
class -- it supports `read()`, `readline()` and so on, but you can't use it
with `for`, so you have to write a `while True` loop. `asyncio for` makes
it possible to add a simple `__anext__` to the `StreamReader` class, as
follows:
```
async def __anext__(self):
line = await self.readline()
if not line:
   raise StopAsyncIteration
return line
```
A similar argument can be made for `async with`; the transaction commit is
pretty convincing, but it also helps to be able to wait e.g. for a
transport to drain upon closing a write stream. As for how to spell these,
I think having `async` at the front makes it most clear that this is a
special form.

(Though maybe we should consider `await for` and `await with`? That would
have the advantage of making it easy to scan for all suspension points by
searching for /await/. But being a verb it doesn't read very well.)

*3. syntactic priority of `await`*

Yury, could you tweak the syntax for `await` so that we can write the most
common usages without parentheses? In particular I'd like to be able to
write
```
return await foo()
with await foo() as bar: ...
foo(await bar(), await bletch())
```
(I don't care about `await foo() + await bar()` but it would be okay.)
```
I think this is reasonable with some tweaks of the grammar (similar to what
Greg did for cocall, but without requiring call syntax at the end).

*4. `cocall` vs. `await`*

Python evolves. We couldn't have PEP 380 (`yield from`) without prior
experience with using generators as coroutines (PEP 342), which in turn
required basic generators (PEP 255), and those were a natural evolution of
Python's earlier `for` loop.

We couldn't PEP 3156 (asyncio) without PEP 380 and all that came before.
The asyncio library is getting plenty of adoption and it has the concept of
separating the *getting* of a future[1] from *waiting* for it.  IIUC this
is also how `await` works in C# (it just requires something with an async
type). This has enabled a variety of operations that take futures and
produce more futures.

[1] I write `future` with a lowercase 'f' to include concepts like
coroutine generator objects.

*I just can't get used to this aspect of PEP 3152, so I'm rejecting it.*
Sorry Greg, but that's the end. We must see `await` as a refinement of
`yield from`, not as an alternative. (Yury: PEP 492 is not accepted yet,
but you're getting closer.)

One more thing: this separation is Pythonic in the sense that it's
similar to the way *getting* a callable object is a separate act from
*calling* it. While this is a cause for newbie bugs (forgetting to call an
argument-less function) it has also enabled the concept of callable as
more general and more powerful in Python: any time you need to pass a
callable, you can pass e.g. a bound method or a class or something you got
from `functools.partial`, and that's a useful thing (other languages
require you to introduce something like a lambda in such cases, which can
be painful if the thing you wrap has a complex signature -- or they don't
support function parameters at all, like Java).

I know that Greg defends it by explaining that `cocal f(args)` is not a
`cocall` operator applied to `f(args)`, it is the *single* operator `cocall
...(args)` applied to `f`. But this is too subtle, and it just doesn't jive
with the long tradition of using `yield from f` where f is some previously
obtained future.

*5. do we really need `__aiter__` and friends*

There's a lot of added complexity, but I think it's worth it. I don't think
we need to make the names longer, the 'a' prefix is fine for these methods.
I think it's all in the protocols: regular `with` uses `__enter__` and
`__exit__`; `async with` uses `__aenter__` and `__aexit__` (which must
return

Re: [Python-Dev] PEP 492 vs. PEP 3152, new round


Guido,

On 2015-04-24 1:03 PM, Guido van Rossum wrote:


*3. syntactic priority of `await`*

Yury, could you tweak the syntax for `await` so that we can write the most
common usages without parentheses? In particular I'd like to be able to
write
```
return await foo()
with await foo() as bar: ...
foo(await bar(), await bletch())
```
(I don't care about `await foo() + await bar()` but it would be okay.)
```
I think this is reasonable with some tweaks of the grammar (similar to what
Greg did for cocall, but without requiring call syntax at the end).

I don't remember the reason why yield requires parentheses
in expressions, hopefully it's not something fundamental.
This has always annoyed me, so let's try to fix that for
await.  I'll experiment.



Ditto for `__aiter__` and `__anext__`. I guess this means that the async
equivalent to obtaining an iterator through `it = iter(xs)` followed by
`for x over it` will have to look like `ait = await aiter(xs)` followed by
`for x over ait`, where an iterator is required to have an `__aiter__`
method that's an async function and returns self immediately. But what if
you left out the `await` from the first call? I.e. can this work?
```
ait = aiter(xs)
async for x in ait:
 print(x)


With the current semantics that PEP 492 proposes, await
for aiter() is mandatory.

You have to write

ait = await aiter(xs)
async for x in ait:
print(c)

We can add some logic that will check that the iterator passed
to 'async for' is not an unresolved awaitable and resolve it
(instead of immediately checking if it has __anext__ method),
but that will complicate the implementation.  It will also
introduce more than one way of doing things.  I think that
users will recognize async builtins (when we add them) by
the first letter a and use them in await expressions
consistently.



```
The question here is whether the object returned by aiter(xs) has an
`__aiter__` method. Since it was intended to be the target of  `await`, it
has an `__await__` method. But that itself is mostly an alias for
`__iter__`, not `__aiter__`. I guess it can be made to work, the object
just has to implement a bunch of different protocols.

Correct.  And yes, we address this all by having iteration
protocols clearly separated.



*6. StopAsyncException*

I'm not sure about this. The motivation given in the PEP seems to focus on
the need for `__anext__` to be async. But is this really the right pattern?
What if we required `ait.__anext__()` to return a future, which can either
raise good old `StopIteration` or return the next value from the iteration
when awaited? I'm wondering if there are a few alternatives to be explored
around the async iterator protocol still.

__anext__ should return an awaitable (following the terminology
of the PEP), which can be a coroutine-object.  I'm not sure that
with semantics of PEP 479 it can actually raise StopIteration
(without some hacks in genobject).

I'm also trying to think forward about how we can add
generator-coroutines (the ones that combine 'await' and some
form of 'yield') to make writing asynchronous iterators
easier. I think that reusing StopIteration on that level
will be a very hard thing to understand and implement.

I'll experiment with reference implementation and update
the PEP.


Thank you,
Yury
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 492 vs. PEP 3152, new round

2015-04-24 Thread Ethan Furman

On 04/24, Yury Selivanov wrote:
 On 2015-04-24 1:03 PM, Guido van Rossum wrote:

 Ditto for `__aiter__` and `__anext__`. I guess this means that the async
 equivalent to obtaining an iterator through `it = iter(xs)` followed by
 `for x over it` will have to look like `ait = await aiter(xs)` followed by
 `for x over ait`, where an iterator is required to have an `__aiter__`
 method that's an async function and returns self immediately. But what if
 you left out the `await` from the first call? I.e. can this work?
 ```
 ait = aiter(xs)
 async for x in ait:
  print(x)
 
 With the current semantics that PEP 492 proposes, await
 for aiter() is mandatory.
 
 You have to write
 
 ait = await aiter(xs)
 async for x in ait:
 print(c)

As a new user to asyncio and this type of programming in general, 'await aiter'
feels terribly redundant.

--
~Ethan~
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] typeshed for 3rd party packages

2015-04-24 Thread Cory Benfield

On 24 April 2015 at 15:21, Steven D'Aprano st...@pearwood.info wrote:

 If the type hints are wrong, there are two errors: false positives, when
 code which should be allowed is flagged as a type error; and false
 negatives, when code which should be flagged as an error is not.
 Ideally, there should be no false positives. But false negatives are not
 so important, since you will still be doing runtime checks. All that
 means is that the static type-checker will be a little less capable of
 picking up type errors at compile time.

I think that's a rational view that will not be shared as widely as I'd like.

Given that the purpose of a type checker is to catch bugs caused by
passing incorrectly typed objects to a function, it seems entirely
reasonable to me to raise a bug against a type hint that allows code
that was of an incorrect type where that incorrectness *could* have
been caught by the type hint. Extending from that into the general
ratio of reports that are actually bugs versus reports that are
errors on the part of the reporter, I can assume that plenty of
people will raise bug reports for incorrect cases as well.

From the perspective of sustainable long-term maintenance, I think the
only way to do type hints is to have them be sufficiently exhaustive
that a user would have to actively *try* to hit an edge case false
negative. I believe that requests' API is too dynamically-typed to fit
into that category at this time.

PS: I should mention that, as Gary Bernhardt pointed out at PyCon,
people often believe (incorrectly) that types are a replacement for
tests. For that reason I feel like underspecified type hints are
something of an attractive nuisance. Again, I really think this is a
case of do it properly or not at all.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Type hints -- a mediocre programmer's reaction

2015-04-24 Thread Paul Sokolovsky

Hello,

On Fri, 24 Apr 2015 18:27:29 +0100
Ronan Lamy ronan.l...@gmail.com wrote:

Also ask why no one used type specifier, they are possible since
Python 3.0 ?
Because it is the wrong way for Python.

That's an example of how perceptions differ. In my list,
everyone(*) uses them - MyPy, MicroPython, etc. Even more should
use them (any JIT module, which are many), but sit in the bushes,
waiting for a kick, like PEP484 provides.

It's OK that type hints are only to assist the programmer.

Yes, it's OK to have a situation where type hints assist only a
programmer. It's not OK to think that type hints may be useful only
for programmer, instead of bunch more purposes, several of which
were already shown in the long previous discussion.

PyPy's FAQ
has an explanation of why type hints are not for performance.
http://pypy.readthedocs.org/en/latest/faq.html#would-type-annotations-help-pypy-s-performance

Sorry, but that's nonsense. PyPy would be a seriously useless
interpreter if running a 10-line script required such a lengthy
compilation, so, obviously, that's not what happens.

--
Best regards,
Paul mailto:pmis...@gmail.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] async/await in Python; v2

2015-04-24 Thread Łukasz Langa


 On Apr 24, 2015, at 6:32 AM, Barry Warsaw ba...@python.org wrote:
 
 On Apr 24, 2015, at 11:17 PM, Steven D'Aprano wrote:
 
 It seems to me that tools that search for r^\s*def\s+spam\s*\( are
 
 They would likely search for something like r^\s*def\s+[a-zA-Z0-9_]+ which
 will hit def async spam but not async def”.

Realistically that can’t be what they’re doing because of multiple string 
literals, internal-scope functions, etc.
But I agree with Steven that guessing here is pointless. More importantly, 
consider:

- if we optimize for some unproven backwards compatibility with tools, we’re 
sacrificing better readability of “async def foo()”
- if that tool wants to work with Python 3.5, it’ll still have to support 
“await” so we’re going to be incompatible anyway; let alone “async for” and 
“async with”

So all in all, I don’t buy this argument.

-- 
Best regards,
Łukasz Langa

WWW: http://lukasz.langa.pl/
Twitter: @llanga
IRC: ambv on #python-dev

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] async/await in Python; v2

2015-04-24 Thread Paul Sokolovsky

Hello,

On Fri, 24 Apr 2015 12:04:27 -0700
Łukasz Langa luk...@langa.pl wrote:

[]
  
  They would likely search for something like
  r^\s*def\s+[a-zA-Z0-9_]+ which will hit def async spam but not
  async def”.
 
 Realistically that can’t be what they’re doing because of multiple
 string literals, internal-scope functions, etc. But I agree with
 Steven that guessing here is pointless. More importantly, consider:
 
 - if we optimize for some unproven backwards compatibility with
 tools, we’re sacrificing better readability of “async def foo()”

Yes, so hopefully another argument prevails: the sooner they break, the
sooner they're fixed (no irony here, I really consider it strange to
optimize language syntax based on background auxiliary utilities'
features or misfeatures).

[]


-- 
Best regards,
 Paul  mailto:pmis...@gmail.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 492 vs. PEP 3152, new round

2015-04-24 Thread Łukasz Langa


 On Apr 24, 2015, at 10:03 AM, Guido van Rossum gu...@python.org wrote:
 
 1. precise syntax of `async def`
 
 So I still prefer `async def`.

Me too. Also, we would use a similar vocabulary to existing users of the 
feature. This is exactly how Hack does it: 
http://docs.hhvm.com/manual/en/hack.async.php 
http://docs.hhvm.com/manual/en/hack.async.php, how ECMAScript 7 proposes it: 
http://wiki.ecmascript.org/doku.php?id=strawman:async_functions 
http://wiki.ecmascript.org/doku.php?id=strawman:async_functions and similarly 
to how C# does it (async comes after the public/private modifier but before the 
return type): https://msdn.microsoft.com/en-us/library/hh156513.aspx 
https://msdn.microsoft.com/en-us/library/hh156513.aspx


 2. do we need `async for` and `async with`
 
 Yes we do.

+1


 (Though maybe we should consider `await for` and `await with`? That would 
 have the advantage of making it easy to scan for all suspension points by 
 searching for /await/. But being a verb it doesn't read very well.)

I’m on the fence here.

OT1H, I think “await for something in a_container” and “await with 
a_context_manager():” also read pretty well. It’s also more consistent with 
being the one way of finding “yield points”.

OTOH, “await with a_context_manager():” suggests the entire statement is 
awaited on, which is not true. “async with” is more opaque in this way, it 
simply states “there are going to be implementation-specific awaits inside”. So 
it’s more consistent with “async def foo()”.

All in all I think I’m leaning towards “async for” and “async with”. More 
importantly though, I’m wondering how obvious will the failure mode be when 
somebody uses a bare “for” instead of an “async for”. Ditto for “with” vs. 
“async with”. How much debugging will be necessary to find that it’s only a 
missing “async” before the loop?


Side note: to add to the confusion about syntax, Hack’s equivalent for “async 
for” - which doesn’t really translate well to Python - uses “await” and ties it 
to the iterable:

foreach ($list await as $input) { … }

The equivalent in Python would be:

for input in list await:

Notably, this is ugly and would be confused with `for input in await list:` 
which means something different.

Also, this particular construct represents less than 0.01% of all “await” 
occurences in Facebook code, suggesting it’s not performance critical.


 3. syntactic priority of `await`
 
 Yury, could you tweak the syntax for `await` so that we can write the most 
 common usages without parentheses?

+1

Yury points out there was likely a reason this wasn’t the case for `yield` in 
the first place. It would be good to revisit that. Maybe for yield itself, too?


 4. `cocall` vs. `await`
 
 I just can't get used to this aspect of PEP 3152, so I'm rejecting it.

+1


 (Yury: PEP 492 is not accepted yet, but you're getting closer.)

May I suggest using the bat-signal to summon Glyph to confirm this is going to 
be helpful/usable with Twisted as well?


 5. do we really need `__aiter__` and friends
 
 There's a lot of added complexity, but I think it's worth it. I don't think 
 we need to make the names longer, the 'a' prefix is fine for these methods.

+1


 6. StopAsyncException
 
 I'm not sure about this. The motivation given in the PEP seems to focus on 
 the need for `__anext__` to be async. But is this really the right pattern? 
 What if we required `ait.__anext__()` to return a future, which can either 
 raise good old `StopIteration` or return the next value from the iteration 
 when awaited? I'm wondering if there are a few alternatives to be explored 
 around the async iterator protocol still.

So are you suggesting to pass the returned value in a future? In this case the 
future would need to be passed to __anext__, so the Cursor example from the PEP 
would look like this:

class Cursor:
def __init__(self):
self.buffer = collections.deque()

def _prefetch(self):
...

async def __aiter__(self):
return self

async def __anext__(self, fut):
if not self.buffer:
self.buffer = await self._prefetch()
if self.buffer:
fut.set_result(self.buffer.popleft())
else:
fut.set_exception(StopIteration)

While this is elegant, my concern is that one-future-per-iteration-step might 
be bad for performance.

Maybe consider the following. The `async def` syntax decouples the concept of a 
coroutine from the implementation. While it’s still based on generators under 
the hood, the user no longer considers his “async function” to be a generator 
or conforming to the generator protocol. From the user’s perpective, it’s 
obvious that the return below means something different than the exception:

async def __anext__(self):
if not self.buffer:
self.buffer = await self._prefetch()
if not self.buffer:
raise StopIteration
return self.buffer.popleft()

So, the same way we

Re: [Python-Dev] PEP 492 vs. PEP 3152, new round


Victor,

On 2015-04-24 5:32 PM, Victor Stinner wrote:

7. compatibility with asyncio and existing users of it

The current state of the PEP makes types.coroutine() mandatory. If a
generator-based coroutine is not modified with types.coroutine, await
cannot be used on it. To be more concrete: asyncio coroutines not
declared with @asyncio.coroutine cannot be used with await.

Would it be crazy to allow waiting on a generator-based coroutine
(current asyncio coroutines) without having to call types.coroutine()
on it?


I'd be big -1 on that.  The current PEP design is all about
strictly prohibiting users from calling regular generators
with 'await' expression.  And if a generator isn't decorated
with @coroutine - then it's a regular generator for us.



Maybe I just missed the purpose of disallow this.

It's also possible to modify asyncio to detect at runtime when an
asyncio coroutine is not decorated by @asyncio.coroutine (emit a
warning or even raise an exception).


I'd be +1 to add a warning to Task and other places where
we accept generator-based coroutines.

Thanks!
Yury

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 492 vs. PEP 3152, new round

2015-04-24 Thread Victor Stinner

Hi,

2015-04-24 19:03 GMT+02:00 Guido van Rossum gu...@python.org:
 1. precise syntax of `async def`

 Of all the places to put `async` I still like *before* the `def` the best.

So do I.


 2. do we need `async for` and `async with`

 Yes we do.

I agree.


 3. syntactic priority of `await`

 Yury, could you tweak the syntax for `await` so that we can write the most
 common usages without parentheses?

IMO another point must be discussed, corner cases in the grammar and
parser of the current PEP  implementation:
https://www.python.org/dev/peps/pep-0492/#transition-period-shortcomings

And the fact the Python 3.7 will make async  await keywords without
proving a way to prepare the code for this major change in the Python
syntax.

According to the PEP, patching the parser to detect async  await as
keywords is almost a hack, and there are corner cases where it doesn't
work as expected.

That's why I suggest to reconsider the idea of supporting an
*optional* from __future__ import async to get async and await as
keywords in the current file. This import would allow all crazy
syntax. The parser might suggest to use the import when it fails to
parse an async or await keyword :-)


If you don't like the compromise of a parser with corner cases and an
optional __future__ to workaround these cases, I'm also ok to
reproduce what we did with the introduction of the with keyword. I
mean not supporting async nor await by default, and maing __future__
mandatory to get the new feature.

= 0 risk of backward compatibility issue
= no more crazy hacks in the parser


 4. `cocall` vs. `await`

 The asyncio library is getting plenty of adoption and it has the concept of
 separating the *getting* of a future[1] from *waiting* for it.

My rationale in my other email was similar (ability to get a function
without calling it, as you wrote, like bounded methods), so obviously
I agree with it :-)

I accept the compromise of creating a coroutine object without wait
for it (obvious and common bug when learning asyncio). Hopefully, we
keep the coroutine wrapper feature (ok, maybe I suggested this idea to
Yury because I suffered so much when I learnt how to use asyncio ;-)),
so it will still be easy to emit a warning in debug mode.

On Stackoverflow, when someone posts a code with bug, I'm now replying
please rerun your code with asyncio debug mode enabled ;-) I also
mentionned it at the *beginning* of the asyncio doc (it's documented
at the end of the asyncio doc!).



 5. do we really need `__aiter__` and friends

 There's a lot of added complexity, but I think it's worth it.

I agree. I don't see how to keep the PEP consistent without having new
dedicated protocols.


 6. StopAsyncException

(Sorry, I have no opinion on this point.)


 7. compatibility with asyncio and existing users of it

The current state of the PEP makes types.coroutine() mandatory. If a
generator-based coroutine is not modified with types.coroutine, await
cannot be used on it. To be more concrete: asyncio coroutines not
declared with @asyncio.coroutine cannot be used with await.

Would it be crazy to allow waiting on a generator-based coroutine
(current asyncio coroutines) without having to call types.coroutine()
on it?

Maybe I just missed the purpose of disallow this.

It's also possible to modify asyncio to detect at runtime when an
asyncio coroutine is not decorated by @asyncio.coroutine (emit a
warning or even raise an exception).

Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 492 vs. PEP 3152, new round