Re: [Python-Dev] A more flexible task creation

[Michel Desmoulin]

> To be honest, I see "async with" being abused everywhere in asyncio,
> lately.  I like to have objects with start() and stop() methods, but
> everywhere I see async context managers.>
> Fine, add nursery or whatever, but please also have a simple start() /
> stop() public API.
> 
> "async with" is only good for functional programming.  If you want to go
> more of an object-oriented style, you tend to have start() and stop()
> methods in your classes, which will call start() & stop() (or close())
> methods recursively on nested resources.  So of the libraries (aiopg,
> I'm looking at you) don't support start/stop or open/close well.

Wouldn't calling __enter__ and __exit__ manually works for you ? I
started coding begin() and stop(), but I removed them, as I couldn't
find a use case for them.

And what exactly is the use case that doesn't work with `async with` ?
The whole point is to spot the boundaries of the tasks execution easily.
If you start()/stop() randomly, it kinda defeat the purpose.

It's a genuine question though. I can totally accept I overlooked a
valid use case.


> 
> I tend to slightly agree, but OTOH if asyncio had been designed to not
> schedule tasks automatically on __init__ I bet there would have been
> other users complaining that "why didn't task XX run?", or "why do tasks
> need a start() method, that is clunky!".  You can't please everyone...

Well, ensure_future([schedule_immediatly=True]) and
asyncio.create_task([schedule_immediatly=True] would take care of that.
They are the entry point for the task creation and scheduling.

> 
> Also, in
>              task_list = run.all(foo(), foo(), foo())
> 
> As soon as you call foo(), you are instantiating a coroutine, which
> consumes memory, while the task may not even be scheduled for a long
> time (if you have 5000 potential tasks but only execute 10 at a time,
> for example).

Yes but this has the benefit of accepting any awaitable, not just
coroutine. You don't have to wonder what to pass, or which form. It's
always the same. Too many APi are hard to understand because you never
know if it accept a callback, a coroutine function, a coroutine, a task,
a future...

For the same reason, request.get() create and destroys a session every
time. It's inefficient, but way easier to understand, and fits the
majority of the use cases.

> 
> But if you do as Yuri suggested, you'll instead accept a function
> reference, foo, which is a singleton, you can have many foo references
> to the function, but they will only create coroutine objects when the
> task is actually about to be scheduled, so it's more efficient in terms
> of memory.

I made some test, and the memory consumption is indeed radically smaller
if you just store references if you just compare it to the same unique
raw coroutine.

However, this is a rare case. It assumes that:

- you have a lot of tasks
- you have a max concurrency
- the max concurrency is very small
- most tasks reuse a similar combination of callables and parameters

It's a very specific narrow case. Also, everything you store on the
scope will be wrapped into a Future object no matter if it's scheduled
or not, so that you can cancel it later. So the scale of the memory
consumption is not as much.

I didn't want to compromise the quality of the current API for the
general case for an edge case optimization.

On the other hand, this is a low hanging fruit and on plateforms such as
raspi where asyncio has a lot to offer, it can make a big difference to
shave up 20 of memory consumption of a specific workload.

So I listened and implemented an escape hatch:

import random
import asyncio

import ayo

async def zzz(seconds):
await asyncio.sleep(seconds)
print(f'Slept for {seconds} seconds')


@ayo.run_as_main()
async def main(run_in_top):

async with ayo.scope(max_concurrency=10) as run:
for _ in range(1):
run.from_callable(zzz, 0.005) # or run.asap(zzz(0.005))

This would only lazily create the awaitable (here the coroutine) on
scheduling. I see a 15% of memory saving for the WHOLE program if using
`from_callable()`.

So definitly a good feature to have, thank you.

But again, and I hope Yuri is reading this because he will implement
that for uvloop, and this will trickles down to asyncio, I think we
should not compromise the main API for this.

asyncio is hard enough to grok, and too many concepts fly around. The
average Python programmer has been experienced way easier things from
past Python encounter.

If we want, one day, that asyncio is consider the clean AND easy way to
do async, we need to work on the API.

asyncio.run() is a step in the right direction (although again I wish we
implemented that 2 years ago when I talked about it instead of telling
me no).

Now if we add nurseries, it should hide the rest of the complexity. Not
add to it.

___
Python-Dev mailing list
Python-Dev@python.org

Re: [Python-Dev] A more flexible task creation

[Michel Desmoulin]

Le 14/06/2018 à 04:09, Nathaniel Smith a écrit :
> How about:
> 
> async def wait_to_run(async_fn, *args):
>     await wait_for_something()
>     return await async_fn(*args)
> 
> task = loop.create_task(wait_to_run(myfunc, ...))
> 

It's quite elegant, although figuring out the wait_for_something() is
going to be tricky.


> -
> 
> Whatever strategy you use, you should also think about what semantics
> you want if one of these delayed tasks is cancelled before it starts.
> 
> For regular, non-delayed tasks, Trio makes sure that even if it gets
> cancelled before it starts, then it still gets scheduled and runs until
> the first cancellation point. This is necessary for correct resource
> hand-off between tasks:
> 
> async def some_task(handle):
>     with handle:
>         await ...
> 
> If we skipped running this task entirely, then the handle wouldn't be
> closed properly; scheduling it once allows the with block to run, and
> then get cleaned up by the cancellation exception. I'm not sure but I
> think asyncio handles pre-cancellation in a similar way. (Yury, do you
> know?
> 
> Now, in delayed task case, there's a similar issue. If you want to keep
> the same solution, then you might want to instead write:
> 
> # asyncio
> async def wait_to_run(async_fn, *args):
>     try:
>         await wait_for_something()
>     except asyncio.CancelledError:
>         # have to create a subtask to make it cancellable
>         subtask = loop.create_task(async_fn(*args))
>         # then cancel it immediately
>         subtask.cancel()
>         # and wait for the cancellation to be processed
>         return await subtask
>     else:
>         return await async_fn(*args)
> 
> In trio, this could be simplified to
> 
> # trio
> async def wait_to_run(async_fn, *args):
>     try:
>         await wait_for_something()
>     except trio.Cancelled:
>         pass
>     return await async_fn(*args)
> 
> (This works because of trio's "stateful cancellation" – if the whole
> thing is cancelled, then as soon as async_fn hits a cancellation point
> the exception will be re-delivered.)

Thanks for the tip. It schedules it in all cases, but I don't know what
asyncio does with it. I'll add a unit test for that.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] A more flexible task creation

[Michel Desmoulin]

> 
> The strict API compatibility requirements of core Python stdlib, coupled
> with the very long feature release life-cycles of Python, make me think
> this sort of thing perhaps is better built in an utility library on top
> of asyncio, rather than inside asyncio itself?  18 months is a long long
> time to iterate on these features.  I can't wait for Python 3.8...
>  

A lot of my late requests come from my attempt to group some of that in
a lib: https://github.com/Tygs/ayo

Most of it works, although I got read of context() recently, but the
lazy task part really fails.


Indeed, the API allows to do:

async with ayo.scope() as run:
task_list = run.all(foo(), foo(), foo())
run.asap(bar())
await task_list.gather()
run.asap(baz())



scope() return a nursery like object, and this works perfectly, with the
usual guaranty of Trio's nursery, but working in asyncio right now.

However, I tried to add to the mix:

async with ayo.scope(max_concurrency=2) as run:
task_list = run.all(foo(), foo(), foo())
run.asap(bar())
await task_list.gather()
run.asap(baz())

And I can get it to work. task_list will right now contains a list of
tasks and None, because some tasks are not scheduled immediately. That's
why I wanted lazytasks. I tried to create my own lazy tasks, but it
never really worked. I'm going to try to go down the road of wrapping
the unscheduled coro in a future-like object as suggested by Yuri. But
having that built-in seems logical, elegant, and just good design in
general: __init__ should not have side effects.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] A more flexible task creation

[Michel Desmoulin]

I was working on a concurrency limiting code for asyncio, so the user
may submit as many tasks as one wants, but only a max number of tasks
will be submitted to the event loop at the same time.

However, I wanted that passing an awaitable would always return a task,
no matter if the task was currently scheduled or not. The goal is that
you could add done callbacks to it, decide to force schedule it, etc

I dug in the asyncio.Task code, and encountered:

def __init__(self, coro, *, loop=None):
...
self._loop.call_soon(self._step)
self.__class__._all_tasks.add(self)

I was surprised to see that instantiating a Task class has any side
effect at all, let alone 2, and one of them being to be immediately
scheduled for execution.

I couldn't find a clean way to do what I wanted: either you
loop.create_task() and you get a task but it runs, or you don't run
anything, but you don't get a nice task object to hold on to.

I tried several alternatives, like returning a future, and binding the
future awaiting to the submission of a task, but that was complicated
code that duplicated a lot of things.

I tried creating a custom task, but it was even harder, setting a custom
event policy, to provide a custom event loop with my own create_task()
accepting parameters. That's a lot to do just to provide a parameter to
Task, especially if you already use a custom event loop (e.g: uvloop). I
was expecting to have to create a task factory only, but task factories
can't get any additional parameters from create_task()).

Additionally I can't use ensure_future(), as it doesn't allow to pass
any parameter to the underlying Task, so if I want to accept any
awaitable in my signature, I need to provide my own custom ensure_future().

All those implementations access a lot of _private_api, and do other
shady things that linters hate; plus they are fragile at best. What's
more, Task being rewritten in C prevents things like setting self._coro,
so we can only inherit from the pure Python slow version.

In the end, I can't even await the lazy task, because it blocks the
entire program.

Hence I have 2 distinct, but independent albeit related, proposals:

- Allow Task to be created but not scheduled for execution, and add a
parameter to ensure_future() and create_task() to control this. Awaiting
such a task would just do like asyncio.sleep(O) until it is scheduled
for execution.

- Add an parameter to ensure_future() and create_task() named "kwargs"
that accept a mapping and will be passed as **kwargs to the underlying
created Task.

I insist on the fact that the 2 proposals are independent, so please
don't reject both if you don't like one or the other. Passing a
parameter to the underlying custom Task is still of value even without
the unscheduled instantiation, and vice versa.

Also, if somebody has any idea on how to make a LazyTask that we can
await on without blocking everything, I'll take it.



___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] The `for y in [x]` idiom in comprehensions

[Michel Desmoulin]

Le 25/02/2018 à 14:11, Nikolaus Rath a écrit :
> On Feb 25 2018, Chris Angelico  wrote:
>> On Sun, Feb 25, 2018 at 11:02 PM, Nikolaus Rath  wrote:
>>> On Feb 22 2018, Serhiy Storchaka  wrote:
 1. Inner generator expression:

 result = [y + g(y) for y in (f(x) for x in range(10))]

>>> [...]



 And maybe there are other ways.
>>>
>>> I think the syntax recently brough up by Nick is still the most
>>> beautiful:
>>>
>>> result = [ (f(x) as y) + g(y) for x in range(10)]

Honestly I find this version the most readable while the double for loop
is completely weird to me, despite doing python for a living for years.

I really hope the later doesn't become a common idiom.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] What's the status of PEP 505: None-aware operators?

[Michel Desmoulin]

Le 29/11/2017 à 19:02, Barry Warsaw a écrit :
> On Nov 29, 2017, at 12:40, David Mertz  wrote:
> 
>> I think some syntax could be possible to only "catch" some exceptions and 
>> let others propagate.  Maybe:
>>
>>val = name.strip()[4:].upper() except (AttributeError, KeyError): -1
>>
>> I don't really like throwing a colon in an expression though.  Perhaps some 
>> other word or symbol could work instead.  How does this read:
>>
>>val = name.strip()[4:].upper() except -1 in (AttributeError, KeyError)
> 
> I don’t know whether I like any of this  but I think a more natural 
> spelling would be:
> 
>val = name.strip()[4:].upper() except (AttributeError, KeyError) as -1
> 
> which could devolve into:
> 
>val = name.strip()[4:].upper() except KeyError as -1
> 
> or:
> 
>val = name.strip()[4:].upper() except KeyError # Implicit `as None`
> 
> I would *not* add any spelling for an explicit bare-except equivalent.  You 
> would have to write:
> 
>val = name.strip()[4:].upper() except Exception as -1
> 
> Cheers,
> -Barry
> 

I really like this one. It's way more general. I can see a use for
IndexError as well (lists don't have the dict.get() method).

Also I would prefer not to use "as" this way. In the context of an
exception, "as" already binds the exception to a variable so it's confusing.

What about:


val = name.strip()[4:].upper() except Exception: -1
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python startup optimization: script vs. service

[Michel Desmoulin]

Maybe it's time to bring back the debate on the "lazy" keyword then ?
Rendering any statement arbitrarily lazy could help with perfs. It would
also make hacks like ugettext_lazy in Django useless. And would render
moot the extensions of f-strings for lazily rendered ones. And bring
lazy imports in the mix.

Le 17/10/2017 à 19:39, Neil Schemenauer a écrit :
> Christian Heimes  wrote:
>> That approach could work, but I think that it is the wrong
>> approach. I'd rather keep Python optimized for long-running
>> processes and introduce a new mode / option to optimize for
>> short-running scripts.
> 
> Another idea is to run a fake transasaction through the process
> before forking.  That will "warm up" things so that most of the lazy
> init is already done.
> 
> After returning from the core sprint, I have gotten over my initial
> enthusiam for my "lazy module defs" idea.  It is just too big of a
> change for Python to accept that this point.  I still hope there
> would be a way to make LOAD_NAME/LOAD_GLOBAL trigger something like
> __getattr__().  That would allow libraries that want to aggressively
> do lazy-init to do so in the clean way.
> 
> The main reason that Python startup is slow is that we do far too
> much work on module import (e.g. initializing data structures that
> never get used).  Reducing that work will almost necessarily impact
> pre-fork model programs (e.g. they expect the init to be done before
> the fork).
> 
> As someone who uses that model heavily, I would still be okay with
> the "lazification" as I think there are many more programs that
> would be helped vs the ones hurt.  Initializing everything that your
> program might possibibly need right at startup time doesn't seem
> like a goal to strive for.  I can understand if you have a different
> opinion though.
> 
> A third approach would be to do more init work at compile time.
> E.g. for re.compile, if the compiled result could be stored in the
> .pyc, that would eliminate a lot of time for short scripts and for
> long-running programs.  Some Lisp systems have "compiler macros".
> They are basically a hook to allow programs to do some work before
> the code is sent to the compiler.  If something like that existed in
> Python, it could be used by re.compile to generate a compiled
> representation of the regex to store in the .pyc file.  That kind of
> behavior is pretty different than the "there is only runtime" model
> that Python generally tries to follow.
> 
> Spit-ball idea, thought up just now:
> 
> PAT = __compiled__(re.compile(...))
> 
> The expression in __compiled__(..) would be evaluated by the
> compiler and the resulting value would become the value to store in
> th .pyc.  If you are running the code as the script, __compiled__
> just returns its argument unchanged.
> 
> Cheers,
> 
>   Neil
> 
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/desmoulinmichel%40gmail.com
> 
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 557: Data Classes

[Michel Desmoulin]

Le 14/09/2017 à 19:24, Mike Miller a écrit :
> 
> On 2017-09-12 21:05, Guido van Rossum wrote:
>> It's ironic that some people dislike "data classes" because these are
>> regular classes, not just for data, while others are proposing
>> alternative names that emphasize the data container aspect. So "data
>> classes" splits the difference, by referring to both data and classes.
> 
> True that these data-classes will be a superset of a traditional
> record.  But, we already have objects and inheritance for those use
> cases.  The data-class is meant to be used primarily like a record, so
> why not name it that way?

Because given how convenient it is, it will most probably becomes the
default way to write classes in Python. Not just for record.

Everybody end up wishing for a less verbose way to write day to day
classes after a while.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 557: Data Classes

[Michel Desmoulin]

Le 10/09/2017 à 18:36, Eric V. Smith a écrit :
> On 9/10/2017 10:00 AM, Michel Desmoulin wrote:
>> The reaction is overwhelmingly positive everywhere: hacker news, reddit,
>> twitter.
> 
> Do you have a pointer to the Hacker News discussion? I missed it.

Err... I may have been over enthusiastic and created the hacker news
thread in my mind.

> 
>> People have been expecting something like that for a long time.
> 
> Me, too!
> 
>> 3 questions:
>>
>> - is providing validation/conversion hooks completely out of the
>> question of still open for debate ? I know it's to keep the
>> implementation simple but have a few callbacks run in the __init__ in a
>> foo loop is not that much complexity. You don't have to provide
>> validators, but having a validators parameters on field() would be a
>> huge time saver. Just a list of callables called when the value is first
>> set, potentially raising an exception that you don't even need to
>> process in any way. It returns the value converted, and voilà. We all do
>> that every day manually.
> 
> I don't particularly want to add validation specifically. I want to make
> it possible to add validation yourself, or via a library.
> 
> What I think I'll do is add a metadata parameter to fields(), defaulting
> to None. Then you could write a post-init hook that does whatever
> single- and multi-field validations you want (or whatever else you want
> to do). Although this plays poorly with "frozen" classes: it's always
> something! I'll think about it.
> 
> To make this most useful, I need to get the post-init hook to take an
> optional parameter so you can get data to it. I don't have a good way to
> do this, yet. Suggestions welcomed.

It doesn't really allow you to do anything you couldn't do as easily as
in __init__.

Alternatively, you could have a "on_set" hooks for field(), that just
take the field value, and return the field value. By default, it's an
identity function and is always called (minus implementation optimizations):

from functools improt reduce
self.foo = reduce((lambda data, next: next(*data)), on_set_hooks,
(field, val))

And people can do whatever they want: default values, factories,
transformers, converters/casters, validation, logging...

> 
> Although if the post-init hook takes a param that you can pass in at
> object creation time, I guess there's really no need for a per-field
> metadata parameter: you could use the field name as a key to look up
> whatever you wanted to know about the field.
> 
>> - I read Guido talking about some base class as alternative to the
>> generator version, but don't see it in the PEP. Is it still considered ?
> 
> I'm going to put some words in explaining why I don't want to use base
> classes (I don't think it buys you anything). Do you have a reason for
> preferring base classes?

Not preferring, but having it as an alternative. Mainly for 2 reasons:

1 - data classes allow one to type in classes very quickly, let's
harvest the benefit from that.

Typing a decorator in a shell is much less comfortable than using
inheritance. Same thing about IDE: all current ones have snippet with
auto-switch to the class parents on tab.

All in all, if you are doing exploratory programming, and thus
disposable code, which data classes are fantastic for, inheritance will
keep you in the flow.

2 - it will help sell the data classes

I train a lot of people to Python each year. I never have to explain
classes to people with any kind of programming background. I _always_
have to explain decorators.

People are not used to it, and even kind fear it for quite some time.

Inheritance however, is familiar, and will not only push people to use
data classes more, but also will let them do less mistakes: they know
the danger of parent ordering, but not the ones of decorators ordering.

> 
>> - any chance it becomes a built in later ? When classes have been
>> improved in Python 2, the object built-in was added. Imagine if we had
>> had to import it every time... Or maybe just plug it to object like
>> @object.dataclass.
> 
> Because of the choice of using module-level functions so as to not
> introduce conflicts in the object's namespace, it would be difficult to
> make this a builtin.
> 
> Although now that I think about it, maybe what are currently
> module-level functions should instead be methods on the "dataclass"
> decorator itself:
> 
> @dataclass
> class C:
>   i: int = dataclass.field(default=1, init=False)
>   j: str
> 
> c = C('hello')
> 
> dataclass.asdict(c)
> {'i': 1, 'j': 'hello'}
> 
> Then, "dataclass" would be the only name the module exports, making it
> easier to someday be a builtin. I'm not sure it's 

Re: [Python-Dev] PEP 557: Data Classes

[Michel Desmoulin]

The reaction is overwhelmingly positive everywhere: hacker news, reddit,
twitter.

People have been expecting something like that for a long time.

3 questions:

- is providing validation/conversion hooks completely out of the
question of still open for debate ? I know it's to keep the
implementation simple but have a few callbacks run in the __init__ in a
foo loop is not that much complexity. You don't have to provide
validators, but having a validators parameters on field() would be a
huge time saver. Just a list of callables called when the value is first
set, potentially raising an exception that you don't even need to
process in any way. It returns the value converted, and voilà. We all do
that every day manually.


- I read Guido talking about some base class as alternative to the
generator version, but don't see it in the PEP. Is it still considered ?

- any chance it becomes a built in later ? When classes have been
improved in Python 2, the object built-in was added. Imagine if we had
had to import it every time... Or maybe just plug it to object like
@object.dataclass.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 559 - built-in noop()

[Michel Desmoulin]

Don't we already have the mock module for that ? A mowk works as a noop,
will be ok with being used as a context manager and allow chaining...

Either way, what would a noop function really give you compared to
lambda *a, **b: None ?

A be bit shorter to write. Maybe faster to run. But do you use it so
much that it needs to be included ? It's not rocket science.

And as a built-in, noon of the less.

If it's coded it in C to gain perfs, then:

- let's put it in functools, not in built-in. I often wish for partial
to be built-in, but it's not. Honestly the fonctools and itertools
module should be autoimported (I always do in my PYTHONSTARTUP). But no
pony for us, let's not clutter the global name spaces.
- provide it with it's little sister, the identity function. They almost
always go hand in hand and doing the whole work of this PEP is a nice
opportunity. sorted/max/min/sort and most validation callbacks have an
identity function as default parameters. We would have then
functools.noop and functools.identity.
- provide a pure Python backport.

Aternatively, just rewrite part of the mock module in C. You'll get a
fast noop, with a lot of features, and as a bonus would speed up a lot
of unit tests around here.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 554 v2 (new "interpreters" module)

[Michel Desmoulin]

Le 09/09/2017 à 01:28, Stefan Krah a écrit :
> On Fri, Sep 08, 2017 at 04:04:27PM -0700, Eric Snow wrote:
>> * "stdlib support for subinterpreters adds extra burden
>>   on C extension authors"
>>
>> In the ``Interpreter Isolation`` section below we identify ways in
>> which isolation in CPython's subinterpreters is incomplete.  Most
>> notable is extension modules that use C globals to store internal
>> state.  PEP 3121 and PEP 489 provide a solution for most of the
>> problem, but one still remains. [petr-c-ext]_  Until that is resolved,
>> C extension authors will face extra difficulty to support
>> subinterpreters.
> 
> It's a bit of a hassle, and the enormous slowdown in some of the existing
> solutions is really a no go [1].
> 
> In the case of _decimal, the tls-context is already subinterpreter safe
> and reasonable fast due to caching.
> 
> 
> The most promising model to me is to put *all* globals in a tls structure
> and cache the whole structure.  Extrapolating from my experiences with the
> context, this might have a slowdown of "only" 4%.
> 
> 
> Still, the argument "who uses subinterpreters?" of course still remains.

For now, nobody. But if we expose it and web frameworks manage to create
workers as fast as multiprocessing and as cheap as threading, you will
find a lot of people starting to want to use it.

We can't know until we got the toy to play with.


> 
> 
> 
> Stefan Krah
> 
> 
> [1] I'm referring to the slowdown of heaptypes + module state.
> 
> 
> 
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/desmoulinmichel%40gmail.com
> 
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] for...else

[Michel Desmoulin]

elif break and elif None: I'd like that very much. It's weird a break
the semantic of break and None, but it's in such a dark corner of Python
anyway I don't bother.

Le 27/07/2017 à 21:19, MRAB a écrit :
> On 2017-07-27 03:34, Mike Miller wrote:
>>
>>
>> On 2017-07-26 16:36, MRAB wrote:
>>> "nobreak" would introduce a new keyword, but "not break" wouldn't.
>>
>> Whenever I've used the for-else, I've put a # no-break right next to
>> it, to
>> remind myself as much as anyone else.
>>
>> for...: not break: is the best alternative I've yet seen, congrats. 
>> Perhaps in
>> Python 5 it can be enabled, with for-else: used instead for empty
>> iterables, as
>> that's what I expected the first few dozen times.
>>
> For empty iterables, how about "elif None:"? :-)
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/desmoulinmichel%40gmail.com
> 
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python startup time

[Michel Desmoulin]

Le 23/07/2017 à 19:36, Brett Cannon a écrit :
> 
> 
> On Sun, Jul 23, 2017, 00:53 Michel Desmoulin, <desmoulinmic...@gmail.com
> <mailto:desmoulinmic...@gmail.com>> wrote:
> 
> 
> 
> > Optimizing startup time is incredibly valuable,
> 
> I've been reading that from the beginning of this thread but I've been
> using python since the 2.4 and I never felt the burden of the
> startup time.
> 
> I'm guessing a lot of people are like me, they just don't express them
> self because "better startup time can't be bad so let's not put a
> barrier on this".
> 
> I'm not against it, but since the necessity of a faster Python in
> general has been a debate for years and is only finally catching up with
> the work of Victor Stinner, can somebody explain me the deal with start
> up time ?
> 
> I understand where it can improve your lives. I just don't get why it's
> suddenly such an explosion of expectations and needs.
> 
> 
> It's actually always been something we have tried to improve, it just
> comes in waves. For instance we occasionally re-examine what modules get
> pulled in during startup. Importlib was optimized to help with startup.
> This just happens to be the latest round of trying to improve the situation.
> 
> As for why we care, every command-line app wants to at least appear
> faster if not be faster because just getting to the point of being able
> to e.g. print a version number is dominated by Python and app start-up.


Fair enought.

> And this is not guessing; I work with a team that puts out a command
> line app and one of the biggest complaints they get is the startup time.

This I don't get. When I run any command line utility in python (grin,
ffind, pyped, django-admin.py...), the execute in a split second.

I can't even SEE the different between:

python3 -c "import os; [print(x) for x in os.listdir('.')]"

and

ls .

I'm having a hard time understanding how the Python VM startup time can
be perceived as a barriere here. I can understand if you have an
application firing Python 1000 times a second, like a CGI service or
some kind of code exec service. But scripting ?

Now I can imagine that a given Python program can be slow to start up,
because it imports a lot of things. But not the VM itself.


> 
> -brett
> 
> ___
> Python-Dev mailing list
> Python-Dev@python.org <mailto:Python-Dev@python.org>
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/brett%40python.org
> 
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python startup time

[Michel Desmoulin]

> Optimizing startup time is incredibly valuable, 

I've been reading that from the beginning of this thread but I've been
using python since the 2.4 and I never felt the burden of the startup time.

I'm guessing a lot of people are like me, they just don't express them
self because "better startup time can't be bad so let's not put a
barrier on this".

I'm not against it, but since the necessity of a faster Python in
general has been a debate for years and is only finally catching up with
the work of Victor Stinner, can somebody explain me the deal with start
up time ?

I understand where it can improve your lives. I just don't get why it's
suddenly such an explosion of expectations and needs.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] API design: where to add async variants of existing stdlib APIs?

[Michel Desmoulin]

Last week I had to download a CSV from an FTP and push any update on it
using websocket so asyncio was a natural fit and the network part went well.

The surprise was that the CSV part would not work as expected. Usually I
read csv doing:

import csv

file_like_object = csv_crawler.get_file()
for row in csv.DictReader(file_like_object)

But it didn't work because file_like_object.read() was a coroutine which
the csv module doesn't handle.

So I had to do:

import csv
import io

raw_bytes = await stream.read(1000)
wrapped_bytes = io.BytesIO(raw_bytes)
text = io.TextIOWrapper(wrapped_bytes, encoding=encoding,
errors='replace')

for i, row in enumerate(csv.DictReader(text)):

Turns out I used asyncio a bit, and I now the stdlib, the io AIP, etc.
But for somebody that doesn't, it's not very easy to figure out. Plus
it's not as elegant as traditional Python. Not to mention it loads the
entire CSV in memory.

So I wondered if I could fix the csv module so it accept async. But the
question arised. Where should I put it ?

- Create AsyncDictReader and AsyncReader ?
- Add inspect.iscoroutine calls widh it in the regular Readers and some
__aiter__ and __aenter__ ?
- add a csv.async namespace ?

What API design are we recommanding for expose both sync and async
behaviors ?


Le 07/03/2017 à 03:08, Guido van Rossum a écrit :
> On Mon, Mar 6, 2017 at 5:57 PM, Raymond Hettinger
> > wrote:
> 
> Of course, it makes sense that anything not specific to asyncio
> should go outside of asyncio.
> 
> What I'm more concerned about is what the other places actually
> are.   Rather than putting async variants of everything sprinkled
> all over the standard library, I suggest collecting them all
> together, perhaps in a new asynctools module.
> 
> 
> That's a tough design choice. I think neither extreme is particularly
> attractive -- having everything in an asynctools package might also
> bundle together thing that are entirely unrelated. In the extreme it
> would be like proposing that all metaclasses should go in a new
> "metaclasstools" package. I think we did a reasonable job with ABCs:
> core support goes in abc.py, support for collections ABCs goes into the
> collections package (in a submodule), and other packages and modules
> sometimes define ABCs for their own users.
> 
> Also, in some cases I expect we'll have to create a whole new module
> instead of updating some ancient piece of code with newfangled async
> variants to its outdated APIs.
> 
> -- 
> --Guido van Rossum (python.org/~guido )
> 
> 
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/desmoulinmichel%40gmail.com
> 
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] we would like to share python articles with you

[Michel Desmoulin]

This mailling list is for coordinating the development of the Python
programming language, not to be used for marketing.

Share your articles on a social network or a forum such as
reddit.com/r/python.

Le 06/03/2017 à 16:53, Yaroslav Lehenchuk a écrit :
> Hi! 
> 
> I like your resource. We in Django Stars writing a lot about
> Python/Django and we would like to share our articles with other
> professionals and geeks.
> Here are two examples of our blog content: 
> http://djangostars.com/blog/continuous-integration-circleci-vs-travisci-vs-jenkins/
> 
>  
> http://djangostars.com/blog/how-to-create-and-deploy-a-telegram-bot/
> 
> 
> And we also have an account on git hub where we sharing our libraries
> and open source projects.
> 
> Tell me please, are you interested in such cooperation? I am talking
> about submitting our post to your tips-digest.
> 
> Waiting for your response. Thank you in advance.
> 
> -- 
> Best Regards,
> Yaroslav Lehenchuk
> Marketer at Django Stars
> 
> Cell: +380730903748
> Skype: yaroslav_le
> Email: yaroslav.lehenc...@djangostars.com
> 
> 
> 
> 
> 
>  
> 
> 
> 
> 
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/desmoulinmichel%40gmail.com
> 
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] When should pathlib stop being provisional?

[Michel Desmoulin]

Fair enough, I stand corrected for both points.

Le 07/04/2016 18:13, Zachary Ware a écrit :
> On Thu, Apr 7, 2016 at 5:50 AM, Michel Desmoulin
> <desmoulinmic...@gmail.com> wrote:
>> Path objects don't have splitext() or and don't allow  "string" / path.
>> Those are the ones bugging me the most.
> 
>>>> import pathlib
>>>> p = '/some/test' / pathlib.Path('path') / 'file_with.ext'
>>>> p
> PosixPath('/some/test/path/file_with.ext')
>>>> p.parent, p.stem, p.suffix
> (PosixPath('/some/test/path'), 'file_with', '.ext')
> 
> 
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] When should pathlib stop being provisional?

[Michel Desmoulin]

Le 06/04/2016 22:47, Sven R. Kunze a écrit :
> On 06.04.2016 07:00, Guido van Rossum wrote:
>> On Tue, Apr 5, 2016 at 9:29 PM, Ethan Furman  wrote:
>>> [...] we can't do:
>>>
>>>  app_root = Path(...)
>>>  config = app_root/'settings.cfg'
>>>  with open(config) as blah:
>>>  # whatever
>>>
>>> It feels like instead of addressing this basic disconnect, the answer
>>> has
>>> instead been:  add that to pathlib!  Which works great -- until a
>>> user or a
>>> library gets this path object and tries to use something from os on it.
>> I agree that asking for config.open() isn't the right answer here
>> (even if it happens to work).
> 
> How come?
> 
>> But in this example, once 3.5.2 is out,
>> the solution would be to use open(config.path), and that will also
>> work when passing it to a library. Is it still unacceptable then?
> 
> I think so. Although in this example I would prefer the shorter
> config.open alternative as I am lazy.
> 
> 
> I still cannot remember what the concrete issue was why we dropped
> pathlib the same day we gave it a try. It was something really stupid
> and although I hoped to reduce the size of the code, it was less
> readable. But it was not the path->str issue but something more mundane.
> It was something that forced us to use os[.path] as Path didn't provide
> something equivalent. Cannot remember.

Path objects don't have splitext() or and don't allow  "string" / path.
Those are the ones bugging me the most.

> 
> 
> Best,
> Sven
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/desmoulinmichel%40gmail.com
> 
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Defining a path protocol

[Michel Desmoulin]

Wouldn't be better to generalize that to a "__location__" protocol,
which allow to return any kind of location, including path, url or
coordinate, ip_address, etc ?

Le 06/04/2016 19:26, Brett Cannon a écrit :
> WIth Ethan volunteering to do the work to help make a path protocol a
> thing -- and I'm willing to help along with propagating this through the
> stdlib where I think Serhiy might be interested in helping as well --
> and a seeming consensus this is a good idea, it seems like this proposal
> has a chance of actually coming to fruition.
> 
> Now we need clear details. :) Some open questions are:
> 
>  1. Name: __path__, __fspath__, or something else?
>  2. Method or attribute? (changes what kind of one-liner you might use
> in libraries, but I think historically all protocols have been
> methods and the serialized string representation might be costly to
> build)
>  3. Built-in? (name is dependent on #1 if we add one)
>  4. Add the method/attribute to str? (I assume so, much like __index__()
> is on int, but I have not seen it explicitly stated so I would
> rather clarify it)
>  5. Expand the C API to have something like PyObject_Path()?
> 
> 
> Some people have asked for the pathlib PEP to have a more flushed out
> reasoning as to why pathlib doesn't inherit from str. If Antoine doesn't
> want to do it I can try to instil my blog post into a more succinct
> paragraph or two and update the PEP myself.
> 
> Is this going to require a PEP or if we can agree on the points here are
> we just going to do it? If we think it requires a PEP I'm willing to
> write it, but I obviously have no issue if we skip that step either. :)
> 
> Oh, and we should resolve this before the next release of Python 3.4,
> 3.5, or 3.6 so that pathlib can be updated in those releases.
> 
> -Brett
> 
> 
> On Wed, 6 Apr 2016 at 08:09 Ethan Furman  > wrote:
> 
> On 04/05/2016 11:57 PM, Nick Coghlan wrote:
> > On 6 April 2016 at 16:53, Nathaniel Smith  > wrote:
> >> On Tue, Apr 5, 2016 at 11:29 PM, Nick Coghlan  > wrote:
> 
> >>> I'd missed the existing precedent in DirEntry.path, so simply taking
> >>> that and running with it sounds good to me.
> >>
> >> This makes me twitch slightly, because NumPy has had a whole set of
> >> problems due to the ancient and minimally-considered decision to
> >> assume a bunch of ad hoc non-namespaced method names fulfilled some
> >> protocol -- like all .sum methods will have a signature that's
> >> compatible with numpy's, and if an object has a .log method then
> >> surely that computes the logarithm (what else in computing could
> "log"
> >> possibly refer to?), etc. This experience may or may not be relevant,
> >> I'm not sure -- sometimes these kinds of twitches are good guides to
> >> intuition, and sometimes they are just knee-jerk responses to an old
> >> and irrelevant problem :-)
> >>
> >> But you might want to at least think about
> >> how common it might be to have existing objects with unrelated
> >> attributes that happen to be called "path", and the bizarro problems
> >> that might be caused if someone accidentally passes one of them to a
> >> function that expects all .path attributes to be instances of
> this new
> >> protocol.
> >
> > sys.path, for example.
> >
> > That's why I'd actually prefer the implicit conversion protocol to be
> > the more explicitly named "__fspath__", with suitable "__fspath__ =
> > path" assignments added to DirEntry and pathlib. However, I'm also not
> > offering to actually *do* the work here, and the casting vote goes to
> > the folks pursuing the implementation effort.
> 
> If we decide upon __fspath__ (or __path__) I will do the work on pathlib
> and scandir to add those attributes. 
> 
> 
> 
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/desmoulinmichel%40gmail.com
> 
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] why we have both re.match and re.string?

[Michel Desmoulin]

Hi,

Le 10/02/2016 22:59, Luca Sangiacomo a écrit :

Hi,
I hope the question is not too silly, but why I would like to 
understand the advantages of having both re.match() and re.search(). 
Wouldn't be more clear to have just one function with one additional 
parameters like this:


re.search(regexp, text, from_beginning=True|False) ?


Actually you can just do

re.search(^regexp, text)

But with match you express the intent to match the text with something, 
while with search, you express that you look for something in the text. 
Maybe that was the idea?




In this way we prevent, as written in the documentation, people 
writing ".*" in front of the regexp used with re.match()


Thanks.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/desmoulin.michel%40gmail.com


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Issue #26204: compiler now emits a SyntaxWarning on constant statement

[Michel Desmoulin]

Hello,

Le 08/02/2016 20:13, Guido van Rossum a écrit :

On Mon, Feb 8, 2016 at 9:44 AM, Victor Stinner  wrote:

I changed the Python compiler to ignore any kind "constant
expressions", whereas it only ignored strings and integers before:
http://bugs.python.org/issue26204

The compiler now also emits a SyntaxWarning on such case. IMHO the
warning can help to detect bugs for developers who just learnt Python.

Hum. I'm not excited by this idea. It is not bad syntax. Have you
actually seen newbies who were confused by such things?




I give regular Python trainings and I see similar errors regularly such as:

- not returning something;
- using something without putting the result back in a variable.

However, these are impossible to warn about.

What's more, I have yet to see somebody creating a constant and not 
doing anything with it. I never worked with Ruby dev though.


My sample of dev is not big enough to be significant, but I haven't met 
this issue yet. I still like the idea, anything making Python easier for 
beginers is a good thing for me.


One particular argument against it is the use of linters, but you must 
realize most beginers don't use linters. Just like they don't use 
virtualenv, pip, pdb, etc. They are part of a toolkit you learn to use 
on the way, but not something you start with. Besides, many people using 
Python are not dev, and will just never take the time to use linters, 
not learn about them.

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com