Re: [Python-ideas] PEP 554: Stdlib Module to Support Multiple Interpreters in Python Code

[Nathaniel Smith]

On Thu, Sep 7, 2017 at 6:14 PM, Matthew Rocklin  wrote:
> Those numbers were for common use in Python tools and reflected my anecdotal
> experience at the time with normal Python tools.  I'm sure that there are
> mechanisms to achieve faster speeds than what I experienced.  That being
> said, here is a small example.
>
>
> In [1]: import multiprocessing
> In [2]: data = b'0' * 1  # 100 MB
> In [3]: from toolz import identity
> In [4]: pool = multiprocessing.Pool()
> In [5]: %time _ = pool.apply_async(identity, (data,)).get()
> CPU times: user 76 ms, sys: 64 ms, total: 140 ms
> Wall time: 252 ms
>
> This is about 400MB/s for a roundtrip

Awesome, thanks for bringing numbers into my wooly-headed theorizing :-).

On my laptop I actually get a worse result from your benchmark: 531 ms
for 100 MB == ~200 MB/s round-trip, or 400 MB/s one-way. So yeah,
transferring data between processes with multiprocessing is slow.

This is odd, though, because on the same machine, using socat to send
1 GiB between processes using a unix domain socket runs at 2 GB/s:

# terminal 1
~$ rm -f /tmp/unix.sock && socat -u -b32768 UNIX-LISTEN:/tmp/unix.sock
"SYSTEM:pv -W > /dev/null"
1.00GiB 0:00:00 [1.89GiB/s] [<=>   ]

# terminal 2
~$ socat -u -b32768 "SYSTEM:dd if=/dev/zero bs=1M count=1024"
UNIX:/tmp/unix.sock
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.529814 s, 2.0 GB/s

(Notice that the pv output is in GiB/s and the dd output is in GB/s.
1.89 GiB/s = 2.03 GB/s, so they actually agree.)

On my system, Python allocates + copies memory at 2.2 GB/s, so bulk
byte-level IPC is within 10% of within-process bulk copying:

# same 100 MB bytestring as above
In [7]: bytearray_data = bytearray(data)

In [8]: %timeit bytearray_data.copy()
45.3 ms ± 540 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)

In [9]: 0.100 / 0.0453  # GB / seconds
Out[9]: 2.207505518763797

I don't know why multiprocessing is so slow -- maybe there's a good
reason, maybe not. But the reason isn't that IPC is intrinsically
slow, and subinterpreters aren't going to automatically be 5x faster
because they can use memcpy.

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PEP 554: Stdlib Module to Support Multiple Interpreters in Python Code

[Eric Snow]

On Thu, Sep 7, 2017 at 12:44 PM, Paul Moore  wrote:
> On 7 September 2017 at 20:14, Eric Snow  wrote:
>> I didn't include such a queue in this proposal because I wanted to
>> keep it as focused as possible.  I'll add a note to the PEP about
>> this.
>
> This all sounds very reasonable. Thanks for the clarification.

Hmm.  Now I'm starting to think some form of basic queue would be
important enough to include in the PEP.  I'll see if that feeling
holds tomorrow.

-eric
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PEP 554: Stdlib Module to Support Multiple Interpreters in Python Code

[Eric Snow]

First of all, thanks for the feedback and encouragement!  Responses
in-line below.

-eric


On Thu, Sep 7, 2017 at 3:48 PM, Nathaniel Smith  wrote:
> My concern about this is the same as it was last time -- the work
> looks neat, but right now, almost no-one uses subinterpreters
> (basically it's Jep and mod_wsgi and that's it?), and therefore many
> packages get away with ignoring subinterpreters.

My concern is that this is a chicken-and-egg problem.  The situation
won't improve until subinterpreters are more readily available.

> Numpy is the one I'm
> most familiar with: when we get subinterpreter bugs we close them
> wontfix, because supporting subinterpreters properly would require
> non-trivial auditing, add overhead for non-subinterpreter use cases,
> and benefit a tiny tiny fraction of our users.

The main problem of which I'm aware is C globals in libraries and
extension modules.  PEPs 489 and 3121 are meant to help but I know
that there is at least one major situation which is still a blocker
for multi-interpreter-safe module state.  Other than C globals, is
there some other issue?

> If we add a friendly python-level API like this, then we're committing
> to this being a part of Python for the long term and encouraging
> people to use it, which puts pressure on downstream packages to do
> that work... but it's still not clear whether any benefits will
> actually materialize.

I'm fine with Nick's idea about making this a "provisional" module.
Would that be enough to ease your concern here?

> I've actually argued with the PyPy devs to try to convince them to add
> subinterpreter support as part of their experiments with GIL-removal,
> because I think the semantics would genuinely be nicer to work with
> than raw threads, but they're convinced that it's impossible to make
> this work. Or more precisely, they think you could make it work in
> theory, but that it would be impossible to make it meaningfully more
> efficient than using multiple processes. I want them to be wrong, but
> I have to admit I can't see a way to make it work either...

Yikes!  Given the people involved I don't find that to be a good sign.
Nevertheless, I still consider my ultimate goals to be tractable and
will press forward.  At each step thus far, the effort has led to
improvements that extend beyond subinterpreters and multi-core.  I see
that trend continuing for the entirety of the project.  Even if my
final goal is not realized, the result will still be significantly net
positive...and I still think it will still work out. :)

> If this is being justified by the multicore use case, and specifically
> by the theory that having two interpreters in the same process will
> allow for more efficient communication than two interpreters in two
> different processes, then... why should we believe that that's
> actually possible? I want your project to succeed, but if it's going
> to fail then it seems better if it fails before we commit to exposing
> new APIs.

The project is partly about performance.  However, it's also
particularly about offering a alternative concurrency model with an
implementation that can run in multiple threads simultaneously in the
same process.

On Thu, Sep 7, 2017 at 5:15 PM, Nathaniel Smith  wrote:
> The slow case is passing
> complicated objects between processes, and it's slow because pickle
> has to walk the object graph to serialize it, and walking the object
> graph is slow. Copying object graphs between subinterpreters has the
> same problem.

The initial goal is to support passing only strings between
interpreters.  Later efforts will involve investigating approaches to
efficiently and safely passing other objects.

> So the only case I can see where I'd expect subinterpreters to make
> communication dramatically more efficient is if you have a "deeply
> immutable" type
> [snip]
> However, it seems impossible to support user-defined deeply-immutable
> types in Python:
> [snip]

I agree that it is currently not an option.  That is part of the
exercise.  There are a number of possible solutions to explore once we
get to that point.  However, this PEP isn't about that.  I'm confident
enough about the possibilities that I'm comfortable with moving
forward here.

> I guess the other case where subprocesses lose to "real" threads is
> startup time on Windows. But starting a subinterpreter is also much
> more expensive than starting a thread, once you take into account the
> cost of loading the application's modules into the new interpreter. In
> both cases you end up needing some kind of process/subinterpreter pool
> or cache to amortize that cost.

Interpreter startup costs (and optimization strategies) are another
aspect of the project which deserve attention.  However, we'll worry
about that after the core functionality has been achieved.

> Obviously I'm committing the cardinal sin of trying to guess about
> performance based on theory instead of measurement, so 

[Python-ideas] Lazy creation of module level functions and classes

[Neil Schemenauer]

This is an idea that come out of the lazy loading modules idea.
Larry Hastings mentioned what a good improvement this was for PHP.
I think it would help a lot of Python too.  Very many functions and
classes are not actually needed but are instantiated anyhow.

Back of napkin idea:

Write AST transformer tool, change top-level functions and classes
to be like properties (can use __class__ I guess)

Transform is something like:

# old code
def inc(x):
return x + 1

# transformed code
def __make_inc(code=):
obj = eval(code)
_ModuleClass.inc = obj # only do eval once
return obj
inc = property(__make_inc)

Totally seat of pants idea but I can't think of a reason why it
shouldn't work.  It seems much more powerful than lazying loading
modules.  In the lazy module case, you load the whole module if any
part is touched.  Many modules only have a small fraction of their
functions and classes actually used.

If this transformer idea works, the standard Python compiler could
be changed to do the above stuff, no transformer needed.
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PEP 554: Stdlib Module to Support Multiple Interpreters in Python Code

[Matthew Rocklin]

Those numbers were for common use in Python tools and reflected my
anecdotal experience at the time with normal Python tools.  I'm sure that
there are mechanisms to achieve faster speeds than what I experienced.
That being said, here is a small example.


In [1]: import multiprocessing
In [2]: data = b'0' * 1  # 100 MB
In [3]: from toolz import identity
In [4]: pool = multiprocessing.Pool()
In [5]: %time _ = pool.apply_async(identity, (data,)).get()
CPU times: user 76 ms, sys: 64 ms, total: 140 ms
Wall time: 252 ms

This is about 400MB/s for a roundtrip


On Thu, Sep 7, 2017 at 9:00 PM, Stephan Hoyer  wrote:

> On Thu, Sep 7, 2017 at 5:15 PM Nathaniel Smith  wrote:
>
>> On Thu, Sep 7, 2017 at 4:23 PM, Nick Coghlan  wrote:
>> > The gist of the idea is that with subinterpreters, your starting point
>> > is multiprocessing-style isolation (i.e. you have to use pickle to
>> > transfer data between subinterpreters), but you're actually running in
>> > a shared-memory threading context from the operating system's
>> > perspective, so you don't need to rely on mmap to share memory over a
>> > non-streaming interface.
>>
>> The challenge is that streaming bytes between processes is actually
>> really fast -- you don't really need mmap for that. (Maybe this was
>> important for X11 back in the 1980s, but a lot has changed since then
>> :-).) And if you want to use pickle and multiprocessing to send, say,
>> a single big numpy array between processes, that's also really fast,
>> because it's basically just a few memcpy's. The slow case is passing
>> complicated objects between processes, and it's slow because pickle
>> has to walk the object graph to serialize it, and walking the object
>> graph is slow. Copying object graphs between subinterpreters has the
>> same problem.
>>
>
> This doesn't match up with my (somewhat limited) experience. For example,
> in this table of bandwidth estimates from Matthew Rocklin (CCed), IPC is
> about 10x slower than a memory copy:
> http://matthewrocklin.com/blog/work/2015/12/29/data-bandwidth
>
> This makes a considerable difference when building a system do to parallel
> data analytics in Python (e.g., on NumPy arrays), which is exactly what
> Matthew has been working on for the past few years.
>
> I'm sure there are other ways to avoid this expensive IPC without using
> sub-interpreters, e.g., by using a tool like Plasma (
> http://arrow.apache.org/blog/2017/08/08/plasma-in-memory-object-store/).
> But I'm skeptical of your assessment that the current multiprocessing
> approach is fast enough.
>
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PEP 554: Stdlib Module to Support Multiple Interpreters in Python Code

[Stephan Hoyer]

On Thu, Sep 7, 2017 at 5:15 PM Nathaniel Smith  wrote:

> On Thu, Sep 7, 2017 at 4:23 PM, Nick Coghlan  wrote:
> > The gist of the idea is that with subinterpreters, your starting point
> > is multiprocessing-style isolation (i.e. you have to use pickle to
> > transfer data between subinterpreters), but you're actually running in
> > a shared-memory threading context from the operating system's
> > perspective, so you don't need to rely on mmap to share memory over a
> > non-streaming interface.
>
> The challenge is that streaming bytes between processes is actually
> really fast -- you don't really need mmap for that. (Maybe this was
> important for X11 back in the 1980s, but a lot has changed since then
> :-).) And if you want to use pickle and multiprocessing to send, say,
> a single big numpy array between processes, that's also really fast,
> because it's basically just a few memcpy's. The slow case is passing
> complicated objects between processes, and it's slow because pickle
> has to walk the object graph to serialize it, and walking the object
> graph is slow. Copying object graphs between subinterpreters has the
> same problem.
>

This doesn't match up with my (somewhat limited) experience. For example,
in this table of bandwidth estimates from Matthew Rocklin (CCed), IPC is
about 10x slower than a memory copy:
http://matthewrocklin.com/blog/work/2015/12/29/data-bandwidth

This makes a considerable difference when building a system do to parallel
data analytics in Python (e.g., on NumPy arrays), which is exactly what
Matthew has been working on for the past few years.

I'm sure there are other ways to avoid this expensive IPC without using
sub-interpreters, e.g., by using a tool like Plasma (
http://arrow.apache.org/blog/2017/08/08/plasma-in-memory-object-store/).
But I'm skeptical of your assessment that the current multiprocessing
approach is fast enough.
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Extension of python/json syntax to support explicitly sets and ordered dict.

[Chris Barker - NOAA Federal]

a json-like format (pyson maybe ?).


I gonna pyson is a fine idea. But don't call it extended JSON ;-)

For me, the point would be to capture Python' s richer data types.

But would you need an OrderedDict?

As pointed out, in recent cPython ( and pypy) dicts are ordered by default,
but it's not part of the language spec)

And you can use ast.literal_eval as a pyson parser, so it's almost ready to
use :-)

-CHB


What do you think about ?

  Kind regards, Matteo.

--
  email:   nast...@alternativeoutput.it,   matteo.nast...@gmail.com
  web:   www.alternativeoutput.it   irc: #linux...@irc.freenode.net
 linkedin: http://lnkd.in/SPQG87
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PEP 554: Stdlib Module to Support Multiple Interpreters in Python Code

[Nathaniel Smith]

On Thu, Sep 7, 2017 at 4:23 PM, Nick Coghlan  wrote:
> On 7 September 2017 at 15:48, Nathaniel Smith  wrote:
>> I've actually argued with the PyPy devs to try to convince them to add
>> subinterpreter support as part of their experiments with GIL-removal,
>> because I think the semantics would genuinely be nicer to work with
>> than raw threads, but they're convinced that it's impossible to make
>> this work. Or more precisely, they think you could make it work in
>> theory, but that it would be impossible to make it meaningfully more
>> efficient than using multiple processes. I want them to be wrong, but
>> I have to admit I can't see a way to make it work either...
>
> The gist of the idea is that with subinterpreters, your starting point
> is multiprocessing-style isolation (i.e. you have to use pickle to
> transfer data between subinterpreters), but you're actually running in
> a shared-memory threading context from the operating system's
> perspective, so you don't need to rely on mmap to share memory over a
> non-streaming interface.

The challenge is that streaming bytes between processes is actually
really fast -- you don't really need mmap for that. (Maybe this was
important for X11 back in the 1980s, but a lot has changed since then
:-).) And if you want to use pickle and multiprocessing to send, say,
a single big numpy array between processes, that's also really fast,
because it's basically just a few memcpy's. The slow case is passing
complicated objects between processes, and it's slow because pickle
has to walk the object graph to serialize it, and walking the object
graph is slow. Copying object graphs between subinterpreters has the
same problem.

So the only case I can see where I'd expect subinterpreters to make
communication dramatically more efficient is if you have a "deeply
immutable" type: one where not only are its instances immutable, but
all objects reachable from those instances are also guaranteed to be
immutable. So like, a tuple except that when you instantiate it it
validates that all of its elements are also marked as deeply
immutable, and errors out if not. Then when you go to send this
between subinterpreters, you can tell by checking the type of the root
object that the whole graph is immutable, so you don't need to walk it
yourself.

However, it seems impossible to support user-defined deeply-immutable
types in Python: types and functions are themselves mutable and hold
tons of references to other potentially mutable objects via __mro__,
__globals__, __weakrefs__, etc. etc., so even if a user-defined
instance can be made logically immutable it's still going to hold
references to mutable things. So the one case where subinterpreters
win is if you have a really big and complicated set of nested
pseudo-tuples of ints and strings and you're bottlenecked on passing
it between interpreters. Maybe frozendicts too. Is that enough to
justify the whole endeavor? It seems dubious to me.

I guess the other case where subprocesses lose to "real" threads is
startup time on Windows. But starting a subinterpreter is also much
more expensive than starting a thread, once you take into account the
cost of loading the application's modules into the new interpreter. In
both cases you end up needing some kind of process/subinterpreter pool
or cache to amortize that cost.

Obviously I'm committing the cardinal sin of trying to guess about
performance based on theory instead of measurement, so maybe I'm
wrong. Or maybe there's some deviously clever trick I'm missing. I
hope so -- a really useful subinterpreter multi-core store would be
awesome.

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PEP 554: Stdlib Module to Support Multiple Interpreters in Python Code

[Nick Coghlan]

On 7 September 2017 at 15:48, Nathaniel Smith  wrote:
> I've actually argued with the PyPy devs to try to convince them to add
> subinterpreter support as part of their experiments with GIL-removal,
> because I think the semantics would genuinely be nicer to work with
> than raw threads, but they're convinced that it's impossible to make
> this work. Or more precisely, they think you could make it work in
> theory, but that it would be impossible to make it meaningfully more
> efficient than using multiple processes. I want them to be wrong, but
> I have to admit I can't see a way to make it work either...

The gist of the idea is that with subinterpreters, your starting point
is multiprocessing-style isolation (i.e. you have to use pickle to
transfer data between subinterpreters), but you're actually running in
a shared-memory threading context from the operating system's
perspective, so you don't need to rely on mmap to share memory over a
non-streaming interface.

It's also definitely the case that to make this viable, we'd need to
provide fast subinterpreter friendly alternatives to C globals for use
by extension modules (otherwise adding subinterpreter compatibility
will be excessively painful), and PEP 550 is likely to be helpful
there.

Personally, I think it would make sense to add the module under PEP
411 provisional status, and make it's continued existence as a public
API contingent on actually delivering on the "lower overhead
multi-core support than multiprocessing" goal (even if it only
delivers on that front on Windows, where process creation is more
expensive and there's no fork() equivalent).

However, I'd also be entirely happy with our adding it as a private
"_subinterpreters" API for testing & experimentation purposes (see
https://bugs.python.org/issue30439 ), and reconsidering introducing it
as a public API after there's more concrete evidence as to what can
actually be achieved based on it.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Adding "View Python 3 Documentation" to all Python 2 documentation URLs

[Nick Coghlan]

On 7 September 2017 at 12:24, Sebastian Krause  wrote:
> Sebastian Krause  wrote:
>> This is why if you search for "python 3 $module" in Google, you'll
>> never see a direct link to the 3.5 or 3.6 versions of the
>> documentation (because Google merges them with the generic
>> docs.python.org/3/), but you still results for versions 3.2, 3.3
>> etc. of the documentation (because the lack the canonical links).
>>
>> A very good step would be to also add this canoncial link to the
>> documentation versions 3.0-3.4, this will make docs.python.org/3.3/
>> etc. vanish from Google and probably rank the generic
>> docs.python.org/3/ higher than now.
>
> Here is Nick Coghlan's bpo issue which added these canonical links:
> https://bugs.python.org/issue26355 - looks like applying this to the
> older doc versions never happened in the end.

Right, as adding those will need to be handled through the web server
and/or by manually regenerating the docs with the additional HTML
headers - making the changes to the CPython repo won't achieve
anything, since the docs for those branches aren't automatically
regenerated anymore.

Another big task that could be undertaken is to start re-routing
unqualified deep links to Python 3 - the reason we still haven't done
that is because the tree layout is actually different (as per the
six.moves module), so it isn't a trivial rewrite rule the way the
current redirection into the Python 2 docs is. Given such a mapping,
it would also be possible to add the corresponding canonical URL
entries to the Python 2.7 documentation to merge their search ranking
in to the corresponding Python 3 pages.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PEP 554: Stdlib Module to Support Multiple Interpreters in Python Code

[Nathaniel Smith]

On Thu, Sep 7, 2017 at 11:26 AM, Eric Snow  wrote:
> Hi all,
>
> As part of the multi-core work I'm proposing the addition of the
> "interpreters" module to the stdlib.  This will expose the existing
> subinterpreters C-API to Python code.  I've purposefully kept the API
> simple.  Please let me know what you think.

My concern about this is the same as it was last time -- the work
looks neat, but right now, almost no-one uses subinterpreters
(basically it's Jep and mod_wsgi and that's it?), and therefore many
packages get away with ignoring subinterpreters. Numpy is the one I'm
most familiar with: when we get subinterpreter bugs we close them
wontfix, because supporting subinterpreters properly would require
non-trivial auditing, add overhead for non-subinterpreter use cases,
and benefit a tiny tiny fraction of our users.

If we add a friendly python-level API like this, then we're committing
to this being a part of Python for the long term and encouraging
people to use it, which puts pressure on downstream packages to do
that work... but it's still not clear whether any benefits will
actually materialize.

I've actually argued with the PyPy devs to try to convince them to add
subinterpreter support as part of their experiments with GIL-removal,
because I think the semantics would genuinely be nicer to work with
than raw threads, but they're convinced that it's impossible to make
this work. Or more precisely, they think you could make it work in
theory, but that it would be impossible to make it meaningfully more
efficient than using multiple processes. I want them to be wrong, but
I have to admit I can't see a way to make it work either...

If this is being justified by the multicore use case, and specifically
by the theory that having two interpreters in the same process will
allow for more efficient communication than two interpreters in two
different processes, then... why should we believe that that's
actually possible? I want your project to succeed, but if it's going
to fail then it seems better if it fails before we commit to exposing
new APIs.

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PEP 554: Stdlib Module to Support Multiple Interpreters in Python Code

[Eric Snow]

On Thu, Sep 7, 2017 at 1:14 PM, Sebastian Krause  wrote:
> How is the GIL situation with subinterpreters these days, is the
> long-term goal still "solving multi-core Python", i.e. using
> multiple CPU cores from within the same process? Or is it mainly
> used for isolation?

The GIL is still process-global.  The goal is indeed to change this to
support actual multi-core parallelism.  However, the benefits of
interpreter isolation are certainly a win otherwise. :)

-eric
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PEP 554: Stdlib Module to Support Multiple Interpreters in Python Code

[Eric Snow]

On Thu, Sep 7, 2017 at 12:44 PM, Paul Moore  wrote:
> Ah, OK. so if I create a new interpreter, none of the classes,
> functions, or objects defined in my calling code will exist within the
> target interpreter? That makes sense, but I'd missed that nuance from
> the description. Again, this is probably worth noting in the PEP.

I'll make sure the PEP is more clear about this.

>
> And for the record, based on that one fact, I'm perfectly OK with the
> initial API being string-only.

Great! :)

-eric
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] lazy import via __future__ or compiler analysis

[Neil Schemenauer]

Barry Warsaw  wrote:
> There are a few other things that might end up marking a module as
> "industrious" (my thesaurus's antonym for "lazy").

Good points.  The analysis can be simple at first and then we can
enhance it to be smarter about what is okay and still lazy load.  We
may evolve it over time too, making things that are not strictly
safe still not trigger the "industrious" load lazy anyhow.

Another idea is to introduce __lazy__ or some such in the global
namespace of the module, if present, e.g.

__lazy__ = True

then the analysis doesn't do anything except return True.  The
module has explicitly stated that side-effects in the top-level code
are okay to be done in a lazy fashion.

Perhaps with a little bit of smarts in the analsis and a little
sprinkling of __lazy__ flags, we can get a big chunk of modules to
lazy load.

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PEP 554: Stdlib Module to Support Multiple Interpreters in Python Code

[Sebastian Krause]

Eric Snow  wrote:
> 1. add a basic queue class for passing objects between interpreters
> * only support strings at first (though Nick pointed out we could
> fall back to pickle or marshal for unsupported objects)
> 2. implement CSP on top of subinterpreters
> 3. expand the queue's supported types
> 4. add something like Interpreter.call()

How is the GIL situation with subinterpreters these days, is the
long-term goal still "solving multi-core Python", i.e. using
multiple CPU cores from within the same process? Or is it mainly
used for isolation?

Sebastian
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] if as

[Chris Angelico]

On Fri, Sep 8, 2017 at 5:40 AM, Philipp A.  wrote:
> sorry, it’s a bit more difficult. this works:
> https://gist.github.com/flying-sheep/86dfcc1bdd71a33fa3483b83e254084c

If this can be made to work - even hackily - can it be added into
contextlib.contextmanager (or technically its helper class)?
Currently, its __enter__ looks like this:

def __enter__(self):
try:
return next(self.gen)
except StopIteration:
raise RuntimeError("generator didn't yield") from None

If that raise were replaced with the hackiness of skipping the body,
we could wrap everything up nicely:

@contextlib.contextmanager
def iff(thing):
if thing: yield thing
# otherwise don't yield

for i in range(1, 11):
with iff(random.randrange(3)) as val:
print(i, val) # won't print any zeroes

ChrisA
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PEP 554: Stdlib Module to Support Multiple Interpreters in Python Code

[Paul Moore]

On 7 September 2017 at 20:14, Eric Snow  wrote:
> On Thu, Sep 7, 2017 at 11:52 AM, Paul Moore  wrote:
>> Is there any reason why passing a callable and args is unsafe, and/or
>> difficult? Naively, I'd assume that
>>
>> interp.call('f(a)')
>>
>> would be precisely as safe as
>>
>> interp.call(f, a)
>
> The problem for now is with sharing objects between interpreters.  The
> simplest safe approach currently is to restrict execution to source
> strings.  Then there are no complications.  Interpreter.call() makes
> sense but I'd like to wait until we get feel for how subinterpreters
> get used and until we address some of the issues with object passing.

Ah, OK. so if I create a new interpreter, none of the classes,
functions, or objects defined in my calling code will exist within the
target interpreter? That makes sense, but I'd missed that nuance from
the description. Again, this is probably worth noting in the PEP.

And for the record, based on that one fact, I'm perfectly OK with the
initial API being string-only.

> FWIW, here are what I see as the next steps for subinterpreters in the stdlib:
>
> 1. add a basic queue class for passing objects between interpreters
> * only support strings at first (though Nick pointed out we could
> fall back to pickle or marshal for unsupported objects)
> 2. implement CSP on top of subinterpreters
> 3. expand the queue's supported types
> 4. add something like Interpreter.call()
>
> I didn't include such a queue in this proposal because I wanted to
> keep it as focused as possible.  I'll add a note to the PEP about
> this.

This all sounds very reasonable. Thanks for the clarification.
Paul
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] if as

[Philipp A.]

sorry, it’s a bit more difficult. this works:
https://gist.github.com/flying-sheep/86dfcc1bdd71a33fa3483b83e254084c

Philipp A.  schrieb am Do., 7. Sep. 2017 um 21:18 Uhr:

> Sadly it’s hard to create a context manager that skips its body like this:
>
> with unpack(computation()) as result:
> do_something_with_result(result)
>
> You can do it with some hackery like described here:
> https://stackoverflow.com/a/12594789/247482
>
> class unpack:
> def __init__(self, pred):
> self.pred = pred
>
> def __enter__(self):
> if self.pred:
> return self.pred
> # else skip the with block’s body
> sys.settrace(lambda *args, **kw: None)
> frame = inspect.currentframe(1)
> frame.f_trace = self.trace
>
> def trace(self, frame, event, arg):
> raise
>
> def __exit__(self, type, value, traceback):
> return True  # suppress the exception
>
> Steven D'Aprano  schrieb am Do., 7. Sep. 2017 um
> 18:26 Uhr:
>
>> On Thu, Sep 07, 2017 at 04:36:40PM +0200, Jason H wrote:
>>
>> > I also often wonder why we are left doing an assignment and test. You
>> have two options:
>> > 1. assign to a variable then test and use
>> > 2. repeat the function call
>>
>> Personally, I don't see what's wrong with the "assign then test" idiom.
>>
>> x = something()
>> if x:
>> do_stuff()
>>
>>
>> > I would offer that 'with' [sh|c]ould be used:
>> > with test() as x:
>> >handle_truthy(x)
>> > else:
>> >handle_falsey() # do we provide x here too? Because None vs False?
>>
>>
>> This would cause confusing errors and mysterious behaviour, depending on
>> whether the test() object was a context manager or not. Which should
>> take priority? If you see:
>>
>> with spam() as x:
>>do_stuff
>>
>> is that a context manager with block (like "with open(...) as f") or
>> your boolean if test in disguise?
>>
>> Having "with" sometimes be a disguised "if" and sometimes a regular
>> "with" will make it really, really hard to reason about code.
>>
>>
>> --
>> Steve
>> ___
>> Python-ideas mailing list
>> Python-ideas@python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Adding "View Python 3 Documentation" to all Python 2 documentation URLs

[Sebastian Krause]

Sebastian Krause  wrote:
> This is why if you search for "python 3 $module" in Google, you'll
> never see a direct link to the 3.5 or 3.6 versions of the
> documentation (because Google merges them with the generic
> docs.python.org/3/), but you still results for versions 3.2, 3.3
> etc. of the documentation (because the lack the canonical links).
>
> A very good step would be to also add this canoncial link to the
> documentation versions 3.0-3.4, this will make docs.python.org/3.3/
> etc. vanish from Google and probably rank the generic
> docs.python.org/3/ higher than now.

Here is Nick Coghlan's bpo issue which added these canonical links:
https://bugs.python.org/issue26355 - looks like applying this to the
older doc versions never happened in the end.
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] if as

[Philipp A.]

Sadly it’s hard to create a context manager that skips its body like this:

with unpack(computation()) as result:
do_something_with_result(result)

You can do it with some hackery like described here:
https://stackoverflow.com/a/12594789/247482

class unpack:
def __init__(self, pred):
self.pred = pred

def __enter__(self):
if self.pred:
return self.pred
# else skip the with block’s body
sys.settrace(lambda *args, **kw: None)
frame = inspect.currentframe(1)
frame.f_trace = self.trace

def trace(self, frame, event, arg):
raise

def __exit__(self, type, value, traceback):
return True  # suppress the exception

Steven D'Aprano  schrieb am Do., 7. Sep. 2017 um
18:26 Uhr:

> On Thu, Sep 07, 2017 at 04:36:40PM +0200, Jason H wrote:
>
> > I also often wonder why we are left doing an assignment and test. You
> have two options:
> > 1. assign to a variable then test and use
> > 2. repeat the function call
>
> Personally, I don't see what's wrong with the "assign then test" idiom.
>
> x = something()
> if x:
> do_stuff()
>
>
> > I would offer that 'with' [sh|c]ould be used:
> > with test() as x:
> >handle_truthy(x)
> > else:
> >handle_falsey() # do we provide x here too? Because None vs False?
>
>
> This would cause confusing errors and mysterious behaviour, depending on
> whether the test() object was a context manager or not. Which should
> take priority? If you see:
>
> with spam() as x:
>do_stuff
>
> is that a context manager with block (like "with open(...) as f") or
> your boolean if test in disguise?
>
> Having "with" sometimes be a disguised "if" and sometimes a regular
> "with" will make it really, really hard to reason about code.
>
>
> --
> Steve
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] lazy import via __future__ or compiler analysis

[Barry Warsaw]

Neil Schemenauer wrote:

> Introduce a lazy module import process that modules can opt-in to.
> The opt-in would either be with a __future__ statement or the
> compiler would statically analyze the module and determine if it is
> safe.  E.g. if the module has no module level statements besides
> imports.

and `def` and `class` of course.

There are a few other things that might end up marking a module as
"industrious" (my thesaurus's antonym for "lazy").  There will likely be
assignments of module global such as:

MY_CONST = 'something'

and it may even be a little more complicated:

COLORS = dict(
red=1,
blue=2,
green=3,
)
REVERSE = {value: key for key, value in COLORS.items()}

A naive evaluation of such a module might not notice them as lazy, but I
think they could still be treated as such.

Function and class decorators might also be false positives.  E.g.

@public
def my_public_function():
pass

or even

@mungify
class Munged:
pass

Maybe that's just the cost of doing business, and if they clear the lazy
flag, so be it.  But it feels like doing so will leave quite a bit of
lazy loading opportunity left on the table.  And I'm not sure you can
solve all of those by moving things to a module level __init__().

Cheers,
-Barry


___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Adding "View Python 3 Documentation" to all Python 2 documentation URLs

[Sebastian Krause]

Guido van Rossum  wrote:
> We just need someone with SEO experience to fix this for us.

I'm not an SEO expert, but I think a possible approach would be
using (or partly abusing) the  element on the
documentation pages:

https://support.google.com/webmasters/answer/139066

Python's documentation is already using these canonical links, just
in an incomplete way: The Python documentation for 2.7 and 3.5 and
later has a  pointing to the generic
documentation URL of the same major Python version, e.g. in
https://docs.python.org/3/library/functools.html you can find this
in the page source:

https://docs.python.org/3/library/functools.html; />

This is why if you search for "python 3 $module" in Google, you'll
never see a direct link to the 3.5 or 3.6 versions of the
documentation (because Google merges them with the generic
docs.python.org/3/), but you still results for versions 3.2, 3.3
etc. of the documentation (because the lack the canonical links).

A very good step would be to also add this canoncial link to the
documentation versions 3.0-3.4, this will make docs.python.org/3.3/
etc. vanish from Google and probably rank the generic
docs.python.org/3/ higher than now.

And now to the abuse part (I'm honestly not sure how well this would
actually work): If you would add such a canonical link in
docs.python.org/2/ pointing to docs.python.org/3/ (at least where
the same module exists in both versions), you would eventually hide
/2/ from the search results. If you don't want to be so extreme
(people still want Python 2 documentation if they search for "python
2 $module") you could remove the canonical link just from the
specific docs.python.org/2.7/ which will then be ranked much lower
than docs.python.org/3/ in the search results, but at least still
show up.

Sebastian
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PEP 554: Stdlib Module to Support Multiple Interpreters in Python Code

[Eric Snow]

On Thu, Sep 7, 2017 at 11:52 AM, Paul Moore  wrote:
> The only quibble I have is that I'd prefer it if we had a
> run(callable, *args, **kwargs) method. Either instead of, or as well
> as, the run(string) one here.
>
> Is there any reason why passing a callable and args is unsafe, and/or
> difficult? Naively, I'd assume that
>
> interp.call('f(a)')
>
> would be precisely as safe as
>
> interp.call(f, a)

The problem for now is with sharing objects between interpreters.  The
simplest safe approach currently is to restrict execution to source
strings.  Then there are no complications.  Interpreter.call() makes
sense but I'd like to wait until we get feel for how subinterpreters
get used and until we address some of the issues with object passing.

FWIW, here are what I see as the next steps for subinterpreters in the stdlib:

1. add a basic queue class for passing objects between interpreters
* only support strings at first (though Nick pointed out we could
fall back to pickle or marshal for unsupported objects)
2. implement CSP on top of subinterpreters
3. expand the queue's supported types
4. add something like Interpreter.call()

I didn't include such a queue in this proposal because I wanted to
keep it as focused as possible.  I'll add a note to the PEP about
this.

>
> Am I missing something? Name visibility or scoping issues come to mind
> as possible complications I'm not seeing. At the least, if we don't
> want a callable-and-args form yet, a note in the PEP explaining why
> it's been omitted would be worthwhile.

I'll add a note to the PEP.  Thanks for pointing this out. :)

-eric
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Extension of python/json syntax to support explicitly sets and ordered dict.

[rym...@gmail.com]

IIRC in CPython 3.6 and PyPy dicts are ordered based on insertion anyway;
although it's an implementation-specific detail, realistically it removes
some of the use cases for ordered dictionary literals.


--
Ryan (ライアン)
Yoko Shimomura, ryo (supercell/EGOIST), Hiroyuki Sawano >> everyone
elsehttp://refi64.com

On Sep 7, 2017 at 6:40 AM, >
wrote:

Hi all,

few days ago I thought about a way to rapresent sets and
ordered dicts using a json compatible syntax, these are my conclusions:

A set could be defined as { item1, item2, item3[...] }
with {,} as an empty set

An ordered dict could be defined as [ item1: value1, item2: value2 ... ]
with [:] ase an empty odered dict

It could be used inside python code or to serialize python structures in
a json-like format (pyson maybe ?).

What do you think about ?

  Kind regards, Matteo.

--
  email:   nast...@alternativeoutput.it,   matteo.nast...@gmail.com
  web:   www.alternativeoutput.it   irc: #linux...@irc.freenode.net
 linkedin: http://lnkd.in/SPQG87
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PEP 554: Stdlib Module to Support Multiple Interpreters in Python Code

[Paul Moore]

On 7 September 2017 at 19:26, Eric Snow  wrote:
> As part of the multi-core work I'm proposing the addition of the
> "interpreters" module to the stdlib.  This will expose the existing
> subinterpreters C-API to Python code.  I've purposefully kept the API
> simple.  Please let me know what you think.

Looks good. I agree with the idea of keeping the interface simple in
the first instance - we can easily add extra functionality later, but
removing stuff (or worse still, finding that stuff we thought was OK
but had missed corner cases of was broken) is much harder.

>run(code):
>
>   Run the provided Python code in the interpreter, in the current
>   OS thread.  Supported code: source text.

The only quibble I have is that I'd prefer it if we had a
run(callable, *args, **kwargs) method. Either instead of, or as well
as, the run(string) one here.

Is there any reason why passing a callable and args is unsafe, and/or
difficult? Naively, I'd assume that

interp.call('f(a)')

would be precisely as safe as

interp.call(f, a)

Am I missing something? Name visibility or scoping issues come to mind
as possible complications I'm not seeing. At the least, if we don't
want a callable-and-args form yet, a note in the PEP explaining why
it's been omitted would be worthwhile.

Paul
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] PEP 554: Stdlib Module to Support Multiple Interpreters in Python Code

[Eric Snow]

Hi all,

As part of the multi-core work I'm proposing the addition of the
"interpreters" module to the stdlib.  This will expose the existing
subinterpreters C-API to Python code.  I've purposefully kept the API
simple.  Please let me know what you think.

-eric

https://www.python.org/dev/peps/pep-0554/
https://github.com/python/peps/blob/master/pep-0554.rst
https://github.com/python/cpython/pull/1748
https://github.com/python/cpython/pull/1802
https://github.com/ericsnowcurrently/cpython/tree/high-level-interpreters-module

**

PEP: 554
Title: Multiple Interpreters in the Stdlib
Author: Eric Snow 
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 2017-09-05
Python-Version: 3.7
Post-History:

Abstract


This proposal introduces the stdlib "interpreters" module.  It exposes
the basic functionality of subinterpreters that exists in the C-API.

Rationale
=

Running code in multiple interpreters provides a useful level of
isolation within the same process.  This can be leveraged in number
of ways.  Furthermore, subinterpreters provide a well-defined framework
in which such isolation may extended.

CPython has supported subinterpreters, with increasing levels of
support, since version 1.5.  While the feature has the potential
to be a powerful tool, subinterpreters have suffered from neglect
because they are not available directly from Python.  Exposing the
existing functionality in the stdlib will help reverse the situation.

Proposal


The "interpreters" module will be added to the stdlib.  It will
provide a high-level interface to subinterpreters and wrap the low-level
"_interpreters" module.  The proposed API is inspired by the
threading module.

The module provides the following functions:

enumerate():

   Return a list of all existing interpreters.

get_current():

   Return the currently running interpreter.

get_main():

   Return the main interpreter.

create():

   Initialize a new Python interpreter and return it.  The
   interpreter will be created in the current thread and will remain
   idle until something is run in it.

The module also provides the following class:

Interpreter(id):

   id:

  The interpreter's ID (read-only).

   is_running():

  Return whether or not the interpreter is currently running.

   destroy():

  Finalize and destroy the interpreter.

   run(code):

  Run the provided Python code in the interpreter, in the current
  OS thread.  Supported code: source text.

Copyright
=

This document has been placed in the public domain.
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] lazy import via __future__ or compiler analysis

[Neil Schemenauer]

This is a half baked idea that perhaps could work.  Maybe call it
2-stage module load instead of lazy.

Introduce a lazy module import process that modules can opt-in to.
The opt-in would either be with a __future__ statement or the
compiler would statically analyze the module and determine if it is
safe.  E.g. if the module has no module level statements besides
imports.

.pyc files get some other bits of information:

A) whether the module has opted for lazy import (IS_LAZY)

B) the modules imported by the module (i.e. top-level imports,
IMPORT_LIST)

Make __import__ understand this data and do lazy loading for modules
that want it.  Sub-modules that have import side-effects will still
execute as normal and the side effects will happen when the parent
module is imported..

This would consist of a recursive process, something like:

def load_module(name):
if not IS_LAZY(name):
import as usual
else:
create lazy version of module 'name'
for subname in IMPORT_LIST(name):
load_module(subname)

An additional idea from Barry W, if a module wants lazy loading but
wants to do some init when the module is "woken up", define a
__init__ top-level function.  Python would call that function when
attributes of the module are first actually used.

My plan was to implement this with a Python __import__
implementation. I would unmarshal the .pyc, compute IS_LAZY and
IMPORT_LIST at import time.  So, not gaining a lot of speedup.  It
would prove if the idea works in terms of not causing application
crashes, etc.  I could try running it with bigger apps and see how
many modules are flagged for lazy loading.
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Adding "View Python 3 Documentation" to all Python 2 documentation URLs

[Guido van Rossum]

We just need someone with SEO experience to fix this for us.

On Thu, Sep 7, 2017 at 9:13 AM, Steven D'Aprano  wrote:

> On Thu, Sep 07, 2017 at 05:51:31AM +, Bar Harel wrote:
> > A bit radical but do you believe we can contact Google to alter the
> search
> > results?
>
> You should try DuckDuckGo, it recognises "python" searches and quotes
> from, and links to, the Python 3 documentation at the top of the page
> before the search results. E.g.:
>
> https://duckduckgo.com/html/?q=python%20functools
>
> Both Bing and Yahoo also gives the Python 2 docs first:
>
> https://www.bing.com/search?q=python+functools
>
> https://au.search.yahoo.com/search?p=python+functools
>
> --
> Steve
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>



-- 
--Guido van Rossum (python.org/~guido)
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] if as

[Steven D'Aprano]

On Thu, Sep 07, 2017 at 04:36:40PM +0200, Jason H wrote:

> I also often wonder why we are left doing an assignment and test. You have 
> two options:
> 1. assign to a variable then test and use
> 2. repeat the function call

Personally, I don't see what's wrong with the "assign then test" idiom.

x = something()
if x:
do_stuff()

 
> I would offer that 'with' [sh|c]ould be used:
> with test() as x:
>handle_truthy(x)
> else:
>handle_falsey() # do we provide x here too? Because None vs False?


This would cause confusing errors and mysterious behaviour, depending on 
whether the test() object was a context manager or not. Which should 
take priority? If you see:

with spam() as x:
   do_stuff

is that a context manager with block (like "with open(...) as f") or 
your boolean if test in disguise?

Having "with" sometimes be a disguised "if" and sometimes a regular 
"with" will make it really, really hard to reason about code.


-- 
Steve
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Adding "View Python 3 Documentation" to all Python 2 documentation URLs

[Steven D'Aprano]

On Thu, Sep 07, 2017 at 05:51:31AM +, Bar Harel wrote:
> A bit radical but do you believe we can contact Google to alter the search
> results?

You should try DuckDuckGo, it recognises "python" searches and quotes 
from, and links to, the Python 3 documentation at the top of the page 
before the search results. E.g.:

https://duckduckgo.com/html/?q=python%20functools

Both Bing and Yahoo also gives the Python 2 docs first:

https://www.bing.com/search?q=python+functools

https://au.search.yahoo.com/search?p=python+functools

-- 
Steve
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Adding "View Python 3 Documentation" to all Python 2 documentation URLs

[Wes Turner]

A sitemap.xml may be helpful for indicating search relevance to search
crawlers?

https://www.google.com/search?q=sphinx+sitemap

On Thursday, September 7, 2017, Bar Harel  wrote:

> A bit radical but do you believe we can contact Google to alter the search
> results?
> It's for the benefit of the user after all, and many just switch to python
> 3 in the version picker anyway.
>
> On Thu, Sep 7, 2017, 4:18 AM Guido van Rossum  > wrote:
>
>> That's a good idea. Could you suggest it to https://github.com/python/
>> pythondotorg/ ? (There is actually a version switcher on every page, but
>> it's rather polite. :-)
>>
>> On Wed, Sep 6, 2017 at 3:52 PM, Ryan Gonzalez > > wrote:
>>
>>> Right now, many Google searches for Python modules return the Python 2
>>> documentation. IMO since 2 will be reaching EOL in around 3 years, it
>>> would be nice to have a giant red box at the top with a link to the
>>> Python 3 documentation.
>>>
>>> SFML already does something like this: https://www.sfml-dev.org/
>>> tutorials/2.3/
>>>
>>> (I mean, it would be even nicer to have a "jump to latest version" for
>>> *all* not-new Python versions, though I figured just 2 would be a lot
>>> easier.)
>>>
>>> --
>>> Ryan (ライアン)
>>> Yoko Shimomura, ryo (supercell/EGOIST), Hiroyuki Sawano >> everyone else
>>> http://refi64.com/
>>> ___
>>> Python-ideas mailing list
>>> Python-ideas@python.org
>>> 
>>> https://mail.python.org/mailman/listinfo/python-ideas
>>> Code of Conduct: http://python.org/psf/codeofconduct/
>>>
>>
>>
>>
>> --
>> --Guido van Rossum (python.org/~guido)
>> ___
>> Python-ideas mailing list
>> Python-ideas@python.org
>> 
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] if as

[Jason H]

> Sent: Thursday, September 07, 2017 at 6:43 AM
> From: "Denis Krienbühl" 
> To: python-ideas@python.org
> Subject: [Python-ideas] if  as 
>
> Hi!
> 
> I’ve been having this idea for a few years and I thought I finally see if 
> what others think of it. I have no experience in language design and I don’t 
> know if this is something I’ve picked up in some other language. I also do 
> not know what the ramifications of implementing this idea would be. I just 
> keep thinking about it :)
> 
> I quite often write code like the following in python:
> 
>   result = computation()
>if result:
> do_something_with_computation(result)
> 
> More often than not this snippet evolves from something like this:
> 
>   if computation():
>   …
> 
> That is, I use the truthiness of the value at first. As the code grows I 
> refactor to actually do something with the result.
> 
> What I would love to see is the following syntax instead, which to me is much 
> cleaner:
> 
>if computation() as result:
>do_something_with_result(result)
> 
> Basically the result variable would be the result of the if condition’s 
> expression and it would be available the same way it would be if we used my 
> initial snippet (much the same way that the result of with expressions also 
> stays around outside the with-block).
> 
> Any feedback is appreciated :)

I also often wonder why we are left doing an assignment and test. You have two 
options:
1. assign to a variable then test and use
2. repeat the function call

I would offer that 'with' [sh|c]ould be used:
with test() as x:
   handle_truthy(x)
else:
   handle_falsey() # do we provide x here too? Because None vs False?


___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Extension of python/json syntax to support explicitly sets and ordered dict.

[Chris Angelico]

On Thu, Sep 7, 2017 at 9:59 PM, Matteo Nastasi
 wrote:
> I was thinking about an extended json too, not just python syntax.
>
> What do you think about it ?
>
> Regards and thank you for your time.

Extend JSON? No thank you. Down the path of extending simple standards
in proprietary ways lies a madness that I do not wish on my best
friends, much less my worst enemies.

ChrisA
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Extension of python/json syntax to support explicitly sets and ordered dict.

[Matteo Nastasi]

I was thinking about an extended json too, not just python syntax.

What do you think about it ?

Regards and thank you for your time.

  Matteo.

On Thu, Sep 07, 2017 at 09:54:15PM +1000, Steven D'Aprano wrote:
> On Thu, Sep 07, 2017 at 01:36:31PM +0200, Matteo Nastasi wrote:
> 
> > A set could be defined as { item1, item2, item3[...] }
> 
> Guido's time machine strikes again. Python 3:
> 
> py> s = {1, 2, 3}
> py> type(s)
> 
> 
> 
> > with {,} as an empty set
> 
> That looks like a typo. I don't think that having a literal for empty 
> sets is important enough that we need worry about the lack.
> 
> 
> > An ordered dict could be defined as [ item1: value1, item2: value2 ... ]
> > with [:] ase an empty odered dict
> 
> I think that's been proposed before.
> 
> I don't hate that suggestion, but I don't think its very useful either. 
> We already have a de facto "Ordered Mapping" literal that can be passed 
> to the OrderedDict constructor:
> 
> OrderedDict(
> [(key1, value1), (key2, value2), (key3, value3), ...]
> )
> 
> Its not quite as compact, but it isn't too awful. And if you really 
> don't like it, try:
> 
> OrderedDict(zip(keys, values))
> 
> 
> 
> -- 
> Steven
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
> 

-- 
  email:   nast...@alternativeoutput.it,   matteo.nast...@gmail.com
  web:   www.alternativeoutput.it   irc: #linux...@irc.freenode.net
 linkedin: http://lnkd.in/SPQG87
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Extension of python/json syntax to support explicitly sets and ordered dict.

[Steven D'Aprano]

On Thu, Sep 07, 2017 at 01:36:31PM +0200, Matteo Nastasi wrote:

> A set could be defined as { item1, item2, item3[...] }

Guido's time machine strikes again. Python 3:

py> s = {1, 2, 3}
py> type(s)



> with {,} as an empty set

That looks like a typo. I don't think that having a literal for empty 
sets is important enough that we need worry about the lack.


> An ordered dict could be defined as [ item1: value1, item2: value2 ... ]
> with [:] ase an empty odered dict

I think that's been proposed before.

I don't hate that suggestion, but I don't think its very useful either. 
We already have a de facto "Ordered Mapping" literal that can be passed 
to the OrderedDict constructor:

OrderedDict(
[(key1, value1), (key2, value2), (key3, value3), ...]
)

Its not quite as compact, but it isn't too awful. And if you really 
don't like it, try:

OrderedDict(zip(keys, values))



-- 
Steven
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Extension of python/json syntax to support explicitly sets and ordered dict.

[Matteo Nastasi]

Hi all,

few days ago I thought about a way to rapresent sets and
ordered dicts using a json compatible syntax, these are my conclusions:

A set could be defined as { item1, item2, item3[...] }
with {,} as an empty set

An ordered dict could be defined as [ item1: value1, item2: value2 ... ]
with [:] ase an empty odered dict

It could be used inside python code or to serialize python structures in
a json-like format (pyson maybe ?).

What do you think about ?

  Kind regards, Matteo.

-- 
  email:   nast...@alternativeoutput.it,   matteo.nast...@gmail.com
  web:   www.alternativeoutput.it   irc: #linux...@irc.freenode.net
 linkedin: http://lnkd.in/SPQG87
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] if as

[Denis Krienbühl]

> On 7 Sep 2017, at 13:11, Paul Moore  wrote:
> 
> On 7 September 2017 at 11:43, Denis Krienbühl  wrote:
>> What I would love to see is the following syntax instead, which to me is 
>> much cleaner:
>> 
>>   if computation() as result:
>>   do_something_with_result(result)
> 
> Hi - thanks for your suggestion! This has actually come up quite a lot
> in the past. Here's a couple of links to threads you might want to
> read (it's not surprising if you missed these, it's not that easy to
> come up with a good search term for this topic).
> 
> https://mail.python.org/pipermail/python-ideas/2012-January/013461.html
> https://mail.python.org/pipermail/python-ideas/2009-March/003423.html
>  (This thread includes a note by Guido that he intentionally left out
> this functionality)
> 
> In summary, it's a reasonably commonly suggested idea, but there's not
> enough benefit to warrant adding it to the language.
> 
> Paul

I see, thank you for digging those threads up. I’ll read them to learn a thing 
or two :)
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] if as

[Paul Moore]

On 7 September 2017 at 11:43, Denis Krienbühl  wrote:
> What I would love to see is the following syntax instead, which to me is much 
> cleaner:
>
>if computation() as result:
>do_something_with_result(result)

Hi - thanks for your suggestion! This has actually come up quite a lot
in the past. Here's a couple of links to threads you might want to
read (it's not surprising if you missed these, it's not that easy to
come up with a good search term for this topic).

https://mail.python.org/pipermail/python-ideas/2012-January/013461.html
https://mail.python.org/pipermail/python-ideas/2009-March/003423.html
  (This thread includes a note by Guido that he intentionally left out
this functionality)

In summary, it's a reasonably commonly suggested idea, but there's not
enough benefit to warrant adding it to the language.

Paul
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] if as

[Denis Krienbühl]

Hi!

I’ve been having this idea for a few years and I thought I finally see if what 
others think of it. I have no experience in language design and I don’t know if 
this is something I’ve picked up in some other language. I also do not know 
what the ramifications of implementing this idea would be. I just keep thinking 
about it :)

I quite often write code like the following in python:

result = computation()
   if result:
do_something_with_computation(result)

More often than not this snippet evolves from something like this:

  if computation():
  …

That is, I use the truthiness of the value at first. As the code grows I 
refactor to actually do something with the result.

What I would love to see is the following syntax instead, which to me is much 
cleaner:

   if computation() as result:
   do_something_with_result(result)

Basically the result variable would be the result of the if condition’s 
expression and it would be available the same way it would be if we used my 
initial snippet (much the same way that the result of with expressions also 
stays around outside the with-block).

Any feedback is appreciated :)

Cheers,

Denis

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Adding "View Python 3 Documentation" to all Python 2 documentation URLs

[Philipp A.]

A progmatic workaround for yourself:
https://addons.mozilla.org/de/firefox/addon/py3direct/
https://chrome.google.com/webstore/detail/py3redirect/codfjigcljdnlklcaopdciclmmdandig

Bar Harel  schrieb am Do., 7. Sep. 2017 um 07:52 Uhr:

> A bit radical but do you believe we can contact Google to alter the search
> results?
> It's for the benefit of the user after all, and many just switch to python
> 3 in the version picker anyway.
>
> On Thu, Sep 7, 2017, 4:18 AM Guido van Rossum  wrote:
>
>> That's a good idea. Could you suggest it to
>> https://github.com/python/pythondotorg/ ? (There is actually a version
>> switcher on every page, but it's rather polite. :-)
>>
>> On Wed, Sep 6, 2017 at 3:52 PM, Ryan Gonzalez  wrote:
>>
>>> Right now, many Google searches for Python modules return the Python 2
>>> documentation. IMO since 2 will be reaching EOL in around 3 years, it
>>> would be nice to have a giant red box at the top with a link to the
>>> Python 3 documentation.
>>>
>>> SFML already does something like this:
>>> https://www.sfml-dev.org/tutorials/2.3/
>>>
>>> (I mean, it would be even nicer to have a "jump to latest version" for
>>> *all* not-new Python versions, though I figured just 2 would be a lot
>>> easier.)
>>>
>>> --
>>> Ryan (ライアン)
>>> Yoko Shimomura, ryo (supercell/EGOIST), Hiroyuki Sawano >> everyone else
>>> http://refi64.com/
>>> ___
>>> Python-ideas mailing list
>>> Python-ideas@python.org
>>> https://mail.python.org/mailman/listinfo/python-ideas
>>> Code of Conduct: http://python.org/psf/codeofconduct/
>>>
>>
>>
>>
>> --
>> --Guido van Rossum (python.org/~guido)
>> ___
>> Python-ideas mailing list
>> Python-ideas@python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/