Re: [Python-ideas] Generator syntax hooks?

[Nick Coghlan]

On 10 August 2017 at 01:49, Soni L.  wrote:
> On 2017-08-09 11:54 AM, Nick Coghlan wrote:
>> Right, I was separating the original request to make "{x for x in
>> integers if 1000 <= x < 100}" work into the concrete proposal to
>> make exactly *that* syntax work (which I don't think is feasible), and
>> the slightly more general notion of offering a more math-like syntax
>> that allows finite sets to be built from infinite iterators by
>> defining a termination condition in addition to a filter condition.
>
> Ok. A concrete proposal would give a read-only 'filter' argument to the
> iterator somehow, which represents some form of simplified AST of the
> condition.
>
> So e.g. {x for x in integers if (lambda v: 1000 <= v < 100)(x)} would
> never complete, but {x for x in integers if 1000 <= x < 100} would. (But
> perhaps lambda objects should include an AST attribute... Having it for
> normal functions would introduce too much overhead tho, and then it would no
> longer be a simplified AST, but rather a complete python AST, which we don't
> want.)

There have been a variety of different "thunking" proposals over the
years, but they've all foundered on the question of what the
*primitive* quoted form should look like, and how the thunks should
subsequently be executed.

For cases like this, where integration with Python's name resolution
mechanism isn't actually required, folks have ended up just using
strings, where the only downside is the fact that syntax highlighters
and other static analysers don't know that the contents are supposed
to be valid Python code. In a case like this, that might look like:

{x for x in integers.build_set("1000 <= x < 100")}

As with regexes, the cost of dynamically parsing such strings can then
be amortised at runtime through the use of an appropriate caching
strategy.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] PyPI JSON Metadata Standardization for Mirrors

[Cooper Ry Lees]

Hi all,

First time emailer, so please be kind. Also, if this is not the right
mailing list for PyPA talk, I apologize. Please point me in the right
direction if so. The main reason I have emailed here is I believe it may be
PEP time to standardize the JSON metadata that PyPI makes available, like
what was done for the `'simple API` described in PEP503.

I've been doing a bit of work on `bandersnatch` (I didn't name it), which
is a PEP 381 mirroring package and wanted to enhance it to also mirror the
handy JSON metadata PyPI generates and makes available @
https://pypi.python.org/pypi/PKG_NAME/json.

I've done a PR on bandersnatch as a POC that mirrors both the PyPI
directory structure (URL/pypi/PKG_NAME/json) and created a standardizable
URL/json/PKG_NAME that the former symlinks to (to be served by NGINX / some
other proxy). I'm also contemplating naming the directory 'metadata' rather
than JSON so if some new hotness / we want to change the format down the
line we're not stuck with json as the dirname. This PR can be found here:
https://bitbucket.org/pypa/bandersnatch/pull-requests/33/save-json-metadata-to-mirror

My main use case is to write a very simple async 'verifier' tool that will
crawl all the JSON files and then ensure the packages directory on each of
my internal mirrors (I have a mirror per region / datacenter) have all the
files they should. I sync centrally (to save resource on the PyPI
infrastructure) and then rsync out all the diffs to each region /
datacenter, and under some failure scenarios I could miss a file or many.
So I feel using JSON pulled down from the authoritative source will allow
an async job to verify the MD5 of all the package files on each mirror.

What are peoples thoughts here? Is it worth a PEP similar to PEP503 going
forward? Can people enhance / share some thoughts on this idea.

Thanks,
Cooper Lees
m...@cooperlees.com 
https://cooperlees.com/
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Generator syntax hooks?

[Nick Coghlan]

On 10 August 2017 at 00:54, Nick Coghlan  wrote:
> Yeah, if we ever did add something like this, I suspect a translation
> using takewhile would potentially be easier for at least some users to
> understand than the one to a break condition:
>
> {x for x in itertools.count(0) if 1000 <= x while x < 100}
>
> <=>
>
> x = set()
> for x in itertools.count(0):
> if 1000 <= x:
> set.add(x)
> # If you've never used the loop-and-a-half idiom, it's
> # not obvious why "while " means "if not : break"
> if not x < 100:
> break
>
> is roughly
>
> {x for x in itertools.takewhile(itertools.count(0), lambda x: x <
> 100) if 1000 <= x}
>
> <=>
>
> x = set()
> for x in takewhile(itertools.count(0), lambda x: x < 100):
> if 1000 <= x:
> set.add(x)

Ugh, this discrepancy is worse than I thought, since the translation
with that clause order is actually wrong (Terry mentioned this by
pointing out that the proposed syntactic translation implemented
"do...while" ordering). The takewhile example is also wrong, since it
has the arguments in the wrong order. Fixing both of those issues
gives the comparison:

{x for x in itertools.count(0) while x < 100 if 1000 <= x}

<=>

x = set()
for x in itertools.count(0):
# If you've never used the loop-and-a-half idiom, it's
# not obvious why "while " means "if : 
else: break"
if x < 100:
if 1000 <= x:
set.add(x)
else:
break

is roughly:

{x for x in itertools.takewhile(lambda x: x < 100,
itertools.count(0)) if 1000 <= x}

<=>

x = set()
for x in takewhile(lambda x: x < 100, itertools.count(0)):
if 1000 <= x:
set.add(x)

And I think that gets me back to pretty much where I was the last time
this came up: a while clause in comprehensions really only makes sense
in combination with a while clause on for loops, where:

for x in itertools.count(0) while x < 100:
...

was roughly equivalent to:

for x in itertools.count(0):
if x < 100:
...
else:

break

(such that there's only one loop from the point of view of
break/continue/else, but the loop may terminate based on either
exhaustion of the underlying iterator *or* some specific condition
becoming false)

While I do think such a clause would be more readable for more people
than the dropwhile/takewhile equivalents (especially when the latter
end up needing to use lambda expressions), I'm still dubious that
these cases come up often enough to justify the addition of a
for-while loop as a composite construct (the old "dropwhile and
takewhile aren't even common enough to justify being builtins, why
should they jump all the way to syntactic support?" question applies).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Mimetypes Include application/json

[Nick Coghlan]

On 10 August 2017 at 04:24, Serhiy Storchaka  wrote:
> 09.08.17 21:17, Brett Cannon пише:
>>
>> On Wed, 9 Aug 2017 at 10:43 Nate. > > wrote:
>> A friend and I have hit a funny situation with the `mimetypes.py`
>> library
>> guessing the type for a '.json' file. Is there a reason why '.json'
>> hasn't been
>> added to the mapping?
>>
>>
>> Probably no one thought about it since the module was added in 1997 which
>> is only 2 years after the creation of JavaScript itself. :)
>
> No one proposed a patch.

That's not *quite* true - there was at least one proposal a few years
to modernise the mimetypes list, but the one I was involved in
reviewing got intertwined with a proposal to completely rewrite the
mimetypes module, and the submitter wasn't interested in creating a
more minimalist patch that solved the specific problem (i.e. the list
was pretty out of date) without all the extraneous changes to how the
module actually worked :(

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Pseudo methods

[Nick Coghlan]

On 9 August 2017 at 18:19, Stephen J. Turnbull
 wrote:
> Nick Coghlan writes:
>
>  > To analyse and investigate this code, we need to "just know" that:
>
> You can of course hope that help(input().has_vowels) will tell you
> where to find it.  If it doesn't, well, shame on you for depending on
> source-unavailable software that you don't understand. ;-)

We can't run "help" when we're reviewing a diff or otherwise reading
code in a situation where interactive help isn't available :)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Generator syntax hooks?

[Terry Reedy]

On 8/9/2017 10:54 AM, Nick Coghlan wrote:

On 9 August 2017 at 15:38, Guido van Rossum  wrote:

On Tue, Aug 8, 2017 at 10:06 PM, Nick Coghlan  wrote:

The OP's proposal doesn't fit into that category though: rather it's
asking about the case where we have an infinite iterator (e.g.
itertools.count(0)), and want to drop items until they start meeting
some condition (i.e. itertools.dropwhile) and then terminate the
iterator as soon as another condition is no longer met (i.e.
itertools.takewhile).


I don't think that's what the OP meant. The original proposal seemed to
assume that it would be somehow reasonable for the input ("integers" in the
example) to be able to see and parse the condition in the generator
expression ("1000 <= x < 10" in the example, with "x" somehow known to
be bound to the iteration value). That's at least what I think the remark "I
like mathy syntax" referred to.


Right, I was separating the original request to make "{x for x in
integers if 1000 <= x < 100}" work into the concrete proposal to
make exactly *that* syntax work (which I don't think is feasible), and
the slightly more general notion of offering a more math-like syntax
that allows finite sets to be built from infinite iterators by
defining a termination condition in addition to a filter condition.


We already have three nice one liners for that, one of which you gave.

x = set(filter(filter_condition, takewhile(continue_condition, source)))
x = set(x for x in takewhile(continue_condition, source) if 
filter_condition)

x = {x for x in takewhile(continue_condition, source) if filter_condition}

Replace takewhile with islice(source, max) if the continue condition is 
(number seen < max).  Add enumerate if the running count is needed 
otherwise.


Terminating an infinite iterator and filtering the initial slice are 
different operations.  The operations are easily composed as they are, 
in multiple ways.  Trying to mix them together in one jumbled special 
syntax is a bad idea to me.



There aren't any technical barriers I'm aware of to implementing that,
with the main historical objection being that instead of the
comprehension level while clause mapping to a while loop directly the
way the for and if clauses map to their statement level counterparts,
it would instead map to the conditional break in the expanded
loop-and-a-half form:

 while True:

 

 if not condition:
 break


In other words, aside from other issues, you would have 'while' mean 
'do...while' in this one special place.  -1.


--
Terry Jan Reedy

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Generator syntax hooks?

[Chris Barker]

On Tue, Aug 8, 2017 at 10:06 PM, Nick Coghlan  wrote:

> On 8 August 2017 at 09:06, Chris Barker  wrote:
> > It would be nice to have an easier access to an "slice iterator" though
> --
> > one of these days I may write up a proposal for that.
>
> An idea I've occasionally toyed with [1] is some kind of "iterview"
> that wraps around an arbitrary iterable and produces lazy itertools
> based results rather than immediate views or copies.
>
> However, my experience is also that folks are *really* accustomed to
> syntactic operations on containers producing either full live views
> (e.g. memoryview or numpy slices, range as a dynamically computed
> container), or actual copies (builtin container types). Having them
> produce consumable iterators instead then gets confusing due to the
> number of operations that will implicitly consume them (including
> simple "x in y" checks).
>

I agree -- which is why I"m thinking only adding a simple "iterable slice",
rather than changing the overall behavior of the container. It would be
quite clear what you are asking for.

Right now, getting the "terminate when false" behaviour requires the
> use of takewhile:
>

I can't recall the use case(s) at the moment, but I have definitely wanted
a way to break out of a comprehension -- and not always with infinite
iterators.

After all, we have "break" in both for and while loops, so clearly there is
the use case...

If someone comes up with a clean and not confusing (and general purpose)
syntax, I think it would be very useful.

-CHB

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Mimetypes Include application/json

[Nate.]

O, fun! Thank you for the guidance. I managed to find a Bug already
created, http://bugs.python.org/issue30824. I'll create a Pull Request
using that Bug.

On Wed, Aug 9, 2017 at 1:18 PM Brett Cannon  wrote:

> On Wed, 9 Aug 2017 at 10:43 Nate.  wrote:
>
>> Hi,
>>
>> A friend and I have hit a funny situation with the `mimetypes.py` library
>> guessing the type for a '.json' file. Is there a reason why '.json'
>> hasn't been
>> added to the mapping?
>>
>
> Probably no one thought about it since the module was added in 1997 which
> is only 2 years after the creation of JavaScript itself. :)
>
>
>>
>> Without `mailcap` installed:
>>
>> [root@de169da8cc46 /]# python3 -m mimetypes build.json
>> I don't know anything about type build.json
>>
>> With `mailcap` installed:
>>
>> [root@de169da8cc46 /]# python3 -m mimetypes build.json
>> type: application/json encoding: None
>>
>> We experimented with adding a mapping for '.json' to 'application/json' to
>> `mimetypes.py` and it seems to work fine for us. It looks like it has been
>> registered with IANA and everything.
>>
>> Proposed diff:
>>
>> ntangsurat@derigible ~/git/e4r7hbug.cpython/Lib master $ git diff
>> diff --git a/Lib/mimetypes.py b/Lib/mimetypes.py
>> index 3d68694864..5919b45a9b 100644
>> --- a/Lib/mimetypes.py
>> +++ b/Lib/mimetypes.py
>> @@ -439,6 +439,7 @@ def _default_mime_types():
>>  '.jpeg'   : 'image/jpeg',
>>  '.jpg': 'image/jpeg',
>>  '.js' : 'application/javascript',
>> +'.json'   : 'application/json',
>>  '.ksh': 'text/plain',
>>  '.latex'  : 'application/x-latex',
>>  '.m1v': 'video/mpeg',
>>
>
> Feel free to file a bug at bugs.python.org and if you aren't too bothered
> then submit a PR to github.com/python/cpython (
> https://devguide.python.org/ has all the details).
>
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Mimetypes Include application/json

[Oleg Broytman]

On Wed, Aug 09, 2017 at 05:42:18PM +, "Nate."  wrote:
> A friend and I have hit a funny situation with the `mimetypes.py` library
> guessing the type for a '.json' file. Is there a reason why '.json' hasn't
> been
> added to the mapping?

   My guess is that nobody uses mimetypes without mailcap.

> Without `mailcap` installed:
> 
> [root@de169da8cc46 /]# python3 -m mimetypes build.json
> I don't know anything about type build.json
> 
> With `mailcap` installed:
> 
> [root@de169da8cc46 /]# python3 -m mimetypes build.json
> type: application/json encoding: None
> 
> We experimented with adding a mapping for '.json' to 'application/json' to
> `mimetypes.py` and it seems to work fine for us. It looks like it has been
> registered with IANA and everything.
> 
> Proposed diff:

   Patches should be published at the issue tracker.

> ntangsurat@derigible ~/git/e4r7hbug.cpython/Lib master $ git diff
> diff --git a/Lib/mimetypes.py b/Lib/mimetypes.py
> index 3d68694864..5919b45a9b 100644
> --- a/Lib/mimetypes.py
> +++ b/Lib/mimetypes.py
> @@ -439,6 +439,7 @@ def _default_mime_types():
>  '.jpeg'   : 'image/jpeg',
>  '.jpg': 'image/jpeg',
>  '.js' : 'application/javascript',
> +'.json'   : 'application/json',
>  '.ksh': 'text/plain',
>  '.latex'  : 'application/x-latex',
>  '.m1v': 'video/mpeg',
> 
> Nate.

Oleg.
-- 
 Oleg Broytmanhttp://phdru.name/p...@phdru.name
   Programmers don't die, they just GOSUB without RETURN.
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Mimetypes Include application/json

[Serhiy Storchaka]

09.08.17 21:17, Brett Cannon пише:
On Wed, 9 Aug 2017 at 10:43 Nate. 
> wrote:

A friend and I have hit a funny situation with the `mimetypes.py`
library
guessing the type for a '.json' file. Is there a reason why '.json'
hasn't been
added to the mapping?


Probably no one thought about it since the module was added in 1997 
which is only 2 years after the creation of JavaScript itself. :)


No one proposed a patch.

Feel free to file a bug at bugs.python.org  and 
if you aren't too bothered then submit a PR to github.com/python/cpython 
 (https://devguide.python.org/ has all 
the details).


https://bugs.python.org/issue30824

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Mimetypes Include application/json

[Brett Cannon]

On Wed, 9 Aug 2017 at 10:43 Nate.  wrote:

> Hi,
>
> A friend and I have hit a funny situation with the `mimetypes.py` library
> guessing the type for a '.json' file. Is there a reason why '.json' hasn't
> been
> added to the mapping?
>

Probably no one thought about it since the module was added in 1997 which
is only 2 years after the creation of JavaScript itself. :)


>
> Without `mailcap` installed:
>
> [root@de169da8cc46 /]# python3 -m mimetypes build.json
> I don't know anything about type build.json
>
> With `mailcap` installed:
>
> [root@de169da8cc46 /]# python3 -m mimetypes build.json
> type: application/json encoding: None
>
> We experimented with adding a mapping for '.json' to 'application/json' to
> `mimetypes.py` and it seems to work fine for us. It looks like it has been
> registered with IANA and everything.
>
> Proposed diff:
>
> ntangsurat@derigible ~/git/e4r7hbug.cpython/Lib master $ git diff
> diff --git a/Lib/mimetypes.py b/Lib/mimetypes.py
> index 3d68694864..5919b45a9b 100644
> --- a/Lib/mimetypes.py
> +++ b/Lib/mimetypes.py
> @@ -439,6 +439,7 @@ def _default_mime_types():
>  '.jpeg'   : 'image/jpeg',
>  '.jpg': 'image/jpeg',
>  '.js' : 'application/javascript',
> +'.json'   : 'application/json',
>  '.ksh': 'text/plain',
>  '.latex'  : 'application/x-latex',
>  '.m1v': 'video/mpeg',
>

Feel free to file a bug at bugs.python.org and if you aren't too bothered
then submit a PR to github.com/python/cpython (https://devguide.python.org/
has all the details).
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Mimetypes Include application/json

[Nate.]

Hi,

A friend and I have hit a funny situation with the `mimetypes.py` library
guessing the type for a '.json' file. Is there a reason why '.json' hasn't
been
added to the mapping?

Without `mailcap` installed:

[root@de169da8cc46 /]# python3 -m mimetypes build.json
I don't know anything about type build.json

With `mailcap` installed:

[root@de169da8cc46 /]# python3 -m mimetypes build.json
type: application/json encoding: None

We experimented with adding a mapping for '.json' to 'application/json' to
`mimetypes.py` and it seems to work fine for us. It looks like it has been
registered with IANA and everything.

Proposed diff:

ntangsurat@derigible ~/git/e4r7hbug.cpython/Lib master $ git diff
diff --git a/Lib/mimetypes.py b/Lib/mimetypes.py
index 3d68694864..5919b45a9b 100644
--- a/Lib/mimetypes.py
+++ b/Lib/mimetypes.py
@@ -439,6 +439,7 @@ def _default_mime_types():
 '.jpeg'   : 'image/jpeg',
 '.jpg': 'image/jpeg',
 '.js' : 'application/javascript',
+'.json'   : 'application/json',
 '.ksh': 'text/plain',
 '.latex'  : 'application/x-latex',
 '.m1v': 'video/mpeg',

Nate.
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Generator syntax hooks?

[Soni L.]

On 2017-08-09 11:54 AM, Nick Coghlan wrote:

On 9 August 2017 at 15:38, Guido van Rossum  wrote:

On Tue, Aug 8, 2017 at 10:06 PM, Nick Coghlan  wrote:

The OP's proposal doesn't fit into that category though: rather it's
asking about the case where we have an infinite iterator (e.g.
itertools.count(0)), and want to drop items until they start meeting
some condition (i.e. itertools.dropwhile) and then terminate the
iterator as soon as another condition is no longer met (i.e.
itertools.takewhile).

I don't think that's what the OP meant. The original proposal seemed to
assume that it would be somehow reasonable for the input ("integers" in the
example) to be able to see and parse the condition in the generator
expression ("1000 <= x < 10" in the example, with "x" somehow known to
be bound to the iteration value). That's at least what I think the remark "I
like mathy syntax" referred to.

Right, I was separating the original request to make "{x for x in
integers if 1000 <= x < 100}" work into the concrete proposal to
make exactly *that* syntax work (which I don't think is feasible), and
the slightly more general notion of offering a more math-like syntax
that allows finite sets to be built from infinite iterators by
defining a termination condition in addition to a filter condition.


Ok. A concrete proposal would give a read-only 'filter' argument to the 
iterator somehow, which represents some form of simplified AST of the 
condition.


So e.g. {x for x in integers if (lambda v: 1000 <= v < 100)(x)} 
would never complete, but {x for x in integers if 1000 <= x < 100} 
would. (But perhaps lambda objects should include an AST attribute... 
Having it for normal functions would introduce too much overhead tho, 
and then it would no longer be a simplified AST, but rather a complete 
python AST, which we don't want.)





There aren't any technical barriers I'm aware of to implementing that,
with the main historical objection being that instead of the
comprehension level while clause mapping to a while loop directly the
way the for and if clauses map to their statement level counterparts,
it would instead map to the conditional break in the expanded
loop-and-a-half form:

 while True:
 if not condition:
 break

While it's taken me a long time to come around to the idea, "Make
subtle infinite loops in mathematical code easier to avoid" *is* a
pretty compelling user-focused justification for incurring that extra
complexity at the language design level.

I haven't come around to this yet. It looks like it will make explaining
comprehensions more complex, since the translation of "while X" into "if not
X: break" feels less direct than the translations of "for x in xs" or "if
pred(x)". (In particular, your proposal seems to require more experience
with mentally translating loops and conditions into jumps -- most regulars
of this forum do that for a living, but I doubt it's second nature for the
OP.)

Yeah, if we ever did add something like this, I suspect a translation
using takewhile would potentially be easier for at least some users to
understand than the one to a break condition:

 {x for x in itertools.count(0) if 1000 <= x while x < 100}

 <=>

 x = set()
 for x in itertools.count(0):
 if 1000 <= x:
 set.add(x)
 # If you've never used the loop-and-a-half idiom, it's
 # not obvious why "while " means "if not : break"
 if not x < 100:
 break

 is roughly

 {x for x in itertools.takewhile(itertools.count(0), lambda x: x <
100) if 1000 <= x}

 <=>

 x = set()
 for x in takewhile(itertools.count(0), lambda x: x < 100):
 if 1000 <= x:
 set.add(x)

However, the break condition is the translation that would make sense
at a language *implementation* level (and would hence be the one that
determined the relative location of the while clause in the expression
form).

That discrepancy *still* sets off alarm bells for me (since it's a
clear sign that "how people would think this works" and "how it would
actually work" probably wouldn't match), I'm also conscious of the
amount of syntactic noise that "takewhile" introduces vs the "while"
keyword.

The counter-argument (which remains valid even against my own change
of heart) is that adding a new comprehension clause doesn't actually
fix the "accidental infinite loop" problem: "{x for x in
itertools.count(0) if 1000 <= x < 100}" will still loop forever,
it would just have a nicer fix to get it to terminate (adding " while
x" to turn the second filter condition into a termination condition).

So while I'm +0 where I used to be a firm -1, it's still only a +0 :)

Cheers,
Nick.



___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: 

Re: [Python-ideas] Generator syntax hooks?

[Nick Coghlan]

On 9 August 2017 at 15:38, Guido van Rossum  wrote:
> On Tue, Aug 8, 2017 at 10:06 PM, Nick Coghlan  wrote:
>> The OP's proposal doesn't fit into that category though: rather it's
>> asking about the case where we have an infinite iterator (e.g.
>> itertools.count(0)), and want to drop items until they start meeting
>> some condition (i.e. itertools.dropwhile) and then terminate the
>> iterator as soon as another condition is no longer met (i.e.
>> itertools.takewhile).
>
> I don't think that's what the OP meant. The original proposal seemed to
> assume that it would be somehow reasonable for the input ("integers" in the
> example) to be able to see and parse the condition in the generator
> expression ("1000 <= x < 10" in the example, with "x" somehow known to
> be bound to the iteration value). That's at least what I think the remark "I
> like mathy syntax" referred to.

Right, I was separating the original request to make "{x for x in
integers if 1000 <= x < 100}" work into the concrete proposal to
make exactly *that* syntax work (which I don't think is feasible), and
the slightly more general notion of offering a more math-like syntax
that allows finite sets to be built from infinite iterators by
defining a termination condition in addition to a filter condition.

>> There aren't any technical barriers I'm aware of to implementing that,
>> with the main historical objection being that instead of the
>> comprehension level while clause mapping to a while loop directly the
>> way the for and if clauses map to their statement level counterparts,
>> it would instead map to the conditional break in the expanded
>> loop-and-a-half form:
>>
>> while True:
>> if not condition:
>> break
>>
>> While it's taken me a long time to come around to the idea, "Make
>> subtle infinite loops in mathematical code easier to avoid" *is* a
>> pretty compelling user-focused justification for incurring that extra
>> complexity at the language design level.
>
> I haven't come around to this yet. It looks like it will make explaining
> comprehensions more complex, since the translation of "while X" into "if not
> X: break" feels less direct than the translations of "for x in xs" or "if
> pred(x)". (In particular, your proposal seems to require more experience
> with mentally translating loops and conditions into jumps -- most regulars
> of this forum do that for a living, but I doubt it's second nature for the
> OP.)

Yeah, if we ever did add something like this, I suspect a translation
using takewhile would potentially be easier for at least some users to
understand than the one to a break condition:

{x for x in itertools.count(0) if 1000 <= x while x < 100}

<=>

x = set()
for x in itertools.count(0):
if 1000 <= x:
set.add(x)
# If you've never used the loop-and-a-half idiom, it's
# not obvious why "while " means "if not : break"
if not x < 100:
break

is roughly

{x for x in itertools.takewhile(itertools.count(0), lambda x: x <
100) if 1000 <= x}

<=>

x = set()
for x in takewhile(itertools.count(0), lambda x: x < 100):
if 1000 <= x:
set.add(x)

However, the break condition is the translation that would make sense
at a language *implementation* level (and would hence be the one that
determined the relative location of the while clause in the expression
form).

That discrepancy *still* sets off alarm bells for me (since it's a
clear sign that "how people would think this works" and "how it would
actually work" probably wouldn't match), I'm also conscious of the
amount of syntactic noise that "takewhile" introduces vs the "while"
keyword.

The counter-argument (which remains valid even against my own change
of heart) is that adding a new comprehension clause doesn't actually
fix the "accidental infinite loop" problem: "{x for x in
itertools.count(0) if 1000 <= x < 100}" will still loop forever,
it would just have a nicer fix to get it to terminate (adding " while
x" to turn the second filter condition into a termination condition).

So while I'm +0 where I used to be a firm -1, it's still only a +0 :)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Argparse argument deprecation

[Tarek Ziadé]

> Another note about the proposal: calling it "deprecated" seems odd,
> since the proposal is really just a general-purpose callback.  argparse
> isn't generating the warning, your callback function would be doing it. 
> Why name it "deprecated"?  How is this different than the "action"
> keyword argument that argparse already provides?

That sounds right. Maybe a better implementation would be  to implement
a custom action by inheriting from argparse.Action

https://docs.python.org/3/library/argparse.html#action

and do all the warning/deprecation job there.

I'll experiment with this idea on my side to see how it goes :)

Cheers
Tarek
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Argparse argument deprecation

[Michel Desmoulin]

Le 09/08/2017 à 12:59, Ned Batchelder a écrit :
> OK, then on a more pragmatic note: why is it easier to write a callback
> than to write a simple if statement after the parsing?  Generating help
> is complex, and a common task that is closely tied to the syntax of the
> options, so it makes sense for argparse to do it.  Deprecation is
> neither complex, common, nor closely tied to the syntax of the options.
> 
> Another note about the proposal: calling it "deprecated" seems odd,
> since the proposal is really just a general-purpose callback.  argparse
> isn't generating the warning, your callback function would be doing it. 
> Why name it "deprecated"?  How is this different than the "action"
> keyword argument that argparse already provides?

I imagine something like:

def _(warn, forbid):
warn('This is deprecated') # for forbid to just put an error

parser.add_option(on_deprecated=deprecationCallback)

This does:

- provide an easy way to warn, or transition to forbid
- allow introspection to list the deprecated options
- deprecated options can be marked as such in the generated --help
- create a complex dynamic deprecation message, or just pass a short lambda

But indeed I'd like it to be able to do:

parser.add_option(on_deprecated=DeprecationWarning('meh'))
parser.add_option(on_deprecated=ValueError('meh'))

As a shortcut for simple use cases.

I still don't know how to make the distinction between deprecated and
removed from the introspection point of view.

All in all, I think it's an interesting proposal, but I'm not going to
fight over it. If it never happens, I can fit a bunch of "if" like you said.

> 
> --Ned.
> 
> 
> On 8/9/17 5:54 AM, Michel Desmoulin wrote:
>> Argparse is not just about parsing, it's about providing convenient
>> tooling associated with parsing.
>>
>> Otherwise you would not have automatically generated a "usage" message
>> or a "--help" command.
>>
>> Following your definition, those are not parsing. But there are here,
>> because we all end up coding them anyway.
>>
>> Le 09/08/2017 à 11:50, Ned Batchelder a écrit :
>>> On 8/9/17 3:56 AM, Tarek Ziadé wrote:
 Hey,

 I don't think there's any helper to deprecate an argument in argparse

 Let's say you have a --foo option in your CLI and want to deprecate it
 in the next release before you completely remove it later.

 My first though on how to do this by adding a new "deprecated" option to
 https://docs.python.org/3/library/argparse.html#argparse.ArgumentParser.add_argument

 "deprecated" would be a callable that is called after the argument has
 been parsed by argparse, 
 so the developer can decide if they want to issue a deprecation warning,
 use the parsed value or override it etc.
>>> I don't see why this is something that argparse has to do.  The
>>> semantics of options is handled by the rest of the program.  Why would
>>> the parser be issuing these warnings?  Let argparse parse the options,
>>> then let other code deal with what they *mean*.
>>>
>>> --Ned.
>>>
>>> ___
>>> Python-ideas mailing list
>>> Python-ideas@python.org
>>> https://mail.python.org/mailman/listinfo/python-ideas
>>> Code of Conduct: http://python.org/psf/codeofconduct/
>>>
>> ___
>> Python-ideas mailing list
>> Python-ideas@python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
> 
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
> 
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Argparse argument deprecation

[Ned Batchelder]

OK, then on a more pragmatic note: why is it easier to write a callback
than to write a simple if statement after the parsing?  Generating help
is complex, and a common task that is closely tied to the syntax of the
options, so it makes sense for argparse to do it.  Deprecation is
neither complex, common, nor closely tied to the syntax of the options.

Another note about the proposal: calling it "deprecated" seems odd,
since the proposal is really just a general-purpose callback.  argparse
isn't generating the warning, your callback function would be doing it. 
Why name it "deprecated"?  How is this different than the "action"
keyword argument that argparse already provides?

--Ned.


On 8/9/17 5:54 AM, Michel Desmoulin wrote:
> Argparse is not just about parsing, it's about providing convenient
> tooling associated with parsing.
>
> Otherwise you would not have automatically generated a "usage" message
> or a "--help" command.
>
> Following your definition, those are not parsing. But there are here,
> because we all end up coding them anyway.
>
> Le 09/08/2017 à 11:50, Ned Batchelder a écrit :
>> On 8/9/17 3:56 AM, Tarek Ziadé wrote:
>>> Hey,
>>>
>>> I don't think there's any helper to deprecate an argument in argparse
>>>
>>> Let's say you have a --foo option in your CLI and want to deprecate it
>>> in the next release before you completely remove it later.
>>>
>>> My first though on how to do this by adding a new "deprecated" option to
>>> https://docs.python.org/3/library/argparse.html#argparse.ArgumentParser.add_argument
>>>
>>> "deprecated" would be a callable that is called after the argument has
>>> been parsed by argparse, 
>>> so the developer can decide if they want to issue a deprecation warning,
>>> use the parsed value or override it etc.
>> I don't see why this is something that argparse has to do.  The
>> semantics of options is handled by the rest of the program.  Why would
>> the parser be issuing these warnings?  Let argparse parse the options,
>> then let other code deal with what they *mean*.
>>
>> --Ned.
>>
>> ___
>> Python-ideas mailing list
>> Python-ideas@python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Argparse argument deprecation

[Ned Batchelder]

On 8/9/17 3:56 AM, Tarek Ziadé wrote:
> Hey,
>
> I don't think there's any helper to deprecate an argument in argparse
>
> Let's say you have a --foo option in your CLI and want to deprecate it
> in the next release before you completely remove it later.
>
> My first though on how to do this by adding a new "deprecated" option to
> https://docs.python.org/3/library/argparse.html#argparse.ArgumentParser.add_argument
>
> "deprecated" would be a callable that is called after the argument has
> been parsed by argparse, 
> so the developer can decide if they want to issue a deprecation warning,
> use the parsed value or override it etc.

I don't see why this is something that argparse has to do.  The
semantics of options is handled by the rest of the program.  Why would
the parser be issuing these warnings?  Let argparse parse the options,
then let other code deal with what they *mean*.

--Ned.

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Argparse argument deprecation

[Michel Desmoulin]

+1, but I would make "deprecated" either a warning, an exception or a
callable.

This way to create a simple deprecation, you just provide
DeprecationWarning('This will be gone in the next release'), or
ValueError('This has been removed in 2.X, use "stuff instead"') if you
decide it's gone for good. But if you need a custom behavior, you pass
in a callable.

Le 09/08/2017 à 09:56, Tarek Ziadé a écrit :
> Hey,
> 
> I don't think there's any helper to deprecate an argument in argparse
> 
> Let's say you have a --foo option in your CLI and want to deprecate it
> in the next release before you completely remove it later.
> 
> My first though on how to do this by adding a new "deprecated" option to
> https://docs.python.org/3/library/argparse.html#argparse.ArgumentParser.add_argument
> 
> "deprecated" would be a callable that is called after the argument has
> been parsed by argparse, 
> so the developer can decide if they want to issue a deprecation warning,
> use the parsed value or override it etc.
> 
> Another interesting approach suggest by Doug Hellman, which I like as
> much, is a set of higher level options that
> provide a deprecation workflow for arguments, see
> 
> https://github.com/openstack/oslo.config/blob/master/oslo_config/cfg.py#L441
> 
> What do you think?
> 
> Cheers
> Tarek
> 
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Pseudo methods

[Stephen J. Turnbull]

Nick Coghlan writes:

 > To analyse and investigate this code, we need to "just know" that:

You can of course hope that help(input().has_vowels) will tell you
where to find it.  If it doesn't, well, shame on you for depending on
source-unavailable software that you don't understand. ;-)

I'm with you on implementing this feature; I don't like it.  But I
don't think the discoverability situation is as dire as you suggest.


___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Generator syntax hooks?

[Stephen J. Turnbull]

Nick Coghlan writes:

 > Right now, getting the "terminate when false" behaviour requires the
 > use of takewhile:
 > 
 > {itertools.takewhile(lambda x: x < 100, itertools.count(1000)}

My objection to this interpretation is different from Guido's (I
think): if you're really thinking in terms of math, sets are
*unordered*, and therefore "takewhile" doesn't guarantee exhaustion of
the desired subset.  Another way to put this is that in order to make
it harder to get bit by subtle infloops, you're going to give more
teeth to "Miller time came early"[1] bugs.

This may be a bigger issue than some may think, because sets and dicts
are iterable, and order of iteration is arbitrary (at best history-
dependent).


Footnotes: 
[1]  American beer commercial claiming that real men go to drink beer
after a full day's work.

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Argparse argument deprecation

[Tarek Ziadé]

Hey,

I don't think there's any helper to deprecate an argument in argparse

Let's say you have a --foo option in your CLI and want to deprecate it
in the next release before you completely remove it later.

My first though on how to do this by adding a new "deprecated" option to
https://docs.python.org/3/library/argparse.html#argparse.ArgumentParser.add_argument

"deprecated" would be a callable that is called after the argument has
been parsed by argparse, 
so the developer can decide if they want to issue a deprecation warning,
use the parsed value or override it etc.

Another interesting approach suggest by Doug Hellman, which I like as
much, is a set of higher level options that
provide a deprecation workflow for arguments, see

https://github.com/openstack/oslo.config/blob/master/oslo_config/cfg.py#L441

What do you think?

Cheers
Tarek

-- 

Tarek Ziadé | coding: https://ziade.org | running: https://foule.es |
twitter: @tarek_ziade
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/