[Python-Dev] Re: PEP 617: New PEG parser for CPython

[Matt Billenstein via Python-Dev]

On Thu, Apr 02, 2020 at 08:57:30PM -0700, Guido van Rossum wrote:
> On Thu, Apr 2, 2020 at 7:55 PM Matt Billenstein  wrote:
> 
> Even just running it in a dev build against the corpus of the top few
> thousand packages on pypi might give enough confidence -- I had a script
> to download the top N packages and run some script over the python files
> contained therein, but I can't seem to find it atm.
> 
> 
> We got that. Check https://github.com/gvanrossum/pegen/tree/master/scripts --
> look at download_pypi_packages.py and test_pypi_packages.py.

Very nice!

m

-- 
Matt Billenstein
m...@vazor.com
http://www.vazor.com/
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/Q5DY4ZNCB7GYGIL5LUWJFJ7GLL5EJMW2/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 617: New PEG parser for CPython

[Guido van Rossum]

On Thu, Apr 2, 2020 at 7:55 PM Matt Billenstein  wrote:

> On Thu, Apr 02, 2020 at 05:17:31PM -0700, Guido van Rossum wrote:
> > On Thu, Apr 2, 2020 at 4:20 PM Nathaniel Smith  wrote:
> >
> > If the AST is supposed to be the same, then would it make sense to
> > temporarily – maybe just during the alpha/beta period – always run
> > *both* parsers and confirm that they match?
> >
> >
> > That's not a bad idea!
> https://github.com/we-like-parsers/cpython/issues/33
>
> Even just running it in a dev build against the corpus of the top few
> thousand packages on pypi might give enough confidence -- I had a script
> to download the top N packages and run some script over the python files
> contained therein, but I can't seem to find it atm.
>

We got that. Check https://github.com/gvanrossum/pegen/tree/master/scripts
-- look at download_pypi_packages.py and test_pypi_packages.py.

-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/P7CPMHFQBTS3SH2A6TOUJNTZI2F25JFH/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 617: New PEG parser for CPython

[Matt Billenstein via Python-Dev]

On Thu, Apr 02, 2020 at 05:17:31PM -0700, Guido van Rossum wrote:
> On Thu, Apr 2, 2020 at 4:20 PM Nathaniel Smith  wrote:
> 
> If the AST is supposed to be the same, then would it make sense to
> temporarily – maybe just during the alpha/beta period – always run
> *both* parsers and confirm that they match?
> 
> 
> That's not a bad idea! https://github.com/we-like-parsers/cpython/issues/33

Even just running it in a dev build against the corpus of the top few
thousand packages on pypi might give enough confidence -- I had a script
to download the top N packages and run some script over the python files
contained therein, but I can't seem to find it atm.

m

-- 
Matt Billenstein
m...@vazor.com
http://www.vazor.com/
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/YCL5HQIV5YIRPZ5VCKV6B7U5XCECFAW2/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 617: New PEG parser for CPython

[Greg Ewing]

On 3/04/20 3:22 pm, Guido van Rossum wrote:
This allows more freedom in designing a grammar. 
For example, it would let a language designer solve the "dangling else" 
problem from the Wikipedia page, by writing the form including the 
"else" clause first .


I'm inclined to think that such problems shouldn't be solved at the
parser level, but rather at the language level, i.e. don't design
the language that way in the first place. After all, if it's
confusing to the computer, it's probably going to be confusing
to humans as well.

(I note that all of Wirth's languages after Pascal changed the syntax
so as not to have a dangling else problem.)

Personally I would rather my parser generator *did* complain about
ambiguities, so that I can facepalm myself for designing my language
in such a stupid way.

--
Greg
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/MKMIDLTI3U7WKJ6PPMHNRQBVZINCJJAQ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 617: New PEG parser for CPython

[Greg Ewing]

On 3/04/20 2:13 pm, Victor Stinner wrote:

"Unlike LL(1) parsers PEG-based parsers cannot be ambiguous: if a
string parses, it has exactly one valid parse tree. This means that a
PEG-based parser cannot suffer from the ambiguity problems described
in the previous section."


That paragraph seems rather confused. I think what it *might* be
trying to say is that a PEG parser allows you to write productions
with overlapping first sets (which would be "ambiguous" for an
LL parser), but still somehow guarantees that a unique parse tree
is produced. The latter suggests that the grammar as a whole still
needs to be unambiguous.

--
Greg
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/K7LH4VHVOZ5ISIUTJ3I7UWEVM3KGK5Y6/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 617: New PEG parser for CPython

[Guido van Rossum]

 Le ven. 3 avr. 2020 à 02:58, Greg Ewing  a
écrit :
> On 3/04/20 10:33 am, Victor Stinner wrote:
> > I also like the fact that PEG is deterministic, whereas
> > LL(1) parsers are not.
>
> Where do you get that LL(1) parsers are not deterministic?
> That's news to me!

On Thu, Apr 2, 2020 at 6:15 PM Victor Stinner  wrote:

> Sorry, I was referring to *ambiguous* grammar rules. Extract of the PEP:
>
> "Unlike LL(1) parsers PEG-based parsers cannot be ambiguous: if a
> string parses, it has exactly one valid parse tree. This means that a
> PEG-based parser cannot suffer from the ambiguity problems described
> in the previous section."
>

Maybe we need to rephrase this a bit. It's more that the LL(1) and PEG
formalisms deal very different with ambiguous *grammars*. An example of an
ambiguous grammar would be:

start: X | Y
X: expr
Y: expr
expr: NAME | NAME '+' NAME

There are probably better examples of ambiguous grammars (see
https://en.wikipedia.org/wiki/Ambiguous_grammar) but I think this will do
to explain the problem.

This is a fine context-free grammar (it accepts strings like "a" and "a+b")
but the LL(1) formalism will reject it because it sees an overlap in FIRST
sets between X and Y -- not surprising because they have the same RHS.
Also, even a more powerful formalism would have to make a choice whether to
choose X or Y, which may matter if the derivation is used to build a parse
tree (like Python's pgen does).

OTOH a PEG parser generator will always take the X alternative -- it
doesn't care that there's more than one derivation, since its '|' operator
is not symmetrical: X|Y and Y|X are not the same, as they are in LL(1) and
most other formalisms. (In fact, the common notation for PEG uses '/' to
emphasize this, but it looks ugly to me so I changed it to '|'.)

That PEG (by definition) always uses the first matching alternative is
actually a blessing as well as a curse. The downside is that PEG can't tell
you when have a real ambiguity in your grammar. But the upside is that it
works like a programmer would write a (recursive descent) parser. Thus it
"solves" the problem of ambiguous grammars by choosing the first
alternative. This allows more freedom in designing a grammar. For example,
it would let a language designer solve the "dangling else" problem from the
Wikipedia page, by writing the form including the "else" clause first .
(Python doesn't have that problem due to the use of indentation, but it
might appear in another disguise.)

I should probably refine this argument and include it in the PEP as one of
the reasons to prefer PEG over LR or LALR (but I need to think more about
that -- it was a very early choice).

-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/3LHZNXC2MJJ2RUMDGRH7L2ZOPH5ZE6QK/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 617: New PEG parser for CPython

[Batuhan Taskaya]

Wonderful news! I'm really excited to see what is coming alongside
this flexible parser.

On Thu, Apr 2, 2020 at 9:16 PM Guido van Rossum  wrote:
>
> Since last fall's core sprint in London, Pablo Galindo Salgado, Lysandros 
> Nikolaou and myself have been working on a new parser for CPython. We are now 
> far enough along that we present a PEP we've written:
>
> https://www.python.org/dev/peps/pep-0617/
>
> Hopefully the PEP speaks for itself. We are hoping for a speedy resolution so 
> we can land the code we've written before 3.9 beta 1.
>
> If people insist I can post a copy of the entire PEP here on the list, but 
> since a lot of it is just background information on the old LL(1) and the new 
> PEG parsing algorithms, I figure I'd spare everyone the need of reading 
> through that. Below is a copy of the most relevant section from the PEP. I'd 
> also like to point out the section on performance (which you can find through 
> the above link) -- basically performance is on a par with that of the old 
> parser.
>
> ==
> Migration plan
> ==
>
> This section describes the migration plan when porting to the new PEG-based 
> parser
> if this PEP is accepted. The migration will be executed in a series of steps 
> that allow
> initially to fallback to the previous parser if needed:
>
> 1.  Before Python 3.9 beta 1, include the new PEG-based parser machinery in 
> CPython
> with a command-line flag and environment variable that allows switching 
> between
> the new and the old parsers together with explicit APIs that allow 
> invoking the
> new and the old parsers independently. At this step, all Python APIs like 
> ``ast.parse``
> and ``compile`` will use the parser set by the flags or the environment 
> variable and
> the default parser will be the current parser.
>
> 2.  After Python 3.9 Beta 1 the default parser will be the new parser.
>
> 3.  Between Python 3.9 and Python 3.10, the old parser and related code (like 
> the
> "parser" module) will be kept until a new Python release happens (Python 
> 3.10). In
> the meanwhile and until the old parser is removed, **no new Python Grammar
> addition will be added that requires the peg parser**. This means that 
> the grammar
> will be kept LL(1) until the old parser is removed.
>
> 4.  In Python 3.10, remove the old parser, the command-line flag, the 
> environment
> variable and the "parser" module and related code.
>
> --
> --Guido van Rossum (python.org/~guido)
> Pronouns: he/him (why is my pronoun here?)
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-dev@python.org/message/HOZ2RI3FXUEMAT4XAX4UHFN4PKG5J5GR/
> Code of Conduct: http://python.org/psf/codeofconduct/
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/PIR74ZYSBM46TW3OZYGPFEBV4I4BZ5MN/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 617: New PEG parser for CPython

[Victor Stinner]

Sorry, I was referring to *ambiguous* grammar rules. Extract of the PEP:

"Unlike LL(1) parsers PEG-based parsers cannot be ambiguous: if a
string parses, it has exactly one valid parse tree. This means that a
PEG-based parser cannot suffer from the ambiguity problems described
in the previous section."

Victor

Le ven. 3 avr. 2020 à 02:58, Greg Ewing  a écrit :
>
> On 3/04/20 10:33 am, Victor Stinner wrote:
> > I also like the fact that PEG is deterministic, whereas
> > LL(1) parsers are not.
>
> Where do you get that LL(1) parsers are not deterministic?
> That's news to me!
>
> --
> Greg
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-dev@python.org/message/JHHLCUF7APZ5BV7C5NUNPPWI264L5XSJ/
> Code of Conduct: http://python.org/psf/codeofconduct/



-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/KGKPKUOG54AD555K7V65XADXWK7MFY4X/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 617: New PEG parser for CPython

[Greg Ewing]

On 3/04/20 10:33 am, Victor Stinner wrote:

I also like the fact that PEG is deterministic, whereas
LL(1) parsers are not.


Where do you get that LL(1) parsers are not deterministic?
That's news to me!

--
Greg
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/JHHLCUF7APZ5BV7C5NUNPPWI264L5XSJ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 617: New PEG parser for CPython

[Greg Ewing]

On 3/04/20 7:10 am, Guido van Rossum wrote:
Since last fall's core sprint in London, Pablo Galindo Salgado, 
Lysandros Nikolaou and myself have been working on a new parser for 
CPython. We are now far enough along that we present a PEP we've written:


https://www.python.org/dev/peps/pep-0617/


Was any consideration given to other types of parser, such
as LR or LALR?

LR parsers handle left recursion naturally, and don't suffer
from any of the drawbacks mentioned in the PEP such as taking
exponential time or requiring all the source to be loaded
into memory.

I think there needs to be a section in the PEP justifying the
choice of PEG over the alternatives.

--
Greg
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/LR6GQNAHMWDN3BZCXEZFSEU2FTK64MS5/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 617: New PEG parser for CPython

[Guido van Rossum]

On Thu, Apr 2, 2020 at 4:20 PM Nathaniel Smith  wrote:

> On Thu, Apr 2, 2020 at 2:48 PM Pablo Galindo Salgado
>  wrote:
> >
> > > About the migration, can I ask who is going to (help to) fix projects
> > which rely on the AST?
> >
> > I think you misunderstood: The AST is exactly the same as the old and
> the new parser. The only
> > the thing that the new parser does is not generate an immediate CST
> (Concrete Syntax Tree) and that
> > is only half-exposed in the parser module.
>
> If the AST is supposed to be the same, then would it make sense to
> temporarily – maybe just during the alpha/beta period – always run
> *both* parsers and confirm that they match?
>

That's not a bad idea! https://github.com/we-like-parsers/cpython/issues/33

-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/TV2C2SVAUEWJ4ICSQ7TLFN3M74Y5W32M/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 617: New PEG parser for CPython

[Guido van Rossum]

On Thu, Apr 2, 2020 at 2:55 PM Pablo Galindo Salgado 
wrote:

> > About the migration, can I ask who is going to (help
> to) fix projects which rely on the AST?
>
> Whoops, I send the latest email before finishing it by mistake. Here is
> the extended version of the answer:
>
> I think there is a misunderstanding here: The new parser generates the
> same AST as the old parser so
> calling ast.parse() or compile() will yield exactly the same result. We
> have extensive testing around that
> and that was a goal from the beginning. Projects using the ast module will
> not need to do anything special.
>

I think that's only half true. It's true if they already work with Python
3.9 (master/HEAD). But probably some of these packages have not yet started
testing with 3.9 nightly runs or even alphas, so it's at least
*conceivable* that some of the fixes we applied to the AST could require
(small) adjustments. And I think *that* was what Victor was referring to.
(For example, I'm not 100% sure that mypy actually works with the latest
3.9. But there seems to be something else wrong there so I can't even test
it.)


> The difference is that the new parser does not generate a CST (Concrete
> Syntax Tree). The concrete syntax tree
> is an immediate structure from where the AST is generated. This structure
> is only partially exposed via the
> "parser" module but otherwise is only used in the parser itself so it
> should not be a problem. On the other hand:
> as explained in the PEP, the lack of the CST greatly simplifies the AST
> generation among other advantages.
>

I just remembered another difference. We haven't really investigated how
good the error reporting is. I'm sure there are cases where the syntax
error points at a *slightly* different position -- sometimes it's a bit
better, sometimes a bit worse. But there could be cases where the PEG
parser reads ahead chasing some alternative that will fail much later, and
then it would be much worse. We should probably explore this.

-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/GUDSLUXXKVGXIO4FQNUW6U6YMYKZLG3W/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 617: New PEG parser for CPython

[Nathaniel Smith]

On Thu, Apr 2, 2020 at 2:48 PM Pablo Galindo Salgado
 wrote:
>
> > About the migration, can I ask who is going to (help to) fix projects
> which rely on the AST?
>
> I think you misunderstood: The AST is exactly the same as the old and the new 
> parser. The only
> the thing that the new parser does is not generate an immediate CST (Concrete 
> Syntax Tree) and that
> is only half-exposed in the parser module.

If the AST is supposed to be the same, then would it make sense to
temporarily – maybe just during the alpha/beta period – always run
*both* parsers and confirm that they match?

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/IRQUITYBQFJYUFQRKTVDXUBX4X42ARMP/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 616 -- String methods to remove prefixes and suffixes

[Nick Coghlan]

On Thu., 2 Apr. 2020, 8:30 am Victor Stinner,  wrote:

> I suggest you to wait one more week to let other people comment the
> PEP. After this delay, if you consider that the PEP is ready for
> pronouncement, you can submit it to the Steering Council, right.
>

Note that the submission to the Steering Council doesn't have to be a
request for immediate pronouncement - it's a notification that the PEP is
mature enough for the Council to decide whether to appoint a Council member
as BDFL-Delegate or to appoint someone else.

The decision on whether to wait for more questions is then up to the
Council and/or the appointed BDFL-Delegate.

PEP 616 definitely looks mature enough for that step to me (and potentially
even immediately accepted - it did get dissected pretty thoroughly, after
all!)

Cheers,
Nick.



> http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/4MJAT2DJAWRSZCG465QQOQRSG3NUIH7D/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 617: New PEG parser for CPython

[Guido van Rossum]

On Thu, Apr 2, 2020 at 12:43 PM Paul Moore  wrote:

> On Thu, 2 Apr 2020 at 19:20, Guido van Rossum  wrote:
> >
> > Since last fall's core sprint in London, Pablo Galindo Salgado,
> Lysandros Nikolaou and myself have been working on a new parser for
> CPython. We are now far enough along that we present a PEP we've written:
> >
> > https://www.python.org/dev/peps/pep-0617/
> >
> > Hopefully the PEP speaks for itself. We are hoping for a speedy
> resolution so we can land the code we've written before 3.9 beta 1.
>
> Excellent news! One question - will there be any user-visible change
> as a result of this PEP other than the removal of the "parser" module?
> From my quick reading of the PEP, I didn't see anything, so I assume
> the answer is "no".
>

I suppose it depends on how deep you dig, but the intention is that the
returned AST is identical in each case. (We've "cheated" a bit by making a
few small changes to the code that produces an AST for the old parser,
mostly bugs related to line/column numbers.)

-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/7CYNFY73LAIPF5L4BYHOX6KVIWLPCTW6/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 616 -- String methods to remove prefixes and suffixes

[Nick Coghlan]

On Sat., 28 Mar. 2020, 8:39 am Guido van Rossum,  wrote:

> On Fri, Mar 27, 2020 at 3:29 PM Dennis Sweeney <
> sweeney.dennis...@gmail.com> wrote:
>
>> > If I saw that in a code review I'd flag it for non-obviousness. One
>> should
>> > use 'string != new_string' unless there is severe pressure to squeeze
>> > every nanosecond out of this particular code (and it better be inside an
>> > inner loop).
>>
>> I thought that someone had suggested that such things go in the PEP,
>
>
> I'm sure someone did.
>

I think that may have been me in a tangent thread where folks were worried
about O(N) checks on long strings.

I know at least I temporarily forgot to account for string equality checks
starting with a few O(1) checks to speed up common cases (IIRC: identity,
length, first code point, last code point), which means explicitly calling
len() is just as likely to slow things down as it is to speed them up.

Cheers,
Nick.


>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/UYA5ICAM6TWREXS7SSA4WKWRA2DAQOMF/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 617: New PEG parser for CPython

[Guido van Rossum]

On Thu, Apr 2, 2020 at 1:21 PM Barry Warsaw  wrote:

> On Apr 2, 2020, at 13:07, Pablo Galindo Salgado 
> wrote:
> >
> >> Just to clarify, this means that 3.9 will ship with the PEG parser as
> default,
> >> right?  If so, this would be a new feature, post beta.  Since that is
> counter to our
> >> general policy, we would need to get explicit RM approval for such a
> change.
> >
> > The idea is merging it *before beta* and make it the default *from the
> beta* (maybe
> > merging that commit making it the default the day before the beta for
> instance).
>
> Yes, sorry I don’t mean to be pedantic, but it would be better policy-wise
> if the new parser were the default in time for the first beta rather than
> after.
>

That was the intention, i.e. releasing beta 1 with the new parser being the
default. The current wording in the PEP are wrong, we'll fix that.

-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/N4QU7AYS7WCSAIQJY7YB3AQBBPLYHEVO/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Debug C++

[Antonio Cavallo]

You can also attach gdb to a running (python) process .. but the
interpreter needs to have the symbol table (eg. not stripped).

I gave a presentation few years back on this exact topic (sorry for the
quality, I was very in-experienced).
  https://skillsmatter.com/skillscasts/4313-intro-to-python-debug

I hope this helps.

PS> I'm working on a tiny "relocatable" python interpreter + toolchain
using docker (sort of miniconda but for plain python)
https://bitbucket.org/cav71/builder3/src/master that might be useful.

On Mon, 30 Mar 2020 at 20:07, Rhodri James  wrote:

> On 30/03/2020 23:20, Leandro Müller wrote:
> > Hi
> > Are there any away to debug C module during python debug?
>
> In a word, gdb.
>
> I've been doing this quite a lot lately.  The trick is to start Python
> up under gdb, set a pending breakpoint in your C module, then carry on
> as normal.  For example:
>
>
> rhodri@Wildebeest:~/hub_module$ gdb python3
> GNU gdb (Ubuntu 8.1-0ubuntu3.2) 8.1.0.20180409-git
> Copyright (C) 2018 Free Software Foundation, Inc.
>
> [...large amounts of boilerplate omitted for brevity...]
>
> Reading symbols from python3...Reading symbols from
>
> /usr/lib/debug/.build-id/28/7763e881de67a59b31b452dd0161047f7c0135.debug...done.
> done.
> (gdb) b Hub_new
> Function "Hub_new" not defined.
> Make breakpoint pending on future shared library load? (y or [n]) y
> Breakpoint 1 (Hub_init) pending.
> (gdb) run
> Starting program: /home/rhodri/Work/Lego/hub_module/hat_env/bin/python3
> [Thread debugging using libthread_db enabled]
> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
> Python 3.6.9 (default, Nov  7 2019, 10:44:02)
> [GCC 8.3.0] on linux
> Type "help", "copyright", "credits" or "license" for more information.
>  >>> from hub import hub
>
> Breakpoint 1, Hub_new (type=0x76236340 , args=(), kwds=0x0)
>  at src/hubmodule.c:164
> 164 HubObject *self = (HubObject *)type->tp_alloc(type, 0);
> (gdb)
>
>
> ...and off you go!
>
>
> --
> Rhodri James *-* Kynesim Ltd
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/4JVUDRD5YZVVZ4FWC3BYN3RJDXBNBTIF/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/N5AEUON35RJXJJ5ENXCXVXYAJXM5TIMO/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 617: New PEG parser for CPython

[Pablo Galindo Salgado]

> About the migration, can I ask who is going to (help
to) fix projects which rely on the AST?

Whoops, I send the latest email before finishing it by mistake. Here is the 
extended version of the answer:

I think there is a misunderstanding here: The new parser generates the same AST 
as the old parser so
calling ast.parse() or compile() will yield exactly the same result. We have 
extensive testing around that
and that was a goal from the beginning. Projects using the ast module will not 
need to do anything special.

The difference is that the new parser does not generate a CST (Concrete Syntax 
Tree). The concrete syntax tree
is an immediate structure from where the AST is generated. This structure is 
only partially exposed via the
"parser" module but otherwise is only used in the parser itself so it should 
not be a problem. On the other hand:
as explained in the PEP, the lack of the CST greatly simplifies the AST 
generation among other advantages.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/52HCDM3QUKQLKKADB4Z3ILP5DN4VTQM2/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 617: New PEG parser for CPython

[Pablo Galindo Salgado]

> About the migration, can I ask who is going to (help to) fix projects
which rely on the AST?

I think you misunderstood: The AST is exactly the same as the old and the new 
parser. The only
the thing that the new parser does is not generate an immediate CST (Concrete 
Syntax Tree) and that
is only half-exposed in the parser module.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/CHH4RV2GAGMMCL3RGUQ4BX6MJQIU6PKS/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 617: New PEG parser for CPython

[Victor Stinner]

Hi,

It's great to see that you finally managed to come up with a PEP, this
work becomes concrete: congrats!

I started to read the PEP, and it's really well written! I heard that
LL(1) parsers have limits, but this PEP explains very well that the
current Python grammar was already "hacked" to work around these
limitations. I also like the fact that PEG is deterministic, whereas
LL(1) parsers are not.

I like to have the new parser being the default, it will ease its
adoption and force users to adapt their code. Otherwise, the migration
may take forever and never complete :-(

--

About the migration, can I ask who is going to (help to) fix projects
which rely on the AST?

I know that the motto was always "we don't provide any backward
compatibility warranty on the AST", *but* more and more projects are
using the Python AST.

Examples of projects relying on the AST:

* gast: used by Pythran
* pylint uses astroid
* Chameleon
* Genshi
* Mako
* pyflakes
* (likely others)

I'm not asking to stop making AST changes. I'm following AST changes,
and the AST is becoming better and better at each Python release!

I'm just asking is there are volunteers around to help to make these
projects compatible with Python 3.9, before the Python 3.9.0 final
release (to accelerate the adoption of Python 3.9). These volunteers
don't have to be the ones behind the PEP 617.

Note: example of previous AST incompatible changes (use ast.Constant,
remove old AST classes) in Python 3.8:
https://bugs.python.org/issue32892 A compatibility layer was added to
ease the migration from old AST classes to the new ast.Constant.

Victor

Le jeu. 2 avr. 2020 à 20:15, Guido van Rossum  a écrit :
>
> Since last fall's core sprint in London, Pablo Galindo Salgado, Lysandros 
> Nikolaou and myself have been working on a new parser for CPython. We are now 
> far enough along that we present a PEP we've written:
>
> https://www.python.org/dev/peps/pep-0617/
>
> Hopefully the PEP speaks for itself. We are hoping for a speedy resolution so 
> we can land the code we've written before 3.9 beta 1.
>
> If people insist I can post a copy of the entire PEP here on the list, but 
> since a lot of it is just background information on the old LL(1) and the new 
> PEG parsing algorithms, I figure I'd spare everyone the need of reading 
> through that. Below is a copy of the most relevant section from the PEP. I'd 
> also like to point out the section on performance (which you can find through 
> the above link) -- basically performance is on a par with that of the old 
> parser.
>
> ==
> Migration plan
> ==
>
> This section describes the migration plan when porting to the new PEG-based 
> parser
> if this PEP is accepted. The migration will be executed in a series of steps 
> that allow
> initially to fallback to the previous parser if needed:
>
> 1.  Before Python 3.9 beta 1, include the new PEG-based parser machinery in 
> CPython
> with a command-line flag and environment variable that allows switching 
> between
> the new and the old parsers together with explicit APIs that allow 
> invoking the
> new and the old parsers independently. At this step, all Python APIs like 
> ``ast.parse``
> and ``compile`` will use the parser set by the flags or the environment 
> variable and
> the default parser will be the current parser.
>
> 2.  After Python 3.9 Beta 1 the default parser will be the new parser.
>
> 3.  Between Python 3.9 and Python 3.10, the old parser and related code (like 
> the
> "parser" module) will be kept until a new Python release happens (Python 
> 3.10). In
> the meanwhile and until the old parser is removed, **no new Python Grammar
> addition will be added that requires the peg parser**. This means that 
> the grammar
> will be kept LL(1) until the old parser is removed.
>
> 4.  In Python 3.10, remove the old parser, the command-line flag, the 
> environment
> variable and the "parser" module and related code.
>
> --
> --Guido van Rossum (python.org/~guido)
> Pronouns: he/him (why is my pronoun here?)
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-dev@python.org/message/HOZ2RI3FXUEMAT4XAX4UHFN4PKG5J5GR/
> Code of Conduct: http://python.org/psf/codeofconduct/



-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/6X7OFZPG7URY5AK3ZKFJNGHUXXHMKNS4/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 617: New PEG parser for CPython

[Barry Warsaw]

On Apr 2, 2020, at 13:07, Pablo Galindo Salgado  wrote:
> 
>> Just to clarify, this means that 3.9 will ship with the PEG parser as 
>> default,
>> right?  If so, this would be a new feature, post beta.  Since that is 
>> counter to our
>> general policy, we would need to get explicit RM approval for such a change.
> 
> The idea is merging it *before beta* and make it the default *from the beta* 
> (maybe
> merging that commit making it the default the day before the beta for 
> instance).

Yes, sorry I don’t mean to be pedantic, but it would be better policy-wise if 
the new parser were the default in time for the first beta rather than after.

Cheers,
-Barry



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/IDHZ3S2UEPNZXPW3E7BZY3LYP5TXAA4B/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 617: New PEG parser for CPython

[Pablo Galindo Salgado]

> Just to clarify, this means that 3.9 will ship with the PEG parser as default,
> right?  If so, this would be a new feature, post beta.  Since that is counter 
> to our
> general policy, we would need to get explicit RM approval for such a change.

The idea is merging it *before beta* and make it the default *from the beta* 
(maybe
merging that commit making it the default the day before the beta for instance).

Pablo
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/VDM7LD4BOXZXMJLUFLOTVBF7B5AULB5W/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 617: New PEG parser for CPython

[Barry Warsaw]

Great to see this new work pay off!

On Apr 2, 2020, at 11:10, Guido van Rossum  wrote:

> 2.  After Python 3.9 Beta 1 the default parser will be the new parser.

Just to clarify, this means that 3.9 will ship with the PEG parser as default, 
right?  If so, this would be a new feature, post beta.  Since that is counter 
to our general policy, we would need to get explicit RM approval for such a 
change.

Cheers,
-Barry



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/EWBDCJVFEI4WASPXPWVIXVDMPI6TDC6Q/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 617: New PEG parser for CPython

[Paul Moore]

On Thu, 2 Apr 2020 at 19:20, Guido van Rossum  wrote:
>
> Since last fall's core sprint in London, Pablo Galindo Salgado, Lysandros 
> Nikolaou and myself have been working on a new parser for CPython. We are now 
> far enough along that we present a PEP we've written:
>
> https://www.python.org/dev/peps/pep-0617/
>
> Hopefully the PEP speaks for itself. We are hoping for a speedy resolution so 
> we can land the code we've written before 3.9 beta 1.

Excellent news! One question - will there be any user-visible change
as a result of this PEP other than the removal of the "parser" module?
>From my quick reading of the PEP, I didn't see anything, so I assume
the answer is "no".

Paul
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/CSUXLFZLP556LCX6MXYITEDQM3EQQZ5K/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 617: New PEG parser for CPython

[Ivan Levkivskyi]

These are good news. I think the new parser is indeed both simpler and more
flexible - great!

--
Ivan



On Thu, 2 Apr 2020 at 19:19, Guido van Rossum  wrote:

> Since last fall's core sprint in London, Pablo Galindo Salgado, Lysandros
> Nikolaou and myself have been working on a new parser for CPython. We are
> now far enough along that we present a PEP we've written:
>
> https://www.python.org/dev/peps/pep-0617/
>
> Hopefully the PEP speaks for itself. We are hoping for a speedy resolution
> so we can land the code we've written before 3.9 beta 1.
>
> If people insist I can post a copy of the entire PEP here on the list, but
> since a lot of it is just background information on the old LL(1) and the
> new PEG parsing algorithms, I figure I'd spare everyone the need of reading
> through that. Below is a copy of the most relevant section from the PEP.
> I'd also like to point out the section on performance (which you can find
> through the above link) -- basically performance is on a par with that of
> the old parser.
>
> ==
> Migration plan
> ==
>
> This section describes the migration plan when porting to the new
> PEG-based parser
> if this PEP is accepted. The migration will be executed in a series of
> steps that allow
> initially to fallback to the previous parser if needed:
>
> 1.  Before Python 3.9 beta 1, include the new PEG-based parser machinery
> in CPython
> with a command-line flag and environment variable that allows
> switching between
> the new and the old parsers together with explicit APIs that allow
> invoking the
> new and the old parsers independently. At this step, all Python APIs
> like ``ast.parse``
> and ``compile`` will use the parser set by the flags or the
> environment variable and
> the default parser will be the current parser.
>
> 2.  After Python 3.9 Beta 1 the default parser will be the new parser.
>
> 3.  Between Python 3.9 and Python 3.10, the old parser and related code
> (like the
> "parser" module) will be kept until a new Python release happens
> (Python 3.10). In
> the meanwhile and until the old parser is removed, **no new Python
> Grammar
> addition will be added that requires the peg parser**. This means that
> the grammar
> will be kept LL(1) until the old parser is removed.
>
> 4.  In Python 3.10, remove the old parser, the command-line flag, the
> environment
> variable and the "parser" module and related code.
>
> --
> --Guido van Rossum (python.org/~guido)
> *Pronouns: he/him **(why is my pronoun here?)*
> 
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/HOZ2RI3FXUEMAT4XAX4UHFN4PKG5J5GR/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/772SIMSA6B7P43ICIHVWPGSB3OYP5SSP/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Moving threadstate to thread-local storage.

[Petr Viktorin]

On 2020-03-24 16:31, Mark Shannon wrote:

Hi,

As an experiment, I thought I would try moving the thread state (what 
you get from _PyThreadState_GET() ) to TLS.


https://github.com/python/cpython/compare/master...markshannon:threadstate_in_tls 



It works, passing all the tests, and seems sound.

It is a small patch (< 50 lines) and doesn't increase the overall code 
size.


My branch is GCC/Clang only, so will need a bit of extra code for 
Windows. It should only need a few more lines; I haven't done it as I 
don't have a Windows machine to test it on.


What about other compilers?

AFAIK, __thread is a a non-standard name for a C11+ feature. Is there a 
good way to do this in C99?

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/ER5FMM2ZDSK2CT7OMDLFPIGIDKG4YHPJ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] PEP 617: New PEG parser for CPython

[Guido van Rossum]

Since last fall's core sprint in London, Pablo Galindo Salgado, Lysandros
Nikolaou and myself have been working on a new parser for CPython. We are
now far enough along that we present a PEP we've written:

https://www.python.org/dev/peps/pep-0617/

Hopefully the PEP speaks for itself. We are hoping for a speedy resolution
so we can land the code we've written before 3.9 beta 1.

If people insist I can post a copy of the entire PEP here on the list, but
since a lot of it is just background information on the old LL(1) and the
new PEG parsing algorithms, I figure I'd spare everyone the need of reading
through that. Below is a copy of the most relevant section from the PEP.
I'd also like to point out the section on performance (which you can find
through the above link) -- basically performance is on a par with that of
the old parser.

==
Migration plan
==

This section describes the migration plan when porting to the new PEG-based
parser
if this PEP is accepted. The migration will be executed in a series of
steps that allow
initially to fallback to the previous parser if needed:

1.  Before Python 3.9 beta 1, include the new PEG-based parser machinery in
CPython
with a command-line flag and environment variable that allows switching
between
the new and the old parsers together with explicit APIs that allow
invoking the
new and the old parsers independently. At this step, all Python APIs
like ``ast.parse``
and ``compile`` will use the parser set by the flags or the environment
variable and
the default parser will be the current parser.

2.  After Python 3.9 Beta 1 the default parser will be the new parser.

3.  Between Python 3.9 and Python 3.10, the old parser and related code
(like the
"parser" module) will be kept until a new Python release happens
(Python 3.10). In
the meanwhile and until the old parser is removed, **no new Python
Grammar
addition will be added that requires the peg parser**. This means that
the grammar
will be kept LL(1) until the old parser is removed.

4.  In Python 3.10, remove the old parser, the command-line flag, the
environment
variable and the "parser" module and related code.

-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/HOZ2RI3FXUEMAT4XAX4UHFN4PKG5J5GR/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Python-Dev Digest, Vol 201, Issue 1

[Ilya Kamenshchikov]

One potentially serious question: what should `enumerate.__reversed__`
> do when given a starting value?
> reversed(enumerate('abc', 1))
> Should that yield...?
> # treat the start value as a start value
> (1, 'c'), (0, 'b'), (-1, 'a')
> # treat the start value as an end value
> (3, 'c'), (2, 'b'), (1, 'a')
> Something else?
> My preference would be to treat the starting value as an ending value.



My idea is largely that as long as any iterable has defined length and
order, it should be reversible, and this should yield the same as if I did
this reverse countdown with some pointer.
It just so happens that `enumerate` depends fully in this regard on the
iterable it enumerates, if it has defined length, then enumerate will also
have defined length. Maybe enumerate should also start defining `len`
whenever underlying iterable supports it.

Regarding the starting value, I think it is important to keep symmetry:
enumerate('abc', 1) is  [ (1, 'a'), (2, 'b'), (3, 'c') ], therefore
reversed(enumerate('abc', 1)) should be [(3, 'c'), (2, 'b'), (1, 'a')]

Best Regards,
--
Ilya Kamen


On Thu, Apr 2, 2020 at 5:55 AM  wrote:

> Send Python-Dev mailing list submissions to
> python-dev@python.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> or, via email, send a message with subject or body 'help' to
> python-dev-requ...@python.org
>
> You can reach the person managing the list at
> python-dev-ow...@python.org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of Python-Dev digest..."
>
> Today's Topics:
>
>1. Re: reversed enumerate (Steven D'Aprano)
>2. Re: [Python-ideas] Re: reversed enumerate (Andrew Barnert)
>3. Re: [Python-ideas] Re: reversed enumerate (Andrew Barnert)
>
>
> --
>
> Date: Thu, 2 Apr 2020 13:20:02 +1100
> From: Steven D'Aprano 
> Subject: [Python-Dev] Re: reversed enumerate
> To: python-dev@python.org
> Cc: python-id...@python.org
> Message-ID: <20200402022000.gc16...@ando.pearwood.info>
> Content-Type: text/plain; charset=us-ascii
>
> Hi Ilya,
>
> I'm not sure that this mailing list (Python-Dev) is the right place for
> this discussion, I think that Python-Ideas (CCed) is the correct place.
>
> For the benefit of Python-Ideas, I have left your entire post below, to
> establish context.
>
> [Ilya]
> > I needed reversed(enumerate(x: list)) in my code, and have discovered
> > that it wound't work. This is disappointing because operation is well
> > defined.
>
> It isn't really well-defined, since enumerate can operate on infinite
> iterators, and you cannot reverse an infinite stream. Consider:
>
> def values():
> while True:
> yield random.random()
>
> a, b = reversed(enumerate(values())
>
> What should the first pair of (a, b) be?
>
> However, having said that, I think that your idea is not unreasonable.
> `enumerate(it)` in the most general case isn't reversable, but if `it`
> is reversable and sized, there's no reason why `enumerate(it)` shouldn't
> be too.
>
> My personal opinion is that this is a fairly obvious and straightforward
> enhancement, one which (hopefully!) shouldn't require much, if any,
> debate. I don't think we need a new class for this, I think enhancing
> enumerate to be reversable if its underlying iterator is reversable
> makes good sense.
>
> But if you can show some concrete use-cases, especially one or two from
> the standard library, that would help your case. Or some other languages
> which offer this functionality as standard.
>
> On the other hand, I think that there is a fairly lightweight work
> around. Define a helper function:
>
> def countdown(n):
> while True:
> yield n
> n -= 1
>
> then call it like this:
>
> # reversed(enumerate(seq))
> zip(countdown(len(seq)-1), reversed(seq)))
>
> So it isn't terribly hard to work around this. But I agree that it would
> be nice if enumerate encapsulated this for the caller.
>
> One potentially serious question: what should `enumerate.__reversed__`
> do when given a starting value?
>
> reversed(enumerate('abc', 1))
>
> Should that yield...?
>
> # treat the start value as a start value
> (1, 'c'), (0, 'b'), (-1, 'a')
>
> # treat the start value as an end value
> (3, 'c'), (2, 'b'), (1, 'a')
>
> Something else?
>
> My preference would be to treat the starting value as an ending value.
>
>
> Steven
>
>
> On Wed, Apr 01, 2020 at 08:45:34PM +0200, Ilya Kamenshchikov wrote:
> > Hi,
> >
> > I needed reversed(enumerate(x: list)) in my code, and have discovered
> that
> > it wound't work. This is disappointing because operation is well defined.
> > It is also well defined for str type, range, and - in principle, but not
> > yet in practice - on dictionary iterators - keys(), values(), items() 

[Python-Dev] Re: reversed enumerate

[Ilya Kamenshchikov]

Re: st...@pearwood.info

> It isn't really well-defined, since enumerate can operate on infinite
> iterators, and you cannot reverse an infinite stream.

It is well defined on any iterable that itself is reversible and has
defined length. For standard types that's lists, strings, dictionary
iterators and some of the collections (deque, dictionary-based counter).


Re: storch...@gmail.com
> Could you please provide evidence that this feature would be quite
> useful? How much usecases can you find in the stdlib?

I needed this while solving a CS problem
https://leetcode.com/problems/container-with-most-water/, which, I agree,
does not tell how useful it is in production projects.

Couldn't find any usecase in stdlib. I could find a usecase in scikit-learn:
https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/pipeline.py#L639
in case file gets changed, here is a copy:

for i in reversed(range(len(estimators))):
name = names[i]
if name in namecount:
names[i] += "-%d" % namecount[name]
namecount[name] -= 1

It can indeed be that one doesn't have to reverse enumeration very
often in practice. It can also be that people just express their ideas
differently knowing it doesn't work in straight-forward way.


Best Regards,
--
Ilya Kamenshchikov
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/W26NX33MDJSDKKEEOWWYFMZN6UKTJUOK/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] reversed enumerate

[Stephen J. Turnbull]

Ilya Kamenshchikov writes:

 > I needed reversed(enumerate(x: list)) in my code,

def reversed_enumerated_list(l):
return zip(reversed(range(len(l)), reversed(l))

 > and have discovered that it wound't work.

I wouldn't have expected it to, because enumerate's result is
unbounded.  But I'm definitely warming to the idea that this should
work for iterables with known length.  Presumably the same
implementation method used for reversed itself would work; not sure if
that is what you proposed.

The method used to implement reversibility should probably be exposed
so that other code that wants a finite enumeration can determine what
enumerate has discovered about the underlying iterable.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/22EP2NZMT624AOL4QTAC6VOKOXOLY7ZM/
Code of Conduct: http://python.org/psf/codeofconduct/