If you look at pegen, that uses the stdlib tokenizer as input, you will see
that the obejct us3d to implement memoization on top of a token stream
simply swallow NL (
https://github.com/we-like-parsers/pegen/blob/main/src/pegen/tokenizer.py#L49).
This is safe since NL has no syntactic meaning only
Hi David,
Could you share what you have so far, perhaps ok GitHub or so? That way
it's easier to diagnose your problems. I'm reasonably familiar with Rust.
Perhaps also add a minimal crashing example?
Cheers,
Matthias.
On Thu, 27 Oct 2022, 04:52 David J W, wrote:
> Pablo,
> Nl and Newline
Hummm… he is also mentioning NL and Newline tokens and if I recall correctly those are tokens that only appear in the Python tokenizer and are emitted differently from the C one (and therefore they are not used in the grammar).Pablo Galindo SalgadoOn 26 Oct 2022, at 21:57, Guido van Rossum wrote:
Thank you very much for your input, I've filled in the bug report:
https://github.com/python/cpython/issues/98740
Best regards,
Piotr
On Wed, Oct 26, 2022 at 6:55 PM Serhiy Storchaka
wrote:
> 26.10.22 11:17, Piotr Waszkiewicz пише:
> > Hi,
> > I would like to ask your guidance as I'm entirely s
I wonder if David may be struggling with the rule that a newline is
significant in the grammar unless it appears inside matching
brackets/parentheses/braces? I think that's in the lexer. Similarly,
multiple newlines are collapsed.
On Wed, Oct 26, 2022 at 1:19 PM Pablo Galindo Salgado
wrote:
> Hi
Pablo,
Nl and Newline are tokens but I am interested in NEWLINE's behavior in
the Python grammar, note the casing.
For example in simple_stmts @
https://github.com/python/cpython/blob/main/Grammar/python.gram#L107
Is that NEWLINE some sort of built in rule to the grammar? In my project
I am
Hi everyone,
Now that the 3.11.0 release is finally done and I can relax a bit, I just
wanted to thank you all
for your fantastic work that has made Python 3.11 such a fantastic release.
No matter if you committed
code to 3.11 or opened a bug, helped with the documentation, reviewed pull
requests,
Hi,
As I mentioned, NEWLINE is a token. All uppercase words in the grammar are
tokens and therefore are produced by the lexer, not the parser. Is not a
built-in rule. In particular, that token is produced here:
https://github.com/python/cpython/blob/6777e09166fc384ea0a4b50202c7b0bd7a23330c/Parser
Hi,
I am not sure I understand exactly what you are asking but NEWLINE is a token,
not a parser rule. What decides when NEWLINE is emitted is the lexer that has
nothing to do with PEG. Normally PEG parsers also acts as tokenizers but the
one in cpython does not.
Also notice that CPython’s pars
I am writing a Rust version of Python for fun and I am at the parser stage
of development.
I copied and modified a PEG grammar ruleset from another open source
project and I've already noticed some problems (ex Newline vs NL) with how
they transcribed things.
I am suspecting that CPython's gramma
26.10.22 11:17, Piotr Waszkiewicz пише:
Hi,
I would like to ask your guidance as I'm entirely sure whether the problem I'm
experiencing should be posted in CPython's repo as a bug issue.
I've tried using newly released Python 3.11 interpreter in some of my projects and one of
them failed to sta
On 2022-10-26 09:17, Piotr Waszkiewicz wrote:
Hi,
I would like to ask your guidance as I'm entirely sure whether the problem I'm
experiencing should be posted in CPython's repo as a bug issue.
I've tried using newly released Python 3.11 interpreter in some of my projects and one of
them failed
Hi,
I would like to ask your guidance as I'm entirely sure whether the problem I'm
experiencing should be posted in CPython's repo as a bug issue.
I've tried using newly released Python 3.11 interpreter in some of my projects
and one of them failed to start with "RuntimeError: invalid SRE code"
I don't remember it being mentioned, but much of the traffic recently
migrated from this list to https://discuss.python.org/c/core-dev/23, which
you may wish to keep in touch with.
Kind regards,
Steve
On Tue, Oct 25, 2022 at 7:53 AM Juan Cristóbal Quesada <
rainonthescarecrowhumanwhe...@gmail.co
14 matches
Mail list logo
