[issue40678] Full list of Python lexical rules

[Ram Rachum]

Ram Rachum  added the comment:

I understand, thank you.

--
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40678] Full list of Python lexical rules

[Terry J. Reedy]

Terry J. Reedy  added the comment:

What you literally seem to ask for does not exist.  If you want to pursue this, 
I suggest posting to python-ideas and you might get support for an acceptable 
alternative.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40678] Full list of Python lexical rules

[Ram Rachum]

Ram Rachum  added the comment:

Hmm, I feel this isn't right, because I still feel like there should be one 
place where one can see the full Python syntax specification, lexing and 
parsing and all. But I'm underqualified to argue because I don't understand the 
details. Is someone more knowledgeable interested in arguing this point?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40678] Full list of Python lexical rules

[Terry J. Reedy]

Change by Terry J. Reedy :


--
versions: +Python 3.10 -Python 3.6, Python 3.7, Python 3.8, Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40678] Full list of Python lexical rules

[Terry J. Reedy]

Terry J. Reedy  added the comment:

First note that 3.8.3 grammar.html is stated to be the actual grammar used by 
the old parser, and is a bit different from the more human readable grammar 
given in the reference manual.  It is a bit different in 3.9 and I expect will 
be much more different in 3.10 with the new PEG parser. 

In the grammar, the CAPITALIZED_NAMES are token names returned by the 
tokenizer/lexer.  This is a standard convention.  

I am pretty sure that the human readable lexing rules in lexical_analysis are 
not what the lexer uses.  I presume the latter uses barely readable RE 
expressions, as does the tokenize module.

Compare the float grammar in 
https://docs.python.org/3/reference/lexical_analysis.html#floating-point-literals
 to the float REs in tokenize.py.

def group(*choices): return '(' + '|'.join(choices) + ')'
def maybe(*choices): return group(*choices) + '?'
# The above are reused for multiple REs.
Exponent = r'[eE][-+]?[0-9](?:_?[0-9])*'
Pointfloat = group(r'[0-9](?:_?[0-9])*\.(?:[0-9](?:_?[0-9])*)?',
   r'\.[0-9](?:_?[0-9])*') + maybe(Exponent)
Expfloat = r'[0-9](?:_?[0-9])*' + Exponent
Floatnumber = group(Pointfloat, Expfloat)

Note that this is (python) code, not a text specification.  You or someone else 
can look at what the C lexer does.  But I think that the proposal should be 
rejected.

--
nosy: +terry.reedy

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40678] Full list of Python lexical rules

[Ram Rachum]

New submission from Ram Rachum :

I'm a noob on parsing, learning about it, so it's possible I've made a mistake 
somewhere.

I know there's this page: https://docs.python.org/3/reference/grammar.html

Which is a full listing of Python's grammar. However, looking at this page: 
https://docs.python.org/3/reference/lexical_analysis.html I see rules that 
aren't written there, like longstringitem.

I'm guessing that's because these are lexing rules, while the former was a list 
of parsing rules?

If that's the case, shouldn't there also be a full, authoritative list of 
Python's lexical rules? Possibly alongside the parsing rules?

--
assignee: docs@python
components: Documentation
messages: 369320
nosy: cool-RR, docs@python, georg.brandl, gvanrossum
priority: normal
severity: normal
status: open
title: Full list of Python lexical rules
type: enhancement
versions: Python 3.6, Python 3.7, Python 3.8, Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com