On 4/1/18 5:11 AM, Eric V. Smith wrote:
> On 3/31/2018 9:48 PM, Steven D'Aprano wrote:
>> On Sun, Apr 01, 2018 at 02:20:16AM +0100, Rob Cliffe via Python-ideas
>> wrote:
>>
>>>> New unordered 'd' and 'D' prefixes, for 'dedent', applied to multiline
>>>> strings only, would multiply the number of alternatives by about 5 and
>>>> would require another rewrite of all code (Python or not) that parses
>>>> Python code (such as in syntax colorizers).
>>>
>>> I think you're exaggerating the difficulty somewhat. Multiplying the
>>> number of alternatives by 5 is not the same thing as increasing the
>>> complexity of code to parse it by 5.
>>
>> Terry didn't say that it would increase the complexity of the code by a
>> factor of five. He said it would multiply the number of alternatives by
>> "about 5". There would be a significant increase in the complexity of
>> the code too, but I wouldn't want to guess how much.
>>
>> Starting with r and f prefixes, in both upper and lower case, we have:
>>
>> 4 single letter prefixes
>> (plus 2 more, u and U, that don't combine with others)
>> 8 double letter prefixes
>>
>> making 14 in total. Adding one more prefix, d|D, increases it to:
>>
>> 6 single letter prefixes
>> (plus 2 more, u and U)
>> 24 double letter prefixes
>> 48 triple letter prefixes
>>
>> making 80 prefixes in total. Terry actually underestimated the explosion
>> in prefixes: it is closer to six times more than five (but who is
>> counting? apart from me *wink*)
>>
>> [Aside: if we add a fourth, the total becomes 634 prefixes.]
>
> Not that it really matters, but there's some code I use whenever I
> feel like playing with adding string prefixes. It usually encourages
> me to not do that!
>
> Lib/tokenize.py:_all_string_prefixes exists just for calculating
> string prefixes. Since it's not what is actually used by the
> tokenizer, I don't claim it's perfect (but I don't know of any errors
> in it).
>
> According to it, and ignoring the empty string, there are currently 24
> prefixes:
> {'B', 'BR', 'Br', 'F', 'FR', 'Fr', 'R', 'RB', 'RF', 'Rb', 'Rf', 'U',
> 'b', 'bR', 'br', 'f', 'fR', 'fr', 'r', 'rB', 'rF', 'rb', 'rf', 'u'}
>
> And if you add 'd', and it can't combine with 'b' or 'u', I count 90:
> {'rdf', 'FR', 'dRF', 'rD', 'FrD', 'DFr', 'frd', 'RDf', 'u', 'DF', 'd',
> 'Frd', 'frD', 'dFr', 'rDF', 'fD', 'rB', 'dFR', 'FD', 'dr', 'Fr',
> 'DfR', 'fdR', 'Rb', 'dfr', 'rdF', 'rf', 'Drf', 'R', 'RB', 'BR', 'FdR',
> 'bR', 'DFR', 'RdF', 'dF', 'F', 'fd', 'Br', 'Dfr', 'Dr', 'r', 'rfd',
> 'RFd', 'Fdr', 'dfR', 'rb', 'fDr', 'rFD', 'fRd', 'Rfd', 'RDF', 'rFd',
> 'Rdf', 'rF', 'FDr', 'drF', 'dR', 'D', 'br', 'fr', 'drf', 'DrF', 'rd',
> 'DRF', 'DR', 'RFD', 'Rf', 'fR', 'RfD', 'Df', 'rDf', 'U', 'f', 'df',
> 'DRf', 'fdr', 'B', 'FRD', 'RF', 'Fd', 'Rd', 'fRD', 'FRd', 'b', 'dRf',
> 'FDR', 'RD', 'fDR', 'rfD'}
>
> I guess it's debatable if you want to count prefixes that contain 'b'
> as string prefixes or not, but the tokenizer thinks they are. If you
> leave them out, you come up with the 14 and 80 that Steven mentions.
>
> I agree with Terry that adding 'F' was a mistake. But since the upper
> case versions of 'r', and 'b' already existed, it was included.
>
> Interestingly, in 2.7 'ur' is a valid prefix, but not in 3.6. I don't
> recall if that was deliberate or not. And 'ru' isn't valid in either
> version.
>
> Eric
One comment about the 'combitorial explosion' is that it sort of assumes
that each individual combination case needs to be handled with distinct
code. My guess is that virtually all of the actual implementation of
these prefixes can be handled by setting a flag for the presence of that
prefix, and at the parsing of each character you need to just check a
flag or two to figure out how to process it. You might get a bit more
complication in determining if a given combination is valid, but if that
gets too complicated it is likely an indication of an inconstancy in the
language definition.
--
Richard Damon
_______________________________________________
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
