[perl #130612] [BUG] LTM doesn't use text order for tie break as expected
Actually...
Rakudo *does* generally follow interpretation (b):
➜ 'x' ~~ / .* { say '*' } | .? { say '?' } /; # *
➜ 'x' ~~ / .? { say '?' } | .* { say '*' } /; # ?
The observed bug is specifically with character classes:
➜ '1' ~~ / { say 'digit' } | <[0..9]> { say '0..9' } /; # 0..9
➜ '1' ~~ /<[0..9]> { say '0..9' } | { say 'digit' } /; # 0..9
Following some more experimentation, here are various atoms for matching the
digit '1', sorted into three categories based on how much LTM favors them in
current Rakudo:
tier 1: '1'
tier 2: . \d \w <[0..9]>
tier 3:<:Number> <:Decimal> etc.
That the literal `1` is preferred over everything else by LTM is to be expected
("longest literal prefix" tie-breaker).
However, that the character classes are split into two tiers - with the
syntactic ones being preferred over the named and uniprop ones - seems strange.
At least I don't see anything in S05 to back that up.
http://design.perl6.org/S05.html#Overview says that LTM is transitive through
subrules, so even if the named character classes are treated as subrule calls
they shouldn't be disfavored, right?
[perl #130612] [BUG] LTM doesn't use text order for tie break as expected
This bug is still present in
Rakudo version 2017.08-104-g76f1d8970
built on MoarVM version 2017.08.1-148-g1059eed1
implementing Perl 6.c.
---
However, there's also a design question to be answered here:
On Sat, 21 Jan 2017 10:32:39 -0800, ronaldxs wrote:
> https://design.perl6.org/S05.html#Longest-token_matching says:
> # For matches of same length ... If the alternatives are in the same
> # grammar file, the textually earlier alternative takes precedence.
This S05 paragraph was clearly written with LTM of `proto` regexes in mind.
What does it mean for LTM of `|` alternations?
I.e., should the test-case above match 'n'...
a) because `token n { ... }` is declared above `token na_2 { ... }`?
b) because in ` | `, ` ` is to the left of ` `?
The test-case at hand does not distinguish between those two interpretations
(Rakudo is currently wrong either way), but it's an important distinction that
needs to be clarified if the bug is to be fixed.
http://design.perl6.org/S05.html#Overview has a sentence on this tie-breaker
which actually mentions alternations:
> Within a given compilation unit, the earlier alternative or multi wins
I'd interpret that as option (b), but am not 100% sure.
[perl #130612] [BUG] LTM doesn't use text order for tie break as expected
# New Ticket Created by Ron Schmidt
# Please include the string: [perl #130612]
# in the subject line of all future correspondence about this issue.
# https://rt.perl.org/Ticket/Display.html?id=130612 >
# https://design.perl6.org/S05.html#Longest-token_matching says:
# For matches of same length ... If the alternatives are in the same
# grammar file, the textually earlier alternative takes precedence.
use Test;
grammar text-order {
token alt-na_1 { | };
token alt-na_2 { | };
token n { + }
token na_1 { <+alpha +digit> + }
token na_2 { <+alpha +[0..9]> + }
}
is text-order.parse('1', :rule).keys[0], 'n',
'Match first textual rule OK';
is text-order.parse('1', :rule).keys[0], 'n',
'Match second textual rule fail';
result:
ok 1 - Match first textual rule OK
not ok 2 - Match second textual rule fail
# Failed test 'Match second textual rule fail'
# at line 17
# expected: 'n
# got: 'na_2'
Posted on IRC, https://irclog.perlgeek.de/perl6/2017-01-20#i_13960886
The IRC discussion notes that the tests should be expected to pass and
the S05 rule is intended for implementation in current Perl 6 / Rakudo.
See also roast issue: https://github.com/perl6/roast/issues/224
