I've just uploaded 2.080000, a new indexed version<https://metacpan.org/release/JKEGL/Marpa-R2-2.080000>, to CPAN. This release is a very big change internally, the first external sign of which is the introduction of efficient Longest Acceptable Tokens Matching (LATM) to the SLIF. LATM is implemented using the "forgiving adverb"<https://metacpan.org/pod/release/JKEGL/Marpa-R2-2.080000/pod/Scanless/DSL.pod#forgiving> .
LATM means that the SLIF, when lexing, considers only those tokens that would actually be acceptable to the top-level structural grammar. Previously, the SLIF used Longet Tokens Matching (LTM), and considered all lexemes, even those that would not be acceptable to the parse. LTM is the tradition and the current standard practice in parsing -- pre-Marpa parsers are not smart enough to tell lexers which lexemes they can accept and which they cannot. LATM is "smarter" than LTM. This "smartness" allows you to be more aggressive in designing your lexer -- it increases the likelihood that the parser will know "what you meant". When context is used to help make the decision, the chance that two of your tokens will treat the same input string in conflicting ways is reduced. With LATM, unacceptable tokens will not cause conflicts. LATM is not the default, because of backward compatibility. But I recommend that everyone, as their preferred choice, start all new scripts with lexeme default = forgiving => 1 You also should be able to convert most old scripts to use LATM without a problem, making them faster and easier to extend. Note that the documentation still contains warning language about the efficiency implications of the "forgiving" adverb. These warnings need not be heeded -- they are no longer true, and will be removed in a future release. -- You received this message because you are subscribed to the Google Groups "marpa parser" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/groups/opt_out.
