On Fri, Oct 1, 2021 at 3:53 PM goodman....@gmail.com <goodman....@gmail.com> wrote:
> I haven't thought deeply about working with tokenized input, but if tokens > are the alphabet for the PEG then it seems to me there's a commitment to > use what the lexer provides > Just wanted to pass on some feedback I received on a paper describing a PEG parser that was able to optionally use a lex preprocessing step. It's worth it to be aware of this: Lexing changes the recognized language, and it is not possible (in the general case, at least) to produce a lexer+parser combo that matches the semantics of a pure lex-less parser for the same target language, because of the limited backtracking behavior of a lex-less parser. In other words, lexing is "greedy", in the sense that it has limited contextual information.
_______________________________________________ PEG mailing list PEG@lists.csail.mit.edu https://lists.csail.mit.edu/mailman/listinfo/peg
