I’ve created a custom language with non-s-expression syntax, so I’ve written a lexer and connected it to the DrRacket syntax highlighter. This works pretty well, but I have a small problem: when modifying text that would cause earlier tokens in the stream to lex differently, I am unsure how to properly communicate that need to the highlighter.
Specifically, this language allows forward slashes within identifiers, like foo/bar/baz, but they cannot be leading or trailing slashes. Therefore, something like “/foo” on its own is a lexer error, so the “/” gets highlighted in red. Normally that’s ok, but when a user types a valid identifier, like “foo/bar”, the highlighter treats “/bar” separately and attempts to lex it in isolation, which produces an error. What I actually want is for typing the forward slash in “foo/bar” to re-lex the entire symbol, producing the correct token. As far as I can tell, there are two mechanisms for this sort of re-lexing, both using the 3-argument get-token function with color:text<%>: - Providing a backup distance, which causes re-lexing when a token is internally modified. - Returning a dont-stop struct as the mode, which prevents the highlighter from being interrupted as long as get-token continues to return dont-stop. I don’t think the backup distance applies here because the token is not being internally modified, it is being appended to. I thought that returning dont-stop when lexing identifiers would solve my problem, but it didn’t seem to affect anything, so I am likely misunderstanding how it works. What is the solution to this sort of problem? What mechanism do I need to use to properly communicate the need to re-lex from the beginning of each identifier when appending text to the end? -- You received this message because you are subscribed to the Google Groups "Racket Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to racket-users+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.