I’ve created a custom language with non-s-expression syntax, so I’ve
written a lexer and connected it to the DrRacket syntax highlighter.
This works pretty well, but I have a small problem: when modifying text
that would cause earlier tokens in the stream to lex differently, I am
unsure how to properly communicate that need to the highlighter.
Specifically, this language allows forward slashes within identifiers,
like foo/bar/baz, but they cannot be leading or trailing slashes.
Therefore, something like “/foo” on its own is a lexer error, so the “/”
gets highlighted in red.
Normally that’s ok, but when a user types a valid identifier, like
“foo/bar”, the highlighter treats “/bar” separately and attempts to lex
it in isolation, which produces an error. What I actually want is for
typing the forward slash in “foo/bar” to re-lex the entire symbol,
producing the correct token.
As far as I can tell, there are two mechanisms for this sort of
re-lexing, both using the 3-argument get-token function with
color:text<%>:
- Providing a backup distance, which causes re-lexing when a token
is internally modified.
- Returning a dont-stop struct as the mode, which prevents the
highlighter from being interrupted as long as get-token continues
to return dont-stop.
I don’t think the backup distance applies here because the token is not
being internally modified, it is being appended to. I thought that
returning dont-stop when lexing identifiers would solve my problem, but
it didn’t seem to affect anything, so I am likely misunderstanding how
it works.
What is the solution to this sort of problem? What mechanism do I need
to use to properly communicate the need to re-lex from the beginning of
each identifier when appending text to the end?
--
You received this message because you are subscribed to the Google Groups
"Racket Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/d/optout.
