I suspect the trickiest part of specifying raw string literals will be the lexer's modal behavior for Unicode escapes. As such, I am going to put the behavior under the microscope. Here is what the JEP has to say:

-----
Unicode escapes, in the form \uxxxx, are processed as part of character input prior to interpretation by the lexer. To support the raw string literal as-is requirement, Unicode escape processing is disabled when the lexer encounters an opening backtick and reenabled when encountering a closing backtick.
-----

I would like to assume that if the lexer comes across the six tokens \ u 0 0 6 0 then it should interpret them as a Unicode escape representing a backtick _and then continue as if consuming the tokens of a raw string literal_. However, the mention of _an_ opening backtick and _a_ closing backtick gave me pause, given that repeated backticks can serve as the opening delimiter and the closing delimiter. For absolute clarity, let's write out examples to confirm intent: (Jim, please confirm or deny as you see fit!)

1.  String s = \u0060`;

Illegal. The RHS is lexed as ``;   which is disallowed by the grammar.

2.  String s = \u0060Hello\u0060;

Illegal. The RHS is lexed as `Hello\u0060; and so on for the rest of the compilation unit -- the six tokens \ u 0 0 6 0 are not treated as a Unicode escape since we're lexing a raw string literal. And without a closing delimiter before the end of the compilation unit, a compile-time error occurs.

3a.  String s = \u0060Hello`;

Legal. The RHS is lexed as `Hello`;   which is well formed.

3b.  String s = \u0060\u0060Hello`;

Depends! If you take the JEP literally, then just the Unicode escape which serves as the first opening backtick ("_an_ opening backtick") is enough to enter raw-string mode. That makes the code legal: the RHS is lexed as `\u0060Hello`; which is well formed. On the other hand, you might think that we shouldn't enter raw-string mode until the lexer in traditional mode has lexed the opening delimiter fully (i.e. ALL the opening backticks). Then, the code in 3b is illegal, because the opening delimiter (``) and the closing delimiter (`) are not symmetric.

I think we should take the JEP literally, so that 3b is legal. And then, some more examples:

4a.  String s = \u0060`Hello``;

Legal. The RHS is lexed as ``Hello``;   which is well formed.

4b.  String s = \u0060\u0060Hello``;

Illegal. The RHS is lexed as `\u0060Hello``; which is disallowed by the grammar. A raw string literal containing 11 tokens is immediately followed by a ` token and a ; token which are not expected.

4c.  String s = \u0060\u0060Hello`\u0060;

Depends! If you take the JEP literally, where _a_ closing backtick is enough to re-enable Unicode escape processing, then the RHS is lexed as `\u0060Hello``; which is illegal per 4b. On the other hand, if you think that we shouldn't re-enter traditional mode until the lexer in raw-string mode has lexed the closing delimiter fully (i.e. ALL the closing backticks), then presumably you think analogously about the opening delimiter, so the RHS would be lexed as ``Hello`\u0060; which is illegal per 2 (no closing delimiter `` before the end of the compilation unit).

5.  String s = \u0060`Hello`\u0060;

I put this here because it looks nice. It hits the same issues as 3b and 4c.

Alex

Reply via email to