On 2/13/2018 2:19 PM, Jim Laskey wrote:
10a. String s = `abc`; 10b. String s = \u0060abc`;
...
So, change the scanner to

A) Peek back to make sure the first open backtick was exactly a
backtick. B) Turn off Unicode escapes immediately so that only
backtick characters can be part of the delimiter. C) Turn on Unicode
escapes only after a valid closing delimiter is encountered.

Based on this all your examples are illegal.

I am not opposed to saying that a delimiter must be constructed from actual ` characters (that is, the RawInputCharacter ` rather than the UnicodeEscape \u0060). It would be silly if the opening delimiter was \u0060 because the closing delimiter cannot be identical -- that hurts readability. (Clearly the six characters \ u 0 0 6 0 inside a raw string literal get no special processing.)

Unfortunately, there is nothing in the lexical grammar that prevents \u0060Hello` or \u0060Hello\u0060 or in fact any of the examples below from being lexed as a RawStringLiteral. The JLS will need a semantic rule to force each RawStringDelimiter to be composed of actual ` characters. As you say, this will make all the examples below illegal.

There is plenty of precedent for semantic rules ("It is a compile-time error ...") in the interpretation of Literal tokens, so that's fine. In fact, JLS 3.10.4 already has a semantic rule that appears to constrain a delimiter in a CharacterLiteral token:

  It is a compile-time error for the character following the
  SingleCharacter or EscapeSequence to be other than a '.

although it doesn't mean to force an actual ' character (that is, the RawInputCharacter ' and not the UnicodeEscape \u0027). It means:

  It is a compile-time error for the character following the
  SingleCharacter or EscapeSequence to be other than a ' (or the
  Unicode escape thereof).

Alex

On Feb 13, 2018, at 1:58 PM, Alex Buckley <alex.buck...@oracle.com>
wrote:

I suspect the trickiest part of specifying raw string literals will
be the lexer's modal behavior for Unicode escapes. As such, I am
going to put the behavior under the microscope. Here is what the
JEP has to say:

----- Unicode escapes, in the form \uxxxx, are processed as part of
character input prior to interpretation by the lexer. To support
the raw string literal as-is requirement, Unicode escape processing
is disabled when the lexer encounters an opening backtick and
reenabled when encountering a closing backtick. -----

I would like to assume that if the lexer comes across the six
tokens \ u 0 0 6 0  then it should interpret them as a Unicode
escape representing a backtick _and then continue as if consuming
the tokens of a raw string literal_. However, the mention of _an_
opening backtick and _a_ closing backtick gave me pause, given that
repeated backticks can serve as the opening delimiter and the
closing delimiter. For absolute clarity, let's write out examples
to confirm intent: (Jim, please confirm or deny as you see fit!)

1.  String s = \u0060`;

Illegal. The RHS is lexed as ``;   which is disallowed by the
grammar.

2.  String s = \u0060Hello\u0060;

Illegal. The RHS is lexed as `Hello\u0060;   and so on for the rest
of the compilation unit -- the six tokens \ u 0 0 6 0 are not
treated as a Unicode escape since we're lexing a raw string
literal. And without a closing delimiter before the end of the
compilation unit, a compile-time error occurs.

3a.  String s = \u0060Hello`;

Legal. The RHS is lexed as `Hello`;   which is well formed.

3b.  String s = \u0060\u0060Hello`;

Depends! If you take the JEP literally, then just the Unicode
escape which serves as the first opening backtick ("_an_ opening
backtick") is enough to enter raw-string mode. That makes the code
legal: the RHS is lexed as `\u0060Hello`;   which is well formed.
On the other hand, you might think that we shouldn't enter
raw-string mode until the lexer in traditional mode has lexed the
opening delimiter fully (i.e. ALL the opening backticks). Then, the
code in 3b is illegal, because the opening delimiter (``) and the
closing delimiter (`) are not symmetric.

I think we should take the JEP literally, so that 3b is legal. And
then, some more examples:

4a.  String s = \u0060`Hello``;

Legal. The RHS is lexed as ``Hello``;   which is well formed.

4b.  String s = \u0060\u0060Hello``;

Illegal. The RHS is lexed as `\u0060Hello``;   which is disallowed
by the grammar. A raw string literal containing 11 tokens is
immediately followed by a ` token and a ; token which are not
expected.

4c.  String s = \u0060\u0060Hello`\u0060;

Depends! If you take the JEP literally, where _a_ closing backtick
is enough to re-enable Unicode escape processing, then the RHS is
lexed as `\u0060Hello``;  which is illegal per 4b. On the other
hand, if you think that we shouldn't re-enter traditional mode
until the lexer in raw-string mode has lexed the closing delimiter
fully (i.e. ALL the closing backticks), then presumably you think
analogously about the opening delimiter, so the RHS would be lexed
as ``Hello`\u0060;   which is illegal per 2 (no closing delimiter
`` before the end of the compilation unit).

5.  String s = \u0060`Hello`\u0060;

I put this here because it looks nice. It hits the same issues as
3b and 4c.

Alex

Reply via email to