Hi László,
Thanks for the pointer, I was taking a look at that very line too.
To clarify, the original Rust grammars by the Rust Team ignore the ">>"
lexer token [1] and let the parser decide [2],[3] if that is valid or
not depending if we're analyzing an expression or a type parameter:
Expression with shift right) "a >> 3;"
Type pameter) "<Vec<u8 >> ();"
From my point of view we should indeed _lex_ the ">>" token (same for
"<<") and decide in the _parser_ whether the ">>" is a valid grammar
construct or not. After all we're building an editor, not a compiler.
In any case we should be ready in "AbstractAntlrLexerBridge" to detect
these sort of hacks and avoid firing the assertion in [4] (taking care
of that assertion in [5] "LexerInputCharStream"?) otherwise bad things
happen to the event dispatch thread and NetBeans freezes somehow.
Cheers,
Antonio
[1]
https://github.com/antlr/grammars-v4/blob/6e0c97aaf0e3149c59b69335f9f75b5a0993e32f/rust/RustLexer.g4#L302
[2]
https://github.com/antlr/grammars-v4/blob/6e0c97aaf0e3149c59b69335f9f75b5a0993e32f/rust/RustParser.g4#L1086
[3]
https://github.com/antlr/grammars-v4/blob/6e0c97aaf0e3149c59b69335f9f75b5a0993e32f/rust/Java/RustParserBase.java#L8
[4]
https://github.com/apache/netbeans/blob/c084119009d2e0f736f225d706bc1827af283501/ide/lexer/src/org/netbeans/spi/lexer/LexerInput.java#L257
[5]
https://github.com/apache/netbeans/blob/d1c39135a852a579f32344e486d72d5e0647317b/ide/lexer.antlr4/src/org/netbeans/spi/lexer/antlr4/LexerInputCharStream.java#L52
On 15/2/23 2:15, László Kishalmi wrote:
That could be due to virtual tokens (ANTLR token with no length)
Maybe I've done something insufficient here:
https://github.com/apache/netbeans/blob/15ad55e3f28727e64a4642971f3a980b538c6d4d/ide/lexer.antlr4/src/org/netbeans/spi/lexer/antlr4/AbstractAntlrLexerBridge.java#L152
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@netbeans.apache.org
For additional commands, e-mail: dev-h...@netbeans.apache.org
For further information about the NetBeans mailing lists, visit:
https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
