Viicos commented on code in PR #2077:
URL:
https://github.com/apache/datafusion-sqlparser-rs/pull/2077#discussion_r2515439716
##########
src/tokenizer.rs:
##########
@@ -449,29 +449,6 @@ impl Word {
}
}
-#[derive(Debug, Clone, PartialEq, PartialOrd, Eq, Ord, Hash)]
-#[cfg_attr(feature = "serde", derive(Serialize, Deserialize))]
-#[cfg_attr(feature = "visitor", derive(Visit, VisitMut))]
-pub enum Whitespace {
- Space,
- Newline,
- Tab,
- SingleLineComment { comment: String, prefix: String },
- MultiLineComment(String),
-}
Review Comment:
> They just should not exist in the form of whitespace.
Agree, maybe they could be made their own token.
> Could you kindly expand on your IDE completions note? I am not sure I
understood it.
I'm currently experimenting with the tokenizer and parser to use it in a SQL
query input we have for our project at work. From a `(line, col)` position, I'm
writing some logic to provide useful completions at this position (by
completions, I mean
[this](https://microsoft.github.io/language-server-protocol/img/vscode-css-code-complete.png)).
If you are currently writing a comment, the token immediately before the
position is going to be a comment token, and in this case I can short-circuit
and returns an empty list of competions.
As an example, this is how the Ruff project (they are working on a Python
language server) [is doing
it](https://github.com/astral-sh/ruff/blob/bd8812127daa556bd86fa81c9a79f5f49a2feaa8/crates/ty_ide/src/completion.rs#L930-L933).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
