LucaCappelletti94 commented on PR #2610:
URL:
https://github.com/apache/datafusion-sqlparser-rs/pull/2610#issuecomment-6132233811
No dialect needs this hook. Presto, the one engine named with
`events:analytics`, lexes it as a single identifier, which `is_identifier_part`
already handles. dbt models are Jinja templates, and `dbt compile` renders `{{
ref('model') }}` into a plain table name, so a SQL parser should be given the
compiled SQL.
The hook also cannot represent what it is for. It returns an `ObjectName`,
which has no `:` separator, so `events:analytics` and `events.analytics` parse
to the same AST and both print as `events.analytics`. It reaches only table
factors, so `INSERT INTO`, `CREATE TABLE`, `DROP TABLE` and `ALTER TABLE` still
reject the same name.
Today `Tokenizer` holds a `&dyn Dialect` and runs a single `next_token` for
every dialect. Each token goes through dozens of `supports_*` checks, and byte
strings, raw strings, `//` and `#` comments and `\r` comment endings are still
hardcoded behind `dialect_of!`. A dialect-generic tokenizer would be
`Tokenizer<D: Dialect + ?Sized>`, with those lexical rules moved onto `Dialect`
as overridable methods. Each built-in dialect would then get a monomorphised
lexer without the branches it never takes, a downstream dialect could add
lexemes of its own, and `Parser` could keep passing `&dyn Dialect` unchanged.
Such a refactoring would be a rather significant challenge, but would fully
address this genre of issues more cleanly, I might look into it in a month or
two if time allows, especially for a potential no-copy aspect.
I will now close this PR.
cc @alamb @iffyio if you have a different opinion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
