I think Andrew’s point is a good one. I am personally not prepared to take on maintenance of sqlparser-rs. I wonder what Luca thinks (I’m not sure if there’s any way to tag them here).
> On Nov 11, 2026, at 7:01 AM, Andy Grove <[email protected]> wrote: > > I created a draft PR against DataFusion to use the proposed v2 parser. > All tests are now passing in DataFusion (except for a dependency check > that cannot pass until there is a published crate for the new parser). > > https://github.com/apache/datafusion/pull/27304 > > Now that this is fully working, I would like to discuss the parser and > dialect design in detail to get feedback on the approach. I've filed > https://github.com/apache/datafusion-sqlparser-rs/issues/2910, > covering: > > - how the parser is generic over the dialect > - how a dialect overrides a grammar rule so the change applies > everywhere the rule is used > - how a dialect reuses another dialect's rules by delegating to them, > one rule at a time > > Perhaps if the approach is acceptable we could start implementing in a > series of self-contained PRs against a new empty branch in the repo so > that we can review each piece as it lands. > > On Mon, Nov 9, 2026 at 1:52 PM Andrew Lamb <[email protected]> wrote: >> >> I think a more important question is who wants to maintain such a codebase. >> If we get buy in for that then I think the actual code is much less >> important >> >> On Mon, Nov 9, 2026 at 3:40 PM Andy Grove <[email protected]> wrote: >> >>> Hi Adrian, >>> >>> You guessed right. >>> >>> The high level design is genuinely mine. When I first implemented the >>> crate I was fresh from Java and was trying to use an object oriented >>> approach in Rust, which I eventually realized does not work. I wanted >>> one base parser then have other parsers extend it and override >>> specific parts of syntax while keeping the main pratt parser and >>> common code in the base. I couldn't achieve what I wanted and wasn't >>> skilled enough with generics and lifetimes. >>> >>> There is some discussion about the use of LLM in the issue as well. >>> This project is quite well suited to LLMs, IMO, because it really is >>> just refactoring existing parsing code and we have a pretty >>> comprehensive test suite, so as long as the new one passes all those >>> tests, we should have confidence that it is as correct as the current >>> version. >>> >>> I know there are lots of different opinions on how far to go with LLM >>> though, so it's a good discussion to have. >>> >>> Thanks, >>> >>> Andy. >>> >>> >>> On Mon, Nov 9, 2026 at 1:29 PM Adrian Garcia Badaracco >>> <[email protected]> wrote: >>>> >>>> I’m guessing this is mostly LLM generated code? >>>> >>>> The question I think we need to answer as a community is if we are okay >>> merging a large chunk of LLM generated code assuming we can put in place >>> the right guardrails, get it to pass all tests, all upstream tests in >>> datafusion, benchmarks look better and the high level design is accepted. >>>> >>>>> On Nov 9, 2026, at 5:46 AM, Andy Grove <[email protected]> wrote: >>>>> >>>>> Hi all, >>>>> >>>>> Over the weekend, I started from a clean slate and implemented a new >>>>> version of the sqlparser-rs crate using the design that I had >>>>> originally wanted but did not know how to build (I was still quite new >>>>> to Rust at the time). >>>>> >>>>> The new version has one trait per dialect and all dialect-specific >>>>> parsing lives in that trait. There is no longer one huge "core" parser >>>>> containing conditional logic for different dialects. Related dialects >>>>> can also share logic. For example, the Hive and Databricks dialects >>>>> both call into the Spark dialect for common pieces. >>>>> >>>>> The new version also has ~10x fewer memory allocations (no more >>>>> copying Strings everywhere) and is ~10x faster. >>>>> >>>>> I am nearly finished porting all of the existing dialects over. The >>>>> actual parsing code is taken from the current implementation, so that >>>>> is not re-written. I have ported over all the tests as well, and am >>>>> working towards 100% parity with v1, although v2 contains quite a lot >>>>> of correctness fixes compared to v1. >>>>> >>>>> What is the appetite in the community for me pushing this to a new 2.0 >>>>> branch in the Apache repo? >>>>> >>>>> Another option would be for me to file a huge issue explaining the >>>>> design in detail, and let the community rebuild it. >>>>> >>>>> WDYT? >>>>> >>>>> I filed an issue for this discssion as well, so that is probably the >>>>> best place to continue the discussion. >>>>> >>>>> https://github.com/apache/datafusion-sqlparser-rs/issues/2893 >>>>> >>>>> Thanks, >>>>> >>>>> Andy. >>>>> >>>>> --------------------------------------------------------------------- >>>>> To unsubscribe, e-mail: [email protected] >>>>> For additional commands, e-mail: [email protected] >>>>> >>>> >>>> >>>> --------------------------------------------------------------------- >>>> To unsubscribe, e-mail: [email protected] >>>> For additional commands, e-mail: [email protected] >>>> >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: [email protected] >>> For additional commands, e-mail: [email protected] >>> >>> > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
