I created a draft PR against DataFusion to use the proposed v2 parser. All tests are now passing in DataFusion (except for a dependency check that cannot pass until there is a published crate for the new parser).
https://github.com/apache/datafusion/pull/27304 Now that this is fully working, I would like to discuss the parser and dialect design in detail to get feedback on the approach. I've filed https://github.com/apache/datafusion-sqlparser-rs/issues/2910, covering: - how the parser is generic over the dialect - how a dialect overrides a grammar rule so the change applies everywhere the rule is used - how a dialect reuses another dialect's rules by delegating to them, one rule at a time Perhaps if the approach is acceptable we could start implementing in a series of self-contained PRs against a new empty branch in the repo so that we can review each piece as it lands. On Mon, Nov 9, 2026 at 1:52 PM Andrew Lamb <[email protected]> wrote: > > I think a more important question is who wants to maintain such a codebase. > If we get buy in for that then I think the actual code is much less > important > > On Mon, Nov 9, 2026 at 3:40 PM Andy Grove <[email protected]> wrote: > > > Hi Adrian, > > > > You guessed right. > > > > The high level design is genuinely mine. When I first implemented the > > crate I was fresh from Java and was trying to use an object oriented > > approach in Rust, which I eventually realized does not work. I wanted > > one base parser then have other parsers extend it and override > > specific parts of syntax while keeping the main pratt parser and > > common code in the base. I couldn't achieve what I wanted and wasn't > > skilled enough with generics and lifetimes. > > > > There is some discussion about the use of LLM in the issue as well. > > This project is quite well suited to LLMs, IMO, because it really is > > just refactoring existing parsing code and we have a pretty > > comprehensive test suite, so as long as the new one passes all those > > tests, we should have confidence that it is as correct as the current > > version. > > > > I know there are lots of different opinions on how far to go with LLM > > though, so it's a good discussion to have. > > > > Thanks, > > > > Andy. > > > > > > On Mon, Nov 9, 2026 at 1:29 PM Adrian Garcia Badaracco > > <[email protected]> wrote: > > > > > > I’m guessing this is mostly LLM generated code? > > > > > > The question I think we need to answer as a community is if we are okay > > merging a large chunk of LLM generated code assuming we can put in place > > the right guardrails, get it to pass all tests, all upstream tests in > > datafusion, benchmarks look better and the high level design is accepted. > > > > > > > On Nov 9, 2026, at 5:46 AM, Andy Grove <[email protected]> wrote: > > > > > > > > Hi all, > > > > > > > > Over the weekend, I started from a clean slate and implemented a new > > > > version of the sqlparser-rs crate using the design that I had > > > > originally wanted but did not know how to build (I was still quite new > > > > to Rust at the time). > > > > > > > > The new version has one trait per dialect and all dialect-specific > > > > parsing lives in that trait. There is no longer one huge "core" parser > > > > containing conditional logic for different dialects. Related dialects > > > > can also share logic. For example, the Hive and Databricks dialects > > > > both call into the Spark dialect for common pieces. > > > > > > > > The new version also has ~10x fewer memory allocations (no more > > > > copying Strings everywhere) and is ~10x faster. > > > > > > > > I am nearly finished porting all of the existing dialects over. The > > > > actual parsing code is taken from the current implementation, so that > > > > is not re-written. I have ported over all the tests as well, and am > > > > working towards 100% parity with v1, although v2 contains quite a lot > > > > of correctness fixes compared to v1. > > > > > > > > What is the appetite in the community for me pushing this to a new 2.0 > > > > branch in the Apache repo? > > > > > > > > Another option would be for me to file a huge issue explaining the > > > > design in detail, and let the community rebuild it. > > > > > > > > WDYT? > > > > > > > > I filed an issue for this discssion as well, so that is probably the > > > > best place to continue the discussion. > > > > > > > > https://github.com/apache/datafusion-sqlparser-rs/issues/2893 > > > > > > > > Thanks, > > > > > > > > Andy. > > > > > > > > --------------------------------------------------------------------- > > > > To unsubscribe, e-mail: [email protected] > > > > For additional commands, e-mail: [email protected] > > > > > > > > > > > > > --------------------------------------------------------------------- > > > To unsubscribe, e-mail: [email protected] > > > For additional commands, e-mail: [email protected] > > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: [email protected] > > For additional commands, e-mail: [email protected] > > > > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
