guan404ming commented on PR #61111: URL: https://github.com/apache/airflow/pull/61111#issuecomment-3825071591
Thanks for the feedback! Totally valid concern regarding the new dependency. > Do you know if there are more popular alternatives? Here is some alternatives but most of them are either support specific dialects or lack of maintenance. After this investigation, I then decide to start to build by myself. - [node-sql-parser](https://github.com/taozhi8833998/node-sql-parser) --> current pkg, large and have some compat issue for different dialects - [pgsql-parser](https://github.com/constructive-io/pgsql-parser) --> only support pg - [dt-sql-parser](https://github.com/DTStack/dt-sql-parser) --> build for flink / spark - [js-sql-parser](https://github.com/JavaScriptor/js-sql-parser) --> outdated, last release is 2 years ago - [antlr4ts](https://github.com/tunnelvisionlabs/antlr4ts) --> need built from scratch, which is high maintenance needed > If there is not what would it take to directly embed datafusion-sqlparser-rs within airflow without relying on an external package I've also consider this option before. But since the core logic of `datafusion-sqlparser-rs` is written on Rust, embedding it directly would force Airflow to introduce a Rust toolchain and WASM build steps (like `wasm-pack`) into the CI/CD pipeline, which increases the barrier and lots code. So packaging it separately to abstracts this complexity away, keeping the Airflow focus on Python/JS is kind of better solution if we want to take speed and size advantage of the `datafusion-sqlparser-rs` > I don't feel super confident adding a third party dependency freshly created and that hasn't a track record of 'working' in production. (and being popular amongst the community). I completely understand the hesitation with relying on a fresh personal repo. I will continue promoting this package to other projects to build a stronger track record. My ideal target is try to donate and transfer this package to the Apache Datafusion upstream. This would make it into an community-governed artifact, removing the dependency on a single individual. However I hope we can still keep this conversation open. I’d really appreciate it if you could give the pkg a spin to see the value it adds. Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
