goldmedal commented on issue #10855: URL: https://github.com/apache/datafusion/issues/10855#issuecomment-2171124147
Hi @alamb, I just want to share how I use user-defined AnalyzerRule. As you know, I'm working on reimplementing the semantic layer engine, [wren-engine](https://github.com/Canner/wren-engine), for LLM using DataFusion. The project is still a work in progress. However, I think I have finished the part of integration with DataFusion. I think it could be a nice use case for `AnalyzerRule`. The basic concept of wren-engine is that user can define a virtual modeling layer to apply on their physical data. For example, given a csv file registered in DataFusion context called, `datafusion.public.customers`. We can wrap a virtual table for it called, `wrenai.default.customers_model`. User can query it like ```sql select * from wrenai.default.customers_model ``` Then , wren engine translate it to a physical query: ```sql SELECT "customers_model"."city", "customers_model"."id", "customers_model"."state" FROM ( SELECT "customers_model"."city", "customers_model"."id", "customers_model"."state" FROM ( SELECT "datafusion"."public"."customers"."city" AS "city", "datafusion"."public"."customers"."id" AS "id", "datafusion"."public"."customers"."state" AS "state" FROM "public"."customers" ) AS "customers_model" ) AS "customers_model" ``` It's a simple use case to show how it works. We have other features, such as `relationship` between models or calculated fields in the model. I don't go into detail about them here. However, I implemented all of them through `AnalyzerRule` and `UserDefinedLogicalNode`. The related PR, https://github.com/Canner/wren-engine/pull/613, is still under review. However, you can find the usage of `AnalyzerRule` in [rule.rs](https://github.com/goldmedal/wren-engine/blob/bugfix/fix-to-one-relation-refactor-chain/wren-modeling-rs/core/src/logical_plan/analyze/rule.rs) and the plan node in [plan.rs](https://github.com/goldmedal/wren-engine/blob/bugfix/fix-to-one-relation-refactor-chain/wren-modeling-rs/core/src/logical_plan/analyze/plan.rs). I also added [some examples](https://github.com/goldmedal/wren-engine/tree/bugfix/fix-to-one-relation-refactor-chain/wren-modeling-rs/wren-example) to show how to use the API or integrate with the DataFusion query flow. Thanks to DataFusion for providing such amazing features, allowing me to implement a more stable and structured converter for the semantic layer. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
