mwiewior opened a new issue, #1077: URL: https://github.com/apache/datafusion-comet/issues/1077
### What is the problem the feature request solves? Extending Comet with custom operators requires two-step procedure: 1. Creating a custom `SparkSessionExtensions` class responsible for modification of SparkPlan object. This is not trivial especially if you would like to apply custom rules on top plan that has been already processed by the rules from `CometSparkSessionExtensions` it requires rerun of native block merging, re-serialization of native operators for a given block, etc - not easy but doable. 2. The problem is with extending `PhysicalPlanner` and `create_plan` function that does the mapping of Opstruct to native implementation of operators. ### Describe the potential solution There is a long discussion (in many threads in the Internet) about the way plugin mechanism should be implemented in Rust applications given the problem of stability of Application Binary Interface (ABI), memory safety aspects, etc. See [this](https://www.arroyo.dev/blog/rust-plugin-systems), [that](https://mayer-pu.medium.com/in-a-recent-project-we-encountered-an-issue-that-required-dynamic-loading-of-different-runtime-2b58aab9f6ad) and [that](https://nullderef.com/series/rust-plugins/) . Seems like all approaches have their pros and cons and I'm too inexperienced in Rust that I could judge which is the best in Comet case. I'm happy to help to implement it once we agree upon the strategy. ### Additional context _No response_ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
