andygrove opened a new issue, #5294: URL: https://github.com/apache/datafusion-comet/issues/5294
Follow-up from review of #4459 ([thread](https://github.com/apache/datafusion-comet/pull/4459#discussion_r3730388880)). `CometRustUdfRegistry` is a process-wide `ConcurrentHashMap[String, RustUdfMetadata]` keyed by bare function name, with no session scoping and no unregister path. As @mbutrovich points out, the [contributor guide's "Global singletons" section](https://datafusion.apache.org/comet/contributor-guide/development.html#global-singletons) lists this shape as one to avoid: the metadata it holds (library path, declared input and return types) is per-registration configuration that varies by caller, which is the guide's "state depends on configuration that can vary between jobs or queries" case. Concretely, in any JVM hosting more than one session — a Spark Connect server, a notebook, a test JVM running several suites — one session registering a Rust UDF named `transform` claims that name for every other session for the life of the process, and the last registration of a name wins for all of them. Registration should be scoped to the session (or `SparkContext`) that performed it, so that a name registered in one session is not visible to another, and entries go away with the session rather than living for the life of the JVM. This is a driver-side concern only: executors never consult the registry, since the library path travels with the plan in the `RustUdfCall` proto. Related to the name-matching half of the same problem (an ordinary Scala UDF registered under a Rust UDF's name is answered out of the Rust library). A fix that gives each registration an identity Comet can recognize may address both. `CometRustUdfRegistry.instance` currently carries a comment explaining the lifetime and bounds, per the guide's requirement for a singleton that is kept anyway; that comment should come out with the fix. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
