jiayuasu commented on issue #3044: URL: https://github.com/apache/sedona/issues/3044#issuecomment-4697881396
Confirmed and root-caused. This is a regression from #2642 (first shipped in 1.9.0). **Root cause.** Sedona registers each aggregate (`ST_Envelope_Aggr`, etc.) in two places: the real UDAF goes into the *session's* function registry via `udf.register`, and a non-invocable placeholder goes into the JVM-global `FunctionRegistry.builtin` (added in #2420 for permanent-view support). Every new `SparkSession` clones `FunctionRegistry.builtin` into its own registry, so any session created after the first `SedonaContext.create()` in the JVM starts out holding that throwing placeholder under the aggregate names. #2642 added a `functionExists()` guard to silence re-registration warnings. That guard sees the cloned placeholder, concludes the function is already registered, and skips `udf.register` — so the real UDAF is never installed and resolution hits the placeholder, producing `Aggregate function ST_Envelope_Aggr cannot be used as a regular function`. In 1.8.1 the registration was unconditional, so it always overwrote the cloned placeholder; that's the behavioral difference your bisect found. **One clarification on the repro:** your `test.py` calls `sedona.stop()` and creates a new session between call 1 and call 2, so call 2 runs in a *second* server-side session — that's the trigger, not "the second call." It isn't Connect-specific either; plain `spark.newSession()` + `SedonaContext.create` reproduces it. Connect just makes new sessions the default. Note also that on a shared Connect server, once any client has connected, even the *first* aggregate call in a subsequent session can fail — so it may appear intermittent. **Verified end-to-end** against a Spark Connect server (Spark 4.0.2): your scenario fails on the released 1.9.0 shaded jar with the exact stack trace above, and passes across repeated sessions on the patched build. Fix in #3050. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
