LuciferYang commented on PR #55410: URL: https://github.com/apache/spark/pull/55410#issuecomment-5927511180
Thanks @HeartSaVioR, that works for me. Added `spark.sql.connect.streaming.foreachBatch.useClonedSession` (default true, so the fix is effective by default). Setting it to false restores the previous behavior: the batch DataFrame stays on the root session, so it can still be combined with the root session, at the cost of not running under the stream session's configuration. The migration guide points at the config, so users who hit the two blocked patterns can disable it and keep their queries while they migrate the cluster. On the close()-recoverability nit: the cloned holder is torn down via the shared `closeSession`, which removes it from the session store before `SessionHolder.close()` runs, and `SessionHolder.close()` is single-shot (the `closedTimeMs` guard). So retaining the reference would not enable a real retry without changing that shared teardown ordering, which is out of scope here. `close()` runs on query termination and is intentionally best-effort: a failure is caught and logged as a possible leak, the same as any session whose `close()` fails. I would keep it as is rather than add a brittle failure-path test that has to mock the global session manager. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
