[
https://issues.apache.org/jira/browse/SPARK-44462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim updated SPARK-44462:
---------------------------------
Priority: Critical (was: Major)
> Fix the session passed to foreachBatch.
> ----------------------------------------
>
> Key: SPARK-44462
> URL: https://issues.apache.org/jira/browse/SPARK-44462
> Project: Spark
> Issue Type: Task
> Components: Connect, Structured Streaming
> Affects Versions: 3.4.1
> Reporter: Raghu Angadi
> Priority: Critical
>
> foreachBatch() in Connect uses initial session used while starting the
> streaming query. But streaming query uses a cloned session, not the the
> original session. We should set up the mapping for the cloned session and
> pass that in. Look for this ticket ID in the code for more context inline.
>
> Another issue with not creating new session id: foreachBatch worker keeps the
> session alive. The session mapping at Connect server does not expire and
> query keeps running even if the original client disappears. This keeps the
> query running.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
