[
https://issues.apache.org/jira/browse/SPARK-44462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18121329#comment-18121329
]
Uroš Bojanić commented on SPARK-44462:
--------------------------------------
Hi folks, [~LuciferYang] [~kabhwan] [~szehon] - should we block 4.2.1, 4.1.4,
etc. maintenance releases on this change
(https://github.com/apache/spark/pull/55410) or not?
> Fix the session passed to foreachBatch.
> ----------------------------------------
>
> Key: SPARK-44462
> URL: https://issues.apache.org/jira/browse/SPARK-44462
> Project: Spark
> Issue Type: Task
> Components: Connect, Structured Streaming
> Affects Versions: 3.4.1
> Reporter: Raghu Angadi
> Priority: Blocker
> Labels: pull-request-available
>
> foreachBatch() in Connect uses initial session used while starting the
> streaming query. But streaming query uses a cloned session, not the the
> original session. We should set up the mapping for the cloned session and
> pass that in. Look for this ticket ID in the code for more context inline.
>
> Another issue with not creating new session id: foreachBatch worker keeps the
> session alive. The session mapping at Connect server does not expire and
> query keeps running even if the original client disappears. This keeps the
> query running.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
