[
https://issues.apache.org/jira/browse/SPARK-44462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Raghu Angadi updated SPARK-44462:
---------------------------------
Description:
foreachBatch() in Connect uses initial session used while starting the
streaming query. But streaming query uses a cloned session, not the the
original session. We should set up the mapping for the cloned session and pass
that in. Look for this ticket ID in the code for more context inline.
Another issue with not creating new session id: foreachBatch worker keeps the
session alive. The session mapping at Connect server does not expire and query
keeps running even if the original client disappears. This keeps the query
running.
was:foreachBatch() in Connect uses initial session used while starting the
streaming query. But streaming query uses a cloned session, not the the
original session. We should set up the mapping for the cloned session and pass
that in. Look for this ticket ID in the code for more context inline.
> Fix the session passed to foreachBatch.
> ----------------------------------------
>
> Key: SPARK-44462
> URL: https://issues.apache.org/jira/browse/SPARK-44462
> Project: Spark
> Issue Type: Task
> Components: Connect, Structured Streaming
> Affects Versions: 3.4.1
> Reporter: Raghu Angadi
> Priority: Major
>
> foreachBatch() in Connect uses initial session used while starting the
> streaming query. But streaming query uses a cloned session, not the the
> original session. We should set up the mapping for the cloned session and
> pass that in. Look for this ticket ID in the code for more context inline.
>
> Another issue with not creating new session id: foreachBatch worker keeps the
> session alive. The session mapping at Connect server does not expire and
> query keeps running even if the original client disappears. This keeps the
> query running.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
