Mukul Murthy created SPARK-26586:
------------------------------------
Summary: Streaming queries should have isolated SparkSessions and
confs
Key: SPARK-26586
URL: https://issues.apache.org/jira/browse/SPARK-26586
Project: Spark
Issue Type: Bug
Components: SQL, Structured Streaming
Affects Versions: 2.4.0, 2.3.0
Reporter: Mukul Murthy
When a stream is started, the stream's config is supposed to be frozen and all
batches run with the config at start time. However, due to a race condition in
creating streams, updating a conf value in the active spark session immediately
after starting a stream can lead to the stream getting that updated value.
The problem is that when StreamingQueryManager creates a MicrobatchExecution
(or ContinuousExecution), it passes in the shared spark session, and the spark
session isn't cloned until StreamExecution.start() is called.
DataStreamWriter.start() should not return until the SparkSession is cloned.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
