HeartSaVioR commented on code in PR #43831:
URL: https://github.com/apache/spark/pull/43831#discussion_r1395304223
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/v2/state/StateDataSourceTestBase.scala:
##########
@@ -18,16 +18,27 @@ package org.apache.spark.sql.execution.datasources.v2.state
import java.sql.Timestamp
+import org.scalatest.BeforeAndAfter
+
import org.apache.spark.sql.{DataFrame, Dataset}
import org.apache.spark.sql.execution.streaming.MemoryStream
+import org.apache.spark.sql.execution.streaming.state.StateStore
import org.apache.spark.sql.functions._
import org.apache.spark.sql.internal.SQLConf
import org.apache.spark.sql.streaming._
import org.apache.spark.sql.streaming.util.StreamManualClock
-trait StateDataSourceTestBase extends StreamTest with StateStoreMetricsTest {
+trait StateDataSourceTestBase extends StreamTest with BeforeAndAfter with
StateStoreMetricsTest {
Review Comment:
The reason we have to clean up StateStore per test is due to maintenance
task. When we run the streaming query, state store is being initialized in to
the executor, and registration is performed against the coordinator in driver.
The lifecycle of the state store provider is not strictly tied to the the
lifecycle of the streaming query - the executor closes the state store provider
when coordinator indicates to the executor that the state store provider is no
longer valid, which is not immediately after the streaming query has stopped.
The lifecycle of the state store provider can overlap among tests.
This means maintenance task against the provider can run after test A. We
are clearing the temp directory in test A after the test A has completed, which
can break the operation being performed against state store provider being used
in test A. E.g. directory does not exist while maintenance task is running.
This won't be an issue in practice because we do not expect the checkpoint
location to be temporary, but it is indeed an issue for how we setup and
cleanup env for tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
