Raviston Thanasekar created SPARK-61788:
-------------------------------------------
Summary: Fix StateStoreInstanceMetricSuite depending on the state
store maintenance task of an earlier test
Key: SPARK-61788
URL: https://issues.apache.org/jira/browse/SPARK-61788
Project: Spark
Issue Type: Test
Components: Structured Streaming
Affects Versions: 5.0.0
Reporter: Raviston Thanasekar
StateStoreInstanceMetricSuiteWithRowChecksum "SPARK-51097: Verify snapshot lag
metrics are updated correctly with RocksDBStateStoreProvider (with changelog
checkpointing)" fails when another test in the same JVM has started the state
store maintenance task before it:
{code}
Assert on query failed: Execute: The code passed to eventually never returned
normally. Attempted 669 times over 10.005817232000002 seconds. Last failure
message: instanceMetrics.forall(((x$2: (String, Long)) =>
scala.Predef.Long2long(x$2._2).>=(0))) was false.
{code}
The maintenance task is shared by the JVM, and
StateStore.startMaintenanceIfNeeded only starts one when none is running. So it
keeps the interval of the test that started it. The test sets
spark.sql.streaming.stateStore.maintenanceInterval to 100 ms, but when a task
with the default interval of 60 seconds is already running, no snapshot is
uploaded during the test, and every SnapshotLastUploaded metric stays -1.
In "sql - other tests", WriteDistributionAndOrderingSuite ("ordered
distribution and sort with same exprs: micro-batch complete") started the task
just before this test, e.g. in
https://github.com/raviston/spark/actions/runs/43067912461 (attempts 10 to 12).
StateStoreInstanceMetricSuiteWithRowChecksum runs in that group because the
ExtendedSQLTest tag of StateStoreInstanceMetricSuite is not inherited.
To reproduce:
{code}
build/sbt 'sql/testOnly
org.apache.spark.sql.connector.WriteDistributionAndOrderingSuite
org.apache.spark.sql.execution.streaming.state.StateStoreInstanceMetricSuiteWithRowChecksum
-- -z "ordered distribution and sort with same exprs: micro-batch complete" -z
"SPARK-51097: Verify snapshot lag metrics are updated correctly with
RocksDBStateStoreProvider (with changelog"'
{code}
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]