Raviston Thanasekar created SPARK-61788:
-------------------------------------------

             Summary: Fix StateStoreInstanceMetricSuite depending on the state 
store maintenance task of an earlier test
                 Key: SPARK-61788
                 URL: https://issues.apache.org/jira/browse/SPARK-61788
             Project: Spark
          Issue Type: Test
          Components: Structured Streaming
    Affects Versions: 5.0.0
            Reporter: Raviston Thanasekar


StateStoreInstanceMetricSuiteWithRowChecksum "SPARK-51097: Verify snapshot lag 
metrics are updated correctly with RocksDBStateStoreProvider (with changelog 
checkpointing)" fails when another test in the same JVM has started the state 
store maintenance task before it:

{code}
Assert on query failed: Execute: The code passed to eventually never returned 
normally. Attempted 669 times over 10.005817232000002 seconds. Last failure 
message: instanceMetrics.forall(((x$2: (String, Long)) => 
scala.Predef.Long2long(x$2._2).>=(0))) was false.
{code}

The maintenance task is shared by the JVM, and 
StateStore.startMaintenanceIfNeeded only starts one when none is running. So it 
keeps the interval of the test that started it. The test sets 
spark.sql.streaming.stateStore.maintenanceInterval to 100 ms, but when a task 
with the default interval of 60 seconds is already running, no snapshot is 
uploaded during the test, and every SnapshotLastUploaded metric stays -1.

In "sql - other tests", WriteDistributionAndOrderingSuite ("ordered 
distribution and sort with same exprs: micro-batch complete") started the task 
just before this test, e.g. in 
https://github.com/raviston/spark/actions/runs/43067912461 (attempts 10 to 12). 
StateStoreInstanceMetricSuiteWithRowChecksum runs in that group because the 
ExtendedSQLTest tag of StateStoreInstanceMetricSuite is not inherited.

To reproduce:
{code}
build/sbt 'sql/testOnly 
org.apache.spark.sql.connector.WriteDistributionAndOrderingSuite 
org.apache.spark.sql.execution.streaming.state.StateStoreInstanceMetricSuiteWithRowChecksum
 -- -z "ordered distribution and sort with same exprs: micro-batch complete" -z 
"SPARK-51097: Verify snapshot lag metrics are updated correctly with 
RocksDBStateStoreProvider (with changelog"'
{code}




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to