[
https://issues.apache.org/jira/browse/SPARK-59784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
sepuri sai krishna updated SPARK-59784:
---------------------------------------
Summary: Fix racy waits for SQL execution metrics in tests (was: Fix flaky
test driver side SQL metrics in SQLAppStatusListenerSuite)
> Fix racy waits for SQL execution metrics in tests
> -------------------------------------------------
>
> Key: SPARK-59784
> URL: https://issues.apache.org/jira/browse/SPARK-59784
> Project: Spark
> Issue Type: Test
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: sepuri sai krishna
> Priority: Major
> Labels: pull-request-available
>
> {{SQLAppStatusListenerSuite}} test "driver side SQL metrics" fails
> intermittently in CI. It has been seen repeatedly in the scheduled master and
> branch-4.x builds, in both the {{SQLAppStatusListenerWithInMemoryStoreSuite}}
> and {{SQLAppStatusListenerWithRocksDBBackendSuite}} variants, and in
> unrelated PR builds. Example failure:
> {noformat}
> - driver side SQL metrics *** FAILED *** (21 milliseconds)
> Map(3105033 -> "2", 3105032 -> "1", 3105031 -> "total (min, med, max
> (stageId: taskId))
> 0 ms (0 ms, 0 ms, 0 ms (stage 5.0: task 10))") did not contain key 3105125
> (SQLAppStatusListenerSuite.scala:607)
> {noformat}
> The wait logic in the test is racy:
> # {{while (statusStore.executionsCount() < oldCount)}} can never be true,
> because the count only grows, so it never waits.
> # {{SQLAppStatusListener}} processes events asynchronously. If it has not yet
> processed the new execution, {{statusStore.executionsList().last}} is an
> execution left over from a previous test. That execution already has metric
> values, so the second wait loop exits immediately and the test reads the
> wrong execution's metrics. The map in the failure above is exactly such a
> previous execution's metrics.
> This can be reproduced deterministically by stalling the status listener
> queue for a couple of seconds while the test runs.
> Proposed fix: capture the execution id inside
> {{SQLExecution.withNewExecutionId}} and use {{eventually}} to wait until that
> specific execution has its metrics, instead of looking at the latest
> execution in the store.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
