sepuri sai krishna created SPARK-59784:
------------------------------------------
Summary: Fix flaky test driver side SQL metrics in
SQLAppStatusListenerSuite
Key: SPARK-59784
URL: https://issues.apache.org/jira/browse/SPARK-59784
Project: Spark
Issue Type: Test
Components: SQL
Affects Versions: 5.0.0
Reporter: sepuri sai krishna
{{SQLAppStatusListenerSuite}} test "driver side SQL metrics" fails
intermittently in CI. It has been seen repeatedly in the scheduled master and
branch-4.x builds, in both the {{SQLAppStatusListenerWithInMemoryStoreSuite}}
and {{SQLAppStatusListenerWithRocksDBBackendSuite}} variants, and in unrelated
PR builds. Example failure:
{noformat}
- driver side SQL metrics *** FAILED *** (21 milliseconds)
Map(3105033 -> "2", 3105032 -> "1", 3105031 -> "total (min, med, max
(stageId: taskId))
0 ms (0 ms, 0 ms, 0 ms (stage 5.0: task 10))") did not contain key 3105125
(SQLAppStatusListenerSuite.scala:607)
{noformat}
The wait logic in the test is racy:
# {{while (statusStore.executionsCount() < oldCount)}} can never be true,
because the count only grows, so it never waits.
# {{SQLAppStatusListener}} processes events asynchronously. If it has not yet
processed the new execution, {{statusStore.executionsList().last}} is an
execution left over from a previous test. That execution already has metric
values, so the second wait loop exits immediately and the test reads the wrong
execution's metrics. The map in the failure above is exactly such a previous
execution's metrics.
This can be reproduced deterministically by stalling the status listener queue
for a couple of seconds while the test runs.
Proposed fix: capture the execution id inside
{{SQLExecution.withNewExecutionId}} and use {{eventually}} to wait until that
specific execution has its metrics, instead of looking at the latest execution
in the store.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
