HyukjinKwon opened a new pull request, #57148: URL: https://github.com/apache/spark/pull/57148
[DO-NOT-MERGE] This is a flaky-test fix under CI verification; the title will be updated with the JIRA id once the fork CI run confirms the fix. Please do not merge yet. ### What changes were proposed in this pull request? Fix the flaky test `pyspark.sql.tests.test_python_datasource.PythonDataSourceTests.test_data_source_writer_with_logging` on `branch-4.1` by backporting the assertion handling already present on `master` and `branch-4.2` (introduced with SPARK-55799). The test writes the abort-path DataFrame with `repartitionByRange(2, "age")`, producing two write tasks that each log `"TestJsonWriter.write: abort test"`, and the expected-logs list hardcodes that warning **twice**. When the first partition's task raises, the executor cancels the remaining task, so whether the second `"abort test"` log record is captured before cancellation is timing dependent. The fix splits the assertion: - `non_abort_logs` (everything except the abort warning) is still asserted exactly, and - `abort_logs.dropDuplicates(["msg"])` is asserted to contain exactly one row, so the deterministic logs are checked precisely while the abort warning is checked for presence rather than an exact, race-dependent count. ### Why are the changes needed? `build_python_3.11.yml` on `branch-4.1` intermittently failed with `[DIFFERENT_ROWS] Results do not match: ( 8.33333 % )` (11 vs 12 rows), the missing row being one of the two identical `"TestJsonWriter.write: abort test"` entries. `master` and `branch-4.2` already handle this; `branch-4.1` was missing the backport. ### Does this PR introduce _any_ user-facing change? No. Test-only change. ### How was this patch tested? Existing test `test_data_source_writer_with_logging`. The assertion block is made identical to the one already on `master`/`branch-4.2`; verified via a `build_python_3.11` run on a fork. ### Was this patch authored or co-authored using generative AI tooling? No. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
