HyukjinKwon opened a new pull request #31723:
URL: https://github.com/apache/spark/pull/31723
### What changes were proposed in this pull request?
`TaskContextTestsWithWorkerReuse.test_task_context_correct_with_python_worker_reuse`
can be flaky and fails sometimes:
```
======================================================================
ERROR [1.798s]: test_task_context_correct_with_python_worker_reuse
(pyspark.tests.test_taskcontext.TaskContextTestsWithWorkerReuse)
...
test_task_context_correct_with_python_worker_reuse
self.assertTrue(pid in worker_pids)
AssertionError: False is not true
----------------------------------------------------------------------
```
I suspect that the Python worker was killed for whatever reason and new
attempt created a new Python worker.
Reusing Python workers actually sort of makes the best effort reuse the
Python workers. For example, if a Python worker is failed, the task will be
relaunched, and then the Python workers could have different process IDs.
This PR fixes the flakiness simply by retrying the test case.
### Why are the changes needed?
To make the tests more robust.
### Does this PR introduce _any_ user-facing change?
No, dev-only.
### How was this patch tested?
Manually tested it by controlling the conditions manually in the test codes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
