Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/22962#discussion_r232634698
--- Diff: python/pyspark/tests.py ---
@@ -618,10 +618,13 @@ def test_barrier_with_python_worker_reuse(self):
"""
Verify that BarrierTaskContext.barrier() with reused python worker.
"""
+ self.sc._conf.set("spark.python.work.reuse", "true")
--- End diff --
I do these 2 check like below:
1. Run this test case without fix in `BarrierTaskContext._getOrCreate`, the
bug can be reproduced.
2. Same code running in pyspark shell and set
`spark.python.work.resue=false`, it return successfully.
Maybe this can prove the UT can cover the issue and also can reuse the
original barrier case code, WDYT?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
