Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/22962#discussion_r233055939
--- Diff: python/pyspark/taskcontext.py ---
@@ -147,8 +147,8 @@ def __init__(self):
@classmethod
def _getOrCreate(cls):
"""Internal function to get or create global BarrierTaskContext."""
- if cls._taskContext is None:
- cls._taskContext = BarrierTaskContext()
+ if not isinstance(cls._taskContext, BarrierTaskContext):
+ cls._taskContext = object.__new__(cls)
--- End diff --
I think the answer is no, maybe I was not clear enough in my previous
explain https://github.com/apache/spark/pull/22962#discussion_r232528333, use
`BarrierTaskContext()` here is my first commit
https://github.com/apache/spark/pull/22962/commits/0cb2cf6e9ece66861073c31b579b595a9de5ce81
, it should also need to rewrite `__new__` for `BarrierTaskContext`, otherwise
the bug still exists cause its parent class `TaskContext` rewrite `__new__()`,
when we call `BarrierTaskContext()` here in a reused worker, a `TaskContext`
instance will be returned in
`TaskContext.__new__()`:https://github.com/apache/spark/blob/c00e72f3d7530eb2ae43d4d45e8efde783daf6ff/python/pyspark/taskcontext.py#L47
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
