HyukjinKwon commented on a change in pull request #28968:
URL: https://github.com/apache/spark/pull/28968#discussion_r479612214
##########
File path: python/pyspark/util.py
##########
@@ -114,6 +117,64 @@ def _parse_memory(s):
raise ValueError("invalid format: " + s)
return int(float(s[:-1]) * units[s[-1].lower()])
+
+class InheritableThread(threading.Thread):
+ """
+ Thread that is recommended to be used in PySpark instead of
:class:`threading.Thread`
+ when the pinned thread mode is enabled. The usage of this class is exactly
same as
+ :class:`threading.Thread` but correctly inherits the inheritable
properties specific
+ to JVM thread such as ``InheritableThreadLocal``.
+
+ Also, note that pinned thread mode does not close the connection from
Python
+ to JVM when the thread is finished in the Python side. With this class,
Python
+ garbage-collects the Python thread instance and also closes the connection
+ which finishes JVM thread correctly.
+
+ When the pinned thread mode is off, this works as
:class:`threading.Thread`.
+
+ .. note:: Experimental
+
+ .. versionadded:: 3.1.0
+ """
+ def __init__(self, target, *args, **kwargs):
+ from pyspark import SparkContext
+
+ sc = SparkContext._active_spark_context
+
+ if isinstance(sc._gateway, ClientServer):
+ # Here's when the pinned-thread mode (PYSPARK_PIN_THREAD) is on.
+ properties = sc._jsc.sc().getLocalProperties().clone()
Review comment:
Actually we're mimicking that behaviour here because the thread in JVM
does not respect the inheritance here since the thread is always sepearately
created via the JVM gateway whereas Scala Java side we can keep the inheritance
by creating a thread within a thread.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
