HyukjinKwon opened a new pull request #32962: URL: https://github.com/apache/spark/pull/32962
### What changes were proposed in this pull request? This PR is a followup of https://github.com/apache/spark/pull/32429 and https://github.com/apache/spark/pull/32644. I was thinking about creating separate PRs but decided to include all in this PR because it shares the same context, and should be easier to review together. This PR includes: - Use `InheritableThread` and `inheritable_thread_target` in the current code base to prevent potential resource leak (since we enabled pinned thread mode by default now at https://github.com/apache/spark/pull/32429) - Enable pinned thread mode by default. The initial PR missed some please at `inheritable_thread_target`. - Change `InheritableThread`: - Copy local properties when `InheritableThread.start` is called to mimic JVM behaviour. Previously it was copied when `InheritableThread` instance was created. - Clean Py4J connections on `__del__` when garbage-collecting instead of each target function invocation. This is a regression from https://github.com/apache/spark/pull/32644. ### Why are the changes needed? To mimic the JVM behaviour about thread lifecycle ### Does this PR introduce _any_ user-facing change? Ideally no. One possible case is that users use `InheritableThread` with pinned thread mode enabled. In this case, the local properties will be copied when starting the thread instead of defining the `InheritableThread` object. This is a small difference that wouldn't likely affect end users. It doesn't affect about cleaning logic because Spark 3.1 already cleans up in `__del__`. It was mistakenly fixed in https://github.com/apache/spark/pull/32644. ### How was this patch tested? Existing tests should cover this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
