HyukjinKwon opened a new pull request, #46993:
URL: https://github.com/apache/spark/pull/46993
### What changes were proposed in this pull request?
This PR propose to avoid having
`ExecutePlanResponseReattachableIterator._release_thread_pool` to initialize
`ThreadPool`.
### Why are the changes needed?
This instance might be dragged in during pickle because it's statically
initialized.
```
_release_thread_pool: Optional[ThreadPool] = ThreadPool(os.cpu_count()
if os.cpu_count() else 8)
File "/usr/lib/python3.10/multiprocessing/pool.py", line 930, in __init__
Pool.__init__(self, processes, initializer, initargs)
File "/usr/lib/python3.10/multiprocessing/pool.py", line 196, in __init__
self._change_notifier = self._ctx.SimpleQueue()
File "/usr/lib/python3.10/multiprocessing/context.py", line 113, in
SimpleQueue
return SimpleQueue(ctx=self.get_context())
File "/usr/lib/python3.10/multiprocessing/queues.py", line 341, in __init__
self._rlock = ctx.Lock()
File "/usr/lib/python3.10/multiprocessing/context.py", line 68, in Lock
return Lock(ctx=self.get_context())
File "/usr/lib/python3.10/multiprocessing/synchronize.py", line 162, in
__init__
SemLock.__init__(self, SEMAPHORE, 1, 1, ctx=ctx)
File "/usr/lib/python3.10/multiprocessing/synchronize.py", line 57, in
__init__
sl = self._semlock = _multiprocessing.SemLock(
PermissionError: [Errno 13] Permission denied
```
which requires to change in OS level.
### Does this PR introduce _any_ user-facing change?
Yeah, potentially this could trigger some random job failures in some
environment like Ubuntu
### How was this patch tested?
Manually tested.
### Was this patch authored or co-authored using generative AI tooling?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
