damccorm opened a new pull request, #40822:
URL: https://github.com/apache/beam/pull/40822
## Summary
Prerequisite `MultiProcessShared(..., spawn_process=True)` improvements for
#40673, split out ahead of the `RunInference` changes:
1. **Serialize constructors with `apache_beam.internal.pickler`**:
`multiprocessing.get_context('spawn')` uses standard-library `pickle` for
`Process(args=...)`, which rejects lambdas, closures, and local callables
(`AttributeError: Can't get local object ...`). Serializing `self._constructor`
with `apache_beam.internal.pickler` allows `MultiProcessShared(lambda: ...,
spawn_process=True)` and objects holding local functions to spawn cleanly.
2. **Make `MultiProcessShared.unsafe_hard_delete()` safe for spawned
processes**:
- Return immediately as a no-op if the tag is not registered locally and
has no `.address` file, avoiding accidentally spawning a new server process
just to delete it.
- Ignore expected `(EOFError, OSError, BrokenPipeError)` disconnects when
the spawned server exits via `os._exit(0)` inside `handle_unsafe_hard_delete`,
and reap exited child processes.
3. **Run custom `__del__` finalizers before `os._exit(0)` during in-flight
calls**: If an in-flight RPC thread (such as a timed-out call) still holds a
stack reference to `self.obj` when `_SingletonEntry.unsafe_hard_delete()` runs
in a spawned server process, `del self.obj` alone does not drop the refcount to
zero before `os._exit(0)` terminates the process. Explicitly invoking
`type(obj).__del__(obj)` ensures custom cleanup hooks still execute.
### Testing
- Added unit tests in `MultiProcessSharedSpawnProcessTest`
(`multi_process_shared_test.py`) covering:
- Dedicated OS PID isolation across tags and shared PID reuse for matching
tags
- Cyclic-GC-on-`Server.accepter` deadlock regression (#40673) with
`always_proxy=True, spawn_process=True`
- Lambda and closure constructors with `spawn_process=True`
- `unsafe_hard_delete()` on non-existent tags (no-op, no extra process
spawned)
- `unsafe_hard_delete()` running `__del__` while a blocked RPC is in flight
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
