damccorm commented on code in PR #37112:
URL: https://github.com/apache/beam/pull/37112#discussion_r2691816552
##########
sdks/python/apache_beam/utils/multi_process_shared.py:
##########
@@ -200,9 +226,99 @@ def __call__(self, *args, **kwargs):
def __getattr__(self, name):
return getattr(self._proxyObject, name)
+ def __setstate__(self, state):
+ self.__dict__.update(state)
+
+ def __getstate__(self):
+ return self.__dict__
Review Comment:
I think it would only be valid if you unpickle onto the same machine (and
maybe even in the same process). Could you remind me what unpickling issues you
ran into?
##########
sdks/python/apache_beam/utils/multi_process_shared.py:
##########
@@ -79,6 +83,10 @@ def singletonProxy_release(self):
assert self._SingletonProxy_valid
self._SingletonProxy_valid = False
+ def unsafe_hard_delete(self):
Review Comment:
Ok - lets at least give it a name like `singletonProxy_unsafe_hard_delete`.
Otherwise we will run into issues if someone has an object with a function or
property called `unsafe_hard_delete`, which seems like it could happen.
##########
sdks/python/apache_beam/utils/multi_process_shared.py:
##########
@@ -200,9 +226,99 @@ def __call__(self, *args, **kwargs):
def __getattr__(self, name):
return getattr(self._proxyObject, name)
+ def __setstate__(self, state):
+ self.__dict__.update(state)
+
+ def __getstate__(self):
+ return self.__dict__
+
def get_auto_proxy_object(self):
return self._proxyObject
+ def unsafe_hard_delete(self):
+ try:
+ self._proxyObject.unsafe_hard_delete()
+ except (EOFError, ConnectionResetError, BrokenPipeError):
+ pass
+ except Exception as e:
+ logging.warning(
+ "Exception %s when trying to hard delete shared object proxy", e)
+
+
+def _run_server_process(address_file, tag, constructor, authkey):
+ """
+ Runs in a separate process.
+ Includes a 'Suicide Pact' monitor: If parent dies, I die.
+ """
+ parent_pid = os.getppid()
+
+ def cleanup_files():
+ logging.info("Server process exiting. Deleting files for %s", tag)
+ try:
+ if os.path.exists(address_file):
+ os.remove(address_file)
+ if os.path.exists(address_file + ".error"):
+ os.remove(address_file + ".error")
+ except Exception:
+ pass
+
+ def handle_unsafe_hard_delete():
+ cleanup_files()
+ os._exit(0)
+
+ def _monitor_parent():
+ """Checks if parent is alive every second."""
+ while True:
+ try:
+ os.kill(parent_pid, 0)
Review Comment:
> if alive nothing happens.
Could you help me understand why this happens?
https://www.geeksforgeeks.org/python/python-os-kill-method/ seems to say this
will actually send the kill signal. Does the parent just ignore it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
