[
https://issues.apache.org/jira/browse/BEAM-3645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16924585#comment-16924585
]
Yifan Mai edited comment on BEAM-3645 at 9/6/19 8:54 PM:
---------------------------------------------------------
While testing this I noticed that the multi-process runner does not handle
SIGINT gracefully. I added the repro steps to BEAM-8149.
was (Author: myffi...@gmail.com):
While testing this I noticed that the multi-process runner does not handle
SIGINT gracefully. To reproduce, run wordcount.py using the "Run with
multiprocessing mode" instructions from the comment above (in Python 3).
Expected: wordcount terminates gracefully when Ctrl-C is pressed during
pipeline execution (similarly to default direct runner)
Actual: wordcount hangs forever after printing the following once per worker:
{code}
Exception in thread run_worker:
Traceback (most recent call last):
File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/usr/lib/python3.6/threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File
"/usr/local/google/home/yifanmai/venv/wordcount/lib/python3.6/site-packages/apache_beam/runners/portability/local_job_service.py",
line 216, in run
'Worker subprocess exited with return code %s' % p.returncode)
RuntimeError: Worker subprocess exited with return code 1
{code}
> Support multi-process execution on the FnApiRunner
> --------------------------------------------------
>
> Key: BEAM-3645
> URL: https://issues.apache.org/jira/browse/BEAM-3645
> Project: Beam
> Issue Type: Improvement
> Components: sdk-py-core
> Affects Versions: 2.2.0, 2.3.0
> Reporter: Charles Chen
> Assignee: Hannah Jiang
> Priority: Major
> Fix For: 2.15.0
>
> Time Spent: 35h 20m
> Remaining Estimate: 0h
>
> https://issues.apache.org/jira/browse/BEAM-3644 gave us a 15x performance
> gain over the previous DirectRunner. We can do even better in multi-core
> environments by supporting multi-process execution in the FnApiRunner, to
> scale past Python GIL limitations.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
