[
https://issues.apache.org/jira/browse/BEAM-3645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16896575#comment-16896575
]
Hannah Jiang edited comment on BEAM-3645 at 7/30/19 10:20 PM:
--------------------------------------------------------------
# pending review to make sure this summary is 100% correct.
Previously, shuffle data, which we call buffer as well with FnApi, is processed
by a single worker. Now, the shuffle data, except WindowGroupingBuffer, is
partitioned to N chunks and each chunk is processed by a worker, thus
processing performance is improved. Depending on running environment,
partitioned shuffles can be multi-thread or multi-process processed.
Fn Api provides following running environment.
1. EmbeddedWorkerHandler
2. EmbeddedGrpcWorkerHandler
3. SubprocessSdkWorkerHandler
4. DockerSdkWorkerHandler
5. ExternalWorkerHandler
Workers running with SubprocessSdkWorkerHandler are running at processes, so if
we want to multi-process shuffle data, we should use this environment. Workers
running with other worker handlers are running at a single process.
*--direct_num_workers* option is used to define number of partitions of
shuffle, default value is 1.
was (Author: hannahjiang):
# pending review to make sure this summary is 100% correct.
Previously, shuffle data, which we call buffer as well with FnApi, is processed
by a single worker. Now, the shuffle data, except WindowGroupingBuffer, is
partitioned to N chunks and each chunk is processed by a worker, thus
processing performance is improved. Depending on running environment,
partitioned shuffles can be multi-thread or multi-process processed.
Fn Api provides following running environment.
1. EmbeddedWorkerHandler
2. EmbeddedGrpcWorkerHandler
3. SubprocessSdkWorkerHandler
4. DockerSdkWorkerHandler
5. ExternalWorkerHandler
Workers running with SubprocessSdkWorkerHandler are running at processes, so if
we want to multi-process shuffle data, we should use this environment. Workers
running with other worker handlers are running at a single process.
*--direct_num_workers* option is used to define number of partitions of
shuffle, default value is 1.
> Support multi-process execution on the FnApiRunner
> --------------------------------------------------
>
> Key: BEAM-3645
> URL: https://issues.apache.org/jira/browse/BEAM-3645
> Project: Beam
> Issue Type: Improvement
> Components: sdk-py-core
> Affects Versions: 2.2.0, 2.3.0
> Reporter: Charles Chen
> Assignee: Hannah Jiang
> Priority: Major
> Fix For: 2.15.0
>
> Time Spent: 35h
> Remaining Estimate: 0h
>
> https://issues.apache.org/jira/browse/BEAM-3644 gave us a 15x performance
> gain over the previous DirectRunner. We can do even better in multi-core
> environments by supporting multi-process execution in the FnApiRunner, to
> scale past Python GIL limitations.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
