[
https://issues.apache.org/jira/browse/FLINK-7019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16078113#comment-16078113
]
ASF GitHub Bot commented on FLINK-7019:
---------------------------------------
GitHub user greghogan opened a pull request:
https://github.com/apache/flink/pull/4282
[FLINK-7019] [gelly] Rework parallelism in Gelly algorithms and examples
Flink job parallelism is set with ExecutionConfig#setParallelism or with -p
on the command-line. The Gelly algorithms JaccardIndex, AdamicAdar,
TriangleListing, and ClusteringCoefficient have intermediate operators which
generate output quadratic in the size of input. These algorithms may need to be
run with a high parallelism but doing so for all operations is wasteful. Thus
was introduced "little parallelism".
This can be simplified by moving the parallelism parameter to the new
common base class with the rule-of-thumb to use the algorithm parallelism for
all normal (small output) operators. The asymptotically large operators will
default to the job parallelism, as will the default algorithm parallelism.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/greghogan/flink
7019_rework_parallelism_in_gelly_algorithms_and_examples
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/4282.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #4282
----
commit 105ddee81892b46f5e9ab0b42bbc9cf206a71414
Author: Greg Hogan <[email protected]>
Date: 2017-06-26T14:21:50Z
[FLINK-7019] [gelly] Rework parallelism in Gelly algorithms and examples
Flink job parallelism is set with ExecutionConfig#setParallelism or with
-p on the command-line. The Gelly algorithms JaccardIndex, AdamicAdar,
TriangleListing, and ClusteringCoefficient have intermediate operators
which generate output quadratic in the size of input. These algorithms
may need to be run with a high parallelism but doing so for all
operations is wasteful. Thus was introduced "little parallelism".
This can be simplified by moving the parallelism parameter to the new
common base class with the rule-of-thumb to use the algorithm
parallelism for all normal (small output) operators. The asymptotically
large operators will default to the job parallelism, as will the default
algorithm parallelism.
----
> Rework parallelism in Gelly algorithms and examples
> ---------------------------------------------------
>
> Key: FLINK-7019
> URL: https://issues.apache.org/jira/browse/FLINK-7019
> Project: Flink
> Issue Type: Sub-task
> Components: Gelly
> Affects Versions: 1.4.0
> Reporter: Greg Hogan
> Assignee: Greg Hogan
> Priority: Minor
> Fix For: 1.4.0
>
>
> Flink job parallelism is set with {{ExecutionConfig#setParallelism}} or when
> {{-p}} on the command-line. The Gelly algorithms {{JaccardIndex}},
> {{AdamicAdar}}, {{TriangleListing}}, and {{ClusteringCoefficient}} have
> intermediate operators which generate output quadratic in the size of input.
> These algorithms may need to be run with a high parallelism but doing so for
> all operations is wasteful. Thus was introduced "little parallelism".
> This can be simplified by moving the parallelism parameter to the new common
> base class with the rule-of-thumb to use the algorithm parallelism for all
> normal (small output) operators. The asymptotically large operators will
> default to the job parallelism, as will the default algorithm parallelism.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
