[
https://issues.apache.org/jira/browse/FLINK-8414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16325157#comment-16325157
]
flora karniav commented on FLINK-8414:
--------------------------------------
Thank you for the information,
I understand the fact that lower parallelism levels are sufficient for these
small datasets. But why would performance decrease with larger parallelism
values? Due to this fact, I cannot measure performance using different datasets
(with sizes that vary from MBs to GBs) with the same Flink setup and
configuration.
In addition, even if I know the Graph size a priori (using VertexMetrics), is
there a formula or some kind of standard way to decide the parallelism level
accordingly? Or is brute force the only way?
Thank you
> Gelly performance seriously decreases when using the suggested parallelism
> configuration
> ----------------------------------------------------------------------------------------
>
> Key: FLINK-8414
> URL: https://issues.apache.org/jira/browse/FLINK-8414
> Project: Flink
> Issue Type: Bug
> Components: Configuration, Documentation, Gelly
> Reporter: flora karniav
> Priority: Minor
>
> I am running Gelly examples with different datasets in a cluster of 5
> machines (1 Jobmanager and 4 Taskmanagers) of 32 cores each.
> The number of Slots parameter is set to 32 (as suggested) and the parallelism
> to 128 (32 cores*4 taskmanagers).
> I observe a vast performance degradation using these suggested settings than
> setting parallelism.default to 16 for example were the same job completes at
> ~60 seconds vs ~140 in the 128 parallelism case.
> Is there something wrong in my configuration? Should I decrease parallelism
> and -if so- will this inevitably decrease CPU utilization?
> Another matter that may be related to this is the number of partitions of the
> data. Is this somehow related to parallelism? How many partitions are created
> in the case of parallelism.default=128?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
