[
https://issues.apache.org/jira/browse/FLINK-1476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14306879#comment-14306879
]
Fabian Hueske commented on FLINK-1476:
--------------------------------------
Hi Xuhong,
your memory configuration does not look optimal.
Your machines have 32GB each and you assign 2GB to each task manager. There is
only one TM running on each node, so the total memory given to Flink is 6GB.
Each TM allocates approx. 70% of its memory for in-memory processing and
divides it among all slots (24 for each TM).
Therefore, the memory for each slot is about (2GB * 0.7) / 24 = 60MB which is
not a lot.
If you run the program with a DOP of 16, Flink will use 60MB * 16 = 960MB for
in-memory processing and put all other data to disk.
> Flink VS Spark on loop test
> ---------------------------
>
> Key: FLINK-1476
> URL: https://issues.apache.org/jira/browse/FLINK-1476
> Project: Flink
> Issue Type: Test
> Affects Versions: 0.7.0-incubating, 0.8
> Environment: 3 machines, every machines has 24 CPU cores and allocate
> 16 CPU cores for the tests. The memory situation is: 3 * 32G
> Reporter: xuhong
> Priority: Critical
>
> In the last days, i did some test on flink and spark. The test results
> shows that flink can do better on many operations, such as GroupBy, Join and
> some complex jobs. But when I do the KMeans, LinearRegression and other loop
> tests, i found that flink is no more excellent than spark. I want to konw,
> whether flink is more comfortable to do the loop jobs with spark.
> I add code: env.setDegreeOfParallelism(16) in each test to allocate same
> CPU cores as in Spark tests.
> My english is not good, i wish you guys can understand me!
> the following is some config of my Flnk:
> jobmanager.rpc.port: 6123
> jobmanager.heap.mb: 2048
> taskmanager.heap.mb: 2048
> taskmanager.numberOfTaskSlots: 24
> parallelization.degree.default: 72
> jobmanager.web.port: 8081
> webclient.port: 8085
> fs.overwrite-files: true
> taskmanager.memory.fraction: 0.8
> taskmanager.network.numberofBuffers: 70000
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
