[
https://issues.apache.org/jira/browse/FLINK-1476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14306863#comment-14306863
]
Stephan Ewen commented on FLINK-1476:
-------------------------------------
This is a known problem in Flink, but we are actually already in the process of
fixing it. You can expect a fix in the next weeks.
Until then, I would suggest to write loops through Flink's built-in iteration
construct http://flink.apache.org/docs/0.8/iterations.html
To support the long pipelines created by for loops, we need to change two
things:
- Flink schedules many tasks together. This technique is what you want for
streaming, but for batch programs, we need to break the job down into multiple
pipelines.
- The memory planning is overly conservative. To guarantee that operators do
not crash with out of memory exceptions, we restrict their allowed memory
budget. Currently, we restrict it too heavily. A fix for that is also in the
making.
> Flink VS Spark on loop test
> ---------------------------
>
> Key: FLINK-1476
> URL: https://issues.apache.org/jira/browse/FLINK-1476
> Project: Flink
> Issue Type: Test
> Affects Versions: 0.7.0-incubating, 0.8
> Environment: 3 machines, every machines has 24 CPU cores and allocate
> 16 CPU cores for the tests. The memory situation is: 3 * 32G
> Reporter: xuhong
> Priority: Critical
>
> In the last days, i did some test on flink and spark. The test results
> shows that flink can do better on many operations, such as GroupBy, Join and
> some complex jobs. But when I do the KMeans, LinearRegression and other loop
> tests, i found that flink is no more excellent than spark. I want to konw,
> whether flink is more comfortable to do the loop jobs with spark.
> I add code: env.setDegreeOfParallelism(16) in each test to allocate same
> CPU cores as in Spark tests.
> My english is not good, i wish you guys can understand me!
> the following is some config of my Flnk:
> jobmanager.rpc.port: 6123
> jobmanager.heap.mb: 2048
> taskmanager.heap.mb: 2048
> taskmanager.numberOfTaskSlots: 24
> parallelization.degree.default: 72
> jobmanager.web.port: 8081
> webclient.port: 8085
> fs.overwrite-files: true
> taskmanager.memory.fraction: 0.8
> taskmanager.network.numberofBuffers: 70000
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
