GitHub user kl0u opened a pull request:
https://github.com/apache/flink/pull/934
Framesize fix
In Apache Flink the results of the collect() call were returned through
akka to the client. This led to an inherent limitation to the size of the
output of a job, as this could not exceed the akka.framesize size. In other
case, akka would drop the message.
To alleviate this, without dropping the benefits brought by akka and its
out-of-the-box efficiency for small-sized results, we decided to keep
forwarding the non-oversized (i.e. smaller than the akka.framesize) results
through akka, and use the BlobCache module for the forwarding the oversized
(large) ones.
Now the JobManager receives end merges the small accumulators (as before),
and simply forwards to the Client the keys to the blobs storing the oversized
ones. Now it is the responsibility of the Client to do the final merging
between oversized and non-oversized accumulators.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/kl0u/flink framesize_fix
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/934.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #934
----
commit fb1fbd6bdcc81acd20d422842789fce0c0872580
Author: Kostas Kloudas <[email protected]>
Date: 2015-07-22T17:53:11Z
Solved the #887 issue: removing the akka.framesize size limitation for the
result of a job.
commit 34d3e433eb0ce976539de166288550c9c7612eb4
Author: Kostas Kloudas <[email protected]>
Date: 2015-07-22T17:53:11Z
Solved the #887 issue: removing the akka.framesize size limitation for the
result of a job.
commit 55aa50c3f3e5c4c3a253b8da68b5ddde9acb307f
Author: Kostas Kloudas <[email protected]>
Date: 2015-07-24T12:02:10Z
Merge branch 'framesize_fix' of https://github.com/kl0u/flink into
framesize_fix
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
