GitHub user kl0u opened a pull request: https://github.com/apache/flink/pull/934
Framesize fix In Apache Flink the results of the collect() call were returned through akka to the client. This led to an inherent limitation to the size of the output of a job, as this could not exceed the akka.framesize size. In other case, akka would drop the message. To alleviate this, without dropping the benefits brought by akka and its out-of-the-box efficiency for small-sized results, we decided to keep forwarding the non-oversized (i.e. smaller than the akka.framesize) results through akka, and use the BlobCache module for the forwarding the oversized (large) ones. Now the JobManager receives end merges the small accumulators (as before), and simply forwards to the Client the keys to the blobs storing the oversized ones. Now it is the responsibility of the Client to do the final merging between oversized and non-oversized accumulators. You can merge this pull request into a Git repository by running: $ git pull https://github.com/kl0u/flink framesize_fix Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/934.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #934 ---- commit fb1fbd6bdcc81acd20d422842789fce0c0872580 Author: Kostas Kloudas <kklou...@gmail.com> Date: 2015-07-22T17:53:11Z Solved the #887 issue: removing the akka.framesize size limitation for the result of a job. commit 34d3e433eb0ce976539de166288550c9c7612eb4 Author: Kostas Kloudas <kklou...@gmail.com> Date: 2015-07-22T17:53:11Z Solved the #887 issue: removing the akka.framesize size limitation for the result of a job. commit 55aa50c3f3e5c4c3a253b8da68b5ddde9acb307f Author: Kostas Kloudas <kklou...@gmail.com> Date: 2015-07-24T12:02:10Z Merge branch 'framesize_fix' of https://github.com/kl0u/flink into framesize_fix ---- --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---