GitHub user mengxr opened a pull request:
https://github.com/apache/spark/pull/1124
[SPARK-1112] use min akka frame size to decide how to send task results
Task results are sent either via akka directly or block manager indirectly,
based on whether the size of the serialized task is smaller than
spark.akka.frameSize. However, the result is actually sent back to the driver
via the backend actor, which is initialized before receiving the SparkConf and
hence it doesn't know the spark.akka.frameSize. That is the root cause of
SPARK-1112.
A quick fix is using the min frame size to decide which route to go.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/mengxr/spark akka-size
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/1124.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1124
----
commit 5ca1edd0d01b66265d306e552c82b49d25537579
Author: Xiangrui Meng <[email protected]>
Date: 2014-06-19T01:48:20Z
use min akka frame size to decide whether to send result via akka or block
manager
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
