Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/1056#issuecomment-48867225
Hi @sryza, I left a couple of comments. In general, I think this patch can
be simplified by using akka only for the driver-executor heartbeats. We should
also clarify that HeartbeatReceiver is only for the driver, and document the
new configs you introduced (`spark.executor.heartbeat.interval`).
Have you looked at how large each new heartbeat is? `TaskMetrics` is
supposed to be small when we first added it, but now it carries many more
things than what it used to. If someone dumps a huge block that kicks out many
old blocks, for instance, the heartbeat would include the list of all the
blocks that are displaced. There has been a problem recently with akka frame
size, and I was wondering whether that might be a concern here too.
Could you upmerge this (again) once you have the chance? Thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
