Github user mateiz commented on the pull request:
https://github.com/apache/spark/pull/694#issuecomment-43038946
Yeah, that was the branch I was talking about above. I'm worried that if
the user has some large variables, we end up with lots of blocks in the block
store that are large and repetitive. So I'd leave that out for now. The best
solution, which will be along those lines, will be to broadcast the body of the
task using a broadcast variable and pass the per-instance pieces (Partition and
such) separately.
This current fix is also smaller, which is good in the meanwhile.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
