Github user davies commented on the pull request:
https://github.com/apache/spark/pull/2681#issuecomment-59464471
This error was not happened in tests of this PR, it happened in tests of
our product, which have similar pattern as streaming, the job was submitted via
py4j.
The PR also check the number of blocks in readBlocks(), will throw a
meaningful exception in case of fail to get cached object in local mode.
TorrentBroadcast is so complicated (including several RPC) that it's not as
stable as HTTPBroadcast or w/o broadcast (we had saw some cases reported by
users in maillist), the motivation of this PR is to remove the complicity for
most cases (serialized task is small), then it will be more stable.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
