Github user tdas commented on the pull request:
https://github.com/apache/spark/pull/4957#issuecomment-93114895
The problem with this approach is that that data is getting copied over
through multiple data structures, first the queue and then the new block. That
is definitely going to hurt performance. That why once things are in the
ArrayBuffer the buffer as a whole in inserted, without further copying (at
least not on the receiver side).
Rather the existing approach with your fix of using the Guava rate limiter
is desirable. Might be better to play around with that approach.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
