GitHub user squito opened a pull request:
https://github.com/apache/spark/pull/15221
[SPARK-17648][CORE] TaskScheduler really needs offers to be an IndexedSeq
## What changes were proposed in this pull request?
The Seq[WorkerOffer] is accessed by index, so it really should be an
IndexedSeq, otherwise an O(n) operation becomes O(n^2). In practice
this hasn't been an issue b/c where these offers are generated, the call
to `.toSeq` just happens to create an IndexedSeq anyway. But its better
I got bitten by this in performance tests I was doing, and its better
for the types to be more precise so eg. a change in Scala doesn't destroy
performance.
## How was this patch tested?
Unit tests via jenkins.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/squito/spark SPARK-17648
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/15221.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #15221
----
commit d6957e2eb220a03dfe92461b182211319b4727bf
Author: Imran Rashid <[email protected]>
Date: 2016-09-23T15:45:16Z
[SPARK-17648][CORE] TaskScheduler really needs offers to be an IndexedSeq
for efficiency
The Seq[WorkerOffer] is accessed by index, so it really should be an
IndexedSeq, otherwise an O(n) operation becomes O(n^2). In practice
this hasn't been an issue b/c where these offers are generated, the call
to `.toSeq` just happens to create an IndexedSeq anyway. But its better
I got bitten by this in performance tests I was doing, and its better
for the types to be more precise so eg. a change in Scala doesn't destroy
performance.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
