GitHub user WeichenXu123 opened a pull request:
https://github.com/apache/spark/pull/19110
[SPARK-21027][ML][PYTHON] Added tunable parallelism to one vs. rest in both
Scala mllib and Pyspark
## What changes were proposed in this pull request?
Added tunable parallelism to the pyspark implementation of one vs. rest
classification. Added a parallelism parameter to the Scala implementation of
one vs. rest along with functionality for using the parameter to tune the level
of parallelism.
I take this PR #18281 over because the original author is busy but we need
merge this PR soon.
## How was this patch tested?
Test suite added.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/WeichenXu123/spark spark-21027
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/19110.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #19110
----
commit b69f201bc51f8de87adc3869d4843e3df6750972
Author: Ajay Saini <[email protected]>
Date: 2017-06-12T21:46:28Z
Added tunable parallelism to the pyspark implementation of one vs. rest
classification. Added a parallelism parameter to the scala implementation of
one vs. rest for python persistence but have not yet used it to tune the scala
parallelism implementation.
commit e750d3ee8eaf75ed8732f5cf1a1904acb1666793
Author: Ajay Saini <[email protected]>
Date: 2017-06-12T22:35:20Z
Fixed python style.
commit 81d458be99cf4f195b761eaa9bcb48ea086cdf61
Author: Ajay Saini <[email protected]>
Date: 2017-06-13T00:10:23Z
Added functionality for tuning parellelism in the Scala implementation of
the one vs. rest algorithm.
commit 213337882a40c63c3a3ef5741c17a6eebd63df0b
Author: Ajay Saini <[email protected]>
Date: 2017-06-13T20:28:49Z
Fixed code according to comments. Added both annotations and unit tests for
testing that parallelism doesn't affect the output.
commit c59b1d897c24d88753f478243ac8428598108da3
Author: Ajay Saini <[email protected]>
Date: 2017-06-22T23:41:15Z
Modified parallel one vs rest to use futures.
commit 5f635a2e3746bb6f1c503ebf48a485c8217f555e
Author: Ajay Saini <[email protected]>
Date: 2017-06-23T00:00:48Z
Put the parallelism parameter as well as the function for getting an
executor service with a given level of parallelism in a separat trait that
OneVsRest inherits from.
commit 4431ffcd38843798db907842799fd81a30c7a5be
Author: Ajay Saini <[email protected]>
Date: 2017-06-23T18:54:32Z
Responded to pull request comments.
commit a841b3ea3c674b790948425c884661063f5e3520
Author: Ajay Saini <[email protected]>
Date: 2017-07-06T23:35:15Z
Made changes based on pull request comments.
commit a95a8af2073b29aac751ae58489b737a3d7a39ae
Author: Ajay Saini <[email protected]>
Date: 2017-07-14T23:49:00Z
Fixed based on pull request comments
commit d45bc233751e63f15d242da702fdad6102f8bd65
Author: Ajay Saini <[email protected]>
Date: 2017-07-18T18:25:50Z
Fixed based on comments
commit 30ac62d476e72fd2f70ef2f4a74a735ea5d98509
Author: Ajay Saini <[email protected]>
Date: 2017-07-19T20:24:48Z
Reverting merge and adding change that would fix merge conflict (making
OneVsRest and OneVsRest model JavaMLReadable and JavaMLWritable)
commit cc634d2034db3144ff54fd9012abc15c57b1a226
Author: Ajay Saini <[email protected]>
Date: 2017-07-19T21:03:43Z
Merge branch 'master' into spark-21027
commit ce14172711b51a4321ed02a3cf8450a54374d4f5
Author: Ajay Saini <[email protected]>
Date: 2017-07-20T00:32:08Z
Style fix with docstring
commit 1c9de16d2919205542712b57de8bf4d866e17d95
Author: Ajay Saini <[email protected]>
Date: 2017-07-27T01:56:31Z
Fixed based on comments.
commit 9f3440412515df1630072c01ef642ee5e3723eb0
Author: Ajay Saini <[email protected]>
Date: 2017-07-27T02:04:43Z
Fixed style issue.
commit 585a3f8ea21359f11cd5a19ba195df88e091d9e0
Author: Ajay Saini <[email protected]>
Date: 2017-08-12T00:22:56Z
Fixed merge conflict
commit f65381afe20f5458ff7d706c6d803355c1b89f00
Author: Ajay Saini <[email protected]>
Date: 2017-08-23T17:51:01Z
Fixed remaining part of merge conflict.
commit 2a335fec1a4527e4970e88c10555d7d3f0375c35
Author: Ajay Saini <[email protected]>
Date: 2017-08-23T18:34:48Z
Fixed style problem
commit 049f371738eddd99d8261aaef02a5b9030843174
Author: WeichenXu <[email protected]>
Date: 2017-09-02T15:43:55Z
Merge branch 'master' into spark-21027
commit ddc2ff4f775358336914e65e228c74af8056e641
Author: WeichenXu <[email protected]>
Date: 2017-09-03T03:13:50Z
address review feedback issues
commit 24f4499702dd6efd80867e7d51a30fc08f2a51cc
Author: WeichenXu <[email protected]>
Date: 2017-09-03T04:26:01Z
update migration guide
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
