GitHub user srowen opened a pull request:
https://github.com/apache/spark/pull/3702
SPARK-4547 [MLLIB] [WIP] OOM when making bins in BinaryClassificationMetrics
Now that I've implemented the basics here, I'm less convinced there is a
need for this change, somehow. Callers can downsample before or after. Really
the OOM is not in the ROC curve code, but in code that might `collect()` it for
local analysis. Still, might be useful to down-sample since the ROC curve
probably never needs millions of points.
This is a first pass. Since the `(score,label)` are already grouped and
sorted, I think it's sufficient to just take every Nth such pair, in order to
downsample by a factor of N? this is just like retaining every Nth point on the
curve, which I think is the goal. All of the data is still used to build the
curve of course.
What do you think about the API, and usefulness?
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/srowen/spark SPARK-4547
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/3702.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3702
----
commit a1c3ba3b87bb779149febc1146d51c4b90b55011
Author: Sean Owen <[email protected]>
Date: 2014-12-15T16:14:59Z
Add downsamplingFactor to BinaryClassificationMetrics
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
