srowen commented on a change in pull request #24775: [SPARK-27925][ML] Better
control numBins of curves in BinaryClassificationMetrics
URL: https://github.com/apache/spark/pull/24775#discussion_r291997764
##########
File path:
mllib/src/main/scala/org/apache/spark/mllib/evaluation/BinaryClassificationMetrics.scala
##########
@@ -163,47 +163,54 @@ class BinaryClassificationMetrics @Since("3.0.0") (
private lazy val (
cumulativeCounts: RDD[(Double, BinaryLabelCounter)],
confusions: RDD[(Double, BinaryConfusionMatrix)]) = {
+
+ val numPartitions = if (numBins == 0) {
+ scoreLabelsWeight.getNumPartitions
+ } else {
+ numBins
Review comment:
I agree, but now a normal-sized input with 100000 bins will result in 100000
partitions. That could really grind to a halt. Even in a more common case with
a smallish number of points and smallish number of partitions and 1000 bins,
1000 partitions could simply be slower.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
