[GitHub] spark pull request #17086: [SPARK-24101][ML][MLLIB] ML Evaluators should use...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/17086 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17086: [SPARK-24101][ML][MLLIB] ML Evaluators should use...
Github user imatiach-msft commented on a diff in the pull request:
https://github.com/apache/spark/pull/17086#discussion_r231760882
--- Diff:
mllib/src/main/scala/org/apache/spark/mllib/evaluation/MulticlassMetrics.scala
---
@@ -27,10 +27,17 @@ import org.apache.spark.sql.DataFrame
/**
* Evaluator for multiclass classification.
*
- * @param predictionAndLabels an RDD of (prediction, label) pairs.
+ * @param predAndLabelsWithOptWeight an RDD of (prediction, label, weight)
or
+ * (prediction, label) pairs.
*/
@Since("1.1.0")
-class MulticlassMetrics @Since("1.1.0") (predictionAndLabels: RDD[(Double,
Double)]) {
+class MulticlassMetrics @Since("3.0.0") (predAndLabelsWithOptWeight:
RDD[_]) {
+ val predLabelsWeight: RDD[(Double, Double, Double)] =
predAndLabelsWithOptWeight.map {
+case (prediction: Double, label: Double, weight: Double) =>
+ (prediction, label, weight)
+case (prediction: Double, label: Double) =>
+ (prediction, label, 1.0)
--- End diff --
wow, great idea! done!
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17086: [SPARK-24101][ML][MLLIB] ML Evaluators should use...
Github user imatiach-msft commented on a diff in the pull request:
https://github.com/apache/spark/pull/17086#discussion_r231760666
--- Diff:
mllib/src/main/scala/org/apache/spark/mllib/evaluation/MulticlassMetrics.scala
---
@@ -27,10 +27,17 @@ import org.apache.spark.sql.DataFrame
/**
* Evaluator for multiclass classification.
*
- * @param predictionAndLabels an RDD of (prediction, label) pairs.
+ * @param predAndLabelsWithOptWeight an RDD of (prediction, label, weight)
or
+ * (prediction, label) pairs.
*/
@Since("1.1.0")
-class MulticlassMetrics @Since("1.1.0") (predictionAndLabels: RDD[(Double,
Double)]) {
+class MulticlassMetrics @Since("3.0.0") (predAndLabelsWithOptWeight:
RDD[_]) {
--- End diff --
done!
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17086: [SPARK-24101][ML][MLLIB] ML Evaluators should use...
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/17086#discussion_r231724314
--- Diff:
mllib/src/main/scala/org/apache/spark/mllib/evaluation/MulticlassMetrics.scala
---
@@ -27,10 +27,17 @@ import org.apache.spark.sql.DataFrame
/**
* Evaluator for multiclass classification.
*
- * @param predictionAndLabels an RDD of (prediction, label) pairs.
+ * @param predAndLabelsWithOptWeight an RDD of (prediction, label, weight)
or
+ * (prediction, label) pairs.
*/
@Since("1.1.0")
-class MulticlassMetrics @Since("1.1.0") (predictionAndLabels: RDD[(Double,
Double)]) {
+class MulticlassMetrics @Since("3.0.0") (predAndLabelsWithOptWeight:
RDD[_]) {
--- End diff --
Nit: I think this Since has to go back to 1.1.0. It hasn't changed
signature after all.
@yanboliang can you comment on the DataFrame constructor? although it might
be a separate issue, would be good to know if we're missing something.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17086: [SPARK-24101][ML][MLLIB] ML Evaluators should use...
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/17086#discussion_r231724984
--- Diff:
mllib/src/main/scala/org/apache/spark/mllib/evaluation/MulticlassMetrics.scala
---
@@ -27,10 +27,17 @@ import org.apache.spark.sql.DataFrame
/**
* Evaluator for multiclass classification.
*
- * @param predictionAndLabels an RDD of (prediction, label) pairs.
+ * @param predAndLabelsWithOptWeight an RDD of (prediction, label, weight)
or
+ * (prediction, label) pairs.
*/
@Since("1.1.0")
-class MulticlassMetrics @Since("1.1.0") (predictionAndLabels: RDD[(Double,
Double)]) {
+class MulticlassMetrics @Since("3.0.0") (predAndLabelsWithOptWeight:
RDD[_]) {
+ val predLabelsWeight: RDD[(Double, Double, Double)] =
predAndLabelsWithOptWeight.map {
+case (prediction: Double, label: Double, weight: Double) =>
+ (prediction, label, weight)
+case (prediction: Double, label: Double) =>
+ (prediction, label, 1.0)
--- End diff --
If you're making one more change, might make an explicit check here on the
type of the RDD, now that it can be anything at compile time. Like `case other
=> throw new IllegalArgumentException(s"Expected tuples, got $other")`
Actually, in order to tighten this back down a little, I wonder if the
method argument can be `RDD[_ <: Product]` ? That includes Tuple2 and Tuple3,
and a lot of other things, but is more specific than 'anything'.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17086: [SPARK-24101][ML][MLLIB] ML Evaluators should use...
Github user imatiach-msft commented on a diff in the pull request:
https://github.com/apache/spark/pull/17086#discussion_r231296074
--- Diff:
mllib/src/main/scala/org/apache/spark/mllib/evaluation/MulticlassMetrics.scala
---
@@ -39,21 +46,28 @@ class MulticlassMetrics @Since("1.1.0")
(predictionAndLabels: RDD[(Double, Doubl
private[mllib] def this(predictionAndLabels: DataFrame) =
this(predictionAndLabels.rdd.map(r => (r.getDouble(0),
r.getDouble(1
- private lazy val labelCountByClass: Map[Double, Long] =
predictionAndLabels.values.countByValue()
- private lazy val labelCount: Long = labelCountByClass.values.sum
- private lazy val tpByClass: Map[Double, Int] = predictionAndLabels
-.map { case (prediction, label) =>
- (label, if (label == prediction) 1 else 0)
+ private lazy val labelCountByClass: Map[Double, Double] =
+predLabelsWeight.map {
+ case (prediction: Double, label: Double, weight: Double) =>
+(label, weight)
+}.reduceByKey(_ + _).collect().toMap
+ private lazy val labelCount: Double = labelCountByClass.values.sum
+ private lazy val tpByClass: Map[Double, Double] = predLabelsWeight
+.map {
--- End diff --
see the test failure here for reference:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98529/testReport/
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17086: [SPARK-24101][ML][MLLIB] ML Evaluators should use...
Github user imatiach-msft commented on a diff in the pull request:
https://github.com/apache/spark/pull/17086#discussion_r231294624
--- Diff:
mllib/src/main/scala/org/apache/spark/mllib/evaluation/MulticlassMetrics.scala
---
@@ -39,21 +46,28 @@ class MulticlassMetrics @Since("1.1.0")
(predictionAndLabels: RDD[(Double, Doubl
private[mllib] def this(predictionAndLabels: DataFrame) =
this(predictionAndLabels.rdd.map(r => (r.getDouble(0),
r.getDouble(1
- private lazy val labelCountByClass: Map[Double, Long] =
predictionAndLabels.values.countByValue()
- private lazy val labelCount: Long = labelCountByClass.values.sum
- private lazy val tpByClass: Map[Double, Int] = predictionAndLabels
-.map { case (prediction, label) =>
- (label, if (label == prediction) 1 else 0)
+ private lazy val labelCountByClass: Map[Double, Double] =
+predLabelsWeight.map {
+ case (prediction: Double, label: Double, weight: Double) =>
+(label, weight)
+}.reduceByKey(_ + _).collect().toMap
+ private lazy val labelCount: Double = labelCountByClass.values.sum
+ private lazy val tpByClass: Map[Double, Double] = predLabelsWeight
+.map {
--- End diff --
it looks like this will actually cause tests to fail, because the key may
become missing if we filter everything out first, whereas we would want it to
be present otherwise but have a 0 value
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17086: [SPARK-24101][ML][MLLIB] ML Evaluators should use...
Github user imatiach-msft commented on a diff in the pull request:
https://github.com/apache/spark/pull/17086#discussion_r231273689
--- Diff:
mllib/src/main/scala/org/apache/spark/mllib/evaluation/MulticlassMetrics.scala
---
@@ -39,21 +46,28 @@ class MulticlassMetrics @Since("1.1.0")
(predictionAndLabels: RDD[(Double, Doubl
private[mllib] def this(predictionAndLabels: DataFrame) =
this(predictionAndLabels.rdd.map(r => (r.getDouble(0),
r.getDouble(1
- private lazy val labelCountByClass: Map[Double, Long] =
predictionAndLabels.values.countByValue()
- private lazy val labelCount: Long = labelCountByClass.values.sum
- private lazy val tpByClass: Map[Double, Int] = predictionAndLabels
-.map { case (prediction, label) =>
- (label, if (label == prediction) 1 else 0)
+ private lazy val labelCountByClass: Map[Double, Double] =
+predLabelsWeight.map {
+ case (prediction: Double, label: Double, weight: Double) =>
+(label, weight)
+}.reduceByKey(_ + _).collect().toMap
+ private lazy val labelCount: Double = labelCountByClass.values.sum
+ private lazy val tpByClass: Map[Double, Double] = predLabelsWeight
+.map {
--- End diff --
done, not sure if more efficient, it seemed to take same time for me, but I
didn't test extensively
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17086: [SPARK-24101][ML][MLLIB] ML Evaluators should use...
Github user imatiach-msft commented on a diff in the pull request:
https://github.com/apache/spark/pull/17086#discussion_r231227854
--- Diff:
mllib/src/main/scala/org/apache/spark/mllib/evaluation/MulticlassMetrics.scala
---
@@ -27,10 +27,17 @@ import org.apache.spark.sql.DataFrame
/**
* Evaluator for multiclass classification.
*
- * @param predictionAndLabels an RDD of (prediction, label) pairs.
+ * @param predAndLabelsWithOptWeight an RDD of (prediction, label, weight)
or
+ * (prediction, label) pairs.
*/
@Since("1.1.0")
-class MulticlassMetrics @Since("1.1.0") (predictionAndLabels: RDD[(Double,
Double)]) {
+class MulticlassMetrics @Since("3.0.0") (predAndLabelsWithOptWeight:
RDD[_]) {
--- End diff --
The python API takes an RDD, creates a DF, and then calls this private
constructor with the DF, but I would think we could just pass the RDD directly
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17086: [SPARK-24101][ML][MLLIB] ML Evaluators should use...
Github user imatiach-msft commented on a diff in the pull request:
https://github.com/apache/spark/pull/17086#discussion_r231227296
--- Diff:
mllib/src/main/scala/org/apache/spark/mllib/evaluation/MulticlassMetrics.scala
---
@@ -27,10 +27,17 @@ import org.apache.spark.sql.DataFrame
/**
* Evaluator for multiclass classification.
*
- * @param predictionAndLabels an RDD of (prediction, label) pairs.
+ * @param predAndLabelsWithOptWeight an RDD of (prediction, label, weight)
or
+ * (prediction, label) pairs.
*/
@Since("1.1.0")
-class MulticlassMetrics @Since("1.1.0") (predictionAndLabels: RDD[(Double,
Double)]) {
+class MulticlassMetrics @Since("3.0.0") (predAndLabelsWithOptWeight:
RDD[_]) {
--- End diff --
"I am not sure what to do about the DataFrame issue though", ah, I think I
see your concern.
But, isn't this dataframe constructor private anyway, so it can't be used
by anyone outside mllib:
private[mllib] def this(predictionAndLabels: DataFrame) =
this(predictionAndLabels.rdd.map(r => (r.getDouble(0), r.getDouble(1
I only modified the RDD part because that is what is used by the ML
evaluator and it is what users outside spark can access. This is to add weight
column for the evaluators.
However, even if we wanted to add weight column support for the private
API, I'm unsure about how to add this. Should I just check if there are 3
columns or two, and if there are 3 use the third one as the weight column? I
guess I am on the fence about this, I could change it but I don't think it is
absolutely necessary, since it's not used anywhere outside spark MLLIB anyway.
Actually, this constructor is a bit weird, it looks like it was added as
part of this PR:
https://github.com/apache/spark/pull/6011/files
It is only used here in the python API:
https://github.com/apache/spark/pull/6011/files#diff-443f766289f8090078531c3e1a1d6027R186
But I don't see why we couldn't just get the rdd there and remove the
private constructor altogether (?)
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17086: [SPARK-24101][ML][MLLIB] ML Evaluators should use...
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/17086#discussion_r231215393
--- Diff:
mllib/src/main/scala/org/apache/spark/mllib/evaluation/MulticlassMetrics.scala
---
@@ -27,10 +27,17 @@ import org.apache.spark.sql.DataFrame
/**
* Evaluator for multiclass classification.
*
- * @param predictionAndLabels an RDD of (prediction, label) pairs.
+ * @param predAndLabelsWithOptWeight an RDD of (prediction, label, weight)
or
+ * (prediction, label) pairs.
*/
@Since("1.1.0")
-class MulticlassMetrics @Since("1.1.0") (predictionAndLabels: RDD[(Double,
Double)]) {
+class MulticlassMetrics @Since("3.0.0") (predAndLabelsWithOptWeight:
RDD[_]) {
--- End diff --
Darn, OK. Hm, so this doesn't actually cause a source or binary change? OK,
that could be fine. I guess MiMa didn't complain. I guess you can now do weird
things like pass `RDD[String]` here and it'll fail quickly. I'm a little uneasy
about it but it's probably acceptable. Any other opinions?
I am not sure what to do about the DataFrame issue though. I suspect most
people will want to call with a DataFrame now.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17086: [SPARK-24101][ML][MLLIB] ML Evaluators should use...
Github user imatiach-msft commented on a diff in the pull request:
https://github.com/apache/spark/pull/17086#discussion_r231214136
--- Diff:
mllib/src/main/scala/org/apache/spark/mllib/evaluation/MulticlassMetrics.scala
---
@@ -39,21 +46,28 @@ class MulticlassMetrics @Since("1.1.0")
(predictionAndLabels: RDD[(Double, Doubl
private[mllib] def this(predictionAndLabels: DataFrame) =
this(predictionAndLabels.rdd.map(r => (r.getDouble(0),
r.getDouble(1
- private lazy val labelCountByClass: Map[Double, Long] =
predictionAndLabels.values.countByValue()
- private lazy val labelCount: Long = labelCountByClass.values.sum
- private lazy val tpByClass: Map[Double, Int] = predictionAndLabels
-.map { case (prediction, label) =>
- (label, if (label == prediction) 1 else 0)
+ private lazy val labelCountByClass: Map[Double, Double] =
+predLabelsWeight.map {
+ case (prediction: Double, label: Double, weight: Double) =>
--- End diff --
done!
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17086: [SPARK-24101][ML][MLLIB] ML Evaluators should use...
Github user imatiach-msft commented on a diff in the pull request:
https://github.com/apache/spark/pull/17086#discussion_r231214331
--- Diff:
mllib/src/main/scala/org/apache/spark/mllib/evaluation/MulticlassMetrics.scala
---
@@ -39,21 +46,28 @@ class MulticlassMetrics @Since("1.1.0")
(predictionAndLabels: RDD[(Double, Doubl
private[mllib] def this(predictionAndLabels: DataFrame) =
this(predictionAndLabels.rdd.map(r => (r.getDouble(0),
r.getDouble(1
- private lazy val labelCountByClass: Map[Double, Long] =
predictionAndLabels.values.countByValue()
- private lazy val labelCount: Long = labelCountByClass.values.sum
- private lazy val tpByClass: Map[Double, Int] = predictionAndLabels
-.map { case (prediction, label) =>
- (label, if (label == prediction) 1 else 0)
+ private lazy val labelCountByClass: Map[Double, Double] =
+predLabelsWeight.map {
+ case (prediction: Double, label: Double, weight: Double) =>
+(label, weight)
+}.reduceByKey(_ + _).collect().toMap
--- End diff --
done!
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17086: [SPARK-24101][ML][MLLIB] ML Evaluators should use...
Github user imatiach-msft commented on a diff in the pull request:
https://github.com/apache/spark/pull/17086#discussion_r231213369
--- Diff:
mllib/src/main/scala/org/apache/spark/mllib/evaluation/MulticlassMetrics.scala
---
@@ -27,10 +27,17 @@ import org.apache.spark.sql.DataFrame
/**
* Evaluator for multiclass classification.
*
- * @param predictionAndLabels an RDD of (prediction, label) pairs.
+ * @param predAndLabelsWithOptWeight an RDD of (prediction, label, weight)
or
+ * (prediction, label) pairs.
*/
@Since("1.1.0")
-class MulticlassMetrics @Since("1.1.0") (predictionAndLabels: RDD[(Double,
Double)]) {
+class MulticlassMetrics @Since("3.0.0") (predAndLabelsWithOptWeight:
RDD[_]) {
--- End diff --
@srowen hmm, this was already suggested, please see this comment:
https://github.com/apache/spark/pull/17086#discussion_r182303815
the build fails with an error due to Java type erasure, so this wouldn't
work... you can't have two constructors with the same type erased signature...
maybe I am misunderstanding something, and you meant something else? Are you
sure this changes the signature in a way that breaks others, it should still
allow RDD with a tuple of 2 Double values.
The error I get is:
[error]
/home/jenkins/workspace/SparkPullRequestBuilder@2/mllib/src/main/scala/org/apache/spark/mllib/evaluation/MulticlassMetrics.scala:35:
double definition:
[error] constructor MulticlassMetrics: (predLabelsWeight:
org.apache.spark.rdd.RDD[(Double, Double,
Double)])org.apache.spark.mllib.evaluation.MulticlassMetrics at line 33 and
[error] constructor MulticlassMetrics: (predAndLabels:
org.apache.spark.rdd.RDD[(Double,
Double)])org.apache.spark.mllib.evaluation.MulticlassMetrics at line 35
[error] have same type after erasure: (predLabelsWeight:
org.apache.spark.rdd.RDD)org.apache.spark.mllib.evaluation.MulticlassMetrics
[error] def this(predAndLabels: RDD[(Double, Double)]) =
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17086: [SPARK-24101][ML][MLLIB] ML Evaluators should use...
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/17086#discussion_r231123811
--- Diff:
mllib/src/main/scala/org/apache/spark/mllib/evaluation/MulticlassMetrics.scala
---
@@ -39,21 +46,28 @@ class MulticlassMetrics @Since("1.1.0")
(predictionAndLabels: RDD[(Double, Doubl
private[mllib] def this(predictionAndLabels: DataFrame) =
this(predictionAndLabels.rdd.map(r => (r.getDouble(0),
r.getDouble(1
- private lazy val labelCountByClass: Map[Double, Long] =
predictionAndLabels.values.countByValue()
- private lazy val labelCount: Long = labelCountByClass.values.sum
- private lazy val tpByClass: Map[Double, Int] = predictionAndLabels
-.map { case (prediction, label) =>
- (label, if (label == prediction) 1 else 0)
+ private lazy val labelCountByClass: Map[Double, Double] =
+predLabelsWeight.map {
+ case (prediction: Double, label: Double, weight: Double) =>
--- End diff --
Nit: you can write prediction as _ here
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17086: [SPARK-24101][ML][MLLIB] ML Evaluators should use...
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/17086#discussion_r231124683
--- Diff:
mllib/src/main/scala/org/apache/spark/mllib/evaluation/MulticlassMetrics.scala
---
@@ -39,21 +46,28 @@ class MulticlassMetrics @Since("1.1.0")
(predictionAndLabels: RDD[(Double, Doubl
private[mllib] def this(predictionAndLabels: DataFrame) =
this(predictionAndLabels.rdd.map(r => (r.getDouble(0),
r.getDouble(1
- private lazy val labelCountByClass: Map[Double, Long] =
predictionAndLabels.values.countByValue()
- private lazy val labelCount: Long = labelCountByClass.values.sum
- private lazy val tpByClass: Map[Double, Int] = predictionAndLabels
-.map { case (prediction, label) =>
- (label, if (label == prediction) 1 else 0)
+ private lazy val labelCountByClass: Map[Double, Double] =
+predLabelsWeight.map {
+ case (prediction: Double, label: Double, weight: Double) =>
+(label, weight)
+}.reduceByKey(_ + _).collect().toMap
+ private lazy val labelCount: Double = labelCountByClass.values.sum
+ private lazy val tpByClass: Map[Double, Double] = predLabelsWeight
+.map {
--- End diff --
I'm not sure, but I wonder if it's more efficient to filter on label ==
prediction, then reduce? The original code didn't do it, but could be worth it.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17086: [SPARK-24101][ML][MLLIB] ML Evaluators should use...
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/17086#discussion_r231123906
--- Diff:
mllib/src/main/scala/org/apache/spark/mllib/evaluation/MulticlassMetrics.scala
---
@@ -39,21 +46,28 @@ class MulticlassMetrics @Since("1.1.0")
(predictionAndLabels: RDD[(Double, Doubl
private[mllib] def this(predictionAndLabels: DataFrame) =
this(predictionAndLabels.rdd.map(r => (r.getDouble(0),
r.getDouble(1
- private lazy val labelCountByClass: Map[Double, Long] =
predictionAndLabels.values.countByValue()
- private lazy val labelCount: Long = labelCountByClass.values.sum
- private lazy val tpByClass: Map[Double, Int] = predictionAndLabels
-.map { case (prediction, label) =>
- (label, if (label == prediction) 1 else 0)
+ private lazy val labelCountByClass: Map[Double, Double] =
+predLabelsWeight.map {
+ case (prediction: Double, label: Double, weight: Double) =>
+(label, weight)
+}.reduceByKey(_ + _).collect().toMap
--- End diff --
.collectAsMap()?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17086: [SPARK-24101][ML][MLLIB] ML Evaluators should use...
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/17086#discussion_r231123729
--- Diff:
mllib/src/main/scala/org/apache/spark/mllib/evaluation/MulticlassMetrics.scala
---
@@ -27,10 +27,17 @@ import org.apache.spark.sql.DataFrame
/**
* Evaluator for multiclass classification.
*
- * @param predictionAndLabels an RDD of (prediction, label) pairs.
+ * @param predAndLabelsWithOptWeight an RDD of (prediction, label, weight)
or
+ * (prediction, label) pairs.
*/
@Since("1.1.0")
-class MulticlassMetrics @Since("1.1.0") (predictionAndLabels: RDD[(Double,
Double)]) {
+class MulticlassMetrics @Since("3.0.0") (predAndLabelsWithOptWeight:
RDD[_]) {
--- End diff --
Oh, wait a sec, this changed the signature. I think you have to retain
both. The `RDD[(Double, Double)]` constructor should stay, one way or the
other, and add a new `RDD[(Double, Double, Double)]` constructor, with
appropriate Since tags on each.
Below there's a `DataFrame` constructor and I'm not sure how to handle
that. It should also handle the case where there's a weight col, but I'm not
sure how to do that cleanly. There can be a second argument like `hasWeightCol`
but that's starting to feel hacky.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17086: [SPARK-24101][ML][MLLIB] ML Evaluators should use...
Github user imatiach-msft commented on a diff in the pull request:
https://github.com/apache/spark/pull/17086#discussion_r230992722
--- Diff:
mllib/src/main/scala/org/apache/spark/ml/evaluation/MulticlassClassificationEvaluator.scala
---
@@ -67,6 +68,10 @@ class MulticlassClassificationEvaluator @Since("1.5.0")
(@Since("1.5.0") overrid
@Since("1.5.0")
def setLabelCol(value: String): this.type = set(labelCol, value)
+ /** @group setParam */
+ @Since("2.4.0")
--- End diff --
done!
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17086: [SPARK-24101][ML][MLLIB] ML Evaluators should use...
Github user imatiach-msft commented on a diff in the pull request:
https://github.com/apache/spark/pull/17086#discussion_r230992384
--- Diff:
mllib/src/main/scala/org/apache/spark/mllib/evaluation/MulticlassMetrics.scala
---
@@ -27,10 +27,17 @@ import org.apache.spark.sql.DataFrame
/**
* Evaluator for multiclass classification.
*
- * @param predictionAndLabels an RDD of (prediction, label) pairs.
+ * @param predAndLabelsWithOptWeight an RDD of (prediction, label, weight)
or
+ * (prediction, label) pairs.
*/
@Since("1.1.0")
-class MulticlassMetrics @Since("1.1.0") (predictionAndLabels: RDD[(Double,
Double)]) {
+class MulticlassMetrics @Since("2.4.0") (predAndLabelsWithOptWeight:
RDD[_]) {
--- End diff --
thanks! yes, it is backwards compatible.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17086: [SPARK-24101][ML][MLLIB] ML Evaluators should use...
Github user imatiach-msft commented on a diff in the pull request:
https://github.com/apache/spark/pull/17086#discussion_r230992285
--- Diff:
mllib/src/test/scala/org/apache/spark/mllib/evaluation/MulticlassMetricsSuite.scala
---
@@ -18,10 +18,14 @@
package org.apache.spark.mllib.evaluation
import org.apache.spark.SparkFunSuite
-import org.apache.spark.mllib.linalg.Matrices
+import org.apache.spark.ml.linalg.Matrices
+import org.apache.spark.ml.util.TestingUtils._
import org.apache.spark.mllib.util.MLlibTestSparkContext
class MulticlassMetricsSuite extends SparkFunSuite with
MLlibTestSparkContext {
+
+ val delta = 1e-7
--- End diff --
done!
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17086: [SPARK-24101][ML][MLLIB] ML Evaluators should use...
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/17086#discussion_r230949468
--- Diff:
mllib/src/main/scala/org/apache/spark/ml/evaluation/MulticlassClassificationEvaluator.scala
---
@@ -67,6 +68,10 @@ class MulticlassClassificationEvaluator @Since("1.5.0")
(@Since("1.5.0") overrid
@Since("1.5.0")
def setLabelCol(value: String): this.type = set(labelCol, value)
+ /** @group setParam */
+ @Since("2.4.0")
--- End diff --
Will need to be 3.0.0
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17086: [SPARK-24101][ML][MLLIB] ML Evaluators should use...
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/17086#discussion_r230949719
--- Diff:
mllib/src/main/scala/org/apache/spark/mllib/evaluation/MulticlassMetrics.scala
---
@@ -27,10 +27,17 @@ import org.apache.spark.sql.DataFrame
/**
* Evaluator for multiclass classification.
*
- * @param predictionAndLabels an RDD of (prediction, label) pairs.
+ * @param predAndLabelsWithOptWeight an RDD of (prediction, label, weight)
or
+ * (prediction, label) pairs.
*/
@Since("1.1.0")
-class MulticlassMetrics @Since("1.1.0") (predictionAndLabels: RDD[(Double,
Double)]) {
+class MulticlassMetrics @Since("2.4.0") (predAndLabelsWithOptWeight:
RDD[_]) {
--- End diff --
Although I think we're not updating .mllib much at all now, I think this is
a simple and backwards-compatible change so think it's OK. It is the
implementation behind the .ml version anyway.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17086: [SPARK-24101][ML][MLLIB] ML Evaluators should use...
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/17086#discussion_r230949430
--- Diff:
mllib/src/test/scala/org/apache/spark/mllib/evaluation/MulticlassMetricsSuite.scala
---
@@ -18,10 +18,14 @@
package org.apache.spark.mllib.evaluation
import org.apache.spark.SparkFunSuite
-import org.apache.spark.mllib.linalg.Matrices
+import org.apache.spark.ml.linalg.Matrices
+import org.apache.spark.ml.util.TestingUtils._
import org.apache.spark.mllib.util.MLlibTestSparkContext
class MulticlassMetricsSuite extends SparkFunSuite with
MLlibTestSparkContext {
+
+ val delta = 1e-7
--- End diff --
`private`?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17086: [SPARK-24101][ML][MLLIB] ML Evaluators should use...
Github user imatiach-msft commented on a diff in the pull request:
https://github.com/apache/spark/pull/17086#discussion_r185163922
--- Diff:
mllib/src/test/scala/org/apache/spark/mllib/evaluation/MulticlassMetricsSuite.scala
---
@@ -95,4 +95,95 @@ class MulticlassMetricsSuite extends SparkFunSuite with
MLlibTestSparkContext {
((4.0 / 9) * f2measure0 + (4.0 / 9) * f2measure1 + (1.0 / 9) *
f2measure2)) < delta)
assert(metrics.labels.sameElements(labels))
}
+
+ test("Multiclass evaluation metrics with weights") {
+/*
+ * Confusion matrix for 3-class classification with total 9 instances
with 2 weights:
+ * |2 * w1|1 * w2 |1 * w1| true class0 (4 instances)
+ * |1 * w2|2 * w1 + 1 * w2|0 | true class1 (4 instances)
+ * |0 |0 |1 * w2| true class2 (1 instance)
+ */
+val w1 = 2.2
+val w2 = 1.5
+val tw = 2.0 * w1 + 1.0 * w2 + 1.0 * w1 + 1.0 * w2 + 2.0 * w1 + 1.0 *
w2 + 1.0 * w2
+val confusionMatrix = Matrices.dense(3, 3,
+ Array(2 * w1, 1 * w2, 0, 1 * w2, 2 * w1 + 1 * w2, 0, 1 * w1, 0, 1 *
w2))
+val labels = Array(0.0, 1.0, 2.0)
+val predictionAndLabelsWithWeights = sc.parallelize(
+ Seq((0.0, 0.0, w1), (0.0, 1.0, w2), (0.0, 0.0, w1), (1.0, 0.0, w2),
+(1.0, 1.0, w1), (1.0, 1.0, w2), (1.0, 1.0, w1), (2.0, 2.0, w2),
+(2.0, 0.0, w1)), 2)
+val metrics = new MulticlassMetrics(predictionAndLabelsWithWeights)
+val delta = 0.001
+val tpRate0 = (2.0 * w1) / (2.0 * w1 + 1.0 * w2 + 1.0 * w1)
+val tpRate1 = (2.0 * w1 + 1.0 * w2) / (2.0 * w1 + 1.0 * w2 + 1.0 * w2)
+val tpRate2 = (1.0 * w2) / (1.0 * w2 + 0)
+val fpRate0 = (1.0 * w2) / (tw - (2.0 * w1 + 1.0 * w2 + 1.0 * w1))
+val fpRate1 = (1.0 * w2) / (tw - (1.0 * w2 + 2.0 * w1 + 1.0 * w2))
+val fpRate2 = (1.0 * w1) / (tw - (1.0 * w2))
+val precision0 = (2.0 * w1) / (2 * w1 + 1 * w2)
+val precision1 = (2.0 * w1 + 1.0 * w2) / (2.0 * w1 + 1.0 * w2 + 1.0 *
w2)
+val precision2 = (1.0 * w2) / (1 * w1 + 1 * w2)
+val recall0 = (2.0 * w1) / (2.0 * w1 + 1.0 * w2 + 1.0 * w1)
+val recall1 = (2.0 * w1 + 1.0 * w2) / (2.0 * w1 + 1.0 * w2 + 1.0 * w2)
+val recall2 = (1.0 * w2) / (1.0 * w2 + 0)
+val f1measure0 = 2 * precision0 * recall0 / (precision0 + recall0)
+val f1measure1 = 2 * precision1 * recall1 / (precision1 + recall1)
+val f1measure2 = 2 * precision2 * recall2 / (precision2 + recall2)
+val f2measure0 = (1 + 2 * 2) * precision0 * recall0 / (2 * 2 *
precision0 + recall0)
+val f2measure1 = (1 + 2 * 2) * precision1 * recall1 / (2 * 2 *
precision1 + recall1)
+val f2measure2 = (1 + 2 * 2) * precision2 * recall2 / (2 * 2 *
precision2 + recall2)
+
+
assert(metrics.confusionMatrix.toArray.sameElements(confusionMatrix.toArray))
--- End diff --
however, I still need to do asML on the metrics.confusionMatrix as that
property is from mllib (in MulticlassMetrics class)
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17086: [SPARK-24101][ML][MLLIB] ML Evaluators should use...
Github user imatiach-msft commented on a diff in the pull request:
https://github.com/apache/spark/pull/17086#discussion_r185163674
--- Diff:
mllib/src/test/scala/org/apache/spark/mllib/evaluation/MulticlassMetricsSuite.scala
---
@@ -55,44 +60,128 @@ class MulticlassMetricsSuite extends SparkFunSuite
with MLlibTestSparkContext {
val f2measure1 = (1 + 2 * 2) * precision1 * recall1 / (2 * 2 *
precision1 + recall1)
val f2measure2 = (1 + 2 * 2) * precision2 * recall2 / (2 * 2 *
precision2 + recall2)
-
assert(metrics.confusionMatrix.toArray.sameElements(confusionMatrix.toArray))
-assert(math.abs(metrics.truePositiveRate(0.0) - tpRate0) < delta)
-assert(math.abs(metrics.truePositiveRate(1.0) - tpRate1) < delta)
-assert(math.abs(metrics.truePositiveRate(2.0) - tpRate2) < delta)
-assert(math.abs(metrics.falsePositiveRate(0.0) - fpRate0) < delta)
-assert(math.abs(metrics.falsePositiveRate(1.0) - fpRate1) < delta)
-assert(math.abs(metrics.falsePositiveRate(2.0) - fpRate2) < delta)
-assert(math.abs(metrics.precision(0.0) - precision0) < delta)
-assert(math.abs(metrics.precision(1.0) - precision1) < delta)
-assert(math.abs(metrics.precision(2.0) - precision2) < delta)
-assert(math.abs(metrics.recall(0.0) - recall0) < delta)
-assert(math.abs(metrics.recall(1.0) - recall1) < delta)
-assert(math.abs(metrics.recall(2.0) - recall2) < delta)
-assert(math.abs(metrics.fMeasure(0.0) - f1measure0) < delta)
-assert(math.abs(metrics.fMeasure(1.0) - f1measure1) < delta)
-assert(math.abs(metrics.fMeasure(2.0) - f1measure2) < delta)
-assert(math.abs(metrics.fMeasure(0.0, 2.0) - f2measure0) < delta)
-assert(math.abs(metrics.fMeasure(1.0, 2.0) - f2measure1) < delta)
-assert(math.abs(metrics.fMeasure(2.0, 2.0) - f2measure2) < delta)
+assert(metrics.confusionMatrix.asML ~== confusionMatrix.asML relTol
delta)
+assert(metrics.truePositiveRate(0.0) ~== tpRate0 absTol delta)
+assert(metrics.truePositiveRate(1.0) ~== tpRate1 absTol delta)
+assert(metrics.truePositiveRate(2.0) ~== tpRate2 absTol delta)
+assert(metrics.falsePositiveRate(0.0) ~== fpRate0 absTol delta)
+assert(metrics.falsePositiveRate(1.0) ~== fpRate1 absTol delta)
+assert(metrics.falsePositiveRate(2.0) ~== fpRate2 absTol delta)
+assert(metrics.precision(0.0) ~== precision0 absTol delta)
+assert(metrics.precision(1.0) ~== precision1 absTol delta)
+assert(metrics.precision(2.0) ~== precision2 absTol delta)
+assert(metrics.recall(0.0) ~== recall0 absTol delta)
+assert(metrics.recall(1.0) ~== recall1 absTol delta)
+assert(metrics.recall(2.0) ~== recall2 absTol delta)
+assert(metrics.fMeasure(0.0) ~== f1measure0 absTol delta)
+assert(metrics.fMeasure(1.0) ~== f1measure1 absTol delta)
+assert(metrics.fMeasure(2.0) ~== f1measure2 absTol delta)
+assert(metrics.fMeasure(0.0, 2.0) ~== f2measure0 absTol delta)
+assert(metrics.fMeasure(1.0, 2.0) ~== f2measure1 absTol delta)
+assert(metrics.fMeasure(2.0, 2.0) ~== f2measure2 absTol delta)
+
+assert(metrics.accuracy ~==
+ (2.0 + 3.0 + 1.0) / ((2 + 3 + 1) + (1 + 1 + 1)) absTol delta)
+assert(metrics.accuracy ~== metrics.precision absTol delta)
+assert(metrics.accuracy ~== metrics.recall absTol delta)
+assert(metrics.accuracy ~== metrics.fMeasure absTol delta)
+assert(metrics.accuracy ~== metrics.weightedRecall absTol delta)
+val weight0 = 4.0 / 9
+val weight1 = 4.0 / 9
+val weight2 = 1.0 / 9
+assert(metrics.weightedTruePositiveRate ~==
+ (weight0 * tpRate0 + weight1 * tpRate1 + weight2 * tpRate2) absTol
delta)
+assert(metrics.weightedFalsePositiveRate ~==
+ (weight0 * fpRate0 + weight1 * fpRate1 + weight2 * fpRate2) absTol
delta)
+assert(metrics.weightedPrecision ~==
+ (weight0 * precision0 + weight1 * precision1 + weight2 * precision2)
absTol delta)
+assert(metrics.weightedRecall ~==
+ (weight0 * recall0 + weight1 * recall1 + weight2 * recall2) absTol
delta)
+assert(metrics.weightedFMeasure ~==
+ (weight0 * f1measure0 + weight1 * f1measure1 + weight2 * f1measure2)
absTol delta)
+assert(metrics.weightedFMeasure(2.0) ~==
+ (weight0 * f2measure0 + weight1 * f2measure1 + weight2 * f2measure2)
absTol delta)
+assert(metrics.labels === labels)
+ }
+
+ test("Multiclass evaluation metrics with weights") {
+/*
+ * Confusion matrix for 3-class classification with total 9 instances
with 2 weights:
+ * |2 * w1|1 * w2 |1 * w1| true class0 (4 instances)
+ * |1 * w2|2 * w1 + 1 * w2|0 | true class1 (4 instances)
+ * |0 |0 |1 * w2| true class2
[GitHub] spark pull request #17086: [SPARK-24101][ML][MLLIB] ML Evaluators should use...
Github user imatiach-msft commented on a diff in the pull request:
https://github.com/apache/spark/pull/17086#discussion_r185163483
--- Diff:
mllib/src/test/scala/org/apache/spark/mllib/evaluation/MulticlassMetricsSuite.scala
---
@@ -95,4 +95,95 @@ class MulticlassMetricsSuite extends SparkFunSuite with
MLlibTestSparkContext {
((4.0 / 9) * f2measure0 + (4.0 / 9) * f2measure1 + (1.0 / 9) *
f2measure2)) < delta)
assert(metrics.labels.sameElements(labels))
}
+
+ test("Multiclass evaluation metrics with weights") {
+/*
+ * Confusion matrix for 3-class classification with total 9 instances
with 2 weights:
+ * |2 * w1|1 * w2 |1 * w1| true class0 (4 instances)
+ * |1 * w2|2 * w1 + 1 * w2|0 | true class1 (4 instances)
+ * |0 |0 |1 * w2| true class2 (1 instance)
+ */
+val w1 = 2.2
+val w2 = 1.5
+val tw = 2.0 * w1 + 1.0 * w2 + 1.0 * w1 + 1.0 * w2 + 2.0 * w1 + 1.0 *
w2 + 1.0 * w2
+val confusionMatrix = Matrices.dense(3, 3,
+ Array(2 * w1, 1 * w2, 0, 1 * w2, 2 * w1 + 1 * w2, 0, 1 * w1, 0, 1 *
w2))
+val labels = Array(0.0, 1.0, 2.0)
+val predictionAndLabelsWithWeights = sc.parallelize(
+ Seq((0.0, 0.0, w1), (0.0, 1.0, w2), (0.0, 0.0, w1), (1.0, 0.0, w2),
+(1.0, 1.0, w1), (1.0, 1.0, w2), (1.0, 1.0, w1), (2.0, 2.0, w2),
+(2.0, 0.0, w1)), 2)
+val metrics = new MulticlassMetrics(predictionAndLabelsWithWeights)
+val delta = 0.001
+val tpRate0 = (2.0 * w1) / (2.0 * w1 + 1.0 * w2 + 1.0 * w1)
+val tpRate1 = (2.0 * w1 + 1.0 * w2) / (2.0 * w1 + 1.0 * w2 + 1.0 * w2)
+val tpRate2 = (1.0 * w2) / (1.0 * w2 + 0)
+val fpRate0 = (1.0 * w2) / (tw - (2.0 * w1 + 1.0 * w2 + 1.0 * w1))
+val fpRate1 = (1.0 * w2) / (tw - (1.0 * w2 + 2.0 * w1 + 1.0 * w2))
+val fpRate2 = (1.0 * w1) / (tw - (1.0 * w2))
+val precision0 = (2.0 * w1) / (2 * w1 + 1 * w2)
+val precision1 = (2.0 * w1 + 1.0 * w2) / (2.0 * w1 + 1.0 * w2 + 1.0 *
w2)
+val precision2 = (1.0 * w2) / (1 * w1 + 1 * w2)
+val recall0 = (2.0 * w1) / (2.0 * w1 + 1.0 * w2 + 1.0 * w1)
+val recall1 = (2.0 * w1 + 1.0 * w2) / (2.0 * w1 + 1.0 * w2 + 1.0 * w2)
+val recall2 = (1.0 * w2) / (1.0 * w2 + 0)
+val f1measure0 = 2 * precision0 * recall0 / (precision0 + recall0)
+val f1measure1 = 2 * precision1 * recall1 / (precision1 + recall1)
+val f1measure2 = 2 * precision2 * recall2 / (precision2 + recall2)
+val f2measure0 = (1 + 2 * 2) * precision0 * recall0 / (2 * 2 *
precision0 + recall0)
+val f2measure1 = (1 + 2 * 2) * precision1 * recall1 / (2 * 2 *
precision1 + recall1)
+val f2measure2 = (1 + 2 * 2) * precision2 * recall2 / (2 * 2 *
precision2 + recall2)
+
+
assert(metrics.confusionMatrix.toArray.sameElements(confusionMatrix.toArray))
--- End diff --
done
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17086: [SPARK-24101][ML][MLLIB] ML Evaluators should use...
Github user WeichenXu123 commented on a diff in the pull request:
https://github.com/apache/spark/pull/17086#discussion_r184584878
--- Diff:
mllib/src/test/scala/org/apache/spark/mllib/evaluation/MulticlassMetricsSuite.scala
---
@@ -55,44 +60,128 @@ class MulticlassMetricsSuite extends SparkFunSuite
with MLlibTestSparkContext {
val f2measure1 = (1 + 2 * 2) * precision1 * recall1 / (2 * 2 *
precision1 + recall1)
val f2measure2 = (1 + 2 * 2) * precision2 * recall2 / (2 * 2 *
precision2 + recall2)
-
assert(metrics.confusionMatrix.toArray.sameElements(confusionMatrix.toArray))
-assert(math.abs(metrics.truePositiveRate(0.0) - tpRate0) < delta)
-assert(math.abs(metrics.truePositiveRate(1.0) - tpRate1) < delta)
-assert(math.abs(metrics.truePositiveRate(2.0) - tpRate2) < delta)
-assert(math.abs(metrics.falsePositiveRate(0.0) - fpRate0) < delta)
-assert(math.abs(metrics.falsePositiveRate(1.0) - fpRate1) < delta)
-assert(math.abs(metrics.falsePositiveRate(2.0) - fpRate2) < delta)
-assert(math.abs(metrics.precision(0.0) - precision0) < delta)
-assert(math.abs(metrics.precision(1.0) - precision1) < delta)
-assert(math.abs(metrics.precision(2.0) - precision2) < delta)
-assert(math.abs(metrics.recall(0.0) - recall0) < delta)
-assert(math.abs(metrics.recall(1.0) - recall1) < delta)
-assert(math.abs(metrics.recall(2.0) - recall2) < delta)
-assert(math.abs(metrics.fMeasure(0.0) - f1measure0) < delta)
-assert(math.abs(metrics.fMeasure(1.0) - f1measure1) < delta)
-assert(math.abs(metrics.fMeasure(2.0) - f1measure2) < delta)
-assert(math.abs(metrics.fMeasure(0.0, 2.0) - f2measure0) < delta)
-assert(math.abs(metrics.fMeasure(1.0, 2.0) - f2measure1) < delta)
-assert(math.abs(metrics.fMeasure(2.0, 2.0) - f2measure2) < delta)
+assert(metrics.confusionMatrix.asML ~== confusionMatrix.asML relTol
delta)
+assert(metrics.truePositiveRate(0.0) ~== tpRate0 absTol delta)
+assert(metrics.truePositiveRate(1.0) ~== tpRate1 absTol delta)
+assert(metrics.truePositiveRate(2.0) ~== tpRate2 absTol delta)
+assert(metrics.falsePositiveRate(0.0) ~== fpRate0 absTol delta)
+assert(metrics.falsePositiveRate(1.0) ~== fpRate1 absTol delta)
+assert(metrics.falsePositiveRate(2.0) ~== fpRate2 absTol delta)
+assert(metrics.precision(0.0) ~== precision0 absTol delta)
+assert(metrics.precision(1.0) ~== precision1 absTol delta)
+assert(metrics.precision(2.0) ~== precision2 absTol delta)
+assert(metrics.recall(0.0) ~== recall0 absTol delta)
+assert(metrics.recall(1.0) ~== recall1 absTol delta)
+assert(metrics.recall(2.0) ~== recall2 absTol delta)
+assert(metrics.fMeasure(0.0) ~== f1measure0 absTol delta)
+assert(metrics.fMeasure(1.0) ~== f1measure1 absTol delta)
+assert(metrics.fMeasure(2.0) ~== f1measure2 absTol delta)
+assert(metrics.fMeasure(0.0, 2.0) ~== f2measure0 absTol delta)
+assert(metrics.fMeasure(1.0, 2.0) ~== f2measure1 absTol delta)
+assert(metrics.fMeasure(2.0, 2.0) ~== f2measure2 absTol delta)
+
+assert(metrics.accuracy ~==
+ (2.0 + 3.0 + 1.0) / ((2 + 3 + 1) + (1 + 1 + 1)) absTol delta)
+assert(metrics.accuracy ~== metrics.precision absTol delta)
+assert(metrics.accuracy ~== metrics.recall absTol delta)
+assert(metrics.accuracy ~== metrics.fMeasure absTol delta)
+assert(metrics.accuracy ~== metrics.weightedRecall absTol delta)
+val weight0 = 4.0 / 9
+val weight1 = 4.0 / 9
+val weight2 = 1.0 / 9
+assert(metrics.weightedTruePositiveRate ~==
+ (weight0 * tpRate0 + weight1 * tpRate1 + weight2 * tpRate2) absTol
delta)
+assert(metrics.weightedFalsePositiveRate ~==
+ (weight0 * fpRate0 + weight1 * fpRate1 + weight2 * fpRate2) absTol
delta)
+assert(metrics.weightedPrecision ~==
+ (weight0 * precision0 + weight1 * precision1 + weight2 * precision2)
absTol delta)
+assert(metrics.weightedRecall ~==
+ (weight0 * recall0 + weight1 * recall1 + weight2 * recall2) absTol
delta)
+assert(metrics.weightedFMeasure ~==
+ (weight0 * f1measure0 + weight1 * f1measure1 + weight2 * f1measure2)
absTol delta)
+assert(metrics.weightedFMeasure(2.0) ~==
+ (weight0 * f2measure0 + weight1 * f2measure1 + weight2 * f2measure2)
absTol delta)
+assert(metrics.labels === labels)
+ }
+
+ test("Multiclass evaluation metrics with weights") {
+/*
+ * Confusion matrix for 3-class classification with total 9 instances
with 2 weights:
+ * |2 * w1|1 * w2 |1 * w1| true class0 (4 instances)
+ * |1 * w2|2 * w1 + 1 * w2|0 | true class1 (4 instances)
+ * |0 |0 |1 * w2| true class2
[GitHub] spark pull request #17086: [SPARK-24101][ML][MLLIB] ML Evaluators should use...
Github user WeichenXu123 commented on a diff in the pull request:
https://github.com/apache/spark/pull/17086#discussion_r184566012
--- Diff:
mllib/src/test/scala/org/apache/spark/mllib/evaluation/MulticlassMetricsSuite.scala
---
@@ -95,4 +95,95 @@ class MulticlassMetricsSuite extends SparkFunSuite with
MLlibTestSparkContext {
((4.0 / 9) * f2measure0 + (4.0 / 9) * f2measure1 + (1.0 / 9) *
f2measure2)) < delta)
assert(metrics.labels.sameElements(labels))
}
+
+ test("Multiclass evaluation metrics with weights") {
+/*
+ * Confusion matrix for 3-class classification with total 9 instances
with 2 weights:
+ * |2 * w1|1 * w2 |1 * w1| true class0 (4 instances)
+ * |1 * w2|2 * w1 + 1 * w2|0 | true class1 (4 instances)
+ * |0 |0 |1 * w2| true class2 (1 instance)
+ */
+val w1 = 2.2
+val w2 = 1.5
+val tw = 2.0 * w1 + 1.0 * w2 + 1.0 * w1 + 1.0 * w2 + 2.0 * w1 + 1.0 *
w2 + 1.0 * w2
+val confusionMatrix = Matrices.dense(3, 3,
+ Array(2 * w1, 1 * w2, 0, 1 * w2, 2 * w1 + 1 * w2, 0, 1 * w1, 0, 1 *
w2))
+val labels = Array(0.0, 1.0, 2.0)
+val predictionAndLabelsWithWeights = sc.parallelize(
+ Seq((0.0, 0.0, w1), (0.0, 1.0, w2), (0.0, 0.0, w1), (1.0, 0.0, w2),
+(1.0, 1.0, w1), (1.0, 1.0, w2), (1.0, 1.0, w1), (2.0, 2.0, w2),
+(2.0, 0.0, w1)), 2)
+val metrics = new MulticlassMetrics(predictionAndLabelsWithWeights)
+val delta = 0.001
+val tpRate0 = (2.0 * w1) / (2.0 * w1 + 1.0 * w2 + 1.0 * w1)
+val tpRate1 = (2.0 * w1 + 1.0 * w2) / (2.0 * w1 + 1.0 * w2 + 1.0 * w2)
+val tpRate2 = (1.0 * w2) / (1.0 * w2 + 0)
+val fpRate0 = (1.0 * w2) / (tw - (2.0 * w1 + 1.0 * w2 + 1.0 * w1))
+val fpRate1 = (1.0 * w2) / (tw - (1.0 * w2 + 2.0 * w1 + 1.0 * w2))
+val fpRate2 = (1.0 * w1) / (tw - (1.0 * w2))
+val precision0 = (2.0 * w1) / (2 * w1 + 1 * w2)
+val precision1 = (2.0 * w1 + 1.0 * w2) / (2.0 * w1 + 1.0 * w2 + 1.0 *
w2)
+val precision2 = (1.0 * w2) / (1 * w1 + 1 * w2)
+val recall0 = (2.0 * w1) / (2.0 * w1 + 1.0 * w2 + 1.0 * w1)
+val recall1 = (2.0 * w1 + 1.0 * w2) / (2.0 * w1 + 1.0 * w2 + 1.0 * w2)
+val recall2 = (1.0 * w2) / (1.0 * w2 + 0)
+val f1measure0 = 2 * precision0 * recall0 / (precision0 + recall0)
+val f1measure1 = 2 * precision1 * recall1 / (precision1 + recall1)
+val f1measure2 = 2 * precision2 * recall2 / (precision2 + recall2)
+val f2measure0 = (1 + 2 * 2) * precision0 * recall0 / (2 * 2 *
precision0 + recall0)
+val f2measure1 = (1 + 2 * 2) * precision1 * recall1 / (2 * 2 *
precision1 + recall1)
+val f2measure2 = (1 + 2 * 2) * precision2 * recall2 / (2 * 2 *
precision2 + recall2)
+
+
assert(metrics.confusionMatrix.toArray.sameElements(confusionMatrix.toArray))
--- End diff --
Oh, that's because you use `Matrices` in `mllib`, change it to `Matrices`
in `ml`, i.e., `import org.apache.spark.ml.linalg.Matrices`
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17086: [SPARK-24101][ML][MLLIB] ML Evaluators should use...
Github user imatiach-msft commented on a diff in the pull request:
https://github.com/apache/spark/pull/17086#discussion_r184454635
--- Diff:
mllib/src/test/scala/org/apache/spark/mllib/evaluation/MulticlassMetricsSuite.scala
---
@@ -95,4 +95,95 @@ class MulticlassMetricsSuite extends SparkFunSuite with
MLlibTestSparkContext {
((4.0 / 9) * f2measure0 + (4.0 / 9) * f2measure1 + (1.0 / 9) *
f2measure2)) < delta)
assert(metrics.labels.sameElements(labels))
}
+
+ test("Multiclass evaluation metrics with weights") {
+/*
+ * Confusion matrix for 3-class classification with total 9 instances
with 2 weights:
+ * |2 * w1|1 * w2 |1 * w1| true class0 (4 instances)
+ * |1 * w2|2 * w1 + 1 * w2|0 | true class1 (4 instances)
+ * |0 |0 |1 * w2| true class2 (1 instance)
+ */
+val w1 = 2.2
+val w2 = 1.5
+val tw = 2.0 * w1 + 1.0 * w2 + 1.0 * w1 + 1.0 * w2 + 2.0 * w1 + 1.0 *
w2 + 1.0 * w2
+val confusionMatrix = Matrices.dense(3, 3,
+ Array(2 * w1, 1 * w2, 0, 1 * w2, 2 * w1 + 1 * w2, 0, 1 * w1, 0, 1 *
w2))
+val labels = Array(0.0, 1.0, 2.0)
+val predictionAndLabelsWithWeights = sc.parallelize(
+ Seq((0.0, 0.0, w1), (0.0, 1.0, w2), (0.0, 0.0, w1), (1.0, 0.0, w2),
+(1.0, 1.0, w1), (1.0, 1.0, w2), (1.0, 1.0, w1), (2.0, 2.0, w2),
+(2.0, 0.0, w1)), 2)
+val metrics = new MulticlassMetrics(predictionAndLabelsWithWeights)
+val delta = 0.001
+val tpRate0 = (2.0 * w1) / (2.0 * w1 + 1.0 * w2 + 1.0 * w1)
+val tpRate1 = (2.0 * w1 + 1.0 * w2) / (2.0 * w1 + 1.0 * w2 + 1.0 * w2)
+val tpRate2 = (1.0 * w2) / (1.0 * w2 + 0)
+val fpRate0 = (1.0 * w2) / (tw - (2.0 * w1 + 1.0 * w2 + 1.0 * w1))
+val fpRate1 = (1.0 * w2) / (tw - (1.0 * w2 + 2.0 * w1 + 1.0 * w2))
+val fpRate2 = (1.0 * w1) / (tw - (1.0 * w2))
+val precision0 = (2.0 * w1) / (2 * w1 + 1 * w2)
+val precision1 = (2.0 * w1 + 1.0 * w2) / (2.0 * w1 + 1.0 * w2 + 1.0 *
w2)
+val precision2 = (1.0 * w2) / (1 * w1 + 1 * w2)
+val recall0 = (2.0 * w1) / (2.0 * w1 + 1.0 * w2 + 1.0 * w1)
+val recall1 = (2.0 * w1 + 1.0 * w2) / (2.0 * w1 + 1.0 * w2 + 1.0 * w2)
+val recall2 = (1.0 * w2) / (1.0 * w2 + 0)
+val f1measure0 = 2 * precision0 * recall0 / (precision0 + recall0)
+val f1measure1 = 2 * precision1 * recall1 / (precision1 + recall1)
+val f1measure2 = 2 * precision2 * recall2 / (precision2 + recall2)
+val f2measure0 = (1 + 2 * 2) * precision0 * recall0 / (2 * 2 *
precision0 + recall0)
+val f2measure1 = (1 + 2 * 2) * precision1 * recall1 / (2 * 2 *
precision1 + recall1)
+val f2measure2 = (1 + 2 * 2) * precision2 * recall2 / (2 * 2 *
precision2 + recall2)
+
+
assert(metrics.confusionMatrix.toArray.sameElements(confusionMatrix.toArray))
--- End diff --
it looks like I needed to change this to an ML matrix instead of MLLIB
matrix in order to make this ~== work, so I used .asML
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
