srowen commented on a change in pull request #24458: [SPARK-27540][MLlib] Add
'meanAveragePrecision_at_k' metric to RankingMetrics
URL: https://github.com/apache/spark/pull/24458#discussion_r278989871
##########
File path:
mllib/src/main/scala/org/apache/spark/mllib/evaluation/RankingMetrics.scala
##########
@@ -92,6 +92,56 @@ class RankingMetrics[T: ClassTag](predictionAndLabels:
RDD[(Array[T], Array[T])]
}.mean()
}
+ /**
+ * Returns the mean average precision (MAP) at ranking position k of all the
queries.
+ * If a query has an empty ground truth set, the average precision will be
zero and a log
+ * warning is generated.
+ * @param k the position to compute the truncated precision, must be positive
+ * @return the mean average precision at first k ranking positions
+ */
+ @Since("3.0.0")
+ def meanAveragePrecisionAt(k: Int): Double = {
+ require(k > 0, "ranking position k should be positive")
+ predictionAndLabels
+ .map {
+ case (pred, lab) =>
+ averagePrecisionAt(pred, lab, k)
+ }
+ .mean()
+ }
+
+ /**
+ * Computes the average precision at first k ranking positions of all the
queries.
+ * If a query has an empty ground truth set, the value will be zero and a log
+ * warning is generated.
+ *
+ * @param pred predicted ranking
+ * @param lab ground truth
+ * @param k use the top k predicted ranking, must be positive
+ * @return average precision at first k ranking positions
+ */
+ private def averagePrecisionAt(pred: Array[T], lab: Array[T], k: Int):
Double = {
+ val labSet = lab.toSet
+
+ if (labSet.nonEmpty) {
+ var i = 0
+ var cnt = 0
+ var precSum = 0.0
+ val n = math.min(k, pred.length)
+ while (i < n) {
+ if (labSet.contains(pred(i))) {
+ cnt += 1
+ precSum += cnt.toDouble / (i + 1)
+ }
+ i += 1
+ }
+ precSum / math.min(labSet.size, k)
Review comment:
That's a good question. I do see that minimum, but it seems inconsistent
with the wikipedia definition they point to (which is what I'm going on for
Spark too). The 'number of relevant documents' doesn't depend on k. If it's
capped at k then I think you'd end up with a weird looking precision-recall
curve that would start at 1 when precision is 1, rather than 0. And average
precision is the area under that curve.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
