[
https://issues.apache.org/jira/browse/SPARK-13435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15157015#comment-15157015
]
zhengruifeng commented on SPARK-13435:
--------------------------------------
I dont think so.
Recently, many Competitions use quadratic weighted kappa as the evaluation
metrics.
such as:
https://www.kaggle.com/c/diabetic-retinopathy-detection/details/evaluation
https://www.kaggle.com/c/prudential-life-insurance-assessment/details/evaluation
...
The unweighted kappa is very easy to compute, especially for binary
classification. But the weighted one is not so obvious, and cause many
confusion. You can find in kaggle's forum that many people are confused with it.
> Add Weighted Cohen's kappa to MulticlassMetrics
> -----------------------------------------------
>
> Key: SPARK-13435
> URL: https://issues.apache.org/jira/browse/SPARK-13435
> Project: Spark
> Issue Type: Improvement
> Components: MLlib
> Reporter: zhengruifeng
> Priority: Minor
>
> Add the missing Weighted Cohen's kappa to MulticlassMetrics.
> Kappa is widely used in Competition and Statistics.
> https://en.wikipedia.org/wiki/Cohen's_kappa
> Some usage examples:
> val metrics = new MulticlassMetrics(predictionAndLabels)
> // The default kappa value (Unweighted kappa)
> val kappa = metrics.kappa
> // Three built-in weighting type ("default":unweighted, "linear":linear
> weighted, "quadratic":quadratic weighted)
> val kappa = metrics.kappa("quadratic")
> // User-defined weighting matrix
> val matrix = Matrices.dense(n, n, values)
> val kappa = metrics.kappa(matrix)
> // User-defined weighting function
> def getWeight(i: Int, j:Int):Double = {
> if (i == j) {
> 0.0
> } else {
> 1.0
> }
> }
> val kappa = metrics.kappa(getWeight) // equals to the unweighted kappa
> The calculation correctness was tested on several small data, and compared to
> two python's package: sklearn and ml_metrics.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
