[
https://issues.apache.org/jira/browse/SPARK-17987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15588085#comment-15588085
]
Sean Owen commented on SPARK-17987:
-----------------------------------
In your example, a predictor is missing, not the predicted value. I am sure
we've had this conversation before with [~josephkb] who indicated that the
current behavior is on purpose.
> ML Evaluator fails to handle null values in the dataset
> -------------------------------------------------------
>
> Key: SPARK-17987
> URL: https://issues.apache.org/jira/browse/SPARK-17987
> Project: Spark
> Issue Type: Improvement
> Components: ML
> Affects Versions: 1.6.2, 2.0.1
> Reporter: bo song
>
> Take the RegressionEvaluator as an example, when the predictionCol is null in
> a row, en exception "scala.MatchEror" will be thrown. The missing null
> prediction is a common case, for example when an predictor is missing, or its
> value is out of bound, almost machine learning models could not produce
> correct predictions, then null predictions would be returned. Evaluators
> should handle the null values instead of an exception thrown, the common way
> to handle missing null values is to ignore them. Besides of the null value,
> the NAN value need to be handled correctly too.
> Those three evaluators RegressionEvaluator, BinaryClassificationEvaluator and
> MulticlassClassificationEvaluator have the same problem.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
