Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/14858#discussion_r77133636
--- Diff:
mllib/src/main/scala/org/apache/spark/ml/feature/QuantileDiscretizer.scala ---
@@ -114,10 +115,10 @@ final class QuantileDiscretizer @Since("1.6.0")
(@Since("1.6.0") override val ui
splits(0) = Double.NegativeInfinity
splits(splits.length - 1) = Double.PositiveInfinity
- val distinctSplits = splits.distinct
+ val distinctSplits = splits.filter(!_.isNaN).distinct
--- End diff --
Ah, I think this is a little bit at odds with the intent. We need to filter
NaN before the data goes to approxQuantile. They should have no input on the
quantiles. Then there's no need to filter out NaN from splits.
The message below can then remain unchanged from what it was before. This
message is not related to the behavior of NaN.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
