Github user VinceShieh commented on a diff in the pull request:
https://github.com/apache/spark/pull/14858#discussion_r77465278
--- Diff:
mllib/src/main/scala/org/apache/spark/ml/feature/QuantileDiscretizer.scala ---
@@ -114,10 +115,10 @@ final class QuantileDiscretizer @Since("1.6.0")
(@Since("1.6.0") override val ui
splits(0) = Double.NegativeInfinity
splits(splits.length - 1) = Double.PositiveInfinity
- val distinctSplits = splits.distinct
+ val distinctSplits = splits.filter(!_.isNaN).distinct
--- End diff --
@srowen then maybe we should, as we discussed earlier on JIRA, align with
R, by having a NaN checker in approxQuantile, that is, having a NaN filter
inside of approxQuantile, rather than ahead of calling approxQuantile.
We can also have a same flag for user to choose to either remove NaN values
or throw an error when there is NaN in data, although, this API change will
introduce collateral impact on several existing function calls.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
