Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/14858#discussion_r76953758
--- Diff:
mllib/src/main/scala/org/apache/spark/ml/feature/QuantileDiscretizer.scala ---
@@ -114,11 +114,9 @@ final class QuantileDiscretizer @Since("1.6.0")
(@Since("1.6.0") override val ui
splits(0) = Double.NegativeInfinity
splits(splits.length - 1) = Double.PositiveInfinity
- val distinctSplits = splits.distinct
- if (splits.length != distinctSplits.length) {
- log.warn(s"Some quantiles were identical. Bucketing to
${distinctSplits.length - 1}" +
- s" buckets as a result.")
- }
+ val distinctSplits = splits.filter(!_.isNaN).distinct
+ log.info(s"The exact number of returned buckets might differ from what
was requested" +
--- End diff --
OK, though now this is logged unconditionally. I think we'd only want to
log (an info-level message) if the number of buckets didn't match the request,
which is what you had previously?
Separately, I think the behavior of NaN has to be documented somewhere in
this class too, to make people aware that it's _always_ possible to get an
extra bucket of data if there are NaNs.
Otherwise looking good to me.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
