[
https://issues.apache.org/jira/browse/SPARK-17141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15427409#comment-15427409
]
Alberto Bonsanto commented on SPARK-17141:
------------------------------------------
Crude data.
| id|chicken|jam|roast beef|
| 1| NaN|2.0| 2.0|
| 2| 2.0|0.0| 2.0|
| 3| NaN|0.0| 2.0|
| 4| 2.0|1.0| -2.0|
| 5| 2.0|2.0| 2.0|
| 6| 2.0|2.0| NaN|
After assemble and normalization, as you can see {{Double.NaN}} are replaced
for {{0.5}}.
|id |chicken|jam|roast beef|features |featuresNorm |
|1 |NaN |2.0|2.0 |[NaN,2.0,2.0] |[0.5,1.0,1.0]|
|2 |2.0 |0.0|2.0 |[2.0,0.0,2.0] |[0.5,0.0,1.0]|
|3 |NaN |0.0|2.0 |[NaN,0.0,2.0] |[0.5,0.0,1.0]|
|4 |2.0 |1.0|-2.0 |[2.0,1.0,-2.0]|[0.5,0.5,0.0]|
|5 |2.0 |2.0|2.0 |[2.0,2.0,2.0] |[0.5,1.0,1.0]|
|6 |2.0 |2.0|NaN |[2.0,2.0,NaN] |[0.5,1.0,NaN]|
> MinMaxScaler behaves weird when min and max have the same value and some
> values are NaN
> ---------------------------------------------------------------------------------------
>
> Key: SPARK-17141
> URL: https://issues.apache.org/jira/browse/SPARK-17141
> Project: Spark
> Issue Type: Bug
> Components: ML
> Affects Versions: 1.6.2, 2.0.0
> Environment: Databrick's Community, Spark 2.0 + Scala 2.10
> Reporter: Alberto Bonsanto
> Priority: Trivial
>
> When you have a {{DataFrame}} with a column named {{features}}, which is a
> {{DenseVector}} and the *maximum* and *minimum* and some values are
> {{Double.NaN}} they get replaced by 0.5, and they should remain with the same
> value, I believe.
> I know how to fix it, but I haven't ever made a pull request. You can check
> the bug in this
> [notebook|https://databricks-prod-cloudfront.cloud.databricks.com/public/4027ec902e239c93eaaa8714f173bcfc/2485090270202665/3126465289264547/8589256059752547/latest.html]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
