[
https://issues.apache.org/jira/browse/SPARK-15528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Owen resolved SPARK-15528.
-------------------------------
Resolution: Fixed
Assignee: Takeshi Yamamuro
Fix Version/s: 2.0.0
1.6.2
Resolved by https://github.com/apache/spark/pull/13391
> conv function returns inconsistent result for the same data
> -----------------------------------------------------------
>
> Key: SPARK-15528
> URL: https://issues.apache.org/jira/browse/SPARK-15528
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 1.6.1
> Reporter: Lior Regev
> Assignee: Takeshi Yamamuro
> Fix For: 1.6.2, 2.0.0
>
>
> When using F.conv to convert a column from a hexadecimal string to an
> integer, the results are inconsistent
> val col = F.conv(df("some_col"), 16, 10)
> val a = df.select(F.countDistinct("some_col"), F.countDistinct(col)).collect()
> val b = df.select(F.countDistinct("some_col"), F.countDistinct(col)).collect()
> returns:
> a: Array[org.apache.spark.sql.Row] = Array([59776,1941936])
> b: Array[org.apache.spark.sql.Row] = Array([59776,1965154])
> P.S.
> "some_col" is a md5 hash of some string column calculated using F.md5
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
