[
https://issues.apache.org/jira/browse/SPARK-32109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17147328#comment-17147328
]
Chen Zhang commented on SPARK-32109:
------------------------------------
The logic in the source code can be represented by the following pseudocode.
{code:scala}
def computeHash(value: Any, hashSeed: Long): Long = {
value match {
case null => hashSeed
case b: Boolean => hashInt(if (b) 1 else 0, hashSeed) // Murmur3Hash
case i: Int => hashInt(i, hashSeed)
...
}
}
val seed = 42L
var hash = seed
var i = 0
val len = columns.length
while (i < len) {
hash = computeHash(columns(i).value, hash)
i += 1
}
hash
{code}
I can solve this problem by modifying the following code.
(eval function and doGenCode function in
org.apache.spark.sql.catalyst.expressions.HashExpression class）
{code:scala}
override def eval(input: InternalRow = null): Any = {
var hash = seed
var i = 0
val len = children.length
while (i < len) {
//hash = computeHash(children(i).eval(input), children(i).dataType, hash)
hash = (31 * hash) + computeHash(children(i).eval(input),
children(i).dataType, hash)
i += 1
}
hash
}
{code}
But I don't think it's necessary to modify the code, and if we do, it will
affect the existing data distribution.
> SQL hash function handling of nulls makes collision too likely
> --------------------------------------------------------------
>
> Key: SPARK-32109
> URL: https://issues.apache.org/jira/browse/SPARK-32109
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.0.0
> Reporter: koert kuipers
> Priority: Minor
>
> this ticket is about org.apache.spark.sql.functions.hash and sparks handling
> of nulls when hashing sequences.
> {code:java}
> scala> spark.sql("SELECT hash('bar', null)").show()
> +---------------+
> |hash(bar, NULL)|
> +---------------+
> | -1808790533|
> +---------------+
> scala> spark.sql("SELECT hash(null, 'bar')").show()
> +---------------+
> |hash(NULL, bar)|
> +---------------+
> | -1808790533|
> +---------------+
> {code}
> these are differences sequences. e.g. these could be positions 0 and 1 in a
> dataframe which are diffferent columns with entirely different meanings. the
> hashes should not be the same.
> another example:
> {code:java}
> scala> Seq(("john", null), (null, "john")).toDF("name",
> "alias").withColumn("hash", hash(col("name"), col("alias"))).show
> +----+-----+---------+
> |name|alias| hash|
> +----+-----+---------+
> |john| null|487839701|
> |null| john|487839701|
> +----+-----+---------+ {code}
> instead of ignoring nulls each null show do a transform to the hash so that
> the order of elements including the nulls matters for the outcome.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
