[
https://issues.apache.org/jira/browse/SPARK-27099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wenchen Fan resolved SPARK-27099.
---------------------------------
Resolution: Fixed
Fix Version/s: 3.0.0
Issue resolved by pull request 24019
[https://github.com/apache/spark/pull/24019]
> Expose xxHash64 as a flexible 64-bit column hash like `hash`
> ------------------------------------------------------------
>
> Key: SPARK-27099
> URL: https://issues.apache.org/jira/browse/SPARK-27099
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 2.3.3, 2.4.0
> Reporter: Huon Wilson
> Assignee: Huon Wilson
> Priority: Major
> Fix For: 3.0.0
>
>
> I’m working on something that requires deterministic randomness, i.e. a row
> gets the same “random” value no matter the order of the DataFrame. A seeded
> hash seems to be the perfect way to do this, but the existing hashes have
> various limitations:
> - hash: 32-bit output (only 4 billion possibilities will result in a lot of
> collisions for many tables: the birthday paradox implies >50% chance of at
> least one for tables larger than 77000 rows, and likely ~1.6 billion
> collisions in a table of size 4 billion)
> - sha1/sha2/md5: single binary column input, string output
> It seems there’s already support for a 64-bit hash function that can work
> with an arbitrary number of arbitrary-typed columns (XxHash64), which could
> be exposed as xxHash64 or xxhash64 (or similar).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
