[
https://issues.apache.org/jira/browse/SPARK-58677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nathan Holland updated SPARK-58677:
-----------------------------------
Description:
This ticket expands on SPARK-45900, to explicitly mention the desire for a
BINARY return type for xxhash3_128 as well as additional functionality that a
modern hashing algorithm on Spark could implement. (such as deterministic
support for complex types)
The original intent of that ticket was to capture a new hashing model through
XXH3 that allowed for high performance hashing using XXH3 with the API of the
earlier XXHash64 hash family.
h2. Return Type
I believe a binary return function should be supported I believe there is
precedence for this decision in other implementations:
* [https://duckdb.org/community_extensions/extensions/hashfuncs] (duckdb
communit hashfuncs sporting *xxh3_128* and {*}xxh3_128_hex{*}) *- I think this
is the best method to allow for a hex code when needed, while still allowing a
byte[] for performance and compact storage when needed.*
*
[https://clickhouse.com/docs/reference/functions/regular-functions/hash-functions#xxh3_128]
(Clickhouse definition 128 unsigned int which without native support is
approximately equivalent to a byte[16])
*
[https://learn.microsoft.com/en-us/dotnet/api/system.io.hashing.xxhash128.hash?view=net-11.0-pp#system-io-hashing-xxhash128-hash(system-byte())]
(C# implementation returning byte[16])
* [https://github.com/Cyan4973/xxHash/blob/dev/xxhash.h#L1378] (the reference
using two long's within a struct, technically we could return a struct with a
high and a low long like this, but I would worry about allocations until
Valhalla drops and I feel like byte[16] is probably a better fit anyway)
I would suggest that adding a "xxh3_128_binary" now that xxh3_128 is stabilised
in 4.4.0
* xxh3_64(...) -> BIGINT \{{ (as current)}}
* xxh3_128_binary(...) -> 16-byte BINARY
* xxh3_128(...) -> 32-character lowercase STRING{{{{}} (as current){}}}
h2. XXHash64 Style Hash API
I believe one aspect that's limiting the XXH3 implementation is supporting only
a single column as input. I believe XXHash64's style of supporting multiple
columns reduces the implementation of brittle solutions (often something like
concat_ws or to_json) and would allow XXH3 to be seen successor to the XXHash64
function while keeping all the semantics and compatibility you'd expect out of
the unitary function.
Links:
- [https://issues.apache.org/jira/browse/SPARK-45900] (Original SPARK-45900
ticket)
was:
This ticket expands on SPARK-45900, to explicitly mention the desire for a
BINARY return type for xxhash3_128 as well as additional functionality that a
modern hashing algorithm on Spark could implement. (such as deterministic
support for complex types)
The original intent of that ticket was to capture a new hashing model through
XXH3 that allowed for high performance hashing using XXH3 with the API of the
earlier XXHash64 hash family.
h2. Return Type
I believe a binary return function should be supported I believe there is
precedence for this decision in other implementations:
* [https://duckdb.org/community_extensions/extensions/hashfuncs] (duckdb
communit hashfuncs sporting *xxh3_128* and {*}xxh3_128_hex{*}) *- I think this
is the best method to allow for a hex code when needed, while still allowing a
byte[] for performance and compact storage when needed.*
*
[https://clickhouse.com/docs/reference/functions/regular-functions/hash-functions#xxh3_128]
(Clickhouse definition 128 unsigned int which without native support is
approximately equivalent to a byte[16])
*
[https://learn.microsoft.com/en-us/dotnet/api/system.io.hashing.xxhash128.hash?view=net-11.0-pp#system-io-hashing-xxhash128-hash(system-byte())]
(C# implementation returning byte[16])
* [https://github.com/Cyan4973/xxHash/blob/dev/xxhash.h#L1378] (the reference
using two long's within a struct, technically we could return a struct with a
high and a low long like this, but I would worry about allocations until
Valhalla drops and I feel like byte[16] is probably a better fit anyway)
I would suggest that adding a "xxh3_128_bin" would probably get us there now
that xxh3_128 is stabilised in 4.4.0
* xxh3_64(...) -> BIGINT {{ (as current)}}
* xxh3_128_bin(...) -> 16-byte BINARY
* xxh3_128(...) -> 32-character lowercase STRING{{{{}} (as current){}}}
h2. XXHash64 Style Hash API
I believe one aspect that's limiting the XXH3 implementation is supporting only
a single column as input. I believe XXHash64's style of supporting multiple
columns reduces the implementation of brittle solutions (often something like
concat_ws or to_json) and would allow XXH3 to be seen successor to the XXHash64
function while keeping all the semantics and compatibility you'd expect out of
the unitary function.
Links:
- [https://issues.apache.org/jira/browse/SPARK-45900] (Original SPARK-45900
ticket)
> Expand hash functionalities of XXH3 to better match XXHash64
> ------------------------------------------------------------
>
> Key: SPARK-58677
> URL: https://issues.apache.org/jira/browse/SPARK-58677
> Project: Spark
> Issue Type: Improvement
> Components: PySpark, SQL
> Affects Versions: 4.5.0
> Reporter: Nathan Holland
> Priority: Major
>
> This ticket expands on SPARK-45900, to explicitly mention the desire for a
> BINARY return type for xxhash3_128 as well as additional functionality that a
> modern hashing algorithm on Spark could implement. (such as deterministic
> support for complex types)
> The original intent of that ticket was to capture a new hashing model through
> XXH3 that allowed for high performance hashing using XXH3 with the API of the
> earlier XXHash64 hash family.
> h2. Return Type
> I believe a binary return function should be supported I believe there is
> precedence for this decision in other implementations:
> * [https://duckdb.org/community_extensions/extensions/hashfuncs] (duckdb
> communit hashfuncs sporting *xxh3_128* and {*}xxh3_128_hex{*}) *- I think
> this is the best method to allow for a hex code when needed, while still
> allowing a byte[] for performance and compact storage when needed.*
> *
> [https://clickhouse.com/docs/reference/functions/regular-functions/hash-functions#xxh3_128]
> (Clickhouse definition 128 unsigned int which without native support is
> approximately equivalent to a byte[16])
> *
> [https://learn.microsoft.com/en-us/dotnet/api/system.io.hashing.xxhash128.hash?view=net-11.0-pp#system-io-hashing-xxhash128-hash(system-byte())]
> (C# implementation returning byte[16])
> * [https://github.com/Cyan4973/xxHash/blob/dev/xxhash.h#L1378] (the
> reference using two long's within a struct, technically we could return a
> struct with a high and a low long like this, but I would worry about
> allocations until Valhalla drops and I feel like byte[16] is probably a
> better fit anyway)
> I would suggest that adding a "xxh3_128_binary" now that xxh3_128 is
> stabilised in 4.4.0
> * xxh3_64(...) -> BIGINT \{{ (as current)}}
> * xxh3_128_binary(...) -> 16-byte BINARY
> * xxh3_128(...) -> 32-character lowercase STRING{{{{}} (as current){}}}
> h2. XXHash64 Style Hash API
> I believe one aspect that's limiting the XXH3 implementation is supporting
> only a single column as input. I believe XXHash64's style of supporting
> multiple columns reduces the implementation of brittle solutions (often
> something like concat_ws or to_json) and would allow XXH3 to be seen
> successor to the XXHash64 function while keeping all the semantics and
> compatibility you'd expect out of the unitary function.
> Links:
> - [https://issues.apache.org/jira/browse/SPARK-45900] (Original SPARK-45900
> ticket)
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]