philipandwaffle opened a new issue, #66704: URL: https://github.com/apache/doris/issues/66704
### Search before asking - [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Description I'm attempting to do duplicate analysis by hashing several fields and storing the hash. If the hash already exists then the item is determined as a duplicate. Using XXHash_64 results in a hash collision chance of ~93% with 10 billion items (based on this calculator https://kevingal.com/apps/collision.html). I could use another hash function such as MD5 or SHA. In the case of MD5 it's much slower and would effect performance. In the case of SHA I would need to store the hash as a string as the largest integer type is 128 bits, which prevents me from using the hash column as a key. From what I can see implementing a XXHASH_128 function would be the best option, though I might be overlooking something as I've only started using Doris. Any suggestions are welcome, thanks. ### Solution Implement XXHASH_128. I'm willing to submit a PR but I'm not sure where to start, any guidance would be appreciated. ### Are you willing to submit PR? - [x] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
