On 09/08/2026 13:00, Leonid Evdokimov wrote:
On Mon, Jul 27, 2026 at 06:27:17PM +0100, Pádraig Brady wrote:
In summary I was thinking dropping support for the buz32 variant
as it would allow us to avoid specifying the 32/64 variants entirely.
Advantages:
* The user selects the hash family and content window,
not needing to worry about implementation width.
_bit_ with correlation with window _byte_ width is confusing anyway.
* BUZHash window and arithmetic width are independent,
so exposing both dimensions provides little practical value.
Performance on 32-bit CPUs is the only reason to keep 32-bit versions,
and I'm unaware of any other good arguments for them.
I'd prefer to drop both 32-bit hashes (gear32 and buz32), assuming
no one uses 32-bit machines to process "large" files in 2026.
However, thinking that might be a bit of a bold move, I went ahead and
implemented the 32-bit versions as well.
Firefox telemetry (https://data.firefox.com/dashboard/hardware) shows
that 32-bit machines account for 6.8% of users. That's more than 64-bit
ARM, but still not a lot. The only 32-bit architecture Debian supports
nowadays is armhf. I'm not aware of any other data points to guide this
decision.
What do you think? Are the 32-bit versions worth maintaining?
Thanks for confirming my thinking on this.
Specialized 32-bit may be appropriate for a library level interface,
but not appropriate for a CLI level one.
Given this is only for performance and not a functionality advantage
I would _not_ present 32 bit variants in the interface.
The dwindling 32-bit platforms are not enough reason for this.
So the interface I presented in my previous mail
would be simpler and preferred interface.
* BUZHash would no longer have a 42 MB average-size restriction.
gear[32] still has it, as the restriction comes from the hash function
bit-width.
I don't expect 42 MB to be a significant limitation, as it's well above
the target sizes I've seen. Academic papers usually target chunk sizes
around 4–16 KiB, BorgBackup targets 2–8 MiB, and the sweet spot for my
Git-based use case was 384 KiB.
Also for split's use cases I'm not use --random-source is required.
The built-in table already gives portable reproducibility,
and I can't think of split(1) needing keep boundary selection secret.
The idea behind --random-source isn't boundary selection secrecy,
but the ability to specify a different seed value if the built-in
table performs poorly on a particular dataset.
My dataset was quite sensitive to the lookup table when I tried slicing
it with the Gear hash. Ultimately, I decided to use BUZHash instead,
as a larger window turned out to be a more robust way to solve the same
problem.
OK, so given the application of --random-source is quite specialized,
and that window size provides the required flexibility,
it's probably best to drop support for that option also.
For reference, while looking at this area I noticed FastCDC,
which seems to have minimum, average, maximum chunk size as the
configurable parameters. The minimum might be inferred, so that
would suggest a possible future split(1) interface of: fastcdc/AVG/MAX.
I think FastCDC is based on gear64 so could reuse the existing
gear64 update and table, with adjusted cut-point logic.
cheers,
Padraig
