Hi all,
Posting before opening a PR per the customary convention. I have a
working, tested implementation that adds SHA-3 (sha3-256, sha3-384,
sha3-512) to the algorithms accepted by --checksum-choice.
I want to lead with the question I expect first, because rsync already
ships a generous set of hashes:
Why add SHA-3 when rsync already has xxh128 / xxh3 / md5 / sha1 /
sha256 / sha512?
Five reasons. None of them is "SHA-2 is broken" -- it isn't, today.
1. Different cryptographic design family.
SHA-2 (sha256, sha512) uses the Merkle-Damgaard construction with
a Davies-Meyer compression function. SHA-3 uses the sponge
construction over the Keccak-f[1600] permutation. These share no
structural elements. A theoretical break in one would not imply a
break in the other. Cryptographic agility -- the ability to swap
to a structurally-independent primitive without redesigning the
surrounding system -- is a property worth having, especially in
tools whose output (rsync's --checksum verification, batch files,
--link-dest farms) can persist for years.
2. Length-extension immunity.
SHA-256 and SHA-512 are vulnerable to length-extension attacks:
given H(s || m), an attacker who does not know s can compute
H(s || m || pad || m') without knowing s. rsync does not directly
use a naive secret-prefix MAC, but operators occasionally do build
integrity-verification pipelines on top of "rsync checksum + extra
data". SHA-3's sponge construction is structurally immune to this
class of attack; SHA-2 requires HMAC-style wrapping to avoid it.
3. FIPS 202 / compliance.
SHA-3 is FIPS 202 (2015). A growing set of regulated environments
(FIPS 140-3 module validations, some federal procurement contexts,
defense-adjacent contracts) explicitly require or prefer SHA-3 for
new deployments. Operators in those environments today either
accept "rsync's strongest hash is SHA-2, that has to be good enough"
or wrap rsync in external integrity tooling. Native SHA-3 removes
the friction.
4. No new dependencies.
This is the boring but important one. SHA-3 has been in OpenSSL's
EVP since 1.1.1 (released September 2018) -- which means it is in
every libcrypto rsync already links against on every supported
distro. No new vendored crypto code, no autoconf detection, no new
build dependency. The implementation is exactly:
{ CSUM_SHA3_512, NNI_EVP, "sha3-512", NULL },
{ CSUM_SHA3_384, NNI_EVP, "sha3-384", NULL },
{ CSUM_SHA3_256, NNI_EVP, "sha3-256", NULL },
plus length/canonical case entries. The existing verify_digest()
probe handles older libcrypto cleanly: if the linked OpenSSL doesn't
expose sha3-*, the entry is demoted to CSUM_gone at init and a user
asking for --checksum-choice=sha3-256 gets the familiar "unknown
checksum name" error.
5. Hardware acceleration is starting to land.
ARMv8.2-A's SHA-3 extension (EOR3, RAX1, XAR, BCAX instructions) is
shipping in production silicon -- Apple M-series, AWS Graviton 3+,
recent Snapdragon and MediaTek parts. On those parts SHA-3 can
outperform software SHA-2. Intel/AMD don't have direct SHA-3
instructions today but OpenSSL's optimized software path is
competitive. No advantage on commodity x86 in 2026, but the
trajectory matters.
A few things SHA-3 is NOT being offered for:
- It is not a replacement for xxh3. xxh3 / xxh128 are
non-cryptographic and very fast; they are the right default for
"detect random corruption in a trusted-LAN backup." SHA-3 is for
cases where the threat model includes adversarial collision
construction or compliance.
- It is not being made the default. The "auto" negotiation path is
untouched. An unmodified peer keeps negotiating xxh128 / xxh3 /
md5 / MD4 as before. SHA-3 has to be requested explicitly via
--checksum-choice=sha3-* on both sides (or via the
RSYNC_CHECKSUM_LIST environment override).
- It is not a protocol bump. Negotiation in 3.2+ is already
capability-based via the name_num_obj table; adding entries to
that table is the entire mechanism that xxh3 / xxh128 use.
DESIGN
Wholly through OpenSSL EVP, no vendored hash code:
- 3 entries in valid_checksums_items[] with NNI_EVP, named
"sha3-256" / "sha3-384" / "sha3-512" so that EVP_get_digestbyname()
resolves them directly with no special-case in csum_evp_md().
- 3 new CSUM_SHA3_{256,384,512} constants in lib/md-defines.h, plus
SHA3_{256,384,512}_DIGEST_LEN literals. (OpenSSL never published
SHA3_*_DIGEST_LENGTH macros, so the literal lengths come from us.)
- Length cases in csum_len_for_type(), and -1 in canonical_checksum()
to match the other strong hashes' byte order.
- No new code paths in get_checksum2(), file_checksum(), or
sum_init/update/end. They all short-circuit to the EVP path when
xfer_sum_evp_md / file_sum_evp_md / cur_sum_evp_md is set, so SHA-3
just rides the existing rails -- same as sha256 and sha512.
VERIFIED
Built on Ubuntu 22.04 / OpenSSL 3.0.2 / gcc 11.4, against upstream
master at b9cc0c6 (3.4.2 + 12 commits, protocol 32).
* rsync --version "Checksum list" now reads:
xxh128 xxh3 xxh64 (xxhash) sha3-512 sha3-384 sha3-256
md5 md4 sha1 none
* FIPS 202 vector match:
SHA3-256("abc") = 3a985da7...1532 (correct)
* 1 GiB urandom file, --checksum-choice={sha3-256,sha3-384,sha3-512}:
Local copy: byte-identical, digest matches reference
openssl dgst output in all three.
rsync:// PULL: same. (daemon and client both built from this
patch, listening on 127.0.0.1:8730.)
rsync:// PUSH: same.
Delta-update path: corrupted 16 x 64 KiB blocks scattered through
a 1 GiB destination, resync with sha3-256 and
--no-whole-file. rsync sent 1.5 MiB literal +
1.7 MiB metadata, matched 1022 MiB of existing
blocks (555x speedup), and the assembled file
was byte-identical with matching sha3-256
digest. Confirms the rolling-block strong-
checksum path is wired correctly.
* With --debug=NSTR, auto negotiation still picks xxh128 by default;
SHA-3 is selected only when explicitly requested.
* `make check` passes the new testsuite/checksum-sha3.test. (The
pre-existing 'devices' failure on the test host is environmental
and reproduces on unmodified master -- it needs root to create
/dev/* nodes.)
DIFFSTAT
NEWS.md | 11 +++++
checksum.c | 20 +++++++++
lib/md-defines.h | 12 +++++
rsync.1.md | 7 +++
testsuite/checksum-sha3.test | 60 ++++++++++++++++++++++++
5 files changed, 110 insertions(+)
OPEN QUESTIONS FOR THE LIST
(a) Algorithm names. I used "sha3-256" / "sha3-384" / "sha3-512"
to mirror OpenSSL's EVP digest names directly so the lookup is
transparent. Happy to switch to "sha3_256" / etc. if the house
style prefers underscores.
(b) Single commit, or split implementation / docs / test into a
three-commit series? I've gone with one commit since the change
is small and tightly coupled, but I'm easy.
(c) Anything missing on the protocol / negotiation side I should
worry about? As far as I can see the name_num_obj negotiation
handles new entries automatically and no protocol-version bump
is required, but I'd rather have that confirmed than assumed.
If reception here is positive I'll open a PR against
RsyncProject/rsync and link this thread from it.
Patch inline below. Also happy to attach as a file or send via
git-send-email if anyone prefers that form.
Thanks,
Deepak Jain
AiNET Factory
[email protected]
---
NEWS.md | 11 +++++
checksum.c | 20 +++++++++
lib/md-defines.h | 12 +++++
rsync.1.md | 7 +++
testsuite/checksum-sha3.test | 60 ++++++++++++++++++++++++
5 files changed, 110 insertions(+)
create mode 100755 testsuite/checksum-sha3.test
diff --git a/NEWS.md b/NEWS.md
index c4a73d6..3c659ba 100644
--- a/NEWS.md
+++ b/NEWS.md
@@ -2,6 +2,17 @@
## Changes in this version:
+### ENHANCEMENTS:
+
+- Added SHA-3 (FIPS 202) as an opt-in `--checksum-choice` algorithm
+ family: `sha3-256`, `sha3-384`, and `sha3-512`. The implementation
+ uses OpenSSL EVP and so requires that rsync be built with OpenSSL
+ support and linked against libcrypto >= 1.1.1. Like `sha256` and
+ `sha512`, SHA-3 is never picked by `auto` negotiation; it must be
+ explicitly requested on both ends (e.g.
+ `--checksum-choice=sha3-256`) or made part of the
+ `RSYNC_CHECKSUM_LIST` environment override.
+
### BUG FIXES:
- Fixed a regression introduced by the 3.4.0 secure_relative_open()
diff --git a/checksum.c b/checksum.c
index 24e46bf..f357aa5 100644
--- a/checksum.c
+++ b/checksum.c
@@ -54,6 +54,15 @@ struct name_num_item valid_checksums_items[] = {
#ifdef SUPPORT_XXHASH
{ CSUM_XXH64, 0, "xxh64", NULL },
{ CSUM_XXH64, 0, "xxhash", NULL },
+#endif
+#ifdef USE_OPENSSL
+ /* SHA-3 is offered only via OpenSSL EVP (sha3-* digests,
OpenSSL >= 1.1.1).
+ * verify_digest() probes each entry at startup and disables any
that the
+ * linked libcrypto does not support, so listing them here is
safe even
+ * when the build environment is older. */
+ { CSUM_SHA3_512, NNI_EVP, "sha3-512", NULL },
+ { CSUM_SHA3_384, NNI_EVP, "sha3-384", NULL },
+ { CSUM_SHA3_256, NNI_EVP, "sha3-256", NULL },
#endif
{ CSUM_MD5, NNI_BUILTIN|NNI_EVP, "md5", NULL },
{ CSUM_MD4, NNI_BUILTIN|NNI_EVP, "md4", NULL },
@@ -237,6 +246,14 @@ int csum_len_for_type(int cst, BOOL flist_csum)
#ifdef SHA512_DIGEST_LENGTH
case CSUM_SHA512:
return SHA512_DIGEST_LENGTH;
+#endif
+#ifdef USE_OPENSSL
+ case CSUM_SHA3_256:
+ return SHA3_256_DIGEST_LEN;
+ case CSUM_SHA3_384:
+ return SHA3_384_DIGEST_LEN;
+ case CSUM_SHA3_512:
+ return SHA3_512_DIGEST_LEN;
#endif
case CSUM_XXH64:
case CSUM_XXH3_64:
@@ -266,6 +283,9 @@ int canonical_checksum(int csum_type)
case CSUM_SHA1:
case CSUM_SHA256:
case CSUM_SHA512:
+ case CSUM_SHA3_256:
+ case CSUM_SHA3_384:
+ case CSUM_SHA3_512:
return -1;
case CSUM_XXH64:
case CSUM_XXH3_64:
diff --git a/lib/md-defines.h b/lib/md-defines.h
index 6ef6a68..0b58c57 100644
--- a/lib/md-defines.h
+++ b/lib/md-defines.h
@@ -10,6 +10,15 @@
#define MD4_DIGEST_LEN 16
#define MD5_DIGEST_LEN 16
+
+/* SHA-3 (FIPS 202). OpenSSL exposes these via EVP only, with no
+ * SHA3_*_DIGEST_LENGTH macros, so define our own. SHA3-512 shares
+ * its 64-byte digest length with SHA-512, so MAX_DIGEST_LEN is
+ * unaffected. */
+#define SHA3_256_DIGEST_LEN 32
+#define SHA3_384_DIGEST_LEN 48
+#define SHA3_512_DIGEST_LEN 64
+
#if defined SHA512_DIGEST_LENGTH
#define MAX_DIGEST_LEN SHA512_DIGEST_LENGTH
#elif defined SHA256_DIGEST_LENGTH
@@ -35,3 +44,6 @@
#define CSUM_SHA1 9
#define CSUM_SHA256 10
#define CSUM_SHA512 11
+#define CSUM_SHA3_256 12
+#define CSUM_SHA3_384 13
+#define CSUM_SHA3_512 14
diff --git a/rsync.1.md b/rsync.1.md
index 2b4b750..0e51d44 100644
--- a/rsync.1.md
+++ b/rsync.1.md
@@ -1781,11 +1781,18 @@ expand it.
- `xxh128`
- `xxh3`
- `xxh64` (aka `xxhash`)
+ - `sha3-512`
+ - `sha3-384`
+ - `sha3-256`
- `md5`
- `md4`
- `sha1`
- `none`
+ The `sha3-*` choices require an rsync built with OpenSSL support and
+ a libcrypto that provides SHA-3 (OpenSSL 1.1.1 or newer). They are
+ never selected by `auto` negotiation and must be requested explicitly.
+
Run `rsync --version` to see the default checksum list compiled into
your
version (which may differ from the list above).
diff --git a/testsuite/checksum-sha3.test b/testsuite/checksum-sha3.test
new file mode 100755
index 0000000..37a4a12
--- /dev/null
+++ b/testsuite/checksum-sha3.test
@@ -0,0 +1,60 @@
+#!/bin/sh
+
+# Test SHA-3 checksum support (sha3-256, sha3-384, sha3-512).
+#
+# Requires rsync to be built with OpenSSL support and linked against a
+# libcrypto that exposes the SHA-3 family (OpenSSL 1.1.1 or newer).
+# The test probes for SHA-3 by asking rsync to use it on a no-op copy;
+# if the local rsync rejects the choice (e.g. no OpenSSL at build time,
+# or older libcrypto), the test is skipped rather than failed.
+
+. "$suitedir/rsync.fns"
+
+probe() {
+ $RSYNC --checksum-choice="$1" --help >/dev/null 2>&1 || return 1
+ # --help doesn't actually parse the choice; force a real parse.
+ mkdir -p "$tmpdir/probe.src"
+ : > "$tmpdir/probe.src/x"
+ rm -rf "$tmpdir/probe.dst"
+ $RSYNC -a --checksum-choice="$1" "$tmpdir/probe.src/"
"$tmpdir/probe.dst/" 2>"$outfile"
+ rc=$?
+ rm -rf "$tmpdir/probe.src" "$tmpdir/probe.dst"
+ if [ $rc -ne 0 ] && grep -q 'unknown checksum name' "$outfile"; then
+ return 1
+ fi
+ return $rc
+}
+
+if ! probe sha3-256; then
+ test_skipped "rsync was not built with OpenSSL SHA-3 support"
+fi
+
+mkdir "$fromdir"
+mkdir "$fromdir/sub"
+# Mix of empty, small, and chunk-spanning sizes (CHUNK_SIZE is 32k
internally).
+: > "$fromdir/empty"
+echo "hello sha-3" > "$fromdir/small"
+dd if=/dev/urandom of="$fromdir/big" bs=1024 count=70 2>/dev/null
+
+for algo in sha3-256 sha3-384 sha3-512; do
+ rm -rf "$todir"
+
+ # Whole-file path (transfer checksum only).
+ checkit "$RSYNC -a --checksum-choice=$algo '$fromdir/' '$todir/'"
"$fromdir" "$todir"
+
+ # Pre-transfer + transfer checksum path (-c forces file_checksum to
run).
+ rm -rf "$todir"
+ checkit "$RSYNC -ac --checksum-choice=$algo '$fromdir/' '$todir/'"
"$fromdir" "$todir"
+
+ # --checksum-choice with distinct xfer,file pair.
+ rm -rf "$todir"
+ checkit "$RSYNC -ac --checksum-choice=$algo,$algo '$fromdir/'
'$todir/'" "$fromdir" "$todir"
+done
+
+# Delta-update path: mutate the destination, re-sync with SHA-3, confirm
match.
+rm -rf "$todir"
+$RSYNC -a --checksum-choice=sha3-256 "$fromdir/" "$todir/" || test_fail
"initial sync failed"
+echo "modified locally" >> "$todir/small"
+checkit "$RSYNC -a --checksum-choice=sha3-256 '$fromdir/' '$todir/'"
"$fromdir" "$todir"
+
+exit 0
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
