[PATCH] Add support for zstd compression
From: Sebastian Andrzej Siewior
zstd compression was announced as "good compression with high
throughput" so I gave it a try. With zlib, on high speed links the CPU
is usually the bottle neck. With zstd I'm able to fill a 200Mbit link :)
zstd detection happens automatically via pkg-config. No zstd header means
no error about missing zstd. So that should be okay. However, pkg-config
is now kind of required…
I duplicated the zlib code and replaced it with zstd hooks once I
understood what was going on. I made a few local tests with and without
tokens and it seems to work. The compression can be selected with `-Z'
option. By default `0' is used as the compression level which is a
special default (it currently maps to 3). The compression level can be
specified by the same option as for zlib.
The compressor feeds data into zstd and starts sending data once the
outgoing buffer is full or when a flush is requested. That flush will
close the current compression block and create a new one for the
following transfer (saving the internal compression / history state).
The decompressor allocates space for two blocks. Should one block
contain more data, then it will loop more often.
Signed-off-by: Sebastian Andrzej Siewior
---
Makefile.in | 4 +-
batch.c | 1 +
configure.ac | 18
options.c| 54 ++-
rsync.yo | 10 ++-
token.c | 248 ++-
6 files changed, 329 insertions(+), 6 deletions(-)
diff --git a/Makefile.in b/Makefile.in
index f912f312ce89a..9bb977eb6b0a8 100644
--- a/Makefile.in
+++ b/Makefile.in
@@ -10,7 +10,7 @@ mandir=@mandir@
LIBS=@LIBS@
CC=@CC@
-CFLAGS=@CFLAGS@
+CFLAGS=@CFLAGS@ @LIBZSTD_CFLAGS@
CPPFLAGS=@CPPFLAGS@
EXEEXT=@EXEEXT@
LDFLAGS=@LDFLAGS@
@@ -91,7 +91,7 @@ install-all: install install-ssl-client install-ssl-daemon
$(MAKE) INSTALL_STRIP='-s' install
rsync$(EXEEXT): $(OBJS)
- $(CC) $(CFLAGS) $(LDFLAGS) -o $@ $(OBJS) $(LIBS)
+ $(CC) $(CFLAGS) $(LDFLAGS) -o $@ $(OBJS) $(LIBS) @LIBZSTD_LIBS@
$(OBJS): $(HEADERS)
$(CHECK_OBJS): $(HEADERS)
diff --git a/batch.c b/batch.c
index 263a9a357d21b..764fd3ebc5acb 100644
--- a/batch.c
+++ b/batch.c
@@ -80,6 +80,7 @@ static char *flag_name[] = {
"--checksum (-c)",
"--dirs (-d)",
"--compress (-z)",
+ "--zstd-compress (-Z)",
"--iconv",
"--acls (-A)",
"--xattrs (-X)",
diff --git a/configure.ac b/configure.ac
index 4f68e98a9e1d1..12db93570d78a 100644
--- a/configure.ac
+++ b/configure.ac
@@ -1038,6 +1038,24 @@ else
esac
fi
+AC_ARG_WITH([libzstd], AS_HELP_STRING([--without-libzstd], [Build without
libzstd (Default: with if possible)]))
+
+AS_IF([test "x$with_libzstd" != "xno"],
+ [want_libzstd=yes],
+ [want_libzstd=no])
+
+AS_IF([test "x$want_libzstd" = "xyes"],
+ [
+ PKG_CHECK_MODULES([LIBZSTD], [libzstd >= 1.3.8],
[AC_DEFINE([HAVE_LIBZSTD], [1], [Use LIBZSTD])])
+ ],
+ [AS_IF([test "x$with_libzstd" = "xyes"],
+ [AC_MSG_ERROR([libzstd requested but not found])
+ ])
+ ])
+
+AC_SUBST([LIBZSTD_CFLAGS])
+AC_SUBST([LIBZSTD_LIBS])
+
#
# check for extended attribute support
AC_MSG_CHECKING(whether to support extended attributes)
diff --git a/options.c b/options.c
index e5b0cb68280ed..07e3f1e5d0ac1 100644
--- a/options.c
+++ b/options.c
@@ -23,6 +23,9 @@
#include "itypes.h"
#include
#include
+#ifdef HAVE_LIBZSTD
+#include
+#endif
extern int module_id;
extern int local_server;
@@ -77,6 +80,7 @@ int protocol_version = PROTOCOL_VERSION;
int sparse_files = 0;
int preallocate_files = 0;
int do_compression = 0;
+int do_compression_zstd = 0;
int def_compress_level = NOT_SPECIFIED;
int am_root = 0; /* 0 = normal, 1 = root, 2 = --super, -1 = --fake-super */
int am_server = 0;
@@ -764,6 +768,9 @@ void usage(enum logcode F)
rprintf(F," --copy-dest=DIR ... and include copies of unchanged
files\n");
rprintf(F," --link-dest=DIR hardlink to files in DIR when
unchanged\n");
rprintf(F," -z, --compress compress file data during the
transfer\n");
+#ifdef HAVE_LIBZSTD
+ rprintf(F," -Z, --zstd-compress compress file data during the
transfer with zstd\n");
+#endif
rprintf(F," --compress-level=NUMexplicitly set compression level\n");
rprintf(F," --skip-compress=LISTskip compressing files with a suffix
in LIST\n");
rprintf(F," -C, --cvs-exclude auto-ignore files the same way CVS
does\n");
@@ -968,6 +975,9 @@ static struct poptOption long_options[] = {
{"no-fuzzy", 0, POPT_ARG_VAL,_basis, 0, 0, 0 },
{"no-y", 0, POPT_ARG_VAL,_basis, 0, 0, 0 },
{"compress",'z', POPT_ARG_NONE, 0, 'z', 0, 0 },
+#ifdef HAVE_LIBZSTD
+ {"zstd-compress", 'Z', POPT_ARG_NONE, 0, 'Z', 0, 0 },
+#endif
{"old-compress", 0, POPT_ARG_VAL,
Re: [RFC PATCH] Add SHA1 support
On 2020-02-20 20:06:39 [+0100], Markus Ueberall wrote: > On 2020-02-09 23:19, Sebastian Andrzej Siewior wrote: > > [...] > > My primar motivation to use SHA1 for checksumming (by default) instead > > of MD5 is not the additional security bits but performance. On a decent > > x86 box the SHA1 performance is almost the same as MD5's but with > > acceleration it outperforms MD5. > > > > The other alternative would be to go for xxHash64 [0] which has the > > superior performance but provides a non-cryptographic hash so I though > > SHA1 would be better here. > > [...] > > With respect to *both* speed and security, wouldn't BLAKE3 be a better, > modern alternative if we're looking at checksumming? > It's "[r]eleased into the public domain with CC0 1.0. Alternatively, it is > licensed under the Apache License 2.0". And the performance (see the chart > at https://github.com/BLAKE3-team/BLAKE3) is *impressive* ... I'm still not sure if rsync requires a cryptographic hash _or_ if a strong hash like xxHash64 would be just fine for the job. > Kind regards, Markus Sebastian -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[RFC PATCH] Add SHA1 support
From: Sebastian Andrzej Siewior
This is a huge all-in-one patch and deserves a little cleanup and
splitting. However, I wanted to get it out here for some feedback.
My primar motivation to use SHA1 for checksumming (by default) instead
of MD5 is not the additional security bits but performance. On a decent
x86 box the SHA1 performance is almost the same as MD5's but with
acceleration it outperforms MD5.
The other alternative would be to go for xxHash64 [0] which has the
superior performance but provides a non-cryptographic hash so I though
SHA1 would be better here.
For linking against OpenSSL as of today the rsync license would need an
"OpenSSL exception" [1]. The master branch of OpenSSL is licensed under
the Apache License 2.0 so we could wait until 3.0 is released and use
the C version of the algorithm in the meantime.
Here are numbers from a ryzen test box:
small file:
|$ dd if=/dev/zero of=/dev/shm/out bs=1073741824 count=1
|1+0 records in
|1+0 records out
|1073741824 bytes (1,1 GB, 1,0 GiB) copied, 0,503252 s, 2,1 GB/s
Old hash:
|$ time ./rsync -c /dev/shm/out --checksum-choice=md4
|-rw-r--r-- 1,073,741,824 2020/02/08 16:34:42 out
|
|real0m1,064s
|user0m0,984s
|sys 0m0,080s
MD5 from openssl (should match built-in speed):
|$ time ./rsync -c /dev/shm/out --checksum-choice=md5
|-rw-r--r-- 1,073,741,824 2020/02/08 16:34:42 out
|
|real0m1,433s
|user0m1,293s
|sys 0m0,140s
SHA1 from openssl:
|$ time ./rsync -c /dev/shm/out --checksum-choice=sha1
|-rw-r--r-- 1,073,741,824 2020/02/08 16:34:42 out
|
|real0m0,619s
|user0m0,524s
|sys 0m0,096s
SHA1 from the built-in code:
|time ./rsync -c /dev/shm/out --checksum-choice=sha1
|-rw-r--r-- 1,073,741,824 2020/02/08 16:34:42 out
|
|real0m1,561s
|user0m1,465s
|sys 0m0,096s
[1]
https://opensource.stackexchange.com/questions/2233/gpl-v3-with-openssl-exception
[0] https://github.com/Cyan4973/xxHash
Signed-off-by: Sebastian Andrzej Siewior
---
Makefile.in | 4 +-
checksum.c| 144 +
configure.ac | 5 +
lib/md32_common.h | 258 +
lib/md5.c | 15 +-
lib/mdigest.h | 77 -
lib/sha1.c| 19 +++
lib/sha1.h| 20 +++
lib/sha_local.h | 401 ++
main.c| 2 +
rsync.h | 2 +-
11 files changed, 902 insertions(+), 45 deletions(-)
create mode 100644 lib/md32_common.h
create mode 100644 lib/sha1.c
create mode 100644 lib/sha1.h
create mode 100644 lib/sha_local.h
diff --git a/Makefile.in b/Makefile.in
index 9bb977eb6b0a8..a390afe4ed829 100644
--- a/Makefile.in
+++ b/Makefile.in
@@ -32,7 +32,7 @@ VERSION=@RSYNC_VERSION@
GENFILES=configure.sh aclocal.m4 config.h.in proto.h proto.h-tstamp rsync.1
rsyncd.conf.5
HEADERS=byteorder.h config.h errcode.h proto.h rsync.h ifuncs.h itypes.h
inums.h \
lib/pool_alloc.h
-LIBOBJ=lib/wildmatch.o lib/compat.o lib/snprintf.o lib/mdfour.o lib/md5.o \
+LIBOBJ=lib/wildmatch.o lib/compat.o lib/snprintf.o lib/mdfour.o lib/md5.o
lib/sha1.o \
lib/permstring.o lib/pool_alloc.o lib/sysacls.o lib/sysxattrs.o
@LIBOBJS@
zlib_OBJS=zlib/deflate.o zlib/inffast.o zlib/inflate.o zlib/inftrees.o \
zlib/trees.o zlib/zutil.o zlib/adler32.o zlib/compress.o zlib/crc32.o
diff --git a/checksum.c b/checksum.c
index 3295252ba0120..77c36b59c93ec 100644
--- a/checksum.c
+++ b/checksum.c
@@ -32,6 +32,7 @@ extern char *checksum_choice;
#define CSUM_MD4_OLD 3
#define CSUM_MD4 4
#define CSUM_MD5 5
+#define CSUM_SHA1 6
int xfersum_type = 0; /* used for the file transfer checksums */
int checksum_type = 0; /* used for the pre-transfer (--checksum) checksums */
@@ -54,6 +55,8 @@ int parse_csum_name(const char *name, int len)
len = strlen(name);
if (!name || (len == 4 && strncasecmp(name, "auto", 4) == 0)) {
+ if (protocol_version >= 31)
+ return CSUM_SHA1;
if (protocol_version >= 30)
return CSUM_MD5;
if (protocol_version >= 27)
@@ -68,6 +71,8 @@ int parse_csum_name(const char *name, int len)
return CSUM_MD5;
if (len == 4 && strncasecmp(name, "none", 4) == 0)
return CSUM_NONE;
+ if (len == 4 && strncasecmp(name, "sha1", 4) == 0)
+ return CSUM_SHA1;
rprintf(FERROR, "unknown checksum name: %s\n", name);
exit_cleanup(RERR_UNSUPPORTED);
@@ -88,6 +93,8 @@ int csum_len_for_type(int cst, BOOL flist_csum)
return MD4_DIGEST_LEN;
case CSUM_MD5:
return MD5_DIGEST_LEN;
+ case CSUM_SHA1:
+ return SHA1_DIGEST_LEN;
default: /* paranoia to prevent missing case values */
exit_cleanup(RERR_UNSUPPORTED);
}
@@ -121,30 +128,48 @@ uint32 get_checksum1(char *buf1, int32 len)
return (s1 & 0x) + (s2 << 16);
}
+static void
Re: [PATCH] SSE2/SSSE3 optimized version of get_checksum1() for x86-64
On 2020-05-18 17:55:58 [+0200], Jorrit Jongma via rsync wrote: > I don't disagree that MD5 could (or even should) be replaced so it is > no longer the bottleneck in several real-world cases (including mine). > > However this patch is not for MD5 performance, rather for the rolling > checksum rsync uses to match blocks on existing files on both ends to > reduce transfer size. Still. You claim in your patch that | Benchmarks C SSE2SSSE3 | - Intel i7-7700hq1850 MB/s 2550 MB/s 4050 MB/s while xxhash [0] claims on a Core i5-3340M @2.7GHz that: |VersionSpeed on 64-bit Speed on 32-bit |XXH64 13.8 GB/s 1.9 GB/s so using xxhash64 for that work would also boost !x86 platforms. However your patch has the benefit that no changes are required on the remote side. I like that. [0] https://github.com/Cyan4973/xxHash#benchmarks Sebastian -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: [PATCH] SSE2/SSSE3 optimized version of get_checksum1() for x86-64
On 2020-05-18 17:06:51 [+0200], Jorrit Jongma via rsync wrote: > diff --git a/checksum.c b/checksum.c > index cd234038..4e696f3d 100644 > --- a/checksum.c > +++ b/checksum.c > @@ -99,6 +99,7 @@ int canonical_checksum(int csum_type) > return csum_type >= CSUM_MD4 ? 1 : 0; > } > > +#ifndef __SSE2__ // see checksum_sse2.c for SSE2/SSSE3 version > /* >a simple 32 bit checksum that can be updated from either end >(inspired by Mark Adler's Adler-32 checksum) > @@ -119,6 +120,7 @@ uint32 get_checksum1(char *buf1, int32 len) > } > return (s1 & 0x) + (s2 << 16); > } > +#endif You can't replace the code like that with SSE2+. You need runtime detection for this. Otherwise it can't be enabled by distros becuase it would fail on CPUs without SSE2+. Only SSE is part of generic x86-64. Sebastian -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: [PATCH] SSE2/SSSE3 optimized version of get_checksum1() for x86-64
On 2020-05-18 21:55:13 [+0200], Jorrit Jongma wrote: > What do you base this on? So my memory was wrong. SSE2 is supported by all x86-64bit CPUs. Sorry for that. > would imply that SSSE3 is enabled out of the box on builds on machines > that support it, this is not the case (it certainly isn't on my Ubuntu > box). It would be preferred to detect this at runtime but getting that > to work on GCC is (apparently) a mess, and would probably require > modifications to configure/Makefile/etc that I'm not comfortable > doing, as my lack of expertise on those would probably lead me to > break the build for somebody else. If someone knowledgable enough in > that area wants to fix it, though... My suggestion would be to have a get_checksum1_sse2() and get_checksum1_sse3() and always build them. The compiler should support it. Then on runtime you would check for sse3 and based on the result get_checksum1() would either invoke the _sse2() or sse3(). Without auto detection it won't be utilized by distros. But yes, this could be improved afterwards. Sebastian -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: [RFC PATCH] Add SHA1 support
On 2020-03-17 00:03:03 [+0100], Dimitrios Apostolou via rsync wrote: > On Thursday, February 20, 2020 10:34:53 PM CET, Sebastian Andrzej Siewior > via rsync wrote: > > > > I'm still not sure if rsync requires a cryptographic hash _or_ if a > > strong hash like xxHash64 would be just fine for the job. > > I'm fairly sure the hash should *not* be easy to spoof, so I'd say a > cryptographic hash is needed. > > As an example, if a file is replaced by a file of the same size and same > hash, > rsync (if -c is in use) will consider the file is the same, and avoid > copying it. correct. The same goes for currently used md5 which has known collision attacks. So if you intend to spoo it, you can manufacture the same hash for two different files for both algorithms. The question is how likely it is that this happens by chance. According to [0] xxhash64 scores a solid 10. It is better than crc32 which has been used a lot as a checksum for files. [0] https://github.com/Cyan4973/xxHash > Dimitris Sebastian -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: [PATCH] Optimized assembler version of md5_process() for x86-64
On 2020-05-22 22:54:18 [-0700], Wayne Davison via rsync wrote: > Thanks for the optimizing patches, Jorrit! I've merged your latest changes > into the git master branch. Wouldn't it be better to add support for a crypto library (like openssl) which would provide optimized algorithms for more than just one platform without the need to maintain it separately? > ..wayne.. Sebastian -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: [PATCH] Optimized assembler version of md5_process() for x86-64
On 2020-05-23 10:21:31 [-0700], Wayne Davison wrote: > > Adding optional support for openssl's crypto library is also a good idea. I posted [0] openssl support with SHA1 support and asked whether openssl is possible. At that time added md5 and I think md4. I received no feedback bach then but if you want me to respin the openssl part (without sha1 now that we have xxhash) I can certainly do that. > I've also added a comment to the checksum.c file about my support of being > able to distribute a dynamically linked version of rsync that links with > the openssl and xxhash libraries (since they are both BSD 2-clause > licensed), especially since this should (IMO) be considered to be covered > under the System Libraries part of the GPL. also, openssl 3.0 will be apache-2 licensed. [0] https://lists.samba.org/archive/rsync/2020-February/032062.html > ..wayne.. Sebastian -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
