Hi,
When using rsync to back up the file system on my laptop, containing a
pretty much default linux desktop, I was wondering how rsync uses over
100MB of RAM it allocates.
It turned out that most of the memory is used for the arrays of file_struct
pointers, most of which end up unused - much more than the actual
file_struct entries. In my case, the peak usage was 135MB of pointers, and
just 1.5MB of the file_struct entries themselves.
The problem seems to be that the default file_list allocation parameters
predate the incremental recursion, which allocates a huge number of small
file lists, while AFAICS originally rsync allocated just one large list.
Applying the attached patch, which reduces the default allocation to 32
pointers, and preallocates 32K pointers only for the main file lists in
send_file_list and recv_file_list, reduces the peak memory usage in my case
from 142MB to 12MB.
Regards,
--
Jindřich Makovička
From ef169c9157d312c63bad00e3bfc1d8eb70d56ccd Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Jind=C5=99ich=20Makovi=C4=8Dka?= <makov...@gmail.com>
Date: Sun, 26 Sep 2021 12:01:21 +0200
Subject: [PATCH] Reduce memory usage
Start only with 32 entries for the partial file lists, instead of 32k.
---
flist.c | 2 ++
rsync.h | 5 +++--
2 files changed, 5 insertions(+), 2 deletions(-)
diff --git a/flist.c b/flist.c
index 3442d868..37f70b69 100644
--- a/flist.c
+++ b/flist.c
@@ -2186,6 +2186,7 @@ struct file_list *send_file_list(int f, int argc, char *argv[])
#endif
flist = cur_flist = flist_new(0, "send_file_list");
+ flist_expand(flist, FLIST_START_LARGE);
if (inc_recurse) {
dir_flist = flist_new(FLIST_TEMP, "send_file_list");
flags |= FLAG_DIVERT_DIRS;
@@ -2541,6 +2542,7 @@ struct file_list *recv_file_list(int f, int dir_ndx)
#endif
flist = flist_new(0, "recv_file_list");
+ flist_expand(flist, FLIST_START_LARGE);
if (inc_recurse) {
if (flist->ndx_start == 1)
diff --git a/rsync.h b/rsync.h
index 2f674bc5..708fd244 100644
--- a/rsync.h
+++ b/rsync.h
@@ -917,8 +917,9 @@ extern int xattrs_ndx;
* Start the flist array at FLIST_START entries and grow it
* by doubling until FLIST_LINEAR then grow by FLIST_LINEAR
*/
-#define FLIST_START (32 * 1024)
-#define FLIST_LINEAR (FLIST_START * 512)
+#define FLIST_START (32)
+#define FLIST_START_LARGE (32 * 1024)
+#define FLIST_LINEAR (FLIST_START_LARGE * 512)
/*
* Extent size for allocation pools: A minimum size of 128KB
--
2.33.0
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
