On Sun, 26 Sep 2021 13:54:13 +0200
Jindřich Makovička <makov...@gmail.com> wrote:
> Hi,
>
> ...
>
> Applying the attached patch, which reduces the default allocation to
> 32 pointers, and preallocates 32K pointers only for the main file
> lists in send_file_list and recv_file_list, reduces the peak memory
> usage in my case from 142MB to 12MB.
The original patch breaks the testsuite due to extra messages in the
output.
Fix attached.
Regards,
--
Jindrich Makovicka
>From aa907eabe701550cd5649b6e3da2b22b79b38a06 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Jind=C5=99ich=20Makovi=C4=8Dka?= <makov...@gmail.com>
Date: Sun, 26 Sep 2021 12:01:21 +0200
Subject: [PATCH] Reduce memory usage
Start only with 32 entries for the partial file lists, instead of 32k.
---
flist.c | 2 ++
rsync.h | 5 +++--
testsuite/rsync.fns | 1 +
3 files changed, 6 insertions(+), 2 deletions(-)
diff --git a/flist.c b/flist.c
index 3442d868..37f70b69 100644
--- a/flist.c
+++ b/flist.c
@@ -2186,6 +2186,7 @@ struct file_list *send_file_list(int f, int argc, char *argv[])
#endif
flist = cur_flist = flist_new(0, "send_file_list");
+ flist_expand(flist, FLIST_START_LARGE);
if (inc_recurse) {
dir_flist = flist_new(FLIST_TEMP, "send_file_list");
flags |= FLAG_DIVERT_DIRS;
@@ -2541,6 +2542,7 @@ struct file_list *recv_file_list(int f, int dir_ndx)
#endif
flist = flist_new(0, "recv_file_list");
+ flist_expand(flist, FLIST_START_LARGE);
if (inc_recurse) {
if (flist->ndx_start == 1)
diff --git a/rsync.h b/rsync.h
index 88319732..17f8700e 100644
--- a/rsync.h
+++ b/rsync.h
@@ -918,8 +918,9 @@ extern int xattrs_ndx;
* Start the flist array at FLIST_START entries and grow it
* by doubling until FLIST_LINEAR then grow by FLIST_LINEAR
*/
-#define FLIST_START (32 * 1024)
-#define FLIST_LINEAR (FLIST_START * 512)
+#define FLIST_START (32)
+#define FLIST_START_LARGE (32 * 1024)
+#define FLIST_LINEAR (FLIST_START_LARGE * 512)
/*
* Extent size for allocation pools: A minimum size of 128KB
diff --git a/testsuite/rsync.fns b/testsuite/rsync.fns
index 1e2b399f..220b9e21 100644
--- a/testsuite/rsync.fns
+++ b/testsuite/rsync.fns
@@ -89,6 +89,7 @@ v_filt() {
-e '/^total: /d' \
-e '/^client charset: /d' \
-e '/^server charset: /d' \
+ -e '/ expand file_list pointer array /d' \
-e '/^$/,$d'
}
--
2.33.0
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
