On Mon, 27 Sep 2021 13:38:22 +0200
Jindřich Makovička <makov...@gmail.com> wrote:
> On Sun, 26 Sep 2021 13:54:13 +0200
> Jindřich Makovička <makov...@gmail.com> wrote:
> > Hi,
> >
> > ...
> >
> > Applying the attached patch, which reduces the default allocation to
> > 32 pointers, and preallocates 32K pointers only for the main file
> > lists in send_file_list and recv_file_list, reduces the peak memory
> > usage in my case from 142MB to 12MB.
>
> The original patch breaks the testsuite due to extra messages in the
> output.
Actually it makes more sense to change the debug print to check that
the original pointer is non-null and leave the testsuite as is.
--
Jindrich Makovicka
>From bdfdef1c5a4437e2492da148b824d39ba235704e Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Jind=C5=99ich=20Makovi=C4=8Dka?= <makov...@gmail.com>
Date: Sun, 26 Sep 2021 12:01:21 +0200
Subject: [PATCH] Reduce memory usage
Start only with 32 entries for the partial file lists, instead of 32k.
---
flist.c | 4 +++-
rsync.h | 5 +++--
2 files changed, 6 insertions(+), 3 deletions(-)
diff --git a/flist.c b/flist.c
index 3442d868..0f7a64e6 100644
--- a/flist.c
+++ b/flist.c
@@ -305,7 +305,7 @@ static void flist_expand(struct file_list *flist, int extra)
new_ptr = realloc_array(flist->files, struct file_struct *, flist->malloced);
- if (DEBUG_GTE(FLIST, 1) && flist->malloced != FLIST_START) {
+ if (DEBUG_GTE(FLIST, 1) && flist->files) {
rprintf(FCLIENT, "[%s] expand file_list pointer array to %s bytes, did%s move\n",
who_am_i(),
big_num(sizeof flist->files[0] * flist->malloced),
@@ -2186,6 +2186,7 @@ struct file_list *send_file_list(int f, int argc, char *argv[])
#endif
flist = cur_flist = flist_new(0, "send_file_list");
+ flist_expand(flist, FLIST_START_LARGE);
if (inc_recurse) {
dir_flist = flist_new(FLIST_TEMP, "send_file_list");
flags |= FLAG_DIVERT_DIRS;
@@ -2541,6 +2542,7 @@ struct file_list *recv_file_list(int f, int dir_ndx)
#endif
flist = flist_new(0, "recv_file_list");
+ flist_expand(flist, FLIST_START_LARGE);
if (inc_recurse) {
if (flist->ndx_start == 1)
diff --git a/rsync.h b/rsync.h
index 88319732..17f8700e 100644
--- a/rsync.h
+++ b/rsync.h
@@ -918,8 +918,9 @@ extern int xattrs_ndx;
* Start the flist array at FLIST_START entries and grow it
* by doubling until FLIST_LINEAR then grow by FLIST_LINEAR
*/
-#define FLIST_START (32 * 1024)
-#define FLIST_LINEAR (FLIST_START * 512)
+#define FLIST_START (32)
+#define FLIST_START_LARGE (32 * 1024)
+#define FLIST_LINEAR (FLIST_START_LARGE * 512)
/*
* Extent size for allocation pools: A minimum size of 128KB
--
2.33.0
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
