On 06/21/2018 10:00 PM, E Cruz wrote:
On 06/21/2018 08:38 PM, Richard Hipp wrote:
Please rebuild your Fossil using the latest trunk check-in, then try
your clone using the new --nocompress option. Report back whether or
not this solves your problem.
Using the new option cuts the cloning time of one of the repositories
from 8 minutes down to 4 minutes. For another repository it went from
2 minutes down to to 1 minute, so basically using "--nocompress" is
cutting the time in half. Most of the remaining time is in the
"rebuilding repository meta-data" phase, but this change is a
significant improvement already. Thanks
On 06/22/2018 09:57 PM, E Cruz wrote:
Delta encoding is still taking a significant amount of time, in
particular the call to content_deltify() by add_one_mlink() in
manifest.c.
Commenting out this call to content_deltify() allows cloning my
smaller repository with "--nocompress" to go from 1 minute down to 4
seconds. I am not familiar enough with fossil to know all the
implications of commenting this call out, but the resulting cloned
repository seems to be fine. If the noCompress flag could be
propagated down so that this particular call to content_deltify() is
skipped when cloning with the flag enabled, the clone operation time
could be reduced to a small fraction of what it is now.
Based on my previous findings, I will like to propose a way to reduce
cloning time of repositories that contain large files with very large
"deltas" between revisions. The change involves saving the
"--nocompress" flag in fossil's global state and using it to skip
calling content_deltify() from add_one_mlink() if the flag is set. I do
not yet understand all the internals of fossil, so I have checked the
proposed changes by cloning fossil's repository with and without
--nocompress, then comparing the outputs of "fossil export --git".
Outputs from both clones were identical. I also checked individual
tables in the two clones and the only differences found were:
1. timestamp of |last-sync-url| entry in |config| table
2. timestamps for the cluster entries in the |tagxref| table
3. |pw| field in the |user| table.
My understanding is these differences are expected to be present between
clones. The way I implemented the proposed changes is shown in the
included patch file. Could you take a look to see if they don't have
any unintended side effect?
Thanks.
Index: src/clone.c
==================================================================
--- src/clone.c
+++ src/clone.c
@@ -128,10 +128,12 @@
const char *zHttpAuth; /* HTTP Authorization user:pass information */
int nErr = 0;
int urlFlags = URL_PROMPT_PW | URL_REMEMBER;
int syncFlags = SYNC_CLONE;
int noCompress = find_option("nocompress",0,0)!=0;
+
+ g.noCompressClone = noCompress;
/* Also clone private branches */
if( find_option("private",0,0)!=0 ) syncFlags |= SYNC_PRIVATE;
if( find_option("once",0,0)!=0) urlFlags &= ~URL_REMEMBER;
if( find_option("verbose","v",0)!=0) syncFlags |= SYNC_VERBOSE;
Index: src/main.c
==================================================================
--- src/main.c
+++ src/main.c
@@ -283,10 +283,11 @@
} reqPayload; /* request payload object (if any) */
cson_array *warnings; /* response warnings */
int timerId; /* fetched from fossil_timer_start() */
} json;
#endif /* FOSSIL_ENABLE_JSON */
+ int noCompressClone; /* True if cloning with --nocompress */
};
/*
** Macro for debugging:
*/
Index: src/manifest.c
==================================================================
--- src/manifest.c
+++ src/manifest.c
@@ -1245,11 +1245,11 @@
db_bind_int(&s1, ":pfn", pfnid);
db_bind_int(&s1, ":mp", mperm);
db_bind_int(&s1, ":isaux", isPrimary==0);
db_exec(&s1);
}
- if( pid && fid ){
+ if( !g.noCompressClone && pid && fid ){
content_deltify(pid, &fid, 1, 0);
}
}
/*
_______________________________________________
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
