Re: [fossil-users] Cloning repository with large files very slow
On 06/21/2018 08:38 PM, Richard Hipp wrote: Please rebuild your Fossil using the latest trunk check-in, then try your clone using the new --nocompress option. Report back whether or not this solves your problem. Using the new option cuts the cloning time of one of the repositories from 8 minutes down to 4 minutes. For another repository it went from 2 minutes down to to 1 minute, so basically using "--nocompress" is cutting the time in half. Most of the remaining time is in the "rebuilding repository meta-data" phase, but this change is a significant improvement already. Thanks! ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] Cloning repository with large files very slow
On 6/21/18, E Cruz wrote: > Is there a way to prevent fossil > from re-applying delta encoding when cloning? Please rebuild your Fossil using the latest trunk check-in, then try your clone using the new --nocompress option. Report back whether or not this solves your problem. -- D. Richard Hipp d...@sqlite.org ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] Cloning repository with large files very slow
On Thu 21 Jun 2018 7:47 PM, E Cruz wrote: > On 06/21/2018 05:06 PM, Warren Young wrote: > > As mentioned in the original post, the majority of the time taken by the > clone operation seems to be spent re-calculating the delta encoding of the > large table definition files. I do not mind much the time it takes to > commit changes to the large tables, although if that can be improved it > would be welcomed. But once that is done, we have to pay for the delta > encoding on every future clone operation. That is the part I would like to > avoid if possible. > Just curious, how often are you cloning and opening the repo? With git, I've had it take a little while to expand the repo. For a comparison, clone the sqlite repo and see if it's quicker/slower/about the same as your repo clone. I believe the sqlite repo is larger than yours. ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] Cloning repository with large files very slow
On 06/21/2018 05:06 PM, Warren Young wrote: Are the differences merely at the binary level or is the semantic content also changing? Thanks for your reply. The files are not binary. They are C source files that define large arrays of floating point values. These arrays are rarely updated, but when they are, most of the values in the array change. Try unsetting repo-cksum on this repository, if it’s enabled: https://fossil-scm.org/index.html/help?cmd=repo-cksum The total size of the fossil repository file is about 65MB, not a huge repository. Following your suggestion, I checked the repo-cksum setting and it was not set. I still tried cloning after running "fossil unset repo-cksum", "fossil setting repo-cksum 0", and "fossil setting repo-cksum 1" on the source repository and all cases took about the same time to complete the clone. That seems to be inline with the documentation you pointed to, where it it mentions that the setting applies to checkouts, and there is no mention of cloning. As mentioned in the original post, the majority of the time taken by the clone operation seems to be spent re-calculating the delta encoding of the large table definition files. I do not mind much the time it takes to commit changes to the large tables, although if that can be improved it would be welcomed. But once that is done, we have to pay for the delta encoding on every future clone operation. That is the part I would like to avoid if possible. ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] Cloning repository with large files very slow
On Jun 21, 2018, at 1:25 PM, E Cruz wrote: > > some of the source files define a few very large tables. These tables do not > change often, but when they do most of their content is replaced with > something completely different from the previous version. Are the differences merely at the binary level or is the semantic content also changing? For instance, these two commands give entirely different output, but with identical semantic content: $ echo "Hello, world!" | bzip2 | od -t x1 $ echo "Hello, world!" | lz4 | od -c The point is, there may be an encoding for your data that reduces the size of the diffs to include only the semantic differences. As an example, storing two different versions of a PNG in Fossil is probably less efficient than storing the same logical image data in Windows BMP form plus a conversion command to PNG during the build process, since the BMP is uncompressed and has very little metadata, so that only the actual pixel differences are stored in Fossil. (If you doubt this, I actually tested it and reported on the differences some years ago here.) Many binary data formats have this same property. They’re optimized for the size of individual files on disk, not for the total size of a delta-and-gzip-compressed version control repository. Yet another example is all of the binary data formats based on ZIP: ODF, JAR, APK… They’d store more efficiently in Fossil if unpacked into a tree before being checked in between changes. > When changes to these files are committed, fossil takes a long time to > process the commit (a couple of minutes for a 20MB table, over 10min for a > 60MB table). That’s superlinear, which is bad. Try unsetting repo-cksum on this repository, if it’s enabled: https://fossil-scm.org/index.html/help?cmd=repo-cksum With that unset, you become responsible for local file integrity. Some of the more modern filesystems obviate the need for manual integrity checks or secondary checks like the one Fossil does: ZFS, APFS, etc. ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
[fossil-users] Cloning repository with large files very slow
I am testing fossil with a repository where some of the source files define a few very large tables. These tables do not change often, but when they do most of their content is replaced with something completely different from the previous version. When changes to these files are committed, fossil takes a long time to process the commit (a couple of minutes for a 20MB table, over 10min for a 60MB table). It would be nice if fossil could handle these commits much faster, but that in itself is not a big problem for us because the tables rarely get changed. What is more problematic is that once a change to the tables has been committed, cloning the repository also takes a very long time. It seems the fossil clone command is attempting to re-apply delta encoding to all files in the repository and that is causing the slowdown. Is re-encoding necessary when performing a clone operation? If it is not necessary, Is there a way to prevent fossil from re-applying delta encoding when cloning? -- Edgardo M. Cruz | edgardo.c...@genkey.com ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] No rule to make target 'src/email.c', ...
On 6/21/18, Warren Young wrote: > > You might want to change the “sendmail -t” default for Windows as well. I need to get it working first. After I get something working, then we can go back and worry about fine-tuning so that the feature to be convenient for windows. Right now, nothing in the email notification logic works. At all. On any platform. So the fact that the defaults are suboptimal is not a distraction that I want to burn time on. -- D. Richard Hipp d...@sqlite.org ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] No rule to make target 'src/email.c', ...
Nice quick fix. Compiles now. Thanks. On Thu, Jun 21, 2018, 10:03 AM Richard Hipp wrote: > On 6/21/18, Eric Dillon wrote: > > Fails to compile on Win32 VS2013 (12.0) > > > > email.obj : error LNK2019: unresolved external symbol _popen referenced > in > > function _email_send > > Thanks for the report. Fixed now. > > -- > D. Richard Hipp > d...@sqlite.org > ___ > fossil-users mailing list > fossil-users@lists.fossil-scm.org > http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users > ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] No rule to make target 'src/email.c', ...
On Jun 21, 2018, at 9:03 AM, Richard Hipp wrote: > > On 6/21/18, Eric Dillon wrote: >> Fails to compile on Win32 VS2013 (12.0) >> >> email.obj : error LNK2019: unresolved external symbol _popen referenced in >> function _email_send > > Thanks for the report. Fixed now. You might want to change the “sendmail -t” default for Windows as well. Requiring that someone use Cygwin, WSL, or a native Windows port of Sendmail/Postfix just to get sendmail(1) on Windows seems like a big “ask”. The more Windows-native methods seem to be one of these options: https://stackoverflow.com/q/15433202/142454 https://superuser.com/q/63081/14927 I think the PowerShell option has the least burdensome requirements, needing only a simple script, which we can provide in the Fossil docs. Here’s a link to a forum thread describing how to read stdin from PowerShell, leaving only the need to parse it to parameters that can be fed to the SmtpClient.Send() API: https://social.technet.microsoft.com/forums/scriptcenter/98a05368-6a0c-4cf3-bf7e-4bb8ae748fe6 Are there any PowerShell scripters here who can stitch this information together into a working script, one that takes RFC 822 via stdin, parses the headers and body into variables, and calls the API correctly? I could probably puzzle it out, but I’d end up writing Perl in PowerShell. A native speaker would do a better job. ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] No rule to make target 'src/email.c', ...
On 6/21/18, Eric Dillon wrote: > Fails to compile on Win32 VS2013 (12.0) > > email.obj : error LNK2019: unresolved external symbol _popen referenced in > function _email_send Thanks for the report. Fixed now. -- D. Richard Hipp d...@sqlite.org ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] No rule to make target 'src/email.c', ...
Fails to compile on Win32 VS2013 (12.0) email.obj : error LNK2019: unresolved external symbol _popen referenced in function _email_send On Wed, Jun 20, 2018, 3:51 PM Florian Balmer wrote: > Richard Hipp: > > > It is reassuring to know that so many people routinely build Fossil > > from the trunk sources :-) > > À propos build: > > I have accidentally set FOSSIL_ENABLE_MINIZ=1 (in win/Makefile.msc), > and this doesn't seem to work, probably because src/shell.c includes > zlib.h without examining FOSSIL_ENABLE_MINIZ. > > Not sure if this is important, or if it's even possible for > src/shell.c to support FOSSIL_ENABLE_MINIZ, but I'd rather let you > know. > > --Florian > ___ > fossil-users mailing list > fossil-users@lists.fossil-scm.org > http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users > ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
[fossil-users] [PATCH] Remove unused option from settings page
It appears the "show-version-diffs" setting was abandoned in commit 0a1f4ed6aa.
Index: src/setup.c
==
--- src/setup.c
+++ src/setup.c
@@ -1463,19 +1463,10 @@
@ in a separate box (using CSS class "timelineDate") whenever the date
changes.
@ With the "-MM-DDHH:MM" and "YYMMDD ..." formats, the complete
date
@ and time is shown on every timeline entry using the CSS class
"timelineTime".
@ (Preperty: "timeline-date-format")
- @
- onoff_attribute("Show version differences by default",
- "show-version-diffs", "vdiff", 0, 0);
- @ The version-information pages linked from the timeline can either
- @ show complete diffs of all file changes, or can just list the names of
- @ the files that have changed. Users can get to either page by
- @ clicking. This setting selects the default.
- @ (Property: "show-version-diffs")
-
@
entry_attribute("Max timeline comment length", 6,
"timeline-max-comment", "tmc", "0", 0);
@ The maximum length of a comment to be displayed in a timeline.
@ "0" there is no length limit.
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
