On Jun 21, 2018, at 1:25 PM, E Cruz <cruz.fossil...@netsiteinnovations.com>
wrote:
>
> some of the source files define a few very large tables. These tables do not
> change often, but when they do most of their content is replaced with
> something completely different from the previous version.
Are the differences merely at the binary level or is the semantic content also
changing?
For instance, these two commands give entirely different output, but with
identical semantic content:
$ echo "Hello, world!" | bzip2 | od -t x1
$ echo "Hello, world!" | lz4 | od -c
The point is, there may be an encoding for your data that reduces the size of
the diffs to include only the semantic differences.
As an example, storing two different versions of a PNG in Fossil is probably
less efficient than storing the same logical image data in Windows BMP form
plus a conversion command to PNG during the build process, since the BMP is
uncompressed and has very little metadata, so that only the actual pixel
differences are stored in Fossil.
(If you doubt this, I actually tested it and reported on the differences some
years ago here.)
Many binary data formats have this same property. They’re optimized for the
size of individual files on disk, not for the total size of a
delta-and-gzip-compressed version control repository.
Yet another example is all of the binary data formats based on ZIP: ODF, JAR,
APK… They’d store more efficiently in Fossil if unpacked into a tree before
being checked in between changes.
> When changes to these files are committed, fossil takes a long time to
> process the commit (a couple of minutes for a 20MB table, over 10min for a
> 60MB table).
That’s superlinear, which is bad.
Try unsetting repo-cksum on this repository, if it’s enabled:
https://fossil-scm.org/index.html/help?cmd=repo-cksum
With that unset, you become responsible for local file integrity. Some of the
more modern filesystems obviate the need for manual integrity checks or
secondary checks like the one Fossil does: ZFS, APFS, etc.
_______________________________________________
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
