Re: [fossil-users] Error during commit
> My guess is that you are probably overflowing a 32-bit integer
> someplace. If so, I fear that this will not be something easily
> fixed.
>
Looks like it's the case at least in one place.
There's int64 => unsigned int overflow at the call to blob_resize()
(blob.c:865
[http://fossil-scm.org/index.html/artifact?udc=1=on=b3ef46413b7313ca]
).
Generally this is tied to the way the Blob type is defined, so the fix
indeed may be painful and involve thorough testing.
This only becomes an issue with LARGE file sizes that overflow the
(unsigned int), which as DRH mentioned is somewhat beyond the expected
Fossil use-cases.
From the code I don't see any easy workaround other than addressing
the sizes of the files stored at the client's end.
May be trying to build a 64-bit version of the Fossil and explicitly
force the int to 64-bit size
BUT this may blow in some other place, so I would not trust it even if
it worked.
I also wonder at effectiveness of tracking the binaries, as these
won't diff properly, thus each new version would likely be stored
entirely, which would kinda turns repo into a gigantic zip archive
with a lot of overhead. I'd rather keep track of a __reference__ to
such binaries, while the binaries could be stored at some other bulk
storage location. Thus a project version would fetch an updated
reference and the pull the resp. binary.
---
src/vfile.c:820
[http://fossil-scm.org/index.html/artifact?udc=1=on=8c891c745e280c02]
rc = blob_read_from_file(, zFullpath, RepoFILE);
---
src/blob.c:843
[http://fossil-scm.org/index.html/artifact?udc=1=on=b3ef46413b7313ca]
sqlite3_int64 blob_read_from_file(
Blob *pBlob, /* The blob to be initialized */
const char *zFilename, /* Extract content from this file */
int eFType /* ExtFILE or RepoFILE - see above */
){
sqlite3_int64 size, got;
...
size = file_size(zFilename, eFType);
...
blob_resize(pBlob, size);
---
src/blob.c:442
[http://fossil-scm.org/index.html/artifact?udc=1=on=b3ef46413b7313ca]
void blob_resize(Blob *pBlob, unsigned int newSize){
pBlob->xRealloc(pBlob, newSize+1);
pBlob->nUsed = newSize;
pBlob->aData[newSize] = 0;
}
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] Error during commit
On Aug 6, 2018, at 11:30 AM, Philip Bennefall wrote: > > a second commit right afterwards caused a segfault again In advance of drh getting time to work on this, maybe you could give two debugging steps on your end: 1. If you’re doing this on a platform that will run Valgrind, try running a debug build of Fossil under it against your repository: $ cd /path/to/fossil/source/tree $ ./configure --fossil-debug $ make clean && make && sudo make install $ cd /path/to/checkout/directory $ valgrind fossil status 2. If you get a vgcore out of that, load it into GDB and produce a backtrace. If not, cores are probably disabled there, so run Fossil under gdb and get a backtrace that way. (The first way is more likely to give a clean backtrace, as the failure gets caught before it can cause damage.) ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] Error during commit
On 8/6/18, Philip Bennefall wrote: > But I wanted to report our experience in case it is of use to the > developers, and in case doing so could give us some assistance with the > immediate issue. Thank you for the report. This is definitely something that should be fixed. But I have a large queue of other issues of relevance to far more users (ex: support for email alerts and forum) that need to be fixed first. -- D. Richard Hipp d...@sqlite.org ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] Error during commit
The following command solved the issue at least for the moment: fossil rebuild --vacuum --analyze --compress I'm not yet sure which of the options made the difference, but I wanted to report back nevertheless. Kind regards, Philip Bennefall On 8/6/2018 6:57 PM, Philip Bennefall wrote: OK. I'll investigate. Thanks for the quick response. Kind regards, Philip Bennefall On 8/6/2018 6:50 PM, Richard Hipp wrote: On 8/6/18, Philip Bennefall wrote: Do you have any recommendations for something we could try in order to get more information, or would you suggest that we switch to another DVCS if we need to store files of these sizes? Regardless of the problem, I don't think *any* DVCS is appropriate for storing gigabyte sized binary files. You should look into a centralized VCS such as subversion, methinks. ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] Error during commit
On 8/6/18, Richard Hipp wrote: > On 8/6/18, Philip Bennefall wrote: >> Do you have any recommendations for something we could try in order to >> get more information, or would you suggest that we switch to another >> DVCS if we need to store files of these sizes? > > Regardless of the problem, I don't think *any* DVCS is appropriate for > storing gigabyte sized binary files. You should look into a > centralized VCS such as subversion, methinks. Further information: Fossil was written for the purpose of managing the SQLite project. The primary SQLite Fossil repository contains about 5.8 GB of content when fully expanded, but that is compressed down to 92 MB. In other words, the whole repository spanning 18 years of development and 20K check-ins and 78K artifacts is less than 1/10th the size of just one of your files. The largest uncompressed artifact in the SQLite repo is 19 MB and the median uncompressed artifact size is 42 KB. With compression the largest artifact is 1.8 MB and median artifact size is a mere 152 bytes. I analyzed a bunch of other Fossil repositories, and they all mostly have similar stats. Except, usually the maximum uncompressed artifact size is about an order of magnitude less than 19 MB (SQLite is has a few exceptionally large artifacts) and in the case of the SQL Logic Test repository, the median artifact size was 1.5 KB. -- D. Richard Hipp d...@sqlite.org ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] Error during commit
OK. I'll investigate. Thanks for the quick response. Kind regards, Philip Bennefall On 8/6/2018 6:50 PM, Richard Hipp wrote: On 8/6/18, Philip Bennefall wrote: Do you have any recommendations for something we could try in order to get more information, or would you suggest that we switch to another DVCS if we need to store files of these sizes? Regardless of the problem, I don't think *any* DVCS is appropriate for storing gigabyte sized binary files. You should look into a centralized VCS such as subversion, methinks. ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] Error during commit
On 8/6/18, Philip Bennefall wrote: > Do you have any recommendations for something we could try in order to > get more information, or would you suggest that we switch to another > DVCS if we need to store files of these sizes? Regardless of the problem, I don't think *any* DVCS is appropriate for storing gigabyte sized binary files. You should look into a centralized VCS such as subversion, methinks. -- D. Richard Hipp d...@sqlite.org ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] Error during commit
Do you have any recommendations for something we could try in order to get more information, or would you suggest that we switch to another DVCS if we need to store files of these sizes? Kind regards, Philip Bennefall On 8/6/2018 6:33 PM, Richard Hipp wrote: On 8/6/18, Philip Bennefall wrote: We have a repository in our organization which stores a number of rather large binary files. When attempting to commit, we sometimes get something like this: ERROR: [largefile.bin is 999378424 bytes on disk but 210746789 in the repository My guess is that you are probably overflowing a 32-bit integer someplace. If so, I fear that this will not be something easily fixed. ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] Error during commit
On 8/6/18, Philip Bennefall wrote: > We have a repository in our organization which stores a number of rather > large binary files. When attempting to commit, we sometimes get > something like this: > > > ERROR: [largefile.bin is 999378424 bytes on disk but 210746789 in the > repository My guess is that you are probably overflowing a 32-bit integer someplace. If so, I fear that this will not be something easily fixed. -- D. Richard Hipp d...@sqlite.org ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] Error during commit
Additional information: The repo checksum is disabled locally. When enabling it, we get: Segmentation fault: 11 Kind regards, Philip Bennefall On 8/6/2018 6:11 PM, Philip Bennefall wrote: We have a repository in our organization which stores a number of rather large binary files. When attempting to commit, we sometimes get something like this: ERROR: [largefile.bin is 999378424 bytes on disk but 210746789 in the repository The file in question was not modified during the given commit, but was checked in months ago. What can cause this, and what steps can we take to investigate? We are using Fossil 2.6. Thanks in advance for any help. Kind regards, Philip Bennefall ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
