Re: [fossil-users] When modifying a file too quickly, it doesn't change
On 12/21/13, David Given d...@cowlark.com wrote: The big gotcha, of course, is scripts, which typically perform several changes on similar files in very rapid succession --- that's precisely how I found this myself. In this case, the script would (should) know which files it had changed so could force the commit. Is Fossil failing to properly perform a forced commit? ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] When modifying a file too quickly, it doesn't change
David Given wrote: The big gotcha, of course, is scripts, which typically perform several changes on similar files in very rapid succession --- that's precisely how I found this myself. I find myself very uneasy about having this defaulting to on. I have no objection to having the option there if people want it, but I do think that the *default* should be the safe option. Particularly since Fossil prides itself on safety. With my particular script I spotted the error because I ended up with an empty checkin and commit bailed. But if that file had been part of a multi-file checkin set, then it would have simply been silently omitted from the checkin and the work lost. As a concrete example: after changing my script to set it to off before starting, my test suite now fails. Ignoring the issue of what the default should be, I've added a --sha1sum option to the commit command. This way you can easily do safe commits in a script without having to detect/modify the mtime-changes setting first. You could have previously achieved the same effect by calling 'fossil changes --sha1sum' right before your commit, but that's very non-obvious and relies on internal implementation details that could theoretically change. ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] When modifying a file too quickly, it doesn't change
I'd agree with David, here. By the time most people have large repositories, they can figure out if the optimization is worth it to them. But the default should always be safety! ../Dave On 21 December 2013 19:47, David Given d...@cowlark.com wrote: On 21/12/13 22:00, Richard Hipp wrote: [...] For a large repo with many files, mtime-changes off means that many operations are slow, because running SHA1 hashes on thousands of files takes significant time. Hence mtime-changes is off by default. Normally this does not cause problems, since it is very rare that a file will actually change content without either its mtime or is size changing. The big gotcha, of course, is scripts, which typically perform several changes on similar files in very rapid succession --- that's precisely how I found this myself. I find myself very uneasy about having this defaulting to on. I have no objection to having the option there if people want it, but I do think that the *default* should be the safe option. Particularly since Fossil prides itself on safety. With my particular script I spotted the error because I ended up with an empty checkin and commit bailed. But if that file had been part of a multi-file checkin set, then it would have simply been silently omitted from the checkin and the work lost. As a concrete example: after changing my script to set it to off before starting, my test suite now fails. -- ┌─── ｄｇ＠ｃｏｗｌａｒｋ．ｃｏｍ ─ http://www.cowlark.com ─ │ There does not now, nor will there ever, exist a programming │ language in which it is the least bit hard to write bad programs. --- │ Flon's Axiom ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
[fossil-users] When modifying a file too quickly, it doesn't change
(Why yes, I am doing something moderately evil with Fossil...)
I have found a rather unpleasant-looking bug where if a file's content
changes too quickly, and its size does not change, it's not considered
to have changed. It smells as if Fossil is using a combination of the
file length and timestamp to determine whether the file has changed.
Test case follows:
echo Test data file
fossil add file
fossil commit file -m Ancestor # succeeds
echo Branch 1 file
fossil commit file -m Initial content # succeeds
echo Branch 2 file
fossil commit file -m Subsequent content # fails
This is on a repo where mtime-changes is unset. If I set it to 'off',
the test passes.
I think the fault is this chunk from vfile.c:
if( (chnged==0 || chnged==2 || chnged==4)
(useMtime==0 || currentMtime!=oldMtime) ){
/* check SHA1 hash here */
}
If the file size hasn't changed, then chnged is 0 as Fossil doesn't know
it's changed yet. If useMtime is 0 and the modification time falls
within the same second window as the original, then the condition never
fires and the SHA1 hash is never calculated.
I think that should be:
if( (chnged==0 || chnged==2 || chnged==4)
|| (useMtime currentMtime!=oldMtime) ){
...that is, check the SHA1 hash if the file is thought to be unchanged
*or* if useMtime is set *and* the mtime has in fact changed.
However I'm sufficiently unconfident about my analysis to actually go
and try it. Thoughts?
--
┌─── ｄｇ＠ｃｏｗｌａｒｋ．ｃｏｍ ─ http://www.cowlark.com ─
│ There does not now, nor will there ever, exist a programming
│ language in which it is the least bit hard to write bad programs. ---
│ Flon's Axiom
signature.asc
Description: OpenPGP digital signature
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] When modifying a file too quickly, it doesn't change
On Sat, Dec 21, 2013 at 09:21:25PM +, David Given wrote: I have found a rather unpleasant-looking bug where if a file's content changes too quickly, and its size does not change, it's not considered to have changed. It smells as if Fossil is using a combination of the file length and timestamp to determine whether the file has changed. Yes, it certainly uses mtime by default to skip the much more expensive hashing step. You can disable that from the settings. Joerg ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] When modifying a file too quickly, it doesn't change
On 21/12/13 21:24, Joerg Sonnenberger wrote: [...] Yes, it certainly uses mtime by default to skip the much more expensive hashing step. You can disable that from the settings. You mean it's *supposed* not to do the SHA1 hash by default? Isn't this horribly unsafe (because you'll miss changes, like is happening in my case)? I was assuming it was using the mtime to detect *changed* files, not *unchanged* files! -- ┌─── ｄｇ＠ｃｏｗｌａｒｋ．ｃｏｍ ─ http://www.cowlark.com ─ │ There does not now, nor will there ever, exist a programming │ language in which it is the least bit hard to write bad programs. --- │ Flon's Axiom signature.asc Description: OpenPGP digital signature ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] When modifying a file too quickly, it doesn't change
On Sat, Dec 21, 2013 at 4:34 PM, David Given d...@cowlark.com wrote: On 21/12/13 21:24, Joerg Sonnenberger wrote: [...] Yes, it certainly uses mtime by default to skip the much more expensive hashing step. You can disable that from the settings. You mean it's *supposed* not to do the SHA1 hash by default? Isn't this horribly unsafe (because you'll miss changes, like is happening in my case)? The command is fossil setting mtime-changes off. Having mtime-changes on (the default) means that fossil will look for changed files by only examining the size and mtime. With it off, Fossil does a full SHA1 hash on every file in the repository. For a large repo with many files, mtime-changes off means that many operations are slow, because running SHA1 hashes on thousands of files takes significant time. Hence mtime-changes is off by default. Normally this does not cause problems, since it is very rare that a file will actually change content without either its mtime or is size changing. mtime-changes used to be off by default, which is manageable for projects of less than 1000 files on a fast workstation. The default got changed to on when people started using Fossil on larger projects. -- D. Richard Hipp d...@sqlite.org ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] When modifying a file too quickly, it doesn't change
On Sat, Dec 21, 2013 at 09:34:45PM +, David Given wrote: On 21/12/13 21:24, Joerg Sonnenberger wrote: [...] Yes, it certainly uses mtime by default to skip the much more expensive hashing step. You can disable that from the settings. You mean it's *supposed* not to do the SHA1 hash by default? Isn't this horribly unsafe (because you'll miss changes, like is happening in my case)? I was assuming it was using the mtime to detect *changed* files, not *unchanged* files! Well, it is looking for what files changed. If a file has potentially changed, it will compute the checksum. While it might be a good idea (and moderately simple) for fossil to support sub-second precision for mtime when the filesystem does so, for normal users it works extremely well. For other cases like fossil stat, you can force it to use hashes with --sha1sum. Joerg ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] When modifying a file too quickly, it doesn't change
On 21/12/13 22:00, Richard Hipp wrote: [...] For a large repo with many files, mtime-changes off means that many operations are slow, because running SHA1 hashes on thousands of files takes significant time. Hence mtime-changes is off by default. Normally this does not cause problems, since it is very rare that a file will actually change content without either its mtime or is size changing. The big gotcha, of course, is scripts, which typically perform several changes on similar files in very rapid succession --- that's precisely how I found this myself. I find myself very uneasy about having this defaulting to on. I have no objection to having the option there if people want it, but I do think that the *default* should be the safe option. Particularly since Fossil prides itself on safety. With my particular script I spotted the error because I ended up with an empty checkin and commit bailed. But if that file had been part of a multi-file checkin set, then it would have simply been silently omitted from the checkin and the work lost. As a concrete example: after changing my script to set it to off before starting, my test suite now fails. -- ┌─── ｄｇ＠ｃｏｗｌａｒｋ．ｃｏｍ ─ http://www.cowlark.com ─ │ There does not now, nor will there ever, exist a programming │ language in which it is the least bit hard to write bad programs. --- │ Flon's Axiom signature.asc Description: OpenPGP digital signature ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
