Re: [fossil-users] Compare by hash

2015-01-15 Thread Ron W
On Thu, Jan 15, 2015 at 5:59 AM, Kelly Dean ke...@prtime.org wrote: My point remains: Fossil unnecessarily uses an extremely slow hash if you're expected to trust the data, and uses one that's too close to being broken to warrant trust if you don't trust the data. Even on 10 year old

Re: [fossil-users] Compare by hash

2015-01-15 Thread Andy Bradford
Thus said Andy Bradford on 15 Jan 2015 13:37:24 -0700: Or would the sync protocol have to be modified so that the client, on push, receives a copy of any existing artifacts that it is intending to push so that it can compare the contents prior to pushing? Not this. This is already what

Re: [fossil-users] Compare by hash

2015-01-15 Thread Andy Bradford
Thus said Kelly Dean on Tue, 13 Jan 2015 23:24:49 +: Yes. And it needs to compare the contents, not just compare hashes as a shortcut. Given Fossil's distributed design, I don't think it is always possible to compare contents, at least not on the client. For example: A clones from S B

Re: [fossil-users] Compare by hash

2015-01-15 Thread Ron W
On Thu, Jan 15, 2015 at 3:46 PM, Andy Bradford amb-fos...@bradfords.org wrote: Thus said Andy Bradford on 15 Jan 2015 13:37:24 -0700: Or would the sync protocol have to be modified so that the client, on push, receives a copy of any existing artifacts that it is intending to push so

Re: [fossil-users] Compare by hash

2015-01-15 Thread Kelly Dean
Andy Bradford wrote: Instead of sha1, use something like a 160-bit version of xxhash, which is 10-20 times faster than a secure hash, and has no more risk of collision than does the latter, assuming you don't commit malicious people's data. The use of SHA1 is detailed here: [snip]

Re: [fossil-users] Compare by hash

2015-01-15 Thread Ron Aaron
On 1/15/15 02:50, Richard Hipp wrote: You are much more likely to be struck by a giant meteor in the millisecond before you press the Enter key than you are to encounter a hash collision. True, though being struck by the giant meteor would

Re: [fossil-users] Compare by hash

2015-01-14 Thread Andy Bradford
Thus said Kelly Dean on Wed, 14 Jan 2015 23:57:57 +: Instead of sha1, use something like a 160-bit version of xxhash, which is 10-20 times faster than a secure hash, and has no more risk of collision than does the latter, assuming you don't commit malicious people's data. The use

Re: [fossil-users] Compare by hash

2015-01-14 Thread Richard Hipp
On 1/14/15, Andy Bradford amb-fos...@bradfords.org wrote: What are the risks of a collision? Perhaps one developer, after spending hours on a solution, will have made a commit without realizing that the SHA1 for the content he is committing already exists and that Fossil will therefore

Re: [fossil-users] Compare by hash

2015-01-14 Thread Kelly Dean
Joerg Sonnenberger wrote: If you don't pull from malicious users, it doesn't happen. If you do, you already have bigger problems. What you want just makes things slower for no good reason. You're right, if you don't commit malicious people's data, then there's no problem. But in that case,

Re: [fossil-users] Compare by hash

2015-01-14 Thread Ron W
On Tue, Jan 13, 2015 at 6:24 PM, Kelly Dean ke...@prtime.org wrote: Ron W wrote: So, you are asking for an option to enable an automatic, post-commit fossil diff $@ (where $@ is the list of files just committed)? Yes. And it needs to compare the contents, not just compare hashes as a

Re: [fossil-users] Compare by hash

2015-01-14 Thread Joerg Sonnenberger
On Tue, Jan 13, 2015 at 10:46:27PM +, Kelly Dean wrote: When you're committing new files to a local repository, obviously you do have both the current data (in the repository) and the new files that you need to ensure don't collide with the current data. In this case, comparing by content

Re: [fossil-users] Compare by hash

2015-01-13 Thread Joerg Sonnenberger
On Tue, Jan 13, 2015 at 11:32:21AM -0500, Ron W wrote: On Tue, Jan 13, 2015 at 5:21 AM, Joerg Sonnenberger jo...@britannica.bec.de wrote: The much more interesting case is getting wrong data send from a 3rd party matching an existing (signed) manifest. Comparing content is not an option

Re: [fossil-users] Compare by hash

2015-01-13 Thread Joerg Sonnenberger
On Tue, Jan 13, 2015 at 01:46:05AM +, Kelly Dean wrote: If you have files foo and bar with different content but the same hash (i.e. there's a hash collision), and you commit foo, then bar, and dedup by hash (which Fossil does, just like Git), then do a checkout and try to verify by

Re: [fossil-users] Compare by hash

2015-01-13 Thread Stephan Beal
On Tue, Jan 13, 2015 at 3:46 AM, Kelly Dean ke...@prtime.org wrote: Stephan Beal wrote: (at least not in the context of fossil). If you can find a preimage, then you certainly can do the attack I described. As my little brother once told me: 'if' is a pretty big word, considering how

Re: [fossil-users] Compare by hash

2015-01-13 Thread Ron W
On Tue, Jan 13, 2015 at 5:21 AM, Joerg Sonnenberger jo...@britannica.bec.de wrote: The much more interesting case is getting wrong data send from a 3rd party matching an existing (signed) manifest. Comparing content is not an option for that as you don't have the original content. As I

Re: [fossil-users] Compare by hash

2015-01-13 Thread Kelly Dean
Joerg Sonnenberger wrote: Either the hash is broken AND you have allowed untrusted third parties to write to your repository or not. In your scenario, you have already lost at the point where you allow untrusted third parties near the the repository, so the strength of the hash function is

Re: [fossil-users] Compare by hash

2015-01-13 Thread Kelly Dean
Joerg Sonnenberger wrote: If you have files foo and bar with different content but the same hash (i.e. there's a hash collision), and you commit foo, then bar, and dedup by hash (which Fossil does, just like Git), then do a checkout and try to verify by checking the hashes, you'll be fooled

Re: [fossil-users] Compare by hash

2015-01-13 Thread Kelly Dean
Ron W wrote: As I understand the Fossil sync protocol, if the remote side offers something whose ID matches the ID of something the local Fossil already has, the local Fossil will decline it. Which means if the attacker wins the race to offer you an artifact, then your local Fossil will

Re: [fossil-users] Compare by hash

2015-01-13 Thread Kelly Dean
I wrote: B. (13.3·10^9 years per eon) / (2^66.7 days) → 2^33 eons. Er, that's 2^24.6 eons. (Forgot to divide by days/year.) Unless you're a young-Earth creationist, in which case it's 2^45.6 eons. ___ fossil-users mailing list

Re: [fossil-users] Compare by hash

2015-01-13 Thread Ron W
On Tue, Jan 13, 2015 at 4:08 PM, Kelly Dean ke...@prtime.org wrote: You could say, well, don't use Fossil for storing evidence. But why not? An option like ZFS's dedup=verify would solve the problem. Without it, Fossil unnecessarily relies on the hash function being secure. Even if a

Re: [fossil-users] Compare by hash

2015-01-13 Thread Kelly Dean
Ron W wrote: So, you are asking for an option to enable an automatic, post-commit fossil diff $@ (where $@ is the list of files just committed)? Yes. And it needs to compare the contents, not just compare hashes as a shortcut. For a simple implementation, it can just unconditionally compare

Re: [fossil-users] Compare by hash

2015-01-12 Thread Richard Hipp
On 1/12/15, Joerg Sonnenberger jo...@britannica.bec.de wrote: On Mon, Jan 12, 2015 at 11:24:58AM +, Kelly Dean wrote: Git does compare-by-hash. This is a mistake, because Valerie Hansen said so. It seems Fossil makes the same mistake. I have no idea who Valerie Hansen is or why I should

Re: [fossil-users] Compare by hash

2015-01-12 Thread Joerg Sonnenberger
On Mon, Jan 12, 2015 at 11:24:58AM +, Kelly Dean wrote: Git does compare-by-hash. This is a mistake, because Valerie Hansen said so. It seems Fossil makes the same mistake. I have no idea who Valerie Hansen is or why I should care about him. Comparing by content is generally not an option

[fossil-users] Compare by hash

2015-01-12 Thread Kelly Dean
Git does compare-by-hash. This is a mistake, because Valerie Hansen said so. It seems Fossil makes the same mistake. If the hash function is broken, you're hosed, and you don't know until it's too late. The checks described at selfcheck.wiki don't detect it. ZFS provides a dedup=verify option

Re: [fossil-users] Compare by hash

2015-01-12 Thread Stephan Beal
On Mon, Jan 12, 2015 at 12:24 PM, Kelly Dean ke...@prtime.org wrote: Git does compare-by-hash. This is a mistake, because Valerie Hansen said so. It seems Fossil makes the same mistake. If the hash function is broken, you're hosed, and you don't know until it's too late. The checks described

Re: [fossil-users] Compare by hash

2015-01-12 Thread Kelly Dean
Richard Hipp wrote: Homework problem: Suppose 100 developers each make 100 file changes every day on the same fossil repository. So 10,000 new hashes enter the repository every day. (A) Compute the expected time until the first SHA1 hash collision is encountered. (B) Rewrite the answer to

Re: [fossil-users] Compare by hash

2015-01-12 Thread Kelly Dean
Richard Hipp wrote: The monotone people addressed this concern over a decade ago: http://www.monotone.ca/docs/Hash-Integrity.html The monotone page says: ⌜the vulnerability is not that the hash might suddenly cease to address benign blocks well, but merely that additional security precautions

Re: [fossil-users] Compare by hash

2015-01-12 Thread Stephan Beal
On Tue, Jan 13, 2015 at 2:49 AM, Kelly Dean ke...@prtime.org wrote: If you allow me to push anything into your repository (even just pushing bug reports; you don't have to trust me), and I can find a hash preimage Prove it can happen first ;). It's a theoretical problem with (AFAIK) no proven

Re: [fossil-users] Compare by hash

2015-01-12 Thread Richard Hipp
On 1/12/15, Kelly Dean ke...@prtime.org wrote: If you have files foo and bar with different content but the same hash (i.e. there's a hash collision), and you commit foo, then bar, and dedup by hash (which Fossil does, just like Git), then do a checkout and try to verify by checking the

Re: [fossil-users] Compare by hash

2015-01-12 Thread Kelly Dean
Stephan Beal wrote: If you allow me to push anything into your repository (even just pushing bug reports; you don't have to trust me), and I can find a hash preimage Prove it can happen first ;). It's a theoretical problem with (AFAIK) no proven attack Are you referring to finding a

Re: [fossil-users] Compare by hash

2015-01-12 Thread Kelly Dean
Joerg Sonnenberger wrote: I have no idea who Valerie Hansen is or why I should care about him. As Richard Hipp pointed out, I misspelled her name. It was ‟Henson” (now ‟Aurora”). (Head hung in shame.) Anyway I was being facetious with the ‟because Valerie said so” remark. Comparing by