Re: SHA-1 collision in repository?

[Myria]

The fulltext whose checksum is 80a10d37de91cadc604ba30e379651b3 I
found out is the first 16384 bytes of the file (see other parts of
this thread).  16384 is SVN__STREAM_CHUNK_SIZE.

On Fri, Mar 2, 2018 at 3:07 PM, Daniel Shahaf <d...@daniel.shahaf.name> wrote:
> Daniel Shahaf wrote on Fri, Mar 02, 2018 at 22:57:51 +:
>> Myria wrote on Mon, Feb 26, 2018 at 13:41:05 -0800:
>> > In other news, unknown whether related to the current problem, my
>> > attempt to clone the repository to my local computer is failing:
>> >
>> > D:\>svnsync sync file:///d:/svnclone
>> > Transmitting file data
>> > .svnsync:
>> > E16: SHA1 of reps '227170 153 193 57465
>> > bb52be764a04d511ebb06e1889910dcf
>> > e6291ab119036eb783d0136afccdb3b445867364 227184-4vap/_4o' and '-1 0
>> > 193 57465 bb52be764a04d511ebb06e1889910dcf
>> > e6291ab119036eb783d0136afccdb3b445867364 227184-4vap/_4o' matches
>> > (e6291ab119036eb783d0136afccdb3b445867364) but contents differ
>> > svnsync: E160004: Filesystem is corrupt
>> > svnsync: E200014: Checksum mismatch while reading representation:
>> >expected:  bb52be764a04d511ebb06e1889910dcf
>> >  actual:  80a10d37de91cadc604ba30e379651b3
>>
>> When this error happens, could you print the first lines of the two reps
>> identical?  The first line is "PLAIN\n" or "DELTA\n" or "DELTA 42 43 44\n".
>> (I wonder whether we have some stray whitespace that's transparent to parsing
>> but breaks checksums.)
>
> In second thought I'm not sure this makes sense.  A better question is: can we
> obtain the fulltext whose checksum is 80a10d37de91cadc604ba30e379651b3?
>
>> Do you happen to have a copy of the repository lying around that you can run
>> 'grep -a 80a10d37de91cadc604ba30e379651b3 db/revs/{0,1,2,...,227}' on?
>> Admittedly that's a bit of a shot in the dark.
>>
>> Cheers,
>>
>> Daniel

Re: SHA-1 collision in repository?

[Myria]

During rep_write_contents_close, there is a call to get_shared_rep.
get_shared_rep calls svn_fs_fs__get_contents_from_file, which has the
code pasted below.


  /* Build the representation list (delta chain). */
  if (rh->type == svn_fs_fs__rep_plain)
{
  rb->rs_list = apr_array_make(pool, 0, sizeof(rep_state_t *));
  rb->src_state = rs;
}
  else if (rh->type == svn_fs_fs__rep_self_delta)
{
  rb->rs_list = apr_array_make(pool, 1, sizeof(rep_state_t *));
  APR_ARRAY_PUSH(rb->rs_list, rep_state_t *) = rs;
  rb->src_state = NULL;
}
  else
{
  representation_t next_rep = { 0 };

  /* skip "SVNx" diff marker */
  rs->current = 4;

  /* REP's base rep is inside a proper revision.
   * It can be reconstructed in the usual way.  */
  next_rep.revision = rh->base_revision;
  next_rep.item_index = rh->base_item_index;
  next_rep.size = rh->base_length;
  svn_fs_fs__id_txn_reset(_rep.txn_id);

  SVN_ERR(build_rep_list(>rs_list, >base_window,
 >src_state, >len, rb->fs, _rep,
 rb->filehandle_pool));


The bug is occurring because build_rep_list is being called with
first_rep->expanded_size set to zero.  Well, the reason it's zero is
because first_rep is the second to last parameter to build_rep_list,
and the above code initialized expanded_size to zero:

representation_t next_rep = { 0 };

Does the code just need this?  I don't know this call >.<

next_rep.expanded_size = rb->rep.expanded_size;

Melissa

On Wed, Mar 7, 2018 at 9:02 AM, Nathan Hartman <hartman.nat...@gmail.com> wrote:
> On Mar 5, 2018, at 10:54 PM, Myria <myriac...@gmail.com> wrote:
>>
>> Final email for the night >.<
>>
>> What's clobbering the expanded_size is this in build_rep_list:
>>
>>  /* The value as stored in the data struct.
>> 0 is either for unknown length or actually zero length. */
>>  *expanded_size = first_rep->expanded_size;
>>
>> first_rep->expanded_size here is zero for the last call to this
>> function before the error.  In every other case before the error, the
>> two values are equal.
>>
>> Then this code executes:
>>
>>  if (*expanded_size == 0)
>>if (rep_header->type == svn_fs_fs__rep_plain || first_rep->size != 4)
>>  *expanded_size = first_rep->size;
>>
>> first_rep->size is 16384, and this is why rb->len becomes 16384,
>> leading to the error.
>>
>> I don't know what all this code is doing, but that's the proximate
>> cause of the failure.
>>
>> Melissa
>
> Has it been possible to determine what is setting expanded_size to 0 before 
> that last call? I wonder if there is specific logic that decides (perhaps 
> incorrectly?) to do that?
>
> Alternatively is it being clobbered by some out-of-bounds access, 
> use-after-free, or another such issue?
>
> Is it possible in your debugger setup to determine the address of that 
> variable and set a breakpoint that triggers when that memory is written? (It 
> may be called a watchpoint?)
>
> Which leads me to another thought: if you can set such a breakpoint / 
> watchpoint and it does not trigger, then this expanded_size might not be the 
> same instance in that final call. Perhaps a shallow copy of an enclosing 
> structure is made which leaves out the known size and sets it to 0 for some 
> reason, and that final call is given that (incomplete) copy.
>
> Caveat: I am not familiar with the codebase but these are my thoughts based 
> on adventures in other code bases.
>

Re: SHA-1 collision in repository?

[Myria]

Final email for the night >.<

What's clobbering the expanded_size is this in build_rep_list:

  /* The value as stored in the data struct.
 0 is either for unknown length or actually zero length. */
  *expanded_size = first_rep->expanded_size;

first_rep->expanded_size here is zero for the last call to this
function before the error.  In every other case before the error, the
two values are equal.

Then this code executes:

  if (*expanded_size == 0)
if (rep_header->type == svn_fs_fs__rep_plain || first_rep->size != 4)
  *expanded_size = first_rep->size;

first_rep->size is 16384, and this is why rb->len becomes 16384,
leading to the error.

I don't know what all this code is doing, but that's the proximate
cause of the failure.

Melissa


On Mon, Mar 5, 2018 at 7:41 PM, Myria <myriac...@gmail.com> wrote:
> When Subversion gets to this part of rep_read_contents, rb->len is
> 16384.  It thinks it is then done reading the entire file, and can
> compare the checksum, but it's not done with the file yet.
>
> rb->rep.expanded_size is correct at the error point, 57465.
> rep_read_get_baton sets rb->len to rb->rep.expanded_size, so I don't
> know why the value changed by the time rep_read_contents got its paws
> on the baton.  I saw that rb->len might be getting clobbered by
> rep_read_content's call to build_rep_list, which has the following
> line of code:
>
> *expanded_size = first_rep->expanded_size;
>
> expanded_size is >len.  I haven't had a chance to debug this area
> yet, so it might be fine.
>
> I verified with sqlite3 that the rep-cache.db has the correct size (57465):
>
> $ sqlite3 /mnt/d/svnclone/db/rep-cache.db "select * from rep_cache
> where hash='e6291ab119036eb783d0136afccdb3b445867364'"
> e6291ab119036eb783d0136afccdb3b445867364|227170|153|193|57465
>
>
> On Mon, Mar 5, 2018 at 6:56 PM, Myria <myriac...@gmail.com> wrote:
>> GMail keeps doing reply instead of reply all.  I'm having to manually
>> add the users list back now.
>>
>> Below is the thread I sent.
>>
>>
>> -- Forwarded message --
>> From: Myria <myriac...@gmail.com>
>> Date: Mon, Mar 5, 2018 at 6:37 PM
>> Subject: Re: SHA-1 collision in repository?
>> To: Philip Martin <phi...@codematters.co.uk>
>>
>>
>> I now know where the checksum error happens, but not why.
>>
>> svn: E200014: Checksum mismatch while reading representation:
>>expected:  bb52be764a04d511ebb06e1889910dcf
>>  actual:  80a10d37de91cadc604ba30e379651b3
>>
>> It's calculating the MD5 of only the first 16 KB of the input file and
>> comparing against the MD5 of the entire file.  The 16 KB number seems
>> to be SVN__STREAM_CHUNK_SIZE.
>>
>> bb52be764a04d511ebb06e1889910dcf is the MD5 of the entire file.
>> 80a10d37de91cadc604ba30e379651b3 is the MD5 of the first 16384 bytes.
>>
>>
>> On Mon, Mar 5, 2018 at 5:23 PM, Myria <myriac...@gmail.com> wrote:
>>> I managed to compile a subversion command line client with debugging
>>> information and optimizations disabled, and can reproduce the problem
>>> with GDB attached.
>>>
>>> Here is a backtrace at the time at which the error occurs.  A few line
>>> numbers in stream.c will be wrong by a few lines due to a few printf's
>>> I added.
>>>
>>> #0  svn_checksum_mismatch_err (expected=0x7ffdcf00,
>>> actual=0x7a0700a0, scratch_pool=0x7a070028,
>>> fmt=0x7c259ac0 "Checksum mismatch while reading
>>> representation") at subversion/libsvn_subr/checksum.c:638
>>> #1  0x7c2123de in rep_read_contents (baton=0x7a1f6190,
>>> buf=0x7a1f66a8 "// "..., len=0x7ffdcf88)
>>> at subversion/libsvn_fs_fs/cached_data.c:2062
>>> #2  0x7e5645fd in svn_stream_read_full (stream=0x7a1f6470,
>>> buffer=0x7a1f66a8 "// "..., len=0x7ffdcf88)
>>> at subversion/libsvn_subr/stream.c:193
>>> #3  0x7e5653f3 in svn_stream_contents_same2
>>> (same=0x7ffdd01c, stream1=0x7a1f6470,
>>> stream2=0x7a1f6650, pool=0x7a1e0028) at
>>> subversion/libsvn_subr/stream.c:589
>>> #4  0x7c247226 in get_shared_rep (old_rep=0x7ffdd188,
>>> fs=0x7f601030, rep=0x7a0e20b8,
>>> file=0x7a1e0390, offset=0, reps_hash=0x0,
>>> result_pool=0x7f5e0028, scratch_pool=0x7a1e0028)
>>> at subversion/libsvn_fs_fs/transaction.c:2280
>>> #5  0x7c247734 in rep_write_contents_close
>>> (baton=0

Re: SHA-1 collision in repository?

[Myria]

When Subversion gets to this part of rep_read_contents, rb->len is
16384.  It thinks it is then done reading the entire file, and can
compare the checksum, but it's not done with the file yet.

rb->rep.expanded_size is correct at the error point, 57465.
rep_read_get_baton sets rb->len to rb->rep.expanded_size, so I don't
know why the value changed by the time rep_read_contents got its paws
on the baton.  I saw that rb->len might be getting clobbered by
rep_read_content's call to build_rep_list, which has the following
line of code:

*expanded_size = first_rep->expanded_size;

expanded_size is >len.  I haven't had a chance to debug this area
yet, so it might be fine.

I verified with sqlite3 that the rep-cache.db has the correct size (57465):

$ sqlite3 /mnt/d/svnclone/db/rep-cache.db "select * from rep_cache
where hash='e6291ab119036eb783d0136afccdb3b445867364'"
e6291ab119036eb783d0136afccdb3b445867364|227170|153|193|57465


On Mon, Mar 5, 2018 at 6:56 PM, Myria <myriac...@gmail.com> wrote:
> GMail keeps doing reply instead of reply all.  I'm having to manually
> add the users list back now.
>
> Below is the thread I sent.
>
>
> -- Forwarded message --
> From: Myria <myriac...@gmail.com>
> Date: Mon, Mar 5, 2018 at 6:37 PM
> Subject: Re: SHA-1 collision in repository?
> To: Philip Martin <phi...@codematters.co.uk>
>
>
> I now know where the checksum error happens, but not why.
>
> svn: E200014: Checksum mismatch while reading representation:
>expected:  bb52be764a04d511ebb06e1889910dcf
>  actual:  80a10d37de91cadc604ba30e379651b3
>
> It's calculating the MD5 of only the first 16 KB of the input file and
> comparing against the MD5 of the entire file.  The 16 KB number seems
> to be SVN__STREAM_CHUNK_SIZE.
>
> bb52be764a04d511ebb06e1889910dcf is the MD5 of the entire file.
> 80a10d37de91cadc604ba30e379651b3 is the MD5 of the first 16384 bytes.
>
>
> On Mon, Mar 5, 2018 at 5:23 PM, Myria <myriac...@gmail.com> wrote:
>> I managed to compile a subversion command line client with debugging
>> information and optimizations disabled, and can reproduce the problem
>> with GDB attached.
>>
>> Here is a backtrace at the time at which the error occurs.  A few line
>> numbers in stream.c will be wrong by a few lines due to a few printf's
>> I added.
>>
>> #0  svn_checksum_mismatch_err (expected=0x7ffdcf00,
>> actual=0x7a0700a0, scratch_pool=0x7a070028,
>> fmt=0x7c259ac0 "Checksum mismatch while reading
>> representation") at subversion/libsvn_subr/checksum.c:638
>> #1  0x7c2123de in rep_read_contents (baton=0x7a1f6190,
>> buf=0x7a1f66a8 "// "..., len=0x7ffdcf88)
>> at subversion/libsvn_fs_fs/cached_data.c:2062
>> #2  0x7e5645fd in svn_stream_read_full (stream=0x7a1f6470,
>> buffer=0x7a1f66a8 "// "..., len=0x7ffdcf88)
>> at subversion/libsvn_subr/stream.c:193
>> #3  0x7e5653f3 in svn_stream_contents_same2
>> (same=0x7ffdd01c, stream1=0x7a1f6470,
>> stream2=0x7a1f6650, pool=0x7a1e0028) at
>> subversion/libsvn_subr/stream.c:589
>> #4  0x7c247226 in get_shared_rep (old_rep=0x7ffdd188,
>> fs=0x7f601030, rep=0x7a0e20b8,
>> file=0x7a1e0390, offset=0, reps_hash=0x0,
>> result_pool=0x7f5e0028, scratch_pool=0x7a1e0028)
>> at subversion/libsvn_fs_fs/transaction.c:2280
>> #5  0x7c247734 in rep_write_contents_close
>> (baton=0x7a232ff0) at subversion/libsvn_fs_fs/transaction.c:2370
>> #6  0x7e56492b in svn_stream_close (stream=0x7a233140) at
>> subversion/libsvn_subr/stream.c:274
>> #7  0x7e841001 in apply_window (window=0x0,
>> baton=0x7a1000a0) at subversion/libsvn_delta/text_delta.c:732
>> #8  0x7c2520d2 in window_consumer (window=0x0,
>> baton=0x7f5f1ab8) at subversion/libsvn_fs_fs/tree.c:2935
>> #9  0x7e8405ef in svn_txdelta_run (source=0x7f5f1a18,
>> target=0x7f5f1298,
>> handler=0x7c25209f ,
>> handler_baton=0x7f5f1ab8, checksum_kind=svn_checksum_md5,
>> checksum=0x7ffdd458, cancel_func=0x0, cancel_baton=0x0,
>> result_pool=0x7f5e0028,
>> scratch_pool=0x7f5e0028) at subversion/libsvn_delta/text_delta.c:454
>> #10 0x7ee98a57 in svn_wc__internal_transmit_text_deltas 
>> (tempfile=0x0,
>> new_text_base_md5_checksum=0x7ffdd5b0,
>> new_text_base_sha1_checksum=0x7ffdd5b8, db=0x7f6c17d8,
>> local_abspath=0x7f672d08
>> "/mnt/d/svntest/repository/directory/Redacted.cpp",
>>

Fwd: SHA-1 collision in repository?

[Myria]

GMail keeps doing reply instead of reply all.  I'm having to manually
add the users list back now.

Below is the thread I sent.


-- Forwarded message --
From: Myria <myriac...@gmail.com>
Date: Mon, Mar 5, 2018 at 6:37 PM
Subject: Re: SHA-1 collision in repository?
To: Philip Martin <phi...@codematters.co.uk>


I now know where the checksum error happens, but not why.

svn: E200014: Checksum mismatch while reading representation:
   expected:  bb52be764a04d511ebb06e1889910dcf
 actual:  80a10d37de91cadc604ba30e379651b3

It's calculating the MD5 of only the first 16 KB of the input file and
comparing against the MD5 of the entire file.  The 16 KB number seems
to be SVN__STREAM_CHUNK_SIZE.

bb52be764a04d511ebb06e1889910dcf is the MD5 of the entire file.
80a10d37de91cadc604ba30e379651b3 is the MD5 of the first 16384 bytes.


On Mon, Mar 5, 2018 at 5:23 PM, Myria <myriac...@gmail.com> wrote:
> I managed to compile a subversion command line client with debugging
> information and optimizations disabled, and can reproduce the problem
> with GDB attached.
>
> Here is a backtrace at the time at which the error occurs.  A few line
> numbers in stream.c will be wrong by a few lines due to a few printf's
> I added.
>
> #0  svn_checksum_mismatch_err (expected=0x7ffdcf00,
> actual=0x7a0700a0, scratch_pool=0x7a070028,
> fmt=0x7c259ac0 "Checksum mismatch while reading
> representation") at subversion/libsvn_subr/checksum.c:638
> #1  0x7c2123de in rep_read_contents (baton=0x7a1f6190,
> buf=0x7a1f66a8 "// "..., len=0x7ffdcf88)
> at subversion/libsvn_fs_fs/cached_data.c:2062
> #2  0x7e5645fd in svn_stream_read_full (stream=0x7a1f6470,
> buffer=0x7a1f66a8 "// "..., len=0x7ffdcf88)
> at subversion/libsvn_subr/stream.c:193
> #3  0x7e5653f3 in svn_stream_contents_same2
> (same=0x7ffdd01c, stream1=0x7a1f6470,
> stream2=0x7a1f6650, pool=0x7a1e0028) at
> subversion/libsvn_subr/stream.c:589
> #4  0x7c247226 in get_shared_rep (old_rep=0x7ffdd188,
> fs=0x7f601030, rep=0x7a0e20b8,
> file=0x7a1e0390, offset=0, reps_hash=0x0,
> result_pool=0x7f5e0028, scratch_pool=0x7a1e0028)
> at subversion/libsvn_fs_fs/transaction.c:2280
> #5  0x7c247734 in rep_write_contents_close
> (baton=0x7a232ff0) at subversion/libsvn_fs_fs/transaction.c:2370
> #6  0x7e56492b in svn_stream_close (stream=0x7a233140) at
> subversion/libsvn_subr/stream.c:274
> #7  0x7e841001 in apply_window (window=0x0,
> baton=0x7a1000a0) at subversion/libsvn_delta/text_delta.c:732
> #8  0x7c2520d2 in window_consumer (window=0x0,
> baton=0x7f5f1ab8) at subversion/libsvn_fs_fs/tree.c:2935
> #9  0x7e8405ef in svn_txdelta_run (source=0x7f5f1a18,
> target=0x7f5f1298,
> handler=0x7c25209f ,
> handler_baton=0x7f5f1ab8, checksum_kind=svn_checksum_md5,
> checksum=0x7ffdd458, cancel_func=0x0, cancel_baton=0x0,
> result_pool=0x7f5e0028,
> scratch_pool=0x7f5e0028) at subversion/libsvn_delta/text_delta.c:454
> #10 0x7ee98a57 in svn_wc__internal_transmit_text_deltas (tempfile=0x0,
> new_text_base_md5_checksum=0x7ffdd5b0,
> new_text_base_sha1_checksum=0x7ffdd5b8, db=0x7f6c17d8,
> local_abspath=0x7f672d08
> "/mnt/d/svntest/repository/directory/Redacted.cpp",
> fulltext=0, editor=0x7f673700, file_baton=0x7f510110,
> result_pool=0x7f6c0028,
> scratch_pool=0x7f5e0028) at subversion/libsvn_wc/adm_crawler.c:1109
> #11 0x7ee98d68 in svn_wc_transmit_text_deltas3
> (new_text_base_md5_checksum=0x7ffdd5b0,
> new_text_base_sha1_checksum=0x7ffdd5b8, wc_ctx=0x7f6c17c0,
> local_abspath=0x7f672d08
> "/mnt/d/svntest/repository/directory/Redacted.cpp",
> fulltext=0, editor=0x7f673700, file_baton=0x7f510110,
> result_pool=0x7f6c0028,
> scratch_pool=0x7f5e0028) at subversion/libsvn_wc/adm_crawler.c:1199
> #12 0x7f18eb12 in svn_client__do_commit (
> base_url=0x7f6142c0 "file:///mnt/d/svntest/repository/directory",
> commit_items=0x7f672c48, editor=0x7f673700,
> edit_baton=0x7f6300a0,
> notify_path_prefix=0x7f672900 "/mnt/d/svntest/repository",
> sha1_checksums=0x7ffdd750,
> ctx=0x7f6c16f0, result_pool=0x7f6c0028, 
> scratch_pool=0x7f650028)
> at subversion/libsvn_client/commit_util.c:1920
> #13 0x7f18a5f9 in svn_client_commit6 (targets=0x7f670a18,
> depth=svn_depth_infinity, keep_locks=0,
> keep_changelists=0, commit_as_operations=1,
> include_file_externals=0, inclu

Re: SHA-1 collision in repository?

[Myria]

How can I dump out the two things that Subversion thinks have the same
SHA-1 checksum but don't match?  This seems to be rather difficult to do.

That said, it's far more likely that there's a bug in Subversion than that
we randomly collided SHA-1.

On Sun, Mar 4, 2018 at 02:29 Philip Martin  wrote:

> Nathan Hartman  writes:
>
> > Does this mean that content being committed to the repository is never
> > elided based on the SHA hash alone but only after a fulltext
> > verification that the content actually already exists in the
> > repository?
>
> That's correct.  Fulltext matching was added in 1.9.6 and 1.8.18, older
> versions of Subversion relied on the SHA1 match alone.
>
> --
> Philip
>

Re: SHA-1 collision in repository?

[Myria]

The problem is identical on Windows command line, Windows TortoiseSVN,
Ubuntu-Linux, Ubuntu-Linux on Windows, and macOS.  I'm just bad at
GDB.

On Thu, Mar 1, 2018 at 9:09 PM, Nico Kadel-Garcia <nka...@gmail.com> wrote:
> On Thu, Mar 1, 2018 at 10:25 PM, Myria <myriac...@gmail.com> wrote:
>> I just found out that the file causing the error from the large commit
>> is not the large file - it's one of the smaller files, about 55 KB.
>> If I commit that single smaller file from the large commit, it errors
>> the same way as the original 227185 would.  This is exactly like the
>> original problem with committing the pixel shader.
>>
>> I managed to get the db/transactions/227184-4vb2.txn directory by
>> breakpointing kernel32!DeleteFileW in TortoiseSVN (so I could get the
>> contents before TortoiseSVN deleted them at failure).  I don't know
>> how they're useful, though.
>>
>> The only way I know how to proceed is to wait until the source code to
>> TortoiseSVN is available so that I can debug it in Visual Studio.  Is
>> there something else I can do?
>
> Sorry that I've not been paying attention to every detail. Do you see
> the same issues if you use the Subversion from CygWin, which is
> proably a lot easier to recompile?

Re: SHA-1 collision in repository?

[Myria]

I just found out that the file causing the error from the large commit
is not the large file - it's one of the smaller files, about 55 KB.
If I commit that single smaller file from the large commit, it errors
the same way as the original 227185 would.  This is exactly like the
original problem with committing the pixel shader.

I managed to get the db/transactions/227184-4vb2.txn directory by
breakpointing kernel32!DeleteFileW in TortoiseSVN (so I could get the
contents before TortoiseSVN deleted them at failure).  I don't know
how they're useful, though.

The only way I know how to proceed is to wait until the source code to
TortoiseSVN is available so that I can debug it in Visual Studio.  Is
there something else I can do?

On Thu, Mar 1, 2018 at 6:45 PM, Myria <myriac...@gmail.com> wrote:
> On Wed, Feb 28, 2018 at 6:17 AM, Nico Kadel-Garcia <nka...@gmail.com> wrote:
>> On Tue, Feb 27, 2018 at 4:09 PM, Myria <myriac...@gmail.com> wrote:
>>
>>> Not to mention that the two revisions complained about are unrelated, and
>>> 2/3 the repository history apart.
>>>
>>> One thing that's interesting is that the commit the svnsync failed on is a
>>> gigantic commit.  It's 1.8 GB.  Maybe that svnsync is failing because of a
>>> Subversion bug with huge files...?
>>
>> Hmm. Could 2 GB filesize limites be involved?
>>
>> When someone starts encountering this kind of issue with such large
>> commits, it leads me to think "what the heck was in that commit"?
>> There are various tools more likely to break when hammered that hard,
>> wuch as pre-commit hooks written carelessly in Python that try to
>> preload a hash with the contents of the file and just say "holy sone
>> of a !@#$, I'm out of resources!!!". Been there, done that, had to
>> explain the concept of reading a text file with a loop to the
>> programmer in question.
>>
>
> The error with the 2 GB file occurred when trying to replicate the
> repository in order to diagnose the original problem.  The original
> problem does not involve large files.
>
> Also, I have no control over what was in the repository five years
> ago.  The huge files were compiled versions of WebKit libraries.  The
> alternative to committing these very large files would have been to
> quadruple the build times, because it takes four times longer to build
> WebKit than it does to build our project.
>
>
> In other news, I can now reproduce the huge file problem in
> TortoiseSVN committing to my "file:" partial copy of the repository.
> However, with SourceForge down due to a DDoS, I cannot get the source
> code to TortoiseSVN in order to debug it.
>
> This does mean that this is very likely to be a Subversion bug,
> probably something in 1.8.x or 1.9.x.  The commit that prevented
> "svnsync" from working was probably during 1.6 or 1.7, which
> succeeded.
>
> Melissa

Re: SHA-1 collision in repository?

[Myria]

On Wed, Feb 28, 2018 at 6:17 AM, Nico Kadel-Garcia <nka...@gmail.com> wrote:
> On Tue, Feb 27, 2018 at 4:09 PM, Myria <myriac...@gmail.com> wrote:
>
>> Not to mention that the two revisions complained about are unrelated, and
>> 2/3 the repository history apart.
>>
>> One thing that's interesting is that the commit the svnsync failed on is a
>> gigantic commit.  It's 1.8 GB.  Maybe that svnsync is failing because of a
>> Subversion bug with huge files...?
>
> Hmm. Could 2 GB filesize limites be involved?
>
> When someone starts encountering this kind of issue with such large
> commits, it leads me to think "what the heck was in that commit"?
> There are various tools more likely to break when hammered that hard,
> wuch as pre-commit hooks written carelessly in Python that try to
> preload a hash with the contents of the file and just say "holy sone
> of a !@#$, I'm out of resources!!!". Been there, done that, had to
> explain the concept of reading a text file with a loop to the
> programmer in question.
>

The error with the 2 GB file occurred when trying to replicate the
repository in order to diagnose the original problem.  The original
problem does not involve large files.

Also, I have no control over what was in the repository five years
ago.  The huge files were compiled versions of WebKit libraries.  The
alternative to committing these very large files would have been to
quadruple the build times, because it takes four times longer to build
WebKit than it does to build our project.


In other news, I can now reproduce the huge file problem in
TortoiseSVN committing to my "file:" partial copy of the repository.
However, with SourceForge down due to a DDoS, I cannot get the source
code to TortoiseSVN in order to debug it.

This does mean that this is very likely to be a Subversion bug,
probably something in 1.8.x or 1.9.x.  The commit that prevented
"svnsync" from working was probably during 1.6 or 1.7, which
succeeded.

Melissa

Re: SHA-1 collision in repository?

[Myria]

On Tue, Feb 27, 2018 at 05:54 Philip Martin <phi...@codematters.co.uk>
wrote:

> Myria <myriac...@gmail.com> writes:
>
> > -bash-4.1$ sqlite3 rep-cache.db "select * from rep_cache where
> > hash='db11617ef1454332336e00abc311d44bc698f3b3'"
> > db11617ef1454332336e00abc311d44bc698f3b3|604440|34|134255|136680
> >
> > The line from the grep -a command containing that hash is below.  They
> > all match.
> > text: 604440 34 134255 136680 c9f4fabc4d093612fece03c339401058
> > db11617ef1454332336e00abc311d44bc698f3b3 604439-cyqm/_13
>
> The rep-cache looks correct.  There doesn't seem to be any corruption in
> the repository: you confirmed that you could retreive the revision in
> question, and that you could verify the revision, and the rep-cache
> looks OK.  So why is the commit that attempts to reuse the data in the
> revision failing?  I don't know :-(
>
> > In other news, unknown whether related to the current problem, my
> > attempt to clone the repository to my local computer is failing:
> >
> > D:\>svnsync sync file:///d:/svnclone
> > Transmitting file data
> >
> .svnsync:
> > E16: SHA1 of reps '227170 153 193 57465
> > bb52be764a04d511ebb06e1889910dcf
> > e6291ab119036eb783d0136afccdb3b445867364 227184-4vap/_4o' and '-1 0
> > 193 57465 bb52be764a04d511ebb06e1889910dcf
> > e6291ab119036eb783d0136afccdb3b445867364 227184-4vap/_4o' matches
> > (e6291ab119036eb783d0136afccdb3b445867364) but contents differ
> > svnsync: E160004: Filesystem is corrupt
> > svnsync: E200014: Checksum mismatch while reading representation:
> >expected:  bb52be764a04d511ebb06e1889910dcf
> >  actual:  80a10d37de91cadc604ba30e379651b3
> >
> > This is odd, because revision 227185 (the revision it's trying to
> > commit) verifies fine on the originating server:
>
> That's an error committing to the new repository on your local machine,
> i.e. the problem is in the new repository not the repository on the
> originating server.  Can you run "svnadmin verify" on the new
> repository?  You may want to use -M to increase the cache size for the
> verify command as the default is small.
>
> It would be odd for svnsync to create a corrupt repository, so I half
> expect verify to report no problems.  If that is the case it seems to be
> the original pproblem again: an apparently valid repository with a
> checksum error only on commit.  So this problem is happening on two
> repositories, on two machines with different OS.


Not to mention that the two revisions complained about are unrelated, and
2/3 the repository history apart.

One thing that's interesting is that the commit the svnsync failed on is a
gigantic commit.  It's 1.8 GB.  Maybe that svnsync is failing because of a
Subversion bug with huge files...?

I started an svnadmin verify on my incomplete local copy last night, and no
problems were reported when it finished this morning.  I'll try again with
this -M option you mention.

I'll also start an svnsync from a Linux machine.

I'm going to see how hard it would be to just copy the 43 GB repository
directly.  We'd have to shut down Subversion service during the copy, so it
might be a while before I have a chance to.

>
>
> --
> Philip
>

Re: SHA-1 collision in repository?

[Myria]

-bash-4.1$ sqlite3 rep-cache.db "select * from rep_cache where
hash='db11617ef1454332336e00abc311d44bc698f3b3'"
db11617ef1454332336e00abc311d44bc698f3b3|604440|34|134255|136680

The line from the grep -a command containing that hash is below.  They
all match.
text: 604440 34 134255 136680 c9f4fabc4d093612fece03c339401058
db11617ef1454332336e00abc311d44bc698f3b3 604439-cyqm/_13


In other news, unknown whether related to the current problem, my
attempt to clone the repository to my local computer is failing:

D:\>svnsync sync file:///d:/svnclone
Transmitting file data
.svnsync:
E16: SHA1 of reps '227170 153 193 57465
bb52be764a04d511ebb06e1889910dcf
e6291ab119036eb783d0136afccdb3b445867364 227184-4vap/_4o' and '-1 0
193 57465 bb52be764a04d511ebb06e1889910dcf
e6291ab119036eb783d0136afccdb3b445867364 227184-4vap/_4o' matches
(e6291ab119036eb783d0136afccdb3b445867364) but contents differ
svnsync: E160004: Filesystem is corrupt
svnsync: E200014: Checksum mismatch while reading representation:
   expected:  bb52be764a04d511ebb06e1889910dcf
 actual:  80a10d37de91cadc604ba30e379651b3

This is odd, because revision 227185 (the revision it's trying to
commit) verifies fine on the originating server:

-bash-4.1$ sudo svnadmin verify -r227170 /srv/subversion/repositories/meow
* Verifying repository metadata ...
* Verifying metadata at revision 227170 ...
* Verified revision 227170.
-bash-4.1$ sudo svnadmin verify -r227185 /srv/subversion/repositories/meow
* Verifying repository metadata ...
* Verified revision 227185.

On Fri, Feb 23, 2018 at 5:42 PM, Philip Martin  wrote:
> Philip Martin  writes:
>
>> There are a couple of options:
>>
>>   A) disable rep-caching by editing fsfs.conf inside the repository
>>
>>   B) reset the mapping by deleting/renaming the file db/rep-cache.db
>>  inside the repository (but please rename rather than delete if you
>>  want to help us identify the corruption)
>>
>> Doing either of these should allow the commit to succeed.
>
> To verify the corruption start with the rep-cache:
>
>   sqlite3 db/rep-cache.db "select * from rep_cache where 
> hash='db11617ef1454332336e00abc311d44bc698f3b3'"
>
> That should give you five numbers: the hash, the revision (604440), the
> offset, the size and the expanded size.
>
> Then examine the revision file for r604440.  It could be unpacked:
>
>   grep -a "^text: 604440.*/_" db/revs/604/604440
>
> or packed:
>
>   grep -a "^text: 604440.*/_" db/revs/604.pack/pack
>
> One of the lines from grep should contain the hash and that line should
> start:
>
>   text: 604440
>
> followed by three more numbers then hashes and other stuff.  The three
> numbers are the offset, size and expanded size and should match the
> values from the rep-cache but I suspect the rep-cache has the wrong
> offset.
>
> --
> Philip

Re: SHA-1 collision in repository?

[Myria]

I'm not subscribed to this mailing list, so I have no standard way to
reply to Philip's email.  I don't even know his email address.

> That pattern, all of MD5, SHA1 and size matching, is exactly what
> happens if a SHA1 collision is committed using an old version of
> Subversion where the rep-cache does not detect collisions.  The first
> part of the collision would have been committed in r604440 and the
> second part in r605556.
>
> If that is the case, and a SHA1 collision did occur, then:
>
>   svnadmin verify -r604440 path/to/repository
>
> will succeed while:
>
>svnadmin verify -r605556 path/to/repository
>
> will fail with an MD5 checksum error.
>
> If this is what you see then unfortunately the colliding r605556 content
> has been elided and the r605556 revision is corrupt.

The revision 605556 is simply the current revision number of the
repository at the time of the attempted commit, and is unrelated to
the problem.  If I attempt the commit now, it's a higher number, but
otherwise the same error message.

Something I did notice is that the commit I'm trying to do is a
reversion to an older version of the same file.  The revision of the
file throwing the error at 604440 is identical to the file I'm trying
to commit, but the file currently in the repository is different.

If I commit a dummy version of the file, then commit the version I
actually want, the latter commit works.  Could the collision be in a
"diff" instead of the files themselves?

Melissa

On Thu, Feb 22, 2018 at 2:45 PM, Matt Simmons <band...@gmail.com> wrote:
> I would get more advice from people here before you invest that time. I'm a
> relative amateur and would listen to people with more experience than
> myself.
>
> --Matt
>
> On Thu, Feb 22, 2018 at 2:29 PM, Myria <myriac...@gmail.com> wrote:
>>
>> That was one document we ran into when searching, yes.
>>
>> We can do an svnsync, but this will take about a week to run--the
>> repository is 43 GB with 600,000 commits.  I guess we'll start it now.
>>
>> On Thu, Feb 22, 2018 at 2:04 PM, Matt Simmons <band...@gmail.com> wrote:
>> > Hi Melissa,
>> >
>> > That definitely is interesting.
>> >
>> > I assume you have read
>> >
>> > http://blogs.collab.net/subversion/subversion-sha1-collision-problem-statement-prevention-remediation-options
>> >
>> > If you do an svnsync to another location and attempt the commit there,
>> > does
>> > the problem replicate itself?
>> >
>> > --Matt
>> >
>> >
>> > On Thu, Feb 22, 2018 at 12:30 PM, Myria <myriac...@gmail.com> wrote:
>> >>
>> >> When we try to commit a very specific version of a very specific
>> >> binary file, we get a SHA-1 collision error from the Subversion
>> >> repository:
>> >>
>> >> D:\confidential>svn commit secret.bin -m "Testing broken commit"
>> >> Sendingsecret.bin
>> >> Transmitting file data .svn: E16: Commit failed (details follow):
>> >> svn: E16: SHA1 of reps '604440 34 134255 136680
>> >> c9f4fabc4d093612fece03c339401058
>> >> db11617ef1454332336e00abc311d44bc698f3b3 605556-czmh/_8' and '-1 0
>> >> 134255 136680 c9f4fabc4d093612fece03c339401058
>> >> db11617ef1454332336e00abc311d44bc698f3b3 605556-czmh/_8' matches
>> >> (db11617ef1454332336e00abc311d44bc698f3b3) but contents differ
>> >>
>> >>
>> >> What can cause this?  This file is a binary pixel shader compiled from
>> >> a build process.  It's most certainly not Google's SHA-1 collision PDF
>> >> files.  We also scanned the repository to confirm that nobody has
>> >> committed Google's collision files.
>> >>
>> >> Occam's Razor suggests that something is wrong with our repository or
>> >> Subversion itself, rather than this being a true SHA-1 collision.  In
>> >> that case, what is wrong with our repository?
>> >>
>> >> If this really is a SHA-1 collision, it would be major cryptography
>> >> news that someone randomly ran into a second collision without even
>> >> trying.  In that case, is there a method by which we could recover the
>> >> two files that supposedly have the same SHA-1?  The collision doesn't
>> >> appear to be in the file itself, but in some sort of diff or revision
>> >> output?
>> >>
>> >> Thanks,
>> >>
>> >> Melissa
>> >
>> >
>> >
>> >
>> > --
>> > "Today, vegetables... Tomorrow, the world!"
>
>
>
>
> --
> "Today, vegetables... Tomorrow, the world!"

Re: SHA-1 collision in repository?

[Myria]

That was one document we ran into when searching, yes.

We can do an svnsync, but this will take about a week to run--the
repository is 43 GB with 600,000 commits.  I guess we'll start it now.

On Thu, Feb 22, 2018 at 2:04 PM, Matt Simmons <band...@gmail.com> wrote:
> Hi Melissa,
>
> That definitely is interesting.
>
> I assume you have read
> http://blogs.collab.net/subversion/subversion-sha1-collision-problem-statement-prevention-remediation-options
>
> If you do an svnsync to another location and attempt the commit there, does
> the problem replicate itself?
>
> --Matt
>
>
> On Thu, Feb 22, 2018 at 12:30 PM, Myria <myriac...@gmail.com> wrote:
>>
>> When we try to commit a very specific version of a very specific
>> binary file, we get a SHA-1 collision error from the Subversion
>> repository:
>>
>> D:\confidential>svn commit secret.bin -m "Testing broken commit"
>> Sendingsecret.bin
>> Transmitting file data .svn: E16: Commit failed (details follow):
>> svn: E16: SHA1 of reps '604440 34 134255 136680
>> c9f4fabc4d093612fece03c339401058
>> db11617ef1454332336e00abc311d44bc698f3b3 605556-czmh/_8' and '-1 0
>> 134255 136680 c9f4fabc4d093612fece03c339401058
>> db11617ef1454332336e00abc311d44bc698f3b3 605556-czmh/_8' matches
>> (db11617ef1454332336e00abc311d44bc698f3b3) but contents differ
>>
>>
>> What can cause this?  This file is a binary pixel shader compiled from
>> a build process.  It's most certainly not Google's SHA-1 collision PDF
>> files.  We also scanned the repository to confirm that nobody has
>> committed Google's collision files.
>>
>> Occam's Razor suggests that something is wrong with our repository or
>> Subversion itself, rather than this being a true SHA-1 collision.  In
>> that case, what is wrong with our repository?
>>
>> If this really is a SHA-1 collision, it would be major cryptography
>> news that someone randomly ran into a second collision without even
>> trying.  In that case, is there a method by which we could recover the
>> two files that supposedly have the same SHA-1?  The collision doesn't
>> appear to be in the file itself, but in some sort of diff or revision
>> output?
>>
>> Thanks,
>>
>> Melissa
>
>
>
>
> --
> "Today, vegetables... Tomorrow, the world!"

SHA-1 collision in repository?

[Myria]

When we try to commit a very specific version of a very specific
binary file, we get a SHA-1 collision error from the Subversion
repository:

D:\confidential>svn commit secret.bin -m "Testing broken commit"
Sendingsecret.bin
Transmitting file data .svn: E16: Commit failed (details follow):
svn: E16: SHA1 of reps '604440 34 134255 136680
c9f4fabc4d093612fece03c339401058
db11617ef1454332336e00abc311d44bc698f3b3 605556-czmh/_8' and '-1 0
134255 136680 c9f4fabc4d093612fece03c339401058
db11617ef1454332336e00abc311d44bc698f3b3 605556-czmh/_8' matches
(db11617ef1454332336e00abc311d44bc698f3b3) but contents differ


What can cause this?  This file is a binary pixel shader compiled from
a build process.  It's most certainly not Google's SHA-1 collision PDF
files.  We also scanned the repository to confirm that nobody has
committed Google's collision files.

Occam's Razor suggests that something is wrong with our repository or
Subversion itself, rather than this being a true SHA-1 collision.  In
that case, what is wrong with our repository?

If this really is a SHA-1 collision, it would be major cryptography
news that someone randomly ran into a second collision without even
trying.  In that case, is there a method by which we could recover the
two files that supposedly have the same SHA-1?  The collision doesn't
appear to be in the file itself, but in some sort of diff or revision
output?

Thanks,

Melissa