Re: SHA-1 collision in repository?

[Johan Corveleyn]

[ Please keep the users list in cc. ]

On Tue, Feb 27, 2018 at 11:38 PM, Myria  wrote:
> On Tue, Feb 27, 2018 at 2:22 PM, Johan Corveleyn  wrote:
>> On Tue, Feb 27, 2018 at 10:09 PM, Myria  wrote:
>>>
>>> On Tue, Feb 27, 2018 at 05:54 Philip Martin 
>>> wrote:

 Myria  writes:

 > -bash-4.1$ sqlite3 rep-cache.db "select * from rep_cache where
 > hash='db11617ef1454332336e00abc311d44bc698f3b3'"
 > db11617ef1454332336e00abc311d44bc698f3b3|604440|34|134255|136680
 >
 > The line from the grep -a command containing that hash is below.  They
 > all match.
 > text: 604440 34 134255 136680 c9f4fabc4d093612fece03c339401058
 > db11617ef1454332336e00abc311d44bc698f3b3 604439-cyqm/_13

 The rep-cache looks correct.  There doesn't seem to be any corruption in
 the repository: you confirmed that you could retreive the revision in
 question, and that you could verify the revision, and the rep-cache
 looks OK.  So why is the commit that attempts to reuse the data in the
 revision failing?  I don't know :-(

 > In other news, unknown whether related to the current problem, my
 > attempt to clone the repository to my local computer is failing:
 >
 > D:\>svnsync sync file:///d:/svnclone
 > Transmitting file data
 >
 > .svnsync:
 > E16: SHA1 of reps '227170 153 193 57465
 > bb52be764a04d511ebb06e1889910dcf
 > e6291ab119036eb783d0136afccdb3b445867364 227184-4vap/_4o' and '-1 0
 > 193 57465 bb52be764a04d511ebb06e1889910dcf
 > e6291ab119036eb783d0136afccdb3b445867364 227184-4vap/_4o' matches
 > (e6291ab119036eb783d0136afccdb3b445867364) but contents differ
 > svnsync: E160004: Filesystem is corrupt
 > svnsync: E200014: Checksum mismatch while reading representation:
 >expected:  bb52be764a04d511ebb06e1889910dcf
 >  actual:  80a10d37de91cadc604ba30e379651b3
 >
 > This is odd, because revision 227185 (the revision it's trying to
 > commit) verifies fine on the originating server:

 That's an error committing to the new repository on your local machine,
 i.e. the problem is in the new repository not the repository on the
 originating server.  Can you run "svnadmin verify" on the new
 repository?  You may want to use -M to increase the cache size for the
 verify command as the default is small.

 It would be odd for svnsync to create a corrupt repository, so I half
 expect verify to report no problems.  If that is the case it seems to be
 the original pproblem again: an apparently valid repository with a
 checksum error only on commit.  So this problem is happening on two
 repositories, on two machines with different OS.
>>>
>>>
>>> Not to mention that the two revisions complained about are unrelated, and
>>> 2/3 the repository history apart.
>>>
>>> One thing that's interesting is that the commit the svnsync failed on is a
>>> gigantic commit.  It's 1.8 GB.  Maybe that svnsync is failing because of a
>>> Subversion bug with huge files...?
>>>
>>> I started an svnadmin verify on my incomplete local copy last night, and no
>>> problems were reported when it finished this morning.  I'll try again with
>>> this -M option you mention.
>>>
>>> I'll also start an svnsync from a Linux machine.
>>>
>>> I'm going to see how hard it would be to just copy the 43 GB repository
>>> directly.  We'd have to shut down Subversion service during the copy, so it
>>> might be a while before I have a chance to.
>>
>> What version of SVN server are you using actually? AFAICS you never
>> mentioned this.
>>
>> I'm wondering whether this is related to the bug that was fixed for 1.8.x 
>> here:
>>
>> http://svn.apache.org/viewvc?view=revision=1803435
>>
>> ... or a similar problem.
>> I'm actually not sure whether that bugfix was released already (it's
>> not mentioned in CHANGES).
>>
>> See also the users@ thread it references (an false positive of a SHA-1
>> collision occurred during 'svnadmin load'):
>> https://lists.apache.org/thread.html/b475d74442bdf93b21c8656ab2289b4c61e0d90efdafc8a16ddca694@%3Cusers.subversion.apache.org%3E
>>
>> OTOH there was also another report of a SHA-1 collision during
>> 'svnadmin load', this time with 1.9.7. We never got to the bottom of
>> that one either:
>> https://svn.haxx.se/users/archive-2018-01/0062.shtml
>>
> Both the server and my desktop system are on Subversion 1.9.7.
>

-- 
Johan

RE: E130003: The XML response contains invalid XML - svn co and log issue on some repos using https/http

[NOCERA, ANDY]

My cert is not valid at this point,  I had turned off ssl and had the same 
error.  I wasn’t sure how to get beyond the 301 on a curl




curl -k -u an2124 --insecure https://scm-uat.web.att.com:6022/svn/asgard
Enter host password for user 'an2124':


301 Moved Permanently

Moved Permanently
The document has moved https://scm-uat.web.att.com:6022/svn/asgard/;>here.



From: Michael Rohan [mailto:mro...@stonepillar.com]
Sent: Tuesday, February 27, 2018 10:11 PM
To: Paul Hammant 
Cc: NOCERA, ANDY ; users@subversion.apache.org
Subject: Re: E130003: The XML response contains invalid XML - svn co and log 
issue on some repos using https/http

Hi,
Might be interesting to see what sort of document you are getting:
$ curl https://scm-uat.web.att.com:6022/svn/asgard
Might need "-k" on that command depending on what sort of certs are in place.
Take care,
Michael.

On Tue, Feb 27, 2018 at 6:44 PM, Paul Hammant 
> wrote:
Do you get the same error with a testfile.txt resource in a freshly created 
(and Apache mounted) Subversion repo?



--
=
Michael Rohan
Stone Pillar Technologies
=

RE: E130003: The XML response contains invalid XML - svn co and log issue on some repos using https/http

[NOCERA, ANDY]

There are several repos work fine and even different version is this repo.



From: Paul Hammant [mailto:p...@hammant.org]
Sent: Tuesday, February 27, 2018 9:44 PM
To: NOCERA, ANDY 
Cc: users@subversion.apache.org
Subject: Re: E130003: The XML response contains invalid XML - svn co and log 
issue on some repos using https/http

Do you get the same error with a testfile.txt resource in a freshly created 
(and Apache mounted) Subversion repo?

Re: E130003: The XML response contains invalid XML - svn co and log issue on some repos using https/http

[Michael Rohan]

Hi,

Might be interesting to see what sort of document you are getting:

$ curl https://scm-uat.web.att.com:6022/svn/asgard

Might need "-k" on that command depending on what sort of certs are in
place.

Take care,
Michael.

On Tue, Feb 27, 2018 at 6:44 PM, Paul Hammant  wrote:

> Do you get the same error with a testfile.txt resource in a freshly
> created (and Apache mounted) Subversion repo?
>



-- 
=
Michael Rohan
Stone Pillar Technologies
=

Re: E130003: The XML response contains invalid XML - svn co and log issue on some repos using https/http

[Paul Hammant]

Do you get the same error with a testfile.txt resource in a freshly created
(and Apache mounted) Subversion repo?

RE: E130003: The XML response contains invalid XML - svn co and log issue on some repos using https/http

[NOCERA, ANDY]

Paul,
Here is the information regarding the logs,
Thanks
Andy

CMD:
svn co https://scm-uat.web.att.com:6022/svn/asgard
svn: E130003: The XML response contains invalid XML


ERROR LOG
[Tue Feb 27 21:31:25.717954 2018] [ssl:info] [pid 18302:tid 140359493441280] 
[client IPADDRESS:55820] AH01964: Connection to child 139 established (server 
x.com:443)
[Tue Feb 27 21:31:25.932500 2018] [authz_svn:info] [pid 18302:tid 
140359482951424] [client IPADDRESS:55820] Access granted: 'an2124' OPTIONS 
asgard:/
[Tue Feb 27 21:31:25.989989 2018] [authz_svn:info] [pid 18302:tid 
140359461971712] [client IPADDRESS:55820] Access granted: 'an2124' OPTIONS 
asgard:/
[Tue Feb 27 21:31:26.045795 2018] [authz_svn:info] [pid 18302:tid 
140359472461568] [client IPADDRESS:55820] Access granted: 'an2124' OPTIONS 
asgard:/
[Tue Feb 27 21:31:26.101858 2018] [authz_svn:info] [pid 18302:tid 
140359451481856] [client IPADDRESS:55820] Access granted: 'an2124' PROPFIND 
asgard:/
[Tue Feb 27 21:31:26.157470 2018] [ssl:info] [pid 18302:tid 140359577360128] 
(70014)End of file found: [client IPADDRESS:55820] AH01991: SSL input filter 
read failed.
[Tue Feb 27 21:31:26.231191 2018] [ssl:info] [pid 31885:tid 140359587849984] 
[client IPADDRESS:55822] AH01964: Connection to child 258 established (server 
scm.test.att.com:443)
[Tue Feb 27 21:31:26.434479 2018] [authz_svn:info] [pid 31885:tid 
140359556380416] [client IPADDRESS:55822] Access granted: 'an2124' OPTIONS 
asgard:/
[Tue Feb 27 21:31:26.490891 2018] [authz_svn:info] [pid 31885:tid 
140359545890560] [client IPADDRESS:55822] Access granted: 'an2124' OPTIONS 
asgard:/
[Tue Feb 27 21:31:26.546663 2018] [authz_svn:info] [pid 31885:tid 
140359472461568] [client IPADDRESS:55822] Access granted: 'an2124' OPTIONS 
asgard:/
[Tue Feb 27 21:31:26.613113 2018] [authz_svn:info] [pid 31885:tid 
140359420012288] [client IPADDRESS:55822] Access granted: 'an2124' REPORT 
asgard:
[Tue Feb 27 21:31:26.613393 2018] [authz_svn:info] [pid 31885:tid 
140359420012288] [client IPADDRESS:55822] Access granted: 'an2124' GET asgard:/
[Tue Feb 27 21:31:26.613665 2018] [authz_svn:info] [pid 31885:tid 
140359420012288] [client IPADDRESS:55822] Access granted: 'an2124' GET asgard:/x
[Tue Feb 27 21:31:26.613800 2018] [authz_svn:info] [pid 31885:tid 
140359420012288] [client IPADDRESS:55822] Access granted: 'an2124' GET 
asgard:/almighty-gab
[Tue Feb 27 21:31:26.614012 2018] [authz_svn:info] [pid 31885:tid 
140359420012288] [client IPADDRESS:55822] Access granted: 'an2124' GET 
asgard:/almighty-gab/test_file_gabalmighty.dat
[Tue Feb 27 21:31:26.614221 2018] [dav:error] [pid 31885:tid 140359420012288] 
[client IPADDRESS:55822] Provider encountered an error while streaming a REPORT 
response.  [500, #0]
[Tue Feb 27 21:31:26.614236 2018] [dav:error] [pid 31885:tid 140359420012288] 
[client IPADDRESS:55822] A failure occurred while driving the update report 
editor  [500, #160053]
[Tue Feb 27 21:31:26.614242 2018] [dav:error] [pid 31885:tid 140359420012288] 
[client IPADDRESS:55822] While reading representation offsets for node-revision 
'2-1026.0.r1115/37':  [500, #160053]
[Tue Feb 27 21:31:26.614248 2018] [dav:error] [pid 31885:tid 140359420012288] 
[client IPADDRESS:55822] malformed txn id 
'1114-99a9211d96cf5c6c693c519a16016f97ab2324bcwn'  [500, #160053]


Server version: Apache/2.4.29 (Unix)
Server built:   Feb 26 2018 12:21:35
$ bin/httpd -V
Server version: Apache/2.4.29 (Unix)
Server built:   Feb 26 2018 12:21:35
Server's Module Magic Number: 20120211:68
Server loaded:  APR 1.6.3, APR-UTIL 1.6.1
Compiled using: APR 1.6.3, APR-UTIL 1.6.1
Architecture:   64-bit
Server MPM: event
  threaded: yes (fixed thread count)
forked: yes (variable process count)
Server compiled with
-D APR_HAS_SENDFILE
-D APR_HAS_MMAP
-D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
-D APR_USE_SYSVSEM_SERIALIZE
-D APR_USE_PTHREAD_SERIALIZE
-D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
-D APR_HAS_OTHER_CHILD
-D AP_HAVE_RELIABLE_PIPED_LOGS
-D DYNAMIC_MODULE_LIMIT=256
-D HTTPD_ROOT="/opt/app/scm/svn/binaries/svn_1.9.7"
-D SUEXEC_BIN="/opt/app/scm/svn/binaries/svn_1.9.7/bin/suexec"
-D DEFAULT_PIDLOG="logs/httpd.pid"
-D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
-D DEFAULT_ERRORLOG="logs/error_log"
-D AP_TYPES_CONFIG_FILE="conf/mime.types"
-D SERVER_CONFIG_FILE="conf/httpd.conf"



From: Paul Hammant [mailto:p...@hammant.org]
Sent: Tuesday, February 27, 2018 7:36 PM
To: NOCERA, ANDY 
Cc: users@subversion.apache.org
Subject: Re: E130003: The XML response contains invalid XML - svn co and log 
issue on some repos using https/http

See what you can see on the server-side Apache logs - 

Re: E130003: The XML response contains invalid XML - svn co and log issue on some repos using https/http

[Paul Hammant]

See what you can see on the server-side Apache logs -
https://unix.stackexchange.com/questions/115972/how-do-i-find-where-apache-keeps-the-log-files
(and others; or adjust for your Linux).

E130003: The XML response contains invalid XML - svn co and log issue on some repos using https/http

[NOCERA, ANDY]

Hi,

I have a recently built SVN 1.9.7 including httpd and I see issues using 
https/https protocol on a few repos and version.  The test repos are 1.8.X and 
work using file level protocol and svnserve. Some Other repos work with no 
issues.

For asgard example below, I was able to checkout from -r r1114  having latest 
r1115 checkout fail. I was able to checkout 1114 and commit to 1116, however 
that didn't fix it, Svnadmin verify returns no errors.

Please advise,
Andy

svn co https://xxx.com:6022/svn/asgard
svn: E130003: The XML response contains invalid XML

svn -v -rr1014 log https://.com:6022/svn/asgard

r1014 | asgardia | 2015-05-08 16:05:07 -0700 (Fri, 08 May 2015) | 1 line
Changed paths:
   A /stash

stuff


svn -v -rr1015 log https://xxx.com:6022/svn/asgard
svn: E175002: Unexpected HTTP status 400 'Bad Request' on 
'/svn/asgard/!svn/rvr/1015'

svn: E160062: Additional errors:
svn: E160062: Malformed node revision ID string


$ svn log -V -r r1116 file:///opt/app/scm/svn/reps/asgard
svn: invalid option character: V
Type 'svn help' for usage.
$ svn log -v -r r1116 file:///opt/app/scm/svn/reps/asgard

r1116 | an2124 | 2018-02-27 16:54:13 -0500 (Tue, 27 Feb 2018) | 2 lines
Changed paths:
   A /x

[TEST] This is a test commit to remove 1015


$ svn log -v -r r1115 file:///opt/app/scm/svn/reps/asgard

r1115 | rv620d | 2016-07-20 11:26:45 -0400 (Wed, 20 Jul 2016) | 1 line
Changed paths:
   M /almighty-gab/test_file_gabalmighty.dat

test

$ svn log -v -r r1114 file:///opt/app/scm/svn/reps/asgard

r1114 | grageda | 2016-06-08 16:00:14 -0400 (Wed, 08 Jun 2016) | 1 line
Changed paths:
   M /developer1/test.sh

COMMIT

#apr-1.6.3.tar - standard apache
#apr-util-1.6.1.tar - standard apache utils
#pcre-8.41.tar - used by httpd

#libtool-2.4.6.tar - used by openldap

#openldap_2_4_26_src.tar - needed for httpd+ldap and svn+https
#  openldap_2_4_26_src.tar - contains 02-berkeleydb  03-sasl2 - we didn't need 
to build.

#
#Python-2.7.13.tar - used to build scons and serf
# scons-local-2.3.0.tar - build script for serf
# serf-1.3.9.zip - enabes SVN https://svncom

#subversion-1.9.7.tar - subversion distribution
# subversion-1.9.7/sqlite-amalgamation - had to be added

# httpd-2.4.29.tar - httpd - standalone https

Re: SHA-1 collision in repository?

[Johan Corveleyn]

On Tue, Feb 27, 2018 at 10:09 PM, Myria  wrote:
>
> On Tue, Feb 27, 2018 at 05:54 Philip Martin 
> wrote:
>>
>> Myria  writes:
>>
>> > -bash-4.1$ sqlite3 rep-cache.db "select * from rep_cache where
>> > hash='db11617ef1454332336e00abc311d44bc698f3b3'"
>> > db11617ef1454332336e00abc311d44bc698f3b3|604440|34|134255|136680
>> >
>> > The line from the grep -a command containing that hash is below.  They
>> > all match.
>> > text: 604440 34 134255 136680 c9f4fabc4d093612fece03c339401058
>> > db11617ef1454332336e00abc311d44bc698f3b3 604439-cyqm/_13
>>
>> The rep-cache looks correct.  There doesn't seem to be any corruption in
>> the repository: you confirmed that you could retreive the revision in
>> question, and that you could verify the revision, and the rep-cache
>> looks OK.  So why is the commit that attempts to reuse the data in the
>> revision failing?  I don't know :-(
>>
>> > In other news, unknown whether related to the current problem, my
>> > attempt to clone the repository to my local computer is failing:
>> >
>> > D:\>svnsync sync file:///d:/svnclone
>> > Transmitting file data
>> >
>> > .svnsync:
>> > E16: SHA1 of reps '227170 153 193 57465
>> > bb52be764a04d511ebb06e1889910dcf
>> > e6291ab119036eb783d0136afccdb3b445867364 227184-4vap/_4o' and '-1 0
>> > 193 57465 bb52be764a04d511ebb06e1889910dcf
>> > e6291ab119036eb783d0136afccdb3b445867364 227184-4vap/_4o' matches
>> > (e6291ab119036eb783d0136afccdb3b445867364) but contents differ
>> > svnsync: E160004: Filesystem is corrupt
>> > svnsync: E200014: Checksum mismatch while reading representation:
>> >expected:  bb52be764a04d511ebb06e1889910dcf
>> >  actual:  80a10d37de91cadc604ba30e379651b3
>> >
>> > This is odd, because revision 227185 (the revision it's trying to
>> > commit) verifies fine on the originating server:
>>
>> That's an error committing to the new repository on your local machine,
>> i.e. the problem is in the new repository not the repository on the
>> originating server.  Can you run "svnadmin verify" on the new
>> repository?  You may want to use -M to increase the cache size for the
>> verify command as the default is small.
>>
>> It would be odd for svnsync to create a corrupt repository, so I half
>> expect verify to report no problems.  If that is the case it seems to be
>> the original pproblem again: an apparently valid repository with a
>> checksum error only on commit.  So this problem is happening on two
>> repositories, on two machines with different OS.
>
>
> Not to mention that the two revisions complained about are unrelated, and
> 2/3 the repository history apart.
>
> One thing that's interesting is that the commit the svnsync failed on is a
> gigantic commit.  It's 1.8 GB.  Maybe that svnsync is failing because of a
> Subversion bug with huge files...?
>
> I started an svnadmin verify on my incomplete local copy last night, and no
> problems were reported when it finished this morning.  I'll try again with
> this -M option you mention.
>
> I'll also start an svnsync from a Linux machine.
>
> I'm going to see how hard it would be to just copy the 43 GB repository
> directly.  We'd have to shut down Subversion service during the copy, so it
> might be a while before I have a chance to.

What version of SVN server are you using actually? AFAICS you never
mentioned this.

I'm wondering whether this is related to the bug that was fixed for 1.8.x here:

http://svn.apache.org/viewvc?view=revision=1803435

... or a similar problem.
I'm actually not sure whether that bugfix was released already (it's
not mentioned in CHANGES).

See also the users@ thread it references (an false positive of a SHA-1
collision occurred during 'svnadmin load'):
https://lists.apache.org/thread.html/b475d74442bdf93b21c8656ab2289b4c61e0d90efdafc8a16ddca694@%3Cusers.subversion.apache.org%3E

OTOH there was also another report of a SHA-1 collision during
'svnadmin load', this time with 1.9.7. We never got to the bottom of
that one either:
https://svn.haxx.se/users/archive-2018-01/0062.shtml

-- 
Johan

Re: Show textual diff in a moved/copied file - how?

[Johan Corveleyn]

On Tue, Feb 27, 2018 at 5:50 PM, Stefan Sperling  wrote:
> On Tue, Feb 27, 2018 at 07:52:00AM -0800, Alexey Neyman wrote:
>> Thanks for bringing up this explanation.
>
> Indeed!
> I had totally forgotten about this conversion from years ago.
>
>> So the second inconsistency is
>> because '-c X' actually defines operative range X-1:X and the source of the
>> copy is X-2 in this case.
>>
>> Indeed, a lot of subtleties and inconsistencies that appear to be bugs.
>>
>> Is there ever going to be SVN 2.0 that can finally break these bug-for-bug
>> compatibility promises? Is there a list of things that are going to be
>> changed in 2.0?
>
> I wouldn't object to changing 'svn diff' to match the behaviour
> of 'svnlook diff' in this particular case. The inconsistency
> does not help anyone, and our compatibilty guarantees aren't
> *that* solid. We've certainly changed some output of our tooling
> when it helped our users, even where doing so hurt scripts.

+1. Backwards compatibility shouldn't block us from fixing bugs.

This certainly feels like a bug to me (the fact that it's inconsistent
depending on the operative revision range, and inconsistent with
'svnlook diff' where the behaviour seems more sane, is a strong
indication).

> I think my concerns were more about the effort involved, rather
> than compatibility. The process of adding --show-copies-as-adds
> was surprisingly difficult. I wouldn't want to go back to that
> code myself. I would review another brave soul's patches, though.
> The effort involved is easy to underestimate, unfortunately.

Putting Julian in cc because he was just talking about the diff code
on IRC today. You never know :-) ...

-- 
Johan

Re: SHA-1 collision in repository?

[Myria]

On Tue, Feb 27, 2018 at 05:54 Philip Martin 
wrote:

> Myria  writes:
>
> > -bash-4.1$ sqlite3 rep-cache.db "select * from rep_cache where
> > hash='db11617ef1454332336e00abc311d44bc698f3b3'"
> > db11617ef1454332336e00abc311d44bc698f3b3|604440|34|134255|136680
> >
> > The line from the grep -a command containing that hash is below.  They
> > all match.
> > text: 604440 34 134255 136680 c9f4fabc4d093612fece03c339401058
> > db11617ef1454332336e00abc311d44bc698f3b3 604439-cyqm/_13
>
> The rep-cache looks correct.  There doesn't seem to be any corruption in
> the repository: you confirmed that you could retreive the revision in
> question, and that you could verify the revision, and the rep-cache
> looks OK.  So why is the commit that attempts to reuse the data in the
> revision failing?  I don't know :-(
>
> > In other news, unknown whether related to the current problem, my
> > attempt to clone the repository to my local computer is failing:
> >
> > D:\>svnsync sync file:///d:/svnclone
> > Transmitting file data
> >
> .svnsync:
> > E16: SHA1 of reps '227170 153 193 57465
> > bb52be764a04d511ebb06e1889910dcf
> > e6291ab119036eb783d0136afccdb3b445867364 227184-4vap/_4o' and '-1 0
> > 193 57465 bb52be764a04d511ebb06e1889910dcf
> > e6291ab119036eb783d0136afccdb3b445867364 227184-4vap/_4o' matches
> > (e6291ab119036eb783d0136afccdb3b445867364) but contents differ
> > svnsync: E160004: Filesystem is corrupt
> > svnsync: E200014: Checksum mismatch while reading representation:
> >expected:  bb52be764a04d511ebb06e1889910dcf
> >  actual:  80a10d37de91cadc604ba30e379651b3
> >
> > This is odd, because revision 227185 (the revision it's trying to
> > commit) verifies fine on the originating server:
>
> That's an error committing to the new repository on your local machine,
> i.e. the problem is in the new repository not the repository on the
> originating server.  Can you run "svnadmin verify" on the new
> repository?  You may want to use -M to increase the cache size for the
> verify command as the default is small.
>
> It would be odd for svnsync to create a corrupt repository, so I half
> expect verify to report no problems.  If that is the case it seems to be
> the original pproblem again: an apparently valid repository with a
> checksum error only on commit.  So this problem is happening on two
> repositories, on two machines with different OS.


Not to mention that the two revisions complained about are unrelated, and
2/3 the repository history apart.

One thing that's interesting is that the commit the svnsync failed on is a
gigantic commit.  It's 1.8 GB.  Maybe that svnsync is failing because of a
Subversion bug with huge files...?

I started an svnadmin verify on my incomplete local copy last night, and no
problems were reported when it finished this morning.  I'll try again with
this -M option you mention.

I'll also start an svnsync from a Linux machine.

I'm going to see how hard it would be to just copy the 43 GB repository
directly.  We'd have to shut down Subversion service during the copy, so it
might be a while before I have a chance to.

>
>
> --
> Philip
>

Re: Show textual diff in a moved/copied file - how?

[Stefan Sperling]

On Tue, Feb 27, 2018 at 07:52:00AM -0800, Alexey Neyman wrote:
> Thanks for bringing up this explanation.

Indeed!
I had totally forgotten about this conversion from years ago.

> So the second inconsistency is
> because '-c X' actually defines operative range X-1:X and the source of the
> copy is X-2 in this case.
> 
> Indeed, a lot of subtleties and inconsistencies that appear to be bugs.
> 
> Is there ever going to be SVN 2.0 that can finally break these bug-for-bug
> compatibility promises? Is there a list of things that are going to be
> changed in 2.0?

I wouldn't object to changing 'svn diff' to match the behaviour
of 'svnlook diff' in this particular case. The inconsistency
does not help anyone, and our compatibilty guarantees aren't
*that* solid. We've certainly changed some output of our tooling
when it helped our users, even where doing so hurt scripts.

I think my concerns were more about the effort involved, rather
than compatibility. The process of adding --show-copies-as-adds
was surprisingly difficult. I wouldn't want to go back to that
code myself. I would review another brave soul's patches, though.
The effort involved is easy to underestimate, unfortunately.

Re: Show textual diff in a moved/copied file - how?

[Alexey Neyman]

On 02/27/2018 03:26 AM, Johan Corveleyn wrote:

On Tue, Feb 27, 2018 at 9:13 AM, Stefan Sperling  wrote:

On Mon, Feb 26, 2018 at 04:52:41PM -0800, Alexey Neyman wrote:

Why is the behavior different in these cases? Isn't that counter-intuitive
as well that the diff's output depends on the source revision of the copy?

I think these differences in behaviour boil down to side-effects of
the implementation.

As I posted before in this thread, this problem was already noted and
discussed before on dev@ (feel free to follow the links I posted :-)).
But I'm happy this issue is brought back to the foreground, because I
too consider this an issue and inconsistent behaviour from the user's
perspective (regardless of the underlying implementation problem).

Back in 2013, you, Stefan, wrote:

On Tue, Jun 25, 2013 at 10:56 PM, Stefan Sperling  wrote:

On Tue, Jun 25, 2013 at 10:26:08PM +0200, Ben Reser wrote:

Back to your issue.  Since Subversion can't represent the copy as part
of the diff it tries to do the interoperable thing which is to
represent the addition of a new file (from a copy) as an addition.

That's not quite true in general. It's more like:

1) If the copyfrom source is part of the operative revision range of
the diff command, show a modification against the copyfrom source.
Unless --show-copies-as-adds was passed, because then we always
show copied files as an addition.

2) If the copyfrom source is not part of the operative revision range,
history of the file isn't traced back to that revision, so it appears
as an addition.

It could be argued that 2) is weird special case, and that it should
behave like 1) (i.e. trace back to the copyfrom source anyway) and
only show an addition with --show-copies-as-adds.

Johan pointed out that svnlook diff seems to traverse to the copyfrom
source even in case 2). If this is indeed the case, these commands are
now behaving in contradictory ways :( However, I think it's too late
to change either command now.
Thanks for bringing up this explanation. So the second inconsistency is 
because '-c X' actually defines operative range X-1:X and the source of 
the copy is X-2 in this case.


Indeed, a lot of subtleties and inconsistencies that appear to be bugs.

Is there ever going to be SVN 2.0 that can finally break these 
bug-for-bug compatibility promises? Is there a list of things that are 
going to be changed in 2.0?


Regards,
Alexey.

Re: SHA-1 collision in repository?

[Philip Martin]

Myria  writes:

> -bash-4.1$ sqlite3 rep-cache.db "select * from rep_cache where
> hash='db11617ef1454332336e00abc311d44bc698f3b3'"
> db11617ef1454332336e00abc311d44bc698f3b3|604440|34|134255|136680
>
> The line from the grep -a command containing that hash is below.  They
> all match.
> text: 604440 34 134255 136680 c9f4fabc4d093612fece03c339401058
> db11617ef1454332336e00abc311d44bc698f3b3 604439-cyqm/_13

The rep-cache looks correct.  There doesn't seem to be any corruption in
the repository: you confirmed that you could retreive the revision in
question, and that you could verify the revision, and the rep-cache
looks OK.  So why is the commit that attempts to reuse the data in the
revision failing?  I don't know :-(

> In other news, unknown whether related to the current problem, my
> attempt to clone the repository to my local computer is failing:
>
> D:\>svnsync sync file:///d:/svnclone
> Transmitting file data
> .svnsync:
> E16: SHA1 of reps '227170 153 193 57465
> bb52be764a04d511ebb06e1889910dcf
> e6291ab119036eb783d0136afccdb3b445867364 227184-4vap/_4o' and '-1 0
> 193 57465 bb52be764a04d511ebb06e1889910dcf
> e6291ab119036eb783d0136afccdb3b445867364 227184-4vap/_4o' matches
> (e6291ab119036eb783d0136afccdb3b445867364) but contents differ
> svnsync: E160004: Filesystem is corrupt
> svnsync: E200014: Checksum mismatch while reading representation:
>expected:  bb52be764a04d511ebb06e1889910dcf
>  actual:  80a10d37de91cadc604ba30e379651b3
> 
> This is odd, because revision 227185 (the revision it's trying to
> commit) verifies fine on the originating server:

That's an error committing to the new repository on your local machine,
i.e. the problem is in the new repository not the repository on the
originating server.  Can you run "svnadmin verify" on the new
repository?  You may want to use -M to increase the cache size for the
verify command as the default is small.

It would be odd for svnsync to create a corrupt repository, so I half
expect verify to report no problems.  If that is the case it seems to be
the original pproblem again: an apparently valid repository with a
checksum error only on commit.  So this problem is happening on two
repositories, on two machines with different OS.

-- 
Philip

Re: Bugreport: invalid xml file produced by: svn log --xml --verbose --use-merge-history --search "msg1"

[Johan Corveleyn]

On Tue, Feb 20, 2018 at 3:17 PM, Martin Obermeir
 wrote:
> Hi all,
>
> I entered this bug to the issue tracker last year:
> https://issues.apache.org/jira/browse/SVN-4711
>
> Should I also post it to the development list (d...@subversion.apache.org
> )?

Hi Martin,

It was good that you tracked this down, and filed an issue (with
reproduction script), but unfortunately this doesn't guarantee that
someone will pick this up and fix it. After all, Apache Subversion is
an open source project largely driven by volunteers these days. So, it
depends ...

Asking again on dev@ doesn't really help either (several devs already
follow this list), unless you have some proposal on how to fix it. For
example, drilling down into the source code, pointing to the place in
the code where it goes wrong, or even proposing a patch, ... those are
certainly useful inputs for a post to dev@.

Or as they say ... patches welcome ;-).

-- 
Johan

Re: Show textual diff in a moved/copied file - how?

[Johan Corveleyn]

On Tue, Feb 27, 2018 at 9:13 AM, Stefan Sperling  wrote:
> On Mon, Feb 26, 2018 at 04:52:41PM -0800, Alexey Neyman wrote:
>> Why is the behavior different in these cases? Isn't that counter-intuitive
>> as well that the diff's output depends on the source revision of the copy?
>
> I think these differences in behaviour boil down to side-effects of
> the implementation.

As I posted before in this thread, this problem was already noted and
discussed before on dev@ (feel free to follow the links I posted :-)).
But I'm happy this issue is brought back to the foreground, because I
too consider this an issue and inconsistent behaviour from the user's
perspective (regardless of the underlying implementation problem).

Back in 2013, you, Stefan, wrote:

On Tue, Jun 25, 2013 at 10:56 PM, Stefan Sperling  wrote:
> On Tue, Jun 25, 2013 at 10:26:08PM +0200, Ben Reser wrote:
>> Back to your issue.  Since Subversion can't represent the copy as part
>> of the diff it tries to do the interoperable thing which is to
>> represent the addition of a new file (from a copy) as an addition.
>
> That's not quite true in general. It's more like:
>
> 1) If the copyfrom source is part of the operative revision range of
>the diff command, show a modification against the copyfrom source.
>Unless --show-copies-as-adds was passed, because then we always
>show copied files as an addition.
>
> 2) If the copyfrom source is not part of the operative revision range,
>history of the file isn't traced back to that revision, so it appears
>as an addition.
>
> It could be argued that 2) is weird special case, and that it should
> behave like 1) (i.e. trace back to the copyfrom source anyway) and
> only show an addition with --show-copies-as-adds.
>
> Johan pointed out that svnlook diff seems to traverse to the copyfrom
> source even in case 2). If this is indeed the case, these commands are
> now behaving in contradictory ways :( However, I think it's too late
> to change either command now.

-- 
Johan

Re: Show textual diff in a moved/copied file - how?

[Stefan Sperling]

On Mon, Feb 26, 2018 at 04:52:41PM -0800, Alexey Neyman wrote:
> Why is the behavior different in these cases? Isn't that counter-intuitive
> as well that the diff's output depends on the source revision of the copy?

I think these differences in behaviour boil down to side-effects of
the implementation.

On the server-side, all of diff/update/merge are driven by the same code,
the "reporter" in libsvn_repos. So when trying to fix diff one needs to be
careful not to break the other two.
I've always had a lot of fun whenever I dove in there...

In other words, it looks like a simple bug on the surface, but when you dig
in to figure out what needs fixing it gets tricky rather quickly.
Maybe diff needs a separate driver implementation on the server.