Re: Questions on migrating SVN (and Trac) to a Google Compute Engine instance

[Daniel Shahaf]

Johan Corveleyn wrote on Sat, 29 Jul 2017 11:39 +0200:
> Two other things which are also not transferred by svnsync (nor by dump/load):
> - locks (server-side locks, of the svn:needs-lock type): these should
> be copied from $OLDREPOS/db/locks to $NEWREPOS/db/locks

It would be best to do this under 'svnadmin freeze'.  Currently, the
failure mode of a non-atomic copy is harmless, but we don't promise
it'll always be this way.

RE: Checksum mismatch bug in 1.8.18

[Bert Huijben]

This specific error message was added in the last release, so yes it is new
in the last versions. The last 1.9.x version also added it and I'm surprised
that you see the error on 1.8 and not 1.9 (or vice versa).

 

It tries to tell you that you have two files with an identical SHA-1 hash,
but different contents. A case that we accidentally didn't catch before,
that might have caused some data loss under extreme circumstances.

 

 

This scenario is pretty unlikely to trigger, unless you specifically try to
do that. I'll try to verify what exactly is in your dump file to see if you
really hit this scenario, or that we have a bug in our code that tries to
catch this scenario. Assuming you properly verified with 1.9.6 (and not
1.9.5 or earlier) I assume that this is a bug in 1.8.18, as other scenarios
are really extremely unlikely.

 

I'm not sure I'll have time to look into this before Monday though, so
perhaps one of the other developers beats me to it.

 

Bert

 

From: David Engel [mailto:den...@magnitude.com] 
Sent: zaterdag 29 juli 2017 00:01
To: users@subversion.apache.org
Subject: Checksum mismatch bug in 1.8.18

 

Hi,

 

I think I'm running into a bug in svnadmin in the latest 1.8 release
(1.8.18).  It is not present in 1.8.17 nor 1.9.5 nor 1.9.6.  I'm guessing
it's related to the recent changes around strict repsharing.  I'm not
subscribed to the mailing list, so please CC me on any responses.

 

Here are the relevant details:

 

OS: Windows 10 and Windows 2012 R2

Subversion release: 1.8.18

Both SlikSVN release:

svnadmin, version 1.8.18-SlikSvn-1.8.18-X64 (SlikSvn/1.8.18) X64

   compiled Jul 17 2017, 13:03:37 on x86_64-microsoft-windows6.2.9200

And Collabnet release:

svnadmin, version 1.8.18 (r1800620)

   compiled Jul  7 2017, 05:51:59 on x86_64-microsoft-windows6.2.9200

 

Performing an svnadmin load results in the following error:

...

<<< Started new transaction, based on original revision 650

 * editing path :
Dev/Common/Utils/External/StampVerData/MetaBuilderVersion.inf ... done.

 * editing path :
Dev/Common/Utils/External/StampVerData/MetaBuilderVersionDBG.inf ... done.

 * editing path :
Dev/Common/Utils/External/StampVerData/OracleVersion.inf ...svnadmin:
E16: SHA1 of reps '-1 0 45 132 a6ea37d29094deece56250ebe167ce46
6a2d6bc4c7e58c0d2627a1b0dd5b23e542aec4d1 650-i2/_8' and '-1 136 45 132
a6ea37d29094deece56250ebe167ce46 6a2d6bc4c7e58c0d2627a1b0dd5b23e542aec4d1
650-i2/_8' matches (6a2d6bc4c7e58c0d2627a1b0dd5b23e542aec4d1) but contents
differ

svnadmin: E160004: Filesystem is corrupt

svnadmin: E200014: Checksum mismatch while reading representation:

   expected:  a6ea37d29094deece56250ebe167ce46

 actual:  5f696f5d0755f3bcb5e927bdfba4bfa8

 

In order to reproduce the error, you can use the attached VersionStamps3
repository dump file.  It was taken with the same version of svnadmin from a
multi-GB repository that's been around for over a decade and pruned
(svndumpfilter) to a small set of files and revisions in one folder while
still encountering the error.  I first encountered the error another place
very quickly in an incremental load on top of a repo that was loaded from
1.8.14, but that one was on a revision north of 120,000+.  This is way
easier to reproduce.  I tried filtering the dump further and dropping empty
revisions, but the error did not occur in those cases.  The error also did
not occur using 1.8.17, 1.9.5., and 1.9.6.  It also did not occur in 1.8.18
if I set enable-rep-sharing = false before performing the load.

 

Repro.bat script:

@echo off

 

:defineCommands

rem You might need to adjust these lines to point to your

rem compiled-from-source Subversion binaries, if using those:

for %%i in (svn.exe) do set SVN="%%~$PATH:i"

for %%i in (svnadmin.exe) do set SVNADMIN="%%~$PATH:i"

 

:defineUrls

rem Only supported access method: file://  . If http:// or svn://,
then

rem you'll have to configure it yourself first.

set URL=file:///%CD%/repos

set URL=%URL:\=/%

echo Base url for repo: %URL%

 

:cleanAllDirsAndCreateRepo

if exist repos rmdir /s /q repos

if exist import-me rmdir /s /q import-me

if exist wc rmdir /s /q wc

%SVNADMIN% create repos

pause

 

:prepareGreekTree

echo Making a Tree for import...

mkdir import-me\Dev\Common\Utils\External

echo Importing it...

cd import-me

%SVN% import -q -m "Initial import." %URL%

cd ..

 

rem This is where your reproduction recipe goes.

svnadmin --version

svnadmin load repos < VersionStamps3

goto :eof

 

 

Thanks,

David

Re: Questions on migrating SVN (and Trac) to a Google Compute Engine instance

[Johan Corveleyn]

On Sat, Jul 29, 2017 at 1:16 AM, James H. H. Lampert
 wrote:
> On 7/27/17, 11:15 PM, Ryan Schmidt wrote:
>>
>> Sounds plausible. An empty pre-revprop-change hook script would allow
>> any revprop change, which you may not want. It's probably possible to
>> write a more-specific script that would allow only the changes
>> svnsync needs and disallow others.
>
> . . .
>>
>> svnsync is probably best, since it allows you to easily incrementally
>> mirror a live read/write repository to another server. It can be slow
>> but once it's done it makes it very quick to switch from the old
>> server to the new one with minimal downtime. Some of the other
>> methods require you to make the source repository read-only or take
>> it offline for the duration of the migration, which could take hours
>> or days depending on how large your repository is.
>
> . . .
> and Arwin and Nico said similar things.
>
> Thanks, Ryan, Arwin, and Nico.
>
> It took a bit of futzing around, but as I type this, I'm replicating a
> repository (the smallest and currently least-active one).
>
> It took me a while to realize that the hooks with .tmpl extensions were
> templates, not live hooks, and I was right on the verge of asking for help
> when I looked at the instructions again, and realized that instead of
> setting up the required null hook, I'd overwritten a template (DUH!)

Indeed, you only need to copy the files under $REPOS/hooks which are
not ending in .tmpl :-). And for the custom hook scripts you have:
maybe review them and compare the comments section at the beginning
with the new corresonding template to see if there are new features
(perhaps copy over the new comment block to your custom hook script).

Two other things which are also not transferred by svnsync (nor by dump/load):
- config files (under $REPOS/conf), for instance authz
- locks (server-side locks, of the svn:needs-lock type): these should
be copied from $OLDREPOS/db/locks to $NEWREPOS/db/locks

FWIW, I've documented some of these things in an extended answer in
our FAQ, about dump/load (some of these things also apply to svnsync):
http://subversion.apache.org/faq.html#dumpload

For the record, you can also perform an incemental dump/load with
minimal downtime (I explained this procedure in the FAQ). But svnsync
is still a lot easier, because of the automatic normalization of
properties (which is not done automatically by 'svnadmin load' at the
moment, which will error out on non-LF line endings in svn:log
messages and things like that, possibly giving a lot of headaches ...
see FAQ).

-- 
Johan

Re: upgrading server

[Johan Corveleyn]

On Tue, Jul 25, 2017 at 9:00 PM, Mark Phippard  wrote:
> On Tue, Jul 25, 2017 at 2:00 PM, Andy So  wrote:
>>
>> We have an old subversion version 1.4.3 (r23084) running on Solaris.
>>
>> We would like to upgrade to use new hardware on Linux based OS (CentOS
>> 6.9), possibly version 1.8.x or 1.9.x
>>
>>
>>
>> Our plan is to installed and configure the latest SVN on CentOS 6.9. Then
>> go through dump and load of the repository as described in various online
>> post and documentations.  The repository is quite large…guessing the size to
>> be in the order of 20-40GB
>>
>>
>>
>> Before we start undertaking such tasks
>>
>> 1.  Does anyone know if there are there any problem/gotcha in
>> migrating the repository?
>>
>> 2.  Does anyone know how long it would take to export the repository
>> of this size?  This will give us an estimate how long to schedule down time
>> and cut off time.
>>
>>
>>
>> Thanks for any insight.
>
>
>
> Dump and load is a good idea because it lets the repository be rewritten
> using the newer code and repository format which will give you a smaller
> repository that will run a little faster.  That said you do not HAVE to
> dump/load.  You have other options:
>
> 1. Just move the repository folder to the new server.  Perhaps using tar and
> then moving the archive.
>
> 2. Instead of using dump/load, use svnsync.  This gives all the benefits of
> the dump/load but allows you to shrink your downtime to almost nothing.
> Just svnsync the repository to the new server.  This will probably take a
> long time, but it does not matter since the original server can be running
> while it happens.  At the time of your choosing, do a final svnsync, and
> then shutdown the old server and use the new one.
>
> 3. Do option 1 now and then do a dump/load or svnsync at some future time
> that is more convenient for downtime.  It will probably run faster since it
> is on new and better hardware too.
>
> There are some gotchas no matter what:
>
> 1.  Does the new server have a new hostname or do you intend to update DNS
> to point to new server?  If you are not doing the latter, then all of your
> existing working copies and scripts have to be modified for the new server.
> Also any use of svn:externals property has to be modified.
>
> 2. With an old repository there is a good chance you will run into bugs in
> your data that cause svnsync or load to fail.  There are workaround for
> different failures but be prepared to run into them and account for finding
> the solutions.  SVN 1.8 and 1.9 have added various options to let you
> workaround some of the known bugs.  A common problem is having svn:
> properties that are not UTF-8 encoded or do not have LF line endings.
> svnsync has this option to work around the encoding problem:
>
> --source-prop-encoding ARG : convert translatable properties from encoding
> ARG
>  to UTF-8. If not specified, then properties are
>  presumed to be encoded in UTF-8.
>
> And it automatically fixes the LF problems.
>
> svnadmin load does not have any options to fix the problems, but you can add
> the   --bypass-prop-validation  option to ignore them and just carry the
> problems into your new repository.

Indeed, dump/load is a bit more limited than svnsync regarding
normalization. Also, be careful: both dump/load and svnsync do not
migrate your hook scripts, config files and locks (server side locks
of the svn:needs-lock type).

I've just extended the FAQ entry about dump/load with some of the
known caveats, and explained a procedure to do dump/load with minimal
downtime. See:

http://subversion.apache.org/faq.html#dumpload

-- 
Johan