Re: Decompression of svndiff data failed
Your history is toast. Stop burning cycles. Lock your working server, Take clean working copies of the important branches if you can get them, the clean working copies, import them to a new server, and start with a much lighter working repository. And *stop* putting binaries in the same repository as source code. History is usually much more important for source, binaries harshly burden all source control systems, and even the smallest errors corrupt them irretrievably. If you *must* store binaries, be sure to commit them in entirely distinct commits from the source code, to avoid precisely this problem, Sent from my iPhone
Re: Decompression of svndiff data failed
The decompression errors only seem to happen when we're sending binary data. For a couple of years our marketing team were storing all of their files in subversion and this seems to be the vast majority of the revisions I'm having to fix with fsfsverify.py. So it could possibly be that they were using a TortoiseSVN version that was built with a buggy library. That said the problematic revision that this thread is about was one created by our engineering team and all of our work is done from linux machines so would have been a different client (cli subversion client). So it sounds more likely that it's something server side. We do still get the issue periodically, so there's a chance it could be related to the NAS that the repo is stored on. We did recently move to a new NAS as the old one was getting a bit slow but we hadn't seen any corruption issues with the old NAS, just slowness. I'm not sure if we've had any svndiff issues since we've moved to the new NAS, I'll find out when I've finally got to the point where it's syncing commits from the last month. Also the issue would happen straight away. i.e. if someone tried to do an svn up immediately after a commit had been made they would get the svndiff error. So it seems like it was an error that occurred when the svnserve process received it or wrote it to the file system. So it's not hard disk corruption after the fact (but again doesn't rule out a NAS issue). Sorry for not being able to provide more specific details. If I see it happening again once we've moved over to a modern OS and Subversion server I'll email back again. Mike On Fri, 30 Aug 2019 at 14:22, Branko Čibej wrote: > On 30.08.2019 15:14, Michael Ditum wrote: > > Hi Brane, > > > > Thanks for the reply. Interestingly Daniel's reply had given me the > > idea to try pretty much what you suggested and I gave it a go this > > morning and it seems to be working. > > > > Stopping svnsync in the right place wasn't hard as i dies as soon as > > it tried to get the binary diff but before it's made any changes. > > > > The one bit I didn't do was update the svnsync metadata. When I > > resumed the sync it just automatically carried on with the revision > > after the one I had just committed. Hopefully that won't cause any > > problems, it seems to be working ok as I'm a lot further syncing than > > I've ever managed before (crazy how many times we've had svndiff > > errors, luckily fsfsverify has fixed all of the others so far). > > > > Thanks for everyone's help! > > Great that it works. :) > > I'm curious though ... have you any idea what caused the decompression > errors? The message you posted came from zlib -- not Subversion's code > -- and that has been very, very stable literally for decades. > > Is it possible that you just had the bad luck to have a broken version > of zlib, way back in the dawn of time? If it had been a problem with the > storage, I'm pretty sure you'd have noticed other issues, too. > > -- Brane > >
Re: Decompression of svndiff data failed
On 30.08.2019 15:14, Michael Ditum wrote: > Hi Brane, > > Thanks for the reply. Interestingly Daniel's reply had given me the > idea to try pretty much what you suggested and I gave it a go this > morning and it seems to be working. > > Stopping svnsync in the right place wasn't hard as i dies as soon as > it tried to get the binary diff but before it's made any changes. > > The one bit I didn't do was update the svnsync metadata. When I > resumed the sync it just automatically carried on with the revision > after the one I had just committed. Hopefully that won't cause any > problems, it seems to be working ok as I'm a lot further syncing than > I've ever managed before (crazy how many times we've had svndiff > errors, luckily fsfsverify has fixed all of the others so far). > > Thanks for everyone's help! Great that it works. :) I'm curious though ... have you any idea what caused the decompression errors? The message you posted came from zlib -- not Subversion's code -- and that has been very, very stable literally for decades. Is it possible that you just had the bad luck to have a broken version of zlib, way back in the dawn of time? If it had been a problem with the storage, I'm pretty sure you'd have noticed other issues, too. -- Brane
Re: Pre-commit hook assistance
On 30.08.2019 15:01, William Muriithi wrote: > Morning, > > I am attempting to write a pre-commit hook that enforces a policy to > some group and warns the rest of the team. The hook is in python and > it just check that jira ticket exist on the commit log. > > The problem I am facing is I am not able to both warn and error for > some reason. If I exit, the failure reason is displayed to the user. > > If I don't exit, but attempts to send a warning to the client, > nothing show up even in the logs. > > I am using sys.stderr.write and sys.stdout.write for error and warning > respectively. Is it technically possible to do this? Would be grateful > for your help on this. This is clearly documented in the pre-commit hook template: # If the hook program exits with success, the txn is committed; but # if it exits with failure (non-zero), the txn is aborted, no commit # takes place, and STDERR is returned to the client. So, no, you can't send "warnings" by writing to stdout, and stderr is only piped to the client if the hook fails. For warnings, you'll have to use some other channel. -- Brane
Re: Decompression of svndiff data failed
Hi Brane, Thanks for the reply. Interestingly Daniel's reply had given me the idea to try pretty much what you suggested and I gave it a go this morning and it seems to be working. Stopping svnsync in the right place wasn't hard as i dies as soon as it tried to get the binary diff but before it's made any changes. The one bit I didn't do was update the svnsync metadata. When I resumed the sync it just automatically carried on with the revision after the one I had just committed. Hopefully that won't cause any problems, it seems to be working ok as I'm a lot further syncing than I've ever managed before (crazy how many times we've had svndiff errors, luckily fsfsverify has fixed all of the others so far). Thanks for everyone's help! Mike On Fri, 30 Aug 2019 at 12:12, Branko Čibej wrote: > On 29.08.2019 20:49, Michael Ditum wrote: > > Apart from using fsfsverify I also tried recreating the diff by > > creating a Fedora 7 VM, running svnsync on it to copy the repo up to > > that point and then manually committing the file and copying the > > revision over to the copy of my original repo. > > Yikes. No, that definitely won't work. > > > Whilst this allows svnsync to get past that revision I then started > > having lots of problems with incorrect byte offsets in later revisions > > and once I (think I correctly) fixed started getting checksum errors. > > And that's why ... binary deltas rely on previously stored data, but > unlike a text diff they have no context. You changed the source of the > delta and that corrupted everything that depends on it in later revisions. > > > > Does anyone have any ideas on how I can fix this revision? As I > > mentioned before, the file gets deleted a couple of revisions later so > > I don't really care about the contents of the revision but I'm > > currently stuck and can't get any further in my svnsync. > > Daniel made the best suggestion, it would work like this: > > * create a new repository > * svnsync up to the revision just before the broken one (stopping > svnsync is the tricky part here) > * commit that one file to the _synced_ repository, and update > svnsync's metadata (in revision properties on r0) to skip the > offending revision on the next run > * svnsync to the end. > > You can do a similar trick with svnadmin dump and (incremental) load; > the benefit of this is that there is no "stopping problem," but it will > be much, much slower than svnsync. You _could_ combine both methods, > i.e., initialize your target repo with a partial dump of the source up > to your offending rX, then commit rX to the target repo, then svnsync > from there. > > -- Brane > >
Pre-commit hook assistance
Morning, I am attempting to write a pre-commit hook that enforces a policy to some group and warns the rest of the team. The hook is in python and it just check that jira ticket exist on the commit log. The problem I am facing is I am not able to both warn and error for some reason. If I exit, the failure reason is displayed to the user. If I don't exit, but attempts to send a warning to the client, nothing show up even in the logs. I am using sys.stderr.write and sys.stdout.write for error and warning respectively. Is it technically possible to do this? Would be grateful for your help on this. Regards, William
Re: Decompression of svndiff data failed
On 29.08.2019 20:49, Michael Ditum wrote: > Apart from using fsfsverify I also tried recreating the diff by > creating a Fedora 7 VM, running svnsync on it to copy the repo up to > that point and then manually committing the file and copying the > revision over to the copy of my original repo. Yikes. No, that definitely won't work. > Whilst this allows svnsync to get past that revision I then started > having lots of problems with incorrect byte offsets in later revisions > and once I (think I correctly) fixed started getting checksum errors. And that's why ... binary deltas rely on previously stored data, but unlike a text diff they have no context. You changed the source of the delta and that corrupted everything that depends on it in later revisions. > Does anyone have any ideas on how I can fix this revision? As I > mentioned before, the file gets deleted a couple of revisions later so > I don't really care about the contents of the revision but I'm > currently stuck and can't get any further in my svnsync. Daniel made the best suggestion, it would work like this: * create a new repository * svnsync up to the revision just before the broken one (stopping svnsync is the tricky part here) * commit that one file to the _synced_ repository, and update svnsync's metadata (in revision properties on r0) to skip the offending revision on the next run * svnsync to the end. You can do a similar trick with svnadmin dump and (incremental) load; the benefit of this is that there is no "stopping problem," but it will be much, much slower than svnsync. You _could_ combine both methods, i.e., initialize your target repo with a partial dump of the source up to your offending rX, then commit rX to the target repo, then svnsync from there. -- Brane
Re: svn x-shelve gives E200014: Checksum mismatch
On 2019-08-30 10:18, Johan Corveleyn wrote: > On Fri, Aug 30, 2019 at 9:12 AM Mark P. wrote: >> >> On 2019-08-30 06:11, Johan Corveleyn wrote: >>> On Wed, Jul 31, 2019 at 7:20 AM Mark P. wrote: svn x-shelve shelf1 I get the following output: Updating '.svn\experimental\shelves\v3\7368656c6631-001.wc': At revision 1. Sendingmain.cpp Transmitting file data .svn: E200014: Checksum mismatch for 'C:\Shared\Test\Test3\.svn\experimental\shelves\v3\7368656c6631-001.wc\main.cpp': expected: b77dd03a6c3f68f496d874153363694e actual: 7f081249d3e4e1383afb2798d46a58f5 >>> >>> Hi Mark, >>> >>> Sorry for the late response. >>> This looks to me like a bug with the experimental shelving feature in 1.12. >>> >>> Is the file main.cpp "translated" into Windows end-of-line style in >>> the Windows working copy (vs. Unix eol-style on the FreeBSD machine), >>> i.e. does it carry the svn property "svn:eol-styl=native"? Perhaps >>> there is a bug when shelving, in that one side is eol-translated (and >>> checksummed) and the other is not. >>> >> >> Hi Johan, >> >> Thanks for the reply. Yes, you're correct, svn:eol-style was set to >> 'native', and I think this is where the bug lies. >> >> I removed the property, ran: >> >> svn x-shelve shelf2 >> >> and it seems to work as expected now, although ideally shelve should be >> able to work with eol-style=native. >> >> Mark > > [ Re-adding the users@s.a.o list back to cc. Please use reply all to > keep the list in the loop. ] Sorry about that. > > Okay, thanks for checking this, and thanks for reporting this in the > first place. > So indeed this seems like a bug in x-shelve when working with > eol-style=native (and shelving it in a Windows working copy). > > Could you please file an issue for that in our issue tracker? > https://issues.apache.org/jira/browse/SVN Thanks for your attention. I've created the issue here: https://issues.apache.org/jira/browse/SVN-4827 > > If you would like to dig in further (going through the source code, > reading our community guide, possibly writing a patch etc. -- see > [1]), you're very welcome to do so. (no problem if you don't, your > report of the issue is already very useful) > > Thanks, > Disclaimer: http://www2.peralex.com/disclaimer.html
Re: svn x-shelve gives E200014: Checksum mismatch
On Fri, Aug 30, 2019 at 9:12 AM Mark P. wrote: > > On 2019-08-30 06:11, Johan Corveleyn wrote: > > On Wed, Jul 31, 2019 at 7:20 AM Mark P. wrote: > >> > >> > >> > >> svn x-shelve shelf1 > >> > >> I get the following output: > >> > >> Updating '.svn\experimental\shelves\v3\7368656c6631-001.wc': > >> At revision 1. > >> Sendingmain.cpp > >> Transmitting file data .svn: E200014: Checksum mismatch for > >> 'C:\Shared\Test\Test3\.svn\experimental\shelves\v3\7368656c6631-001.wc\main.cpp': > >>expected: b77dd03a6c3f68f496d874153363694e > >> actual: 7f081249d3e4e1383afb2798d46a58f5 > >> > > > > Hi Mark, > > > > Sorry for the late response. > > This looks to me like a bug with the experimental shelving feature in 1.12. > > > > Is the file main.cpp "translated" into Windows end-of-line style in > > the Windows working copy (vs. Unix eol-style on the FreeBSD machine), > > i.e. does it carry the svn property "svn:eol-styl=native"? Perhaps > > there is a bug when shelving, in that one side is eol-translated (and > > checksummed) and the other is not. > > > > Hi Johan, > > Thanks for the reply. Yes, you're correct, svn:eol-style was set to > 'native', and I think this is where the bug lies. > > I removed the property, ran: > > svn x-shelve shelf2 > > and it seems to work as expected now, although ideally shelve should be > able to work with eol-style=native. > > Mark [ Re-adding the users@s.a.o list back to cc. Please use reply all to keep the list in the loop. ] Okay, thanks for checking this, and thanks for reporting this in the first place. So indeed this seems like a bug in x-shelve when working with eol-style=native (and shelving it in a Windows working copy). Could you please file an issue for that in our issue tracker? https://issues.apache.org/jira/browse/SVN If you would like to dig in further (going through the source code, reading our community guide, possibly writing a patch etc. -- see [1]), you're very welcome to do so. (no problem if you don't, your report of the issue is already very useful) Thanks, -- Johan [1] http://subversion.apache.org/contributing.html
