Hi guys This is just for closure
The main reason was RestrictSUIDSGID=yes, I had to override it for apache to allow chmod fro svn /etc/systemd/system/httpd.service.d/override.conf [Service] ReadWritePaths=/var/repos/svn UMask=0002 RestrictSUIDSGID=no On Tue, 21 Jul 2026 at 09:36, Barry <[email protected]> wrote: > > > On 25 May 2026, at 19:45, William Muriithi <[email protected]> > wrote: > > > Hi Daniel > > If you look at one of your existing working copies (run for example svn >> info from the command line), what is the highest revision number you have >> there? >> > Have the same revision # on the client too > > > william@william-ryzen:~/Documents/application$ svn info > Path: . > Working Copy Root Path: /home/william/Documents/application > URL: https://carbon.example.com/svn/application > Relative URL: ^/ > Repository Root: https://carbon.example.com/svn/application > Repository UUID: 02a4eb3e-e1fa-4ead-9f6a-c97b0409a8a7 > Revision: 269999 > Node Kind: directory > Schedule: normal > Last Changed Author: nathan > Last Changed Rev: 269999 > > > I hit a bug like this with a repo that jammed on 1999. > I used something to force in r2000 and then everything started working > again. > > I suspect I use root and committed a trivial change use file:// url to the > repo. > But it was a while ago and I did not keep notes. > > Barty > > Last Changed Date: 2026-05-11 15:54:29 -0400 (Mon, 11 May 2026) > > william@william-ryzen:~/Documents/application$ > > >> >> I have a feeling that something has caused the 270 directory to >> disappear, thus your repository is now in an inconsistent state. I'm not >> sure if you can recover except restoring a backup. If you can run svnadmin >> dump and then svnadmin load to a new repository, you can probably get a >> working repository with the first 269999 revisions, but you of course loose >> everything later than that. >> > > Possibly, I am open to that being the case. Is there a way to be > confident this is what happened? > > - I have two systems - the backup which was using svnsync is also in the > same state. I suspect having a replica doesn't change much as cron was > scheduled to run every 5 minutes, so if the directory 270 was created and > destroyed within 5 minutes, the replica wouldn't have picked the change > [root@boron ~]# cat /usr/local/bin/svnsync.sh > /usr/bin/svnsync sync file:///var/repos/svn/projects > - The last svndump restore leaves the repo in the same state. Wouldn't > the repo dump have flagged that something is off? Wouldn't the restore the > restore fail when it try to replay the entry 270080 that SVN seems to > assume exit? > - The svnadmin verify pass all the commits up to 269999 > > What else can we do to figure out where the corruption is? > >> >> To answer the question about maximum revision number: The ASF main >> repository (https://svn.apache.org/repos/asf) is currently at revision >> 1934607, so over 7 times your repository. I'm not sure about size but I >> believe it is quite huge. >> >> > Ok, so there exit repo bigger than this then, thanks. > > > Regards, > William > >> Kind regards, >> Daniel >> >> >> >> Den mån 25 maj 2026 kl 19:13 skrev William Muriithi < >> [email protected]>: >> >>> Hi guys and girls >>> >>> >>> Something I missed to mention. Originally, the file contents looked like >>> this:- >>> >>> [root@repos ~]# cat /var/repos/svn/projects/db/current >>> 269999 >>> [root@repos ~]# cat /var/repos/svn/projects/db/txn-current >>> 5sbz >>> >>> These files seem to be important on how SVN makes the decision of which >>> directory to dump/look for a file. So I looked around and google result >>> suggested that the last file may be incorrect. >>> >>> So I changed it to:- >>> >>> [root@repos ~]# cat /var/repos/svn/projects/db/txn-current >>> 5r3j >>> >>> MOVE forward: >>> I then created the files strace point as SVN access points. All below >>> commands were executed as user apache >>> >>> mkdir /var/repos/svn/projects/db/revs/270 >>> cd /var/repos/svn/projects/db/revs/270 >>> touch 270000 270016 270080 270336 >>> svnadmin recover /var/repos/svn/projects/ >>> >>> When I tried committing a new, I got a new error:- >>> >>> svnadmin: E160004: Revision 270336 has a revs file but no revprops file >>> >>> MOVE backward: >>> [root@repos ~]# cat /var/repos/svn/projects/db/current >>> 269999 >>> [root@repos ~]# cat /var/repos/svn/projects/db/txn-current >>> 5r3j >>> >>> svnadmin recover /var/repos/svn/projects/ (As apache user) - It run >>> successfully. >>> >>> Then we went back to SVN client and got the old error >>> >>> Basically, we can't seem to move it back or forward. >>> >>> Regards, >>> William >>> >>> >>> >>> On Mon, 25 May 2026 at 12:24, William Muriithi < >>> [email protected]> wrote: >>> >>>> Hi guys, >>>> >>>> [root@repos ~]# rpm -qa | grep subversion >>>> subversion-libs-1.14.5-3.el10.x86_64_v2 >>>> subversion-1.14.5-3.el10.x86_64_v2 >>>> subversion-tools-1.14.5-3.el10.x86_64_v2 >>>> [root@repos ~]# >>>> [root@repos ~]# >>>> [root@repos ~]# cat /etc/redhat-release >>>> AlmaLinux release 10.1 (Heliotrope Lion) >>>> [root@repos ~]# >>>> [root@repos ~]# rpm -qa | grep svn >>>> mod_dav_svn-1.14.5-3.el10.x86_64_v2 >>>> >>>> >>>> I think I have hit a bug. We have a relatively large SVN repository - >>>> About 2TB uncompressed. 2 weeks ago, someone noticed that we can't commit >>>> any more. When we looked at it, it looked like a permission issue. >>>> >>>> This is the error we are getting from the SVN client. >>>> >>>> svn: E000001: Commit failed (details follow): >>>> svn: E000001: Can't set permissions on >>>> '/var/repos/svn/application/db/revs/270' >>>> william@william-ryzen:~/Documents/application$ >>>> [email protected] >>>> >>>> So we first attempted to "FIX" permissions. Didn't work. Then, when >>>> reading, I noticed opening up the permission is not a good idea as I can >>>> see from strace that svn does try to narrow down the file permissions. So >>>> we decided to restore a fresh to remove all the chmod we had done to the >>>> files and directory. >>>> >>>> This time, when restoring, we created the repo using an apache >>>> account. We also loaded the dump file using the svn account. This removed >>>> all possibility this is a file permission issue. The repo was restored >>>> successfully, BUT the problem remained. >>>> >>>> We looked at the file limit too and to make sure that isn't causing any >>>> issues, we changed it to unlimited for the apache user. The problem >>>> remained. >>>> >>>> It's at this point we attempted to use strace to see what svn is >>>> doing. Oddly, it seemed to be looking for files from directory 270, which >>>> didn't exist at the time. We came to the conclusion that was what was >>>> happening because these came up when one attempts to commit. >>>> >>>> openat(AT_FDCWD, "/var/repos/svn/projects/db/revs/270/270336", >>>> O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) >>>> openat(AT_FDCWD, "/var/repos/svn/projects/db/revs/270/270336", >>>> O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) >>>> openat(AT_FDCWD, "/var/repos/svn/projects/db/revs/270/270080", >>>> O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) >>>> openat(AT_FDCWD, "/var/repos/svn/projects/db/revs/270/270080", >>>> O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) >>>> openat(AT_FDCWD, "/var/repos/svn/projects/db/revs/270/270016", >>>> O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) >>>> openat(AT_FDCWD, "/var/repos/svn/projects/db/revs/270/270016", >>>> O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) >>>> openat(AT_FDCWD, "/var/repos/svn/projects/db/revs/270/270000", >>>> O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) >>>> openat(AT_FDCWD, "/var/repos/svn/projects/db/revs/270/270000", >>>> O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) >>>> >>>> Note, the directory 270 didn't exist. According to google, this can >>>> happen due to either a hook or CI committing too often creating >>>> corruption. We can't seem to clear the corruption though. We are also a >>>> bit skeptical of its corruption as it should have cleared on a freshly >>>> restored repository. >>>> >>>> >>>> So over the weekend, we attempted to separate the projects on the repo >>>> to individual repos. This is to reduce the size of the repo as it could >>>> have been causing the issue. We however kept the revision numbers as they >>>> are in documentations and tags. However, on the now small repo, with only >>>> 187 GB in size, but with the last successful commit # 269999, we still >>>> see the problem. >>>> >>>> This now looks more like a software bug than a setup issue. We have >>>> "yum updated" the system in an attempt to make sure we have all the bug >>>> fixes out there but nothing works. >>>> >>>> Would anyone know if there is a transaction limit in the current SVN >>>> code? Anyone have a repository with more than 270000 revisions out there >>>> please? Any pointers to something we may have overlooked so far? >>>> Something else, the svndump is cleanly at 2 TB, any chance there may be a >>>> limit of how big a repo glow?? This was our first thought but since we >>>> have created smaller repos and the issue remains, it now looks far more >>>> like a sharding logic error to us. >>>> >>>> >>>> Regards, >>>> William >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>>
