Re: Adjust revision in new repository
On Mon, Sep 4, 2023 at 8:36 AM Magnus Lyrberg wrote: > > On Fri, Sep 1, 2023 at 3:15 PM Johan Corveleyn wrote: >> >> On Fri, Sep 1, 2023 at 2:47 PM Magnus Lyrberg >> wrote: >> > Thank you. This is very similar to our current solution. It would >> > however be nice to avoid a lot of empty commits, hence my >> > engagement in this list asking for alternative solutions. >> > >> > But perhaps there are none. >> >> I'm not sure, but it sounds like that would be quite a hack, and I >> don't think it will be possible. >> >> The repository still has to give a sane reply if a user asks for "svn >> update -r 2" or "svn ls https://server/svn@1;. If those revisions >> really don't exist, what should the server answer? So I don't think >> you can avoid creating those revisions, but you can leave them empty, >> as suggested. >> >> What is the problem in having those empty revisions anyway? I assume >> they hardly take up any diskspace. If that's the only price you have >> to pay for having this "old cruft removed but still original >> rev-numbers repository", it sounds like a good deal to me (and it's >> still a correctly working repository that behaves as designed). > > > Theoretically you could get the same answer from the server as when you > ask for a revision that does not yet exist. I see your point though. > > There is some concern that having 18 empty commits might impact > the performance of the repository. I assume each commit still has an entry > in the database etc, even if it does not use a lot of disk. Then it's time to revisit your basic architecture to point to a new repo with a wildly reduced history, and *use tags*. Subversion was written to preserve history and treat it as immutable. Replacing 10,000 commits with empty commits just to preserve the numbering for a later commit is. very much against its design goals, and likely to be much more expensive than switching to tag bsed references ASAP. I sympathize with the desire to take what seems like a simple path, but it sounds like you have to switch to a new repo *anyway* Take advantage of the opportunity to strip it for performance. And oh, the fastest way to strip is probably to pull the repo into a git clone, discard unwelcome branches, run "git gc ---aggressive" and push it back to a new repo.. Subversion was not written to discard unused history easily, git does a fair job and does pretty well publishing back to another subversion repo.
Re: Adjust revision in new repository
On Thu, Aug 31, 2023 at 8:21 AM Magnus Lyrberg wrote: > > Hello. > > I'm not subscribed so please CC me in any response. > > We have, most unfortunately and for historical reasons, > a very hard dependency on the revision number in subversion. > > Our current repository has become large and unwieldy, and we > want to move to a fresh repository. The history is not very important, > we are ok to start from scratch, but loosing the revision number > would be incredibly costly. Stop. Do not pass go, do not collect $200. "Incredibly costly" is unlikely to be as expensive as supporting a jury-rig like this. Go back and rewrite your base software to use tags If you can't rewrite it, make a copy of the repo somewhere else to play with it,. And slap the wrist of the silly person who originally decided to use revision numbers. And that said, you may actually be able to strip down the repo by a large factor and set a tag to match that old revision number, skipping right past the size issue.
Re: Adjust revision in new repository
On Mon, Sep 4, 2023 at 8:36 AM Magnus Lyrberg wrote: > There is some concern that having 18 empty commits might impact > the performance of the repository. I assume each commit still has an entry > in the database etc, even if it does not use a lot of disk. Are you doing this for performance? I doubt you will see any improvements even if you did not care about the revision numbers and just created a new repo with a handful of revisions. SVN performance is not impacted by the number of revisions with the only exception being the tools you use to backup and restore the repository. If anything, having a single revision with a lot of changes in it hurts performance of some commands. For example svn log -v is really harmed by having a single revision with a lot of changed paths. Mark
Re: Adjust revision in new repository
On Fri, Sep 1, 2023 at 3:15 PM Johan Corveleyn wrote: > On Fri, Sep 1, 2023 at 2:47 PM Magnus Lyrberg > wrote: > > Thank you. This is very similar to our current solution. It would > > however be nice to avoid a lot of empty commits, hence my > > engagement in this list asking for alternative solutions. > > > > But perhaps there are none. > > I'm not sure, but it sounds like that would be quite a hack, and I > don't think it will be possible. > > The repository still has to give a sane reply if a user asks for "svn > update -r 2" or "svn ls https://server/svn@1;. If those revisions > really don't exist, what should the server answer? So I don't think > you can avoid creating those revisions, but you can leave them empty, > as suggested. > > What is the problem in having those empty revisions anyway? I assume > they hardly take up any diskspace. If that's the only price you have > to pay for having this "old cruft removed but still original > rev-numbers repository", it sounds like a good deal to me (and it's > still a correctly working repository that behaves as designed). Theoretically you could get the same answer from the server as when you ask for a revision that does not yet exist. I see your point though. There is some concern that having 18 empty commits might impact the performance of the repository. I assume each commit still has an entry in the database etc, even if it does not use a lot of disk. Best regards Magnus Lyrberg
Re: Adjust revision in new repository
On Fri, Sep 1, 2023 at 2:47 PM Magnus Lyrberg
wrote:
>
> On Fri, Sep 1, 2023 at 2:05 PM Yasuhito FUTATSUKI
> wrote:
> >
> > Although it makes a bunch of dummy empty commit, a dump file contains
> > empty revisions can increase youngest revision number keeping content
> > of the tree empty.
> >
> > e.g. A shell script to make a repositoy contains
> > 10 empty revisions:
> > [[[
> > #!/bin/sh
> >
> > repos_uuid='c68ffc52-e6c1-4053-9b3b-6e67a0e223f2'
> > rev0_date='2023-09-01T10:22:50.036256Z' # This should be just 27 letters.
> > number_of_commits=10
> >
> > produce_dump()
> > {
> > cat < > SVN-fs-dump-format-version: 2
> >
> > UUID: ${repos_uuid}
> >
> > Revision-number: 0
> > Prop-content-length: 56
> > Content-length: 56
> >
> > K 8
> > svn:date
> > V 27
> > ${rev0_date}
> > PROPS-END
> >
> > DUMP_HEAD_END
> >
> > rev=0
> > while [ $rev -lt $1 ]; do
> > rev=$(($rev+1))
> > cat < > Revision-number: $rev
> > Prop-content-length: 10
> > Content-length: 10
> >
> > PROPS-END
> >
> > REV_END
> > done
> > }
> >
> > svnadmin create new_repo
> > produce_dump "${number_of_commits}" | svnadmin load -q new_repo
> > svnadmin info new_repo
> > ]]]
>
> Thank you. This is very similar to our current solution. It would
> however be nice to avoid a lot of empty commits, hence my
> engagement in this list asking for alternative solutions.
>
> But perhaps there are none.
I'm not sure, but it sounds like that would be quite a hack, and I
don't think it will be possible.
The repository still has to give a sane reply if a user asks for "svn
update -r 2" or "svn ls https://server/svn@1;. If those revisions
really don't exist, what should the server answer? So I don't think
you can avoid creating those revisions, but you can leave them empty,
as suggested.
What is the problem in having those empty revisions anyway? I assume
they hardly take up any diskspace. If that's the only price you have
to pay for having this "old cruft removed but still original
rev-numbers repository", it sounds like a good deal to me (and it's
still a correctly working repository that behaves as designed).
--
Johan
Re: Adjust revision in new repository
On Fri, Sep 1, 2023 at 2:05 PM Yasuhito FUTATSUKI
wrote:
>
> Although it makes a bunch of dummy empty commit, a dump file contains
> empty revisions can increase youngest revision number keeping content
> of the tree empty.
>
> e.g. A shell script to make a repositoy contains
> 10 empty revisions:
> [[[
> #!/bin/sh
>
> repos_uuid='c68ffc52-e6c1-4053-9b3b-6e67a0e223f2'
> rev0_date='2023-09-01T10:22:50.036256Z' # This should be just 27 letters.
> number_of_commits=10
>
> produce_dump()
> {
> cat < SVN-fs-dump-format-version: 2
>
> UUID: ${repos_uuid}
>
> Revision-number: 0
> Prop-content-length: 56
> Content-length: 56
>
> K 8
> svn:date
> V 27
> ${rev0_date}
> PROPS-END
>
> DUMP_HEAD_END
>
> rev=0
> while [ $rev -lt $1 ]; do
> rev=$(($rev+1))
> cat < Revision-number: $rev
> Prop-content-length: 10
> Content-length: 10
>
> PROPS-END
>
> REV_END
> done
> }
>
> svnadmin create new_repo
> produce_dump "${number_of_commits}" | svnadmin load -q new_repo
> svnadmin info new_repo
> ]]]
Thank you. This is very similar to our current solution. It would
however be nice to avoid a lot of empty commits, hence my
engagement in this list asking for alternative solutions.
But perhaps there are none.
Best regards
Magnus Lyrberg
Re: Adjust revision in new repository
On Fri, Sep 1, 2023 at 2:00 PM Johan Corveleyn wrote: > On Fri, Sep 1, 2023 at 11:05 AM Magnus Lyrberg > wrote: > > On Thu, Aug 31, 2023 at 4:51 PM Johan Corveleyn > wrote: > > > On Thu, Aug 31, 2023 at 2:21 PM Magnus Lyrberg > > > > Is there some other way besides making a bunch of dummy commits > > > > to reach our current revision number in a new repository? > > > > > > Maybe you can give svndumpfilter [1] a try to filter out the "cruft" > > > (after creating a dump file from the original repository). > > > Unfortunately 'svnadmin load' ignores the revision information in the > dump > > file. If the new repository is empty it will start with revision 1 > > regardless of > > what the dump file says. > > > > I made a quick test to verify this: > > > > svnadmin load /var/svn/repo2 < partial_dump.dump > > <<< Started new transaction, based on original revision 7 > > * adding path : test.txt ... done. > > > > --- Committed new rev 1 (loaded from original rev 7) >>> > > But what is partial_dump.dump? Can you show us the exact commands you > used to create it? > > Did you create this by dumping only revisions 7 and 8 from the > original repository, as in 'svnadmin dump /path/to/orig_repos -r7:8 > > partial_dump.dump'? In that case, it is expected to behave like you > described. > > Or did you create it by dumping the entire repository, and then > svndumpfilter-ing out the unneeded stuff, as in 'svnadmin dump > /path/to/orig_repos | svndumpfilter exclude /path/to/unneededstuff > > partial_dump.dump'? > > It is the last technique that I was suggesting, and I would expect > that to retain all the existing revision numbers (some revisions might > become empty revs because of the filtering, but they should still > "count"). > You are completely correct that the partial_dump.dump contained only revision 7 and 8, and that a filtered dump like you suggest will result in the correct revision number in the new repository. This does however also result in a number of empty revisions (dummy commits as I called them in the initial mail), which is what we want to avoid. What we would like to achieve is to have the current HEAD, without the history, in a new repository while still maintaining the revision number. Preferably without (quite a lot of) empty commits. Best regards Magnus Lyrberg
Re: Adjust revision in new repository
Hello,
On 2023/09/01 18:04, Magnus Lyrberg wrote:
Unfortunately 'svnadmin load' ignores the revision information in the dump
file. If the new repository is empty it will start with revision 1
regardless of
what the dump file says.
I made a quick test to verify this:
svnadmin load /var/svn/repo2 < partial_dump.dump
<<< Started new transaction, based on original revision 7
* adding path : test.txt ... done.
--- Committed new rev 1 (loaded from original rev 7) >>>
<<< Started new transaction, based on original revision 8
* editing path : test.txt ... done.
--- Committed new rev 2 (loaded from original rev 8) >>>
I want to achieve the new repository ending up at revision 8 (or even slightly
above 8), where my old repository was. But as not all revisions are in the
dump file it ended up on revision 2.
Although it makes a bunch of dummy empty commit, a dump file contains
empty revisions can increase youngest revision number keeping content
of the tree empty.
e.g. A shell script to make a repositoy contains
10 empty revisions:
[[[
#!/bin/sh
repos_uuid='c68ffc52-e6c1-4053-9b3b-6e67a0e223f2'
rev0_date='2023-09-01T10:22:50.036256Z' # This should be just 27 letters.
number_of_commits=10
produce_dump()
{
cat <
Re: Adjust revision in new repository
On Fri, Sep 1, 2023 at 11:05 AM Magnus Lyrberg wrote: > On Thu, Aug 31, 2023 at 4:51 PM Johan Corveleyn wrote: > > On Thu, Aug 31, 2023 at 2:21 PM Magnus Lyrberg > > wrote: > > > > > > Hello. > > > > > > I'm not subscribed so please CC me in any response. > > > > > > Is there some other way besides making a bunch of dummy commits > > > to reach our current revision number in a new repository? > > > > Maybe you can give svndumpfilter [1] a try to filter out the "cruft" > > (after creating a dump file from the original repository). It has > > options like: > > > > --drop-empty-revs > > If the current filtering invocation causes any revision to be > > empty (i.e., the revision causes no change to the repository), removes > > these revisions from the final dump file. > > > > --renumber-revs > > Renumbers revisions that remain after filtering. > > > > which seems to indicate that by default (not using these options) it > > will keep the original revision numbering (even if you filter out > > entire revisions so they become empty). > > > > You'll have to create a dump file first, by using 'svnadmin dump' if > > you have file access to the repo, or 'svnrdump dump' if you only have > > remote access. Then svndumpfilter it, and 'svnadmin load' it into a > > new repository. Maybe first try it with a small test sample to see if > > it works as expected. > > Unfortunately 'svnadmin load' ignores the revision information in the dump > file. If the new repository is empty it will start with revision 1 > regardless of > what the dump file says. > > I made a quick test to verify this: > > svnadmin load /var/svn/repo2 < partial_dump.dump > <<< Started new transaction, based on original revision 7 > * adding path : test.txt ... done. > > --- Committed new rev 1 (loaded from original rev 7) >>> But what is partial_dump.dump? Can you show us the exact commands you used to create it? Did you create this by dumping only revisions 7 and 8 from the original repository, as in 'svnadmin dump /path/to/orig_repos -r7:8 > partial_dump.dump'? In that case, it is expected to behave like you described. Or did you create it by dumping the entire repository, and then svndumpfilter-ing out the unneeded stuff, as in 'svnadmin dump /path/to/orig_repos | svndumpfilter exclude /path/to/unneededstuff > partial_dump.dump'? It is the last technique that I was suggesting, and I would expect that to retain all the existing revision numbers (some revisions might become empty revs because of the filtering, but they should still "count"). -- Johan
Re: Adjust revision in new repository
On Thu, Aug 31, 2023 at 4:51 PM Johan Corveleyn wrote: > On Thu, Aug 31, 2023 at 2:21 PM Magnus Lyrberg > wrote: > > > > Hello. > > > > I'm not subscribed so please CC me in any response. > > > > Is there some other way besides making a bunch of dummy commits > > to reach our current revision number in a new repository? > > Maybe you can give svndumpfilter [1] a try to filter out the "cruft" > (after creating a dump file from the original repository). It has > options like: > > --drop-empty-revs > If the current filtering invocation causes any revision to be > empty (i.e., the revision causes no change to the repository), removes > these revisions from the final dump file. > > --renumber-revs > Renumbers revisions that remain after filtering. > > which seems to indicate that by default (not using these options) it > will keep the original revision numbering (even if you filter out > entire revisions so they become empty). > > You'll have to create a dump file first, by using 'svnadmin dump' if > you have file access to the repo, or 'svnrdump dump' if you only have > remote access. Then svndumpfilter it, and 'svnadmin load' it into a > new repository. Maybe first try it with a small test sample to see if > it works as expected. Unfortunately 'svnadmin load' ignores the revision information in the dump file. If the new repository is empty it will start with revision 1 regardless of what the dump file says. I made a quick test to verify this: svnadmin load /var/svn/repo2 < partial_dump.dump <<< Started new transaction, based on original revision 7 * adding path : test.txt ... done. --- Committed new rev 1 (loaded from original rev 7) >>> <<< Started new transaction, based on original revision 8 * editing path : test.txt ... done. --- Committed new rev 2 (loaded from original rev 8) >>> I want to achieve the new repository ending up at revision 8 (or even slightly above 8), where my old repository was. But as not all revisions are in the dump file it ended up on revision 2. Best regards Magnus Lyrberg
Re: Adjust revision in new repository
On Thu, Aug 31, 2023 at 2:21 PM Magnus Lyrberg wrote: > > Hello. > > I'm not subscribed so please CC me in any response. > > We have, most unfortunately and for historical reasons, > a very hard dependency on the revision number in subversion. > > Our current repository has become large and unwieldy, and we > want to move to a fresh repository. The history is not very important, > we are ok to start from scratch, but loosing the revision number > would be incredibly costly. > > Is there some other way besides making a bunch of dummy commits > to reach our current revision number in a new repository? Maybe you can give svndumpfilter [1] a try to filter out the "cruft" (after creating a dump file from the original repository). It has options like: --drop-empty-revs If the current filtering invocation causes any revision to be empty (i.e., the revision causes no change to the repository), removes these revisions from the final dump file. --renumber-revs Renumbers revisions that remain after filtering. which seems to indicate that by default (not using these options) it will keep the original revision numbering (even if you filter out entire revisions so they become empty). You'll have to create a dump file first, by using 'svnadmin dump' if you have file access to the repo, or 'svnrdump dump' if you only have remote access. Then svndumpfilter it, and 'svnadmin load' it into a new repository. Maybe first try it with a small test sample to see if it works as expected. [1] https://svnbook.red-bean.com/nightly/en/svn.ref.svndumpfilter.html -- Johan
