Re: Sysadmin Load Reduction: Subversion Infrastructure
Hi all, This has been quite the long thread for one that was started only around 2 days ago, so thought I would summarise things: On the topic of removal of the Subversion mirrors, there have been no objections, so we will proceed with this. With regards to removal of access for everyone but translators, there have been two objections noted to this. One concerned compliance with the Manifesto, whilst the other was concerned with creating obstacles. Given that restricting access to the websites is still compliant with the Manifesto despite the lack of a clear clause permitting this I don't believe that not granting access by default to Subversion should be an issue, so would like to proceed with this as well. That leaves the last point, which has generated the most contention, the removal of WebSVN (websvn.kde.org). Based on the responses, it would appear that translation workflows are currently heavily embedded with and tightly dependent on this service. This would indicate that we would need to keep this service around for now. With just a few websites left on Subversion though, I would like to know if the translation teams have a plan for how they will continue to do things in the long term, and whether this includes a migration away from Subversion (as otherwise we are maintaining indefinitely a Subversion repository, customised hooks, mirroring infrastructure, WebSVN, and support for the email notifications from those customised hooks within other systems such as our IRC Bots and Bugzilla) Regards, Ben
Re: Sysadmin Load Reduction: Subversion Infrastructure
On Mon, Nov 11, 2019 at 3:58 AM Luigi Toscano wrote: > > Ben Cooksley ha scritto: > > On Sun, Nov 10, 2019 at 11:19 AM Albert Astals Cid wrote: > >> > >> El dissabte, 9 de novembre de 2019, a les 21:22:16 CET, Ben Cooksley va > >> escriure: > >>> On Sun, Nov 10, 2019 at 8:37 AM Albert Astals Cid wrote: > > El dissabte, 9 de novembre de 2019, a les 19:27:07 CET, Ben Cooksley va > escriure: > > On Sun, Nov 10, 2019 at 7:18 AM Alexander Potashev > > wrote: > >> > >> сб, 9 нояб. 2019 г. в 03:20, Ben Cooksley : > >>> This would include the shutdown of WebSVN in particular, which when > >>> coupled with the shutdown of our two CGit instances would also allow > >>> for us to eliminate an entire virtual machine from our systems. > >> > >> Will there be any web interface for SVN after shutdown of WebSVN? > >> > >> Can we assume https://phabricator.kde.org/source/svn/ remains > >> available during the next 10 years? > >> > > > > Phabricator's browser will be retired as part of the shutdown of > > Phabricator, which will take place once Gitlab has assumed > > responsibility for code hosting and review, and the tasks have been > > migrated from Phabricator. > > > > Should WebSVN be shutdown as well, then there would be no web > > interface to our SVN repository. > > That's not acceptable. > >>> > >>> Mind explaining why? > >> > >> Because we use it in l10n.kde.org to link to po files. > > > > Mind detailing what those links are used for? > > > >> > >>> Bear in mind that there is a cost both in terms of infrastructure, and > >>> people time to maintain a service such as WebSVN. > >> > >> We have money, we don't have to shut down things we use because there is a > >> cost. > > > > I wasn't referring to monetary cost there, I was referring to the flow > > on effects (such as having to maintain the necessary components on the > > master server to allow for the Subversion repository to be mirrored). > > > > Note also the "people time" component there. > > Sure, but please see my previous questions: > - can we extend the space of rosetta? It already has a partial checkout, and > 100 GB of free space (which can be kept down We would need to ask the system hosters to provide this, as Rosetta is a donated system (if memory serves, Rosetta is currently IOPS constrainted, so using it to host WebSVN may over-burden the system) > - if that's not enough, can we simply setup a machine which periodically sync > from the svn repository? You are probably going to tell me that it does not > work without server support, but from what I'm reading about svnsync, I don't > think it should overload the server if executed every 30 minutes. > Are we sure that we still need something on the master server? Let's try it > first. I'm afraid you cannot use svnsync with KDE's Subversion repository, that utility hasn't been able to handle it for many, many years now (it crashes if memory serves - [ade] hit this and documented it on his blog back around the time it broke which was before the switch to Git got underway) We have custom tooling which uses rsync instead to mirror the repository. Custom tooling is necessary because plain rsync cannot be used to reliably mirror the repository - because it has 1.5 million commits in it, which means over 3 million inodes on disk. Each of these would have to be checked by plain usage of rsync, which takes a substantial amount of time even on NVMe SSD storage - during which time the local mirror of the repository may be inconsistent (rendering it unusable), and which also generates a matching disk load on the master server, reducing it's performance. The RSync endpoint for the mirrors to contact is the component on the master server that has to be maintained. > - in case it's not clear, I'm volunteering to maintain that system. > > Ciao > -- > Luigi Cheers, Ben
Re: Sysadmin Load Reduction: Subversion Infrastructure
svētd., 2019. g. 10. nov., plkst. 04:09 — lietotājs Alexander Potashev
() rakstīja:
>
> вс, 10 нояб. 2019 г. в 04:10, Ben Cooksley :
> >
> > On Sun, Nov 10, 2019 at 11:19 AM Albert Astals Cid wrote:
> > >
> > > El dissabte, 9 de novembre de 2019, a les 21:22:16 CET, Ben Cooksley va
> > > escriure:
> > > Because we use it in l10n.kde.org to link to po files.
> >
> > Mind detailing what those links are used for?
>
> A common workflow suggested for translators without SVN account
> (and/or those who find it hard to use SVN) is as folllows:
>
> 1. Go to https://l10n.kde.org/stats/gui/trunk-kf5/team/ru/ ("ru" for
> Russian) and pick a .po translation file that needs update - e.g.
> incomplete translation or one containing mistakes.
>
> 2. Click on the .po file at l10n.kde.org to downloaded. WebSVN is
> used for this.
>
> 3. Edit the downloaded .po file, send for review, etc.
Besides easier way of handling some corner cases, as exampled by
Friedrich W. H. Kossebau, this is the main reason. If WebSVN goes
away, it means redesign of l10n.kde.org to extract PO/POT files to
some storage place for serving. It might take less space than a full
copy of SVN repo, but who volunteers to write the code (before WebSVN
is shut down)?
Māris Nartišs.
Re: Sysadmin Load Reduction: Subversion Infrastructure
Ben Cooksley ha scritto: > On Sun, Nov 10, 2019 at 11:19 AM Albert Astals Cid wrote: >> >> El dissabte, 9 de novembre de 2019, a les 21:22:16 CET, Ben Cooksley va >> escriure: >>> On Sun, Nov 10, 2019 at 8:37 AM Albert Astals Cid wrote: El dissabte, 9 de novembre de 2019, a les 19:27:07 CET, Ben Cooksley va escriure: > On Sun, Nov 10, 2019 at 7:18 AM Alexander Potashev > wrote: >> >> сб, 9 нояб. 2019 г. в 03:20, Ben Cooksley : >>> This would include the shutdown of WebSVN in particular, which when >>> coupled with the shutdown of our two CGit instances would also allow >>> for us to eliminate an entire virtual machine from our systems. >> >> Will there be any web interface for SVN after shutdown of WebSVN? >> >> Can we assume https://phabricator.kde.org/source/svn/ remains >> available during the next 10 years? >> > > Phabricator's browser will be retired as part of the shutdown of > Phabricator, which will take place once Gitlab has assumed > responsibility for code hosting and review, and the tasks have been > migrated from Phabricator. > > Should WebSVN be shutdown as well, then there would be no web > interface to our SVN repository. That's not acceptable. >>> >>> Mind explaining why? >> >> Because we use it in l10n.kde.org to link to po files. > > Mind detailing what those links are used for? > >> >>> Bear in mind that there is a cost both in terms of infrastructure, and >>> people time to maintain a service such as WebSVN. >> >> We have money, we don't have to shut down things we use because there is a >> cost. > > I wasn't referring to monetary cost there, I was referring to the flow > on effects (such as having to maintain the necessary components on the > master server to allow for the Subversion repository to be mirrored). > > Note also the "people time" component there. Sure, but please see my previous questions: - can we extend the space of rosetta? It already has a partial checkout, and 100 GB of free space (which can be kept down - if that's not enough, can we simply setup a machine which periodically sync from the svn repository? You are probably going to tell me that it does not work without server support, but from what I'm reading about svnsync, I don't think it should overload the server if executed every 30 minutes. Are we sure that we still need something on the master server? Let's try it first. - in case it's not clear, I'm volunteering to maintain that system. Ciao -- Luigi
Re: Sysadmin Load Reduction: Subversion Infrastructure
El diumenge, 10 de novembre de 2019, a les 3:04:03 CET, Alexander Potashev va
escriure:
> вс, 10 нояб. 2019 г. в 04:10, Ben Cooksley :
> >
> > On Sun, Nov 10, 2019 at 11:19 AM Albert Astals Cid wrote:
> > >
> > > El dissabte, 9 de novembre de 2019, a les 21:22:16 CET, Ben Cooksley va
> > > escriure:
> > > > On Sun, Nov 10, 2019 at 8:37 AM Albert Astals Cid wrote:
> > > > >
> > > > > El dissabte, 9 de novembre de 2019, a les 19:27:07 CET, Ben Cooksley
> > > > > va escriure:
> > > > > > On Sun, Nov 10, 2019 at 7:18 AM Alexander Potashev
> > > > > > wrote:
> > > > > > >
> > > > > > > сб, 9 нояб. 2019 г. в 03:20, Ben Cooksley :
> > > > > > > > This would include the shutdown of WebSVN in particular, which
> > > > > > > > when
> > > > > > > > coupled with the shutdown of our two CGit instances would also
> > > > > > > > allow
> > > > > > > > for us to eliminate an entire virtual machine from our systems.
> > > > > > >
> > > > > > > Will there be any web interface for SVN after shutdown of WebSVN?
> > > > > > >
> > > > > > > Can we assume https://phabricator.kde.org/source/svn/ remains
> > > > > > > available during the next 10 years?
> > > > > > >
> > > > > >
> > > > > > Phabricator's browser will be retired as part of the shutdown of
> > > > > > Phabricator, which will take place once Gitlab has assumed
> > > > > > responsibility for code hosting and review, and the tasks have been
> > > > > > migrated from Phabricator.
> > > > > >
> > > > > > Should WebSVN be shutdown as well, then there would be no web
> > > > > > interface to our SVN repository.
> > > > >
> > > > > That's not acceptable.
> > > >
> > > > Mind explaining why?
> > >
> > > Because we use it in l10n.kde.org to link to po files.
> >
> > Mind detailing what those links are used for?
>
> A common workflow suggested for translators without SVN account
> (and/or those who find it hard to use SVN) is as folllows:
>
> 1. Go to https://l10n.kde.org/stats/gui/trunk-kf5/team/ru/ ("ru" for
> Russian) and pick a .po translation file that needs update - e.g.
> incomplete translation or one containing mistakes.
>
> 2. Click on the .po file at l10n.kde.org to downloaded. WebSVN is
> used for this.
>
> 3. Edit the downloaded .po file, send for review, etc.
>
>
> WebSVN is even mentioned in KDE's primary translation how-to:
> https://l10n.kde.org/docs/translation-howto/gui-translation.html
In addition to that we also use it in check-kde-tp-results to link to the
particular version the check was run
e.g.
https://l10n.kde.org/check-kde-tp-results/trunk-kf5/et/messages.html
links to
https://websvn.kde.org/trunk/l10n-kf5/et/messages/kdemultimedia/kwave.po?revision=1555006=markup
in case it was fixed later since the checks only run weekly.
Of course all all of this can be "fixed" by us providing having a "mirror of
the l10n files" on l10n.kde.org but without having a clue on how websvn works
it seems it'd be much easier to just keep running that instead of having to
code our own mirror.
Cheers,
Albert
Re: Sysadmin Load Reduction: Subversion Infrastructure
El diumenge, 10 de novembre de 2019, a les 2:09:19 CET, Ben Cooksley va escriure: > On Sun, Nov 10, 2019 at 11:19 AM Albert Astals Cid wrote: > > > > El dissabte, 9 de novembre de 2019, a les 21:22:16 CET, Ben Cooksley va > > escriure: > > > On Sun, Nov 10, 2019 at 8:37 AM Albert Astals Cid wrote: > > > > > > > > El dissabte, 9 de novembre de 2019, a les 19:27:07 CET, Ben Cooksley va > > > > escriure: > > > > > On Sun, Nov 10, 2019 at 7:18 AM Alexander Potashev > > > > > wrote: > > > > > > > > > > > > сб, 9 нояб. 2019 г. в 03:20, Ben Cooksley : > > > > > > > This would include the shutdown of WebSVN in particular, which > > > > > > > when > > > > > > > coupled with the shutdown of our two CGit instances would also > > > > > > > allow > > > > > > > for us to eliminate an entire virtual machine from our systems. > > > > > > > > > > > > Will there be any web interface for SVN after shutdown of WebSVN? > > > > > > > > > > > > Can we assume https://phabricator.kde.org/source/svn/ remains > > > > > > available during the next 10 years? > > > > > > > > > > > > > > > > Phabricator's browser will be retired as part of the shutdown of > > > > > Phabricator, which will take place once Gitlab has assumed > > > > > responsibility for code hosting and review, and the tasks have been > > > > > migrated from Phabricator. > > > > > > > > > > Should WebSVN be shutdown as well, then there would be no web > > > > > interface to our SVN repository. > > > > > > > > That's not acceptable. > > > > > > Mind explaining why? > > > > Because we use it in l10n.kde.org to link to po files. > > Mind detailing what those links are used for? > > > > > > Bear in mind that there is a cost both in terms of infrastructure, and > > > people time to maintain a service such as WebSVN. > > > > We have money, we don't have to shut down things we use because there is a > > cost. > > I wasn't referring to monetary cost there, I was referring to the flow > on effects (such as having to maintain the necessary components on the > master server to allow for the Subversion repository to be mirrored). > > Note also the "people time" component there. That can also be solved with money :) Cheers, Albert
Re: Sysadmin Load Reduction: Subversion Infrastructure
вс, 10 нояб. 2019 г. в 04:10, Ben Cooksley :
>
> On Sun, Nov 10, 2019 at 11:19 AM Albert Astals Cid wrote:
> >
> > El dissabte, 9 de novembre de 2019, a les 21:22:16 CET, Ben Cooksley va
> > escriure:
> > > On Sun, Nov 10, 2019 at 8:37 AM Albert Astals Cid wrote:
> > > >
> > > > El dissabte, 9 de novembre de 2019, a les 19:27:07 CET, Ben Cooksley va
> > > > escriure:
> > > > > On Sun, Nov 10, 2019 at 7:18 AM Alexander Potashev
> > > > > wrote:
> > > > > >
> > > > > > сб, 9 нояб. 2019 г. в 03:20, Ben Cooksley :
> > > > > > > This would include the shutdown of WebSVN in particular, which
> > > > > > > when
> > > > > > > coupled with the shutdown of our two CGit instances would also
> > > > > > > allow
> > > > > > > for us to eliminate an entire virtual machine from our systems.
> > > > > >
> > > > > > Will there be any web interface for SVN after shutdown of WebSVN?
> > > > > >
> > > > > > Can we assume https://phabricator.kde.org/source/svn/ remains
> > > > > > available during the next 10 years?
> > > > > >
> > > > >
> > > > > Phabricator's browser will be retired as part of the shutdown of
> > > > > Phabricator, which will take place once Gitlab has assumed
> > > > > responsibility for code hosting and review, and the tasks have been
> > > > > migrated from Phabricator.
> > > > >
> > > > > Should WebSVN be shutdown as well, then there would be no web
> > > > > interface to our SVN repository.
> > > >
> > > > That's not acceptable.
> > >
> > > Mind explaining why?
> >
> > Because we use it in l10n.kde.org to link to po files.
>
> Mind detailing what those links are used for?
A common workflow suggested for translators without SVN account
(and/or those who find it hard to use SVN) is as folllows:
1. Go to https://l10n.kde.org/stats/gui/trunk-kf5/team/ru/ ("ru" for
Russian) and pick a .po translation file that needs update - e.g.
incomplete translation or one containing mistakes.
2. Click on the .po file at l10n.kde.org to downloaded. WebSVN is
used for this.
3. Edit the downloaded .po file, send for review, etc.
WebSVN is even mentioned in KDE's primary translation how-to:
https://l10n.kde.org/docs/translation-howto/gui-translation.html
--
Alexander Potashev
Re: Sysadmin Load Reduction: Subversion Infrastructure
On Sun, Nov 10, 2019 at 11:19 AM Albert Astals Cid wrote: > > El dissabte, 9 de novembre de 2019, a les 21:22:16 CET, Ben Cooksley va > escriure: > > On Sun, Nov 10, 2019 at 8:37 AM Albert Astals Cid wrote: > > > > > > El dissabte, 9 de novembre de 2019, a les 19:27:07 CET, Ben Cooksley va > > > escriure: > > > > On Sun, Nov 10, 2019 at 7:18 AM Alexander Potashev > > > > wrote: > > > > > > > > > > сб, 9 нояб. 2019 г. в 03:20, Ben Cooksley : > > > > > > This would include the shutdown of WebSVN in particular, which when > > > > > > coupled with the shutdown of our two CGit instances would also allow > > > > > > for us to eliminate an entire virtual machine from our systems. > > > > > > > > > > Will there be any web interface for SVN after shutdown of WebSVN? > > > > > > > > > > Can we assume https://phabricator.kde.org/source/svn/ remains > > > > > available during the next 10 years? > > > > > > > > > > > > > Phabricator's browser will be retired as part of the shutdown of > > > > Phabricator, which will take place once Gitlab has assumed > > > > responsibility for code hosting and review, and the tasks have been > > > > migrated from Phabricator. > > > > > > > > Should WebSVN be shutdown as well, then there would be no web > > > > interface to our SVN repository. > > > > > > That's not acceptable. > > > > Mind explaining why? > > Because we use it in l10n.kde.org to link to po files. Mind detailing what those links are used for? > > > Bear in mind that there is a cost both in terms of infrastructure, and > > people time to maintain a service such as WebSVN. > > We have money, we don't have to shut down things we use because there is a > cost. I wasn't referring to monetary cost there, I was referring to the flow on effects (such as having to maintain the necessary components on the master server to allow for the Subversion repository to be mirrored). Note also the "people time" component there. > > Cheers, > Albert Thanks, Ben > > > This includes having to maintain a mirror of the repository on the > > machine that runs WebSVN, along with the associated infrastructure for > > allowing that mirroring to happen on the master server. > > > > > > > > Cheers, > > > Albert > > > > Regards, > > Ben > > > > > > > > > > > > > > > > > > > I'm not sure what use case Luigi has in mind, but among Russian l10n > > > > > team we often use WebSVN to refer to specific SVN revisions, for > > > > > example: https://websvn.kde.org/?view=revision=1555342 > > > > > > > > > > -- > > > > > Alexander Potashev > > > > > > > > Regards, > > > > Ben > > > > > > > > > > > > > > > > > > > > > >
Re: Sysadmin Load Reduction: Subversion Infrastructure
On Sun, Nov 10, 2019 at 11:47 AM Friedrich W. H. Kossebau wrote: > > Am Samstag, 9. November 2019, 21:22:16 CET schrieb Ben Cooksley: > > On Sun, Nov 10, 2019 at 8:37 AM Albert Astals Cid wrote: > > > El dissabte, 9 de novembre de 2019, a les 19:27:07 CET, Ben Cooksley va > escriure: > > > > On Sun, Nov 10, 2019 at 7:18 AM Alexander Potashev > wrote: > > > > > сб, 9 нояб. 2019 г. в 03:20, Ben Cooksley : > > > > > > This would include the shutdown of WebSVN in particular, which when > > > > > > coupled with the shutdown of our two CGit instances would also allow > > > > > > for us to eliminate an entire virtual machine from our systems. > > > > > > > > > > Will there be any web interface for SVN after shutdown of WebSVN? > > > > > > > > > > Can we assume https://phabricator.kde.org/source/svn/ remains > > > > > available during the next 10 years? > > > > > > > > Phabricator's browser will be retired as part of the shutdown of > > > > Phabricator, which will take place once Gitlab has assumed > > > > responsibility for code hosting and review, and the tasks have been > > > > migrated from Phabricator. > > > > > > > > Should WebSVN be shutdown as well, then there would be no web > > > > interface to our SVN repository. > > > > > > That's not acceptable. > > > > Mind explaining why? > > FWIW, everytime I had to deal with translations as developer (like checking > pot files as well as .po files contents) I found having the web interface and > its browsing feature very valuable to quickly find what I was looking for, > over having to locally mess around with svn commands and juggling between > commandline & file viewers. Including url bookmarks for quick access to > browsing certain sets of files. > > Incidents which I remember right now included: > * finding out whether extraction scripts were working as intended > * comparing translations seen by users over what they should see > > Are there any other KDE clients of the svn repos still around, besides > translation system? The only other clients I am aware of are a small number of websites, which can be found at /trunk/www/ in SVN. The list of ones we serve can be found at https://invent.kde.org/sysadmin/binary-factory-tooling/blob/master/staticweb/standard-jobs.yaml#L69 > Perhaps the "full clone" needed for WebSVN could be reduced to the translation > subtrees, would that improve situation to a degree if possible? (well, you > surely thought of this yourself, just in case) Unfortunately WebSVN needs a full copy of the underlying repository to be able to work. > > For me as developer contributor to projects in KDE spheres, losing the web > browsing interface for raw translation files would be a regression in > developer experience. > > Cheers > Friedrich > > Regards, Ben
Re: Sysadmin Load Reduction: Subversion Infrastructure
El sáb., 9 de nov. de 2019 a la(s) 19:48, Friedrich W. H. Kossebau (kosse...@kde.org) escribió: > FWIW, everytime I had to deal with translations as developer (like checking > pot files as well as .po files contents) I found having the web interface and > its browsing feature very valuable to quickly find what I was looking for, > over having to locally mess around with svn commands and juggling between > commandline & file viewers. Including url bookmarks for quick access to > browsing certain sets of files. > > Incidents which I remember right now included: > * finding out whether extraction scripts were working as intended > * comparing translations seen by users over what they should see > > Are there any other KDE clients of the svn repos still around, besides > translation system? > Perhaps the "full clone" needed for WebSVN could be reduced to the translation > subtrees, would that improve situation to a degree if possible? (well, you > surely thought of this yourself, just in case) This is unfortunately not possible. WebSVN needs a full copy of the repository with its history, not a svn checkout, and that can't be trimmed to a subtree. Or maybe you *can* extract a subtree, but then you can't keep that in sync with new changes that appear in the master repository. Even in git that's a giant pain. -- Nicolás
Re: Sysadmin Load Reduction: Subversion Infrastructure
Am Samstag, 9. November 2019, 21:22:16 CET schrieb Ben Cooksley: > On Sun, Nov 10, 2019 at 8:37 AM Albert Astals Cid wrote: > > El dissabte, 9 de novembre de 2019, a les 19:27:07 CET, Ben Cooksley va escriure: > > > On Sun, Nov 10, 2019 at 7:18 AM Alexander Potashev wrote: > > > > сб, 9 нояб. 2019 г. в 03:20, Ben Cooksley : > > > > > This would include the shutdown of WebSVN in particular, which when > > > > > coupled with the shutdown of our two CGit instances would also allow > > > > > for us to eliminate an entire virtual machine from our systems. > > > > > > > > Will there be any web interface for SVN after shutdown of WebSVN? > > > > > > > > Can we assume https://phabricator.kde.org/source/svn/ remains > > > > available during the next 10 years? > > > > > > Phabricator's browser will be retired as part of the shutdown of > > > Phabricator, which will take place once Gitlab has assumed > > > responsibility for code hosting and review, and the tasks have been > > > migrated from Phabricator. > > > > > > Should WebSVN be shutdown as well, then there would be no web > > > interface to our SVN repository. > > > > That's not acceptable. > > Mind explaining why? FWIW, everytime I had to deal with translations as developer (like checking pot files as well as .po files contents) I found having the web interface and its browsing feature very valuable to quickly find what I was looking for, over having to locally mess around with svn commands and juggling between commandline & file viewers. Including url bookmarks for quick access to browsing certain sets of files. Incidents which I remember right now included: * finding out whether extraction scripts were working as intended * comparing translations seen by users over what they should see Are there any other KDE clients of the svn repos still around, besides translation system? Perhaps the "full clone" needed for WebSVN could be reduced to the translation subtrees, would that improve situation to a degree if possible? (well, you surely thought of this yourself, just in case) For me as developer contributor to projects in KDE spheres, losing the web browsing interface for raw translation files would be a regression in developer experience. Cheers Friedrich
Re: Sysadmin Load Reduction: Subversion Infrastructure
El dissabte, 9 de novembre de 2019, a les 21:22:16 CET, Ben Cooksley va escriure: > On Sun, Nov 10, 2019 at 8:37 AM Albert Astals Cid wrote: > > > > El dissabte, 9 de novembre de 2019, a les 19:27:07 CET, Ben Cooksley va > > escriure: > > > On Sun, Nov 10, 2019 at 7:18 AM Alexander Potashev > > > wrote: > > > > > > > > сб, 9 нояб. 2019 г. в 03:20, Ben Cooksley : > > > > > This would include the shutdown of WebSVN in particular, which when > > > > > coupled with the shutdown of our two CGit instances would also allow > > > > > for us to eliminate an entire virtual machine from our systems. > > > > > > > > Will there be any web interface for SVN after shutdown of WebSVN? > > > > > > > > Can we assume https://phabricator.kde.org/source/svn/ remains > > > > available during the next 10 years? > > > > > > > > > > Phabricator's browser will be retired as part of the shutdown of > > > Phabricator, which will take place once Gitlab has assumed > > > responsibility for code hosting and review, and the tasks have been > > > migrated from Phabricator. > > > > > > Should WebSVN be shutdown as well, then there would be no web > > > interface to our SVN repository. > > > > That's not acceptable. > > Mind explaining why? Because we use it in l10n.kde.org to link to po files. > Bear in mind that there is a cost both in terms of infrastructure, and > people time to maintain a service such as WebSVN. We have money, we don't have to shut down things we use because there is a cost. Cheers, Albert > This includes having to maintain a mirror of the repository on the > machine that runs WebSVN, along with the associated infrastructure for > allowing that mirroring to happen on the master server. > > > > > Cheers, > > Albert > > Regards, > Ben > > > > > > > > > > > > > > I'm not sure what use case Luigi has in mind, but among Russian l10n > > > > team we often use WebSVN to refer to specific SVN revisions, for > > > > example: https://websvn.kde.org/?view=revision=1555342 > > > > > > > > -- > > > > Alexander Potashev > > > > > > Regards, > > > Ben > > > > > > > > > > > >
Re: Sysadmin Load Reduction: Subversion Infrastructure
On Sun, Nov 10, 2019 at 9:32 AM Albert Vaca Cintora wrote: > > On Sat, Nov 9, 2019 at 6:53 PM Ben Cooksley wrote: > > > > Unfortunately Rosetta does not have the necessary free disk space, as > > the Subversion repository is approximately 80GB these days in size, > > and operating WebSVN requires a full copy of the Subversion repository > > locally in order to operate. > > Is this also the case if you run WebSVN next to the SVN server itself > (ie: on the same machine)? If a single repo copy can be shared by > WebSVN and the actual SVN server, that would reduce the storage needs. Running WebSVN on the same machine is not safe for us to do, because certain requests cause WebSVN to generate a substantial amount of system load, and have caused out of memory events on systems running it in the past. This poses an unacceptable risk to the canonical copy of our repositories, as well as to Gitlab (which will be hosted on the same machine as the master Subversion repository) > > Albert Regards, Ben
Re: Sysadmin Load Reduction: Subversion Infrastructure
On Sat, Nov 9, 2019 at 6:53 PM Ben Cooksley wrote: > > Unfortunately Rosetta does not have the necessary free disk space, as > the Subversion repository is approximately 80GB these days in size, > and operating WebSVN requires a full copy of the Subversion repository > locally in order to operate. Is this also the case if you run WebSVN next to the SVN server itself (ie: on the same machine)? If a single repo copy can be shared by WebSVN and the actual SVN server, that would reduce the storage needs. Albert
Re: Sysadmin Load Reduction: Subversion Infrastructure
On Sun, Nov 10, 2019 at 8:37 AM Albert Astals Cid wrote: > > El dissabte, 9 de novembre de 2019, a les 19:27:07 CET, Ben Cooksley va > escriure: > > On Sun, Nov 10, 2019 at 7:18 AM Alexander Potashev > > wrote: > > > > > > сб, 9 нояб. 2019 г. в 03:20, Ben Cooksley : > > > > This would include the shutdown of WebSVN in particular, which when > > > > coupled with the shutdown of our two CGit instances would also allow > > > > for us to eliminate an entire virtual machine from our systems. > > > > > > Will there be any web interface for SVN after shutdown of WebSVN? > > > > > > Can we assume https://phabricator.kde.org/source/svn/ remains > > > available during the next 10 years? > > > > > > > Phabricator's browser will be retired as part of the shutdown of > > Phabricator, which will take place once Gitlab has assumed > > responsibility for code hosting and review, and the tasks have been > > migrated from Phabricator. > > > > Should WebSVN be shutdown as well, then there would be no web > > interface to our SVN repository. > > That's not acceptable. Mind explaining why? Bear in mind that there is a cost both in terms of infrastructure, and people time to maintain a service such as WebSVN. This includes having to maintain a mirror of the repository on the machine that runs WebSVN, along with the associated infrastructure for allowing that mirroring to happen on the master server. > > Cheers, > Albert Regards, Ben > > > > > > > > > I'm not sure what use case Luigi has in mind, but among Russian l10n > > > team we often use WebSVN to refer to specific SVN revisions, for > > > example: https://websvn.kde.org/?view=revision=1555342 > > > > > > -- > > > Alexander Potashev > > > > Regards, > > Ben > > > > > >
Re: Sysadmin Load Reduction: Subversion Infrastructure
El dissabte, 9 de novembre de 2019, a les 19:27:07 CET, Ben Cooksley va escriure: > On Sun, Nov 10, 2019 at 7:18 AM Alexander Potashev > wrote: > > > > сб, 9 нояб. 2019 г. в 03:20, Ben Cooksley : > > > This would include the shutdown of WebSVN in particular, which when > > > coupled with the shutdown of our two CGit instances would also allow > > > for us to eliminate an entire virtual machine from our systems. > > > > Will there be any web interface for SVN after shutdown of WebSVN? > > > > Can we assume https://phabricator.kde.org/source/svn/ remains > > available during the next 10 years? > > > > Phabricator's browser will be retired as part of the shutdown of > Phabricator, which will take place once Gitlab has assumed > responsibility for code hosting and review, and the tasks have been > migrated from Phabricator. > > Should WebSVN be shutdown as well, then there would be no web > interface to our SVN repository. That's not acceptable. Cheers, Albert > > > > > I'm not sure what use case Luigi has in mind, but among Russian l10n > > team we often use WebSVN to refer to specific SVN revisions, for > > example: https://websvn.kde.org/?view=revision=1555342 > > > > -- > > Alexander Potashev > > Regards, > Ben >
Re: Sysadmin Load Reduction: Subversion Infrastructure
Ben Cooksley ha scritto: > On Sun, Nov 10, 2019 at 12:20 AM Luigi Toscano > wrote: >> >> Ben Cooksley ha scritto: >>> Hi all, >>> >>> When KDE committed to performing a migration to Git back in 2010, one >>> of the things that was agreed at the time was that translators could >>> remain on Subversion to avoid disrupting their workflows. >>> >>> This however has led to a certain amount of additional infrastructure >>> which Sysadmin needs to continue to maintain. >>> >>> In recognition of the fact that with few exceptions, everything has >>> now migrated from Subversion aside from Translations, i'd like to >>> reduce the level of infrastructure supporting our Subversion >>> repository to the bare minimum necessary. >>> >>> This would include the shutdown of WebSVN in particular, which when >>> coupled with the shutdown of our two CGit instances would also allow >>> for us to eliminate an entire virtual machine from our systems. >> >> As websvn has proven to be useful, would it help the sysadmin (pending >> agreement with aacid and the other people with root access on rosetta) to >> move >> websvn to rosetta and under localization team's responsibility? > > Unfortunately Rosetta does not have the necessary free disk space, as > the Subversion repository is approximately 80GB these days in size, > and operating WebSVN requires a full copy of the Subversion repository > locally in order to operate. Can we extend rosetta's disk then, or add an external volume? In case it's not clear, I'd like to do everything possible to have websvn still up. -- Luigi
Re: Sysadmin Load Reduction: Subversion Infrastructure
сб, 9 нояб. 2019 г. в 03:20, Ben Cooksley : > This would include the shutdown of WebSVN in particular, which when > coupled with the shutdown of our two CGit instances would also allow > for us to eliminate an entire virtual machine from our systems. Will there be any web interface for SVN after shutdown of WebSVN? Can we assume https://phabricator.kde.org/source/svn/ remains available during the next 10 years? I'm not sure what use case Luigi has in mind, but among Russian l10n team we often use WebSVN to refer to specific SVN revisions, for example: https://websvn.kde.org/?view=revision=1555342 -- Alexander Potashev
Re: Sysadmin Load Reduction: Subversion Infrastructure
On Sun, Nov 10, 2019 at 12:20 AM Luigi Toscano wrote: > > Ben Cooksley ha scritto: > > Hi all, > > > > When KDE committed to performing a migration to Git back in 2010, one > > of the things that was agreed at the time was that translators could > > remain on Subversion to avoid disrupting their workflows. > > > > This however has led to a certain amount of additional infrastructure > > which Sysadmin needs to continue to maintain. > > > > In recognition of the fact that with few exceptions, everything has > > now migrated from Subversion aside from Translations, i'd like to > > reduce the level of infrastructure supporting our Subversion > > repository to the bare minimum necessary. > > > > This would include the shutdown of WebSVN in particular, which when > > coupled with the shutdown of our two CGit instances would also allow > > for us to eliminate an entire virtual machine from our systems. > > As websvn has proven to be useful, would it help the sysadmin (pending > agreement with aacid and the other people with root access on rosetta) to move > websvn to rosetta and under localization team's responsibility? Unfortunately Rosetta does not have the necessary free disk space, as the Subversion repository is approximately 80GB these days in size, and operating WebSVN requires a full copy of the Subversion repository locally in order to operate. > > Ciao > -- > Luigi Cheers, Ben
Re: Sysadmin Load Reduction: Subversion Infrastructure
Ben Cooksley ha scritto: > Hi all, > > When KDE committed to performing a migration to Git back in 2010, one > of the things that was agreed at the time was that translators could > remain on Subversion to avoid disrupting their workflows. > > This however has led to a certain amount of additional infrastructure > which Sysadmin needs to continue to maintain. > > In recognition of the fact that with few exceptions, everything has > now migrated from Subversion aside from Translations, i'd like to > reduce the level of infrastructure supporting our Subversion > repository to the bare minimum necessary. > > This would include the shutdown of WebSVN in particular, which when > coupled with the shutdown of our two CGit instances would also allow > for us to eliminate an entire virtual machine from our systems. As websvn has proven to be useful, would it help the sysadmin (pending agreement with aacid and the other people with root access on rosetta) to move websvn to rosetta and under localization team's responsibility? Ciao -- Luigi
Sysadmin Load Reduction: Subversion Infrastructure
Hi all, When KDE committed to performing a migration to Git back in 2010, one of the things that was agreed at the time was that translators could remain on Subversion to avoid disrupting their workflows. This however has led to a certain amount of additional infrastructure which Sysadmin needs to continue to maintain. In recognition of the fact that with few exceptions, everything has now migrated from Subversion aside from Translations, i'd like to reduce the level of infrastructure supporting our Subversion repository to the bare minimum necessary. This would include the shutdown of WebSVN in particular, which when coupled with the shutdown of our two CGit instances would also allow for us to eliminate an entire virtual machine from our systems. On top of this, i'd also like to remove commit access to it for everyone but translators and those who need to work on the small number of websites remaining on Subversion and only provision this for people on an as-needed basis. In the next year or so i'd expect the remaining websites to complete their migrations to Git, after which only translators would receive access. We would also cease providing geographically distributed anonsvn service, with anonymous access only being provided by the master server going forward. Any comments? Thanks, Ben
