Re: Help needed: conflicting interests between Salsa admins and Salsa users (Re: Git Packaging Round 2: When to Salsa)
Hi! On 06.01.20 19:56, Alexander Wirt wrote: > On Mon, 06 Jan 2020, Ulrike Uhlig wrote: >>> On 28.12.19 18:16, Alexander Wirt wrote: >> >From your replies to emails in this thread I was wondering: do you mean >> that the Salsa team does not need, or does not want, help? Or does not >> need, or want, help outside of a sprint? > That basically means: yes we probably need help. But it also means that > getting help should be done in a coordinated way, like introducing one or two > team members during a sprint. Getting someone new involved always means > overhead and should happen when there is time for such overhead. In my > experience sprints are ideal for it. I also talked to some people about > getting them involved in salsa, but there will also be a call for help. > > I / we plan to add at least one global admin and maybe one or two assistants > that help with "user" support. > > We just need some time to plan and coordinate those things (around christmas > is really a bad timing for such discussions) Sounds like a good plan! :) -- ulrike
Re: Help needed: conflicting interests between Salsa admins and Salsa users (Re: Git Packaging Round 2: When to Salsa)
On Mon, 06 Jan 2020, Ulrike Uhlig wrote: > Hi formorer, > > > On 28.12.19 18:16, Alexander Wirt wrote: > >> On Thu, 26 Dec 2019, Otto Kekäläinen wrote: > >> I am sure there are many ways to help the team and it is not just > >> about Salsa/Gitlab admin stuff, but also about creating structure in > >> the team, triaging issues, spreading best practices for users and > >> helping the most advanced users to grow into admins of Salsa etc. > >> Right now we don't even have any kind of salsa-related discussion > >> list on lists.debian.org. Thus I wanted to raise discussion on > >> debian-devel. In my opinion Salsa is becoming a very central piece of > >> the Debian infrastructure and it should have more attention on > >> debian-devel and from the project leader. > > > We are working on it and after my holidays are over I will plan > > another sprint for improving salsa. > > >From your replies to emails in this thread I was wondering: do you mean > that the Salsa team does not need, or does not want, help? Or does not > need, or want, help outside of a sprint? That basically means: yes we probably need help. But it also means that getting help should be done in a coordinated way, like introducing one or two team members during a sprint. Getting someone new involved always means overhead and should happen when there is time for such overhead. In my experience sprints are ideal for it. I also talked to some people about getting them involved in salsa, but there will also be a call for help. I / we plan to add at least one global admin and maybe one or two assistants that help with "user" support. We just need some time to plan and coordinate those things (around christmas is really a bad timing for such discussions) Alex
Re: Help needed: conflicting interests between Salsa admins and Salsa users (Re: Git Packaging Round 2: When to Salsa)
Hi formorer, > On 28.12.19 18:16, Alexander Wirt wrote: >> On Thu, 26 Dec 2019, Otto Kekäläinen wrote: >> I am sure there are many ways to help the team and it is not just >> about Salsa/Gitlab admin stuff, but also about creating structure in >> the team, triaging issues, spreading best practices for users and >> helping the most advanced users to grow into admins of Salsa etc. >> Right now we don't even have any kind of salsa-related discussion >> list on lists.debian.org. Thus I wanted to raise discussion on >> debian-devel. In my opinion Salsa is becoming a very central piece of >> the Debian infrastructure and it should have more attention on >> debian-devel and from the project leader. > We are working on it and after my holidays are over I will plan > another sprint for improving salsa. >From your replies to emails in this thread I was wondering: do you mean that the Salsa team does not need, or does not want, help? Or does not need, or want, help outside of a sprint? -- ulrike
Re: Help needed: conflicting interests between Salsa admins and Salsa users (Re: Git Packaging Round 2: When to Salsa)
On Mon, 06 Jan 2020, Sam Hartman wrote: > > "Alexander" == Alexander Wirt writes: > > Alexander> For everything else: we are working on it. > > I just want to confirm that part of the things that you are working on > is documenting the issues. At a number of points you've talked about > how people are misunderstanding the issues or are thinking it's simply > more CPU etc. > > That's all doubtless true. > But part of being in a community is communicating with that community. > People would almost certainly be more understanding if they had more > information. just for the record, I am doing all of this in my spare time and I prefer to decide on my own what is in my queue and what the priority is. And yes, I do prefer to fix things instead of documenting what needs to get fixed. And if everyone would behave sane instead of st* flame wars, insulting each other and so on (it was a listmaster month to forget) my motivation to work with that specific community would be a lot better. And for everyones sake: when announcing CI support we told everyone that the ressources are limited and everyone should play nice with the CI. As often, other people decided to make the CI and the runners a quasi standard, adding it to their workflows and so on. Now everyone is surprised if things are as we always told everybody. What a surprise. Thanks for listening. Alex signature.asc Description: PGP signature
Re: Help needed: conflicting interests between Salsa admins and Salsa users (Re: Git Packaging Round 2: When to Salsa)
> "Alexander" == Alexander Wirt writes: Alexander> For everything else: we are working on it. I just want to confirm that part of the things that you are working on is documenting the issues. At a number of points you've talked about how people are misunderstanding the issues or are thinking it's simply more CPU etc. That's all doubtless true. But part of being in a community is communicating with that community. People would almost certainly be more understanding if they had more information. Yes, that sort of communication does involve time, and yes that time could be spent fixing things. There comes a point though where the communication becomes at least as important as the fix. I think I've heard several requests for more information here, and I just want to confirm that too is in your queue. --Sam
Re: Help needed: conflicting interests between Salsa admins and Salsa users (Re: Git Packaging Round 2: When to Salsa)
On Wed, 01 Jan 2020, Otto Kekäläinen wrote: > Hello! > > ti 31. jouluk. 2019 klo 14.55 Alexander Wirt (formo...@debian.org) kirjoitti: > > On Mon, 30 Dec 2019, Bernd Zeimetz wrote: > > > Also, if resources are an issue: I've offered several times to see if I > > > can get some k8s resources for gitlab runners, but never got a reply. > > > Not even a no. > > It is not a problem on the runner side. > > > > And as said, we are working on the other problems until we can improve > > something in that part. > > If it is not a problem on the runner side and you don't need more > runner resources, what is the reason the runner is capped at 1h? > MariaDB is a huge beast and building it and running all tests take > 1,5h for completely valid reasons (also note we have ccache on > Salsa-CI so re-builds are much faster). We updated the timeout to three hours. However, if that leads to problems on salsa side we will have to set it back. So please ensure not to create too much load / jobs that use that extended limit. For everything else: we are working on it. Alex
Re: Help needed: conflicting interests between Salsa admins and Salsa users (Re: Git Packaging Round 2: When to Salsa)
Hello! ti 31. jouluk. 2019 klo 14.55 Alexander Wirt (formo...@debian.org) kirjoitti: > On Mon, 30 Dec 2019, Bernd Zeimetz wrote: > > Also, if resources are an issue: I've offered several times to see if I > > can get some k8s resources for gitlab runners, but never got a reply. > > Not even a no. > It is not a problem on the runner side. > > And as said, we are working on the other problems until we can improve > something in that part. If it is not a problem on the runner side and you don't need more runner resources, what is the reason the runner is capped at 1h? MariaDB is a huge beast and building it and running all tests take 1,5h for completely valid reasons (also note we have ccache on Salsa-CI so re-builds are much faster). Could you be kind at set back the default runner time limit to 2h as it was some weeks ago? This is stopping me and our contributors from working on putting mariadb-10.4 in Debian. It you don't intend to revert the change back to how it was? Or could you please at least state the reasons at https://salsa.debian.org/salsa/support/issues/184 ? So far you have not commented anything in your own bug tracker at salsa/support.. Thanks again for everybody maintaining and developing Salsa, the CI and all that comes with them. That has been a huge boost in my motivation to continue Debian work and makes me much more productive than ever before. - Otto
Re: Help needed: conflicting interests between Salsa admins and Salsa users (Re: Git Packaging Round 2: When to Salsa)
On Mon, 30 Dec 2019, Bernd Zeimetz wrote: > > > On 12/30/19 11:29 AM, Raphael Hertzog wrote: > > I don't think that salsa-ci is particularly problematic in terms of > > "efficiency". With the exception of the LXC usage, there's not much > > that can be "cut" to save resources. > > Also, if resources are an issue: I've offered several times to see if I > can get some k8s resources for gitlab runners, but never got a reply. > Not even a no. It is not a problem on the runner side. And as said, we are working on the other problems until we can improve something in that part. Alex
Re: Help needed: conflicting interests between Salsa admins and Salsa users (Re: Git Packaging Round 2: When to Salsa)
On 12/30/19 11:29 AM, Raphael Hertzog wrote: > I don't think that salsa-ci is particularly problematic in terms of > "efficiency". With the exception of the LXC usage, there's not much > that can be "cut" to save resources. Also, if resources are an issue: I've offered several times to see if I can get some k8s resources for gitlab runners, but never got a reply. Not even a no. -- Bernd ZeimetzDebian GNU/Linux Developer http://bzed.dehttp://www.debian.org GPG Fingerprint: ECA1 E3F2 8E11 2432 D485 DD95 EB36 171A 6FF9 435F
Re: Help needed: conflicting interests between Salsa admins and Salsa users (Re: Git Packaging Round 2: When to Salsa)
On Mon, 30 Dec 2019, Raphael Hertzog wrote: > On Sat, 28 Dec 2019, Alexander Wirt wrote: > > On Thu, 26 Dec 2019, Otto Kekäläinen wrote: > > > > > Hello! > > > > > > I've seen many times before statements like these so I'd like to raise > > > some discussion around the topic: > > > > > > pe 13. syysk. 2019 klo 16.36 Bastian Blank (wa...@debian.org) kirjoitti: > > > > On Sun, Sep 08, 2019 at 05:35:10PM -0400, Sam Hartman wrote: > > > > > The Salsa CA pipeline is recommended. > > > > > > > > For this I need to use my veto as Salsa admin. With the CI people we > > > > have to work through too much problems first. > > The salsa ci pipeline itself has some problematic implementation details > > waldi lined out in the past. Afaik nothing changed, we had several reports > > This is not really true: > https://salsa.debian.org/salsa-ci-team/pipeline/issues?scope=all=%E2%9C%93=opened_username=waldi > > Out of 12 issues reported by waldi, 8 have been fixed/closed. > > Among the remaining ones: > > - https://salsa.debian.org/salsa-ci-team/pipeline/issues/93 > (usage of LXC for autopkgtest) > is likely the most problematic one even though you never explained > what's the real issue... yeah it's virtualization over virtualization > and it downloads a root tarball to do its work, but is this worth than > downloading a docker image? It uses more resources than direct execution > of autopkgtest but it hasn't broken anything so far? that second level of virtualisation caused problems where people told us they are not able to do things in their ci jobs. *snip* > > where people telling us things are not possible on our runners. In the end > > most of them turned out to be limitations of salsa ci. Salsa ci is also > > not very efficent, therefore the veto. > > Claims like "salsa ci is not very efficient" are not actionable. Bugs like > those above are more useful but you should at least take the time to > respond to queries of people, otherwise no progress can be made. > > I don't think that salsa-ci is particularly problematic in terms of > "efficiency". With the exception of the LXC usage, there's not much > that can be "cut" to save resources. Thats probably true, but if it is inefficent and may cause problems on our current architecture / ressources - that can't get fixed easily - a veto is the only thing we have. > > We are working on it and after my holidays are over I will plan another > > sprint for improving salsa. > > \o/ Alex - forgive the shortness, I am on vacation signature.asc Description: PGP signature
Re: Help needed: conflicting interests between Salsa admins and Salsa users (Re: Git Packaging Round 2: When to Salsa)
On Sat, 28 Dec 2019, Alexander Wirt wrote: > On Thu, 26 Dec 2019, Otto Kekäläinen wrote: > > > Hello! > > > > I've seen many times before statements like these so I'd like to raise > > some discussion around the topic: > > > > pe 13. syysk. 2019 klo 16.36 Bastian Blank (wa...@debian.org) kirjoitti: > > > On Sun, Sep 08, 2019 at 05:35:10PM -0400, Sam Hartman wrote: > > > > The Salsa CA pipeline is recommended. > > > > > > For this I need to use my veto as Salsa admin. With the CI people we > > > have to work through too much problems first. > The salsa ci pipeline itself has some problematic implementation details > waldi lined out in the past. Afaik nothing changed, we had several reports This is not really true: https://salsa.debian.org/salsa-ci-team/pipeline/issues?scope=all=%E2%9C%93=opened_username=waldi Out of 12 issues reported by waldi, 8 have been fixed/closed. Among the remaining ones: - https://salsa.debian.org/salsa-ci-team/pipeline/issues/93 (usage of LXC for autopkgtest) is likely the most problematic one even though you never explained what's the real issue... yeah it's virtualization over virtualization and it downloads a root tarball to do its work, but is this worth than downloading a docker image? It uses more resources than direct execution of autopkgtest but it hasn't broken anything so far? - https://salsa.debian.org/salsa-ci-team/pipeline/issues/94 (docker images accumulating in forks) this one should have improved a lot AFAIK as GitLab now supports what's required to remove images from the CI environment too and there's WIP on that front (it might even be live without anyone updating that bug, I'm not sure) - https://salsa.debian.org/salsa-ci-team/pipeline/issues/116 This one is not clear to me. What jobs are using "docker-in-docker" without any legitimate use ? - https://salsa.debian.org/salsa-ci-team/pipeline/issues/121 (split into source and build) This one seems like wishlist and has no real impact on resources as long as we build for a single architecture... > where people telling us things are not possible on our runners. In the end > most of them turned out to be limitations of salsa ci. Salsa ci is also > not very efficent, therefore the veto. Claims like "salsa ci is not very efficient" are not actionable. Bugs like those above are more useful but you should at least take the time to respond to queries of people, otherwise no progress can be made. I don't think that salsa-ci is particularly problematic in terms of "efficiency". With the exception of the LXC usage, there's not much that can be "cut" to save resources. > We are working on it and after my holidays are over I will plan another > sprint for improving salsa. \o/ Cheers, -- ⢀⣴⠾⠻⢶⣦⠀ Raphaël Hertzog ⣾⠁⢠⠒⠀⣿⡁ ⢿⡄⠘⠷⠚⠋The Debian Handbook: https://debian-handbook.info/get/ ⠈⠳⣄ Debian Long Term Support: https://deb.li/LTS signature.asc Description: PGP signature
Re: Help needed: conflicting interests between Salsa admins and Salsa users (Re: Git Packaging Round 2: When to Salsa)
On Thu, 26 Dec 2019, Otto Kekäläinen wrote: > Hello! > > I've seen many times before statements like these so I'd like to raise > some discussion around the topic: > > pe 13. syysk. 2019 klo 16.36 Bastian Blank (wa...@debian.org) kirjoitti: > > On Sun, Sep 08, 2019 at 05:35:10PM -0400, Sam Hartman wrote: > > > The Salsa CA pipeline is recommended. > > > > For this I need to use my veto as Salsa admin. With the CI people we > > have to work through too much problems first. The salsa ci pipeline itself has some problematic implementation details waldi lined out in the past. Afaik nothing changed, we had several reports where people telling us things are not possible on our runners. In the end most of them turned out to be limitations of salsa ci. Salsa ci is also not very efficent, therefore the veto. > There seems to be a conflict between the Salsa admins and users of > Salsa: the more Salsa is used, the bigger becomes the maintenance > burden and the more computing resources Salsa needs. There is however > no inherent growth feedback loop in the system that would increase > maintenance commitments as usage commitments grow. In economic terms > one could say that the Salsa admins don't profit from maintaining > Salsa and as demand grows there is nothing that grows the supply at > the moment. > > The reason for Salsa popularity to grow all the time is simply because > it is such a brilliant service and many Debian Developers and aspiring > new contributors love to use it. Personally I've had all my packages > on Salsa since early 2018 and I would never want to go back to the mix > of Github and Alioth I used before. Using Gitlab-CI is nowadays an > inherent part of my packaging workflow to test contributions before > merging them and to do QA before uploads. Any disruptions to Salsa > basically grinds by packaging work to a halt[1], it is so central for > me nowadays. > > Since Salsa was officially launched in 2018 there has not been any [2] > new members to the Salsa admins group [3]. Alexander, Joerg and > Bastian have done a great job maintaining our Gitlab installation. The > software suite is a beast and keeping it running well is a major > effort in itself. > > They need help going forward. The sentiment of restricting vital use > of Salsa is a sign of them trying to keep things under control. But > Salsa usage needs to grow, as that is good for Debian as a project. > For the Debian project I think it would be a priority to find more > resources to the Salsa admin team. I think that would be the ultimate > solution to the current conflict. For more performane salsa would need a proper redesign by moving it from its monolithic system to a more distributed system. In fact we are already talking about it for some time. But in fact you - the users - should not think that everything is as easy as just adding some cpus, disks or workers. Things are often more complicated and - in the end - everything should be maintainable by DSA too. > Personally I cannot commit to maintain Salsa, unfortunately. If Salsa > is out of computing resources I can however help find more sponsors > for public runners. But I have the understanding that Google has > donated plenty of cloud computing time and the root cause is not in > lack of computing resources, but in the human scalability aspects of > Salsa operations. in fact thats only partially true. We are working on it and after my holidays are over I will plan another sprint for improving salsa. Things are not always as easy as it seems. Alex signature.asc Description: PGP signature
Re: Help needed: conflicting interests between Salsa admins and Salsa users (Re: Git Packaging Round 2: When to Salsa)
On Thu, 26 Dec 2019, Otto Kekäläinen wrote: > I am sure there are many ways to help the team and it is not just > about Salsa/Gitlab admin stuff, but also about creating structure in > the team, triaging issues, spreading best practices for users and > helping the most advanced users to grow into admins of Salsa etc. > Right now we don't even have any kind of salsa-related discussion list > on lists.debian.org. Thus I wanted to raise discussion on > debian-devel. In my opinion Salsa is becoming a very central piece of > the Debian infrastructure and it should have more attention on > debian-devel and from the project leader. +1 on everything you said. (And IMO you should have started a new thread instead of replying to an old one) I do hope we can find some constructive way to go forward because the current settings are now too strict and are effectively hindering big packages (exactly those that need help!) and some other advanced use of the service. I would also like to mention that we should have #salsa or #debian-salsa and drop #alioth on IRC, sure it made sense at the start to continue to use the same place as where we used to be but now it just makes it harder to find the correct place to discuss issues with Salsa. Cheers, -- ⢀⣴⠾⠻⢶⣦⠀ Raphaël Hertzog ⣾⠁⢠⠒⠀⣿⡁ ⢿⡄⠘⠷⠚⠋The Debian Handbook: https://debian-handbook.info/get/ ⠈⠳⣄ Debian Long Term Support: https://deb.li/LTS
Help needed: conflicting interests between Salsa admins and Salsa users (Re: Git Packaging Round 2: When to Salsa)
Hello! I've seen many times before statements like these so I'd like to raise some discussion around the topic: pe 13. syysk. 2019 klo 16.36 Bastian Blank (wa...@debian.org) kirjoitti: > On Sun, Sep 08, 2019 at 05:35:10PM -0400, Sam Hartman wrote: > > The Salsa CA pipeline is recommended. > > For this I need to use my veto as Salsa admin. With the CI people we > have to work through too much problems first. There seems to be a conflict between the Salsa admins and users of Salsa: the more Salsa is used, the bigger becomes the maintenance burden and the more computing resources Salsa needs. There is however no inherent growth feedback loop in the system that would increase maintenance commitments as usage commitments grow. In economic terms one could say that the Salsa admins don't profit from maintaining Salsa and as demand grows there is nothing that grows the supply at the moment. The reason for Salsa popularity to grow all the time is simply because it is such a brilliant service and many Debian Developers and aspiring new contributors love to use it. Personally I've had all my packages on Salsa since early 2018 and I would never want to go back to the mix of Github and Alioth I used before. Using Gitlab-CI is nowadays an inherent part of my packaging workflow to test contributions before merging them and to do QA before uploads. Any disruptions to Salsa basically grinds by packaging work to a halt[1], it is so central for me nowadays. Since Salsa was officially launched in 2018 there has not been any [2] new members to the Salsa admins group [3]. Alexander, Joerg and Bastian have done a great job maintaining our Gitlab installation. The software suite is a beast and keeping it running well is a major effort in itself. They need help going forward. The sentiment of restricting vital use of Salsa is a sign of them trying to keep things under control. But Salsa usage needs to grow, as that is good for Debian as a project. For the Debian project I think it would be a priority to find more resources to the Salsa admin team. I think that would be the ultimate solution to the current conflict. Personally I cannot commit to maintain Salsa, unfortunately. If Salsa is out of computing resources I can however help find more sponsors for public runners. But I have the understanding that Google has donated plenty of cloud computing time and the root cause is not in lack of computing resources, but in the human scalability aspects of Salsa operations. I hope somebody else on the debian-devel list would respond to this call for help. I am sure there are many ways to help the team and it is not just about Salsa/Gitlab admin stuff, but also about creating structure in the team, triaging issues, spreading best practices for users and helping the most advanced users to grow into admins of Salsa etc. Right now we don't even have any kind of salsa-related discussion list on lists.debian.org. Thus I wanted to raise discussion on debian-devel. In my opinion Salsa is becoming a very central piece of the Debian infrastructure and it should have more attention on debian-devel and from the project leader. Thanks again for current Salsa admins for the work you've done! Salsa is amazing and I hope it will get broader attention and help so it scales to support our packaging work far into the future. [1] https://salsa.debian.org/salsa/support/issues/184 [2] https://wiki.debian.org/Salsa?action=diff=37=17 [3] https://wiki.debian.org/Salsa#Maintenance
