Jarek, I would suggest you have a direct chat with Greg Stein.
Best Regards, Dave Sent from my iPhone > On Jan 10, 2021, at 9:08 AM, Jarek Potiuk <[email protected]> wrote: > > On Sun, Jan 10, 2021 at 5:28 PM Matt Sicker <[email protected]> wrote: > >> If we can get GA to handle our use case properly, that would be awesome. >> Being in the security engineering domain, though, I’m generally pessimistic >> about security, so please excuse my cynicism. >> > > I totally understand. i am a bit more optimistic, especially that we > potentially could > throw some heavyweight - like a number of Serious Apache project making a > common and coordinated action of "We can either praise GitHub for their > cooperation" or "They are not secure and not willing to improve". > That is a publicity they would either love (the former) or hate (the > latter). > > I am happy to help in any way I can - represent INFRA in talks, describe the > problems and propose solutions, word the "carrot" and "stick opttions and > even prepare how they could look like - I coudl take part in discussions > with GitHub, maybe even escalate this to Microsoft if they will not show > they are cooperating - butI have no legitimation for doing so. > I have no power to throw the weight of ASF in the discussion. But I would > love to do that and lead that if only I had this kind of power at least > delegated > to me and provided with the means of contacting GitHub and representing > the ASF (but I doubt anyone would give me that power, it is a bit risky as > with big power I would have no big responsibility. > > Tough call - I am not sure how else I can help INFRA/ASF to help me and > others. > > J. > > >> >>> On Sun, Jan 10, 2021 at 03:43 Jarek Potiuk <[email protected]> >>> wrote: >>> >>> I have a feeling (though I cannot know for sure) >>> that you are underestimating the power of an organization like ASF in >>> actually 'stating' their requirements and 'expectations' towards GitHub. >>> >>> I am now an engineer, but I used to be CTO, CEO, Head of IT, Head of >>> Technology >>> and I know that a lot can be achieved by proper communication, stating >> your >>> expectations clearly and follow-up and pushing when you are dealing with >>> partners like that - and engineering excellence or security perfection is >>> not the only >>> the thing that matters. Usability, maintenance, streamlining development >>> matter and if you >>> have "good enough security", they are more important for users. >>> >>> I know if you look at it from an "infrastructure security Jenkins" point >> of >>> view - the Jenkins >>> you manage is superior when it comes to security. >>> This is perfectly clear, and I have no intention to question that or >>> disagree with you. >>> And yes - in this aspect I fully agree with you. >>> >>> But there are other aspects which I see (and try to explain). >>> While I deeply care about security (as probably you could see from my >>> earlier >>> communication). Just limiting the discussion to "who is more secure" is a >>> terrible, >>> terrible oversimplification. >>> >>> I encourage you to exercise empathy and see it from the side I was >>> explaining - >>> maintenance, features, integration, streamlining development. Those are >>> important >>> things for developers. Less important for security engineers of course, >> but >>> if >>> we can satisfy security, those are the things that matter. >>> >>> I think currently we have mitigations for all the security problems we >>> found at the project >>> level. Also (as I mentioned before) we will have good leverage - via >> social >>> media pressure >>> to push GA into solving those that are 'systemic' problems we found. They >>> are not >>> necessary for our project to solve, but it would simplify your life as >> you >>> take care of so >>> many projects. So the security bounties that I opened are not for me - >> they >>> are for the >>> ASF as a whole and for the security team of ASF. I exercised a lot of >>> empathy to your >>> team that rather than only solving my problem, I also spend time and >> effort >>> to push >>> GA into solving it for all ASF projects and in the way that ASF infra >>> security will be satisfied. >>> I did not have to do that. Yet I try to think about your needs there. >>> >>> And to be honest I expect something in return. Empathy and understanding >>> other needs >>> I have - performance, usability, streamlining development, minimum >>> engineering effort >>> to solve our problems is the least I can ask for. Help in dealing >>> with GitHub and >>> exercising ASF powers would be great. >>> >>> Maybe with GitHub, the problem is that organizations like ASF do not >>> exercise >>> their leverage and do not clearly state what is essential for them while >>> working with >>> partners like them? >>> >>> Did the ASF explicitly contacted GA and firmly stated that solving the >>> problem of >>> self-hosted runnines is an absolute top priority to solve our performance >>> issues? >>> >>> I do not know. >>> >>> Did anyone from ASF contacted GA and firmly stated that the two bounties >> I >>> created are essential for the security team to be able to provide >> security >>> for >>> the organization? >>> >>> I do not know. >>> >>> Did the ASF push GA in any way in this direction stating that >>> ASF is considering alternatives? (The "stick" in this discussion) >>> >>> I do not know. >>> >>> Did the ASF propose GA that we can endorse their service, write blogs, >> and >>> ask >>> the 100s of projects that will use GA to endorse their service publicly >>> once they >>> start addressing our firmly stated needs and expectations? (The "carrot" >> in >>> this >>> discussion) >>> >>> I do not know. >>> >>> This is what I would do if I were at INFRA. I am not. I am not even an >> ASF >>> member to >>> have more insight and visibility into it. >>> >>> The only thing I can do is to ask for help and see if the ASF Infra is >>> willing to help in >>> the situation by exercising the powers that I do not have. >>> >>> For me, this is really a test, whether the ASF has the power to negotiate >>> with such >>> partners. If not - maybe it's time to think that everything (including >>> GitHub repos) >>> should be self-hosted by INFRA, because if you are dealing with partners >>> like that >>> you should have some negotiating power, otherwise, you put yourself in a >>> loosing >>> position. >>> >>> But again - I do not know much. This is what I would do If I had the >>> powers. >>> On my side, I think I've shown that I do above and beyond what you might >>> expect >>> from a PMC of one of the ASF projects, and asking for help from the >>> organization, >>> I - so far - proudly belong to, is the only thing left I have. I run out >>> of all ammo. >>> >>> So again - please help! >>> >>> J. >>> >>> >>>> On Sat, Jan 9, 2021 at 11:00 PM Matt Sicker <[email protected]> wrote: >>> >>>> I work on the Jenkins security team. We don’t have embarrassing >> security >>>> failures like this anymore, but part of that is due to the added >>> complexity >>>> of a secure configuration. By the time GA meets your security >> standards, >>>> it’ll likely either require non-trivial changes to your CI scripts, or >>>> it’ll break various use cases that you otherwise considered to be >>> usability >>>> enhancements. It’s really getting annoying how every complaint you have >>>> about every non-Jenkins system isn’t a problem in Jenkins. We have more >>>> expertise available to customize things in such a way that works for >>>> non-proprietary SaaS that most services are optimized for (which is why >>>> their security models tend to fall short once a large organization like >>>> Apache tries using something). >>>> >>>> Many of the features you’re asking from GA are likely non-trivial >>>> architecture changes they’ll have to make to accommodate the >> non-trivial >>>> use cases we have. Or maybe it isn’t and they’re just incompetent? >>>> >>>> On Sat, Jan 9, 2021 at 05:58 Jarek Potiuk <[email protected]> wrote: >>>> >>>>>> >>>>>>> >>>>>>> The multiple threads about how shitty those are in practice for >>> your >>>>>>> needs seem to indicate otherwise. Security and easy learning >> curves >>>>>>> don't seem to get along too well, do they? >>>>>> >>>>> >>>>> The usabilty, integration level (especially GitHub Actions), >>> maintenance >>>>> effort needed >>>>> - thi is far, far superior. If only we could solve one simple >> problem - >>>>> securely running >>>>> the self-hosted runners for GA - all our problems are solved >>> INSTANTLY. >>>>> >>>>> Security issues happen everywhere, at least if they happen in such >>>> services >>>>> you can >>>>> mitigate (we just did it in Airflow- we mitigated all the security >>> issues >>>>> we found), >>>>> open bounty requests (I did - I opened two bounty requests) and then >>>>> escalate. >>>>> If I do not hear about my 2 security bounties from GitHub shortly, >>>>> I am going to start a hell of a social media campaign about it >>>>> using all the means I can. I tried to responsibly disclose it but I >> am >>>>> going to write a nice >>>>> blog post about "How to exploit Github Actions" and I am going to >> tell >>>> them >>>>> that before >>>>> I publish it and give them a chance to fix it. >>>>> >>>>> So you have many ways to influence the security of public services >> like >>>>> that. I think it's >>>>> much better than when you have to manage security yourselves. >>>>> >>>>> >>>>>>> >>>>>>> That would all be possible in Jenkins, some of it would be fairly >>>>>>> simple to integrate, others would indeed be non-trivial rewrites. >>>>>>> >>>>>> >>>>> >>>>> Yep. The non-trivial ones I am afraid of. It took me a year to >> perfect >>>> and >>>>> optimise >>>>> a number of steps in our CI and the problem was - it worked really >> well >>>>> until it stopped >>>>> because of uncontrolled increase of usage from other projects and no >>>> secure >>>>> way >>>>> to add extra resources needed (even if we have all the funds - we now >>>> have >>>>> 8000 USD >>>>> secured from our stakeholders - with outlook for more) to run those. >>> But >>>> if >>>>> you add the >>>>> engineering effort needed to migrate, the engineering time for that >>> costs >>>>> FAR more than >>>>> just that - enabling compute resources to use all the engineering >>> efforts >>>>> you've already >>>>> spent. This is no brainer which way is simpler, cheaper and can be >> done >>>>> faster. >>>>> We just need to have a secure way of doing it. >>>>> >>>>> >>>>>>> >>>>>>> >>>>>>> You can have your own Jenkins controller for your PMC. This is >>> vastly >>>>>>> simpler for you to administer than a super time-shared >> environment >>>>>>> like GA. CI systems seem to be a dime a dozen nowadays, yet not a >>>>>>> single one seems to implement job scheduling in a sufficiently >>>>>>> customizable way that scales. >>>>>> >>>>> >>>>> I do not want nor need to administer my CI. And I've done that many >>> times >>>>> in >>>>> the past - Jenkins, Bamboo, GitLab - you name it. >>>>> Heck - I built and maintained my first custom CI framework for my >>> company >>>>> some 20 years ago when the "CI" was just being coined. >>>>> With CI as a service, I do not want to do it anymore. At all. CI for >> me >>>>> should just 'be there'. >>>>> Great CI is one that you are not aware of its existence until your >> test >>>>> fail - and >>>>> even then you just want to see the logs of your failed tests and >> figure >>>>> out the reason >>>>> This is what you want from CI system. I do not want to learn how >>>>> to manage Jenkins, install plugins, configure that etc. This is not >> my >>>> job, >>>>> nor any >>>>> one in our project. This requires far more than just setting it >>>>> up - it is making sure that it is secure, that it runs 24/7, that it >>> gets >>>>> updated etc. etc. >>>>> This is far more complex than 'just use CI'. We have enough trouble >>> with >>>>> setting up >>>>> and maintaining runners (once we get them securely connected). >>>>> >>>>> I know it looks differently from the infrastructure person point of >>> view >>>> - >>>>> running >>>>> Jenkins is pretty much core part of what you do. But for project >>>>> developers, CI >>>>> should just 'work'. This is what I get from GitHub Actions. It just >>>>> 'works'. I have >>>>> to spend 0 effort to maintain it. Sometimes when it does not work, it >>>>> pains, but >>>>> then it's their problem to fix - and they have to fix it eventually >>>> because >>>>> they get >>>>> pressure from all their customers. In case I run my own jenkins >> install >>>> and >>>>> administer it - all those problems fall on us. I do not want that. >> This >>>>> moves us >>>>> away from doing what we should - develop our product. >>>>> >>>>> >>>>>>> >>>>>>> Definitely valid points. Any CI migrations are non-trivial, >>>> especially >>>>>>> once you've set up nice workflows. Perhaps there are some >>>> alternatives >>>>>>> that can help bridge the gap if GA still can't meet your needs. >>> I've >>>>>>> seen prow [1] used in various projects in the Kubernetes >>> communities, >>>>>>> and I'm sure there must be plenty of others. >>>>>> >>>>> >>>>> GA meets all my needs. Except one that I am asking ASF to help with - >>>>> make GitHub focus on making a secure way of working with self-hosted >>>>> runners. That's it . We even (In November) opened a PR to Github >>> Actions >>>>> Runner to enable it: https://github.com/actions/runner/pull/783 >>>>> But we have not heard anything since. >>>>> >>>>> This is what I ask INFRA to help with - put pressure on GitHub to >> make >>> it >>>>> happen. I need nothing more - no money, no Jenkins, nothing like >> that. >>>>> I just want to be able to spend the money we managed to secure. >>>>> >>>>> J. >>>>> >>>>> >>>>> -- >>>>> +48 660 796 129 >>>>> >>>> >>> >>> >>> -- >>> >>> Jarek Potiuk >>> Polidea <https://www.polidea.com/> | Principal Software Engineer >>> >>> M: +48 660 796 129 <+48660796129> >>> [image: Polidea] <https://www.polidea.com/> >>> >> > > > -- > +48 660 796 129
