Re: Diamond Dependencies and Upstreams at Scale
On Wed, May 20, 2020 at 8:39 AM Gerald Wiltse wrote: > > it still captures the problem with visuals, and helps demonstrate demand for > a solution. When we were using Build Flow before the creation of Pipeline, my users really liked the visualization provided by the buildgraph-view plugin. I got a lot of complaints from my users when we switched to Pipeline and this UI was removed. Basil -- You received this message because you are subscribed to the Google Groups "Jenkins Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAFwNDjovetPo0PwywsY8EhV7ywFz-biSEyhwV-yQ-n%3DYay64xQ%40mail.gmail.com.
Re: Diamond Dependencies and Upstreams at Scale
Upon further review, this uses an observer pattern which is just completely different from what I was envisioning. I'm still glad I found it because it still captures the problem with visuals, and helps demonstrate demand for a solution. Gerald R. Wiltse jerrywil...@gmail.com On Wed, May 20, 2020 at 11:09 AM Gerald Wiltse wrote: > It's worth noting that I think a lot of good code and abstraction and > capturing of the problem already exists here: > > https://plugins.jenkins.io/join/ > > I'm not sure if Stefan Wolf still watches this list or actively > contributes to Jenkins, but his insight would probably be invaluable. I > will study his work and try to reach out to him. > > Regards, > Gerald R. Wiltse > jerrywil...@gmail.com > > > > On Tue, May 19, 2020 at 10:30 AM Gerald Wiltse > wrote: > >> I find this feedback very encouraging. It definitely does seem to be a >> good candidate for JEP proposal. I will plan for that in the >> mid-term future. >> >> Your suggestions about alternatives are all right on. In one large >> environment, I created a solution with a metajob that received webhooks >> from all jobs, used the Jenkins REST API's to query "all job >> configurations" and then correlate the hooks to metadata, and use build() >> to trigger the appropriate jobs. This effectively represented an >> alternative downstream mapping mechanism. It worked for it's purpose and >> is still in production today. In the end, we looked back and squinted at >> it, and could see that with a few very deep, yet reasonable (likely >> non-breaking) changes to the upstream/downstream system , Jenkins could do >> the same logic natively. That largely led to this thread. Right now, I'm >> engineering a solution for a different use case which is similar-in-scope, >> related to the topic, yet different enough to learn some new things. At >> the end of this, I think I will have an even better mix of perspectives to >> guide me through a JEP. I apologize in advance for bothering everyone in >> the future with my struggles on creating the reference implementation. >> >> To everyone reading, I would still like to collect support for this >> effort in terms of votes and comments and other peoples struggles in the >> Issue I linked. >> >> Regards, >> Jerry >> >> Gerald R. Wiltse >> jerrywil...@gmail.com >> >> >> >> On Mon, May 18, 2020 at 5:55 PM Jesse Glick wrote: >> >>> As far as I know there is no serious work in progress in this area, >>> and no particular plan for work on it from the “core team” (maybe a >>> misleading phrase). >>> >>> Indeed `DependencyGraph` as currently defined is very rigid and could >>> not work well even for moderately subtle Pipeline scenarios, so it >>> does not seem worth trying to adapt. >>> >>> You can define more sophisticated variants of `ReverseBuildTrigger` in >>> plugins, though I would tend to discourage doing this sort of thing at >>> the Jenkins level to begin with. Instead it is likely more scalable to >>> have “downstream” builds be triggered by some external event, such an >>> artifact appearing in Nexus or an image in a Docker registry. >>> >>> Alternatively, you can keep trigger management outside of component >>> Pipelines altogether, defining some sort of orchestration project that >>> uses the `build` step internally but in a computed graph. Or this >>> orchestration can be done by external tools designed for that purpose, >>> for example using the Jenkins REST API to trigger builds. >>> >>> If some larger and more intrusive concept of dependency graphs needs >>> to make its way into fundamental APIs so that a variety of plugins can >>> interoperate based on a common understanding of project relationships >>> (for example so the graph can be displayed in build visualizations), >>> then someone would need to file a JEP for it and commit to writing a >>> reference implementation and driving integrations. The added >>> complexity would need to be justified by new abilities that a lot of >>> people could enjoy without too much migration effort. >>> >>> Some inertia stems from the fact there is no obvious, straightforward, >>> single best practice for doing CI when you have hundreds of >>> interrelated components. Some organizations use a monorepo and use >>> various tools to cache partial build results. Others prefer microrepos >>> with subtle triggering relationships and special workflows. The build >>> system often frames the problem. If you have a particular model in >>> mind then you are in a position to sketch a tool which would help you >>> and others in the same situation. >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Jenkins Developers" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to jenkinsci-dev+unsubscr...@googlegroups.com. >>> To view this discussion on the web visit >>>
Re: Diamond Dependencies and Upstreams at Scale
It's worth noting that I think a lot of good code and abstraction and capturing of the problem already exists here: https://plugins.jenkins.io/join/ I'm not sure if Stefan Wolf still watches this list or actively contributes to Jenkins, but his insight would probably be invaluable. I will study his work and try to reach out to him. Regards, Gerald R. Wiltse jerrywil...@gmail.com On Tue, May 19, 2020 at 10:30 AM Gerald Wiltse wrote: > I find this feedback very encouraging. It definitely does seem to be a > good candidate for JEP proposal. I will plan for that in the > mid-term future. > > Your suggestions about alternatives are all right on. In one large > environment, I created a solution with a metajob that received webhooks > from all jobs, used the Jenkins REST API's to query "all job > configurations" and then correlate the hooks to metadata, and use build() > to trigger the appropriate jobs. This effectively represented an > alternative downstream mapping mechanism. It worked for it's purpose and > is still in production today. In the end, we looked back and squinted at > it, and could see that with a few very deep, yet reasonable (likely > non-breaking) changes to the upstream/downstream system , Jenkins could do > the same logic natively. That largely led to this thread. Right now, I'm > engineering a solution for a different use case which is similar-in-scope, > related to the topic, yet different enough to learn some new things. At > the end of this, I think I will have an even better mix of perspectives to > guide me through a JEP. I apologize in advance for bothering everyone in > the future with my struggles on creating the reference implementation. > > To everyone reading, I would still like to collect support for this effort > in terms of votes and comments and other peoples struggles in the Issue I > linked. > > Regards, > Jerry > > Gerald R. Wiltse > jerrywil...@gmail.com > > > > On Mon, May 18, 2020 at 5:55 PM Jesse Glick wrote: > >> As far as I know there is no serious work in progress in this area, >> and no particular plan for work on it from the “core team” (maybe a >> misleading phrase). >> >> Indeed `DependencyGraph` as currently defined is very rigid and could >> not work well even for moderately subtle Pipeline scenarios, so it >> does not seem worth trying to adapt. >> >> You can define more sophisticated variants of `ReverseBuildTrigger` in >> plugins, though I would tend to discourage doing this sort of thing at >> the Jenkins level to begin with. Instead it is likely more scalable to >> have “downstream” builds be triggered by some external event, such an >> artifact appearing in Nexus or an image in a Docker registry. >> >> Alternatively, you can keep trigger management outside of component >> Pipelines altogether, defining some sort of orchestration project that >> uses the `build` step internally but in a computed graph. Or this >> orchestration can be done by external tools designed for that purpose, >> for example using the Jenkins REST API to trigger builds. >> >> If some larger and more intrusive concept of dependency graphs needs >> to make its way into fundamental APIs so that a variety of plugins can >> interoperate based on a common understanding of project relationships >> (for example so the graph can be displayed in build visualizations), >> then someone would need to file a JEP for it and commit to writing a >> reference implementation and driving integrations. The added >> complexity would need to be justified by new abilities that a lot of >> people could enjoy without too much migration effort. >> >> Some inertia stems from the fact there is no obvious, straightforward, >> single best practice for doing CI when you have hundreds of >> interrelated components. Some organizations use a monorepo and use >> various tools to cache partial build results. Others prefer microrepos >> with subtle triggering relationships and special workflows. The build >> system often frames the problem. If you have a particular model in >> mind then you are in a position to sketch a tool which would help you >> and others in the same situation. >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Jenkins Developers" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to jenkinsci-dev+unsubscr...@googlegroups.com. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr3oAz%2BFL%3D8FOXufOe0frmXiKeTP7WD9Jis%2BkwFCAkVvLw%40mail.gmail.com >> . >> > -- You received this message because you are subscribed to the Google Groups "Jenkins Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAML1RCAtwPJs8gq%3DCKzdefMpHjubjjRF-KM%3DX6%2B0yMs-C22w2g%40mail.gmail.com.
Re: Diamond Dependencies and Upstreams at Scale
I find this feedback very encouraging. It definitely does seem to be a good candidate for JEP proposal. I will plan for that in the mid-term future. Your suggestions about alternatives are all right on. In one large environment, I created a solution with a metajob that received webhooks from all jobs, used the Jenkins REST API's to query "all job configurations" and then correlate the hooks to metadata, and use build() to trigger the appropriate jobs. This effectively represented an alternative downstream mapping mechanism. It worked for it's purpose and is still in production today. In the end, we looked back and squinted at it, and could see that with a few very deep, yet reasonable (likely non-breaking) changes to the upstream/downstream system , Jenkins could do the same logic natively. That largely led to this thread. Right now, I'm engineering a solution for a different use case which is similar-in-scope, related to the topic, yet different enough to learn some new things. At the end of this, I think I will have an even better mix of perspectives to guide me through a JEP. I apologize in advance for bothering everyone in the future with my struggles on creating the reference implementation. To everyone reading, I would still like to collect support for this effort in terms of votes and comments and other peoples struggles in the Issue I linked. Regards, Jerry Gerald R. Wiltse jerrywil...@gmail.com On Mon, May 18, 2020 at 5:55 PM Jesse Glick wrote: > As far as I know there is no serious work in progress in this area, > and no particular plan for work on it from the “core team” (maybe a > misleading phrase). > > Indeed `DependencyGraph` as currently defined is very rigid and could > not work well even for moderately subtle Pipeline scenarios, so it > does not seem worth trying to adapt. > > You can define more sophisticated variants of `ReverseBuildTrigger` in > plugins, though I would tend to discourage doing this sort of thing at > the Jenkins level to begin with. Instead it is likely more scalable to > have “downstream” builds be triggered by some external event, such an > artifact appearing in Nexus or an image in a Docker registry. > > Alternatively, you can keep trigger management outside of component > Pipelines altogether, defining some sort of orchestration project that > uses the `build` step internally but in a computed graph. Or this > orchestration can be done by external tools designed for that purpose, > for example using the Jenkins REST API to trigger builds. > > If some larger and more intrusive concept of dependency graphs needs > to make its way into fundamental APIs so that a variety of plugins can > interoperate based on a common understanding of project relationships > (for example so the graph can be displayed in build visualizations), > then someone would need to file a JEP for it and commit to writing a > reference implementation and driving integrations. The added > complexity would need to be justified by new abilities that a lot of > people could enjoy without too much migration effort. > > Some inertia stems from the fact there is no obvious, straightforward, > single best practice for doing CI when you have hundreds of > interrelated components. Some organizations use a monorepo and use > various tools to cache partial build results. Others prefer microrepos > with subtle triggering relationships and special workflows. The build > system often frames the problem. If you have a particular model in > mind then you are in a position to sketch a tool which would help you > and others in the same situation. > > -- > You received this message because you are subscribed to the Google Groups > "Jenkins Developers" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to jenkinsci-dev+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr3oAz%2BFL%3D8FOXufOe0frmXiKeTP7WD9Jis%2BkwFCAkVvLw%40mail.gmail.com > . > -- You received this message because you are subscribed to the Google Groups "Jenkins Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAML1RCB636jwU3ZodNFvk08_XoGLb7jrSs2wit-7j5dXiuVN%2Bw%40mail.gmail.com.
Re: Diamond Dependencies and Upstreams at Scale
As far as I know there is no serious work in progress in this area, and no particular plan for work on it from the “core team” (maybe a misleading phrase). Indeed `DependencyGraph` as currently defined is very rigid and could not work well even for moderately subtle Pipeline scenarios, so it does not seem worth trying to adapt. You can define more sophisticated variants of `ReverseBuildTrigger` in plugins, though I would tend to discourage doing this sort of thing at the Jenkins level to begin with. Instead it is likely more scalable to have “downstream” builds be triggered by some external event, such an artifact appearing in Nexus or an image in a Docker registry. Alternatively, you can keep trigger management outside of component Pipelines altogether, defining some sort of orchestration project that uses the `build` step internally but in a computed graph. Or this orchestration can be done by external tools designed for that purpose, for example using the Jenkins REST API to trigger builds. If some larger and more intrusive concept of dependency graphs needs to make its way into fundamental APIs so that a variety of plugins can interoperate based on a common understanding of project relationships (for example so the graph can be displayed in build visualizations), then someone would need to file a JEP for it and commit to writing a reference implementation and driving integrations. The added complexity would need to be justified by new abilities that a lot of people could enjoy without too much migration effort. Some inertia stems from the fact there is no obvious, straightforward, single best practice for doing CI when you have hundreds of interrelated components. Some organizations use a monorepo and use various tools to cache partial build results. Others prefer microrepos with subtle triggering relationships and special workflows. The build system often frames the problem. If you have a particular model in mind then you are in a position to sketch a tool which would help you and others in the same situation. -- You received this message because you are subscribed to the Google Groups "Jenkins Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr3oAz%2BFL%3D8FOXufOe0frmXiKeTP7WD9Jis%2BkwFCAkVvLw%40mail.gmail.com.
Diamond Dependencies and Upstreams at Scale
TLDR; This ticket suggests that Jenkins team open a discussion and planning to spend significant times on the problems mentioned above. If you have experienced issues with diamond dependencies between jenkins jobs, or want more features around upstream() and downstream() dependency handling, please read on and consider adding some comments/votes in this ticket below. There are a lot of other related issues where you may have commented in the past. Please use this ticket to update/aggregate those comments (maybe you've worked around the issue since, or found ). https://issues.jenkins-ci.org/browse/JENKINS-57648 Disclaimer: The message below is based on my teams current understanding of Jenkins and the current Plugin ecosystem for Pipeline jobs specifically. The call to action is based on this understanding. It's possible that there's some critical piece of information which we are not aware of, which renders this discussion completely or partially invalid. If so, we apologize, please let us know and help guide the discussion forward. Introduction: Jenkins upstream and downstream triggers seem to be a critical and fundamental abstraction for Jenkins when used at enterprise scale. In the abstract sense, they are the declarative primitives for defining a "dependency graph". There's another non-declarative primitive, the build() step, which we can consider a "dynamic"/"runtime" version of downstream(). There are a number of Jira Issues surrounding features for the job dependency graph referenced in the citations at the bottom of this message. Some points in this message are borrowed from those issues. For the purpose of this message, we'll define one acronym: - "JDG" = job dependency graph Focus: This message is focused on the interaction between two related concepts: - Upstream job triggers in pipelines - The diamond dependency problem (locking/blocking/de-duplicating/etc of downstream builds) History: Jenkins has a very robust API for plugins and jobs to work with the job dependency graph. This API evolved over a long period of time and became very mature. Unfortunately, for the new pipeline model, this entire body of work has been rendered unusable. Plugins based on DependencyGraph do not work with pipelines. So, currently, in terms of "dependency graph" features for pipelines, Jenkins lost a lot of ground. If we had a number of the old API features on the new model, it's likely this message would be unnecessary. There's a decent reason for this gap. It seems virtually impossible to update the existing API to support the new model, especially in a way that would be non-breaking. Also, the old DependencyGraph API had some fundamental limitations which had been identified and needed to be overcome anyway. So, it makes more sense to re-think any DependencyGraph-related features for pipelines, and consult the old API for guidance where applicable. With so much going on in Jenkins evolution, a massive project like a "new from-scratch dependency graph API" this is not easy to justify without overwhelming public support which simply hasn't existed yet. The Hard Part for Jenkins: It turns out that an overwhelming number of non-trivial pipeline cases rely on the "build()" step as the cornerstone of constructing their pipelines because it offers the unique power and flexibility of controlling the entire build tree from a single "job". It seems likely that this is one major contributing factor to the lack of public demand for more attention to supporting API's for working with upstream() and downstream() constructs. Without conditional logic, upstream() and downstream() declarations just seem like non-starters compared to using the "build()" step. Many organizations are now likely to be deeply entrenched in pipelines based on build() step and dis-incentivized to rethink them, or be involved with this discussion. The Hard Part for Jenkins Users: The hardest cases are enterprise organizations with large, complex JDG's with many hundreds of inter-related components. In these environments, designing, maintaining, and evolving the JDG is one of the most significant and challenging aspects of the Jenkins administration role. Smaller organizations have many of the same challenges, but design compromises and imperfections are less painful and costly. A lot of "annoyances" are acceptable up to a certain scale. BUT, at some scale become "completely unmanageable", and users in those environments are the ones targeted in this message. At some scale, the build() step goes from being a powerful, flexible "bread and butter" feature, to becoming the "GOTO" of CI. It encourages large monolithic pipelines which are difficult to debug and manage, and which are anathema to goal of modularity of components and their builds. In the OSS world, each component has separate builds on one-or-more build services because that's a good general strategy for building components. This is not to say that the build() step is bad and we
