My excuses Brian I had not seen your question: > - Will dependabot work ok with the version ranges that we specify? For example some Python dependencies have upper bounds for the next major version, some for the next minor version. Is dependabot smart enough to try bumping the appropriate version number?
Yes, it does and we can also explicitly set it to ignore certain versions or a all for each dependency if we don't want to have any PR upgrade for it. As a follow up on this I received an email from my Beam fork this morning reporting a CVE issue on one of the website dependencies, it is a moderate issue since this is a dep for the website generation, so it won't affect Beam users) but it is a clear example of the utility of dependabot. So the question is how do we proceed? Do I contact INFRA to enable it for the main repo? and more concretely how do we deal with these PRs in a practical sense? Do we rename them and create an associated JIRA for tracking? Other opinions? Ismaël On Fri, Apr 16, 2021 at 5:36 PM Brian Hulette <bhule...@google.com> wrote: > Yeah I can see the advantage in tooling like this for easy upgrades. I > suspect many of the outdated Python dependencies fall under this category, > but the toil of creating a PR and verifying it passes tests is enough of a > barrier that we just haven't done it. Having a bot create the PR and > trigger CI to verify it would be helpful IMO. > > Some questions/concerns I have: > - I think many python upgrades will still require manual work: > - We also have pinned versions for some Python dependencies in > base_image_requirements.txt [1] > - We test with multiple major versions of pyarrow. We'd want to add a > new test environment [2] when bumping to the next major version > - Will dependabot work ok with the version ranges that we specify? For > example some Python dependencies have upper bounds for the next major > version, some for the next minor version. Is dependabot smart enough to try > bumping the appropriate version number? > > Brian > > [1] > https://github.com/apache/beam/blob/master/sdks/python/container/base_image_requirements.txt > > [2] > https://github.com/apache/beam/blob/985e2f095d150261e998f58cf048e48a909d5b2b/sdks/python/tox.ini#L231 > > On Fri, Apr 16, 2021 at 7:16 AM Ismaël Mejía <ieme...@gmail.com> wrote: > >> Oh forgot to mention one alternative that we do in the Avro project, >> it is that we don't create issues for the dependabot PRs and then we >> search all the commits authored by dependabot and include them in the >> release notes to track dependency upgrades. >> >> On Fri, Apr 16, 2021 at 4:02 PM Ismaël Mejía <ieme...@gmail.com> wrote: >> > >> > > Quite often, dependency upgrade to latest versions leads to either >> compilation errors or failed tests and it should be resolved manually or >> declined. Having this, maybe I miss something, but I don’t see what kind of >> advantages automatic upgrade will bring to us except that we don’t need to >> create a PR manually (which is a not big deal). >> > >> > The advantage is exactly that, that we don't have to create and track >> > dependency updates manually, it will be done by the bot and we will >> > only have to do the review and guarantee that no issues are >> > introduced. I forgot to mention but we can create exception rules so >> > no further upgrades will be proposed for some dependencies e.g. >> > Hadoop, Netty (Java 11 flavor) etc. I forgot to mention another big >> > advantage that is the detailed security report that will help us >> > prioritize dependency upgrades. >> > >> > > Regarding another issue - it’s already a problem, imho. Since we have >> a one Jira per package upgrade now and usually it “accumulates” all package >> upgrades and it’s not closed once upgrade is done, we don’t have a reliable >> way to notify in release notes about all dependency upgrades for current >> release. One of the way is to mention the package upgrade in CHANGES.md >> which seems not very relible because it's quite easy to forget to do. I’d >> prefer to have a dedicated Jira issue for every upgrade and it will be >> included into releases notes almost automatically. >> > >> > Yes it seems the best for release note tracking to create the issue >> > and rename the PR title for this, but that would be part of the >> > review/merge process, so up to the Beam committers to do it >> > systematically and given how well we respect the commit naming / >> > squashing rules I am not sure if we will win much by having another >> > useless rule. >> > >> > On Fri, Apr 16, 2021 at 3:24 PM Alexey Romanenko >> > <aromanenko....@gmail.com> wrote: >> > > >> > > Quite often, dependency upgrade to latest versions leads to either >> compilation errors or failed tests and it should be resolved manually or >> declined. Having this, maybe I miss something, but I don’t see what kind of >> advantages automatic upgrade will bring to us except that we don’t need to >> create a PR manually (which is a not big deal). >> > > >> > > Regarding another issue - it’s already a problem, imho. Since we have >> a one Jira per package upgrade now and usually it “accumulates” all package >> upgrades and it’s not closed once upgrade is done, we don’t have a reliable >> way to notify in release notes about all dependency upgrades for current >> release. One of the way is to mention the package upgrade in CHANGES.md >> which seems not very relible because it's quite easy to forget to do. I’d >> prefer to have a dedicated Jira issue for every upgrade and it will be >> included into releases notes almost automatically. >> > > >> > > > On 16 Apr 2021, at 14:15, Ismaël Mejía <ieme...@gmail.com> wrote: >> > > > >> > > > Hello, >> > > > >> > > > Github has a bot that creates automatically Dependency Update PRs >> and >> > > > report security issues called dependabot. >> > > > >> > > > I was wondering if we should enable it for Beam. I tested it in my >> > > > personal Beam fork and it seems to be working well, it created >> > > > dependency updates for both Python and JS (website) dependencies. >> > > > The bot seems to be having problems to understand our gradle >> > > > dependency definitions for Java but that's something we can address >> in >> > > > the future to benefit of the updates. Also it did not propose >> go-lang >> > > > updates (probably for the same reason). >> > > > >> > > > If the community agrees I will create a ticket for INFRA to enable >> it. >> > > > We might be getting extra PRs (at the beginning) and we have to be >> > > > cautious about updates that might have unintended consequences for >> > > > example we should not merge non stable dependency updates (those >> > > > ending on -rc1 or -beta on Java) that >> > > > might be proposed or dependencies that committers are aware we >> should >> > > > not update for example projects where their main stable version is >> not >> > > > the most recent one like Hadoop or dependencies that do not support >> > > > our ongoing language target version (e.g. Java 11 only deps). >> > > > >> > > > Another issue is that these dependency updates might not get a JIRA >> > > > associated with them so we need to decide if (1) we create one and >> > > > rename/associate the PR with it, or (2) we just decide not to have >> > > > JIRAs for dependency updates. >> > > > >> > > > WDYT? other pros/cons that I can be missing? >> > > > >> > > > Ismaël >> > > >> >
