Thanks again for investigating. > I can try that out and if it's actually helpful add something to our > development docs. Ideally we have simple commands to run for all GLVs to > keep them updated in the future.
+1 for this approach for sure though :) On Fri, Apr 26, 2019 at 11:49 AM Florian Hockmann <[email protected]> wrote: > I just reached out to ASF Infra and that basically concluded this > discussion as they simply don't allow any GitHub bots that require write > access to the repo which includes dependabot unfortunately. > > We could only enable it on a fork and then forward the PRs from there to > the main repo which we could also automate in principal, but at least > for now this sounds like too much work. > > So, I guess we have to find other ways to keep the dependencies up to > date, especially for the GLVs. > > At least for .NET, I know that there is a nice tool we could use that > can list outdated dependencies and it should also be able to perform an > update: https://github.com/jerriep/dotnet-outdated > > I can try that out and if it's actually helpful add something to our > development docs. Ideally we have simple commands to run for all GLVs to > keep them updated in the future. > > Am 16.04.2019 um 13:39 schrieb Stephen Mallette: > >> We can define the target branch for the PRs. So, we could set that to > > tp33 and then merge though to master like we usually do it for other PRs. > > (Maybe it's also possible to create different configurations for the > > different branches if we want to get one PR per branch.) > > > > There are different dependencies on tp33 vs master so it would be nice to > > know for sure how they handle that. Of course, you allude to a > > proliferation of PRs that would result....ugh. Anyway, I guess I'd like > to > > know what the options really are there... > > > >> I assume that Apacha Infra will have to activate GitHub bots like > > dependabot for us as that requires ownership of the GitHub organization. > I > > hope that they can afterwards give us permissions to change the settings, > > but I'm not sure about that. Maybe these permissions also only work on an > > organization basis (which would include all ASF repos). Worst case would > be > > that ASF Infra has to configure such a bot for us. > > > > all good questions to answer. no one seem to really be objecting to this > > direction (i only want to know more about the details of how it will > work) > > so I guess we're at a point where you could try to move things forward > with > > infra and the bot folks to see how these final issues we're discussing > > would work out. please let us know what you find out. thanks. > > > > On Wed, Apr 10, 2019 at 9:52 AM Florian Hockmann <[email protected] > > > > wrote: > > > >>> always using the most recent has been disastrous in python. our build > >> breaks all the time with no changes from us because of that style where > we > >> don't pin to specific dependencies. i don't understand that model at > all. > >> i > >> know you're not saying that we blindly upgrade, i was just making a > point > >> about python that is semi-related. > >> > >> Understood and I agree completely that we should pin versions. > >> > >>> we're not always sure of what a change in dependency will bring in > terms > >> of change to the API but i agree that upgrades can take place (and as > you > >> pointed out at the start, are already taking place in a more manual > >> fashion). > >> > >> IF our dependencies use semantic versioning, then this shouldn't be a > >> problem as long as we don't do major version updates in our patch > >> versions. > >> > >>> Why no PR for spark, hadoop, etc? not that they would pass compilation > - > >> i'd expect failures, but I'm just wondering if you knew the reason it > >> didn't catch those? > >> > >> I also wondered about that as I had exactly the same expectation of > >> immediately getting a failed build for these dependencies. Maybe they > have > >> a list of dependencies where they don't attempt an update? We can of > >> course > >> ask them about this if we decide to use this bot. > >> > >>> it looks like it submits all PRs against master. what about tp33? will > >> we have to cherry-pick out of the PR to tp33, test and then merge > forward > >> to master? > >> > >> We can define the target branch for the PRs. So, we could set that to > tp33 > >> and then merge though to master like we usually do it for other PRs. > >> (Maybe > >> it's also possible to create different configurations for the different > >> branches if we want to get one PR per branch.) > >> > >>> i assume that you setup an account for dependabot that gets it > >> configured at our repo? I assume that the account is bound to your > github > >> account? > >> > >> Yes, I enabled dependabot for my account and could then give it access > to > >> my own repos which is a fork of our TinkerPop repo in this case. > >> > >>> can that dependabot dashboard for "tinkerpop" be accessed by others or > >> just your account? how will that work? > >> > >> I assume that Apacha Infra will have to activate GitHub bots like > >> dependabot for us as that requires ownership of the GitHub > organization. I > >> hope that they can afterwards give us permissions to change the > settings, > >> but I'm not sure about that. Maybe these permissions also only work on > an > >> organization basis (which would include all ASF repos). Worst case > would > >> be > >> that ASF Infra has to configure such a bot for us. > >> > >>> i don't see much information about what the dashboard allows users to > do > >> - could you summarize some of the options available and what we see > there? > >> > >> It allows to add different configurations for different languages or > >> directories in a repo. We can then configure the following aspects per > >> directory / language: > >> > >> Update schedule: daily / weekly / monthly > >> Directory (important for the GLVs as that directory should contain the > >> project file, like the .sln file for .NET) > >> Target branch > >> Filters: They currently only have a filter to only get security updates. > >> Automatic PR merging: Can be configured for only some versions (like > patch > >> versions, only security fixes, ...) and only for specific dependencies. > (I > >> don't think that we want automatic merging.) > >> For the GitHub PRs, we can also add default reviewers, assignees, and > >> labels. > >> > >> On top of that, the UI also allows to manually trigger an update run, > >> irrespective of the configured update schedule. I used that on the first > >> day as dependabot by default only creates 5 PRs per directory/language > and > >> per update run, but I wanted to get more PRs without having to wait > >> several > >> days. > >> > >> -----Ursprüngliche Nachricht----- > >> Von: Stephen Mallette <[email protected]> > >> Gesendet: Mittwoch, 10. April 2019 14:41 > >> An: [email protected] > >> Betreff: Re: Using a bot to keep dependencies up to date > >> > >>> I'm still not that experienced with Java/Maven, but the reasoning is > >>> that we want to avoid dependency version conflicts for our users, > right? > >>> > >> well, we just try to keep things stable as possible without introducing > a > >> breaking change by way of a dependency. it's not to say that we never do > >> it, but it's always with done with careful consideration. > >> > >> > >>> In that case, it's specific to Java I'd say. At least for .NET, I > >>> would expect a library to use recent versions of its dependencies. > >>> > >> always using the most recent has been disastrous in python. our build > >> breaks all the time with no changes from us because of that style where > we > >> don't pin to specific dependencies. i don't understand that model at > all. > >> i > >> know you're not saying that we blindly upgrade, i was just making a > point > >> about python that is semi-related. > >> > >> > >>> Since we add new features in our "patch versions" (we aren't doing > >>> semantic versioning so those aren't actually patch versions), I think > >>> it's also OK in general to update dependencies in those versions, at > >>> least to a higher patch version, but I think a higher minor version > >>> should also be OK if we also add features ourselves in those versions. > >>> > >> We allow new features into patch versions because we control the new > >> features and make choices as to whether they are "safe" or not. we're > not > >> always sure of what a change in dependency will bring in terms of > change > >> to > >> the API but i agree that upgrades can take place (and as you pointed > out > >> at > >> the start, are already taking place in a more manual fashion). Again, I > >> just feel like we need to take each upgrade on a case by case basis. > >> > >> > >>>> We also typically have JIRAs for version bumps of dependencies - not > >>> always plugins - so if we continued with that those would be need to > >>> be created and CHANGELOG entries added manually after a merge i guess. > >>> > >>> Do we need a JIRA + CHANGELOG entry for all dependency updates or only > >>> for important ones? > >>> If we need them, then the bot at least notifies us of an available > >>> update and provides a PR for the update itself. We can then still > >>> create a JIRA issue + CHANGELOG entry manually if necessary. > >>> The notification should also include fixes for security problems in > >>> which case the bot sometimes also includes this information directly in > >> the PR. > >> maybe we just need it for "important" ones. i guess that's how we've > been > >> doing it so far....... > >> > >> > >>>> kuppitz makes the good point of the mess Travis has been in lately. > >>> Is that a reason against using a bot for automation or am I > >>> misunderstanding your/Daniel's point here? If a Travis build fails > >>> non-deterministically for a PR created by a bot, then I would simply > >>> trigger a rebuild manually. > >>> > >> i guess that's just us railing about travis........not a reason to avoid > >> automation. > >> > >> > >>> We could also use such a bot only for the GLVs / some of the GLVs if > >>> we don't want to update dependencies in general outside of major > >>> releases (like 3.5.0). At least for .NET I would appreciate such a > >>> bot, but I can't say much for Python / JavaScript as I don't know > >>> enough about how they handle dependency updates. > >> > >> if we use this, i think we should use it across the project. i looked at > >> the list of PRs that were generated by it in your fork more carefully > and > >> it was curious that the larger version bumps weren't present. Why no PR > >> for > >> spark, hadoop, etc? not that they would pass compilation - i'd expect > >> failures, but I'm just wondering if you knew the reason it didn't catch > >> those? > >> > >> some more questions: > >> > >> 1. I see the list of commands that dependabot has - it looks like it > >> submits all PRs against master. what about tp33? will we have to > >> cherry-pick out of the PR to tp33, test and then merge forward to > master? > >> or is there something easier? > >> 2. i assume that you setup an account for dependabot that gets it > >> configured at our repo? I assume that the account is bound to your > github > >> account? can that dependabot dashboard for "tinkerpop" be accessed by > >> others or just your account? how will that work? > >> 3. i don't see much information about what the dashboard allows users > to > >> do > >> - could you summarize some of the options available and what we see > there? > >> > >> thanks! > >> > >> > >> On Wed, Apr 10, 2019 at 7:59 AM Florian Hockmann < > [email protected]> > >> wrote: > >> > >>>> we don't usually upgrade them in a release line unless > >>> I'm still not that experienced with Java/Maven, but the reasoning is > >>> that we want to avoid dependency version conflicts for our users, > right? > >>> > >>> In that case, it's specific to Java I'd say. At least for .NET, I > >>> would expect a library to use recent versions of its dependencies. > >>> > >>> Since we add new features in our "patch versions" (we aren't doing > >>> semantic versioning so those aren't actually patch versions), I think > >>> it's also OK in general to update dependencies in those versions, at > >>> least to a higher patch version, but I think a higher minor version > >>> should also be OK if we also add features ourselves in those versions. > >>> > >>>> We also typically have JIRAs for version bumps of dependencies - not > >>> always plugins - so if we continued with that those would be need to > >>> be created and CHANGELOG entries added manually after a merge i guess. > >>> > >>> Do we need a JIRA + CHANGELOG entry for all dependency updates or only > >>> for important ones? > >>> If we need them, then the bot at least notifies us of an available > >>> update and provides a PR for the update itself. We can then still > >>> create a JIRA issue + CHANGELOG entry manually if necessary. > >>> The notification should also include fixes for security problems in > >>> which case the bot sometimes also includes this information directly in > >> the PR. > >>>> kuppitz makes the good point of the mess Travis has been in lately. > >>> Is that a reason against using a bot for automation or am I > >>> misunderstanding your/Daniel's point here? If a Travis build fails > >>> non-deterministically for a PR created by a bot, then I would simply > >>> trigger a rebuild manually. > >>> > >>> We could also use such a bot only for the GLVs / some of the GLVs if > >>> we don't want to update dependencies in general outside of major > >>> releases (like 3.5.0). At least for .NET I would appreciate such a > >>> bot, but I can't say much for Python / JavaScript as I don't know > >>> enough about how they handle dependency updates. > >>> > >>> -----Ursprüngliche Nachricht----- > >>> Von: Stephen Mallette <[email protected]> > >>> Gesendet: Mittwoch, 3. April 2019 20:27 > >>> An: [email protected] > >>> Betreff: Re: Using a bot to keep dependencies up to date > >>> > >>> hmm - so far i've been the bot. i run maven commands to find out what > >>> needs upgrading and then make decisions on when to do those upgrades > >>> as we don't usually upgrade them in a release line unless: > >>> > >>> 1. they are maven plugins > >>> 2. they are fixes for security problems 3. they introduce something > >>> important for our code base - like a performance fix or other > >>> enhancements > >>> > >>> That said, my maven commands only deal with Java and not the GLVs so > >>> not much attention has been paid there unfortunately now that I think > >> about it. > >>> We also typically have JIRAs for version bumps of dependencies - not > >>> always plugins - so if we continued with that those would be need to > >>> be created and CHANGELOG entries added manually after a merge i guess. > >>> > >>> I'm not completely against the idea, just pointing out some points of > >>> friction with what we currently do and kuppitz makes the good point of > >>> the mess Travis has been in lately. Utterly useless the last few > >>> weeks. If we didn't use a bot, at minimum it would be nice to document > >>> the manual methods for getting reports for the GLVs that show upgrade > >>> paths (I assume that those exist for other language ecosystems as they > >>> do for maven). it would be nice to hear additional thoughts on the > >> matter. > >>> Florian, thanks for taking the time to look into improving our build > >>> process. > >>> > >>> > >>> > >>> On Wed, Apr 3, 2019 at 1:29 PM Daniel Kuppitz <[email protected]> wrote: > >>> > >>>> Pretty cool, I like that (if only Travis would be a little more > >>> reliable). > >>>> Cheers, > >>>> Daniel > >>>> > >>>> > >>>> On Wed, Apr 3, 2019 at 9:43 AM Florian Hockmann > >>>> <[email protected]> > >>>> wrote: > >>>> > >>>>> Hi, > >>>>> > >>>>> we have a lot of dependencies in TinkerPop in different projects > >>>>> and even across different languages. That makes it hard to keep > >>>>> them updated which sometimes has security implications. > >>>>> > >>>>> I recently noticed that other open source projects use a bot that > >>>>> regularly checks whether any updates are available for their > >>>>> dependencies and then creates one PR per dependency. Just to try > >>>>> it out with TinkerPop, I activated such a bot on my fork: > >>>>> > >>>>> https://github.com/florianhockmann/tinkerpop/pulls > >>>>> > >>>>> and the overall result looks quite good in my opinion. It created > >>>>> a lot of PRs* and most could probably be directly merged. The bot > >>>>> can also be easily configured just by adding comments to its PR, > >>>>> for example to ignore a certain (major/minor/patch) version of a > >>> dependency: > >>>>> > >>>> https://github.com/FlorianHockmann/tinkerpop/pull/24#issuecomment-47 > >>>> 39 > >>>> 36360 > >>>>> What do you think about adding such a bot for our repo? > >>>>> > >>>>> > >>>>> * This is limited to only 5 PRs per day at first to not overwhelm > >>>>> a project with PRs. > >>>>> > >>>>> > >>>>> > >>> > >> > >
