| The install-plugins.sh script will always download the latest version of all of the dependencies of a specified plugin. This lead to a couple issues: in some cases, the latest version of a dependency would be incompatible with the user's version of Jenkins; because the "latest" is dependent on whatever is currently in the update center at the time, the set of plugins that is installed may not be easily reproduced. The new plugin management CLI tool does not by default download the latest versions of dependent plugins - instead, it downloads the versions of dependent plugins listed either in the update center metadata or the plugin manifest file. Although this leads to better reproducibility, the dependent plugin versions are often conservative, so the user may get a set of plugins that's far from the most up-to-date. Ideally, the user should be able to have both options. Below are some of the conversations and points that have been brought up involving this issue: stopalopa @stopalopa Aug 22 20:17
@casz RE: PR 61 Honestly I had also been thinking about adding another CLI option where users could specify that they wanted the latest of all plugins _and_their dependencies. I think right now they can specify that they want the latest version of a particular plugin but the version of the dependencies that are downloaded are dependent on whatever is in the update center metadata, which might not be the latest. Thoughts?
Joseph Petersen
@caszAug 22 20:19
both for yaml and plugin.txt it should be very easy to assume latest if version or second position in split is not provided ![]()
you could also add --update-version-to-latest which will find the newest version and put it into the file for you
stopalopa
@stopalopaAug 22 20:20
I guess my question is, how important is it that if you specify that you want the latest version of a particular plugin, that the dependenices of that plugin are also the latest version?
Joseph Petersen
@caszAug 22 20:21
Even for dependencies, I would also say the latest is ![]() that's mostly how plugins.sh worked.
but that's my opinion ![]()
I'd prefer to deal with the problem upfront instead of months or years down the line.
stopalopa
@stopalopaAug 22 20:24
I think that was also one of the issues with install-plugins.sh - people wanted a little bit more control of the plugin versions that got installled.
Joseph Petersen
@caszAug 22 20:27
But it shouldn't be too hard to add the option to download latest dependencies as well ![]()
Kristin Whetstone
@kwhetstoneAug 22 23:54
Right, the whole point is people wanted more control so just downloading the latest version isn't very helpful
Joseph Petersen
@caszAug 23 07:41
okay excluding the jenkinsci/plugin-installation-manager-tool#61
the master branch of the tool gives me a working instance but somewhat sad that I have to specify that I want the latest of icon-shim or other such plugins.
Or latest of pubsub-light without specifying it ![]()
stopalopa
@stopalopaAug 23 07:45
Ok…so the way you would want it to work would be that if a user specifies “latest” for a plugin, it would download the latest version of that plugin and the latest version of all of its dependencies by default, and if the user doesn’t want the latest version of the dependencies, they should have to specify that?
Kristin Whetstone
@kwhetstoneAug 23 09:20
I think so
stopalopa
@stopalopaAug 23 09:21
Ok. And is that something that should be included in the first official version?
Kristin Whetstone
@kwhetstoneAug 23 09:35
Paart of me leans towards downloading latest plugins is actually against part of core of the tool. Part of the point is that it pulls in the specified versions of plugins across the board. {{-useLatest}}could be a flag to signify that that you want to download the latest version of the dependencies of ones without a version.
This is a tough one; I'm trying to think of where the behavior would most factor in.
Tim Jacomb
@timjaAug 23 09:38
I wouldn't want an old dependency pulled in...
Plugins tend to have very conservative dependency versions. But I also specify all plugins in my file so I get the versions of what I want ![]()
Kristin Whetstone
@kwhetstoneAug 23 09:39
Yeah, but an "old dependency" wouldn't get pulled in right? The tool has already calculated the newest possible dependency
Tim Jacomb
@timjaAug 23 09:39
wouldn't that be latest then?
Martin d'Anjou
@martindaAug 23 09:39
I think the way I would use this is 1) use Jenkins LTS and get all latest plugins. Then I would list the plugin versions, and create a "frozen list" with all plugin versions hardcoded.
Kristin Whetstone
@kwhetstoneAug 23 09:39
Not always
Tim Jacomb
@timjaAug 23 09:40
that's roughly what I do @martinda
Kristin Whetstone
@kwhetstoneAug 23 09:40
That's how I thought most people were going to use it.
Which is why I thought a --useLatest flag (or some sort of config) would be a great for the other case
Martin d'Anjou
@martindaAug 23 09:42
Do you mean --use-latest instead of having to explicitely say latest for every plugin?
Kristin Whetstone
@kwhetstoneAug 23 09:44
no --use-latest for downloading the latest versions of the dependencies of a plugin with version latest
The other main usecase of the tool is going to be someone saying generically "I want these plugins installed" and they might not care about versions, so they're going to just want behavior similar to how Jenkins calculates plugin dependencies today
Martin d'Anjou
@martindaAug 23 09:50
I do not know how Jenkins calculates dependencies today... does it ignore dependencies versions? Like if I ask for plugin-a:1.1.0, does it use 1.1.0 version for plugin-a with "latest" for its dependencies, or does it use the dependency versions as specified by plugin-a:1.1.0's pom.xml ?
Kristin Whetstone
@kwhetstoneAug 23 09:51
Jenkins takes the latest version of everything, it doesn't respect you requesting plugin versions
Kristin Whetstone
@kwhetstoneAug 23 10:01
So since in this tool, the focus has been on specific versions of plugins, I think it makes sense to download latest for dependencies of latest as an option, not the default
stopalopa
@stopalopaAug 23 11:19
Ok, so it seems like we all agree that users want both the ability to control which plugins get installed as well also having the ability to get all of the latest of the plugins they specify and the dependencies if they - the disagreement is about what the default should be. I’m not sure how to make a decision. Take a vote? Reach out to the dev mailing list?
Kristin Whetstone
@kwhetstoneAug 23 11:23
I'd use a flag. Let's keep it from getting too complicated. We can all vote here; give the people who have been involved early on some preference ![]()
stopalopa
@stopalopaAug 23 11:31
It sounded like @casz wanted getting the latest dependencies to be the default and have a flag only if a user didn’t want that. Opinions @martinda, @timja, @slide or anyone else?
Alex Earl
@slideAug 23 11:36
If not specified explicitly, I would say get latest
Joseph Petersen @casz Aug 23 17:59
@stopalopa I have a very good use case for when I want latest... I always want the latest blue ocean... Doesn't matter but the problem is blue ocean has such a huge dependency tree and usually they aren't that up to date. I would hate to have redundant entries for all blue ocean dependencies to just say I want the latest of blue ocean.
Joseph Petersen @casz Aug 23 18:04
what if you could use the .yaml file to provide conditions instead of only being cli based?
so you could have downloadStrategy: latest or something
or configure the logger from a yaml structure? Martin d'Anjou @martinda Aug 24 12:17
@stopalopa Just saw your ping. Sounds like there are two modes that people want the most: latest of everything, and at the opposite end of the spectrum, reproducibility through immutable dependencies. I think you are trying to answer the question of what should the default behavior be. For that I do not really care, IMO because it's new tool you can make the default whaterver you or others want, as long as both modes are available. In principle in the reproducible mode of operation, I should only have to list the top plugins versions and transitive dependencies should always be the same no matter when I run the tool. I hope I make sense. Thanks a lot. Kristin Whetstone @kwhetstone Aug 25 23:50
Yeah, I think that taking a param for determining that you want to use the latest download strategy is advisable. It doesn't change current behavior, and it can be added after doing a release. Kristin Whetstone @kwhetstone Aug 26 07:54
For some reason I was thinking the tool already read config options from a file, but it doesn't, and that's a good ticket Joseph Petersen @casz Aug 27 09:43
@kwhetstone @stopalopa I think one way to solve all of this is using a Lock file
I would very much like to just use latest and have a lock file generated I can commit into the repo ![]()
And I can in my plugins file always say I want to use git plugin 2.7.3 and the rest could be latest but I then ask plugin tool to generate my lock file which I can decide if it is too aggressive and dial it in to my needs and commit it to git and have plugin installer use the lock file ![]() Ok, thanks for the feedback. I’ll create a Jira ticket for that feature Jesse Glick @jglick Aug 23 13:37
For context, this is an example of a DiY system I have used in the past out of frustration with the limitations of install-plugins.sh (both the lack of cache sharing with Maven, and the fact that it picks up transitive plugin dependencies in nondeterministic versions acc. to whatever is on the UC that day): https://github.com/jenkinsci/parallel-test-executor-plugin/blob/f50c3d8b8a7d3223e8575247ff4303975aa293e0/demo/pom.xml#L32-L213 (which could now be simplified using https://github.com/jenkinsci/bom/#usage) and https://github.com/jenkinsci/parallel-test-executor-plugin/blob/f50c3d8b8a7d3223e8575247ff4303975aa293e0/demo/Makefile#L8-L17 stopalopa @stopalopa Aug 23 13:59
@jglick for your point about
fact that it picks up transitive plugin dependencies in nondeterministic versions acc. to whatever is on the UC that day
this is because in install-plugins.sh by default pulls in the latest version of the dependencies? Jesse Glick @jglick Aug 23 15:06
Yes. I have a general aversion to any build process that might behave differently at different times! You should need to specify the versions of all plugins you want to include. (And it should be easy to update those versions to the latest available…so long as that becomes an SCM commit you can test in a pull request. Hence @dependabot.) Jesse Glick @jglick Aug 23 15:25
Of course you might want to make different tradeoffs depending on your typical use case. I was just letting you know about some prior art that might have been missed otherwise. stopalopa @stopalopa Aug 23 15:25
@jglick So your comments very much tie into some discussions we were having on the gitter channel earlier today and I’d love to get your feedback. The way I currently have it implemented is different from the original bash script in that instead of downloading the latest version of the transitive dependencies, the versions of the dependent plugins are determined by what’s specified in either the update center metadata or the plugin manifest for the original plugin that was requested. On the one hand, the versions of the listed dependent plugins listed are probably somewhat conservative and the user wouldn’t necessarily get the latest versions. On the other, it does lead to a consistent set of plugins across different runs. We were talking that we should probably also add the ability for users to specify that they want the latest of all plugins (or at least, if they specify the latest version of one plugin, then it dependencies should also be the latest). There was some debate about what the default behavior should be, so I’d like to get your Jesse Glick @jglick Aug 26 10:58
@stopalopa sorry for delayed response. My opinion is that a tool like this should always use deterministic plugin versions—i.e., given a particular configuration file as input, and assuming that metadata of released plugins is immutable (which Artifactory ought to enforce), you will always get the same output—unless the user explicitly opts into nondeterministic behavior and accepts the corresponding risk. The least-upper-bound behavior I think you describe as the current default seems reasonable as a starting point. Yes some of the dependencies are then going to be pretty old, and there is no deterministic way to get later versions of dependency plugins without mentioning them, so you need to think about how to make a good developer experience when the input file lists all plugins:
When this mode is opted into, signal an error if any transitive dependency is not explicitly listed.
Textually differentiate “feature” plugins that the user specifically wants, vs. plugins in that list that are only there as dependencies, and signal an error if there is a dependency entry which is not in the transitive dependency closure of any feature entry. (See discussion in JENKINS-53506.)
Make it possible to use @dependabot to get push notifications for newer versions of any plugin in the list. (Or, make it possible to reuse the BOM, which already uses Dependabot and runs plugin-compat-tester against the entire set to ensure a basic level of mutual compatibility.)
By the way see jenkins-x/jenkins-x-serverless#56 where JX in “Jenkinsfile-runner” (JFR) mode was having trouble keeping a reasonable list of plugins. |
