Re: Third party dependencies, APIs, and Jenkins: Proposals

[Jesse Glick]

On Wed, Apr 10, 2019 at 6:44 PM Stephen Connolly
 wrote:
> If the “classboxed” dependencies have scope=provided then they shouldn’t be 
> transitive and you could leave them unshaded.

True. `war/pom.xml` would need to be massaged to bundle them.

> The JenkinsRule would just need to setup the classloaders from the war and 
> all would be fine

That “just” is JENKINS-41827. I believe writing such a harness is
feasible—I have done so in the past (see link in JIRA)—but migrating
existing functional tests to run in a new harness may not be
straightforward.

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr3QdyFTegLWWYxuuro6GFhim9QFk1D3_GuAAK61p1Rwbg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Third party dependencies, APIs, and Jenkins: Proposals

[Jesse Glick]

On Thu, Apr 11, 2019 at 4:10 AM Ullrich Hafner  wrote:
> [shading does not] prevent developers to use the shaded classes, they are 
> still there, just with another package name

True, but the Jenkins plugin class loader can refuse to serve such
classes, so any such mistake would immediately fail at runtime.

> We can add an architecture test or check style rule to prevent those imports 
> though.

Yes, you can use the Enforcer plugin to check this sort of thing at build time.

> Maybe we can use an enforcer rule […] that checks that no transitive 
> dependencies are used at all? Then developers need to provide explicit 
> dependencies.

We may want to do this anyway. (The Maven plugin for NetBeans modules
enforces such a rule.)

> The only problem I see here is if classes of the dependency are part of the 
> plugin API.

You can use explicit dependencies when an API includes other APIs in
its signatures. It just means that clients need to declare
dependencies on both, or they will not be able to compile.

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr0bxVjsVgN5dt7cH7Ohdi%2Bg0VZ79VKTte%3DnssT6MocdUA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Third party dependencies, APIs, and Jenkins: Proposals

[Stephen Connolly]

On Wed 10 Apr 2019 at 21:55, Jesse Glick  wrote:

> On Wed, Apr 10, 2019 at 10:48 AM Matt Sicker 
> wrote:
> > 1. Shade in third party dependencies of Jenkins core into Jenkins with
> > a package rename. This will allow core to use the dependencies, but
> > plugins will still need to include the dependencies explicitly. This
> > could potentially be combined with a compatibility shim for plugins
> > that try to load a class that doesn't exist.
>
> You do not necessarily need to rename packages. It suffices for the
> plugin class loader defined in Jenkins core to decline to forward
> class loading requests for these libraries to the parent. (The
> implementation already exists to some extent, in the
> `pluginFirstClassLoader` option, but this is too extreme because it
> applies to _all_ class loads whereas we only want to decline certain
> names.) Combined with the existing detached plugin system for backward
> compatibility, you have a complete runtime solution, and rather easily
> at that.
>
> The problems are in the developer tooling. `compiler:compile` and
> `JenkinsRule` via `surefire:test` are going to pick up transitive
> dependencies of `jenkins-core` in a flat classpath. Thus if a plugin
> built against a new core baseline (one after the change) _does_
> declare an explicit dependency on the new library wrapper plugin,
> Maven will pick one or the other version to build & test against
> (depending on POM details of where this dependency appears in the tree
> relative to `jenkins-core`); if it does _not_, Maven will silently
> build & test against the library as bundled in core, even though at
> runtime this would be a `NoClassDefFoundError`. These problems make me
> think that shading is more practical.


If the “classboxed” dependencies have scope=provided then they shouldn’t be
transitive and you could leave them unshaded.

The JenkinsRule would just need to setup the classloaders from the war and
all would be fine

>
>
> > 4.alt. I don't know enough about JPMS, but perhaps it can be used to
> > help enforce something similar?
>
> Last I checked it was much less flexible than what we already have in
> Jenkins core, so it would be hard to migrate to. (Requiring Java 11+
> would be the least of our worries.)
>
> --
> You received this message because you are subscribed to the Google Groups
> "Jenkins Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to jenkinsci-dev+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr285%2Bc7FnDwmes-H5T04kNX%2BhP5z_8JkY2heHYH_VXhzw%40mail.gmail.com
> .
> For more options, visit https://groups.google.com/d/optout.
>
-- 
Sent from my phone

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-dev/CA%2BnPnMzO%3D3Jxgzs_OaMP6w%3DsLccfZgAYycZbGW03MYepQ4ECoA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Third party dependencies, APIs, and Jenkins: Proposals

[Daniel Beck]

> On 10. Apr 2019, at 22:55, Jesse Glick  wrote:
> 
> You do not necessarily need to rename packages. It suffices for the
> plugin class loader defined in Jenkins core to decline to forward
> class loading requests for these libraries to the parent.

https://issues.jenkins-ci.org/browse/JENKINS-30685

> The problems are in the developer tooling. `compiler:compile` and
> `JenkinsRule` via `surefire:test` are going to pick up transitive
> dependencies of `jenkins-core` in a flat classpath. Thus if a plugin
> built against a new core baseline (one after the change) _does_
> declare an explicit dependency on the new library wrapper plugin,
> Maven will pick one or the other version to build & test against
> (depending on POM details of where this dependency appears in the tree
> relative to `jenkins-core`); if it does _not_, Maven will silently
> build & test against the library as bundled in core, even though at
> runtime this would be a `NoClassDefFoundError`. These problems make me
> think that shading is more practical.

https://issues.jenkins-ci.org/browse/JENKINS-41827

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-dev/55F43228-C285-4377-B4B9-DC9BD27565A7%40beckweb.net.
For more options, visit https://groups.google.com/d/optout.

Third party dependencies, APIs, and Jenkins: Proposals

[Matt Sicker]

I've been struggling for a while now on how to best approach
dependency management in Jenkins. One of our key pain points has been
in shared libraries like Jackson, HttpComponents, AsyncHTTPClient, and
various other commonly used libraries. As we've already found, simply
including these common dependencies in Jenkins itself leads to
problems such as never being able to upgrade that library again if it
breaks backward compatibility in the slightest.

In order to address these types of issues, I have four different proposals:

1. Shade in third party dependencies of Jenkins core into Jenkins with
a package rename. This will allow core to use the dependencies, but
plugins will still need to include the dependencies explicitly. This
could potentially be combined with a compatibility shim for plugins
that try to load a class that doesn't exist. This would allow for the
core dependencies to upgrade more easily as well. This is certainly a
hacky way of approaching it, but it's fairly common for low level
libraries to package rename their dependencies to avoid JAR hell.

2. Create facade APIs for common third party libraries such as a JSON
API, an HTTP client, etc. This approach could at least work the most
Jenkins-native way, though it seems like a rather large task.

3. Adopt JakartaEE (formerly known as JavaEE) tech via microprofile
 or similar. This is fairly similar to #2,
but instead, we delegate the design of these facade APIs to the
experts. I like this idea myself, though I'm not sure how well this
works with Jetty and Tomcat.

4. Utilize OSGi in the plugin system. We can dynamically add OSGi
metadata via bndtools at load time, and the hpi plugin could be
updated to include those metadata for faster startup. Hell, I'm pretty
sure the access modifier annotations could even be implemented via
OSGi bundle wiring. Based on my previous experience with OSGi, I
believe we would only be able to utilize up to and including the
module/bundle layer of OSGi. Higher level APIs from OSGi tend to have
duplicate functionality in Jenkins, so that's not necessary. The
advantages to this approach would be the ability to run multiple
versions of a library in the same JVM if necessary, the potential
ability to install and update plugins without restarting Jenkins (!),
and a proper shared library system we could use instead of needing to
package API plugins.

4.alt. I don't know enough about JPMS, but perhaps it can be used to
help enforce something similar? It would only be available in Jenkins
running on Java 9 or higher, though.

-- 
Matt Sicker
Senior Software Engineer, Jenkins Security, CloudBees

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-dev/CAEot4oxxFb%2BCFAttEeN51iqjCD%2BJc4QYh%2Bu%2BW3GsHk6v-vpkyw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.