Re: Resolving classpath hell issues by allowing libraries to only use the version they request?
On 6 Jan 2015, at 12:29, Kevin Burton wrote: I agree. in this case the issue is testing. I need to embed cassandra so I can test it but it’s conflicting with spark. As a total side issue - have you considered using Docker? http://blog.xebia.com/2014/10/13/fast-and-easy-integration-testing-with-docker-and-overcast/ I'm sorely tempted to switch a lot of our integration tests over to using overcast, assuming I can create a local registry of docker images ( I believe Artifactory can do it, but not nexus (yet?) ). Mark -- Mark Derricutt http://www.theoryinpractice.net http://plus.google.com/+MarkDerricutt http://twitter.com/talios http://facebook.com/mderricutt signature.asc Description: OpenPGP digital signature
Re: Resolving classpath hell issues by allowing libraries to only use the version they request?
You are right that jar hell and dll hell are not that easy to deal with. OSGi is supposed to be a way to reduce this problem but IMHUO brings it own problems. Most reputable projects are pretty careful about compatibility since it is really tough on developers if new versions break existing code. Developers will just refuse to use the library if updates repeatedly break their code. Breaking your application into smaller independent services may give you a way to reduce the number of conflicts related to these different transitive dependencies but that may not be as easy as testing your app with the updated versions or doing some research with the teams that are building with the old libraries to see if they will upgrade and run their acceptance tests within a time frame that meets your project's timeline requirements. If the project is open source, you may be able to run the acceptance tests for the dependencies that use old libraries after you change their build to use later versions. That would give you a higher level of confidence and perhaps be a bit of a pay-back to the team that is giving you a helpful library, if you give them the feedback. Ron On 05/01/2015 4:03 PM, Kevin Burton wrote: I spent a ton of time tonight in classpath hell. Basically, Apache Spark, Cassandra, and Cassandra Unit, and Guava, Jackson JSON, and Jetty have an INSANE dependency graph. They're all trampling on each other with broken dependencies. This results in a lot of exclusion work to get them to use sane bindings. And it’s not necessarily anyone’s fault. Library A will use library B which depend on library C version 1.1 but then library D will use library C which depends on 1.2… I *mostly* end up with NoClassDefFoundException. But I’m worried about other smaller bugs. For example, if my library wasn’t tested with a later version of a library it isn’t necessarily as reliable out of the box since it wasn’t tested with that config. Has anyone done any work with only allowing a library to use the version of a class it requested? If you release your library fully tested on version 1.1.1 of some library, and someone accidentally starts using 1.1.2 becuase of dependency hell, then we're going to break the build and introduce bugs. So what would happen is every dependency would only be sourced from the version you require. This could be done via a hierarchical class loader. Of course this would probably end up burning up some memory but probably worth it. The main issue, is it WORTH it.. would this introduce new bugs? For example, it’s possible that libraries could step on each other by creating duplicate threads with the same names, or messing up system properties. -- Ron Wheeler President Artifact Software Inc email: rwhee...@artifact-software.com skype: ronaldmwheeler phone: 866-970-2435, ext 102 - To unsubscribe, e-mail: users-unsubscr...@maven.apache.org For additional commands, e-mail: users-h...@maven.apache.org
Re: Resolving classpath hell issues by allowing libraries to only use the version they request?
Hi Kevin, I agree with Steven. One way of resolving this sort of problem without OSGi is to use a consistent, meaningful versioning system such as Semantic Versioning (http://semver.org/). Once you have the ability to reason about forwards and backwards compatibility, it is easier to resolve version conflicts in a more reliable (ideally automated) way. Of course, all the tools you are using do not necessarily abide by such versioning standards. So then conflict resolution becomes more complicated. Steven's suggestion of managing them upwards to the latest needed is often good enough... Regards, Curtis On Mon, Jan 5, 2015 at 3:20 PM, Steven Schlansker stevenschlans...@gmail.com wrote: On Jan 5, 2015, at 1:03 PM, Kevin Burton bur...@spinn3r.com wrote: I spent a ton of time tonight in classpath hell. Basically, Apache Spark, Cassandra, and Cassandra Unit, and Guava, Jackson JSON, and Jetty have an INSANE dependency graph. They're all trampling on each other with broken dependencies. This results in a lot of exclusion work to get them to use sane bindings. Has anyone done any work with only allowing a library to use the version of a class it requested? So what would happen is every dependency would only be sourced from the version you require. This could be done via a hierarchical class loader. Of course this would probably end up burning up some memory but probably worth it. The main issue, is it WORTH it.. would this introduce new bugs? You are basically talking about OSGi or some other such module system. https://en.wikipedia.org/wiki/OSGi In my opinion, it is actually easier to resolve the dependency conflicts among the modules (usually by managing them upwards to the latest needed by any dependency) and tracking this via a plugin [1] Having such complicated class loader situations just leads to even more subtle and confusing bugs IMO. [1] https://github.com/ning/maven-dependency-versions-check-plugin - To unsubscribe, e-mail: users-unsubscr...@maven.apache.org For additional commands, e-mail: users-h...@maven.apache.org
Re: Resolving classpath hell issues by allowing libraries to only use the version they request?
Breaking your application into smaller independent services may give you a way to reduce the number of conflicts related to these different transitive dependencies but that may not be as easy as testing your app with the updated versions or doing some research with the teams that are building with the old libraries to see if they will upgrade and run their acceptance tests within a time frame that meets your project's timeline requirements. I agree. in this case the issue is testing. I need to embed cassandra so I can test it but it’s conflicting with spark. Ideally I could just start a container and I think that’s the way we’re going to evolve things. Essentially, start a container for the daemons I need to test against and then make sure all my components work. Then just kill the containers. — Founder/CEO Spinn3r.com Location: *San Francisco, CA* blog: http://burtonator.wordpress.com … or check out my Google+ profile https://plus.google.com/102718274791889610666/posts http://spinn3r.com
Re: Resolving classpath hell issues by allowing libraries to only use the version they request?
On 05/01/2015 6:29 PM, Kevin Burton wrote: Breaking your application into smaller independent services may give you a way to reduce the number of conflicts related to these different transitive dependencies but that may not be as easy as testing your app with the updated versions or doing some research with the teams that are building with the old libraries to see if they will upgrade and run their acceptance tests within a time frame that meets your project's timeline requirements. I agree. in this case the issue is testing. I need to embed cassandra so I can test it but it’s conflicting with spark. These are both Apache projects that I don't know. Have you asked the one using the older dependencies if they can upgrade their dependencies and run their acceptance tests? Even if you have to download the sources and upgrade the dependencies and run their acceptance tests, this might be easier than trying to test your own application to the extent required to ensure that the software has no problems with the updated versions. This assumes that these projects have strong test suites. I would state your problems in the forum of the offending project. There might be a lot of enthusiasm for an upgrade. Ron Ideally I could just start a container and I think that’s the way we’re going to evolve things. Essentially, start a container for the daemons I need to test against and then make sure all my components work. Then just kill the containers. — Founder/CEO Spinn3r.com Location: *San Francisco, CA* blog: http://burtonator.wordpress.com … or check out my Google+ profile https://plus.google.com/102718274791889610666/posts http://spinn3r.com -- Ron Wheeler President Artifact Software Inc email: rwhee...@artifact-software.com skype: ronaldmwheeler phone: 866-970-2435, ext 102 - To unsubscribe, e-mail: users-unsubscr...@maven.apache.org For additional commands, e-mail: users-h...@maven.apache.org
Re: Resolving classpath hell issues by allowing libraries to only use the version they request?
On 6 January 2015 at 14:50, Ron Wheeler rwhee...@artifact-software.com wrote: On 05/01/2015 6:29 PM, Kevin Burton wrote: Breaking your application into smaller independent services may give you a way to reduce the number of conflicts related to these different transitive dependencies but that may not be as easy as testing your app with the updated versions or doing some research with the teams that are building with the old libraries to see if they will upgrade and run their acceptance tests within a time frame that meets your project's timeline requirements. I agree. in this case the issue is testing. I need to embed cassandra so I can test it but it’s conflicting with spark. These are both Apache projects that I don't know. Same. Or alternatively from my brief reading of these two it appears that they can run independently. If you can avoid having them running in the same process you can separate out your dependency hell. This may be as simple as creating a new module to do your integration testing and to control starting the external dependencies needed to do your integration testing.
[ANN] Apache Maven Plugin Tools (with Maven Plugin Plugin) 3.4 Released
The Apache Maven team is pleased to announce the release of the Apache Maven Plugin Tools, version 3.4 The Maven Plugin Tools contains the necessary tools to generate rebarbative content like descriptor, help and documentation. http://maven.apache.org/plugin-tools/ http://maven.apache.org/plugin-tools/maven-plugin-plugin/ You should specify the version in your project's plugin configuration: plugin groupIdorg.apache.maven.plugins/groupId artifactIdmaven-plugin-plugin/artifactId version3.4/version /plugin Release Notes - Maven Plugin Tools - Version 3.4 ** Bug * [MPLUGIN-238] - HelpMojo phases fails when using java-annotations * [MPLUGIN-266] - Incorrect warning comment about deprecated @component usage for maven objects * [MPLUGIN-270] - Deprecation of classical Maven objects as components is broken when using Javadoc tags * [MPLUGIN-272] - Descriptor goal fail with java 8 and interface with default method: upgrade QDox * [MPLUGIN-274] - generated HelpMojo source contains tabs and other errors reported by Checkstyle * [MPLUGIN-284] - don't try to extract mojos from Beanshell by default * [MPLUGIN-286] - Failure during build via mvn -Prun-its clean verfiy based on missing dependencies ** Improvement * [MPLUGIN-267] - document how to change descriptor phase instead of running it twice with skipErrorNoDescriptorsFound * [MPLUGIN-279] - improve jdk requirement when m-compiler-p not explicitely configured: use default properties * [MPLUGIN-281] - Removed dependency plexus-container-default:1.0-alpha-9- stable-1 * [MPLUGIN-283] - Upgrade plexus-utils to 3.0.20 ** Task * [MPLUGIN-278] - Upgrade to Maven 2.2.1 minimum requirement * [MPLUGIN-280] - remove xdoc goal since the content is generated during report * [MPLUGIN-285] - change java extractor id to java-javadoc to match java-annotations * [MPLUGIN-287] - refactoring: consistent package name for extractor and javadoc Enjoy, -The Apache Maven team - To unsubscribe, e-mail: users-unsubscr...@maven.apache.org For additional commands, e-mail: users-h...@maven.apache.org
Resolving classpath hell issues by allowing libraries to only use the version they request?
I spent a ton of time tonight in classpath hell. Basically, Apache Spark, Cassandra, and Cassandra Unit, and Guava, Jackson JSON, and Jetty have an INSANE dependency graph. They're all trampling on each other with broken dependencies. This results in a lot of exclusion work to get them to use sane bindings. And it’s not necessarily anyone’s fault. Library A will use library B which depend on library C version 1.1 but then library D will use library C which depends on 1.2… I *mostly* end up with NoClassDefFoundException. But I’m worried about other smaller bugs. For example, if my library wasn’t tested with a later version of a library it isn’t necessarily as reliable out of the box since it wasn’t tested with that config. Has anyone done any work with only allowing a library to use the version of a class it requested? If you release your library fully tested on version 1.1.1 of some library, and someone accidentally starts using 1.1.2 becuase of dependency hell, then we're going to break the build and introduce bugs. So what would happen is every dependency would only be sourced from the version you require. This could be done via a hierarchical class loader. Of course this would probably end up burning up some memory but probably worth it. The main issue, is it WORTH it.. would this introduce new bugs? For example, it’s possible that libraries could step on each other by creating duplicate threads with the same names, or messing up system properties. -- Founder/CEO Spinn3r.com Location: *San Francisco, CA* blog: http://burtonator.wordpress.com … or check out my Google+ profile https://plus.google.com/102718274791889610666/posts http://spinn3r.com
Re: Resolving classpath hell issues by allowing libraries to only use the version they request?
On Jan 5, 2015, at 1:03 PM, Kevin Burton bur...@spinn3r.com wrote: I spent a ton of time tonight in classpath hell. Basically, Apache Spark, Cassandra, and Cassandra Unit, and Guava, Jackson JSON, and Jetty have an INSANE dependency graph. They're all trampling on each other with broken dependencies. This results in a lot of exclusion work to get them to use sane bindings. Has anyone done any work with only allowing a library to use the version of a class it requested? So what would happen is every dependency would only be sourced from the version you require. This could be done via a hierarchical class loader. Of course this would probably end up burning up some memory but probably worth it. The main issue, is it WORTH it.. would this introduce new bugs? You are basically talking about OSGi or some other such module system. https://en.wikipedia.org/wiki/OSGi In my opinion, it is actually easier to resolve the dependency conflicts among the modules (usually by managing them upwards to the latest needed by any dependency) and tracking this via a plugin [1] Having such complicated class loader situations just leads to even more subtle and confusing bugs IMO. [1] https://github.com/ning/maven-dependency-versions-check-plugin - To unsubscribe, e-mail: users-unsubscr...@maven.apache.org For additional commands, e-mail: users-h...@maven.apache.org
