Interesting proposal Yonghao. I think it's worth pointing out that today, a Flume build will work "out of the box" with any combination of supported plugins since they are all already in the Flume classpath. The downside of that, of course, is that the dependency hell makes it very hard to upgrade modules. The benefit is that it's nice from a usability perspective.
However, if we are willing to drop this "out of the box" usage feature, then we could allow for separate modules to have conflicting dependencies (i.e. Solr and ElasticSearch) as long as they are not loaded in the same Flume agent. What do you folks think about removing the Solr Sink and the ElasticSearch Sink from the default distribution and instead distribute them as separate (fat) jars? If we do it for those two then it might also make sense to do it for other modules as well. I am not sure this is very user friendly in the common case where people just want to use the HDFS and HBase sink out of the box. So maybe we leave those as part of the "default" module setup? Obviously, changing how modules get included into the classpath would be a breaking change. I think a change that drastic would call for a Flume 2.0.0 release in order to indicate that it's not backwards compatible. Thoughts? Mike On Sat, Jan 27, 2018 at 10:50 PM, Yonghao Zou <yonghaoz1...@gmail.com> wrote: > Maybe the Maven shade plugin can solve this, we can package each module to > a fat jar and they will not depend on other modules. > > 2018-01-27 6:14 GMT+08:00 Wahrmann, Helmut <helmut.wahrm...@rsa.com>: > > > fully agreed. > > But we shouldn't do that do support a 4 years old component. > > The goal should be to support the latest releases. > > It is ridiculous to release flume 1.9 with elasticsearch 0.90.1 support, > > when 6.x is the current version. > > > > > > > > Best regards, > > > > Helmut Wahrmann > > > > > > -------- Ursprüngliche Nachricht -------- > > Von: Ralph Goers <ralph.go...@dslextreme.com> > > Datum: 26.01.18 22:12 (GMT+01:00) > > An: dev@flume.apache.org > > Betreff: Re: Merging to trunk? > > > > The “right” way to deal with this from a Maven perspective is to declare > > the version you want in the dependencyManagement section of the parent > pom > > and to not specify any versions in child poms. Then use the Maven > enforcer > > plugin to make sure everything required has a declaration in the > > dependencyManagement section. > > > > Ralph > > > > > On Jan 26, 2018, at 8:28 AM, Wahrmann, Helmut <helmut.wahrm...@rsa.com > > > > wrote: > > > > > > Hi Mike, > > > > > > This only shows that some other component needs to be updated as well. > > > Which software relies on Lucene 4.3.0, which was released in 2013? > > > It is most probably not used by anyone else anymore. > > > > > > This needs to be solved. It makes no sense to release Flume 1.9 with > > support for ElasticSearch 0.90.1. > > > And it doesn't make sense either to release Flume 1.9 with support for > a > > component that relies on Lucene 4.3.0. > > > > > > We will always have dependency problems, if one component is updated to > > a newer version and others not. > > > > > > To get rid of the conflict is probably to set "optional" to true in the > > pom.xml. > > > Then we don't distribute those Jars in the flume lib and the users need > > to set their class path to point e.g. to Elasticsearch and then the > correct > > libs will be used. > > > I doubt that a Elasticsearch user will also use some other component > > with e.g. this outdated Lucene. > > > > > > best regards, > > > > > > Helmut > > > > > > > > > -----Original Message----- > > > From: Mike Percy [mailto:mpe...@apache.org] > > > Sent: Donnerstag, 25. Jänner 2018 23:01 > > > To: dev@flume.apache.org > > > Subject: Re: Merging to trunk? > > > > > > Hi Helmut, > > > I wrote a small Perl script to identify dependency conflicts based on > > output from mvn dependency:tree and posted it here: > > https://gist.github.com/ > > > mpercy/39614d770864bdd0c386befd5e8a1840 > > > > > > I ran that on the current trunk and it actually found some errors which > > should be fixed: > > > > > > Version conflict: package com.google.guava:guava:jar needed in 2 > > different > > > versions: (11.0.2, 18.0) > > > Version conflict: package commons-httpclient:commons-httpclient:jar > > needed in 2 different versions: (3.0.1, 3.1) Version conflict: package > > commons-logging:commons-logging:jar needed in 2 different versions: > > (1.1.3, 1.2) Version conflict: package commons-pool:commons-pool:jar > needed > > in 2 different versions: (1.5.4, 1.6) Version conflict: package > > net.sf.jopt-simple:jopt-simple:jar needed in 2 different versions: (3.2, > > 4.7) Version conflict: package org.codehaus.jackson:jackson-jaxrs:jar > > needed in > > > 2 different versions: (1.8.3, 1.8.8) > > > ERROR: 6 package version conflicts identified > > > > > > However, after applying your patch on top of trunk and running it > again, > > it identified new conflicts: > > > > > > Version conflict: package com.google.guava:guava:jar needed in 2 > > different > > > versions: (11.0.2, 18.0) > > > Version conflict: package commons-httpclient:commons-httpclient:jar > > needed in 2 different versions: (3.0.1, 3.1) Version conflict: package > > commons-logging:commons-logging:jar needed in 2 different versions: > > (1.1.3, 1.2) Version conflict: package commons-pool:commons-pool:jar > needed > > in 2 different versions: (1.5.4, 1.6) Version conflict: package > > net.sf.jopt-simple:jopt-simple:jar needed in 3 different versions: (3.2, > > 4.7, 5.0.2) Version conflict: package org.apache.lucene:lucene- > > analyzers-common:jar > > > needed in 2 different versions: (4.3.0, 7.1.0) Version conflict: > package > > org.apache.lucene:lucene-core:jar needed in 2 different versions: > (4.3.0, > > 7.1.0) Version conflict: package org.apache.lucene:lucene-grouping:jar > > needed in 2 different versions: (4.3.0, 7.1.0) Version conflict: package > > org.apache.lucene:lucene-highlighter:jar needed in 2 different versions: > > (4.3.0, 7.1.0) Version conflict: package org.apache.lucene:lucene- > memory:jar > > needed in 2 different versions: (4.3.0, 7.1.0) Version conflict: package > > org.apache.lucene:lucene-misc:jar needed in 2 different versions: > (4.3.0, > > 7.1.0) Version conflict: package org.apache.lucene:lucene-queries:jar > > needed in 2 different versions: (4.3.0, 7.1.0) Version conflict: package > > org.apache.lucene:lucene-queryparser:jar needed in 2 different versions: > > (4.3.0, 7.1.0) Version conflict: package org.apache.lucene:lucene- > spatial:jar > > needed in 2 different versions: (4.3.0, 7.1.0) Version conflict: package > > org.apache.lucene:lucene-suggest:jar needed in 2 different versions: > > (4.3.0, 7.1.0) Version conflict: package org.codehaus.jackson:jackson- > jaxrs:jar > > needed in > > > 2 different versions: (1.8.3, 1.8.8) > > > Version conflict: package org.yaml:snakeyaml:jar needed in 2 different > > > versions: (1.10, 1.17) > > > ERROR: 17 package version conflicts identified > > > > > > So while it appears Guava is no longer a concern, there are other > > conflicting dependencies involved in the conflict here. > > > > > > I'm interested to hear your thoughts on this. > > > > > > Regards, > > > Mike > > > > > > On Wed, Jan 24, 2018 at 7:57 AM, Wahrmann, Helmut < > > helmut.wahrm...@rsa.com> > > > wrote: > > > > > >> Hi Mike, > > >> > > >> Thanks for the quick response. > > >> > > >> I fully agree that we need to take care about interop between the > > >> different components and dependencies. > > >> But if you look at the patch in FLUME-3021, it only puts in > > >> Elasticsearch dependencies. > > >> > > >> The issue with Guava, which you saw in FLUME-2921 is no longer there, > > >> because since mid-2016 a lot has changed in the Flume trunk and all > > >> those dependencies are fixed. > > >> It won't even be possible gto apply this patch anymore. > > >> > > >> The patch in FLUME-3021 works with the latest Elasticsearch versions, > > >> without introducing additional dependencies, which might cause > > >> problems to other projects. > > >> So I think it can be safely merged into trunk. > > >> > > >> best regards, > > >> > > >> Helmut > > >> > > >> > > >> -----Original Message----- > > >> From: Mike Percy [mailto:mpe...@apache.org] > > >> Sent: Dienstag, 23. Jänner 2018 20:39 > > >> To: dev@flume.apache.org > > >> Subject: Re: Merging to trunk? > > >> > > >> Hi Helmut, > > >> Thank you for bringing this up on dev@ and thank you for the patch. I > > >> see there are other people people interested in this component upgrade > > as well. > > >> > > >> As you are probably aware, a Flume committer will need to approve the > > >> change before I gets merged to trunk. > > >> > > >> My primary concern w/ merging this would be compatibility of the > > >> dependencies. Flume suffers from a kind of a "dependency hell" because > > >> of a lack of classloading support (either via OSGI or Java modules). > > >> See > > >> https://issues.apache.org/jira/browse/FLUME-2293 for a tracking > ticket > > >> about that issue. What that means is that all of the components that > > >> Flume ships must have compatible dependencies with each other which > > >> makes changes like this more complex. Therefore I would like someone > > >> to verify that mvn dependency:tree does not show conflicts when run > > >> from the top level with the new patches. If memory serves, I believe > > >> Google Guava is likely to conflict. > > >> > > >> Also, I think that https://issues.apache.org/jira/browse/FLUME-3021 > is > > >> a duplicate of https://issues.apache.org/jira/browse/FLUME-2921 which > > >> has additional information about the various issues that we need to > > >> solve to do this upgrade. > > >> > > >> I'd be happy to discuss this issue some more on this thread. > > >> > > >> Mike > > >> > > >> > > >> On Tue, Jan 23, 2018 at 12:57 AM, Wahrmann, Helmut < > > >> helmut.wahrm...@rsa.com> > > >> wrote: > > >> > > >>> Hi, > > >>> > > >>> Who decides if/when a patch or pull request gets merged to trunk? > > >>> Reason I am asking is for the Elasticsearch support. The current > > >>> code in trunk does not work with ES > 2.x. > > >>> Currently Elasticsearch is at 6.1. > > >>> > > >>> In FLUME-3021 we have several patches since March last year. > > >>> I have a patched Flume running at a customer since April last year > > >>> without any problems. > > >>> > > >>> So why not merging those changes into the trunk? > > >>> As I stated in my comment in FLUME-3021, it will not cause any > > >>> problems, cause the current trunk won't work with newer ES versions > > >> anyhow. > > >>> I doubt that someone is out there still running ES 0.95. > > >>> > > >>> best regards, > > >>> > > >>> Helmut > > >>> > > >> > > > > > > > > > -- > *Yonghao Zou* > *Shanghai China* > > *China Merchants Bank* > > *EN_BLOG <http://blog.haozi1994.com/>* > *CH_BLOG* <http://www.fxiaoqing.com/> >
