I believe, that a clean and maintainable solution would be if the engine/framework itself could be separated from everything else, the dependency directions would be fixed, plugins would depend only on api-s and after that, any source/sink/etc implementation would come as an individual module/plugin and would be loaded with an isolated classloader.
This way anyone could implement anything with any version. It would be possible to use a different version of ES at the same time if you need it. On Mon, Jan 29, 2018 at 10:48 AM, Wahrmann, Helmut <helmut.wahrm...@rsa.com> wrote: > I think it is not a problem to distribute the modules together if we have > maintainers for them. > It doesn't make sense to release Flume 1.9 in 2018 with support for module > versions which were current in 2013. > > Best way would be to identify people willing to upgrade the modules to > their latest version. > If we can't find someone, we should distribute it separately. > > best regards, > > Helmut > > > -----Original Message----- > From: Mike Percy [mailto:mpe...@apache.org] > Sent: Montag, 29. Jänner 2018 09:12 > To: dev@flume.apache.org > Subject: Re: Merging to trunk? > > Interesting proposal Yonghao. > > I think it's worth pointing out that today, a Flume build will work "out > of the box" with any combination of supported plugins since they are all > already in the Flume classpath. The downside of that, of course, is that > the dependency hell makes it very hard to upgrade modules. The benefit is > that it's nice from a usability perspective. > > However, if we are willing to drop this "out of the box" usage feature, > then we could allow for separate modules to have conflicting dependencies > (i.e. Solr and ElasticSearch) as long as they are not loaded in the same > Flume agent. > > What do you folks think about removing the Solr Sink and the ElasticSearch > Sink from the default distribution and instead distribute them as separate > (fat) jars? If we do it for those two then it might also make sense to do > it for other modules as well. > > I am not sure this is very user friendly in the common case where people > just want to use the HDFS and HBase sink out of the box. So maybe we leave > those as part of the "default" module setup? > > Obviously, changing how modules get included into the classpath would be a > breaking change. I think a change that drastic would call for a Flume 2.0.0 > release in order to indicate that it's not backwards compatible. > > Thoughts? > > Mike > > On Sat, Jan 27, 2018 at 10:50 PM, Yonghao Zou <yonghaoz1...@gmail.com> > wrote: > > > Maybe the Maven shade plugin can solve this, we can package each > > module to a fat jar and they will not depend on other modules. > > > > 2018-01-27 6:14 GMT+08:00 Wahrmann, Helmut <helmut.wahrm...@rsa.com>: > > > > > fully agreed. > > > But we shouldn't do that do support a 4 years old component. > > > The goal should be to support the latest releases. > > > It is ridiculous to release flume 1.9 with elasticsearch 0.90.1 > > > support, when 6.x is the current version. > > > > > > > > > > > > Best regards, > > > > > > Helmut Wahrmann > > > > > > > > > -------- Ursprüngliche Nachricht -------- > > > Von: Ralph Goers <ralph.go...@dslextreme.com> > > > Datum: 26.01.18 22:12 (GMT+01:00) > > > An: dev@flume.apache.org > > > Betreff: Re: Merging to trunk? > > > > > > The “right” way to deal with this from a Maven perspective is to > > > declare the version you want in the dependencyManagement section of > > > the parent > > pom > > > and to not specify any versions in child poms. Then use the Maven > > enforcer > > > plugin to make sure everything required has a declaration in the > > > dependencyManagement section. > > > > > > Ralph > > > > > > > On Jan 26, 2018, at 8:28 AM, Wahrmann, Helmut > > > > <helmut.wahrm...@rsa.com > > > > > > wrote: > > > > > > > > Hi Mike, > > > > > > > > This only shows that some other component needs to be updated as > well. > > > > Which software relies on Lucene 4.3.0, which was released in 2013? > > > > It is most probably not used by anyone else anymore. > > > > > > > > This needs to be solved. It makes no sense to release Flume 1.9 > > > > with > > > support for ElasticSearch 0.90.1. > > > > And it doesn't make sense either to release Flume 1.9 with support > > > > for > > a > > > component that relies on Lucene 4.3.0. > > > > > > > > We will always have dependency problems, if one component is > > > > updated to > > > a newer version and others not. > > > > > > > > To get rid of the conflict is probably to set "optional" to true > > > > in the > > > pom.xml. > > > > Then we don't distribute those Jars in the flume lib and the users > > > > need > > > to set their class path to point e.g. to Elasticsearch and then the > > correct > > > libs will be used. > > > > I doubt that a Elasticsearch user will also use some other > > > > component > > > with e.g. this outdated Lucene. > > > > > > > > best regards, > > > > > > > > Helmut > > > > > > > > > > > > -----Original Message----- > > > > From: Mike Percy [mailto:mpe...@apache.org] > > > > Sent: Donnerstag, 25. Jänner 2018 23:01 > > > > To: dev@flume.apache.org > > > > Subject: Re: Merging to trunk? > > > > > > > > Hi Helmut, > > > > I wrote a small Perl script to identify dependency conflicts based > > > > on > > > output from mvn dependency:tree and posted it here: > > > https://gist.github.com/ > > > > mpercy/39614d770864bdd0c386befd5e8a1840 > > > > > > > > I ran that on the current trunk and it actually found some errors > > > > which > > > should be fixed: > > > > > > > > Version conflict: package com.google.guava:guava:jar needed in 2 > > > different > > > > versions: (11.0.2, 18.0) > > > > Version conflict: package > > > > commons-httpclient:commons-httpclient:jar > > > needed in 2 different versions: (3.0.1, 3.1) Version conflict: > > > package commons-logging:commons-logging:jar needed in 2 different > versions: > > > (1.1.3, 1.2) Version conflict: package commons-pool:commons-pool:jar > > needed > > > in 2 different versions: (1.5.4, 1.6) Version conflict: package > > > net.sf.jopt-simple:jopt-simple:jar needed in 2 different versions: > > > (3.2, > > > 4.7) Version conflict: package > > > org.codehaus.jackson:jackson-jaxrs:jar > > > needed in > > > > 2 different versions: (1.8.3, 1.8.8) > > > > ERROR: 6 package version conflicts identified > > > > > > > > However, after applying your patch on top of trunk and running it > > again, > > > it identified new conflicts: > > > > > > > > Version conflict: package com.google.guava:guava:jar needed in 2 > > > different > > > > versions: (11.0.2, 18.0) > > > > Version conflict: package > > > > commons-httpclient:commons-httpclient:jar > > > needed in 2 different versions: (3.0.1, 3.1) Version conflict: > > > package commons-logging:commons-logging:jar needed in 2 different > versions: > > > (1.1.3, 1.2) Version conflict: package commons-pool:commons-pool:jar > > needed > > > in 2 different versions: (1.5.4, 1.6) Version conflict: package > > > net.sf.jopt-simple:jopt-simple:jar needed in 3 different versions: > > > (3.2, 4.7, 5.0.2) Version conflict: package > > > org.apache.lucene:lucene- analyzers-common:jar > > > > needed in 2 different versions: (4.3.0, 7.1.0) Version conflict: > > package > > > org.apache.lucene:lucene-core:jar needed in 2 different versions: > > (4.3.0, > > > 7.1.0) Version conflict: package > > > org.apache.lucene:lucene-grouping:jar > > > needed in 2 different versions: (4.3.0, 7.1.0) Version conflict: > > > package org.apache.lucene:lucene-highlighter:jar needed in 2 > different versions: > > > (4.3.0, 7.1.0) Version conflict: package org.apache.lucene:lucene- > > memory:jar > > > needed in 2 different versions: (4.3.0, 7.1.0) Version conflict: > > > package org.apache.lucene:lucene-misc:jar needed in 2 different > versions: > > (4.3.0, > > > 7.1.0) Version conflict: package > > > org.apache.lucene:lucene-queries:jar > > > needed in 2 different versions: (4.3.0, 7.1.0) Version conflict: > > > package org.apache.lucene:lucene-queryparser:jar needed in 2 > different versions: > > > (4.3.0, 7.1.0) Version conflict: package org.apache.lucene:lucene- > > spatial:jar > > > needed in 2 different versions: (4.3.0, 7.1.0) Version conflict: > > > package org.apache.lucene:lucene-suggest:jar needed in 2 different > versions: > > > (4.3.0, 7.1.0) Version conflict: package > > > org.codehaus.jackson:jackson- > > jaxrs:jar > > > needed in > > > > 2 different versions: (1.8.3, 1.8.8) Version conflict: package > > > > org.yaml:snakeyaml:jar needed in 2 different > > > > versions: (1.10, 1.17) > > > > ERROR: 17 package version conflicts identified > > > > > > > > So while it appears Guava is no longer a concern, there are other > > > conflicting dependencies involved in the conflict here. > > > > > > > > I'm interested to hear your thoughts on this. > > > > > > > > Regards, > > > > Mike > > > > > > > > On Wed, Jan 24, 2018 at 7:57 AM, Wahrmann, Helmut < > > > helmut.wahrm...@rsa.com> > > > > wrote: > > > > > > > >> Hi Mike, > > > >> > > > >> Thanks for the quick response. > > > >> > > > >> I fully agree that we need to take care about interop between the > > > >> different components and dependencies. > > > >> But if you look at the patch in FLUME-3021, it only puts in > > > >> Elasticsearch dependencies. > > > >> > > > >> The issue with Guava, which you saw in FLUME-2921 is no longer > > > >> there, because since mid-2016 a lot has changed in the Flume > > > >> trunk and all those dependencies are fixed. > > > >> It won't even be possible gto apply this patch anymore. > > > >> > > > >> The patch in FLUME-3021 works with the latest Elasticsearch > > > >> versions, without introducing additional dependencies, which > > > >> might cause problems to other projects. > > > >> So I think it can be safely merged into trunk. > > > >> > > > >> best regards, > > > >> > > > >> Helmut > > > >> > > > >> > > > >> -----Original Message----- > > > >> From: Mike Percy [mailto:mpe...@apache.org] > > > >> Sent: Dienstag, 23. Jänner 2018 20:39 > > > >> To: dev@flume.apache.org > > > >> Subject: Re: Merging to trunk? > > > >> > > > >> Hi Helmut, > > > >> Thank you for bringing this up on dev@ and thank you for the > > > >> patch. I see there are other people people interested in this > > > >> component upgrade > > > as well. > > > >> > > > >> As you are probably aware, a Flume committer will need to approve > > > >> the change before I gets merged to trunk. > > > >> > > > >> My primary concern w/ merging this would be compatibility of the > > > >> dependencies. Flume suffers from a kind of a "dependency hell" > > > >> because of a lack of classloading support (either via OSGI or Java > modules). > > > >> See > > > >> https://issues.apache.org/jira/browse/FLUME-2293 for a tracking > > ticket > > > >> about that issue. What that means is that all of the components > > > >> that Flume ships must have compatible dependencies with each > > > >> other which makes changes like this more complex. Therefore I > > > >> would like someone to verify that mvn dependency:tree does not > > > >> show conflicts when run from the top level with the new patches. > > > >> If memory serves, I believe Google Guava is likely to conflict. > > > >> > > > >> Also, I think that > > > >> https://issues.apache.org/jira/browse/FLUME-3021 > > is > > > >> a duplicate of https://issues.apache.org/jira/browse/FLUME-2921 > > > >> which has additional information about the various issues that we > > > >> need to solve to do this upgrade. > > > >> > > > >> I'd be happy to discuss this issue some more on this thread. > > > >> > > > >> Mike > > > >> > > > >> > > > >> On Tue, Jan 23, 2018 at 12:57 AM, Wahrmann, Helmut < > > > >> helmut.wahrm...@rsa.com> > > > >> wrote: > > > >> > > > >>> Hi, > > > >>> > > > >>> Who decides if/when a patch or pull request gets merged to trunk? > > > >>> Reason I am asking is for the Elasticsearch support. The current > > > >>> code in trunk does not work with ES > 2.x. > > > >>> Currently Elasticsearch is at 6.1. > > > >>> > > > >>> In FLUME-3021 we have several patches since March last year. > > > >>> I have a patched Flume running at a customer since April last > > > >>> year without any problems. > > > >>> > > > >>> So why not merging those changes into the trunk? > > > >>> As I stated in my comment in FLUME-3021, it will not cause any > > > >>> problems, cause the current trunk won't work with newer ES > > > >>> versions > > > >> anyhow. > > > >>> I doubt that someone is out there still running ES 0.95. > > > >>> > > > >>> best regards, > > > >>> > > > >>> Helmut > > > >>> > > > >> > > > > > > > > > > > > > > > -- > > *Yonghao Zou* > > *Shanghai China* > > > > *China Merchants Bank* > > > > *EN_BLOG <http://blog.haozi1994.com/>* > > *CH_BLOG* <http://www.fxiaoqing.com/> > > >
