RE: Merging to trunk?

[Wahrmann, Helmut]

Hi Mike,

I gave it a try and changed the Solr version to 7.2.1. It all compiled well.
Only the tests were failing, because SolrServer, which is needed in the test 
cases to connect to the embedded Test Server is no longer available in 
org.apache.solr.client.solrj.
But that's not needed in the "production" version.
I cannot test however if it really works, since I don't have access to a  Solr 
installation.

I understand your discussion with the dependency problem, but I need to move 
ahead.
It doesn't help me to have a Flume 1.9 or 2.0 released with no dependency 
problems, but it doesn't support the latest Elasticsearch client, which are 
needed by my customer.

And by the time, you will release Flume 1.9, the ES project is at version 8, 
where they will drop support for the Transport client.
So I am in the urgent need to develop support for the new ElasticSearch Rest 
Client.
For that I need the changes which are done in Flume-3021.

So I continue my work in a fork and will publish Pull requests . Once the 
dependencies problems are sorted out, you're welcome to include my changes in 
your trunk. 

best regards,

Helmut
 

-Original Message-
From: Mike Percy [mailto:mpe...@apache.org] 
Sent: Dienstag, 30. Jänner 2018 04:57
To: dev@flume.apache.org
Subject: Re: Merging to trunk?

On Mon, Jan 29, 2018 at 10:48 AM, Wahrmann, Helmut <helmut.wahrm...@rsa.com
> wrote:
>
> I think it is not a problem to distribute the modules together if we 
> have maintainers for them.
> It doesn't make sense to release Flume 1.9 in 2018 with support for 
> module versions which were current in 2013.


> Best way would be to identify people willing to upgrade the modules to 
> their latest version.
> If we can't find someone, we should distribute it separately.


I would be OK with upgrading Solr and ES simultaneously if we can do it in a 
compatible way. However, there is no guarantee that the latest versions of Solr 
and ES will be dependency-compatible. If they are, I would certainly be in 
favor of merging such a patch.

On Mon, Jan 29, 2018 at 2:49 AM, Ferenc Szabo <szabofe...@apache.org> wrote:

> I believe, that a clean and maintainable solution would be if the 
> engine/framework itself could be separated from everything else, the 
> dependency directions would be fixed, plugins would depend only on 
> api-s  and after that, any source/sink/etc implementation would come 
> as an individual module/plugin and would be loaded with an isolated 
> classloader.
>

I agree that isolating Flume plugins from each other would solve this problem 
once and for all. I wonder if we can do both things: come up with a short-term 
"band-aid" patch to allow us to upgrade ES / Solr and also come up with a 
longer-term plan to solve the underlying problem.

Mike

Re: Merging to trunk?

[Mike Percy]

On Mon, Jan 29, 2018 at 10:48 AM, Wahrmann, Helmut  wrote:
>
> I think it is not a problem to distribute the modules together if we have
> maintainers for them.
> It doesn't make sense to release Flume 1.9 in 2018 with support for module
> versions which were current in 2013.


> Best way would be to identify people willing to upgrade the modules to
> their latest version.
> If we can't find someone, we should distribute it separately.


I would be OK with upgrading Solr and ES simultaneously if we can do it in
a compatible way. However, there is no guarantee that the latest versions
of Solr and ES will be dependency-compatible. If they are, I would
certainly be in favor of merging such a patch.

On Mon, Jan 29, 2018 at 2:49 AM, Ferenc Szabo  wrote:

> I believe, that a clean and maintainable solution would be if the
> engine/framework itself could be separated from everything else,
> the dependency directions would be fixed, plugins would depend only on
> api-s  and after that, any source/sink/etc implementation would come as an
> individual module/plugin and would be loaded with an isolated classloader.
>

I agree that isolating Flume plugins from each other would solve this
problem once and for all. I wonder if we can do both things: come up with a
short-term "band-aid" patch to allow us to upgrade ES / Solr and also come
up with a longer-term plan to solve the underlying problem.

Mike

Re: Merging to trunk?

[Ferenc Szabo]

I believe, that a clean and maintainable solution would be if the
engine/framework itself could be separated from everything else,
the dependency directions would be fixed, plugins would depend only on
api-s  and after that, any source/sink/etc implementation would come as an
individual module/plugin and would be loaded with an isolated classloader.

This way anyone could implement anything with any version. It would be
possible to use a different version of ES at the same time if you need it.

On Mon, Jan 29, 2018 at 10:48 AM, Wahrmann, Helmut <helmut.wahrm...@rsa.com>
wrote:

> I think it is not a problem to distribute the modules together if we have
> maintainers for them.
> It doesn't make sense to release Flume 1.9 in 2018 with support for module
> versions which were current in 2013.
>
> Best way would be to identify people willing to upgrade the modules to
> their latest version.
> If we can't find someone, we should distribute it separately.
>
> best regards,
>
> Helmut
>
>
> -Original Message-
> From: Mike Percy [mailto:mpe...@apache.org]
> Sent: Montag, 29. Jänner 2018 09:12
> To: dev@flume.apache.org
> Subject: Re: Merging to trunk?
>
> Interesting proposal Yonghao.
>
> I think it's worth pointing out that today, a Flume build will work "out
> of the box" with any combination of supported plugins since they are all
> already in the Flume classpath. The downside of that, of course, is that
> the dependency hell makes it very hard to upgrade modules. The benefit is
> that it's nice from a usability perspective.
>
> However, if we are willing to drop this "out of the box" usage feature,
> then we could allow for separate modules to have conflicting dependencies
> (i.e. Solr and ElasticSearch) as long as they are not loaded in the same
> Flume agent.
>
> What do you folks think about removing the Solr Sink and the ElasticSearch
> Sink from the default distribution and instead distribute them as separate
> (fat) jars? If we do it for those two then it might also make sense to do
> it for other modules as well.
>
> I am not sure this is very user friendly in the common case where people
> just want to use the HDFS and HBase sink out of the box. So maybe we leave
> those as part of the "default" module setup?
>
> Obviously, changing how modules get included into the classpath would be a
> breaking change. I think a change that drastic would call for a Flume 2.0.0
> release in order to indicate that it's not backwards compatible.
>
> Thoughts?
>
> Mike
>
> On Sat, Jan 27, 2018 at 10:50 PM, Yonghao Zou <yonghaoz1...@gmail.com>
> wrote:
>
> > Maybe the Maven shade plugin can solve this, we can package each
> > module to a fat jar and they will not depend on other modules.
> >
> > 2018-01-27 6:14 GMT+08:00 Wahrmann, Helmut <helmut.wahrm...@rsa.com>:
> >
> > >  fully agreed.
> > > But we shouldn't do that do support a 4 years old component.
> > > The goal should be to support the latest releases.
> > > It is ridiculous to release flume 1.9 with elasticsearch 0.90.1
> > > support, when 6.x is the current version.
> > >
> > >
> > >
> > > Best regards,
> > >
> > > Helmut Wahrmann
> > >
> > >
> > >  Ursprüngliche Nachricht 
> > > Von: Ralph Goers <ralph.go...@dslextreme.com>
> > > Datum: 26.01.18 22:12 (GMT+01:00)
> > > An: dev@flume.apache.org
> > > Betreff: Re: Merging to trunk?
> > >
> > > The “right” way to deal with this from a Maven perspective is to
> > > declare the version you want in the dependencyManagement section of
> > > the parent
> > pom
> > > and to not specify any versions in child poms. Then use the Maven
> > enforcer
> > > plugin to make sure everything required has a declaration in the
> > > dependencyManagement section.
> > >
> > > Ralph
> > >
> > > > On Jan 26, 2018, at 8:28 AM, Wahrmann, Helmut
> > > > <helmut.wahrm...@rsa.com
> > >
> > > wrote:
> > > >
> > > > Hi Mike,
> > > >
> > > > This only shows that some other component needs to be updated as
> well.
> > > > Which software relies on Lucene 4.3.0, which was released in 2013?
> > > > It is most probably not used by anyone else anymore.
> > > >
> > > > This needs to be solved. It makes no sense to release Flume 1.9
> > > > with
> > > support for ElasticSearch 0.90.1.
> > > > And it doesn't make sense either to relea

RE: Merging to trunk?

[Wahrmann, Helmut]

I think it is not a problem to distribute the modules together if we have 
maintainers for them.
It doesn't make sense to release Flume 1.9 in 2018 with support for module 
versions which were current in 2013.

Best way would be to identify people willing to upgrade the modules to their 
latest version.
If we can't find someone, we should distribute it separately.

best regards,

Helmut
 

-Original Message-
From: Mike Percy [mailto:mpe...@apache.org] 
Sent: Montag, 29. Jänner 2018 09:12
To: dev@flume.apache.org
Subject: Re: Merging to trunk?

Interesting proposal Yonghao.

I think it's worth pointing out that today, a Flume build will work "out of the 
box" with any combination of supported plugins since they are all already in 
the Flume classpath. The downside of that, of course, is that the dependency 
hell makes it very hard to upgrade modules. The benefit is that it's nice from 
a usability perspective.

However, if we are willing to drop this "out of the box" usage feature, then we 
could allow for separate modules to have conflicting dependencies (i.e. Solr 
and ElasticSearch) as long as they are not loaded in the same Flume agent.

What do you folks think about removing the Solr Sink and the ElasticSearch Sink 
from the default distribution and instead distribute them as separate
(fat) jars? If we do it for those two then it might also make sense to do it 
for other modules as well.

I am not sure this is very user friendly in the common case where people just 
want to use the HDFS and HBase sink out of the box. So maybe we leave those as 
part of the "default" module setup?

Obviously, changing how modules get included into the classpath would be a 
breaking change. I think a change that drastic would call for a Flume 2.0.0 
release in order to indicate that it's not backwards compatible.

Thoughts?

Mike

On Sat, Jan 27, 2018 at 10:50 PM, Yonghao Zou <yonghaoz1...@gmail.com>
wrote:

> Maybe the Maven shade plugin can solve this, we can package each 
> module to a fat jar and they will not depend on other modules.
>
> 2018-01-27 6:14 GMT+08:00 Wahrmann, Helmut <helmut.wahrm...@rsa.com>:
>
> >  fully agreed.
> > But we shouldn't do that do support a 4 years old component.
> > The goal should be to support the latest releases.
> > It is ridiculous to release flume 1.9 with elasticsearch 0.90.1 
> > support, when 6.x is the current version.
> >
> >
> >
> > Best regards,
> >
> > Helmut Wahrmann
> >
> >
> >  Ursprüngliche Nachricht ----
> > Von: Ralph Goers <ralph.go...@dslextreme.com>
> > Datum: 26.01.18 22:12 (GMT+01:00)
> > An: dev@flume.apache.org
> > Betreff: Re: Merging to trunk?
> >
> > The “right” way to deal with this from a Maven perspective is to 
> > declare the version you want in the dependencyManagement section of 
> > the parent
> pom
> > and to not specify any versions in child poms. Then use the Maven
> enforcer
> > plugin to make sure everything required has a declaration in the 
> > dependencyManagement section.
> >
> > Ralph
> >
> > > On Jan 26, 2018, at 8:28 AM, Wahrmann, Helmut 
> > > <helmut.wahrm...@rsa.com
> >
> > wrote:
> > >
> > > Hi Mike,
> > >
> > > This only shows that some other component needs to be updated as well.
> > > Which software relies on Lucene 4.3.0, which was released in 2013?
> > > It is most probably not used by anyone else anymore.
> > >
> > > This needs to be solved. It makes no sense to release Flume 1.9 
> > > with
> > support for ElasticSearch 0.90.1.
> > > And it doesn't make sense either to release Flume 1.9 with support 
> > > for
> a
> > component that relies on Lucene 4.3.0.
> > >
> > > We will always have dependency problems, if one component is 
> > > updated to
> > a newer version and others not.
> > >
> > > To get rid of the conflict is probably to set "optional" to true 
> > > in the
> > pom.xml.
> > > Then we don't distribute those Jars in the flume lib and the users 
> > > need
> > to set their class path to point e.g. to Elasticsearch and then the
> correct
> > libs will be used.
> > > I doubt that a Elasticsearch user will also use some other 
> > > component
> > with e.g. this outdated Lucene.
> > >
> > > best regards,
> > >
> > > Helmut
> > >
> > >
> > > -Original Message-
> > > From: Mike Percy [mailto:mpe...@apache.org]
> > > Sent: Donnerstag, 25. Jänner 2018 23:01
> > > To: dev@flume.apache.org
> &g

Re: Merging to trunk?

[Mike Percy]

Interesting proposal Yonghao.

I think it's worth pointing out that today, a Flume build will work "out of
the box" with any combination of supported plugins since they are all
already in the Flume classpath. The downside of that, of course, is that
the dependency hell makes it very hard to upgrade modules. The benefit is
that it's nice from a usability perspective.

However, if we are willing to drop this "out of the box" usage feature,
then we could allow for separate modules to have conflicting dependencies
(i.e. Solr and ElasticSearch) as long as they are not loaded in the same
Flume agent.

What do you folks think about removing the Solr Sink and the ElasticSearch
Sink from the default distribution and instead distribute them as separate
(fat) jars? If we do it for those two then it might also make sense to do
it for other modules as well.

I am not sure this is very user friendly in the common case where people
just want to use the HDFS and HBase sink out of the box. So maybe we leave
those as part of the "default" module setup?

Obviously, changing how modules get included into the classpath would be a
breaking change. I think a change that drastic would call for a Flume 2.0.0
release in order to indicate that it's not backwards compatible.

Thoughts?

Mike

On Sat, Jan 27, 2018 at 10:50 PM, Yonghao Zou <yonghaoz1...@gmail.com>
wrote:

> Maybe the Maven shade plugin can solve this, we can package each module to
> a fat jar and they will not depend on other modules.
>
> 2018-01-27 6:14 GMT+08:00 Wahrmann, Helmut <helmut.wahrm...@rsa.com>:
>
> >  fully agreed.
> > But we shouldn't do that do support a 4 years old component.
> > The goal should be to support the latest releases.
> > It is ridiculous to release flume 1.9 with elasticsearch 0.90.1 support,
> > when 6.x is the current version.
> >
> >
> >
> > Best regards,
> >
> > Helmut Wahrmann
> >
> >
> >  Ursprüngliche Nachricht ----
> > Von: Ralph Goers <ralph.go...@dslextreme.com>
> > Datum: 26.01.18 22:12 (GMT+01:00)
> > An: dev@flume.apache.org
> > Betreff: Re: Merging to trunk?
> >
> > The “right” way to deal with this from a Maven perspective is to declare
> > the version you want in the dependencyManagement section of the parent
> pom
> > and to not specify any versions in child poms. Then use the Maven
> enforcer
> > plugin to make sure everything required has a declaration in the
> > dependencyManagement section.
> >
> > Ralph
> >
> > > On Jan 26, 2018, at 8:28 AM, Wahrmann, Helmut <helmut.wahrm...@rsa.com
> >
> > wrote:
> > >
> > > Hi Mike,
> > >
> > > This only shows that some other component needs to be updated as well.
> > > Which software relies on Lucene 4.3.0, which was released in 2013?
> > > It is most probably not used by anyone else anymore.
> > >
> > > This needs to be solved. It makes no sense to release Flume 1.9 with
> > support for ElasticSearch 0.90.1.
> > > And it doesn't make sense either to release Flume 1.9 with support for
> a
> > component that relies on Lucene 4.3.0.
> > >
> > > We will always have dependency problems, if one component is updated to
> > a newer version and others not.
> > >
> > > To get rid of the conflict is probably to set "optional" to true in the
> > pom.xml.
> > > Then we don't distribute those Jars in the flume lib and the users need
> > to set their class path to point e.g. to Elasticsearch and then the
> correct
> > libs will be used.
> > > I doubt that a Elasticsearch user will also use some other component
> > with e.g. this outdated Lucene.
> > >
> > > best regards,
> > >
> > > Helmut
> > >
> > >
> > > -Original Message-
> > > From: Mike Percy [mailto:mpe...@apache.org]
> > > Sent: Donnerstag, 25. Jänner 2018 23:01
> > > To: dev@flume.apache.org
> > > Subject: Re: Merging to trunk?
> > >
> > > Hi Helmut,
> > > I wrote a small Perl script to identify dependency conflicts based on
> > output from mvn dependency:tree and posted it here:
> > https://gist.github.com/
> > > mpercy/39614d770864bdd0c386befd5e8a1840
> > >
> > > I ran that on the current trunk and it actually found some errors which
> > should be fixed:
> > >
> > > Version conflict: package com.google.guava:guava:jar needed in 2
> > different
> > > versions: (11.0.2, 18.0)
> > > Version conflict: package commons-httpclient:commons-httpclient:jar
> > need

Re: Merging to trunk?

[Yonghao Zou]

Maybe the Maven shade plugin can solve this, we can package each module to
a fat jar and they will not depend on other modules.

2018-01-27 6:14 GMT+08:00 Wahrmann, Helmut <helmut.wahrm...@rsa.com>:

>  fully agreed.
> But we shouldn't do that do support a 4 years old component.
> The goal should be to support the latest releases.
> It is ridiculous to release flume 1.9 with elasticsearch 0.90.1 support,
> when 6.x is the current version.
>
>
>
> Best regards,
>
> Helmut Wahrmann
>
>
>  Ursprüngliche Nachricht 
> Von: Ralph Goers <ralph.go...@dslextreme.com>
> Datum: 26.01.18 22:12 (GMT+01:00)
> An: dev@flume.apache.org
> Betreff: Re: Merging to trunk?
>
> The “right” way to deal with this from a Maven perspective is to declare
> the version you want in the dependencyManagement section of the parent pom
> and to not specify any versions in child poms. Then use the Maven enforcer
> plugin to make sure everything required has a declaration in the
> dependencyManagement section.
>
> Ralph
>
> > On Jan 26, 2018, at 8:28 AM, Wahrmann, Helmut <helmut.wahrm...@rsa.com>
> wrote:
> >
> > Hi Mike,
> >
> > This only shows that some other component needs to be updated as well.
> > Which software relies on Lucene 4.3.0, which was released in 2013?
> > It is most probably not used by anyone else anymore.
> >
> > This needs to be solved. It makes no sense to release Flume 1.9 with
> support for ElasticSearch 0.90.1.
> > And it doesn't make sense either to release Flume 1.9 with support for a
> component that relies on Lucene 4.3.0.
> >
> > We will always have dependency problems, if one component is updated to
> a newer version and others not.
> >
> > To get rid of the conflict is probably to set "optional" to true in the
> pom.xml.
> > Then we don't distribute those Jars in the flume lib and the users need
> to set their class path to point e.g. to Elasticsearch and then the correct
> libs will be used.
> > I doubt that a Elasticsearch user will also use some other component
> with e.g. this outdated Lucene.
> >
> > best regards,
> >
> > Helmut
> >
> >
> > -Original Message-
> > From: Mike Percy [mailto:mpe...@apache.org]
> > Sent: Donnerstag, 25. Jänner 2018 23:01
> > To: dev@flume.apache.org
> > Subject: Re: Merging to trunk?
> >
> > Hi Helmut,
> > I wrote a small Perl script to identify dependency conflicts based on
> output from mvn dependency:tree and posted it here:
> https://gist.github.com/
> > mpercy/39614d770864bdd0c386befd5e8a1840
> >
> > I ran that on the current trunk and it actually found some errors which
> should be fixed:
> >
> > Version conflict: package com.google.guava:guava:jar needed in 2
> different
> > versions: (11.0.2, 18.0)
> > Version conflict: package commons-httpclient:commons-httpclient:jar
> needed in 2 different versions: (3.0.1, 3.1) Version conflict: package
> commons-logging:commons-logging:jar needed in 2 different versions:
> (1.1.3, 1.2) Version conflict: package commons-pool:commons-pool:jar needed
> in 2 different versions: (1.5.4, 1.6) Version conflict: package
> net.sf.jopt-simple:jopt-simple:jar needed in 2 different versions: (3.2,
> 4.7) Version conflict: package org.codehaus.jackson:jackson-jaxrs:jar
> needed in
> > 2 different versions: (1.8.3, 1.8.8)
> > ERROR: 6 package version conflicts identified
> >
> > However, after applying your patch on top of trunk and running it again,
> it identified new conflicts:
> >
> > Version conflict: package com.google.guava:guava:jar needed in 2
> different
> > versions: (11.0.2, 18.0)
> > Version conflict: package commons-httpclient:commons-httpclient:jar
> needed in 2 different versions: (3.0.1, 3.1) Version conflict: package
> commons-logging:commons-logging:jar needed in 2 different versions:
> (1.1.3, 1.2) Version conflict: package commons-pool:commons-pool:jar needed
> in 2 different versions: (1.5.4, 1.6) Version conflict: package
> net.sf.jopt-simple:jopt-simple:jar needed in 3 different versions: (3.2,
> 4.7, 5.0.2) Version conflict: package org.apache.lucene:lucene-
> analyzers-common:jar
> > needed in 2 different versions: (4.3.0, 7.1.0) Version conflict: package
> org.apache.lucene:lucene-core:jar needed in 2 different versions: (4.3.0,
> 7.1.0) Version conflict: package org.apache.lucene:lucene-grouping:jar
> needed in 2 different versions: (4.3.0, 7.1.0) Version conflict: package
> org.apache.lucene:lucene-highlighter:jar needed in 2 different versions:
> (4.3.0, 7.1.0) Version conflict: package org.apache.lucene:lucene-mem

Re: Merging to trunk?

[Wahrmann, Helmut]

 fully agreed.
But we shouldn't do that do support a 4 years old component.
The goal should be to support the latest releases.
It is ridiculous to release flume 1.9 with elasticsearch 0.90.1 support, when 
6.x is the current version.



Best regards,

Helmut Wahrmann


 Ursprüngliche Nachricht 
Von: Ralph Goers <ralph.go...@dslextreme.com>
Datum: 26.01.18 22:12 (GMT+01:00)
An: dev@flume.apache.org
Betreff: Re: Merging to trunk?

The “right” way to deal with this from a Maven perspective is to declare the 
version you want in the dependencyManagement section of the parent pom and to 
not specify any versions in child poms. Then use the Maven enforcer plugin to 
make sure everything required has a declaration in the dependencyManagement 
section.

Ralph

> On Jan 26, 2018, at 8:28 AM, Wahrmann, Helmut <helmut.wahrm...@rsa.com> wrote:
>
> Hi Mike,
>
> This only shows that some other component needs to be updated as well.
> Which software relies on Lucene 4.3.0, which was released in 2013?
> It is most probably not used by anyone else anymore.
>
> This needs to be solved. It makes no sense to release Flume 1.9 with support 
> for ElasticSearch 0.90.1.
> And it doesn't make sense either to release Flume 1.9 with support for a 
> component that relies on Lucene 4.3.0.
>
> We will always have dependency problems, if one component is updated to a 
> newer version and others not.
>
> To get rid of the conflict is probably to set "optional" to true in the 
> pom.xml.
> Then we don't distribute those Jars in the flume lib and the users need to 
> set their class path to point e.g. to Elasticsearch and then the correct libs 
> will be used.
> I doubt that a Elasticsearch user will also use some other component with 
> e.g. this outdated Lucene.
>
> best regards,
>
> Helmut
>
>
> -Original Message-
> From: Mike Percy [mailto:mpe...@apache.org]
> Sent: Donnerstag, 25. Jänner 2018 23:01
> To: dev@flume.apache.org
> Subject: Re: Merging to trunk?
>
> Hi Helmut,
> I wrote a small Perl script to identify dependency conflicts based on output 
> from mvn dependency:tree and posted it here: https://gist.github.com/
> mpercy/39614d770864bdd0c386befd5e8a1840
>
> I ran that on the current trunk and it actually found some errors which 
> should be fixed:
>
> Version conflict: package com.google.guava:guava:jar needed in 2 different
> versions: (11.0.2, 18.0)
> Version conflict: package commons-httpclient:commons-httpclient:jar needed in 
> 2 different versions: (3.0.1, 3.1) Version conflict: package 
> commons-logging:commons-logging:jar needed in 2 different versions: (1.1.3, 
> 1.2) Version conflict: package commons-pool:commons-pool:jar needed in 2 
> different versions: (1.5.4, 1.6) Version conflict: package 
> net.sf.jopt-simple:jopt-simple:jar needed in 2 different versions: (3.2, 4.7) 
> Version conflict: package org.codehaus.jackson:jackson-jaxrs:jar needed in
> 2 different versions: (1.8.3, 1.8.8)
> ERROR: 6 package version conflicts identified
>
> However, after applying your patch on top of trunk and running it again, it 
> identified new conflicts:
>
> Version conflict: package com.google.guava:guava:jar needed in 2 different
> versions: (11.0.2, 18.0)
> Version conflict: package commons-httpclient:commons-httpclient:jar needed in 
> 2 different versions: (3.0.1, 3.1) Version conflict: package 
> commons-logging:commons-logging:jar needed in 2 different versions: (1.1.3, 
> 1.2) Version conflict: package commons-pool:commons-pool:jar needed in 2 
> different versions: (1.5.4, 1.6) Version conflict: package 
> net.sf.jopt-simple:jopt-simple:jar needed in 3 different versions: (3.2, 4.7, 
> 5.0.2) Version conflict: package org.apache.lucene:lucene-analyzers-common:jar
> needed in 2 different versions: (4.3.0, 7.1.0) Version conflict: package 
> org.apache.lucene:lucene-core:jar needed in 2 different versions: (4.3.0, 
> 7.1.0) Version conflict: package org.apache.lucene:lucene-grouping:jar needed 
> in 2 different versions: (4.3.0, 7.1.0) Version conflict: package 
> org.apache.lucene:lucene-highlighter:jar needed in 2 different versions: 
> (4.3.0, 7.1.0) Version conflict: package org.apache.lucene:lucene-memory:jar 
> needed in 2 different versions: (4.3.0, 7.1.0) Version conflict: package 
> org.apache.lucene:lucene-misc:jar needed in 2 different versions: (4.3.0, 
> 7.1.0) Version conflict: package org.apache.lucene:lucene-queries:jar needed 
> in 2 different versions: (4.3.0, 7.1.0) Version conflict: package 
> org.apache.lucene:lucene-queryparser:jar needed in 2 different versions: 
> (4.3.0, 7.1.0) Version conflict: package org.apache.lucene:lucene-spatial:jar 
> needed in 2 different versions: (4.3.0, 7.1.0) Version conflict: pac

Re: Merging to trunk?

[Ralph Goers]

The “right” way to deal with this from a Maven perspective is to declare the 
version you want in the dependencyManagement section of the parent pom and to 
not specify any versions in child poms. Then use the Maven enforcer plugin to 
make sure everything required has a declaration in the dependencyManagement 
section. 

Ralph

> On Jan 26, 2018, at 8:28 AM, Wahrmann, Helmut <helmut.wahrm...@rsa.com> wrote:
> 
> Hi Mike,
> 
> This only shows that some other component needs to be updated as well.
> Which software relies on Lucene 4.3.0, which was released in 2013?
> It is most probably not used by anyone else anymore.
> 
> This needs to be solved. It makes no sense to release Flume 1.9 with support 
> for ElasticSearch 0.90.1.
> And it doesn't make sense either to release Flume 1.9 with support for a 
> component that relies on Lucene 4.3.0.
> 
> We will always have dependency problems, if one component is updated to a 
> newer version and others not.
> 
> To get rid of the conflict is probably to set "optional" to true in the 
> pom.xml.
> Then we don't distribute those Jars in the flume lib and the users need to 
> set their class path to point e.g. to Elasticsearch and then the correct libs 
> will be used.
> I doubt that a Elasticsearch user will also use some other component with 
> e.g. this outdated Lucene.
> 
> best regards,
> 
> Helmut
> 
> 
> -Original Message-
> From: Mike Percy [mailto:mpe...@apache.org] 
> Sent: Donnerstag, 25. Jänner 2018 23:01
> To: dev@flume.apache.org
> Subject: Re: Merging to trunk?
> 
> Hi Helmut,
> I wrote a small Perl script to identify dependency conflicts based on output 
> from mvn dependency:tree and posted it here: https://gist.github.com/
> mpercy/39614d770864bdd0c386befd5e8a1840
> 
> I ran that on the current trunk and it actually found some errors which 
> should be fixed:
> 
> Version conflict: package com.google.guava:guava:jar needed in 2 different
> versions: (11.0.2, 18.0)
> Version conflict: package commons-httpclient:commons-httpclient:jar needed in 
> 2 different versions: (3.0.1, 3.1) Version conflict: package 
> commons-logging:commons-logging:jar needed in 2 different versions: (1.1.3, 
> 1.2) Version conflict: package commons-pool:commons-pool:jar needed in 2 
> different versions: (1.5.4, 1.6) Version conflict: package 
> net.sf.jopt-simple:jopt-simple:jar needed in 2 different versions: (3.2, 4.7) 
> Version conflict: package org.codehaus.jackson:jackson-jaxrs:jar needed in
> 2 different versions: (1.8.3, 1.8.8)
> ERROR: 6 package version conflicts identified
> 
> However, after applying your patch on top of trunk and running it again, it 
> identified new conflicts:
> 
> Version conflict: package com.google.guava:guava:jar needed in 2 different
> versions: (11.0.2, 18.0)
> Version conflict: package commons-httpclient:commons-httpclient:jar needed in 
> 2 different versions: (3.0.1, 3.1) Version conflict: package 
> commons-logging:commons-logging:jar needed in 2 different versions: (1.1.3, 
> 1.2) Version conflict: package commons-pool:commons-pool:jar needed in 2 
> different versions: (1.5.4, 1.6) Version conflict: package 
> net.sf.jopt-simple:jopt-simple:jar needed in 3 different versions: (3.2, 4.7, 
> 5.0.2) Version conflict: package org.apache.lucene:lucene-analyzers-common:jar
> needed in 2 different versions: (4.3.0, 7.1.0) Version conflict: package 
> org.apache.lucene:lucene-core:jar needed in 2 different versions: (4.3.0, 
> 7.1.0) Version conflict: package org.apache.lucene:lucene-grouping:jar needed 
> in 2 different versions: (4.3.0, 7.1.0) Version conflict: package 
> org.apache.lucene:lucene-highlighter:jar needed in 2 different versions: 
> (4.3.0, 7.1.0) Version conflict: package org.apache.lucene:lucene-memory:jar 
> needed in 2 different versions: (4.3.0, 7.1.0) Version conflict: package 
> org.apache.lucene:lucene-misc:jar needed in 2 different versions: (4.3.0, 
> 7.1.0) Version conflict: package org.apache.lucene:lucene-queries:jar needed 
> in 2 different versions: (4.3.0, 7.1.0) Version conflict: package 
> org.apache.lucene:lucene-queryparser:jar needed in 2 different versions: 
> (4.3.0, 7.1.0) Version conflict: package org.apache.lucene:lucene-spatial:jar 
> needed in 2 different versions: (4.3.0, 7.1.0) Version conflict: package 
> org.apache.lucene:lucene-suggest:jar needed in 2 different versions: (4.3.0, 
> 7.1.0) Version conflict: package org.codehaus.jackson:jackson-jaxrs:jar 
> needed in
> 2 different versions: (1.8.3, 1.8.8)
> Version conflict: package org.yaml:snakeyaml:jar needed in 2 different
> versions: (1.10, 1.17)
> ERROR: 17 package version conflicts identified
> 
> So while it appears Guava is no longer a conce

RE: Merging to trunk?

[Wahrmann, Helmut]

Hi Mike,

This only shows that some other component needs to be updated as well.
Which software relies on Lucene 4.3.0, which was released in 2013?
It is most probably not used by anyone else anymore.

This needs to be solved. It makes no sense to release Flume 1.9 with support 
for ElasticSearch 0.90.1.
And it doesn't make sense either to release Flume 1.9 with support for a 
component that relies on Lucene 4.3.0.

We will always have dependency problems, if one component is updated to a newer 
version and others not.

To get rid of the conflict is probably to set "optional" to true in the pom.xml.
Then we don't distribute those Jars in the flume lib and the users need to set 
their class path to point e.g. to Elasticsearch and then the correct libs will 
be used.
I doubt that a Elasticsearch user will also use some other component with e.g. 
this outdated Lucene.

best regards,

Helmut
 

-Original Message-
From: Mike Percy [mailto:mpe...@apache.org] 
Sent: Donnerstag, 25. Jänner 2018 23:01
To: dev@flume.apache.org
Subject: Re: Merging to trunk?

Hi Helmut,
I wrote a small Perl script to identify dependency conflicts based on output 
from mvn dependency:tree and posted it here: https://gist.github.com/
mpercy/39614d770864bdd0c386befd5e8a1840

I ran that on the current trunk and it actually found some errors which should 
be fixed:

Version conflict: package com.google.guava:guava:jar needed in 2 different
versions: (11.0.2, 18.0)
Version conflict: package commons-httpclient:commons-httpclient:jar needed in 2 
different versions: (3.0.1, 3.1) Version conflict: package 
commons-logging:commons-logging:jar needed in 2 different versions: (1.1.3, 
1.2) Version conflict: package commons-pool:commons-pool:jar needed in 2 
different versions: (1.5.4, 1.6) Version conflict: package 
net.sf.jopt-simple:jopt-simple:jar needed in 2 different versions: (3.2, 4.7) 
Version conflict: package org.codehaus.jackson:jackson-jaxrs:jar needed in
2 different versions: (1.8.3, 1.8.8)
ERROR: 6 package version conflicts identified

However, after applying your patch on top of trunk and running it again, it 
identified new conflicts:

Version conflict: package com.google.guava:guava:jar needed in 2 different
versions: (11.0.2, 18.0)
Version conflict: package commons-httpclient:commons-httpclient:jar needed in 2 
different versions: (3.0.1, 3.1) Version conflict: package 
commons-logging:commons-logging:jar needed in 2 different versions: (1.1.3, 
1.2) Version conflict: package commons-pool:commons-pool:jar needed in 2 
different versions: (1.5.4, 1.6) Version conflict: package 
net.sf.jopt-simple:jopt-simple:jar needed in 3 different versions: (3.2, 4.7, 
5.0.2) Version conflict: package org.apache.lucene:lucene-analyzers-common:jar
needed in 2 different versions: (4.3.0, 7.1.0) Version conflict: package 
org.apache.lucene:lucene-core:jar needed in 2 different versions: (4.3.0, 
7.1.0) Version conflict: package org.apache.lucene:lucene-grouping:jar needed 
in 2 different versions: (4.3.0, 7.1.0) Version conflict: package 
org.apache.lucene:lucene-highlighter:jar needed in 2 different versions: 
(4.3.0, 7.1.0) Version conflict: package org.apache.lucene:lucene-memory:jar 
needed in 2 different versions: (4.3.0, 7.1.0) Version conflict: package 
org.apache.lucene:lucene-misc:jar needed in 2 different versions: (4.3.0, 
7.1.0) Version conflict: package org.apache.lucene:lucene-queries:jar needed in 
2 different versions: (4.3.0, 7.1.0) Version conflict: package 
org.apache.lucene:lucene-queryparser:jar needed in 2 different versions: 
(4.3.0, 7.1.0) Version conflict: package org.apache.lucene:lucene-spatial:jar 
needed in 2 different versions: (4.3.0, 7.1.0) Version conflict: package 
org.apache.lucene:lucene-suggest:jar needed in 2 different versions: (4.3.0, 
7.1.0) Version conflict: package org.codehaus.jackson:jackson-jaxrs:jar needed 
in
2 different versions: (1.8.3, 1.8.8)
Version conflict: package org.yaml:snakeyaml:jar needed in 2 different
versions: (1.10, 1.17)
ERROR: 17 package version conflicts identified

So while it appears Guava is no longer a concern, there are other conflicting 
dependencies involved in the conflict here.

I'm interested to hear your thoughts on this.

Regards,
Mike

On Wed, Jan 24, 2018 at 7:57 AM, Wahrmann, Helmut <helmut.wahrm...@rsa.com>
wrote:

> Hi Mike,
>
> Thanks for the quick response.
>
> I fully agree that we need to take care about interop between the 
> different components and dependencies.
> But if you look at the patch in FLUME-3021, it only puts in 
> Elasticsearch dependencies.
>
> The issue with Guava, which you saw in FLUME-2921 is no longer there, 
> because since mid-2016 a lot has changed in the Flume trunk and all 
> those dependencies are fixed.
> It won't even be possible gto apply this patch anymore.
>
> The patch in FLUME-3021 works with the latest Elasticsearch versions, 
> without introducing

Re: Merging to trunk?

[Mike Percy]

Hi Helmut,
I wrote a small Perl script to identify dependency conflicts based on
output from mvn dependency:tree and posted it here: https://gist.github.com/
mpercy/39614d770864bdd0c386befd5e8a1840

I ran that on the current trunk and it actually found some errors which
should be fixed:

Version conflict: package com.google.guava:guava:jar needed in 2 different
versions: (11.0.2, 18.0)
Version conflict: package commons-httpclient:commons-httpclient:jar needed
in 2 different versions: (3.0.1, 3.1)
Version conflict: package commons-logging:commons-logging:jar needed in 2
different versions: (1.1.3, 1.2)
Version conflict: package commons-pool:commons-pool:jar needed in 2
different versions: (1.5.4, 1.6)
Version conflict: package net.sf.jopt-simple:jopt-simple:jar needed in 2
different versions: (3.2, 4.7)
Version conflict: package org.codehaus.jackson:jackson-jaxrs:jar needed in
2 different versions: (1.8.3, 1.8.8)
ERROR: 6 package version conflicts identified

However, after applying your patch on top of trunk and running it again, it
identified new conflicts:

Version conflict: package com.google.guava:guava:jar needed in 2 different
versions: (11.0.2, 18.0)
Version conflict: package commons-httpclient:commons-httpclient:jar needed
in 2 different versions: (3.0.1, 3.1)
Version conflict: package commons-logging:commons-logging:jar needed in 2
different versions: (1.1.3, 1.2)
Version conflict: package commons-pool:commons-pool:jar needed in 2
different versions: (1.5.4, 1.6)
Version conflict: package net.sf.jopt-simple:jopt-simple:jar needed in 3
different versions: (3.2, 4.7, 5.0.2)
Version conflict: package org.apache.lucene:lucene-analyzers-common:jar
needed in 2 different versions: (4.3.0, 7.1.0)
Version conflict: package org.apache.lucene:lucene-core:jar needed in 2
different versions: (4.3.0, 7.1.0)
Version conflict: package org.apache.lucene:lucene-grouping:jar needed in 2
different versions: (4.3.0, 7.1.0)
Version conflict: package org.apache.lucene:lucene-highlighter:jar needed
in 2 different versions: (4.3.0, 7.1.0)
Version conflict: package org.apache.lucene:lucene-memory:jar needed in 2
different versions: (4.3.0, 7.1.0)
Version conflict: package org.apache.lucene:lucene-misc:jar needed in 2
different versions: (4.3.0, 7.1.0)
Version conflict: package org.apache.lucene:lucene-queries:jar needed in 2
different versions: (4.3.0, 7.1.0)
Version conflict: package org.apache.lucene:lucene-queryparser:jar needed
in 2 different versions: (4.3.0, 7.1.0)
Version conflict: package org.apache.lucene:lucene-spatial:jar needed in 2
different versions: (4.3.0, 7.1.0)
Version conflict: package org.apache.lucene:lucene-suggest:jar needed in 2
different versions: (4.3.0, 7.1.0)
Version conflict: package org.codehaus.jackson:jackson-jaxrs:jar needed in
2 different versions: (1.8.3, 1.8.8)
Version conflict: package org.yaml:snakeyaml:jar needed in 2 different
versions: (1.10, 1.17)
ERROR: 17 package version conflicts identified

So while it appears Guava is no longer a concern, there are other
conflicting dependencies involved in the conflict here.

I'm interested to hear your thoughts on this.

Regards,
Mike

On Wed, Jan 24, 2018 at 7:57 AM, Wahrmann, Helmut <helmut.wahrm...@rsa.com>
wrote:

> Hi Mike,
>
> Thanks for the quick response.
>
> I fully agree that we need to take care about interop between the
> different components and dependencies.
> But if you look at the patch in FLUME-3021, it only puts in Elasticsearch
> dependencies.
>
> The issue with Guava, which you saw in FLUME-2921 is no longer there,
> because since mid-2016 a lot has changed in the Flume trunk and all those
> dependencies are fixed.
> It won't even be possible gto apply this patch anymore.
>
> The patch in FLUME-3021 works with the latest Elasticsearch versions,
> without introducing additional dependencies, which might cause problems to
> other projects.
> So I think it can be safely merged into trunk.
>
> best regards,
>
> Helmut
>
>
> -Original Message-
> From: Mike Percy [mailto:mpe...@apache.org]
> Sent: Dienstag, 23. Jänner 2018 20:39
> To: dev@flume.apache.org
> Subject: Re: Merging to trunk?
>
> Hi Helmut,
> Thank you for bringing this up on dev@ and thank you for the patch. I see
> there are other people people interested in this component upgrade as well.
>
> As you are probably aware, a Flume committer will need to approve the
> change before I gets merged to trunk.
>
> My primary concern w/ merging this would be compatibility of the
> dependencies. Flume suffers from a kind of a "dependency hell" because of a
> lack of classloading support (either via OSGI or Java modules). See
> https://issues.apache.org/jira/browse/FLUME-2293 for a tracking ticket
> about that issue. What that means is that all of the components that Flume
> ships must have compatible dependencies with eac

RE: Merging to trunk?

[Wahrmann, Helmut]

Hi Mike,

Thanks for the quick response.

I fully agree that we need to take care about interop between the different 
components and dependencies.
But if you look at the patch in FLUME-3021, it only puts in Elasticsearch 
dependencies.

The issue with Guava, which you saw in FLUME-2921 is no longer there, because 
since mid-2016 a lot has changed in the Flume trunk and all those dependencies 
are fixed.
It won't even be possible gto apply this patch anymore.

The patch in FLUME-3021 works with the latest Elasticsearch versions, without 
introducing additional dependencies, which might cause problems to other 
projects.
So I think it can be safely merged into trunk.

best regards,

Helmut
 

-Original Message-
From: Mike Percy [mailto:mpe...@apache.org] 
Sent: Dienstag, 23. Jänner 2018 20:39
To: dev@flume.apache.org
Subject: Re: Merging to trunk?

Hi Helmut,
Thank you for bringing this up on dev@ and thank you for the patch. I see there 
are other people people interested in this component upgrade as well.

As you are probably aware, a Flume committer will need to approve the change 
before I gets merged to trunk.

My primary concern w/ merging this would be compatibility of the dependencies. 
Flume suffers from a kind of a "dependency hell" because of a lack of 
classloading support (either via OSGI or Java modules). See
https://issues.apache.org/jira/browse/FLUME-2293 for a tracking ticket about 
that issue. What that means is that all of the components that Flume ships must 
have compatible dependencies with each other which makes changes like this more 
complex. Therefore I would like someone to verify that mvn dependency:tree does 
not show conflicts when run from the top level with the new patches. If memory 
serves, I believe Google Guava is likely to conflict.

Also, I think that https://issues.apache.org/jira/browse/FLUME-3021 is a 
duplicate of https://issues.apache.org/jira/browse/FLUME-2921 which has 
additional information about the various issues that we need to solve to do 
this upgrade.

I'd be happy to discuss this issue some more on this thread.

Mike


On Tue, Jan 23, 2018 at 12:57 AM, Wahrmann, Helmut <helmut.wahrm...@rsa.com>
wrote:

> Hi,
>
> Who decides if/when a patch or pull request gets merged to trunk?
> Reason I am asking is for the Elasticsearch support. The current code 
> in trunk does not work with ES > 2.x.
> Currently Elasticsearch is at 6.1.
>
> In FLUME-3021 we have several patches since March last year.
> I have a patched Flume running at a customer since April last year 
> without any problems.
>
> So why not merging those changes into the trunk?
> As I stated in my comment in FLUME-3021, it will not cause any 
> problems, cause the current trunk won't work with newer ES versions anyhow.
> I doubt that someone is out there still running ES 0.95.
>
> best regards,
>
> Helmut
>

Re: Merging to trunk?

[Mike Percy]

Hi Helmut,
Thank you for bringing this up on dev@ and thank you for the patch. I see
there are other people people interested in this component upgrade as well.

As you are probably aware, a Flume committer will need to approve the
change before I gets merged to trunk.

My primary concern w/ merging this would be compatibility of the
dependencies. Flume suffers from a kind of a "dependency hell" because of a
lack of classloading support (either via OSGI or Java modules). See
https://issues.apache.org/jira/browse/FLUME-2293 for a tracking ticket
about that issue. What that means is that all of the components that Flume
ships must have compatible dependencies with each other which makes changes
like this more complex. Therefore I would like someone to verify that mvn
dependency:tree does not show conflicts when run from the top level with
the new patches. If memory serves, I believe Google Guava is likely to
conflict.

Also, I think that https://issues.apache.org/jira/browse/FLUME-3021 is a
duplicate of https://issues.apache.org/jira/browse/FLUME-2921 which has
additional information about the various issues that we need to solve to do
this upgrade.

I'd be happy to discuss this issue some more on this thread.

Mike


On Tue, Jan 23, 2018 at 12:57 AM, Wahrmann, Helmut 
wrote:

> Hi,
>
> Who decides if/when a patch or pull request gets merged to trunk?
> Reason I am asking is for the Elasticsearch support. The current code in
> trunk does not work with ES > 2.x.
> Currently Elasticsearch is at 6.1.
>
> In FLUME-3021 we have several patches since March last year.
> I have a patched Flume running at a customer since April last year without
> any problems.
>
> So why not merging those changes into the trunk?
> As I stated in my comment in FLUME-3021, it will not cause any problems,
> cause the current trunk won't work with newer ES versions anyhow.
> I doubt that someone is out there still running ES 0.95.
>
> best regards,
>
> Helmut
>