RE: Updating 3rd party dependencies
The Elasticsearch sink shouldn't be a problem. No one Is using that anyhow, since Flume is supporting version 0.9 of Elastic and Elastic is at version 7.0. best regards, Helmut -Original Message- From: Ferenc Szabo Sent: Montag, 6. Mai 2019 09:13 To: dev@flume.apache.org Subject: Re: Updating 3rd party dependencies [EXTERNAL EMAIL] I meant the other way around as Java 8 is EOL and users have to upgrade to Java 11. I would like to achieve Java 11 support and now Flume does _not_ run with Java 11 because of its dependencies. for example from the unit tests: - Hive sink does not work with Java 11 - Elasticsearch sink does not work with Java 11 here is my branch where I did some work on Java 11 support: https://github.com/apache/flume/compare/trunk...szaboferee:j11?expand=1 On Fri, May 3, 2019 at 6:58 PM Ralph Goers wrote: > I don’t think a major version change should be necessary just because > the JDK dependency changed. Changing the major version number should > tied to major changes in the project itself. > > I am curious to know what dependencies now only run on Java 11. > > Ralph > > > On May 3, 2019, at 5:02 AM, Ferenc Szabo wrote: > > > > Hello Flume devs > > > > I was looking into the Java 11 support in Flume and some of the > > dependencies are not compatible with Java 11 and does not have > > compatible versions within the same major version. > > > > Also, some of them have security vulnerabilities that have been > > fixed in another major version. > > > > What do you think about updating dependencies to a higher major > > version > and > > possibly losing some backward compatibility in 1.10? Or should we > > move to > > 2.0 with more effort and updating there? Also adding some framework > changes > > that would enable us to implement things we could not do in a minor > version. > > > > We should create a roadmap for us containing library updates, new > > components and framework improvements. > > > > Regards, > > Ferenc Szabo > > >
What do we do with Integrations with no maintainer?
Hi all, What are we doing with integrations having no maintainer? An example is the morphline sink. It supports Solr 4.3 and Apache has already relased Solr 7.5.0. Kite SDK is at 1.1. Seems that no one is taking care of it. On the other site we are still "supporting" Elasticsearch 0.90.1, while ElasticSearch is already on 6.5. Having a dependency of Lucene between Solr and Elastic I cannot push my changes to flume. So while there would be a maintainer for ElasticSearch, new features cannot be introduced, because of the above. Shouldn't we deprecate old stuff, where no maintainer is active? If 1.9 is released with support of outdated SOLR and ElasticSearch, no one would use Flume anyhow. I doubt that someone would downgrade the ElasticSearch cluster to 0.90 just to be able to pump in events via Flume. I keep a working ElasticSearch 6.x Fork anyhow, but would like to contribute back to the project. Note: I have tried to upgrade SOLR support to the newest version, but it fails on the test cases and I don't have anything to test. Thanks, Helmut
RE: Merge of patch in Flume-3021?
Hi Mike,
Did you have a chance to look at that?
best regards,
Helmut
-Original Message-
From: Wahrmann, Helmut
Sent: Dienstag, 17. April 2018 18:53
To: 'dev@flume.apache.org'
Subject: RE: Merge of patch in Flume-3021?
Hi Mike,
Did you have a chance to look into this?
thanks,
Helmut
-Original Message-
From: Mike Percy [mailto:mpe...@apache.org]
Sent: Montag, 26. März 2018 21:46
To: dev@flume.apache.org
Subject: Re: Merge of patch in Flume-3021?
I haven't figured this out yet but I'll look into it this week.
Mike
On Mon, Mar 19, 2018 at 2:41 AM, Wahrmann, Helmut
wrote:
> Hi Mike,
>
> With the help of Ferenc, I got rid of the initial errors.
>
> Only one is remaining now:
>
> [INFO] ---
> [INFO] T E S T S
> [INFO] ---
> [INFO] Running
> org.apache.flume.sink.solr.morphline.TestBlobDeserializer
> [INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
> 1.126 s - in org.apache.flume.sink.solr.morphline.TestBlobDeserializer
> [INFO] Running org.apache.flume.sink.solr.morphline.TestBlobHandler
> [INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
> 1.125 s - in org.apache.flume.sink.solr.morphline.TestBlobHandler
> [INFO] Running org.apache.flume.sink.solr.morphline.
> TestMorphlineInterceptor
> [INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
> 7.724 s - in
> org.apache.flume.sink.solr.morphline.TestMorphlineInterceptor
> [INFO] Running
> org.apache.flume.sink.solr.morphline.TestMorphlineSolrSink
> [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
> 1.797 s <<< FAILURE! - in org.apache.flume.sink.solr.morphline.
> TestMorphlineSolrSink
> [ERROR] org.apache.flume.sink.solr.morphline.TestMorphlineSolrSink
> Time
> elapsed: 1.797 s <<< ERROR!
> java.util.IllformedLocaleException: Invalid subtag: en_us [at index 0]
>
> [INFO] Running
> org.apache.flume.sink.solr.morphline.TestUUIDInterceptor
> [INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
> 0.391 s - in org.apache.flume.sink.solr.morphline.TestUUIDInterceptor
> [INFO]
> [INFO] Results:
> [INFO]
> [ERROR] Errors:
> [ERROR] TestMorphlineSolrSink>LuceneTestCase.localeForLanguageTag:1588
> ╗ IllformedLocale
> [INFO]
> [ERROR] Tests run: 17, Failures: 0, Errors: 1, Skipped: 0 [INFO]
> [INFO]
> --
> --
> [INFO] BUILD FAILURE
>
>
> The LuceneTestCase is extended by SolrTestCaseJ4.
> No idea, what I could do against this.
>
> best regards,
>
> Helmut
>
> -Original Message-
> From: Mike Percy [mailto:mpe...@apache.org]
> Sent: Montag, 19. März 2018 05:12
> To: dev@flume.apache.org
> Subject: Re: Merge of patch in Flume-3021?
>
> Nice! Thanks Ferenc. Answered better than I could have done. :)
>
> Sorry, I meant to send this last week but I just found it in my drafts.
>
> Helmut, please let us know if you need more help with this.
>
> Mike
>
> On Tue, Mar 6, 2018 at 7:25 AM, Ferenc Szabo wrote:
>
> > the createJetty method of a Test class became final in the new
> > versions of solr.
> > the kite sdk test-jar has to be removed because it depends on a
> > different incompatible version of solr
> >
> >
> > org.kitesdk
> > kite-morphlines-solr-core
> > ${kite.version}
> > test-jar
> > test
> >
> >
> > TestEnvironment.java has to be removed as well because it depends on
> > the incompatible dependency
> >
> > then we need this class:
> > https://github.com/kite-sdk/kite/blob/master/kite-morphlines
> > /kite-morphlines-solr-core/src/test/java/org/kitesdk/
> > morphline/solr/TestEmbeddedSolrServer.java
> > I believe it is ok to have a copy of this because it is part of the
> > incompatible test dependency we just removed
> >
> > then we need a newer version of commons-compress:
> > 1.10
> >
> > from here you can fix the actual solr related test errors :)
> >
> >
> >
> > On Tue, Mar 6, 2018 at 11:56 AM, Wahrmann, Helmut
> > > >
> > wrote:
> >
> > > Hi Ferenc,
> > >
> > > Thanks for offering help.
> > > In agreement with Mike I want to have support for Solr 7.2.1 in
> > > the morphline solr sink, so that we can easily upgrade the
> > > Elasticsearch
> > sink.
> > >
> > > My updates are here: https://github.com/hwahrmann/
> > > flume/tree/Upgrade_Morphline_Sink
> >
RE: Merge of patch in Flume-3021?
Hi Mike,
Did you have a chance to look into this?
thanks,
Helmut
-Original Message-
From: Mike Percy [mailto:mpe...@apache.org]
Sent: Montag, 26. März 2018 21:46
To: dev@flume.apache.org
Subject: Re: Merge of patch in Flume-3021?
I haven't figured this out yet but I'll look into it this week.
Mike
On Mon, Mar 19, 2018 at 2:41 AM, Wahrmann, Helmut <helmut.wahrm...@rsa.com>
wrote:
> Hi Mike,
>
> With the help of Ferenc, I got rid of the initial errors.
>
> Only one is remaining now:
>
> [INFO] ---
> [INFO] T E S T S
> [INFO] ---
> [INFO] Running
> org.apache.flume.sink.solr.morphline.TestBlobDeserializer
> [INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
> 1.126 s - in org.apache.flume.sink.solr.morphline.TestBlobDeserializer
> [INFO] Running org.apache.flume.sink.solr.morphline.TestBlobHandler
> [INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
> 1.125 s - in org.apache.flume.sink.solr.morphline.TestBlobHandler
> [INFO] Running org.apache.flume.sink.solr.morphline.
> TestMorphlineInterceptor
> [INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
> 7.724 s - in
> org.apache.flume.sink.solr.morphline.TestMorphlineInterceptor
> [INFO] Running
> org.apache.flume.sink.solr.morphline.TestMorphlineSolrSink
> [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
> 1.797 s <<< FAILURE! - in org.apache.flume.sink.solr.morphline.
> TestMorphlineSolrSink
> [ERROR] org.apache.flume.sink.solr.morphline.TestMorphlineSolrSink
> Time
> elapsed: 1.797 s <<< ERROR!
> java.util.IllformedLocaleException: Invalid subtag: en_us [at index 0]
>
> [INFO] Running
> org.apache.flume.sink.solr.morphline.TestUUIDInterceptor
> [INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
> 0.391 s - in org.apache.flume.sink.solr.morphline.TestUUIDInterceptor
> [INFO]
> [INFO] Results:
> [INFO]
> [ERROR] Errors:
> [ERROR] TestMorphlineSolrSink>LuceneTestCase.localeForLanguageTag:1588
> ╗ IllformedLocale
> [INFO]
> [ERROR] Tests run: 17, Failures: 0, Errors: 1, Skipped: 0 [INFO]
> [INFO]
> --
> --
> [INFO] BUILD FAILURE
>
>
> The LuceneTestCase is extended by SolrTestCaseJ4.
> No idea, what I could do against this.
>
> best regards,
>
> Helmut
>
> -Original Message-
> From: Mike Percy [mailto:mpe...@apache.org]
> Sent: Montag, 19. März 2018 05:12
> To: dev@flume.apache.org
> Subject: Re: Merge of patch in Flume-3021?
>
> Nice! Thanks Ferenc. Answered better than I could have done. :)
>
> Sorry, I meant to send this last week but I just found it in my drafts.
>
> Helmut, please let us know if you need more help with this.
>
> Mike
>
> On Tue, Mar 6, 2018 at 7:25 AM, Ferenc Szabo <fsz...@cloudera.com> wrote:
>
> > the createJetty method of a Test class became final in the new
> > versions of solr.
> > the kite sdk test-jar has to be removed because it depends on a
> > different incompatible version of solr
> >
> >
> > org.kitesdk
> > kite-morphlines-solr-core
> > ${kite.version}
> > test-jar
> > test
> >
> >
> > TestEnvironment.java has to be removed as well because it depends on
> > the incompatible dependency
> >
> > then we need this class:
> > https://github.com/kite-sdk/kite/blob/master/kite-morphlines
> > /kite-morphlines-solr-core/src/test/java/org/kitesdk/
> > morphline/solr/TestEmbeddedSolrServer.java
> > I believe it is ok to have a copy of this because it is part of the
> > incompatible test dependency we just removed
> >
> > then we need a newer version of commons-compress:
> > 1.10
> >
> > from here you can fix the actual solr related test errors :)
> >
> >
> >
> > On Tue, Mar 6, 2018 at 11:56 AM, Wahrmann, Helmut
> > <helmut.wahrm...@rsa.com
> > >
> > wrote:
> >
> > > Hi Ferenc,
> > >
> > > Thanks for offering help.
> > > In agreement with Mike I want to have support for Solr 7.2.1 in
> > > the morphline solr sink, so that we can easily upgrade the
> > > Elasticsearch
> > sink.
> > >
> > > My updates are here: https://github.com/hwahrmann/
> > > flume/tree/Upgrade_Morphline_Sink
> > >
> > > I changed the solr version to 7.2.1 and was able to compile the
> > > sink withput any problems.
&g
RE: Merge of patch in Flume-3021?
Hi Mike,
With the help of Ferenc, I got rid of the initial errors.
Only one is remaining now:
[INFO] ---
[INFO] T E S T S
[INFO] ---
[INFO] Running org.apache.flume.sink.solr.morphline.TestBlobDeserializer
[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.126 s
- in org.apache.flume.sink.solr.morphline.TestBlobDeserializer
[INFO] Running org.apache.flume.sink.solr.morphline.TestBlobHandler
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.125 s
- in org.apache.flume.sink.solr.morphline.TestBlobHandler
[INFO] Running org.apache.flume.sink.solr.morphline.TestMorphlineInterceptor
[INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.724 s
- in org.apache.flume.sink.solr.morphline.TestMorphlineInterceptor
[INFO] Running org.apache.flume.sink.solr.morphline.TestMorphlineSolrSink
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.797 s
<<< FAILURE! - in org.apache.flume.sink.solr.morphline.TestMorphlineSolrSink
[ERROR] org.apache.flume.sink.solr.morphline.TestMorphlineSolrSink Time
elapsed: 1.797 s <<< ERROR!
java.util.IllformedLocaleException: Invalid subtag: en_us [at index 0]
[INFO] Running org.apache.flume.sink.solr.morphline.TestUUIDInterceptor
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.391 s
- in org.apache.flume.sink.solr.morphline.TestUUIDInterceptor
[INFO]
[INFO] Results:
[INFO]
[ERROR] Errors:
[ERROR] TestMorphlineSolrSink>LuceneTestCase.localeForLanguageTag:1588 ╗
IllformedLocale
[INFO]
[ERROR] Tests run: 17, Failures: 0, Errors: 1, Skipped: 0 [INFO] [INFO]
[INFO] BUILD FAILURE
The LuceneTestCase is extended by SolrTestCaseJ4.
No idea, what I could do against this.
best regards,
Helmut
-Original Message-
From: Mike Percy [mailto:mpe...@apache.org]
Sent: Montag, 19. März 2018 05:12
To: dev@flume.apache.org
Subject: Re: Merge of patch in Flume-3021?
Nice! Thanks Ferenc. Answered better than I could have done. :)
Sorry, I meant to send this last week but I just found it in my drafts.
Helmut, please let us know if you need more help with this.
Mike
On Tue, Mar 6, 2018 at 7:25 AM, Ferenc Szabo <fsz...@cloudera.com> wrote:
> the createJetty method of a Test class became final in the new
> versions of solr.
> the kite sdk test-jar has to be removed because it depends on a
> different incompatible version of solr
>
>
> org.kitesdk
> kite-morphlines-solr-core
> ${kite.version}
> test-jar
> test
>
>
> TestEnvironment.java has to be removed as well because it depends on
> the incompatible dependency
>
> then we need this class:
> https://github.com/kite-sdk/kite/blob/master/kite-morphlines
> /kite-morphlines-solr-core/src/test/java/org/kitesdk/
> morphline/solr/TestEmbeddedSolrServer.java
> I believe it is ok to have a copy of this because it is part of the
> incompatible test dependency we just removed
>
> then we need a newer version of commons-compress:
> 1.10
>
> from here you can fix the actual solr related test errors :)
>
>
>
> On Tue, Mar 6, 2018 at 11:56 AM, Wahrmann, Helmut
> <helmut.wahrm...@rsa.com
> >
> wrote:
>
> > Hi Ferenc,
> >
> > Thanks for offering help.
> > In agreement with Mike I want to have support for Solr 7.2.1 in the
> > morphline solr sink, so that we can easily upgrade the Elasticsearch
> sink.
> >
> > My updates are here: https://github.com/hwahrmann/
> > flume/tree/Upgrade_Morphline_Sink
> >
> > I changed the solr version to 7.2.1 and was able to compile the sink
> > withput any problems.
> > I can also compile the tests, but when running, I get multiple
> > errors
> like
> > this:
> >
> > [INFO] Running org.apache.flume.sink.solr.morphline.
> > TestMorphlineInterceptor
> > [ERROR] Tests run: 66, Failures: 0, Errors: 66, Skipped: 0, Time elapsed:
> > 24.438 s <<< FAILURE! - in org.apache.flume.sink.solr.morphline.
> > TestMorphlineInterceptor
> > [ERROR] testIfDetectMimeTypeRouteToNorthPole(org.apache.flume.sink.
> > solr.morphline.TestMorphlineInterceptor) Time elapsed: 1.985 s <<<
> > ERROR!
> > java.lang.VerifyError: class org.kitesdk.morphline.solr.Abs
> tractSolrMorphlineZkTest
> > overrides final method createJetty.(Ljava/io/File;
> > Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;
> > Ljava/lang/String;)Lorg/apache/solr/client/solrj/embedded/
> JettySolrRunner;
> > at
> > org.apache.flume.sink.solr.morphline.TestMorphlineIntercepto
> r
RE: Merge of patch in Flume-3021?
Hi Ferenc, Thanks for offering help. In agreement with Mike I want to have support for Solr 7.2.1 in the morphline solr sink, so that we can easily upgrade the Elasticsearch sink. My updates are here: https://github.com/hwahrmann/flume/tree/Upgrade_Morphline_Sink I changed the solr version to 7.2.1 and was able to compile the sink withput any problems. I can also compile the tests, but when running, I get multiple errors like this: [INFO] Running org.apache.flume.sink.solr.morphline.TestMorphlineInterceptor [ERROR] Tests run: 66, Failures: 0, Errors: 66, Skipped: 0, Time elapsed: 24.438 s <<< FAILURE! - in org.apache.flume.sink.solr.morphline.TestMorphlineInterceptor [ERROR] testIfDetectMimeTypeRouteToNorthPole(org.apache.flume.sink.solr.morphline.TestMorphlineInterceptor) Time elapsed: 1.985 s <<< ERROR! java.lang.VerifyError: class org.kitesdk.morphline.solr.AbstractSolrMorphlineZkTest overrides final method createJetty.(Ljava/io/File;Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;)Lorg/apache/solr/client/solrj/embedded/JettySolrRunner; at org.apache.flume.sink.solr.morphline.TestMorphlineInterceptor.build(TestMorphlineInterceptor.java:151) at org.apache.flume.sink.solr.morphline.TestMorphlineInterceptor.testIfDetectMimeTypeRouteToNorthPole(TestMorphlineInterceptor.java:139) [ERROR] testGrokIfNotMatchDropEventRetain(org.apache.flume.sink.solr.morphline.TestMorphlineInterceptor) Time elapsed: 0.363 s <<< ERROR! java.lang.VerifyError: class org.kitesdk.morphline.solr.AbstractSolrMorphlineZkTest overrides final method createJetty.(Ljava/io/File;Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;)Lorg/apache/solr/client/solrj/embedded/JettySolrRunner; at org.apache.flume.sink.solr.morphline.TestMorphlineInterceptor.build(TestMorphlineInterceptor.java:151) at org.apache.flume.sink.solr.morphline.TestMorphlineInterceptor.testGrokIfNotMatchDropEventRetain(TestMorphlineInterceptor.java:83) The all have problems with createJetty. So it seems that I maybe need a different version of jetty or something like that. And for that I have too less knowledge about maven. thx, Helmut -Original Message- From: Ferenc Szabo [mailto:fsz...@cloudera.com] Sent: Dienstag, 6. März 2018 11:04 To: dev@flume.apache.org Subject: Re: Merge of patch in Flume-3021? Hi Helmut, let me know what can I help You with. share your current code on a github fork and describe the issue. I will see what can we do to solve it. On Tue, Mar 6, 2018 at 10:50 AM, Wahrmann, Helmut <helmut.wahrm...@rsa.com> wrote: > Hi Mike, > > I am stuck with Solr. > The morphline-solr sink compiles without any problems, but I am > struggling with the tests. > Seems I need to exclude some stuff from maven, but my knowledge about > maven is not good enough to figure out what to do. > Anyone able to help? > > regards, > > Helmut > > -Original Message- > From: Mike Percy [mailto:mpe...@apache.org] > Sent: Mittwoch, 14. Februar 2018 19:19 > To: dev@flume.apache.org > Subject: Re: Merge of patch in Flume-3021? > > Hi Helmut, > As long as the integration tests still pass and the packaging issues > are not exacerbated, I don't see why we couldn't merge an upgrade > patch, barring any serious concerns with the patch. > > Mike > > On Wed, Feb 14, 2018 at 1:47 AM, Wahrmann, Helmut > <helmut.wahrm...@rsa.com > > > wrote: > > > Hi Mike, > > > > I won't have a problem upgrading the Solr sink to the latest version. > > I am missing test environment however. > > So while it may build correctly and all integration tests work, I > > have no real environment to test with. > > > > best regards, > > Helmut > > > > -Original Message- > > From: Mike Percy [mailto:mpe...@apache.org] > > Sent: Mittwoch, 14. Februar 2018 00:38 > > To: dev@flume.apache.org > > Subject: Re: Merge of patch in Flume-3021? > > > > OK. In the pull request, it would be nice if whoever submits or > > merges it mentions all of the contributors to the patch in the commit > > message. > > > > I asked Wolfgang H. about the SolrServer thing and this is what he > > told > me: > > > > Hi Mike, the class has been renamed to "SolrClient" (which > > unfortunately > > > breaks compat). It's just a class rename. The functionality is the > > > same as before. It was called SolrServer in Solr4 because it was a > > > client proxy that sends RPCs to a Solr server, but calling it > > > SolrClient is more straightforward to understand, hence the > > > community decided to rename the class. > > > It's possible to spawn an embedded S
RE: Merge of patch in Flume-3021?
Hi Mike, I am stuck with Solr. The morphline-solr sink compiles without any problems, but I am struggling with the tests. Seems I need to exclude some stuff from maven, but my knowledge about maven is not good enough to figure out what to do. Anyone able to help? regards, Helmut -Original Message- From: Mike Percy [mailto:mpe...@apache.org] Sent: Mittwoch, 14. Februar 2018 19:19 To: dev@flume.apache.org Subject: Re: Merge of patch in Flume-3021? Hi Helmut, As long as the integration tests still pass and the packaging issues are not exacerbated, I don't see why we couldn't merge an upgrade patch, barring any serious concerns with the patch. Mike On Wed, Feb 14, 2018 at 1:47 AM, Wahrmann, Helmut <helmut.wahrm...@rsa.com> wrote: > Hi Mike, > > I won't have a problem upgrading the Solr sink to the latest version. > I am missing test environment however. > So while it may build correctly and all integration tests work, I have > no real environment to test with. > > best regards, > Helmut > > -Original Message- > From: Mike Percy [mailto:mpe...@apache.org] > Sent: Mittwoch, 14. Februar 2018 00:38 > To: dev@flume.apache.org > Subject: Re: Merge of patch in Flume-3021? > > OK. In the pull request, it would be nice if whoever submits or merges > it mentions all of the contributors to the patch in the commit message. > > I asked Wolfgang H. about the SolrServer thing and this is what he told me: > > Hi Mike, the class has been renamed to "SolrClient" (which > unfortunately > > breaks compat). It's just a class rename. The functionality is the > > same as before. It was called SolrServer in Solr4 because it was a > > client proxy that sends RPCs to a Solr server, but calling it > > SolrClient is more straightforward to understand, hence the > > community decided to rename the class. > > It's possible to spawn an embedded Solr server, for example for > > testing purposes, via class EmbeddedSolrServer (a class that retains > > the same name in Solr7 and Solr4), which extends the SolrClient class. > > > Hope this helps, > Mike > > On Tue, Feb 13, 2018 at 4:41 AM, Wahrmann, Helmut > <helmut.wahrm...@rsa.com > > > wrote: > > > Hi Mike, > > > > Thanks for the response. Would be cool if we get that sorted out. > > > > I've asked Yonghao Zou to submit the Pull Request, since he did most > > of the work and should get the credit. > > He'll do so after the Chinese New Year's Eve. > > > > I will then issue a Pull request for the new ES Rest client, which > > is dependent on the above work. > > > > best regards, > > Helmut > > > > -Original Message- > > From: Mike Percy [mailto:mpe...@apache.org] > > Sent: Dienstag, 13. Februar 2018 04:30 > > To: dev@flume.apache.org > > Subject: Re: Merge of patch in Flume-3021? > > > > Hi Helmut, > > I see that I neglected to follow up on the other thread on this > > topic after your reply about SolrServer missing from the solrj jar. > > Let me ask around w/ some folks I know that work on Solr and see if > > there is any way to retain the SolrServer for our tests after > > upgrading to the > new version. > > > > Thank you very much for working on upgrading Solr. Would you mind > > submitting a pull request with your (apparently work-in-progress) > > patch to upgrade both Solr and ES? > > > > To reply to your email in this thread, the JAR packaging situation > > is largely the same after merging FLUME-2957 so unfortunately most > > of what I noted in my reply in the other thread ( > > https://s.apache.org/GqcX ) still holds. > > > > I hope that we can upgrade the Solr dependencies as part of the same > > commit as the ES dependencies to avoid worrying about which lucene > > jar is first in the classpath, and ensure we are not adding any > > additional dependency conflicts to mvn dependency:tree. > > > > Regards, > > Mike > > > > On Mon, Feb 12, 2018 at 12:54 AM, Wahrmann, Helmut < > > helmut.wahrm...@rsa.com> > > wrote: > > > > > Hi, > > > > > > now that the blocker for FLUME-3021 is removed by committing > > > FLUME-2957, can we get the patch from 3021 merged to trunk? > > > > > > Thanks, > > > > > > Helmut > > > > > >
RE: Merge of patch in Flume-3021?
Hi Mike, I won't have a problem upgrading the Solr sink to the latest version. I am missing test environment however. So while it may build correctly and all integration tests work, I have no real environment to test with. best regards, Helmut -Original Message- From: Mike Percy [mailto:mpe...@apache.org] Sent: Mittwoch, 14. Februar 2018 00:38 To: dev@flume.apache.org Subject: Re: Merge of patch in Flume-3021? OK. In the pull request, it would be nice if whoever submits or merges it mentions all of the contributors to the patch in the commit message. I asked Wolfgang H. about the SolrServer thing and this is what he told me: Hi Mike, the class has been renamed to "SolrClient" (which unfortunately > breaks compat). It's just a class rename. The functionality is the > same as before. It was called SolrServer in Solr4 because it was a > client proxy that sends RPCs to a Solr server, but calling it > SolrClient is more straightforward to understand, hence the community > decided to rename the class. > It's possible to spawn an embedded Solr server, for example for > testing purposes, via class EmbeddedSolrServer (a class that retains > the same name in Solr7 and Solr4), which extends the SolrClient class. Hope this helps, Mike On Tue, Feb 13, 2018 at 4:41 AM, Wahrmann, Helmut <helmut.wahrm...@rsa.com> wrote: > Hi Mike, > > Thanks for the response. Would be cool if we get that sorted out. > > I've asked Yonghao Zou to submit the Pull Request, since he did most > of the work and should get the credit. > He'll do so after the Chinese New Year's Eve. > > I will then issue a Pull request for the new ES Rest client, which is > dependent on the above work. > > best regards, > Helmut > > -Original Message- > From: Mike Percy [mailto:mpe...@apache.org] > Sent: Dienstag, 13. Februar 2018 04:30 > To: dev@flume.apache.org > Subject: Re: Merge of patch in Flume-3021? > > Hi Helmut, > I see that I neglected to follow up on the other thread on this topic > after your reply about SolrServer missing from the solrj jar. Let me > ask around w/ some folks I know that work on Solr and see if there is > any way to retain the SolrServer for our tests after upgrading to the new > version. > > Thank you very much for working on upgrading Solr. Would you mind > submitting a pull request with your (apparently work-in-progress) > patch to upgrade both Solr and ES? > > To reply to your email in this thread, the JAR packaging situation is > largely the same after merging FLUME-2957 so unfortunately most of > what I noted in my reply in the other thread ( > https://s.apache.org/GqcX ) still holds. > > I hope that we can upgrade the Solr dependencies as part of the same > commit as the ES dependencies to avoid worrying about which lucene jar > is first in the classpath, and ensure we are not adding any additional > dependency conflicts to mvn dependency:tree. > > Regards, > Mike > > On Mon, Feb 12, 2018 at 12:54 AM, Wahrmann, Helmut < > helmut.wahrm...@rsa.com> > wrote: > > > Hi, > > > > now that the blocker for FLUME-3021 is removed by committing > > FLUME-2957, can we get the patch from 3021 merged to trunk? > > > > Thanks, > > > > Helmut > > >
RE: Store password for config safely?
That sounds good. Need to have a closer look tough, how it can be used. Best regards, Helmut -Original Message- From: Mike Percy [mailto:mpe...@apache.org] Sent: Mittwoch, 14. Februar 2018 00:32 To: dev@flume.apache.org Subject: Re: Store password for config safely? I think Ferenc has been looking at something related to this, or perhaps is trying to get an existing patch merged (FLUME-2442 <https://issues.apache.org/jira/browse/FLUME-2442>, PR 197 <https://github.com/apache/flume/pull/197>). I haven't been following that work closely so I don't know if it's exactly what you're looking for, but maybe he can chime in here. Mike On Mon, Feb 12, 2018 at 1:16 AM, Wahrmann, Helmut <helmut.wahrm...@rsa.com> wrote: > Hi, > > Do we have a way of storing a password safely, i.e. not in clear text? > When e.g. an Elasticsearch cluster is protected by X-Pack Security, I > need to specify a userid / password when connecting. > The userid / password could be specified in the config, but then the > password would be available in readable form. > > Do we have other sinks or sources, where we are dealing with passwords > and were a suitable method exists? > > best regards, > > Helmut >
RE: Merge of patch in Flume-3021?
Hi Mike, Thanks for the response. Would be cool if we get that sorted out. I've asked Yonghao Zou to submit the Pull Request, since he did most of the work and should get the credit. He'll do so after the Chinese New Year's Eve. I will then issue a Pull request for the new ES Rest client, which is dependent on the above work. best regards, Helmut -Original Message- From: Mike Percy [mailto:mpe...@apache.org] Sent: Dienstag, 13. Februar 2018 04:30 To: dev@flume.apache.org Subject: Re: Merge of patch in Flume-3021? Hi Helmut, I see that I neglected to follow up on the other thread on this topic after your reply about SolrServer missing from the solrj jar. Let me ask around w/ some folks I know that work on Solr and see if there is any way to retain the SolrServer for our tests after upgrading to the new version. Thank you very much for working on upgrading Solr. Would you mind submitting a pull request with your (apparently work-in-progress) patch to upgrade both Solr and ES? To reply to your email in this thread, the JAR packaging situation is largely the same after merging FLUME-2957 so unfortunately most of what I noted in my reply in the other thread ( https://s.apache.org/GqcX ) still holds. I hope that we can upgrade the Solr dependencies as part of the same commit as the ES dependencies to avoid worrying about which lucene jar is first in the classpath, and ensure we are not adding any additional dependency conflicts to mvn dependency:tree. Regards, Mike On Mon, Feb 12, 2018 at 12:54 AM, Wahrmann, Helmut <helmut.wahrm...@rsa.com> wrote: > Hi, > > now that the blocker for FLUME-3021 is removed by committing > FLUME-2957, can we get the patch from 3021 merged to trunk? > > Thanks, > > Helmut >
Store password for config safely?
Hi, Do we have a way of storing a password safely, i.e. not in clear text? When e.g. an Elasticsearch cluster is protected by X-Pack Security, I need to specify a userid / password when connecting. The userid / password could be specified in the config, but then the password would be available in readable form. Do we have other sinks or sources, where we are dealing with passwords and were a suitable method exists? best regards, Helmut
Merge of patch in Flume-3021?
Hi, now that the blocker for FLUME-3021 is removed by committing FLUME-2957, can we get the patch from 3021 merged to trunk? Thanks, Helmut
RE: Merging to trunk?
Hi Mike, I gave it a try and changed the Solr version to 7.2.1. It all compiled well. Only the tests were failing, because SolrServer, which is needed in the test cases to connect to the embedded Test Server is no longer available in org.apache.solr.client.solrj. But that's not needed in the "production" version. I cannot test however if it really works, since I don't have access to a Solr installation. I understand your discussion with the dependency problem, but I need to move ahead. It doesn't help me to have a Flume 1.9 or 2.0 released with no dependency problems, but it doesn't support the latest Elasticsearch client, which are needed by my customer. And by the time, you will release Flume 1.9, the ES project is at version 8, where they will drop support for the Transport client. So I am in the urgent need to develop support for the new ElasticSearch Rest Client. For that I need the changes which are done in Flume-3021. So I continue my work in a fork and will publish Pull requests . Once the dependencies problems are sorted out, you're welcome to include my changes in your trunk. best regards, Helmut -Original Message- From: Mike Percy [mailto:mpe...@apache.org] Sent: Dienstag, 30. Jänner 2018 04:57 To: dev@flume.apache.org Subject: Re: Merging to trunk? On Mon, Jan 29, 2018 at 10:48 AM, Wahrmann, Helmut <helmut.wahrm...@rsa.com > wrote: > > I think it is not a problem to distribute the modules together if we > have maintainers for them. > It doesn't make sense to release Flume 1.9 in 2018 with support for > module versions which were current in 2013. > Best way would be to identify people willing to upgrade the modules to > their latest version. > If we can't find someone, we should distribute it separately. I would be OK with upgrading Solr and ES simultaneously if we can do it in a compatible way. However, there is no guarantee that the latest versions of Solr and ES will be dependency-compatible. If they are, I would certainly be in favor of merging such a patch. On Mon, Jan 29, 2018 at 2:49 AM, Ferenc Szabo <szabofe...@apache.org> wrote: > I believe, that a clean and maintainable solution would be if the > engine/framework itself could be separated from everything else, the > dependency directions would be fixed, plugins would depend only on > api-s and after that, any source/sink/etc implementation would come > as an individual module/plugin and would be loaded with an isolated > classloader. > I agree that isolating Flume plugins from each other would solve this problem once and for all. I wonder if we can do both things: come up with a short-term "band-aid" patch to allow us to upgrade ES / Solr and also come up with a longer-term plan to solve the underlying problem. Mike
RE: Merging to trunk?
I think it is not a problem to distribute the modules together if we have maintainers for them. It doesn't make sense to release Flume 1.9 in 2018 with support for module versions which were current in 2013. Best way would be to identify people willing to upgrade the modules to their latest version. If we can't find someone, we should distribute it separately. best regards, Helmut -Original Message- From: Mike Percy [mailto:mpe...@apache.org] Sent: Montag, 29. Jänner 2018 09:12 To: dev@flume.apache.org Subject: Re: Merging to trunk? Interesting proposal Yonghao. I think it's worth pointing out that today, a Flume build will work "out of the box" with any combination of supported plugins since they are all already in the Flume classpath. The downside of that, of course, is that the dependency hell makes it very hard to upgrade modules. The benefit is that it's nice from a usability perspective. However, if we are willing to drop this "out of the box" usage feature, then we could allow for separate modules to have conflicting dependencies (i.e. Solr and ElasticSearch) as long as they are not loaded in the same Flume agent. What do you folks think about removing the Solr Sink and the ElasticSearch Sink from the default distribution and instead distribute them as separate (fat) jars? If we do it for those two then it might also make sense to do it for other modules as well. I am not sure this is very user friendly in the common case where people just want to use the HDFS and HBase sink out of the box. So maybe we leave those as part of the "default" module setup? Obviously, changing how modules get included into the classpath would be a breaking change. I think a change that drastic would call for a Flume 2.0.0 release in order to indicate that it's not backwards compatible. Thoughts? Mike On Sat, Jan 27, 2018 at 10:50 PM, Yonghao Zou <yonghaoz1...@gmail.com> wrote: > Maybe the Maven shade plugin can solve this, we can package each > module to a fat jar and they will not depend on other modules. > > 2018-01-27 6:14 GMT+08:00 Wahrmann, Helmut <helmut.wahrm...@rsa.com>: > > > fully agreed. > > But we shouldn't do that do support a 4 years old component. > > The goal should be to support the latest releases. > > It is ridiculous to release flume 1.9 with elasticsearch 0.90.1 > > support, when 6.x is the current version. > > > > > > > > Best regards, > > > > Helmut Wahrmann > > > > > > Ursprüngliche Nachricht > > Von: Ralph Goers <ralph.go...@dslextreme.com> > > Datum: 26.01.18 22:12 (GMT+01:00) > > An: dev@flume.apache.org > > Betreff: Re: Merging to trunk? > > > > The “right” way to deal with this from a Maven perspective is to > > declare the version you want in the dependencyManagement section of > > the parent > pom > > and to not specify any versions in child poms. Then use the Maven > enforcer > > plugin to make sure everything required has a declaration in the > > dependencyManagement section. > > > > Ralph > > > > > On Jan 26, 2018, at 8:28 AM, Wahrmann, Helmut > > > <helmut.wahrm...@rsa.com > > > > wrote: > > > > > > Hi Mike, > > > > > > This only shows that some other component needs to be updated as well. > > > Which software relies on Lucene 4.3.0, which was released in 2013? > > > It is most probably not used by anyone else anymore. > > > > > > This needs to be solved. It makes no sense to release Flume 1.9 > > > with > > support for ElasticSearch 0.90.1. > > > And it doesn't make sense either to release Flume 1.9 with support > > > for > a > > component that relies on Lucene 4.3.0. > > > > > > We will always have dependency problems, if one component is > > > updated to > > a newer version and others not. > > > > > > To get rid of the conflict is probably to set "optional" to true > > > in the > > pom.xml. > > > Then we don't distribute those Jars in the flume lib and the users > > > need > > to set their class path to point e.g. to Elasticsearch and then the > correct > > libs will be used. > > > I doubt that a Elasticsearch user will also use some other > > > component > > with e.g. this outdated Lucene. > > > > > > best regards, > > > > > > Helmut > > > > > > > > > -Original Message- > > > From: Mike Percy [mailto:mpe...@apache.org] > > > Sent: Donnerstag, 25. Jänner 2018 23:01 > > > To: dev@flume.apache.org > &g
Re: Merging to trunk?
fully agreed. But we shouldn't do that do support a 4 years old component. The goal should be to support the latest releases. It is ridiculous to release flume 1.9 with elasticsearch 0.90.1 support, when 6.x is the current version. Best regards, Helmut Wahrmann Ursprüngliche Nachricht Von: Ralph Goers <ralph.go...@dslextreme.com> Datum: 26.01.18 22:12 (GMT+01:00) An: dev@flume.apache.org Betreff: Re: Merging to trunk? The “right” way to deal with this from a Maven perspective is to declare the version you want in the dependencyManagement section of the parent pom and to not specify any versions in child poms. Then use the Maven enforcer plugin to make sure everything required has a declaration in the dependencyManagement section. Ralph > On Jan 26, 2018, at 8:28 AM, Wahrmann, Helmut <helmut.wahrm...@rsa.com> wrote: > > Hi Mike, > > This only shows that some other component needs to be updated as well. > Which software relies on Lucene 4.3.0, which was released in 2013? > It is most probably not used by anyone else anymore. > > This needs to be solved. It makes no sense to release Flume 1.9 with support > for ElasticSearch 0.90.1. > And it doesn't make sense either to release Flume 1.9 with support for a > component that relies on Lucene 4.3.0. > > We will always have dependency problems, if one component is updated to a > newer version and others not. > > To get rid of the conflict is probably to set "optional" to true in the > pom.xml. > Then we don't distribute those Jars in the flume lib and the users need to > set their class path to point e.g. to Elasticsearch and then the correct libs > will be used. > I doubt that a Elasticsearch user will also use some other component with > e.g. this outdated Lucene. > > best regards, > > Helmut > > > -Original Message- > From: Mike Percy [mailto:mpe...@apache.org] > Sent: Donnerstag, 25. Jänner 2018 23:01 > To: dev@flume.apache.org > Subject: Re: Merging to trunk? > > Hi Helmut, > I wrote a small Perl script to identify dependency conflicts based on output > from mvn dependency:tree and posted it here: https://gist.github.com/ > mpercy/39614d770864bdd0c386befd5e8a1840 > > I ran that on the current trunk and it actually found some errors which > should be fixed: > > Version conflict: package com.google.guava:guava:jar needed in 2 different > versions: (11.0.2, 18.0) > Version conflict: package commons-httpclient:commons-httpclient:jar needed in > 2 different versions: (3.0.1, 3.1) Version conflict: package > commons-logging:commons-logging:jar needed in 2 different versions: (1.1.3, > 1.2) Version conflict: package commons-pool:commons-pool:jar needed in 2 > different versions: (1.5.4, 1.6) Version conflict: package > net.sf.jopt-simple:jopt-simple:jar needed in 2 different versions: (3.2, 4.7) > Version conflict: package org.codehaus.jackson:jackson-jaxrs:jar needed in > 2 different versions: (1.8.3, 1.8.8) > ERROR: 6 package version conflicts identified > > However, after applying your patch on top of trunk and running it again, it > identified new conflicts: > > Version conflict: package com.google.guava:guava:jar needed in 2 different > versions: (11.0.2, 18.0) > Version conflict: package commons-httpclient:commons-httpclient:jar needed in > 2 different versions: (3.0.1, 3.1) Version conflict: package > commons-logging:commons-logging:jar needed in 2 different versions: (1.1.3, > 1.2) Version conflict: package commons-pool:commons-pool:jar needed in 2 > different versions: (1.5.4, 1.6) Version conflict: package > net.sf.jopt-simple:jopt-simple:jar needed in 3 different versions: (3.2, 4.7, > 5.0.2) Version conflict: package org.apache.lucene:lucene-analyzers-common:jar > needed in 2 different versions: (4.3.0, 7.1.0) Version conflict: package > org.apache.lucene:lucene-core:jar needed in 2 different versions: (4.3.0, > 7.1.0) Version conflict: package org.apache.lucene:lucene-grouping:jar needed > in 2 different versions: (4.3.0, 7.1.0) Version conflict: package > org.apache.lucene:lucene-highlighter:jar needed in 2 different versions: > (4.3.0, 7.1.0) Version conflict: package org.apache.lucene:lucene-memory:jar > needed in 2 different versions: (4.3.0, 7.1.0) Version conflict: package > org.apache.lucene:lucene-misc:jar needed in 2 different versions: (4.3.0, > 7.1.0) Version conflict: package org.apache.lucene:lucene-queries:jar needed > in 2 different versions: (4.3.0, 7.1.0) Version conflict: package > org.apache.lucene:lucene-queryparser:jar needed in 2 different versions: > (4.3.0, 7.1.0) Version conflict: package org.apache.lucene:lucene-spatial:jar > needed in 2 different versions: (4.3.0, 7.1.0) Version conflict: pac
RE: Merging to trunk?
Hi Mike, This only shows that some other component needs to be updated as well. Which software relies on Lucene 4.3.0, which was released in 2013? It is most probably not used by anyone else anymore. This needs to be solved. It makes no sense to release Flume 1.9 with support for ElasticSearch 0.90.1. And it doesn't make sense either to release Flume 1.9 with support for a component that relies on Lucene 4.3.0. We will always have dependency problems, if one component is updated to a newer version and others not. To get rid of the conflict is probably to set "optional" to true in the pom.xml. Then we don't distribute those Jars in the flume lib and the users need to set their class path to point e.g. to Elasticsearch and then the correct libs will be used. I doubt that a Elasticsearch user will also use some other component with e.g. this outdated Lucene. best regards, Helmut -Original Message- From: Mike Percy [mailto:mpe...@apache.org] Sent: Donnerstag, 25. Jänner 2018 23:01 To: dev@flume.apache.org Subject: Re: Merging to trunk? Hi Helmut, I wrote a small Perl script to identify dependency conflicts based on output from mvn dependency:tree and posted it here: https://gist.github.com/ mpercy/39614d770864bdd0c386befd5e8a1840 I ran that on the current trunk and it actually found some errors which should be fixed: Version conflict: package com.google.guava:guava:jar needed in 2 different versions: (11.0.2, 18.0) Version conflict: package commons-httpclient:commons-httpclient:jar needed in 2 different versions: (3.0.1, 3.1) Version conflict: package commons-logging:commons-logging:jar needed in 2 different versions: (1.1.3, 1.2) Version conflict: package commons-pool:commons-pool:jar needed in 2 different versions: (1.5.4, 1.6) Version conflict: package net.sf.jopt-simple:jopt-simple:jar needed in 2 different versions: (3.2, 4.7) Version conflict: package org.codehaus.jackson:jackson-jaxrs:jar needed in 2 different versions: (1.8.3, 1.8.8) ERROR: 6 package version conflicts identified However, after applying your patch on top of trunk and running it again, it identified new conflicts: Version conflict: package com.google.guava:guava:jar needed in 2 different versions: (11.0.2, 18.0) Version conflict: package commons-httpclient:commons-httpclient:jar needed in 2 different versions: (3.0.1, 3.1) Version conflict: package commons-logging:commons-logging:jar needed in 2 different versions: (1.1.3, 1.2) Version conflict: package commons-pool:commons-pool:jar needed in 2 different versions: (1.5.4, 1.6) Version conflict: package net.sf.jopt-simple:jopt-simple:jar needed in 3 different versions: (3.2, 4.7, 5.0.2) Version conflict: package org.apache.lucene:lucene-analyzers-common:jar needed in 2 different versions: (4.3.0, 7.1.0) Version conflict: package org.apache.lucene:lucene-core:jar needed in 2 different versions: (4.3.0, 7.1.0) Version conflict: package org.apache.lucene:lucene-grouping:jar needed in 2 different versions: (4.3.0, 7.1.0) Version conflict: package org.apache.lucene:lucene-highlighter:jar needed in 2 different versions: (4.3.0, 7.1.0) Version conflict: package org.apache.lucene:lucene-memory:jar needed in 2 different versions: (4.3.0, 7.1.0) Version conflict: package org.apache.lucene:lucene-misc:jar needed in 2 different versions: (4.3.0, 7.1.0) Version conflict: package org.apache.lucene:lucene-queries:jar needed in 2 different versions: (4.3.0, 7.1.0) Version conflict: package org.apache.lucene:lucene-queryparser:jar needed in 2 different versions: (4.3.0, 7.1.0) Version conflict: package org.apache.lucene:lucene-spatial:jar needed in 2 different versions: (4.3.0, 7.1.0) Version conflict: package org.apache.lucene:lucene-suggest:jar needed in 2 different versions: (4.3.0, 7.1.0) Version conflict: package org.codehaus.jackson:jackson-jaxrs:jar needed in 2 different versions: (1.8.3, 1.8.8) Version conflict: package org.yaml:snakeyaml:jar needed in 2 different versions: (1.10, 1.17) ERROR: 17 package version conflicts identified So while it appears Guava is no longer a concern, there are other conflicting dependencies involved in the conflict here. I'm interested to hear your thoughts on this. Regards, Mike On Wed, Jan 24, 2018 at 7:57 AM, Wahrmann, Helmut <helmut.wahrm...@rsa.com> wrote: > Hi Mike, > > Thanks for the quick response. > > I fully agree that we need to take care about interop between the > different components and dependencies. > But if you look at the patch in FLUME-3021, it only puts in > Elasticsearch dependencies. > > The issue with Guava, which you saw in FLUME-2921 is no longer there, > because since mid-2016 a lot has changed in the Flume trunk and all > those dependencies are fixed. > It won't even be possible gto apply this patch anymore. > > The patch in FLUME-3021 works with the latest Elasticsearch versions, > without introducing
RE: Merging to trunk?
Hi Mike, Thanks for the quick response. I fully agree that we need to take care about interop between the different components and dependencies. But if you look at the patch in FLUME-3021, it only puts in Elasticsearch dependencies. The issue with Guava, which you saw in FLUME-2921 is no longer there, because since mid-2016 a lot has changed in the Flume trunk and all those dependencies are fixed. It won't even be possible gto apply this patch anymore. The patch in FLUME-3021 works with the latest Elasticsearch versions, without introducing additional dependencies, which might cause problems to other projects. So I think it can be safely merged into trunk. best regards, Helmut -Original Message- From: Mike Percy [mailto:mpe...@apache.org] Sent: Dienstag, 23. Jänner 2018 20:39 To: dev@flume.apache.org Subject: Re: Merging to trunk? Hi Helmut, Thank you for bringing this up on dev@ and thank you for the patch. I see there are other people people interested in this component upgrade as well. As you are probably aware, a Flume committer will need to approve the change before I gets merged to trunk. My primary concern w/ merging this would be compatibility of the dependencies. Flume suffers from a kind of a "dependency hell" because of a lack of classloading support (either via OSGI or Java modules). See https://issues.apache.org/jira/browse/FLUME-2293 for a tracking ticket about that issue. What that means is that all of the components that Flume ships must have compatible dependencies with each other which makes changes like this more complex. Therefore I would like someone to verify that mvn dependency:tree does not show conflicts when run from the top level with the new patches. If memory serves, I believe Google Guava is likely to conflict. Also, I think that https://issues.apache.org/jira/browse/FLUME-3021 is a duplicate of https://issues.apache.org/jira/browse/FLUME-2921 which has additional information about the various issues that we need to solve to do this upgrade. I'd be happy to discuss this issue some more on this thread. Mike On Tue, Jan 23, 2018 at 12:57 AM, Wahrmann, Helmut <helmut.wahrm...@rsa.com> wrote: > Hi, > > Who decides if/when a patch or pull request gets merged to trunk? > Reason I am asking is for the Elasticsearch support. The current code > in trunk does not work with ES > 2.x. > Currently Elasticsearch is at 6.1. > > In FLUME-3021 we have several patches since March last year. > I have a patched Flume running at a customer since April last year > without any problems. > > So why not merging those changes into the trunk? > As I stated in my comment in FLUME-3021, it will not cause any > problems, cause the current trunk won't work with newer ES versions anyhow. > I doubt that someone is out there still running ES 0.95. > > best regards, > > Helmut >
Merging to trunk?
Hi, Who decides if/when a patch or pull request gets merged to trunk? Reason I am asking is for the Elasticsearch support. The current code in trunk does not work with ES > 2.x. Currently Elasticsearch is at 6.1. In FLUME-3021 we have several patches since March last year. I have a patched Flume running at a customer since April last year without any problems. So why not merging those changes into the trunk? As I stated in my comment in FLUME-3021, it will not cause any problems, cause the current trunk won't work with newer ES versions anyhow. I doubt that someone is out there still running ES 0.95. best regards, Helmut
RE: Working with Jira
Hi Mike, Thanks for your quick response. It worked perfect. best regards, Helmut -Original Message- From: Mike Percy [mailto:mpe...@apache.org] Sent: Donnerstag, 18. Jänner 2018 23:02 To: dev@flume.apache.org Subject: Re: Working with Jira Hi Helmut, I think I just added you as a contributor in Flume which would allow you to assign yourself a JIRA. If it didn't work, send me your username in JIRA. Sending a GitHub pull request is the preferred method to contribute a patch these days. Best, Mike On Thu, Jan 18, 2018 at 5:55 AM, Wahrmann, Helmut <helmut.wahrm...@rsa.com> wrote: > Hi, > > In your developing guidelines you state, that if I would like to work > on an issue I could assign it to myself: > > https://github.com/apache/flume/blob/trunk/CONTRIBUTING.md > > Well, I was able to create a Jira, but I am not able to assign it to > myself. Seems I am missing authorization. > > Also, what do you consider better way? > > - Submitting a patch, which is attached to the Jira or Submitting a > pull request and update the Jira that a Pull request has been > submitted. > > Thanks, > Helmut > >
Working with Jira
Hi, In your developing guidelines you state, that if I would like to work on an issue I could assign it to myself: https://github.com/apache/flume/blob/trunk/CONTRIBUTING.md Well, I was able to create a Jira, but I am not able to assign it to myself. Seems I am missing authorization. Also, what do you consider better way? - Submitting a patch, which is attached to the Jira or Submitting a pull request and update the Jira that a Pull request has been submitted. Thanks, Helmut
