Re: [VOTE] Release Apache Flume version 1.8.0 RC2

[Attila Simon]

+1 on this Release Candidate

Please see inline below the checks performed.

Cheers,
Attila


*Attila Simon*
Software Engineer
Email:   s...@cloudera.com

[image: Cloudera Inc.]

On Mon, Sep 18, 2017 at 2:46 PM, Denes Arvay <de...@cloudera.com> wrote:

> Hi All,
>
> I'd like to encourage everybody to check the RC2 and provide feedback about
> it. It's quite simple, just do the following steps:
> - Download the artifacts from the people.apache.org link I've sent in the
> initial email (see below)
>

downloaded


> - Check the md5/sha1 checksums
>

checked and verified


> - Verify the signature: import the KEYS file (link below, howto in the KEYS
> file), then verify with gpg --verify apache-flume-1.8.0-src.tar.gz.asc
> and gpg --verify apache-flume-1.8.0-bin.tar.gz.asc
>

gpg: assuming signed data in 'apache-flume-1.8.0-src.tar.gz'
gpg: Signature made Fri Sep 15 15:04:39 2017 CEST using RSA key ID 4199ACFF
gpg: Good signature from "Denes Arvay <de...@apache.org>" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg:  There is no indication that the signature belongs to the
owner.
Primary key fingerprint: 4CF1 5CF1 525F CE29 F66C  08FA 302C B2A8 4199 ACFF

Fingerprint matches with your key from the KEY file obtained via a trusted
https connection.


- Check that the downloaded artifacts match the ones in the maven staging
> repo (org.apache.flume:flume-ng-dist:1.8.0, link to the repo below)
>

checked (besides file name mismatch the content is identical)


> - Extract the source tarball, compile and run the unit tests (note: some
> flaky tests might break, you might want to use
> the surefire.rerunFailingTestsCount flag)
>

mvn clean install -Dsurefire.rerunFailingTestsCount=10 -> passes

[INFO] 

[INFO] Reactor Summary:
[INFO]
[INFO] Flume checkstyle project ... SUCCESS [
 1.301 s]
[INFO] Apache Flume ... SUCCESS [
 1.443 s]
[INFO] Flume NG SDK ... SUCCESS [01:15
min]
[INFO] Flume NG Configuration . SUCCESS [
 2.722 s]
[INFO] Flume Auth . SUCCESS [
 7.071 s]
[INFO] Flume NG Core .. SUCCESS [08:17
min]
[INFO] Flume NG Sinks . SUCCESS [
 0.231 s]
[INFO] Flume NG HDFS Sink . SUCCESS [02:40
min]
[INFO] Flume NG IRC Sink .. SUCCESS [
 2.977 s]
[INFO] Flume NG Channels .. SUCCESS [
 0.432 s]
[INFO] Flume NG JDBC channel .. SUCCESS [
35.675 s]
[INFO] Flume NG file-based channel  SUCCESS [05:26
min]
[INFO] Flume NG Spillable Memory channel .. SUCCESS [
56.480 s]
[INFO] Flume NG Node .. SUCCESS [
41.525 s]
[INFO] Flume NG Embedded Agent  SUCCESS [
23.380 s]
[INFO] Flume NG HBase Sink  SUCCESS [04:54
min]
[INFO] Flume NG ElasticSearch Sink  SUCCESS [01:03
min]
[INFO] Flume NG Morphline Solr Sink ... SUCCESS [
16.488 s]
[INFO] Flume Shared Utils . SUCCESS [
 0.186 s]
[INFO] Flume Shared Kafka Test Utils .. SUCCESS [
 0.749 s]
[INFO] Flume Kafka Sink ... SUCCESS [
41.873 s]
[INFO] Flume HTTP/S Sink .. SUCCESS [
12.705 s]
[INFO] Flume NG Kite Dataset Sink . SUCCESS [
11.923 s]
[INFO] Flume NG Hive Sink . SUCCESS [
40.225 s]
[INFO] Flume Sources .. SUCCESS [
 0.204 s]
[INFO] Flume Scribe Source  SUCCESS [
 5.345 s]
[INFO] Flume JMS Source ... SUCCESS [
23.410 s]
[INFO] Flume Twitter Source ... SUCCESS [
 1.390 s]
[INFO] Flume Kafka Source . SUCCESS [02:09
min]
[INFO] Flume Taildir Source ... SUCCESS [
17.342 s]
[INFO] flume-kafka-channel  SUCCESS [03:36
min]
[INFO] Flume legacy Sources ... SUCCESS [
 0.194 s]
[INFO] Flume legacy Avro source ... SUCCESS [
 2.025 s]
[INFO] Flume legacy Thrift Source . SUCCESS [
 1.828 s]
[INFO] Flume NG Clients ... SUCCESS [
 0.178 s]
[INFO] Flume NG Log4j Appender  SUCCESS [
36.688 s]
[INFO] Flume NG Tools . SUCCESS [
 2.793 s]
[INFO] Flume NG distribution .. SUCCESS [
16.495 s]
[IN

Re: [DISCUSS] Flume 1.8 release proposal

[Attila Simon]

Hi Denes,

I think this is a great idea! I can offer help in reviewing some of the
inflight patches/pull requests.

Cheers,
Attila


*Attila Simon*
Software Engineer
Email:   s...@cloudera.com

[image: Cloudera Inc.]

On Tue, Sep 5, 2017 at 11:34 PM, Mike Percy <mpe...@apache.org> wrote:

> Hi Denes,
> +1 from me for releasing a Flume 1.8.0 and for you taking on the RM role
> for Flume 1.8.0. The timeline for a first RC seems fine.
>
> It sounds like you will have some help, which is good. Let me know if you
> need anything from me.
>
> Regards,
> Mike
>
> On Mon, Sep 4, 2017 at 1:21 AM, Denes Arvay <de...@cloudera.com> wrote:
>
> > Hi Flume Community,
> >
> > Almost a year passed since we've released Flume 1.7.
> > More than 50 commits were pushed since then, including documentation
> fixes,
> > many critical bug fixes and several important features, so I'd like to
> > propose to publish the next minor release of Flume.
> >
> > I'd be happy to be the Release Manager with the help of Ferenc Szabo and
> > Marcell Hegedus who have been quite active recently, and Balazs Donat
> > Bessenyei who took the lion's share of the work during the previous
> release
> > - if both community and they are OK with it.
> >
> > Among others the following major changes will be included in the next
> > release:
> >
> > Fixed bugs:
> > - FLUME-2857. Make Kafka Source/Channel/Sink restore default values when
> > live updating config
> > - FLUME-2812. Fix semaphore leak causing java.lang.Error: Maximum permit
> > count exceeded in MemoryChannel
> > - FLUME-3020. Improve HDFS Sink escape sequence substitution
> > - FLUME-3027. Change Kafka Channel to clear offsets map after commit
> > - FLUME-3049. Make HDFS sink rotate more reliably in secure mode
> > - FLUME-3080. Close failure in HDFS Sink might cause data loss
> > - FLUME-3085. HDFS Sink can skip flushing some BucketWriters, might lead
> to
> > data loss
> > - FLUME-2752. Fix AvroSource startup resource leaks
> > - FLUME-2905. Fix NetcatSource file descriptor leak if startup fails
> >
> > New features:
> > - FLUME-2171. Add Interceptor to remove headers from event
> > - FLUME-2993. Add support for environment variables in configuration
> files
> > - New component: HTTP Sink
> > - FLUME-3100. Support arbitrary header substitution for topic of Kafka
> Sink
> > - FLUME-2917. Provide netcat UDP source as alternative to TCP
> >
> > There are 35 open tickets targeted for 1.8 in patch available state:
> > https://s.apache.org/flume-1.8-target-tickets
> >
> > Plus we also have quite a lot (~65) open pull requests on GitHub:
> > https://github.com/apache/flume/pulls
> >
> > Some of the above mentioned tickets/pull requests already have some
> review
> > comments, so at least part of this list can get into this release beside
> > the already pushed ones.
> >
> > I'd like to propose to target the week of 11th of September with the
> first
> > release candidate. That'd mean that the branch date would be the 11th,
> any
> > significant code change should get in by that date.
> >
> > If nobody has any concerns then I'm going to create an umbrella ticket to
> > track the release process.
> >
> > Kind regards,
> > Denes
> >
>

[jira] [Commented] (FLUME-3057) Build fails due to unsupported snappy-java version on ppc64le

[Attila Simon (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLUME-3057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16140398#comment-16140398
 ] 

Attila Simon commented on FLUME-3057:
-

Hi [~pravindsilva], 
Since this has been committed, could you please close the reviewboard review: 
https://reviews.apache.org/r/57676/

> Build fails due to unsupported snappy-java version on ppc64le
> -
>
> Key: FLUME-3057
> URL: https://issues.apache.org/jira/browse/FLUME-3057
> Project: Flume
>  Issue Type: Bug
>  Components: Build
>Affects Versions: 1.8.0
> Environment: $ uname -a
> Linux 2f63413ff231 4.4.0-31-generic #50-Ubuntu SMP Wed Jul 13 00:05:18 UTC 
> 2016 ppc64le ppc64le ppc64le GNU/Linux
>Reporter: Pravin Dsilva
>  Labels: powerpc, ppc64le
> Fix For: 1.8.0
>
> Attachments: FLUME-3057-1.patch, FLUME-3057.patch
>
>
> Flume has a snappy-java dependency with version 1.1.0. Upon building Flume on 
> ppc64le architecture, errors such as "[FAILED_TO_LOAD_NATIVE_LIBRARY] no 
> native library is found for os.name=Linux and os.arch=ppc64le" are seen
>  Native libraries for ppc64le were added in snappy-java version 1.1.1. Hence 
> Flume needs to have this higher version of snappy-java.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Comment Edited] (FLUME-3057) Build fails due to unsupported snappy-java version on ppc64le

[Attila Simon (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLUME-3057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16137204#comment-16137204
 ] 

Attila Simon edited comment on FLUME-3057 at 8/22/17 7:01 PM:
--

I have the same issue and updating snappy solved it. Unfortunately the patch 
doesn't apply any more. Since this ticket is unassigned at the moment: to move 
things forward I created a new patch (FLUME-3057-1.patch) which is ready to be 
committed on latest trunk (as of now). I cannot add it to your reviewboard 
review but please feel free to do that. Also you can look at the updated patch 
as a commit here (I'm happy to create a pull request if that helps resolving 
this issue): 
https://github.com/simonati/flume/commit/5dd54881be4a752ea458d54089dc9cc7816568fa
 


was (Author: sati):
I have the same issue and updating snappy solved it. Unfortunately the patch 
doesn't apply any more. Since this ticket is unassigned at the moment: to move 
things forward I created a new patch which is ready to be committed on latest 
trunk (as of now). I cannot add it to your reviewboard review but please feel 
free to do that. Also you can look at the updated patch as a commit here (I'm 
happy to create a pull request if that helps resolving this issue): 
https://github.com/simonati/flume/commit/5dd54881be4a752ea458d54089dc9cc7816568fa
 

> Build fails due to unsupported snappy-java version on ppc64le
> -
>
> Key: FLUME-3057
> URL: https://issues.apache.org/jira/browse/FLUME-3057
> Project: Flume
>  Issue Type: Bug
>  Components: Build
>Affects Versions: 1.8.0
> Environment: $ uname -a
> Linux 2f63413ff231 4.4.0-31-generic #50-Ubuntu SMP Wed Jul 13 00:05:18 UTC 
> 2016 ppc64le ppc64le ppc64le GNU/Linux
>Reporter: Pravin Dsilva
>  Labels: powerpc, ppc64le
> Fix For: 1.8.0
>
> Attachments: FLUME-3057-1.patch, FLUME-3057.patch
>
>
> Flume has a snappy-java dependency with version 1.1.0. Upon building Flume on 
> ppc64le architecture, errors such as "[FAILED_TO_LOAD_NATIVE_LIBRARY] no 
> native library is found for os.name=Linux and os.arch=ppc64le" are seen
>  Native libraries for ppc64le were added in snappy-java version 1.1.1. Hence 
> Flume needs to have this higher version of snappy-java.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLUME-3057) Build fails due to unsupported snappy-java version on ppc64le

[Attila Simon (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLUME-3057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16137204#comment-16137204
 ] 

Attila Simon commented on FLUME-3057:
-

I have the same issue and updating snappy solved it. Unfortunately the patch 
doesn't apply any more. Since this ticket is unassigned at the moment: to move 
things forward I created a new patch which is ready to be committed on latest 
trunk (as of now). I cannot add it to your reviewboard review but please feel 
free to do that. Also you can look at the updated patch as a commit here (I'm 
happy to create a pull request if that helps resolving this issue): 
https://github.com/simonati/flume/commit/5dd54881be4a752ea458d54089dc9cc7816568fa
 

> Build fails due to unsupported snappy-java version on ppc64le
> -
>
> Key: FLUME-3057
> URL: https://issues.apache.org/jira/browse/FLUME-3057
> Project: Flume
>  Issue Type: Bug
>  Components: Build
>Affects Versions: 1.8.0
> Environment: $ uname -a
> Linux 2f63413ff231 4.4.0-31-generic #50-Ubuntu SMP Wed Jul 13 00:05:18 UTC 
> 2016 ppc64le ppc64le ppc64le GNU/Linux
>Reporter: Pravin Dsilva
>  Labels: powerpc, ppc64le
> Fix For: 1.8.0
>
> Attachments: FLUME-3057-1.patch, FLUME-3057.patch
>
>
> Flume has a snappy-java dependency with version 1.1.0. Upon building Flume on 
> ppc64le architecture, errors such as "[FAILED_TO_LOAD_NATIVE_LIBRARY] no 
> native library is found for os.name=Linux and os.arch=ppc64le" are seen
>  Native libraries for ppc64le were added in snappy-java version 1.1.1. Hence 
> Flume needs to have this higher version of snappy-java.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (FLUME-3057) Build fails due to unsupported snappy-java version on ppc64le

[Attila Simon (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLUME-3057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Attila Simon updated FLUME-3057:

Attachment: FLUME-3057-1.patch

> Build fails due to unsupported snappy-java version on ppc64le
> -
>
> Key: FLUME-3057
> URL: https://issues.apache.org/jira/browse/FLUME-3057
> Project: Flume
>  Issue Type: Bug
>  Components: Build
>Affects Versions: 1.8.0
> Environment: $ uname -a
> Linux 2f63413ff231 4.4.0-31-generic #50-Ubuntu SMP Wed Jul 13 00:05:18 UTC 
> 2016 ppc64le ppc64le ppc64le GNU/Linux
>Reporter: Pravin Dsilva
>  Labels: powerpc, ppc64le
> Fix For: 1.8.0
>
> Attachments: FLUME-3057-1.patch, FLUME-3057.patch
>
>
> Flume has a snappy-java dependency with version 1.1.0. Upon building Flume on 
> ppc64le architecture, errors such as "[FAILED_TO_LOAD_NATIVE_LIBRARY] no 
> native library is found for os.name=Linux and os.arch=ppc64le" are seen
>  Native libraries for ppc64le were added in snappy-java version 1.1.1. Hence 
> Flume needs to have this higher version of snappy-java.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLUME-3143) Fix undeclared and unused dependencies

[Attila Simon (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLUME-3143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16134964#comment-16134964
 ] 

Attila Simon commented on FLUME-3143:
-

Removing the "unused" slf4j-log4j12 dependency from the flume-ng-sdk module 
would result in empty test output files. A logger backend implementation 
slf4j-log4j12 is a runtime resolved dependency. AFAK dependency analysis only 
checks statically the compiled bytecode thus it cannot be perfect around 
dependencies used via runtime/reflection (and causing false positives which 
would prevent enforcement). 
However I recommended to ignoreNonCompile (since the check cannot be perfect 
for those dependencies anyway) but in the end it is implementer's call to use 
that recommendation or put the known to be runtime to the "exclude from 
analysis" list. 
If we don't want to enforce the good state at the end then I would see less 
value doing this cleanup.

> Fix undeclared and unused dependencies
> --
>
> Key: FLUME-3143
> URL: https://issues.apache.org/jira/browse/FLUME-3143
> Project: Flume
>  Issue Type: Bug
>Affects Versions: 1.7.0
>Reporter: Ferenc Szabo
>Assignee: Ferenc Szabo
>  Labels: dependency
> Fix For: 1.8.0
>
>
> Dependency analysis shows some warnings in the project that should be fixed
> {noformat}
> [INFO] 
> 
> [INFO] Building Flume NG SDK 1.8.0-SNAPSHOT
> [INFO] 
> 
> [WARNING] Used undeclared dependencies found:
> [WARNING]commons-lang:commons-lang:jar:2.5:compile
> [WARNING] Unused declared dependencies found:
> [WARNING]org.slf4j:slf4j-log4j12:jar:1.6.1:compile
> [INFO]
>  
> [INFO] 
> 
> [INFO] Building Flume NG Configuration 1.8.0-SNAPSHOT
> [INFO] 
> 
> [WARNING] Unused declared dependencies found:
> [WARNING]org.slf4j:slf4j-log4j12:jar:1.6.1:compile
> [INFO]
>  
> [INFO] 
> 
> [INFO] Building Flume Auth 1.8.0-SNAPSHOT
> [INFO] 
> 
> [WARNING] Used undeclared dependencies found:
> [WARNING]org.apache.thrift:libthrift:jar:0.9.0:compile
> [WARNING] Unused declared dependencies found:
> [WARNING]org.slf4j:slf4j-log4j12:jar:1.6.1:compile
> [INFO]
>  
> [INFO] 
> 
> [INFO] Building Flume NG Core 1.8.0-SNAPSHOT
> [INFO] 
> 
> [WARNING] Used undeclared dependencies found:
> [WARNING]com.google.code.findbugs:jsr305:jar:1.3.9:compile
> [WARNING]org.apache.httpcomponents:httpcore:jar:4.2.1:compile
> [WARNING] Unused declared dependencies found:
> [WARNING]org.mortbay.jetty:jetty-util:jar:6.1.26:compile
> [WARNING]log4j:log4j:jar:1.2.17:compile
> [WARNING]org.slf4j:slf4j-log4j12:jar:1.6.1:compile
> [INFO]
>  
> [INFO] 
> 
> [INFO] Building Flume NG HDFS Sink 1.8.0-SNAPSHOT
> [INFO] 
> 
> [WARNING] Used undeclared dependencies found:
> [WARNING]org.apache.hadoop:hadoop-hdfs:test-jar:tests:2.4.0:test
> [WARNING]org.apache.flume:flume-ng-auth:jar:1.8.0-SNAPSHOT:compile
> [WARNING]org.apache.avro:avro:jar:1.7.4:compile
> [WARNING] Unused declared dependencies found:
> [WARNING]org.apache.hadoop:hadoop-minicluster:jar:2.4.0:test
> [WARNING]org.apache.hadoop:hadoop-auth:jar:2.4.0:compile
> [WARNING]org.slf4j:slf4j-log4j12:jar:1.6.1:test
> [INFO]
>  
> [INFO] 
> 
> [INFO] Building Flume NG IRC Sink 1.8.0-SNAPSHOT
> [INFO] 
> 
> [WARNING] Used undeclared dependencies found:
> [WARNING]com.google.guava:gua

[jira] [Commented] (FLUME-3143) Fix undeclared and unused dependencies

[Attila Simon (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLUME-3143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16133987#comment-16133987
 ] 

Attila Simon commented on FLUME-3143:
-

This output seems partly incorrect since it lists slf4j-log4j12 as unused but 
slf4j-log4j12 is used in generating the test log thus it's declaration in the 
pom is wrong it should have been a runtime only scope eg test. With that said 
{{mvn dependency:analyze -DignoreNonCompile}} might be a better way to run this 
report. I don't know whether that would miss some other misconfiguration.

Once the cleanup is complete we may want to enforce the correct state as part 
of our build:
https://maven.apache.org/plugins/maven-dependency-plugin/analyze-only-mojo.html

> Fix undeclared and unused dependencies
> --
>
> Key: FLUME-3143
> URL: https://issues.apache.org/jira/browse/FLUME-3143
> Project: Flume
>  Issue Type: Bug
>Affects Versions: 1.7.0
>Reporter: Ferenc Szabo
>Assignee: Ferenc Szabo
>  Labels: dependency
> Fix For: 1.8.0
>
>
> Dependency analysis shows some warnings in the project that should be fixed
> {noformat}
> [INFO] 
> 
> [INFO] Building Flume NG SDK 1.8.0-SNAPSHOT
> [INFO] 
> 
> [WARNING] Used undeclared dependencies found:
> [WARNING]commons-lang:commons-lang:jar:2.5:compile
> [WARNING] Unused declared dependencies found:
> [WARNING]org.slf4j:slf4j-log4j12:jar:1.6.1:compile
> [INFO]
>  
> [INFO] 
> 
> [INFO] Building Flume NG Configuration 1.8.0-SNAPSHOT
> [INFO] 
> 
> [WARNING] Unused declared dependencies found:
> [WARNING]org.slf4j:slf4j-log4j12:jar:1.6.1:compile
> [INFO]
>  
> [INFO] 
> 
> [INFO] Building Flume Auth 1.8.0-SNAPSHOT
> [INFO] 
> 
> [WARNING] Used undeclared dependencies found:
> [WARNING]org.apache.thrift:libthrift:jar:0.9.0:compile
> [WARNING] Unused declared dependencies found:
> [WARNING]org.slf4j:slf4j-log4j12:jar:1.6.1:compile
> [INFO]
>  
> [INFO] 
> 
> [INFO] Building Flume NG Core 1.8.0-SNAPSHOT
> [INFO] 
> 
> [WARNING] Used undeclared dependencies found:
> [WARNING]com.google.code.findbugs:jsr305:jar:1.3.9:compile
> [WARNING]org.apache.httpcomponents:httpcore:jar:4.2.1:compile
> [WARNING] Unused declared dependencies found:
> [WARNING]org.mortbay.jetty:jetty-util:jar:6.1.26:compile
> [WARNING]log4j:log4j:jar:1.2.17:compile
> [WARNING]org.slf4j:slf4j-log4j12:jar:1.6.1:compile
> [INFO]
>  
> [INFO] 
> 
> [INFO] Building Flume NG HDFS Sink 1.8.0-SNAPSHOT
> [INFO] 
> 
> [WARNING] Used undeclared dependencies found:
> [WARNING]org.apache.hadoop:hadoop-hdfs:test-jar:tests:2.4.0:test
> [WARNING]org.apache.flume:flume-ng-auth:jar:1.8.0-SNAPSHOT:compile
> [WARNING]org.apache.avro:avro:jar:1.7.4:compile
> [WARNING] Unused declared dependencies found:
> [WARNING]org.apache.hadoop:hadoop-minicluster:jar:2.4.0:test
> [WARNING]org.apache.hadoop:hadoop-auth:jar:2.4.0:compile
> [WARNING]org.slf4j:slf4j-log4j12:jar:1.6.1:test
> [INFO]
>  
> [INFO] 
> 
> [INFO] Building Flume NG IRC Sink 1.8.0-SNAPSHOT
> [INFO] 
> 
> [WARNING] Used undeclared dependencies found:
> [WARNING]com.google.guava:guava:jar:11.0.2:compile
> [WARNING]commons-io:commons-io:jar:2.1:compile
> [WARNING] Unused declared dependencies found:
> [WARNING]org.slf4j:slf

[jira] [Commented] (FLUME-3112) Upgrade jackson-core library dependency

[Attila Simon (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLUME-3112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16128706#comment-16128706
 ] 

Attila Simon commented on FLUME-3112:
-

I concluded the review will be done on github. If that is indeed the case could 
you please close the review on reviewboard?

> Upgrade jackson-core library dependency
> ---
>
> Key: FLUME-3112
> URL: https://issues.apache.org/jira/browse/FLUME-3112
> Project: Flume
>  Issue Type: Bug
>Affects Versions: 1.7.0
>    Reporter: Attila Simon
>Assignee: Ferenc Szabo
>Priority: Critical
>  Labels: dependency
> Fix For: 1.8.0
>
> Attachments: FLUME-3112.patch
>
>
> ||Group||Artifact||Version used||Upgrade target||
> |com.fasterxml.jackson.core|jackson-core|2.3.1|2.8.9|
> Security vulnerability: http://www.cvedetails.com/cve/CVE-2016-7051/
> Please do:
> - double check the newest version. 
> - consider to remove a dependency if better alternative is available.
> - check whether the lib change would introduce a backward incompatibility (in 
> which case please add this label `breaking_change` and fix version should be 
> the next major)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 61431: FLUME-3112 Upgrade jackson-core library dependency

[Attila Simon]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61431/#review182279
---



Could you please do a rebase? Unfortunately the patch doesn't apply after 
63aabc4dae977e1e46a76b0b259afb778f3d2754

- Attila Simon


On Aug. 4, 2017, 5:34 p.m., Ferenc Szabo wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61431/
> ---
> 
> (Updated Aug. 4, 2017, 5:34 p.m.)
> 
> 
> Review request for Flume.
> 
> 
> Repository: flume-git
> 
> 
> Description
> ---
> 
> Updated the version numbers in the parent poms dependency management part.
> 
> 
> Diffs
> -
> 
>   pom.xml 5730db0c 
> 
> 
> Diff: https://reviews.apache.org/r/61431/diff/1/
> 
> 
> Testing
> ---
> 
> tested with mvn clean install
> Build and unit tests were successful.
> 
> Dependency tree only changed in version numbers and not in its form.
> 
> diff of mvn dependency:list | grep jar | sort | uniq command before and after 
> the patch:
> 
> 23,26c23,26
> < [INFO]com.fasterxml.jackson.core:jackson-annotations:jar:2.3.0:compile
> < [INFO]com.fasterxml.jackson.core:jackson-annotations:jar:2.4.2:test
> < [INFO]com.fasterxml.jackson.core:jackson-core:jar:2.3.1:compile
> < [INFO]com.fasterxml.jackson.core:jackson-core:jar:2.4.2:test
> ---
> > [INFO]com.fasterxml.jackson.core:jackson-annotations:jar:2.8.9:compile
> > [INFO]com.fasterxml.jackson.core:jackson-annotations:jar:2.8.9:test
> > [INFO]com.fasterxml.jackson.core:jackson-core:jar:2.8.9:compile
> > [INFO]com.fasterxml.jackson.core:jackson-core:jar:2.8.9:test
> 
> 
> Thanks,
> 
> Ferenc Szabo
> 
>

[jira] [Commented] (FLUME-3132) Upgrade tomcat jasper library dependencies

[Attila Simon (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLUME-3132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16104663#comment-16104663
 ] 

Attila Simon commented on FLUME-3132:
-

Hi [~fszabo],
Thank you for looking into this. Indeed it is included as a transitive 
dependency via hadoop-common only. As long as we understand the exact impact 
I'm fine with either case. Within our company we ship flume with a slightly 
enhanced hadoop-common:2.6.0 so I cannot help you saying that 2.7.0 will be 
fine for sure so that has to be heavily verified. By writing this it looks like 
that upgrading hadoop-common would be a longer and separate task thus should 
deserve a separate jira (and close this one once that has been resolved). I 
guess impact of exclusion has a pretty limited scope so sounds easier to me, 
but essentially it is your call which way to go.

> Upgrade tomcat jasper library dependencies
> --
>
> Key: FLUME-3132
> URL: https://issues.apache.org/jira/browse/FLUME-3132
> Project: Flume
>  Issue Type: Bug
>Affects Versions: 1.7.0
>    Reporter: Attila Simon
>Assignee: Ferenc Szabo
>Priority: Critical
>  Labels: dependency
> Fix For: 1.8.0
>
>
> ||Group||Artifact||Version used||Upgrade target||
> |tomcat|jasper-compiler|5.5.23|8.5.x|
> |tomcat|jasper-runtime|5.5.23|8.5.x|
> Security vulnerability: 
> - https://www.cvedetails.com/cve/CVE-2011-1318/
> - 
> http://www.cvedetails.com/vulnerability-list/vendor_id-45/product_id-887/Apache-Tomcat.html
> Maven repositories: 
> - https://mvnrepository.com/artifact/org.apache.tomcat/tomcat-jasper
> Note: These artifacts were moved to:
> * New Group   org.apache.tomcat
> * New Artifact
> Please do:
> - CVE might be a false alarm or mistake. Please double check.
> - double check the newest version. 
> - consider to remove a dependency if better alternative is available.
> - check whether the lib change would introduce a backward incompatibility (in 
> which case please add this label `breaking_change` and fix version should be 
> the next major)
> Excerpt from mvn dependency:tree
> {noformat}
> org.apache.flume:flume-ng-auth:jar:1.8.0-SNAPSHOT
> +- org.apache.hadoop:hadoop-common:jar:2.4.0:compile
> |  +- tomcat:jasper-compiler:jar:5.5.23:runtime
> |  +- tomcat:jasper-runtime:jar:5.5.23:runtime
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 61014: FLUME-3131 Upgrade spring framework library dependencies

[Attila Simon]

ou considered removing the 
unused javax.jms:jms:1.1 dependency definition? It is not a biggie I think it 
might be even better to submit this in a (follow-up) separate commit.


- Attila Simon


On July 23, 2017, 11:13 a.m., Ferenc Szabo wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61014/
> ---
> 
> (Updated July 23, 2017, 11:13 a.m.)
> 
> 
> Review request for Flume.
> 
> 
> Repository: flume-git
> 
> 
> Description
> ---
> 
> Moved activemq to test scope so the vulnerable spring dependencies move out 
> from production and added javax.jms to handle the jms dependencies
> 
> 
> Diffs
> -
> 
>   flume-ng-sources/flume-jms-source/pom.xml cbae702 
>   pom.xml 5730db0 
> 
> 
> Diff: https://reviews.apache.org/r/61014/diff/1/
> 
> 
> Testing
> ---
> 
> mvn clean install
> Build and unit tests were successful.
> 
> 
> Thanks,
> 
> Ferenc Szabo
> 
>

[jira] [Updated] (FLUME-3093) Groundwork for version changes in root pom

[Attila Simon (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLUME-3093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Attila Simon updated FLUME-3093:

Labels: dependency  (was: )

> Groundwork for version changes in root pom
> --
>
> Key: FLUME-3093
> URL: https://issues.apache.org/jira/browse/FLUME-3093
> Project: Flume
>  Issue Type: Task
>Affects Versions: 1.7.0
>Reporter: Miklos Csanady
>Assignee: Miklos Csanady
>  Labels: dependency
>
> Flume's root pom should have a parameter block where all the dependency 
> versions are listed. 
> This is fundamental to later version changes required to time effectively 
> overcame 3rd party security vulnerabilities.
> This should be done in two steps: first just refactoring with no change, the 
> second is getting rid of unnecessary different versions.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLUME-3131) Upgrade spring framework library dependencies

[Attila Simon (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLUME-3131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16095935#comment-16095935
 ] 

Attila Simon commented on FLUME-3131:
-

After looking at your patch now it is clear that you wanted to achieve what I 
wrote above. Have you considered pulling in the 
https://search.maven.org/#artifactdetails%7Cjavax.jms%7Cjms-api%7C1.1-rev-1%7Cjar
 instead of the geronimo shaded version?

> Upgrade spring framework library dependencies
> -
>
> Key: FLUME-3131
> URL: https://issues.apache.org/jira/browse/FLUME-3131
> Project: Flume
>  Issue Type: Bug
>Affects Versions: 1.7.0
>    Reporter: Attila Simon
>Assignee: Ferenc Szabo
>Priority: Critical
>  Labels: dependency
> Fix For: 1.8.0
>
> Attachments: FLUME-3131.patch
>
>
> ||Group||Artifact||Version used||Upgrade target||
> |org.springframework|spring-aop|3.0.7.RELEASE|4.3.9.RELEASE,|
> |org.springframework|spring-context|3.0.7.RELEASE|4.3.9.RELEASE,|
> |org.springframework|spring-core|3.0.7.RELEASE|4.3.9.RELEASE,|
> Security vulnerability: 
> https://www.cvedetails.com/vulnerability-list/vendor_id-9664/product_id-17274/Springsource-Spring-Framework.html
> Maven repositories: 
> - https://mvnrepository.com/artifact/org.springframework/spring-aop
> - https://mvnrepository.com/artifact/org.springframework/spring-context
> - https://mvnrepository.com/artifact/org.springframework/spring-core
> Please do:
> - CVE might be a false alarm or mistake. Please double check.
> - double check the newest version. 
> - consider to remove a dependency if better alternative is available.
> - check whether the lib change would introduce a backward incompatibility (in 
> which case please add this label `breaking_change` and fix version should be 
> the next major)
> Excerpt from mvn dependency:tree
> {noformat}
> org.apache.flume.flume-ng-sources:flume-jms-source:jar:1.8.0-SNAPSHOT
> \- org.apache.activemq:activemq-core:jar:5.7.0:provided
>+- org.springframework:spring-context:jar:3.0.7.RELEASE:provided
>|  +- org.springframework:spring-aop:jar:3.0.7.RELEASE:provided
>|  +- org.springframework:spring-beans:jar:3.0.7.RELEASE:provided
>|  +- org.springframework:spring-core:jar:3.0.7.RELEASE:provided
>|  +- org.springframework:spring-expression:jar:3.0.7.RELEASE:provided
>|  \- org.springframework:spring-asm:jar:3.0.7.RELEASE:provided
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Comment Edited] (FLUME-3131) Upgrade spring framework library dependencies

[Attila Simon (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLUME-3131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16095870#comment-16095870
 ] 

Attila Simon edited comment on FLUME-3131 at 7/21/17 6:56 AM:
--

Hi [~fszabo],
In general I'm fine with any approach which getting us closer to the state that 
flume is not vulnerable based on our understanding. 

Indeed it looks like test only. But having a closer look it seems like that 
activemq (parent dependency of spring and also brings in geronimo) also falls 
into the same category. I would also consider update the version of the 
activemq in case it still passes testing and doesn't bring in undesired 
dependencies transitively. (This in turn might help resolving this ticket by 
either removing the spring dependency completely or pulling in a "better" one)

{noformat}
⏚ [~/ws/apache/flume] trunk ± ag activemq *
flume-ng-doc/sphinx/FlumeUserGuide.rst
932:application it should work with any JMS provider but has only been tested 
with ActiveMQ.
945:**initialContextFactory**   --   Inital Context Factory, e.g: 
org.apache.activemq.jndi.ActiveMQInitialContextFactory
994:  a1.sources.r1.initialContextFactory = 
org.apache.activemq.jndi.ActiveMQInitialContextFactory

flume-ng-sources/flume-jms-source/pom.xml
74:  org.apache.activemq
75:  activemq-core

flume-ng-sources/flume-jms-source/src/test/java/org/apache/flume/source/jms/TestIntegrationActiveMQ.java
37:import org.apache.activemq.ActiveMQConnectionFactory;
38:import org.apache.activemq.broker.BrokerPlugin;
39:import org.apache.activemq.broker.BrokerService;
40:import org.apache.activemq.security.AuthenticationUser;
41:import org.apache.activemq.security.SimpleAuthenticationPlugin;
57:public class TestIntegrationActiveMQ {
60:  "org.apache.activemq.jndi.ActiveMQInitialContextFactory";
65:  // specific for dynamic queues on ActiveMq
133:ConnectionFactory factory = new ActiveMQConnectionFactory(USERNAME,
154:ConnectionFactory factory = new ActiveMQConnectionFactory(USERNAME,

pom.xml
1081:org.apache.activemq
1082:activemq-core
{noformat}


was (Author: sati):
Hi [~fszabo],
In general I'm fine with any approach which getting us closer to the state that 
flume is not vulnerable based on our understanding. 

Indeed it looks like test only. But having a closer look it seems like that 
activemq (parent dependency of geronimo) also falls into the same category. I 
would also consider update the version of the activemq in case it still passes 
testing and doesn't bring in undesired dependencies transitively. (This in turn 
might help resolving this ticket by either removing the spring dependency 
completely or pulling in a "better" one)

{noformat}
⏚ [~/ws/apache/flume] trunk ± ag activemq *
flume-ng-doc/sphinx/FlumeUserGuide.rst
932:application it should work with any JMS provider but has only been tested 
with ActiveMQ.
945:**initialContextFactory**   --   Inital Context Factory, e.g: 
org.apache.activemq.jndi.ActiveMQInitialContextFactory
994:  a1.sources.r1.initialContextFactory = 
org.apache.activemq.jndi.ActiveMQInitialContextFactory

flume-ng-sources/flume-jms-source/pom.xml
74:  org.apache.activemq
75:  activemq-core

flume-ng-sources/flume-jms-source/src/test/java/org/apache/flume/source/jms/TestIntegrationActiveMQ.java
37:import org.apache.activemq.ActiveMQConnectionFactory;
38:import org.apache.activemq.broker.BrokerPlugin;
39:import org.apache.activemq.broker.BrokerService;
40:import org.apache.activemq.security.AuthenticationUser;
41:import org.apache.activemq.security.SimpleAuthenticationPlugin;
57:public class TestIntegrationActiveMQ {
60:  "org.apache.activemq.jndi.ActiveMQInitialContextFactory";
65:  // specific for dynamic queues on ActiveMq
133:ConnectionFactory factory = new ActiveMQConnectionFactory(USERNAME,
154:ConnectionFactory factory = new ActiveMQConnectionFactory(USERNAME,

pom.xml
1081:org.apache.activemq
1082:activemq-core
{noformat}

> Upgrade spring framework library dependencies
> -
>
> Key: FLUME-3131
> URL: https://issues.apache.org/jira/browse/FLUME-3131
> Project: Flume
>  Issue Type: Bug
>Affects Versions: 1.7.0
>Reporter: Attila Simon
>Assignee: Ferenc Szabo
>Priority: Critical
>  Labels: dependency
> Fix For: 1.8.0
>
> Attachments: FLUME-3131.patch
>
>
> ||Group||Artifact||Version used||Upgrade target||
> |org.springframework|spring-aop|3.0.7.RELEASE|4.3.9.RELEASE,|
> |org.springframework|spring-context|3.0.7.RELEASE|4.3.9.RELEASE,|
> |org.springframework|spring-core|3.0.7.RELEASE|4.3.9.RELEASE,|
> Security vulnerability: 
> https://www

[jira] [Commented] (FLUME-3131) Upgrade spring framework library dependencies

[Attila Simon (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLUME-3131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16095870#comment-16095870
 ] 

Attila Simon commented on FLUME-3131:
-

Hi [~fszabo],
In general I'm fine with any approach which getting us closer to the state that 
flume is not vulnerable based on our understanding. 

Indeed it looks like test only. But having a closer look it seems like that 
activemq (parent dependency of geronimo) also falls into the same category. I 
would also consider update the version of the activemq in case it still passes 
testing and doesn't bring in undesired dependencies transitively. (This in turn 
might help resolving this ticket by either removing the spring dependency 
completely or pulling in a "better" one)

{noformat}
⏚ [~/ws/apache/flume] trunk ± ag activemq *
flume-ng-doc/sphinx/FlumeUserGuide.rst
932:application it should work with any JMS provider but has only been tested 
with ActiveMQ.
945:**initialContextFactory**   --   Inital Context Factory, e.g: 
org.apache.activemq.jndi.ActiveMQInitialContextFactory
994:  a1.sources.r1.initialContextFactory = 
org.apache.activemq.jndi.ActiveMQInitialContextFactory

flume-ng-sources/flume-jms-source/pom.xml
74:  org.apache.activemq
75:  activemq-core

flume-ng-sources/flume-jms-source/src/test/java/org/apache/flume/source/jms/TestIntegrationActiveMQ.java
37:import org.apache.activemq.ActiveMQConnectionFactory;
38:import org.apache.activemq.broker.BrokerPlugin;
39:import org.apache.activemq.broker.BrokerService;
40:import org.apache.activemq.security.AuthenticationUser;
41:import org.apache.activemq.security.SimpleAuthenticationPlugin;
57:public class TestIntegrationActiveMQ {
60:  "org.apache.activemq.jndi.ActiveMQInitialContextFactory";
65:  // specific for dynamic queues on ActiveMq
133:ConnectionFactory factory = new ActiveMQConnectionFactory(USERNAME,
154:ConnectionFactory factory = new ActiveMQConnectionFactory(USERNAME,

pom.xml
1081:org.apache.activemq
1082:activemq-core
{noformat}

> Upgrade spring framework library dependencies
> -
>
> Key: FLUME-3131
> URL: https://issues.apache.org/jira/browse/FLUME-3131
> Project: Flume
>  Issue Type: Bug
>Affects Versions: 1.7.0
>Reporter: Attila Simon
>Assignee: Ferenc Szabo
>Priority: Critical
>  Labels: dependency
> Fix For: 1.8.0
>
> Attachments: FLUME-3131.patch
>
>
> ||Group||Artifact||Version used||Upgrade target||
> |org.springframework|spring-aop|3.0.7.RELEASE|4.3.9.RELEASE,|
> |org.springframework|spring-context|3.0.7.RELEASE|4.3.9.RELEASE,|
> |org.springframework|spring-core|3.0.7.RELEASE|4.3.9.RELEASE,|
> Security vulnerability: 
> https://www.cvedetails.com/vulnerability-list/vendor_id-9664/product_id-17274/Springsource-Spring-Framework.html
> Maven repositories: 
> - https://mvnrepository.com/artifact/org.springframework/spring-aop
> - https://mvnrepository.com/artifact/org.springframework/spring-context
> - https://mvnrepository.com/artifact/org.springframework/spring-core
> Please do:
> - CVE might be a false alarm or mistake. Please double check.
> - double check the newest version. 
> - consider to remove a dependency if better alternative is available.
> - check whether the lib change would introduce a backward incompatibility (in 
> which case please add this label `breaking_change` and fix version should be 
> the next major)
> Excerpt from mvn dependency:tree
> {noformat}
> org.apache.flume.flume-ng-sources:flume-jms-source:jar:1.8.0-SNAPSHOT
> \- org.apache.activemq:activemq-core:jar:5.7.0:provided
>+- org.springframework:spring-context:jar:3.0.7.RELEASE:provided
>|  +- org.springframework:spring-aop:jar:3.0.7.RELEASE:provided
>|  +- org.springframework:spring-beans:jar:3.0.7.RELEASE:provided
>|  +- org.springframework:spring-core:jar:3.0.7.RELEASE:provided
>|  +- org.springframework:spring-expression:jar:3.0.7.RELEASE:provided
>|  \- org.springframework:spring-asm:jar:3.0.7.RELEASE:provided
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Comment Edited] (FLUME-3134) Enforce version of library dependencies during mvn build

[Attila Simon (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLUME-3134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16094523#comment-16094523
 ] 

Attila Simon edited comment on FLUME-3134 at 7/20/17 10:55 AM:
---

The idea came from a discussion on the dev@ list between 
[~ralph.go...@dslextreme.com] and [~mpercy]:


{quote}
> Mike,
>
> In all the projects I actively work on I put a dependencyManagement
> section in the parent pom and try to specify the version of every
> dependency there, whether direct or transitive. I also add the Maven
> enforcer plugin to make sure that subproject aren’t specifying differing
> versions of the same dependency.
>
> If one of the dependencies uses a library that is incompatible with a
> later version then you should create a bug report for that dependency so
> they can get it fixed.
>
> Ralph
>

Hey Ralph,
Nice. I didn't know enforcer could do that. After looking at its usage page
it seems easy to set up, too:
http://maven.apache.org/enforcer/enforcer-rules/dependencyConvergence.html

I would definitely be +1 for harmonizing Flume's dependencies and adding a
dependencyConvergence check to the Maven build. That first step should be
done with some care, though.

Mike
{quote}




was (Author: sati):
The idea came from a discussion on the dev@ list:

{noformat}
> Mike,
>
> In all the projects I actively work on I put a dependencyManagement
> section in the parent pom and try to specify the version of every
> dependency there, whether direct or transitive. I also add the Maven
> enforcer plugin to make sure that subproject aren’t specifying differing
> versions of the same dependency.
>
> If one of the dependencies uses a library that is incompatible with a
> later version then you should create a bug report for that dependency so
> they can get it fixed.
>
> Ralph
>

Hey Ralph,
Nice. I didn't know enforcer could do that. After looking at its usage page
it seems easy to set up, too:
http://maven.apache.org/enforcer/enforcer-rules/dependencyConvergence.html

I would definitely be +1 for harmonizing Flume's dependencies and adding a
dependencyConvergence check to the Maven build. That first step should be
done with some care, though.

Mike
{noformat}


> Enforce version of library dependencies during mvn build
> 
>
> Key: FLUME-3134
> URL: https://issues.apache.org/jira/browse/FLUME-3134
>     Project: Flume
>  Issue Type: Bug
>Affects Versions: 1.7.0
>Reporter: Attila Simon
>  Labels: dependency
>
> http://maven.apache.org/enforcer/enforcer-rules/dependencyConvergence.html



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLUME-3134) Enforce version of library dependencies during mvn build

[Attila Simon (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLUME-3134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16094523#comment-16094523
 ] 

Attila Simon commented on FLUME-3134:
-

The idea came from a discussion on the dev@ list:

{noformat}
> Mike,
>
> In all the projects I actively work on I put a dependencyManagement
> section in the parent pom and try to specify the version of every
> dependency there, whether direct or transitive. I also add the Maven
> enforcer plugin to make sure that subproject aren’t specifying differing
> versions of the same dependency.
>
> If one of the dependencies uses a library that is incompatible with a
> later version then you should create a bug report for that dependency so
> they can get it fixed.
>
> Ralph
>

Hey Ralph,
Nice. I didn't know enforcer could do that. After looking at its usage page
it seems easy to set up, too:
http://maven.apache.org/enforcer/enforcer-rules/dependencyConvergence.html

I would definitely be +1 for harmonizing Flume's dependencies and adding a
dependencyConvergence check to the Maven build. That first step should be
done with some care, though.

Mike
{noformat}


> Enforce version of library dependencies during mvn build
> 
>
> Key: FLUME-3134
> URL: https://issues.apache.org/jira/browse/FLUME-3134
> Project: Flume
>  Issue Type: Bug
>Affects Versions: 1.7.0
>Reporter: Attila Simon
>  Labels: dependency
>
> http://maven.apache.org/enforcer/enforcer-rules/dependencyConvergence.html



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (FLUME-3134) Enforce version of library dependencies during mvn build

[Attila Simon (JIRA)]

Attila Simon created FLUME-3134:
---

 Summary: Enforce version of library dependencies during mvn build
 Key: FLUME-3134
 URL: https://issues.apache.org/jira/browse/FLUME-3134
 Project: Flume
  Issue Type: Bug
Affects Versions: 1.7.0
Reporter: Attila Simon


http://maven.apache.org/enforcer/enforcer-rules/dependencyConvergence.html



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (FLUME-1520) Timestamp interceptor should support custom headers

[Attila Simon (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLUME-1520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Attila Simon updated FLUME-1520:

Attachment: FLUME-1520-3.patch

> Timestamp interceptor should support custom headers
> ---
>
> Key: FLUME-1520
> URL: https://issues.apache.org/jira/browse/FLUME-1520
> Project: Flume
>  Issue Type: Improvement
>Reporter: Hari Shreedharan
>Assignee: Hari Shreedharan
> Fix For: 1.8.0
>
> Attachments: FLUME-1520-2.patch, FLUME-1520-3.patch, FLUME-1520.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLUME-1520) Timestamp interceptor should support custom headers

[Attila Simon (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLUME-1520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16094497#comment-16094497
 ] 

Attila Simon commented on FLUME-1520:
-

Hi [~tmgstev],

The 2nd patch contains an extra file. I guess it was added by mistake: 
flume-ng-core/src/main/java/org/apache/flume/interceptor/TimestampInterceptor.java.orig
Unfortunately the patch fails maven build on checkstyle. 

Overall the doc, test and functionality looks good to me. 
I'm only nitpicking on the naming:
{code}
public class TimestampInterceptor implements Interceptor {
private final String header;
...
  public static class Constants {
public static final String CONFIG_PRESERVE = "preserveExisting";
public static final boolean DEFAULT_PRESERVE = false;
public static final String CONFIG_HEADER_NAME = "headerName";
public static final String DEFAULT_HEADER_NAME = "timestamp";
  }
{code}

I have this renaming and using final for constants as well as removed some 
unused local variables from the tests. I upload a patch which compiles and 
passes junit as well. Please feel free to use it (completely or parts) if you 
like.


> Timestamp interceptor should support custom headers
> ---
>
> Key: FLUME-1520
> URL: https://issues.apache.org/jira/browse/FLUME-1520
> Project: Flume
>  Issue Type: Improvement
>Reporter: Hari Shreedharan
>Assignee: Hari Shreedharan
> Fix For: 1.8.0
>
> Attachments: FLUME-1520-2.patch, FLUME-1520-3.patch, FLUME-1520.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (FLUME-3132) Upgrade tomcat jasper library dependencies

[Attila Simon (JIRA)]

Attila Simon created FLUME-3132:
---

 Summary: Upgrade tomcat jasper library dependencies
 Key: FLUME-3132
 URL: https://issues.apache.org/jira/browse/FLUME-3132
 Project: Flume
  Issue Type: Bug
Affects Versions: 1.7.0
Reporter: Attila Simon
Priority: Critical
 Fix For: 1.8.0


||Group||Artifact||Version used||Upgrade target||
|tomcat|jasper-compiler|5.5.23|8.5.x|
|tomcat|jasper-runtime|5.5.23|8.5.x|

Security vulnerability: 
- https://www.cvedetails.com/cve/CVE-2011-1318/
- 
http://www.cvedetails.com/vulnerability-list/vendor_id-45/product_id-887/Apache-Tomcat.html
Maven repositories: 
- https://mvnrepository.com/artifact/org.apache.tomcat/tomcat-jasper

Note: These artifacts were moved to:
* New Group org.apache.tomcat
* New Artifact  

Please do:
- CVE might be a false alarm or mistake. Please double check.
- double check the newest version. 
- consider to remove a dependency if better alternative is available.
- check whether the lib change would introduce a backward incompatibility (in 
which case please add this label `breaking_change` and fix version should be 
the next major)

Excerpt from mvn dependency:tree
{noformat}
org.apache.flume:flume-ng-auth:jar:1.8.0-SNAPSHOT
+- org.apache.hadoop:hadoop-common:jar:2.4.0:compile
|  +- tomcat:jasper-compiler:jar:5.5.23:runtime
|  +- tomcat:jasper-runtime:jar:5.5.23:runtime
{noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (FLUME-3131) Upgrade spring framework library dependencies

[Attila Simon (JIRA)]

Attila Simon created FLUME-3131:
---

 Summary: Upgrade spring framework library dependencies
 Key: FLUME-3131
 URL: https://issues.apache.org/jira/browse/FLUME-3131
 Project: Flume
  Issue Type: Bug
Affects Versions: 1.7.0
Reporter: Attila Simon
Priority: Critical
 Fix For: 1.8.0


||Group||Artifact||Version used||Upgrade target||
|org.springframework|spring-aop|3.0.7.RELEASE|4.3.9.RELEASE,|
|org.springframework|spring-context|3.0.7.RELEASE|4.3.9.RELEASE,|
|org.springframework|spring-core|3.0.7.RELEASE|4.3.9.RELEASE,|

Security vulnerability: 
https://www.cvedetails.com/vulnerability-list/vendor_id-9664/product_id-17274/Springsource-Spring-Framework.html
Maven repositories: 
- https://mvnrepository.com/artifact/org.springframework/spring-aop
- https://mvnrepository.com/artifact/org.springframework/spring-context
- https://mvnrepository.com/artifact/org.springframework/spring-core

Please do:
- CVE might be a false alarm or mistake. Please double check.
- double check the newest version. 
- consider to remove a dependency if better alternative is available.
- check whether the lib change would introduce a backward incompatibility (in 
which case please add this label `breaking_change` and fix version should be 
the next major)

Excerpt from mvn dependency:tree
{noformat}
org.apache.flume.flume-ng-sources:flume-jms-source:jar:1.8.0-SNAPSHOT
\- org.apache.activemq:activemq-core:jar:5.7.0:provided
   +- org.springframework:spring-context:jar:3.0.7.RELEASE:provided
   |  +- org.springframework:spring-aop:jar:3.0.7.RELEASE:provided
   |  +- org.springframework:spring-beans:jar:3.0.7.RELEASE:provided
   |  +- org.springframework:spring-core:jar:3.0.7.RELEASE:provided
   |  +- org.springframework:spring-expression:jar:3.0.7.RELEASE:provided
   |  \- org.springframework:spring-asm:jar:3.0.7.RELEASE:provided
{noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (FLUME-3130) Upgrade restlet library dependency

[Attila Simon (JIRA)]

Attila Simon created FLUME-3130:
---

 Summary: Upgrade restlet library dependency
 Key: FLUME-3130
 URL: https://issues.apache.org/jira/browse/FLUME-3130
 Project: Flume
  Issue Type: Bug
Affects Versions: 1.7.0
Reporter: Attila Simon
Priority: Critical
 Fix For: 1.8.0


||Group||Artifact||Version used||Upgrade target||
|org.restlet.jee|org.restlet|2.1.1|2.3.10|

Security vulnerability: 
http://www.cvedetails.com/vulnerability-list/vendor_id-12911/product_id-26316/Restlet-Restlet.html
Maven: https://mvnrepository.com/artifact/org.restlet.jee/org.restlet

Please do:
- CVE might be a false alarm or mistake. Please double check.
- double check the newest version. 
- consider to remove a dependency if better alternative is available.
- check whether the lib change would introduce a backward incompatibility (in 
which case please add this label `breaking_change` and fix version should be 
the next major)

Excerpt from mvn dependency:tree
{noformat}
org.apache.flume.flume-ng-sinks:flume-ng-morphline-solr-sink:jar:1.8.0-SNAPSHOT
+- org.apache.solr:solr-test-framework:jar:4.3.0:test
|  +- org.apache.solr:solr-core:jar:4.3.0:compile
|  |  +- org.restlet.jee:org.restlet:jar:2.1.1:compile
|  |  +- org.restlet.jee:org.restlet.ext.servlet:jar:2.1.1:compile
{noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (FLUME-2698) Upgrade Jetty Version

[Attila Simon (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLUME-2698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Attila Simon updated FLUME-2698:

Labels: dependency  (was: )

> Upgrade Jetty Version
> -
>
> Key: FLUME-2698
> URL: https://issues.apache.org/jira/browse/FLUME-2698
> Project: Flume
>  Issue Type: Bug
>  Components: Web
>Affects Versions: 1.6.0, 1.5.1, 1.7.0
>Reporter: Joakim Erdfelt
>Assignee: Tristan Stevens
>  Labels: dependency
>
> Flume depends on Jetty 6
> {code:xml}
>   
> org.mortbay.jetty
> jetty-util
> 6.1.26
>   
> {code}
> Which was EOL (End of Life) back in 2010 and is no longer fit for production 
> use (without heavy customizations and modifications like Google does for GAE, 
> just to keep it safe and vulnerability free)
> Jetty was moved to Eclipse.org back during the Jetty 7 days.
> http://eclipse.org/jetty/
> Note that [Jetty 7 and Jetty 8 are now also 
> EOL|https://dev.eclipse.org/mhonarc/lists/jetty-announce/msg00069.html] (as 
> of 2014)
> Jetty 9 is the only stable and supported version of Jetty now.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (FLUME-3129) Upgrade bouncycastle library dependencies

[Attila Simon (JIRA)]

Attila Simon created FLUME-3129:
---

 Summary: Upgrade bouncycastle library dependencies
 Key: FLUME-3129
 URL: https://issues.apache.org/jira/browse/FLUME-3129
 Project: Flume
  Issue Type: Bug
Affects Versions: 1.7.0
Reporter: Attila Simon
Priority: Critical
 Fix For: 1.8.0


||Group||Artifact||Version used||Upgrade target||
|org.bouncycastle|bcprov-jdk15|1.45|1.57|
|org.bouncycastle|bcmail-jdk15|1.45|1.57|

Security vulnerability: 
https://www.cvedetails.com/vulnerability-list/vendor_id-7637/Bouncycastle.html
Maven repository: 
https://mvnrepository.com/artifact/org.bouncycastle/bcprov-jdk15

Please do:
- CVE might be a false alarm or mistake. Please double check.
- double check the newest version. 
- consider to remove a dependency if better alternative is available.
- check whether the lib change would introduce a backward incompatibility (in 
which case please add this label `breaking_change` and fix version should be 
the next major)

Excerpt from mvn dependency:tree
{noformat}
org.apache.flume.flume-ng-sinks:flume-ng-morphline-solr-sink:jar:1.8.0-SNAPSHOT
+- org.kitesdk:kite-morphlines-all:pom:1.0.0:compile
|  +- org.kitesdk:kite-morphlines-solr-cell:jar:1.0.0:compile
|  |  +- org.apache.tika:tika-xmp:jar:1.5:compile
|  |  |  +- org.apache.tika:tika-parsers:jar:1.5:compile
|  |  |  |  +- org.bouncycastle:bcmail-jdk15:jar:1.45:compile
|  |  |  |  +- org.bouncycastle:bcprov-jdk15:jar:1.45:compile
{noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (FLUME-3127) Upgrade libfb303 library dependency

[Attila Simon (JIRA)]

Attila Simon created FLUME-3127:
---

 Summary: Upgrade libfb303 library dependency
 Key: FLUME-3127
 URL: https://issues.apache.org/jira/browse/FLUME-3127
 Project: Flume
  Issue Type: Bug
Affects Versions: 1.7.0
Reporter: Attila Simon
Priority: Critical
 Fix For: 1.8.0


||Group||Artifact||Version used||Upgrade target||
|org.apache.thrift|libthrift|0.9.0|0.9.3,0.10.0|
|org.apache.thrift|libfb303|0.9.0|0.9.3|

Security vulnerability: http://www.cvedetails.com/cve/CVE-2015-3254/
Maven repository: 
- https://mvnrepository.com/artifact/org.apache.thrift/libthrift
- https://mvnrepository.com/artifact/org.apache.thrift/libfb303

Please do:
- CVE might be a false alarm or mistake. Please double check.
- double check the newest version. 
- consider to remove a dependency if better alternative is available.
- check whether the lib change would introduce a backward incompatibility (in 
which case please add this label `breaking_change` and fix version should be 
the next major)

Excerpt from mvn dependency:tree
{noformat}
org.apache.flume:flume-ng-sdk:jar:1.8.0-SNAPSHOT
\- org.apache.thrift:libthrift:jar:0.9.0:compile

org.apache.flume.flume-ng-sinks:flume-hive-sink:jar:1.8.0-SNAPSHOT
+- org.apache.hive.hcatalog:hive-hcatalog-streaming:jar:1.0.0:provided
|  +- org.apache.hive:hive-metastore:jar:1.0.0:provided
|  |  \- org.apache.thrift:libfb303:jar:0.9.0:provided
{noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (FLUME-3126) Upgrade apache poi library dependencies

[Attila Simon (JIRA)]

Attila Simon created FLUME-3126:
---

 Summary: Upgrade apache poi library dependencies
 Key: FLUME-3126
 URL: https://issues.apache.org/jira/browse/FLUME-3126
 Project: Flume
  Issue Type: Bug
Affects Versions: 1.7.0
Reporter: Attila Simon
Priority: Critical
 Fix For: 1.8.0


||Group||Artifact||Version used||Upgrade target||
|org.apache.poi|poi|3.10-beta2|3.15-beta2|
|org.apache.poi|poi-ooxml|3.10-beta2|3.15-beta2|
|org.apache.poi|poi-scratchpad|3.10-beta2|3.15-beta2|

Security vulnerability: 
https://www.cvedetails.com/vulnerability-list/vendor_id-45/product_id-22766/Apache-POI.html
Maven repositories: 
- https://mvnrepository.com/artifact/org.apache.poi/poi-ooxml
- https://mvnrepository.com/artifact/org.apache.poi/poi
- https://mvnrepository.com/artifact/org.apache.poi/poi

Please do:
- CVE might be a false alarm or mistake. Please double check.
- double check the newest version. 
- consider to remove a dependency if better alternative is available.
- check whether the lib change would introduce a backward incompatibility (in 
which case please add this label `breaking_change` and fix version should be 
the next major)

Excerpt from mvn dependency:tree
{noformat}
org.apache.flume.flume-ng-sinks:flume-ng-morphline-solr-sink:jar:1.8.0-SNAPSHOT
+- org.kitesdk:kite-morphlines-all:pom:1.0.0:compile
|  +- org.kitesdk:kite-morphlines-solr-cell:jar:1.0.0:compile
|  |  +- org.apache.tika:tika-xmp:jar:1.5:compile
|  |  |  +- org.apache.tika:tika-parsers:jar:1.5:compile
|  |  |  |  +- org.apache.poi:poi:jar:3.10-beta2:compile
|  |  |  |  +- org.apache.poi:poi-scratchpad:jar:3.10-beta2:compile
|  |  |  |  +- org.apache.poi:poi-ooxml:jar:3.10-beta2:compile
{noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (FLUME-3125) Upgrade fontbox library dependency

[Attila Simon (JIRA)]

Attila Simon created FLUME-3125:
---

 Summary: Upgrade fontbox library dependency
 Key: FLUME-3125
 URL: https://issues.apache.org/jira/browse/FLUME-3125
 Project: Flume
  Issue Type: Bug
Affects Versions: 1.7.0
Reporter: Attila Simon
Priority: Critical
 Fix For: 1.8.0


||Group||Artifact||Version used||Upgrade target||
|org.apache.pdfbox|fontbox|1.8.4|2.0.6|

Security vulnerability: http://www.cvedetails.com/cve/CVE-2016-2175/
Maven repository: https://mvnrepository.com/artifact/org.apache.pdfbox/fontbox

Please do:
- CVE might be a false alarm or mistake. Please double check.
- double check the newest version. 
- consider to remove a dependency if better alternative is available.
- check whether the lib change would introduce a backward incompatibility (in 
which case please add this label `breaking_change` and fix version should be 
the next major)

Excerpt from mvn dependency:tree
{noformat}
org.apache.flume.flume-ng-sinks:flume-ng-morphline-solr-sink:jar:1.8.0-SNAPSHOT
+- org.kitesdk:kite-morphlines-all:pom:1.0.0:compile
|  +- org.kitesdk:kite-morphlines-solr-cell:jar:1.0.0:compile
|  |  +- org.apache.tika:tika-xmp:jar:1.5:compile
|  |  |  +- org.apache.tika:tika-parsers:jar:1.5:compile
|  |  |  |  +- org.apache.pdfbox:pdfbox:jar:1.8.4:compile
|  |  |  |  |  +- org.apache.pdfbox:fontbox:jar:1.8.4:compile
{noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (FLUME-3124) Upgrade apache-mime4j-core library dependency

[Attila Simon (JIRA)]

Attila Simon created FLUME-3124:
---

 Summary: Upgrade apache-mime4j-core library dependency
 Key: FLUME-3124
 URL: https://issues.apache.org/jira/browse/FLUME-3124
 Project: Flume
  Issue Type: Bug
Affects Versions: 1.7.0
Reporter: Attila Simon
Priority: Critical
 Fix For: 1.8.0


||Group||Artifact||Version used||Upgrade target||
|org.apache.james|apache-mime4j-core|0.7.2|0.8.1|

Security vulnerability: 
https://www.cvedetails.com/vulnerability-list/vendor_id-45/product_id-4526/Apache-James.html
 
Maven repository: 
https://mvnrepository.com/artifact/org.apache.james/apache-mime4j

Please do:
- CVE might be a false alarm or mistake. Please double check.
- double check the newest version. 
- consider to remove a dependency if better alternative is available.
- check whether the lib change would introduce a backward incompatibility (in 
which case please add this label `breaking_change` and fix version should be 
the next major)

Excerpt from mvn dependency:tree
{noformat}
org.apache.flume.flume-ng-sinks:flume-ng-morphline-solr-sink:jar:1.8.0-SNAPSHOT
+- org.kitesdk:kite-morphlines-all:pom:1.0.0:compile
|  +- org.kitesdk:kite-morphlines-solr-cell:jar:1.0.0:compile
|  |  +- org.apache.tika:tika-xmp:jar:1.5:compile
|  |  |  +- org.apache.tika:tika-parsers:jar:1.5:compile
|  |  |  |  +- org.apache.james:apache-mime4j-core:jar:0.7.2:compile
|  |  |  |  +- org.apache.james:apache-mime4j-dom:jar:0.7.2:compile
{noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLUME-3114) Upgrade commons-httpclient library dependency

[Attila Simon (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLUME-3114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16093050#comment-16093050
 ] 

Attila Simon commented on FLUME-3114:
-

Linking related tickets. Please note that both 
commons-httpclient:commons-httpclient and org.apache.httpcomponents:httpclient 
(new maven group/artifact name) are loaded into flume classpath. Ideal state 
would be to depend only this one:
https://mvnrepository.com/artifact/org.apache.httpcomponents/httpclient

Excerpt from dependecy:tree (it appears multiple places in the dep:tree I 
copied only a single location)
{noformat}
org.apache.flume.flume-ng-sinks:flume-ng-elasticsearch-sink:jar:1.8.0-SNAPSHOT
+- org.apache.httpcomponents:httpclient:jar:4.2.1:compile
{noformat}


> Upgrade commons-httpclient library dependency
> -
>
> Key: FLUME-3114
> URL: https://issues.apache.org/jira/browse/FLUME-3114
> Project: Flume
>  Issue Type: Bug
>Affects Versions: 1.7.0
>    Reporter: Attila Simon
>Priority: Critical
>  Labels: dependency
> Fix For: 1.8.0
>
>
> ||Group||Artifact||Version used||Upgrade target||
> |commons-httpclient|commons-httpclient|3.1,3.0.1|4.5.2|
> Note: This artifact was moved to:
> * New Group   org.apache.httpcomponents
> * New Artifacthttpclient
> Security vulnerability: https://www.cvedetails.com/cve/CVE-2012-5783/
> Please do:
> - double check the newest version. 
> - consider to remove a dependency if better alternative is available.
> - check whether the lib change would introduce a backward incompatibility (in 
> which case please add this label `breaking_change` and fix version should be 
> the next major)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (FLUME-3121) Upgrade javax.mail:test dependency

[Attila Simon (JIRA)]

Attila Simon created FLUME-3121:
---

 Summary: Upgrade javax.mail:test dependency
 Key: FLUME-3121
 URL: https://issues.apache.org/jira/browse/FLUME-3121
 Project: Flume
  Issue Type: Bug
Affects Versions: 1.7.0
Reporter: Attila Simon
Priority: Critical
 Fix For: 1.8.0


||Group||Artifact||Version used||Upgrade target||
|javax.mail|mail|1.4.1|1.5.6|

Note: This artifact was moved to:
- New Group javax.mail
- New Artifact  javax.mail-api

Security vulnerability: 
https://www.cvedetails.com/vulnerability-list/vendor_id-5/product_id-5085/SUN-Javamail.html
 
Note: this might be a false alarm. Please double check the CVE. 

Please do:
- double check the newest version. 
- consider to remove a dependency if better alternative is available.
- check whether the lib change would introduce a backward incompatibility (in 
which case please add this label `breaking_change` and fix version should be 
the next major)

Excerpt from mvn dependency:tree
{noformat}
+- org.apache.hive:hive-cli:jar:1.0.0:test
|  +- org.apache.hive:hive-service:jar:1.0.0:test
|  |  +- org.eclipse.jetty.aggregate:jetty-all:jar:7.6.0.v20120127:test
|  |  |  +- javax.mail:mail:jar:1.4.1:test
{noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLUME-2977) Upgrade RAT to 0.12

[Attila Simon (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLUME-2977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16091741#comment-16091741
 ] 

Attila Simon commented on FLUME-2977:
-

RAT-222 is resolved. 0.12 is available : 
http://creadur.apache.org/rat/download_rat.cgi

> Upgrade RAT to 0.12
> ---
>
> Key: FLUME-2977
> URL: https://issues.apache.org/jira/browse/FLUME-2977
> Project: Flume
>  Issue Type: Improvement
>  Components: Build
>    Reporter: Attila Simon
>Assignee: Attila Simon
>Priority: Minor
> Fix For: 1.8.0
>
> Attachments: FLUME-2977.patch
>
>
> Before RAT check mvn install prints out the following warnings:
> {noformat}
> Warning:  org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser: Property 
> 'http://www.oracle.com/xml/jaxp/properties/entityExpansionLimit' is not 
> recognized.
> Compiler warnings:
>   WARNING:  'org.apache.xerces.jaxp.SAXParserImpl: Property 
> 'http://javax.xml.XMLConstants/property/accessExternalDTD' is not recognized.'
> Warning:  org.apache.xerces.parsers.SAXParser: Feature 
> 'http://javax.xml.XMLConstants/feature/secure-processing' is not recognized.
> Warning:  org.apache.xerces.parsers.SAXParser: Property 
> 'http://javax.xml.XMLConstants/property/accessExternalDTD' is not recognized.
> Warning:  org.apache.xerces.parsers.SAXParser: Property 
> 'http://www.oracle.com/xml/jaxp/properties/entityExpansionLimit' is not 
> recognized.
> [INFO] Rat check: Summary of files. Unapproved: 0 unknown: 0 generated: 0 
> approved: 9 licence.
> {noformat} 
> It doesn't break the build but seems misleading.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLUME-3114) Upgrade commons-httpclient library dependency

[Attila Simon (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLUME-3114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16091725#comment-16091725
 ] 

Attila Simon commented on FLUME-3114:
-

Excerpt from dependency:tree
{noformat}
org.apache.flume:flume-ng-auth:jar:1.8.0-SNAPSHOT
+- org.apache.hadoop:hadoop-common:jar:2.4.0:compile
|  +- commons-httpclient:commons-httpclient:jar:3.1:compile
{noformat}

> Upgrade commons-httpclient library dependency
> -
>
> Key: FLUME-3114
> URL: https://issues.apache.org/jira/browse/FLUME-3114
> Project: Flume
>  Issue Type: Bug
>Affects Versions: 1.7.0
>    Reporter: Attila Simon
>Priority: Critical
>  Labels: dependency
> Fix For: 1.8.0
>
>
> ||Group||Artifact||Version used||Upgrade target||
> |commons-httpclient|commons-httpclient|3.1,3.0.1|4.5.2|
> Note: This artifact was moved to:
> * New Group   org.apache.httpcomponents
> * New Artifacthttpclient
> Security vulnerability: https://www.cvedetails.com/cve/CVE-2012-5783/
> Please do:
> - double check the newest version. 
> - consider to remove a dependency if better alternative is available.
> - check whether the lib change would introduce a backward incompatibility (in 
> which case please add this label `breaking_change` and fix version should be 
> the next major)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLUME-3113) Upgrade commons-beanutils library dependency

[Attila Simon (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLUME-3113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16091724#comment-16091724
 ] 

Attila Simon commented on FLUME-3113:
-

Excerpt from dependency:tree
{noformat}
org.apache.flume:flume-ng-auth:jar:1.8.0-SNAPSHOT
+- org.apache.hadoop:hadoop-common:jar:2.4.0:compile
|  +- commons-configuration:commons-configuration:jar:1.6:compile
|  |  +- commons-digester:commons-digester:jar:1.8:compile
|  |  |  \- commons-beanutils:commons-beanutils:jar:1.7.0:compile
|  |  \- commons-beanutils:commons-beanutils-core:jar:1.8.0:compile
{noformat}


> Upgrade commons-beanutils library dependency
> 
>
> Key: FLUME-3113
> URL: https://issues.apache.org/jira/browse/FLUME-3113
> Project: Flume
>  Issue Type: Bug
>Affects Versions: 1.7.0
>    Reporter: Attila Simon
>Priority: Critical
>  Labels: dependency
> Fix For: 1.8.0
>
>
> ||Group||Artifact||Version used||Upgrade target||
> |commons-beanutils|commons-beanutils|1.7.0|1.9.3|
> |commons-beanutils|commons-beanutils-core|1.8.0|1.8.3|
> Security vulnerability: https://www.cvedetails.com/cve/CVE-2014-0114/
> Please do:
> - double check the newest version. 
> - consider to remove a dependency if better alternative is available.
> - check whether the lib change would introduce a backward incompatibility (in 
> which case please add this label `breaking_change` and fix version should be 
> the next major)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (FLUME-3115) Upgrade netty library dependency

[Attila Simon (JIRA)]

Attila Simon created FLUME-3115:
---

 Summary: Upgrade netty library dependency
 Key: FLUME-3115
 URL: https://issues.apache.org/jira/browse/FLUME-3115
 Project: Flume
  Issue Type: Bug
Affects Versions: 1.7.0
Reporter: Attila Simon
Priority: Critical
 Fix For: 1.8.0


||Group||Artifact||Version used||Upgrade target||
|io.netty|netty|3.2.2.Final, 3.9.4.Final|4.1.12.Final|

Note: This artifact was moved to:
- New Group io.netty
- New Artifact  netty-all

Security vulnerability: http://www.cvedetails.com/cve/CVE-2014-3488/

Please do:
- double check the newest version. 
- consider to remove a dependency if better alternative is available.
- check whether the lib change would introduce a backward incompatibility (in 
which case please add this label `breaking_change` and fix version should be 
the next major)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLUME-1732) Build is failing due to netty problems

[Attila Simon (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLUME-1732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16075044#comment-16075044
 ] 

Attila Simon commented on FLUME-1732:
-

Interestingly it seems like it has been committed:
{noformat}
trunk(964bcf56)$ git log --oneline --grep FLUME-1732
750809c7 FLUME-1732: SpoolableDirectorySource should have configurable support 
for deleting files it has already completed instead of renaming
{noformat}
Unfortunately this is just a mistake in commit message. That change belongs to 
FLUME-1731 instead.

I followed [~mpercy]'s steps from above but no failure for me. I conclude 
everything went back to normal since this ticket was opened. So marking this 
ticket as resolved. 
{noformat}
trunk(964bcf56)$ mvn -version
Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5; 
2015-11-10T17:41:47+01:00)
Maven home: /usr/local/Cellar/maven/3.3.9/libexec
Java version: 1.8.0_101, vendor: Oracle Corporation
Java home: /Library/Java/JavaVirtualMachines/jdk1.8.0_101.jdk/Contents/Home/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "mac os x", version: "10.12.5", arch: "x86_64", family: "mac"
{noformat} 

> Build is failing due to netty problems
> --
>
> Key: FLUME-1732
> URL: https://issues.apache.org/jira/browse/FLUME-1732
> Project: Flume
>  Issue Type: Bug
>Affects Versions: 1.4.0
>Reporter: Brock Noland
>Assignee: Mike Percy
> Attachments: FLUME-1732-3.patch, FLUME-1732.patch
>
>
> FLUME-1723 changed how we bring in netty and that seems to have broken the 
> build https://builds.apache.org/job/flume-trunk/330/#showFailuresLink



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (FLUME-2501) Updating HttpClient lib version to ensure compat with Solr

[Attila Simon (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLUME-2501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Attila Simon updated FLUME-2501:

Labels: dependency  (was: )

> Updating HttpClient lib version to ensure compat with Solr
> --
>
> Key: FLUME-2501
> URL: https://issues.apache.org/jira/browse/FLUME-2501
> Project: Flume
>  Issue Type: Bug
>  Components: Sinks+Sources
>Affects Versions: 1.5.0.1
>Reporter: Roshan Naik
>Assignee: Roshan Naik
>  Labels: dependency
> Attachments: FLUME-2501.patch, FLUME-2501.v2.patch
>
>
> Mismatch in httpclient and http core libs pulled by flume v/s the ones that 
> come with Solr causes errors at runtime
> {code}
> 2014-10-13 19:52:32,042 (lifecycleSupervisor-1-1) [DEBUG - 
> org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:106)]
>  Creating new http client, 
> config:maxConnections=128=32=false
> 2014-10-13 19:52:32,225 (lifecycleSupervisor-1-1) [ERROR - 
> org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:253)]
>  Unable to start SinkRunner: { 
> policy:org.apache.flume.sink.DefaultSinkProcessor@4752b854 counterGroup:{ 
> name:null counters:{} } } - Exception follows.
> java.lang.NoSuchFieldError: DEF_CONTENT_CHARSET
>   at 
> org.apache.http.impl.client.DefaultHttpClient.setDefaultHttpParams(DefaultHttpClient.java:175)
>   at 
> org.apache.http.impl.client.DefaultHttpClient.createHttpParams(DefaultHttpClient.java:158)
>   at 
> org.apache.http.impl.client.AbstractHttpClient.getParams(AbstractHttpClient.java:448)
>   at 
> org.apache.solr.client.solrj.impl.HttpClientUtil.setFollowRedirects(HttpClientUtil.java:251)
>   at 
> org.apache.solr.client.solrj.impl.HttpClientConfigurer.configure(HttpClientConfigurer.java:58)
>   at 
> org.apache.solr.client.solrj.impl.HttpClientUtil.configureClient(HttpClientUtil.java:133)
>   at 
> org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:109)
>   at 
> org.apache.solr.client.solrj.impl.HttpSolrServer.(HttpSolrServer.java:161)
>   at 
> org.apache.solr.client.solrj.impl.HttpSolrServer.(HttpSolrServer.java:138)
>   at 
> org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer.(ConcurrentUpdateSolrServer.java:122)
>   at 
> org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer.(ConcurrentUpdateSolrServer.java:114)
>   at 
> org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer.(ConcurrentUpdateSolrServer.java:104)
>   at 
> org.kitesdk.morphline.solr.SafeConcurrentUpdateSolrServer.(SafeConcurrentUpdateSolrServer.java:39)
>   at 
> org.kitesdk.morphline.solr.SafeConcurrentUpdateSolrServer.(SafeConcurrentUpdateSolrServer.java:35)
>   at 
> org.kitesdk.morphline.solr.SolrLocator.getLoader(SolrLocator.java:116)
>   at 
> org.kitesdk.morphline.solr.LoadSolrBuilder$LoadSolr.(LoadSolrBuilder.java:70)
>   at 
> org.kitesdk.morphline.solr.LoadSolrBuilder.build(LoadSolrBuilder.java:52)
>   at 
> org.kitesdk.morphline.base.AbstractCommand.buildCommand(AbstractCommand.java:303)
>   at 
> org.kitesdk.morphline.base.AbstractCommand.buildCommandChain(AbstractCommand.java:250)
>   at org.kitesdk.morphline.stdlib.Pipe.(Pipe.java:46)
>   at org.kitesdk.morphline.stdlib.PipeBuilder.build(PipeBuilder.java:40)
>   at org.kitesdk.morphline.base.Compiler.compile(Compiler.java:126)
>   at org.kitesdk.morphline.base.Compiler.compile(Compiler.java:55)
>   at 
> org.apache.flume.sink.solr.morphline.MorphlineHandlerImpl.configure(MorphlineHandlerImpl.java:101)
>   at 
> org.apache.flume.sink.solr.morphline.MorphlineSink.start(MorphlineSink.java:97)
>   at 
> org.apache.flume.sink.DefaultSinkProcessor.start(DefaultSinkProcessor.java:46)
>   at org.apache.flume.SinkRunner.start(SinkRunner.java:79)
>   at 
> org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>   at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
>   at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
>   at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:745)
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLUME-3114) Upgrade commons-httpclient library dependency

[Attila Simon (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLUME-3114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16074995#comment-16074995
 ] 

Attila Simon commented on FLUME-3114:
-

We have a very similar jira where patch is ready to upgrade to an older than 
the currently proposed version. The proposed version there doesn't seem to have 
any CVE yet. 

> Upgrade commons-httpclient library dependency
> -
>
> Key: FLUME-3114
> URL: https://issues.apache.org/jira/browse/FLUME-3114
> Project: Flume
>  Issue Type: Bug
>Affects Versions: 1.7.0
>    Reporter: Attila Simon
>Priority: Critical
>  Labels: dependency
> Fix For: 1.8.0
>
>
> ||Group||Artifact||Version used||Upgrade target||
> |commons-httpclient|commons-httpclient|3.1,3.0.1|4.5.2|
> Note: This artifact was moved to:
> * New Group   org.apache.httpcomponents
> * New Artifacthttpclient
> Security vulnerability: https://www.cvedetails.com/cve/CVE-2012-5783/
> Please do:
> - double check the newest version. 
> - consider to remove a dependency if better alternative is available.
> - check whether the lib change would introduce a backward incompatibility (in 
> which case please add this label `breaking_change` and fix version should be 
> the next major)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (FLUME-3114) Upgrade commons-httpclient library dependency

[Attila Simon (JIRA)]

Attila Simon created FLUME-3114:
---

 Summary: Upgrade commons-httpclient library dependency
 Key: FLUME-3114
 URL: https://issues.apache.org/jira/browse/FLUME-3114
 Project: Flume
  Issue Type: Bug
Affects Versions: 1.7.0
Reporter: Attila Simon
Priority: Critical
 Fix For: 1.8.0


||Group||Artifact||Version used||Upgrade target||
|commons-httpclient|commons-httpclient|3.1,3.0.1|4.5.2|

Note: This artifact was moved to:
* New Group org.apache.httpcomponents
* New Artifact  httpclient

Security vulnerability: https://www.cvedetails.com/cve/CVE-2012-5783/

Please do:
- double check the newest version. 
- consider to remove a dependency if better alternative is available.
- check whether the lib change would introduce a backward incompatibility (in 
which case please add this label `breaking_change` and fix version should be 
the next major)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (FLUME-3113) Upgrade commons-beanutils library dependency

[Attila Simon (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLUME-3113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Attila Simon updated FLUME-3113:

Description: 
||Group||Artifact||Version used||Upgrade target||
|commons-beanutils|commons-beanutils|1.7.0|1.9.3|
|commons-beanutils|commons-beanutils-core|1.8.0|1.8.3|

Security vulnerability: https://www.cvedetails.com/cve/CVE-2014-0114/

Please do:
- double check the newest version. 
- consider to remove a dependency if better alternative is available.
- check whether the lib change would introduce a backward incompatibility (in 
which case please add this label `breaking_change` and fix version should be 
the next major)

  was:
||Group||Artifact||Version used||Upgrade target||
|commons-beanutils|commons-beanutils|1.7.0|1.9.3|

Security vulnerability: https://www.cvedetails.com/cve/CVE-2014-0114/

Please do:
- double check the newest version. 
- consider to remove a dependency if better alternative is available.
- check whether the lib change would introduce a backward incompatibility (in 
which case please add this label `breaking_change` and fix version should be 
the next major)


> Upgrade commons-beanutils library dependency
> 
>
> Key: FLUME-3113
> URL: https://issues.apache.org/jira/browse/FLUME-3113
> Project: Flume
>  Issue Type: Bug
>Affects Versions: 1.7.0
>    Reporter: Attila Simon
>Priority: Critical
>  Labels: dependency
> Fix For: 1.8.0
>
>
> ||Group||Artifact||Version used||Upgrade target||
> |commons-beanutils|commons-beanutils|1.7.0|1.9.3|
> |commons-beanutils|commons-beanutils-core|1.8.0|1.8.3|
> Security vulnerability: https://www.cvedetails.com/cve/CVE-2014-0114/
> Please do:
> - double check the newest version. 
> - consider to remove a dependency if better alternative is available.
> - check whether the lib change would introduce a backward incompatibility (in 
> which case please add this label `breaking_change` and fix version should be 
> the next major)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLUME-3112) Upgrade jackson-core library dependency

[Attila Simon (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLUME-3112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16074953#comment-16074953
 ] 

Attila Simon commented on FLUME-3112:
-

Excerpted transitive dependency tree from `mvn dependency:tree` 

{noformat}
org.apache.flume.flume-ng-sinks:flume-dataset-sink:jar:1.8.0-SNAPSHOT
org.kitesdk:kite-data-core:jar:1.0.0:compile
com.fasterxml.jackson.core:jackson-databind:jar:2.3.1:compile
com.fasterxml.jackson.core:jackson-annotations:jar:2.3.0:compile

com.fasterxml.jackson.core:jackson-core:jar:2.3.1:compile
{noformat}

{noformat}
org.apache.flume.flume-ng-sinks:flume-ng-morphline-solr-sink:jar:1.8.0-SNAPSHOT
org.kitesdk:kite-morphlines-all:pom:1.0.0:compile
org.kitesdk:kite-morphlines-json:jar:1.0.0:compile
com.fasterxml.jackson.core:jackson-databind:jar:2.3.1:compile
com.fasterxml.jackson.core:jackson-annotations:jar:2.3.0:compile
com.fasterxml.jackson.core:jackson-core:jar:2.3.1:compile
{noformat}

> Upgrade jackson-core library dependency
> ---
>
> Key: FLUME-3112
> URL: https://issues.apache.org/jira/browse/FLUME-3112
> Project: Flume
>  Issue Type: Bug
>Affects Versions: 1.7.0
>    Reporter: Attila Simon
>Priority: Critical
>  Labels: dependency
> Fix For: 1.8.0
>
>
> ||Group||Artifact||Version used||Upgrade target||
> |com.fasterxml.jackson.core|jackson-core|2.3.1|2.8.9|
> Security vulnerability: http://www.cvedetails.com/cve/CVE-2016-7051/
> Please do:
> - double check the newest version. 
> - consider to remove a dependency if better alternative is available.
> - check whether the lib change would introduce a backward incompatibility (in 
> which case please add this label `breaking_change` and fix version should be 
> the next major)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (FLUME-3113) Upgrade commons-beanutils library dependency

[Attila Simon (JIRA)]

Attila Simon created FLUME-3113:
---

 Summary: Upgrade commons-beanutils library dependency
 Key: FLUME-3113
 URL: https://issues.apache.org/jira/browse/FLUME-3113
 Project: Flume
  Issue Type: Bug
Affects Versions: 1.7.0
Reporter: Attila Simon
Priority: Critical
 Fix For: 1.8.0


||Group||Artifact||Version used||Upgrade target||
|commons-beanutils|commons-beanutils|1.7.0|1.9.3|

Security vulnerability: https://www.cvedetails.com/cve/CVE-2014-0114/

Please do:
- double check the newest version. 
- consider to remove a dependency if better alternative is available.
- check whether the lib change would introduce a backward incompatibility (in 
which case please add this label `breaking_change` and fix version should be 
the next major)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (FLUME-3112) Upgrade jackson-core library dependency

[Attila Simon (JIRA)]

Attila Simon created FLUME-3112:
---

 Summary: Upgrade jackson-core library dependency
 Key: FLUME-3112
 URL: https://issues.apache.org/jira/browse/FLUME-3112
 Project: Flume
  Issue Type: Bug
Affects Versions: 1.7.0
Reporter: Attila Simon
Priority: Critical
 Fix For: 1.8.0


||Group||Artifact||Version used||Upgrade target||
|com.fasterxml.jackson.core|jackson-core|2.3.1|2.8.9|

Security vulnerability: http://www.cvedetails.com/cve/CVE-2016-7051/

Please do:
- double check the newest version. 
- consider to remove a dependency if better alternative is available.
- check whether the lib change would introduce a backward incompatibility (in 
which case please add this label `breaking_change` and fix version should be 
the next major)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Update 3rd party dependencies

[Attila Simon]

Hi,

My colleague made a through analysis on our library dependencies and
classified them using the following notation:


G = Good - we're on the latest version
M = Mostly good - we're only a patch version behind
O = Old - not on latest version, but on same major version
B = Bad - one or more major versions behind - you may want to strongly
consider upgrading
S = Security - potential security vulnerability reported against this or a
newer version
A = Abandonware - no longer supported in any form - these must be addressed
? = Needs further research


As a result I would like to create a jira for each upgrade to track the
Security vulnerability categories so please expect some noise from that
direction today and tomorrow. (If we can make a good progress there then we
can continue with the A,B,O,M categories later)


I'll somehow collect these into an epic or umbrella jira or label (if no
suggestion then I would pick one of them) including the existing library
upgrade jiras (eg FLUME-2914
<https://issues.apache.org/jira/browse/FLUME-2914>)

So the candidates for this iteration are the following. I think what we
should consider (will be part of the description of the newly created
jiras) is
- double check the existence of security vulnerability and
- double check the newest version.
- We might also want to consider to remove a dependency if better
alternative is available.
- check whether the lib change would introduce a backward incompatibility
(I think that would be marked as a label "breaking-change" and a fix
version for flume-ng 2.0.0)

Group Artifact Version used Version(s) available at search.maven.org
com.fasterxml.jackson.core jackson-core 2.3.1 2.8.1,
commons-beanutils commons-beanutils 1.7.0 1.9.2
commons-beanutils commons-beanutils-core 1.8.0 1.8.3,
commons-daemon commons-daemon 1.0.13 1.0.15
commons-httpclient commons-httpclient 3.1, 3.0.1 4.5.2
io.netty netty 3.2.2.Final, 3.9.4.Final 4.1.4
javax.mail mail 1.4.1 1.5.0-b01,
javax.servlet servlet-api 2.5 3.0-alpha-1, 2.5
javax.xml.bind jaxb-api 2.2.2 2.2.12,
org.apache.curator curator-framework 2.6.0 3.2.0,
org.apache.htrace htrace-core 3.1.0-incubating 4.0.0-incubating,
org.apache.httpcomponents httpclient 4.2.1 4.5.2,
org.apache.httpcomponents httpmime 4.2.5 4.5.2,
org.apache.james apache-mime4j-core 0.7.2 0.7.2,
org.apache.pdfbox fontbox 1.8.4 2.0.2,
org.apache.poi poi 3.10-beta2 3.15-beta2,
org.apache.poi poi-ooxml 3.10-beta2 3.15-beta2,
org.apache.poi poi-scratchpad 3.10-beta2 3.15-beta2,
org.apache.thrift libfb303 0.9.0 0.9.3,
org.bouncycastle bcprov-jdk15 1.45 1.46,
org.codehaus.jackson jackson-core-asl 1.9.3 1.9.13,
org.mortbay.jetty jetty 6.1.26 7.0.0.pre5,
org.mortbay.jetty jetty-util 6.1.26 7.0.0.pre5,
org.mortbay.jetty servlet-api 2.5-20110124 3.0.20100224,
org.restlet.jee org.restlet 2.1.1 2.3.4
org.springframework spring-aop 3.0.7.RELEASE 4.3.2.RELEASE,
org.springframework spring-context 3.0.7.RELEASE 4.3.2.RELEASE,
org.springframework spring-core 3.0.7.RELEASE 4.3.2.RELEASE,
tomcat jasper-compiler 5.5.23 5.5.23,
tomcat jasper-runtime 5.5.23 5.5.23,

Comments are very welcomed.

Cheers,
Attila


*Attila Simon*
Software Engineer
Email:   s...@cloudera.com

[image: Cloudera Inc.]

Re: Review Request 15956: Adding notes to Developer Guide upgrading Protocol Buffer version

[Attila Simon]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15956/#review179557
---


Ship it!




Ship It!

- Attila Simon


On Dec. 3, 2013, 4:08 a.m., Roshan Naik wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/15956/
> ---
> 
> (Updated Dec. 3, 2013, 4:08 a.m.)
> 
> 
> Review request for Flume.
> 
> 
> Bugs: FLUME-2175
> https://issues.apache.org/jira/browse/FLUME-2175
> 
> 
> Repository: flume-git
> 
> 
> Description
> ---
> 
> Adding notes to Developer Guide upgrading Protocol Buffer version
> 
> 
> Diffs
> -
> 
>   flume-ng-doc/sphinx/FlumeDeveloperGuide.rst 2be9c68 
> 
> 
> Diff: https://reviews.apache.org/r/15956/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Roshan Naik
> 
>

[jira] [Commented] (FLUME-2752) Flume AvroSource will leak the memory and the OOM will be happened.

[Attila Simon (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLUME-2752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16066342#comment-16066342
 ] 

Attila Simon commented on FLUME-2752:
-

I experienced similar issues on startup. If there is any exception during 
startup then the resources won't be released (memory, threadpool, open files, 
etc). I have an implementation which conforms with the proposed idea here: call 
this.stop() if there were any error during startup and make sure that 
this.stop() releases the resources. I plan to open a pull request for this 
change soon.

> Flume AvroSource will leak the memory and the OOM will be happened.
> ---
>
> Key: FLUME-2752
> URL: https://issues.apache.org/jira/browse/FLUME-2752
> Project: Flume
>  Issue Type: Bug
>  Components: Sinks+Sources
>Affects Versions: 1.6.0
>Reporter: yinghua_zh
>    Assignee: Attila Simon
>
> If the flume agent config the nonexist IP for the avro source,the exception 
> will be happened as follow:
> 2015-07-21 19:57:47,054 | ERROR | [lifecycleSupervisor-1-2] |  Unable to 
> start EventDrivenSourceRunner: { source:Avro source avro_source_21155: { 
> bindAddress: 51.196.27.32, port: 21155 } } - Exception follows.  | 
> org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:253)
> org.jboss.netty.channel.ChannelException: Failed to bind to: 
> /51.196.27.32:21155
>   at 
> org.jboss.netty.bootstrap.ServerBootstrap.bind(ServerBootstrap.java:297)
>   at org.apache.avro.ipc.NettyServer.(NettyServer.java:106)
>   at org.apache.flume.source.AvroSource.start(AvroSource.java:294)
>   at 
> org.apache.flume.source.EventDrivenSourceRunner.start(EventDrivenSourceRunner.java:44)
>   at 
> org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
>   at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>   at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: java.net.BindException: Cannot assign requested address
>   at sun.nio.ch.Net.bind0(Native Method)
>   at sun.nio.ch.Net.bind(Net.java:437)
>   at sun.nio.ch.Net.bind(Net.java:429)
>   at 
> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
>   at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
>   at 
> org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.bind(NioServerSocketPipelineSink.java:140)
>   at 
> org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.handleServerSocket(NioServerSocketPipelineSink.java:90)
>   at 
> org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.eventSunk(NioServerSocketPipelineSink.java:64)
>   at org.jboss.netty.channel.Channels.bind(Channels.java:569)
>   at 
> org.jboss.netty.channel.AbstractChannel.bind(AbstractChannel.java:189)
>   at 
> org.jboss.netty.bootstrap.ServerBootstrap$Binder.channelOpen(ServerBootstrap.java:342)
>   at org.jboss.netty.channel.Channels.fireChannelOpen(Channels.java:170)
>   at 
> org.jboss.netty.channel.socket.nio.NioServerSocketChannel.(NioServerSocketChannel.java:80)
>   at 
> org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.newChannel(NioServerSocketChannelFactory.java:158)
>   at 
> org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.newChannel(NioServerSocketChannelFactory.java:86)
>   at 
> org.jboss.netty.bootstrap.ServerBootstrap.bind(ServerBootstrap.java:276)
> if the above exception happened for 2 hours,and the agent JVM -Xxx is 4G,the 
> OutOfMemory will be happened.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (FLUME-2752) Flume AvroSource will leak the memory and the OOM will be happened.

[Attila Simon (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLUME-2752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Attila Simon updated FLUME-2752:

Summary: Flume AvroSource will leak the memory and the OOM will be 
happened.  (was: Flume AvroSoucr will leak the memory and the OOM will be 
happened.)

> Flume AvroSource will leak the memory and the OOM will be happened.
> ---
>
> Key: FLUME-2752
> URL: https://issues.apache.org/jira/browse/FLUME-2752
> Project: Flume
>  Issue Type: Bug
>  Components: Sinks+Sources
>Affects Versions: 1.6.0
>Reporter: yinghua_zh
>    Assignee: Attila Simon
>
> If the flume agent config the nonexist IP for the avro source,the exception 
> will be happened as follow:
> 2015-07-21 19:57:47,054 | ERROR | [lifecycleSupervisor-1-2] |  Unable to 
> start EventDrivenSourceRunner: { source:Avro source avro_source_21155: { 
> bindAddress: 51.196.27.32, port: 21155 } } - Exception follows.  | 
> org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:253)
> org.jboss.netty.channel.ChannelException: Failed to bind to: 
> /51.196.27.32:21155
>   at 
> org.jboss.netty.bootstrap.ServerBootstrap.bind(ServerBootstrap.java:297)
>   at org.apache.avro.ipc.NettyServer.(NettyServer.java:106)
>   at org.apache.flume.source.AvroSource.start(AvroSource.java:294)
>   at 
> org.apache.flume.source.EventDrivenSourceRunner.start(EventDrivenSourceRunner.java:44)
>   at 
> org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
>   at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>   at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: java.net.BindException: Cannot assign requested address
>   at sun.nio.ch.Net.bind0(Native Method)
>   at sun.nio.ch.Net.bind(Net.java:437)
>   at sun.nio.ch.Net.bind(Net.java:429)
>   at 
> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
>   at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
>   at 
> org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.bind(NioServerSocketPipelineSink.java:140)
>   at 
> org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.handleServerSocket(NioServerSocketPipelineSink.java:90)
>   at 
> org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.eventSunk(NioServerSocketPipelineSink.java:64)
>   at org.jboss.netty.channel.Channels.bind(Channels.java:569)
>   at 
> org.jboss.netty.channel.AbstractChannel.bind(AbstractChannel.java:189)
>   at 
> org.jboss.netty.bootstrap.ServerBootstrap$Binder.channelOpen(ServerBootstrap.java:342)
>   at org.jboss.netty.channel.Channels.fireChannelOpen(Channels.java:170)
>   at 
> org.jboss.netty.channel.socket.nio.NioServerSocketChannel.(NioServerSocketChannel.java:80)
>   at 
> org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.newChannel(NioServerSocketChannelFactory.java:158)
>   at 
> org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.newChannel(NioServerSocketChannelFactory.java:86)
>   at 
> org.jboss.netty.bootstrap.ServerBootstrap.bind(ServerBootstrap.java:276)
> if the above exception happened for 2 hours,and the agent JVM -Xxx is 4G,the 
> OutOfMemory will be happened.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (FLUME-2752) Flume AvroSoucr will leak the memory and the OOM will be happened.

[Attila Simon (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLUME-2752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Attila Simon reassigned FLUME-2752:
---

Assignee: Attila Simon

> Flume AvroSoucr will leak the memory and the OOM will be happened.
> --
>
> Key: FLUME-2752
> URL: https://issues.apache.org/jira/browse/FLUME-2752
> Project: Flume
>  Issue Type: Bug
>  Components: Sinks+Sources
>Affects Versions: 1.6.0
>Reporter: yinghua_zh
>    Assignee: Attila Simon
>
> If the flume agent config the nonexist IP for the avro source,the exception 
> will be happened as follow:
> 2015-07-21 19:57:47,054 | ERROR | [lifecycleSupervisor-1-2] |  Unable to 
> start EventDrivenSourceRunner: { source:Avro source avro_source_21155: { 
> bindAddress: 51.196.27.32, port: 21155 } } - Exception follows.  | 
> org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:253)
> org.jboss.netty.channel.ChannelException: Failed to bind to: 
> /51.196.27.32:21155
>   at 
> org.jboss.netty.bootstrap.ServerBootstrap.bind(ServerBootstrap.java:297)
>   at org.apache.avro.ipc.NettyServer.(NettyServer.java:106)
>   at org.apache.flume.source.AvroSource.start(AvroSource.java:294)
>   at 
> org.apache.flume.source.EventDrivenSourceRunner.start(EventDrivenSourceRunner.java:44)
>   at 
> org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
>   at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>   at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: java.net.BindException: Cannot assign requested address
>   at sun.nio.ch.Net.bind0(Native Method)
>   at sun.nio.ch.Net.bind(Net.java:437)
>   at sun.nio.ch.Net.bind(Net.java:429)
>   at 
> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
>   at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
>   at 
> org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.bind(NioServerSocketPipelineSink.java:140)
>   at 
> org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.handleServerSocket(NioServerSocketPipelineSink.java:90)
>   at 
> org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.eventSunk(NioServerSocketPipelineSink.java:64)
>   at org.jboss.netty.channel.Channels.bind(Channels.java:569)
>   at 
> org.jboss.netty.channel.AbstractChannel.bind(AbstractChannel.java:189)
>   at 
> org.jboss.netty.bootstrap.ServerBootstrap$Binder.channelOpen(ServerBootstrap.java:342)
>   at org.jboss.netty.channel.Channels.fireChannelOpen(Channels.java:170)
>   at 
> org.jboss.netty.channel.socket.nio.NioServerSocketChannel.(NioServerSocketChannel.java:80)
>   at 
> org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.newChannel(NioServerSocketChannelFactory.java:158)
>   at 
> org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.newChannel(NioServerSocketChannelFactory.java:86)
>   at 
> org.jboss.netty.bootstrap.ServerBootstrap.bind(ServerBootstrap.java:276)
> if the above exception happened for 2 hours,and the agent JVM -Xxx is 4G,the 
> OutOfMemory will be happened.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 60357: NetcatSource - Socket not closed when an exception is encountered during start() leading to file descriptor leaks

[Attila Simon]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60357/#review179084
---


Ship it!




Ship It!

- Attila Simon


On June 28, 2017, 4:58 a.m., Siddharth Ahuja wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60357/
> ---
> 
> (Updated June 28, 2017, 4:58 a.m.)
> 
> 
> Review request for Flume and Attila Simon.
> 
> 
> Repository: flume-git
> 
> 
> Description
> ---
> 
> Review request for: https://issues.apache.org/jira/browse/FLUME-2905 which is 
> trying to prevent socket leaks when a netcat port is already bound to an 
> existing process.
> 
> 
> Diffs
> -
> 
>   flume-ng-core/src/main/java/org/apache/flume/source/NetcatSource.java 
> 9513902 
>   flume-ng-core/src/test/java/org/apache/flume/source/TestNetcatSource.java 
> 99d413a 
> 
> 
> Diff: https://reviews.apache.org/r/60357/diff/2/
> 
> 
> Testing
> ---
> 
> I have tested flume-ng executable generated from my changes and I can confirm 
> from the lsof output that the sockets do not keep increasing if the port to 
> which netcat source is trying to bind to is already in use.
> The junits are also passing for me for the NetcatSource.
> 
> 
> Thanks,
> 
> Siddharth Ahuja
> 
>

[jira] [Commented] (FLUME-2905) NetcatSource - Socket not closed when an exception is encountered during start() leading to file descriptor leaks

[Attila Simon (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLUME-2905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16059612#comment-16059612
 ] 

Attila Simon commented on FLUME-2905:
-

It looks good, thanks! You can check others eg: 
https://reviews.apache.org/r/48161/

I also left some comments there (you can upload your corrections there, I guess 
attaching the final version of the patch here would be enough)

> NetcatSource - Socket not closed when an exception is encountered during 
> start() leading to file descriptor leaks
> -
>
> Key: FLUME-2905
> URL: https://issues.apache.org/jira/browse/FLUME-2905
> Project: Flume
>  Issue Type: Bug
>  Components: Sinks+Sources
>Affects Versions: 1.6.0
>Reporter: Siddharth Ahuja
>Assignee: Siddharth Ahuja
> Attachments: FLUME-2905-0.patch, FLUME-2905-1.patch, 
> FLUME-2905-2.patch, FLUME-2905-3.patch, FLUME-2905-4.patch, FLUME-2905-5.patch
>
>
> During the flume agent start-up, the flume configuration containing the 
> NetcatSource is parsed and the source's start() is called. If there is an 
> issue while binding the channel's socket to a local address to configure the 
> socket to listen for connections following exception is thrown but the socket 
> open just before is not closed. 
> {code}
> 2016-05-01 03:04:37,273 ERROR org.apache.flume.lifecycle.LifecycleSupervisor: 
> Unable to start EventDrivenSourceRunner: { 
> source:org.apache.flume.source.NetcatSource{name:src-1,state:IDLE} } - 
> Exception follows.
> org.apache.flume.FlumeException: java.net.BindException: Address already in 
> use
> at org.apache.flume.source.NetcatSource.start(NetcatSource.java:173)
> at 
> org.apache.flume.source.EventDrivenSourceRunner.start(EventDrivenSourceRunner.java:44)
> at 
> org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
> at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
> at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.net.BindException: Address already in use
> at sun.nio.ch.Net.bind0(Native Method)
> at sun.nio.ch.Net.bind(Net.java:444)
> at sun.nio.ch.Net.bind(Net.java:436)
> at 
> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)
> at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
> at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:67)
> at org.apache.flume.source.NetcatSource.start(NetcatSource.java:167)
> ... 9 more
> {code}
> The source's start() is then called again leading to another socket being 
> opened but not closed and so on. This leads to file descriptor (socket) leaks.
> This can be easily reproduced as follows:
> 1. Set Netcat as the source in flume agent configuration.
> 2. Set the bind port for the netcat source to a port which is already in use. 
> e.g. in my case I used 50010 which is the port for DataNode's XCeiver 
> Protocol in use by the HDFS service.
> 3. Start flume agent and perform "lsof -p  | wc -l". Notice 
> the file descriptors keep on growing due to socket leaks with errors like: 
> "can't identify protocol".



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 60357: NetcatSource - Socket not closed when an exception is encountered during start() leading to file descriptor leaks

[Attila Simon]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60357/#review178679
---




flume-ng-core/src/main/java/org/apache/flume/source/NetcatSource.java
Lines 176 (patched)
<https://reviews.apache.org/r/60357/#comment252825>

Please remove the unneccessary whitespaces (highlighted in red). Here and 
in the test class.



flume-ng-core/src/test/java/org/apache/flume/source/TestNetcatSource.java
Lines 322-324 (patched)
<https://reviews.apache.org/r/60357/#comment252849>

Nit: You can merge line #322 and #324 and use try-with-resources
```
try (ServerSocketChannel dummyServerSocket = ServerSocketChannel.open()) {
...
}
```
and remove the dummyServerSocket.close() from the end



flume-ng-core/src/test/java/org/apache/flume/source/TestNetcatSource.java
Lines 330 (patched)
<https://reviews.apache.org/r/60357/#comment252848>

I think you should avoid using the startSource() for this test as it would 
be responsible for selecting a free port (even though it is buggy and doesn't 
do that). This would be the replacement: 
```
Context context = new Context();
context.put("port", String.valueOf(PORT));
context.put("bind", "0.0.0.0");
context.put("ack-every-event", "false");
Configurables.configure(source, context);

source.start();
```


- Attila Simon


On June 22, 2017, 5:48 a.m., Siddharth Ahuja wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60357/
> ---
> 
> (Updated June 22, 2017, 5:48 a.m.)
> 
> 
> Review request for Flume and Attila Simon.
> 
> 
> Repository: flume-git
> 
> 
> Description
> ---
> 
> Review request for: https://issues.apache.org/jira/browse/FLUME-2905 which is 
> trying to prevent socket leaks when a netcat port is already bound to an 
> existing process.
> 
> 
> Diffs
> -
> 
>   flume-ng-core/src/main/java/org/apache/flume/source/NetcatSource.java 
> 9513902 
>   flume-ng-core/src/test/java/org/apache/flume/source/TestNetcatSource.java 
> 99d413a 
> 
> 
> Diff: https://reviews.apache.org/r/60357/diff/1/
> 
> 
> Testing
> ---
> 
> I have tested flume-ng executable generated from my changes and I can confirm 
> from the lsof output that the sockets do not keep increasing if the port to 
> which netcat source is trying to bind to is already in use.
> The junits are also passing for me for the NetcatSource.
> 
> 
> Thanks,
> 
> Siddharth Ahuja
> 
>

[jira] [Commented] (FLUME-2905) NetcatSource - Socket not closed when an exception is encountered during start() leading to file descriptor leaks

[Attila Simon (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLUME-2905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16058287#comment-16058287
 ] 

Attila Simon commented on FLUME-2905:
-

Hi [~sahuja],
Junit passed for me but hasn't run exhaustively yet. I checked the 
FLUME-2905-4.patch and would have some minor suggestions. Would you mind 
uploading to https://reviews.apache.org/ for a code review? Please let me know 
if you have trouble with it (I'm also happy to upload it there for you). 

In a nutshell:
* I would recommend calling stop() after writing out the exception. 
* I would recommend removing the return statement from that part of the stop 
function which catches an Exception from Socket close. That currently prevents 
the rest of the stop() to be executed including super.stop() which then can't 
set the Lifecycle state.
* Test currently checks that stop was executed properly expecting the source to 
be in LifecycleState.STOP state. I think that is fine since it is internal to 
the NetcatSource that it opens a ServerSocket. What we have to check whether 
the Source is stopped on error. And this is indeed checked by the test. 

As a side note: It seems like that startSource() doesn't behave as expected 
(should pick a new port on error). This is a separate issue so I guess it 
should be fixed with a separate jira/PR.

> NetcatSource - Socket not closed when an exception is encountered during 
> start() leading to file descriptor leaks
> -
>
> Key: FLUME-2905
> URL: https://issues.apache.org/jira/browse/FLUME-2905
> Project: Flume
>  Issue Type: Bug
>  Components: Sinks+Sources
>Affects Versions: 1.6.0
>Reporter: Siddharth Ahuja
>Assignee: Siddharth Ahuja
> Attachments: FLUME-2905-0.patch, FLUME-2905-1.patch, 
> FLUME-2905-2.patch, FLUME-2905-3.patch, FLUME-2905-4.patch
>
>
> During the flume agent start-up, the flume configuration containing the 
> NetcatSource is parsed and the source's start() is called. If there is an 
> issue while binding the channel's socket to a local address to configure the 
> socket to listen for connections following exception is thrown but the socket 
> open just before is not closed. 
> {code}
> 2016-05-01 03:04:37,273 ERROR org.apache.flume.lifecycle.LifecycleSupervisor: 
> Unable to start EventDrivenSourceRunner: { 
> source:org.apache.flume.source.NetcatSource{name:src-1,state:IDLE} } - 
> Exception follows.
> org.apache.flume.FlumeException: java.net.BindException: Address already in 
> use
> at org.apache.flume.source.NetcatSource.start(NetcatSource.java:173)
> at 
> org.apache.flume.source.EventDrivenSourceRunner.start(EventDrivenSourceRunner.java:44)
> at 
> org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
> at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
> at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.net.BindException: Address already in use
> at sun.nio.ch.Net.bind0(Native Method)
> at sun.nio.ch.Net.bind(Net.java:444)
> at sun.nio.ch.Net.bind(Net.java:436)
> at 
> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)
> at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
> at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:67)
> at org.apache.flume.source.NetcatSource.start(NetcatSource.java:167)
> ... 9 more
> {code}
> The source's start() is then called again leading to another socket being 
> opened but not closed and so on. This leads to file descriptor (socket) leaks.
> This can be easily reproduced as follows:
> 1. Set Netcat as the source in flume agent configuration.
> 2. Set the bind port for the netcat source to a port which is already in use. 
> e.g. in my case I used 50010 which is the port for DataNode's XCeiver 
> Protocol in use by the HDFS service.
> 3. Start flume agent and perform "lsof -p  | wc -l". Notice 
> the file descriptors keep on growing due to socket leaks with errors like: 
> "can't identify protocol".



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: [ANNOUNCE] New Flume committer - Denes Arvay

[Attila Simon]

Kudos to Denes! Well deserved!

Cheers,
Attila

On Mon, May 22, 2017 at 7:07 AM, Mike Percy  wrote:

> On behalf of the Apache Flume PMC, I am very pleased to welcome Denes Arvay
> as a committer on the Apache Flume project.
>
> Denes has put a lot of effort into improving the stability of Flume, most
> recently focusing on identifying and fixing serious and hard-to-diagnose
> issues including several bugs that could cause data loss.
>
> Congratulations and welcome, Denes!
>
> Best,
> Mike
>

Re: I want discuss something about flume

[Attila Simon]

Hi Lisheng Xia,

I like this feature! I would like to add the dev list to this conversation
so others can express their opinion as well. After all what community says
is what really matters. We can discuss there your proposal in detail as
well as whether there is a library which can help you in the unit
conversion eg javax.measure
<http://jscience.org/api/javax/measure/unit/package-summary.html>.

In my opinion it is appropriate :

   - to have a configuration property name with "byte" and still allowing
   to specify the value with units, if it is clear from the documentation what
   would that mean (eg please see the GiB vs GB definitions here
   <https://en.wikipedia.org/wiki/Gigabyte>).
   - extending this feature with time values (Hour,Minute,Second,Milisec...)
   - having the conversation failures instantly and clearly visible to the
   user by throwing an exception. I think "< Agent >.configuration.error =
   defaultValue/stopRunning" would be even better but much harder to
   implement.


Cheers,
Attila


On Sun, May 14, 2017 at 1:02 PM, 小夏 <iai...@163.com> wrote:

> Dear Attila Simon:
>  I use the flume in my work,when I was in the configuration of the
> flume, found that some of the parameters of configuration is very
> troublesome, and readability is very poor like the Memory Channel's
> byteCapacity ,File Channel's transactionCapacity and maxFileSize etc.,
> basic it is about the capacity configuration.Generally when I was in the
> configuration that, I need a special calculation of byte after
> transformation, such as 2g into 2147483648 <(214)%20748-3648>, and must
> be withing annotated, otherwise the next time I read, don't know this is 2g
> intuitive
> So I wrote a simple method used for converting readable capacity
> allocation into corresponding byte code is as follows.
> --
> public class Utils {
> private static final String REGULAR="((?\\d+(\\.\\d+)?
> )(g|G|gb|GB))?((?\\d+(\\.\\d+)?)(m|M|mb|MB))?((?\\d+
> (\\.\\d+)?)(k|K|kb|KB))?((?\\d+)(b|B|byte|BYTE)?)?";
> private static final int rate=1024;
> public static Long string2Bytes(String in){
> return string2Bytes(in,null);
> }
> public static Long string2Bytes(String in,Long defaultValue){
> if(in==null || in.trim().length()==0){
> return defaultValue;
> }
> in=in.trim();
> Pattern pattern = Pattern.compile(REGULAR);
> Matcher matcher = pattern.matcher(in);
> if(matcher.matches()){
> long bytes=0;
> String gb=matcher.group("gb");
> String mb=matcher.group("mb");
> String kb=matcher.group("kb");
> String b=matcher.group("b");
> if(gb!=null){
> bytes+=Math.round(Double.parseDouble(gb)*Math.pow(rate, 3));
> }
> if(mb!=null){
> bytes+=Math.round(Double.parseDouble(mb)*Math.pow(rate, 2));
> }
> if(kb!=null){
> bytes+=Math.round(Double.parseDouble(kb)*Math.pow(rate, 1));
> }
> if(b!=null){
> bytes+=Integer.parseInt(b);
> }
> return bytes;
> }else{
> throw new IllegalArgumentException("the param "+in+" is not a right");
> }
> }
> }
> Below is the test class
> @RunWith(Parameterized.class)
> public class UtilsTest {
> private String param;
> private Long result;
> public UtilsTest(String param,Long result){
> this.param=param;
> this.result=result;
> }
> @Parameters
> public static Collection<Object[]> data() {
> return Arrays.asList(new Object[][]{
> {"", null},
> {"  ", null},
> {"2g", 1L*2*1024*1024*1024},
> {"2G", 1L*2*1024*1024*1024},
> {"2gb", 1L*2*1024*1024*1024},
> {"2GB", 1L*2*1024*1024*1024},
> {"2000m", 1L*2000*1024*1024},
> {"2000mb", 1L*2000*1024*1024},
> {"2000M", 1L*2000*1024*1024},
> {"2000MB", 1L*2000*1024*1024},
> {"1000k", 1L*1000*1024},
> {"1000kb", 1L*1000*1024},
> {"1000K", 1L*1000*1024},
> {"1000KB", 1L*1000*1024},
> {"1000", 1L*1000},
> {"1.5GB", 1L*Math.round(1.5*1024*1024*1024)},
> {"1.38g", 1L*Math.round(1.38*1024*1024*1024)},
> {"1g500MB", 1L*1024*1024*1024+500*1024*1024},
> {"20MB512", 1L*20*1024*1024+512},
> {"0.5g", 1L*Math.round(0.5*1024*1024*1024)},
> {"0.5g0.5m", 1L*Math.round(0.

Re: [ANNOUNCE] Bessenyei Balázs Donát joining the Flume PMC

[Attila Simon]

Congrats Donat!


On Sat, Mar 11, 2017 at 11:35 PM, iain wright  wrote:

> Congrats! As a user -- thank you!
>
> -iain
>
> Sent from my iPhone
>
> > On Mar 11, 2017, at 2:21 PM, Bessenyei Balázs Donát 
> wrote:
> >
> > Hi Flume community and Mike,
> >
> > Thank you very much for this honour and opportunity.
> >
> > I'd like to say thank you to all who helped me along the way. Extra
> > kudos to Mike who has been a great mentor!
> >
> > Also, I'm very much looking forward to be contributing to Flume as a PMC
> member.
> >
> >
> > Donat
> >
> > 2017-03-11 10:09 GMT+01:00 Mike Percy :
> >>> On Mar 11, 2017, at 12:57 AM, Mike Percy  wrote:
> >>>
> >>> Donat was also instrumental to the latest release by acting as the
> release manager for Flume 1.6.0.
> >>
> >> Oops, I meant Flume 1.7.0 :)
> >>
> >> Mike
> >>
> >>
>

[jira] [Commented] (FLUME-3050) add error stats to monitor URL

[Attila Simon (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLUME-3050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15892039#comment-15892039
 ] 

Attila Simon commented on FLUME-3050:
-

Hi [~yuvalif], 

Thanks for your comments. Most of these metrics indeed look useful. I hope it 
will get picked soon. I think however picks it up can discuss further which 
ones are possible/makes sense and do the implementation based on that agreement.

> add error stats to monitor URL
> --
>
> Key: FLUME-3050
> URL: https://issues.apache.org/jira/browse/FLUME-3050
> Project: Flume
>  Issue Type: Improvement
>  Components: Channel, Shell, Sinks+Sources
>Affects Versions: v1.7.0
>Reporter: Yuval Lifshitz
>  Labels: features
>
> currently error counters are not present when getting stats. for example:
> {code}
>  > curl http://my-flume-host:4/metrics
> {"SINK.k1":{"ConnectionCreatedCount":"1","ConnectionClosedCount":"0","Type":"SINK","BatchCompleteCount":"0","BatchEmptyCount":"4","EventDrainAttemptCount":"10","StartTime":"1485348138992","EventDrainSuccessCount":"10","BatchUnderflowCount":"1","StopTime":"0","ConnectionFailedCount":"0"},"CHANNEL.c1":{"ChannelCapacity":"100","ChannelFillPercentage":"0.0","Type":"CHANNEL","ChannelSize":"0","EventTakeSuccessCount":"10","EventTakeAttemptCount":"15","StartTime":"1485348138990","EventPutAttemptCount":"10","EventPutSuccessCount":"10","StopTime":"0"},"SOURCE.r1":{"EventReceivedCount":"10","AppendBatchAcceptedCount":"0","Type":"SOURCE","AppendReceivedCount":"0","EventAcceptedCount":"10","StartTime":"1485348138993","AppendAcceptedCount":"0","OpenConnectionCount":"0","AppendBatchReceivedCount":"0","StopTime":"0"}}
> {code}
> return only "good" stats for source, channel and sink.
> to get error you need to look into the log file. this makes it hard to 
> integrate flume into automatic monitoring systems, NMS etc.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLUME-3050) add error stats to monitor URL

[Attila Simon (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLUME-3050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15878576#comment-15878576
 ] 

Attila Simon commented on FLUME-3050:
-

Hi [~yuvalif],

Could you please mention some example counters (or the full list would be even 
more helpful) you expect to see?

> add error stats to monitor URL
> --
>
> Key: FLUME-3050
> URL: https://issues.apache.org/jira/browse/FLUME-3050
> Project: Flume
>  Issue Type: Improvement
>  Components: Channel, Shell, Sinks+Sources
>Affects Versions: v1.7.0
>Reporter: Yuval Lifshitz
>  Labels: features
>
> currently error counters are not present when getting stats. for example:
> {code}
>  > curl http://my-flume-host:4/metrics
> {"SINK.k1":{"ConnectionCreatedCount":"1","ConnectionClosedCount":"0","Type":"SINK","BatchCompleteCount":"0","BatchEmptyCount":"4","EventDrainAttemptCount":"10","StartTime":"1485348138992","EventDrainSuccessCount":"10","BatchUnderflowCount":"1","StopTime":"0","ConnectionFailedCount":"0"},"CHANNEL.c1":{"ChannelCapacity":"100","ChannelFillPercentage":"0.0","Type":"CHANNEL","ChannelSize":"0","EventTakeSuccessCount":"10","EventTakeAttemptCount":"15","StartTime":"1485348138990","EventPutAttemptCount":"10","EventPutSuccessCount":"10","StopTime":"0"},"SOURCE.r1":{"EventReceivedCount":"10","AppendBatchAcceptedCount":"0","Type":"SOURCE","AppendReceivedCount":"0","EventAcceptedCount":"10","StartTime":"1485348138993","AppendAcceptedCount":"0","OpenConnectionCount":"0","AppendBatchReceivedCount":"0","StopTime":"0"}}
> {code}
> return only "good" stats for source, channel and sink.
> to get error you need to look into the log file. this makes it hard to 
> integrate flume into automatic monitoring systems, NMS etc.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: how to get log information dynamically from flume engine

[Attila Simon]

I know logback and log4j2 can do this. Flume is shipped with log4j 1.2 but
in theory nothing  prevents you to use a different backend for logging. It
is not guaranteed that it will work in practice, but it may worth a try.
http://logback.qos.ch/manual/configuration.html#autoScan
https://logging.apache.org/log4j/2.x/manual/configuration.html (search for
"automatic reconfiguration")

Laxman's finding is correct but it will possibly  add a code binding on a
particular logger backend. It has benefits and downsides so worth
discussing. I would rather investigate in upgrading to log4j2 because log4j
1.x reached end of life.

Cheers,
Attila


On 2016. Dec 31., Sat at 4:21, Laxman Ch  wrote:

> We may just need to change the flume's bootstrap class (Application.java)
>
> to do this. Please file a jira.
>
>
>
>
>
> On 30-Dec-2016 8:37 PM, "Naveenkumar"  wrote:
>
>
>
> Hi Laxman
>
> I have checked the documentation what you have suggested but the thing is I
>
> don't want to do flume source code modification because, according to the
>
> documentation we have to add method called configureandwatch in classes
>
> wherever we are using Log4j so ,if we go with this. then we need to change
>
> each and every class and above all internally flume source code uses slf4j
>
> framework that to with lig4j so,is there any other way to handle this out
>
> Regards
>
> Naveen
>
>
>
> Sent from my iPhone
>
>
>
> > On 30-Dec-2016, at 4:22 PM, Laxman Ch  wrote:
>
> >
>
> > Naveen,
>
> >
>
> > This is more of a log4j question and this can be done with configuration.
>
> > Please check log4j documentation.
>
> > https://logging.apache.org/log4j/1.2/faq.html#a3.6
>
> >
>
> > Flume uses log4j 1.2.
>
> >
>
> > --
>
> > Thanks,
>
> > Laxman
>
> >
>
> > On 30 December 2016 at 15:00, Naveenkumar Panchappanavar <
>
> nave...@gmail.com>
>
> > wrote:
>
> >
>
> >> Hi Team,
>
> >>
>
> >> I am try to get the dynamic log information from the flume engine *ie*:
>
> >> lets say i have started the flume agent and in meanwhile if i change the
>
> >> log level in log4j.properties it should dynamically take affect
> according
>
> >> to the log level which i configured without restarting the flume engine.
>
> >>
>
> >> can any please help regarding this.
>
> >>
>
> >> Regards
>
> >> Naveen.
>
> >>
>
>

[jira] [Commented] (FLUME-3032) taildir source sleeps frequently.

[Attila Simon (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLUME-3032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15767508#comment-15767508
 ] 

Attila Simon commented on FLUME-3032:
-

I would refactor {{tailFileProcess()}} to a separate class which would get its 
dependencies via constructor and define this function as part of its public 
API. That class could be tested as a unit by mocking its constructor params and 
see whether all the events was passed to the channel just by invoking the 
{{tailFileProcess()}} once. Mock which will be needed: 
{{getChannelProcessor().processEventBatch()}} should also modify the number of 
{{events}} beside count how many arrived (to imitate an interceptor which 
dropped one and also to verify that all events arrived). I think mocking the 
{{reader}} to produce {{events}} simply would be a convenient choice over 
setting up temporal files. I know this would be a relatively bigger change 
compared to your existing patch but would make an improvement on code quality. 
If you think you need help with it I'm happy to answer your questions or 
provide more details. 

Also another option would be to test this functionality via the public 
{{process()}} function by mocking all the relevant objects as written above 
plus the ones required by {{process()}} itself to work. 

There might be other options I guess every idea is welcomed.


Just double checked and you missed a space before '=' here: {{long 
receivedSize= 0;}}
and missed a semicolon here: {{receivedSize = events.size()}}

> taildir source sleeps frequently.
> -
>
> Key: FLUME-3032
> URL: https://issues.apache.org/jira/browse/FLUME-3032
> Project: Flume
>  Issue Type: Bug
>  Components: Sinks+Sources
>Affects Versions: v1.7.0
> Environment: CentOS Linux release 7.2.1511 (Core) 
> java version "1.7.0_80"
>Reporter: Liu Tianhao
>  Labels: newbie
> Attachments: FLUME-3032.patch
>
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> Test configuration.
> source - taildir
> interceptor -  The custom interceptor drops some events
> channel - anyone
> sink - none
> I found that taildir source sleeps frequently.
> The tailFileProcess() function in TaildirSource.java break the loop by 
> (events.size() < batchSize), but interceptor may change events.size().
> I think the events.size() should be used before interceptor processing. 
> Avoid unnecessary sleep.
> {code:title=TaildirSource.java|borderStyle=solid}
> private void tailFileProcess(TailFile tf, boolean backoffWithoutNL)
> throws IOException, InterruptedException {
> long receivedSize = 0;
> while (true) {
> reader.setCurrentFile(tf);
> List events = reader.readEvents(batchSize, 
> backoffWithoutNL);
> if (events.isEmpty()) {
> break;
> }
> receivedSize = events.size();
> sourceCounter.addToEventReceivedCount(receivedSize);
> sourceCounter.incrementAppendBatchReceivedCount();
> try {
> getChannelProcessor().processEventBatch(events);
> reader.commit();
> } catch (ChannelException ex) {
> logger.warn("The channel is full or unexpected failure. "
> + "The source will try again after " + retryInterval
> + " ms");
> TimeUnit.MILLISECONDS.sleep(retryInterval);
> retryInterval = retryInterval << 1;
> retryInterval = Math.min(retryInterval, maxRetryInterval);
> continue;
> }
> retryInterval = 1000;
> sourceCounter.addToEventAcceptedCount(events.size());
> sourceCounter.incrementAppendBatchAcceptedCount();
> if (receivedSize < batchSize) {
> break;
> }
> }
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLUME-2988) Kafka Sink metrics missing eventDrainAttemptCount

[Attila Simon (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLUME-2988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15726404#comment-15726404
 ] 

Attila Simon commented on FLUME-2988:
-

Hi [~udai],
Sorry for not getting back to you earlier. The change looks good with two small 
corrections:
 - as [~denes] pointed out Long.valueOf is not needed
 - I would recommend adding this to closer to the top of process() function eg: 
where we know for sure that we have a reference to an event from the 
channel.take() (== where flume first attempt to drain an event and knows that 
the channel was not empty). More specifically in an else path at 
https://github.com/apache/flume/blob/trunk/flume-ng-sinks/flume-ng-kafka-sink/src/main/java/org/apache/flume/sink/kafka/KafkaSink.java#L169
 

I think if you open a pull request for this on https://github.com/apache/flume 
would help the reviewing process.

> Kafka Sink metrics missing eventDrainAttemptCount
> -
>
> Key: FLUME-2988
> URL: https://issues.apache.org/jira/browse/FLUME-2988
> Project: Flume
>  Issue Type: Bug
>  Components: Sinks+Sources
>Reporter: Denes Arvay
>Assignee: Udai Kiran Potluri
>Priority: Minor
> Attachments: FLUME-2988.1.patch
>
>
> {{eventDrainAttemptCount}} doesn't get incremented in Kafka Sink, only the 
> {{eventDrainSuccessCount}} 
> (https://github.com/apache/flume/blob/trunk/flume-ng-sinks/flume-ng-kafka-sink/src/main/java/org/apache/flume/sink/kafka/KafkaSink.java#L210)
>  resulting in misleading statistics (e.g. when calculating {{attemptCount - 
> successCount}})



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLUME-3032) taildir source sleeps frequently.

[Attila Simon (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLUME-3032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15699547#comment-15699547
 ] 

Attila Simon commented on FLUME-3032:
-

Nice finding and the change also looks good to me. Would you consider adding a 
junit test to cover this change? 

> taildir source sleeps frequently.
> -
>
> Key: FLUME-3032
> URL: https://issues.apache.org/jira/browse/FLUME-3032
> Project: Flume
>  Issue Type: Bug
>  Components: Sinks+Sources
>Affects Versions: v1.7.0
> Environment: CentOS Linux release 7.2.1511 (Core) 
> java version "1.7.0_80"
>Reporter: Liu Tianhao
>  Labels: newbie
> Attachments: FLUME-3032.patch
>
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> Test configuration.
> source - taildir
> interceptor -  The custom interceptor drops some events
> channel - anyone
> sink - none
> I found that taildir source sleeps frequently.
> The tailFileProcess() function in TaildirSource.java break the loop by 
> (events.size() < batchSize), but interceptor may change events.size().
> I think the events.size() should be used before interceptor processing. 
> Avoid unnecessary sleep.
> {code:title=TaildirSource.java|borderStyle=solid}
> private void tailFileProcess(TailFile tf, boolean backoffWithoutNL)
> throws IOException, InterruptedException {
> long receivedSize = 0;
> while (true) {
> reader.setCurrentFile(tf);
> List events = reader.readEvents(batchSize, 
> backoffWithoutNL);
> if (events.isEmpty()) {
> break;
> }
> receivedSize = events.size();
> sourceCounter.addToEventReceivedCount(receivedSize);
> sourceCounter.incrementAppendBatchReceivedCount();
> try {
> getChannelProcessor().processEventBatch(events);
> reader.commit();
> } catch (ChannelException ex) {
> logger.warn("The channel is full or unexpected failure. "
> + "The source will try again after " + retryInterval
> + " ms");
> TimeUnit.MILLISECONDS.sleep(retryInterval);
> retryInterval = retryInterval << 1;
> retryInterval = Math.min(retryInterval, maxRetryInterval);
> continue;
> }
> retryInterval = 1000;
> sourceCounter.addToEventAcceptedCount(events.size());
> sourceCounter.incrementAppendBatchAcceptedCount();
> if (receivedSize < batchSize) {
> break;
> }
> }
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLUME-3031) sequence source should reset its counter for event body on channel exception

[Attila Simon (JIRA)]

Attila Simon created FLUME-3031:
---

 Summary: sequence source should reset its counter for event body 
on channel exception
 Key: FLUME-3031
 URL: https://issues.apache.org/jira/browse/FLUME-3031
 Project: Flume
  Issue Type: Bug
  Components: Sinks+Sources, Test
Affects Versions: v1.7.0
Reporter: Attila Simon
Assignee: Attila Simon


SequenceGeneratorSource uses a counter to construct the body of the generated 
Events which counter is not reseted when writing event to channel failed. This 
can lead to a situation that the total number of unique events at destination 
(if deduplication relies on msg body) is bigger than the totalEvents 
configuration parameter due to the retries.

Fix should make sure that number of events at destination after filtering out 
the duplicates is equal to the configured value.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 53231: FLUME-2857 Kafka Source/Channel/Sink does not restore default values when live update config

[Attila Simon]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53231/#review154111
---


Ship it!




LGTM

- Attila Simon


On Oct. 28, 2016, 10:24 a.m., Tristan Stevens wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/53231/
> ---
> 
> (Updated Oct. 28, 2016, 10:24 a.m.)
> 
> 
> Review request for Flume.
> 
> 
> Repository: flume-git
> 
> 
> Description
> ---
> 
> Change to fix error where Kafka Channel, Sink and Source sub-configurations 
> aren't tolerant of the configure() method being called more than once (as 
> happens when a Live Config Update happens)
> 
> 
> Diffs
> -
> 
>   
> flume-ng-channels/flume-kafka-channel/src/main/java/org/apache/flume/channel/kafka/KafkaChannel.java
>  47c0634 
>   
> flume-ng-channels/flume-kafka-channel/src/test/java/org/apache/flume/channel/kafka/TestKafkaChannel.java
>  276fee1 
>   
> flume-ng-sinks/flume-ng-kafka-sink/src/main/java/org/apache/flume/sink/kafka/KafkaSink.java
>  dd40224 
>   
> flume-ng-sinks/flume-ng-kafka-sink/src/test/java/org/apache/flume/sink/kafka/TestKafkaSink.java
>  7eccf76 
>   
> flume-ng-sources/flume-kafka-source/src/main/java/org/apache/flume/source/kafka/KafkaSource.java
>  195eca3 
>   
> flume-ng-sources/flume-kafka-source/src/test/java/org/apache/flume/source/kafka/TestKafkaSource.java
>  9554201 
> 
> Diff: https://reviews.apache.org/r/53231/diff/
> 
> 
> Testing
> ---
> 
> Added unit tests which all would have failed and now pass.
> Also tested manually for Channel and Sink (Producer settings only), Consumer 
> settings are a little harder as you don't get the verbose logs, however unit 
> tests pass.
> 
> 
> Thanks,
> 
> Tristan Stevens
> 
>

Re: Review Request 51802: FLUME-2976: Exception when JMS source tries to connect to a Weblogic server without authentication

[Attila Simon]

> On Oct. 27, 2016, 1:37 p.m., Attila Simon wrote:
> > flume-ng-sources/flume-jms-source/src/main/java/org/apache/flume/source/jms/JMSSource.java,
> >  line 127
> > <https://reviews.apache.org/r/51802/diff/1/?file=1496685#file1496685line127>
> >
> > Non breaking change as the password can still be specified as "". (eg 
> > by creating an empty password file with touch and specifying that in flume 
> > conf)

So the change itself looks good to me.


- Attila


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51802/#review153276
---


On Sept. 12, 2016, 1:08 p.m., Denes Arvay wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51802/
> ---
> 
> (Updated Sept. 12, 2016, 1:08 p.m.)
> 
> 
> Review request for Flume, Balázs Donát Bessenyei, Mike Percy, and Attila 
> Simon.
> 
> 
> Bugs: FLUME-2976
> https://issues.apache.org/jira/browse/FLUME-2976
> 
> 
> Repository: flume-git
> 
> 
> Description
> ---
> 
> If no `userName` and `passwordFile` is set for the JMS source it sets the 
> password to `Optional("")`. This leads to an exception in the weblogic jndi 
> context implementation when trying to connect to a weblogic jms server.
> 
> 
> Diffs
> -
> 
>   
> flume-ng-sources/flume-jms-source/src/main/java/org/apache/flume/source/jms/JMSSource.java
>  7631827 
>   
> flume-ng-sources/flume-jms-source/src/test/java/org/apache/flume/source/jms/TestIntegrationActiveMQ.java
>  53cc09a 
> 
> Diff: https://reviews.apache.org/r/51802/diff/
> 
> 
> Testing
> ---
> 
> - `flume-ng-sources/flume-jms-source` tests pass
> - `TestIntegrationActiveMQ` is now a parameterized test to test with and 
> without authentication.
> 
> 
> Thanks,
> 
> Denes Arvay
> 
>

Re: Review Request 51802: FLUME-2976: Exception when JMS source tries to connect to a Weblogic server without authentication

[Attila Simon]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51802/#review153276
---




flume-ng-sources/flume-jms-source/src/main/java/org/apache/flume/source/jms/JMSSource.java
 (line 127)
<https://reviews.apache.org/r/51802/#comment223529>

Non breaking change as the password can still be specified as "". (eg by 
creating an empty password file with touch and specifying that in flume conf)



flume-ng-sources/flume-jms-source/src/test/java/org/apache/flume/source/jms/TestIntegrationActiveMQ.java
 (lines 87 - 102)
<https://reviews.apache.org/r/51802/#comment222556>

Could you please use an Enum instead of boolean flags?


- Attila Simon


On Sept. 12, 2016, 1:08 p.m., Denes Arvay wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51802/
> ---
> 
> (Updated Sept. 12, 2016, 1:08 p.m.)
> 
> 
> Review request for Flume, Balázs Donát Bessenyei, Mike Percy, and Attila 
> Simon.
> 
> 
> Bugs: FLUME-2976
> https://issues.apache.org/jira/browse/FLUME-2976
> 
> 
> Repository: flume-git
> 
> 
> Description
> ---
> 
> If no `userName` and `passwordFile` is set for the JMS source it sets the 
> password to `Optional("")`. This leads to an exception in the weblogic jndi 
> context implementation when trying to connect to a weblogic jms server.
> 
> 
> Diffs
> -
> 
>   
> flume-ng-sources/flume-jms-source/src/main/java/org/apache/flume/source/jms/JMSSource.java
>  7631827 
>   
> flume-ng-sources/flume-jms-source/src/test/java/org/apache/flume/source/jms/TestIntegrationActiveMQ.java
>  53cc09a 
> 
> Diff: https://reviews.apache.org/r/51802/diff/
> 
> 
> Testing
> ---
> 
> - `flume-ng-sources/flume-jms-source` tests pass
> - `TestIntegrationActiveMQ` is now a parameterized test to test with and 
> without authentication.
> 
> 
> Thanks,
> 
> Denes Arvay
> 
>

Re: jira reference is missing from git commit messages

[Attila Simon]

Hi Mike,

I'm a big fan of having the site in scm so I guess I can volunteer
after Donat finishes that movement.
I have no strong opinion on whether we should have a jira or not (thus
the proposal of FLUME-PR70). I just "I found this habit very useful".
I have tooling which depends on the commit titles are unique with high
probability. Most likely these tools can be upgraded with some extra
effort.
Maybe having jira for each change is an overkill for small changes.
What do you think what can be considered as a small change? eg the
ones for which the "how to contribute" guide doesn't require review
board?

Cheers,
Attila

On Fri, Oct 21, 2016 at 12:54 PM, Mike Percy <mpe...@apache.org> wrote:
>
> Hi Attila,
> Thanks for raising this concern of yours. Please see inline.
>
> On Fri, Oct 21, 2016 at 12:17 PM, Attila Simon <s...@cloudera.com> wrote:
>
> > The how to contribute page asks to have a jira first for each change. I
> > guess with allowing pull requests we have to update the how to contribute
> > page as well (which only describes attaching patch and using review board).
> >
>
> Yes, it appeared that we had consensus for allowing PR on a couple separate
> dev@ threads a while back [1], [2].
>
> The Flume contributor docs are currently a bit stale. I have recently
> updated the How to Release cwiki page but if you want to volunteer to
> update some of the other docs that would be very welcome! Please let me
> know if you want to help and I can give you cwiki edit access if you send
> me your account id. Side note: Donat has recently proposed moving those
> docs from cwiki to the Git repo which I think would be a big improvement.
>
> I would like to ask committers to reestablish the habit of having a jira
> > for each commit and start the message with that jira.
> >
>
> Forcing people to file a JIRA when they are already doing a PR feels like
> pointless extra paperwork to me. There is certainly a place for JIRA in a
> software project, but I think that is to track unfixed bugs, ongoing tasks
> / projects, etc.
>
>
> > If we would like to relax the have jira for each change (for pull requests)
> > then I would suggest putting the request id as the first thing in the
> > commit message.
> >
>
> Why?
>
> If you click on this:
> https://github.com/apache/flume/commit/87d4c2c13862144eb578b211bcf800b2206834ff
>
> You will see that the text "Closes #70" creates a clickable hyperlink to
> the PR. Is this not sufficient for tracking purposes?
>
> Mike
>
> [1] https://s.apache.org/k31f
> [2] https://s.apache.org/Skm2

[jira] [Comment Edited] (FLUME-3015) Fix flaky org.apache.flume.channel.jdbc.TestJdbcChannelProviderNoFK.testEventWithSimulatedSourceAndSinks

[Attila Simon (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLUME-3015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15595616#comment-15595616
 ] 

Attila Simon edited comment on FLUME-3015 at 10/21/16 4:54 PM:
---

TestJdbcChannelProvider.testEventWithSimulatedSourceAndSinks might also be 
flaky. If you found the root cause for this ticket please double check whether 
it also applicable to 
TestJdbcChannelProvider.testEventWithSimulatedSourceAndSinks. Stack traces and 
exception is the similar to NoFK.


was (Author: sati):
TestJdbcChannelProvider.testEventWithSimulatedSourceAndSinks might also be 
flaky. If you found the root cause for this ticket please double check whether 
it also applicable to 
TestJdbcChannelProvider.testEventWithSimulatedSourceAndSinks

> Fix flaky 
> org.apache.flume.channel.jdbc.TestJdbcChannelProviderNoFK.testEventWithSimulatedSourceAndSinks
> 
>
> Key: FLUME-3015
> URL: https://issues.apache.org/jira/browse/FLUME-3015
> Project: Flume
>  Issue Type: Bug
>  Components: Test
>Affects Versions: v1.7.0
>    Reporter: Attila Simon
>
> testEventWithSimulatedSourceAndSinks(org.apache.flume.channel.jdbc.TestJdbcChannelProviderNoFK):
>  org.apache.flume.channel.jdbc.JdbcChannelException: Failed to persist event
> {noformat}
>  
> testEventWithSimulatedSourceAndSinks(org.apache.flume.channel.jdbc.TestJdbcChannelProviderNoFK)
>   Time elapsed: 310978 sec  <<< ERROR!
>  java.util.concurrent.ExecutionException: 
> org.apache.flume.channel.jdbc.JdbcChannelException: Failed to persist event
>   at java.util.concurrent.FutureTask.report(FutureTask.java:122)
>   at java.util.concurrent.FutureTask.get(FutureTask.java:188)
>   at 
> org.apache.flume.channel.jdbc.BaseJdbcChannelProviderTest.testEventWithSimulatedSourceAndSinks(BaseJdbcChannelProviderTest.java:202)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
>   at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
>   at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
>   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
>   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
>   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
>   at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at 
> org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
>   at 
> org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
>   at 
> org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.

[jira] [Created] (FLUME-3017) Fix flaky org.apache.flume.sink.hbase.TestHBaseSink

[Attila Simon (JIRA)]

Attila Simon created FLUME-3017:
---

 Summary: Fix flaky org.apache.flume.sink.hbase.TestHBaseSink
 Key: FLUME-3017
 URL: https://issues.apache.org/jira/browse/FLUME-3017
 Project: Flume
  Issue Type: Bug
  Components: Test
Affects Versions: v1.7.0
Reporter: Attila Simon


{noformat}
ERROR [main] hbase.MiniHBaseCluster (MiniHBaseCluster.java:init(230)) - Error 
starting cluster
java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/procedure2/Procedure
at java.lang.Class.getDeclaredConstructors0(Native Method)
at java.lang.Class.privateGetDeclaredConstructors(Class.java:2671)
at java.lang.Class.getConstructor0(Class.java:3075)
at java.lang.Class.getConstructor(Class.java:1825)
at 
org.apache.hadoop.hbase.util.JVMClusterUtil.createMasterThread(JVMClusterUtil.java:139)
at 
org.apache.hadoop.hbase.LocalHBaseCluster.addMaster(LocalHBaseCluster.java:219)
at 
org.apache.hadoop.hbase.LocalHBaseCluster.(LocalHBaseCluster.java:155)
at 
org.apache.hadoop.hbase.MiniHBaseCluster.init(MiniHBaseCluster.java:214)
at 
org.apache.hadoop.hbase.MiniHBaseCluster.(MiniHBaseCluster.java:94)
at 
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniHBaseCluster(HBaseTestingUtility.java:1036)
at 
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:996)
at 
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:868)
at 
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:862)
at 
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:806)
at 
org.apache.flume.sink.hbase.TestHBaseSink.setUpOnce(TestHBaseSink.java:81)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
at 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:27)
at 
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30)
at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
at 
org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
at 
org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
at 
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)
at 
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)
Caused by: java.lang.ClassNotFoundException: 
org.apache.hadoop.hbase.procedure2.Procedure
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 37 more
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLUME-3016) Fix flaky org.apache.flume.sink.hbase.TestAsyncHBaseSink

[Attila Simon (JIRA)]

Attila Simon created FLUME-3016:
---

 Summary: Fix flaky org.apache.flume.sink.hbase.TestAsyncHBaseSink 
 Key: FLUME-3016
 URL: https://issues.apache.org/jira/browse/FLUME-3016
 Project: Flume
  Issue Type: Bug
  Components: Test
Affects Versions: v1.7.0
Reporter: Attila Simon


{noformat}
2016-09-26 05:11:33,321 ERROR [main] hbase.MiniHBaseCluster 
(MiniHBaseCluster.java:init(230)) - Error starting cluster
java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/procedure2/Procedure
at java.lang.Class.getDeclaredConstructors0(Native Method)
at java.lang.Class.privateGetDeclaredConstructors(Class.java:2671)
at java.lang.Class.getConstructor0(Class.java:3075)
at java.lang.Class.getConstructor(Class.java:1825)
at 
org.apache.hadoop.hbase.util.JVMClusterUtil.createMasterThread(JVMClusterUtil.java:139)
at 
org.apache.hadoop.hbase.LocalHBaseCluster.addMaster(LocalHBaseCluster.java:219)
at 
org.apache.hadoop.hbase.LocalHBaseCluster.(LocalHBaseCluster.java:155)
at 
org.apache.hadoop.hbase.MiniHBaseCluster.init(MiniHBaseCluster.java:214)
at 
org.apache.hadoop.hbase.MiniHBaseCluster.(MiniHBaseCluster.java:94)
at 
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniHBaseCluster(HBaseTestingUtility.java:1036)
at 
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:996)
at 
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:868)
at 
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:862)
at 
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:806)
at 
org.apache.flume.sink.hbase.TestAsyncHBaseSink.setUp(TestAsyncHBaseSink.java:73)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
at 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:27)
at 
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30)
at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
at 
org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
at 
org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
at 
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)
at 
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)
Caused by: java.lang.ClassNotFoundException: 
org.apache.hadoop.hbase.procedure2.Procedure
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLUME-3015) Fix flaky org.apache.flume.channel.jdbc.TestJdbcChannelProviderNoFK.testEventWithSimulatedSourceAndSinks

[Attila Simon (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLUME-3015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15595616#comment-15595616
 ] 

Attila Simon commented on FLUME-3015:
-

TestJdbcChannelProvider.testEventWithSimulatedSourceAndSinks might also be 
flaky. If you found the root cause for this ticket please double check whether 
it also applicable to 
TestJdbcChannelProvider.testEventWithSimulatedSourceAndSinks

> Fix flaky 
> org.apache.flume.channel.jdbc.TestJdbcChannelProviderNoFK.testEventWithSimulatedSourceAndSinks
> 
>
> Key: FLUME-3015
> URL: https://issues.apache.org/jira/browse/FLUME-3015
> Project: Flume
>  Issue Type: Bug
>  Components: Test
>Affects Versions: v1.7.0
>    Reporter: Attila Simon
>
> testEventWithSimulatedSourceAndSinks(org.apache.flume.channel.jdbc.TestJdbcChannelProviderNoFK):
>  org.apache.flume.channel.jdbc.JdbcChannelException: Failed to persist event
> {noformat}
>  
> testEventWithSimulatedSourceAndSinks(org.apache.flume.channel.jdbc.TestJdbcChannelProviderNoFK)
>   Time elapsed: 310978 sec  <<< ERROR!
>  java.util.concurrent.ExecutionException: 
> org.apache.flume.channel.jdbc.JdbcChannelException: Failed to persist event
>   at java.util.concurrent.FutureTask.report(FutureTask.java:122)
>   at java.util.concurrent.FutureTask.get(FutureTask.java:188)
>   at 
> org.apache.flume.channel.jdbc.BaseJdbcChannelProviderTest.testEventWithSimulatedSourceAndSinks(BaseJdbcChannelProviderTest.java:202)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
>   at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
>   at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
>   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
>   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
>   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
>   at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at 
> org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
>   at 
> org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
>   at 
> org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)
>  Caused by: org.apache.flume.channel.jdbc.JdbcChannelException: Failed to 
> persist event
>   at 
> org.apache.flume.channel.jdbc.impl.JdbcChannelProviderImpl.persistEvent(JdbcChannelProviderImpl.java:312)
>   at 
> org.apache.flume.channel.jdbc.BaseJdbcChannelProvide

[jira] [Created] (FLUME-3015) Fix flaky org.apache.flume.channel.jdbc.TestJdbcChannelProviderNoFK.testEventWithSimulatedSourceAndSinks

[Attila Simon (JIRA)]

Attila Simon created FLUME-3015:
---

 Summary: Fix flaky 
org.apache.flume.channel.jdbc.TestJdbcChannelProviderNoFK.testEventWithSimulatedSourceAndSinks
 Key: FLUME-3015
 URL: https://issues.apache.org/jira/browse/FLUME-3015
 Project: Flume
  Issue Type: Bug
  Components: Test
Affects Versions: v1.7.0
Reporter: Attila Simon


testEventWithSimulatedSourceAndSinks(org.apache.flume.channel.jdbc.TestJdbcChannelProviderNoFK):
 org.apache.flume.channel.jdbc.JdbcChannelException: Failed to persist event

{noformat}
 
testEventWithSimulatedSourceAndSinks(org.apache.flume.channel.jdbc.TestJdbcChannelProviderNoFK)
  Time elapsed: 310978 sec  <<< ERROR!
 java.util.concurrent.ExecutionException: 
org.apache.flume.channel.jdbc.JdbcChannelException: Failed to persist event
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:188)
at 
org.apache.flume.channel.jdbc.BaseJdbcChannelProviderTest.testEventWithSimulatedSourceAndSinks(BaseJdbcChannelProviderTest.java:202)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
at 
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at 
org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
at 
org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
at 
org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
at 
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)
at 
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)
 Caused by: org.apache.flume.channel.jdbc.JdbcChannelException: Failed to 
persist event
at 
org.apache.flume.channel.jdbc.impl.JdbcChannelProviderImpl.persistEvent(JdbcChannelProviderImpl.java:312)
at 
org.apache.flume.channel.jdbc.BaseJdbcChannelProviderTest$MockSource.call(BaseJdbcChannelProviderTest.java:368)
at 
org.apache.flume.channel.jdbc.BaseJdbcChannelProviderTest$MockSource.call(BaseJdbcChannelProviderTest.java:345)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
 Caused by: org.apache.flume.channel.jdbc.JdbcChannelException: Unable to 
persist event: org.apache.flume.channel.jdbc.impl.PersistableEvent@4fbafc05
at 
org.apache.flume.channel.jdbc.impl.DerbySchemaHandler.storeEvent(DerbySchemaHandler.java:733)
at 
org.apache.flume.channel.jdbc.impl.JdbcChannelProviderImpl.persistEvent(JdbcChannelProviderImpl

[jira] [Created] (FLUME-3014) fix flaky test org.apache.flume.source.TestSyslogUdpSource.testSourceCounter

[Attila Simon (JIRA)]

Attila Simon created FLUME-3014:
---

 Summary: fix flaky test 
org.apache.flume.source.TestSyslogUdpSource.testSourceCounter
 Key: FLUME-3014
 URL: https://issues.apache.org/jira/browse/FLUME-3014
 Project: Flume
  Issue Type: Bug
  Components: Test
Affects Versions: v1.7.0
Reporter: Attila Simon


{noformat}
java.lang.AssertionError: expected:<1> but was:<0>
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.failNotEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:128)
at org.junit.Assert.assertEquals(Assert.java:472)
at org.junit.Assert.assertEquals(Assert.java:456)
at 
org.apache.flume.source.TestSyslogUdpSource.testSourceCounter(TestSyslogUdpSource.java:205)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
at 
org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
at 
org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
at 
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)
at 
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLUME-3013) Fix flaky testShutdown(org.apache.flume.source.TestExecSource)

[Attila Simon (JIRA)]

Attila Simon created FLUME-3013:
---

 Summary: Fix flaky 
testShutdown(org.apache.flume.source.TestExecSource)
 Key: FLUME-3013
 URL: https://issues.apache.org/jira/browse/FLUME-3013
 Project: Flume
  Issue Type: Bug
  Components: Test
Affects Versions: v1.7.0
Reporter: Attila Simon


{noformat}
java.lang.IllegalStateException: Command [ps -ef] exited with 143
at org.apache.flume.source.TestExecSource.exec(TestExecSource.java:460)
at
org.apache.flume.source.TestExecSource.testShutdown(TestExecSource.java:406)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
at
org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
at
org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
at
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)

testShutdown(org.apache.flume.source.TestExecSource)  Time elapsed: 621 sec
 <<< ERROR!
java.lang.NullPointerException
at org.apache.flume.source.ExecSource.stop(ExecSource.java:202)
at org.apache.flume.source.TestExecSource.tearDown(TestExecSource.java:83)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:36)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImp

Re: Review Request 52510: Some tests in TestBucketWriter are flaky

[Attila Simon]

> On Oct. 21, 2016, 1:21 p.m., Attila Simon wrote:
> > Ship It!

Discussed offline that there will be a follow up patch to use builder pattern 
for constructing BucketWriter. (So that further testing only instance mutators 
can be eliminated)


- Attila


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52510/#review153538
---


On Oct. 4, 2016, 11:11 a.m., Balázs Donát Bessenyei wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/52510/
> ---
> 
> (Updated Oct. 4, 2016, 11:11 a.m.)
> 
> 
> Review request for Flume.
> 
> 
> Bugs: FLUME-3002
> https://issues.apache.org/jira/browse/FLUME-3002
> 
> 
> Repository: flume-git
> 
> 
> Description
> ---
> 
> testFileSuffixNotGiven (and probably a few other tests) are flaky
> 
> 
> Diffs
> -
> 
>   
> c/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java
>  b096410 
>   
> c/flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestBucketWriter.java
>  742deb0 
> 
> Diff: https://reviews.apache.org/r/52510/diff/
> 
> 
> Testing
> ---
> 
> mvn clean install runs successfully in flume-ng-sinks
> 
> 
> Thanks,
> 
> Balázs Donát Bessenyei
> 
>

Re: Review Request 52510: Some tests in TestBucketWriter are flaky

[Attila Simon]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52510/#review153538
---


Ship it!




Ship It!

- Attila Simon


On Oct. 4, 2016, 11:11 a.m., Balázs Donát Bessenyei wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/52510/
> ---
> 
> (Updated Oct. 4, 2016, 11:11 a.m.)
> 
> 
> Review request for Flume.
> 
> 
> Bugs: FLUME-3002
> https://issues.apache.org/jira/browse/FLUME-3002
> 
> 
> Repository: flume-git
> 
> 
> Description
> ---
> 
> testFileSuffixNotGiven (and probably a few other tests) are flaky
> 
> 
> Diffs
> -
> 
>   
> c/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java
>  b096410 
>   
> c/flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestBucketWriter.java
>  742deb0 
> 
> Diff: https://reviews.apache.org/r/52510/diff/
> 
> 
> Testing
> ---
> 
> mvn clean install runs successfully in flume-ng-sinks
> 
> 
> Thanks,
> 
> Balázs Donát Bessenyei
> 
>

Re: Review Request 53056: FLUME-2945: Bump java target version to 1.8

[Attila Simon]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53056/#review153527
---



Hi Lior,
conf/flume-env.sh.template#22 is still referenceing to java-6-sun. Could you 
please include that as well.

- Attila Simon


On Oct. 20, 2016, 4:50 p.m., Lior Zeno wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/53056/
> ---
> 
> (Updated Oct. 20, 2016, 4:50 p.m.)
> 
> 
> Review request for Flume.
> 
> 
> Repository: flume-git
> 
> 
> Description
> ---
> 
> We should move to Java 8 as a minimum requirement.
> 
> 1. Java 7 is EOL'ed http://www.oracle.com/technetwork/java/eol-135779.html.
> 2. Many projects are adopting java 8 as a minimum requirement, for instance:
> 
> * Solr 6: https://issues.apache.org/jira/browse/LUCENE-6722
> * Hbase 2: https://issues.apache.org/jira/browse/HBASE-15624
> * elasticsearch 5: 
> https://www.elastic.co/guide/en/elasticsearch/reference/master/setup.html
> 
> 
> Diffs
> -
> 
>   README.md 9ebb2a3 
>   flume-ng-doc/sphinx/FlumeUserGuide.rst cd76bb3 
>   flume-ng-sources/flume-taildir-source/pom.xml 96c2468 
>   pom.xml f62c99a 
> 
> Diff: https://reviews.apache.org/r/53056/diff/
> 
> 
> Testing
> ---
> 
> The attached patch was tested on Ubuntu 16.04 with openjdk version "1.8.0_91".
> All unit tests, except for the known flaky ones, successfully passed.
> 
> 
> Thanks,
> 
> Lior Zeno
> 
>

Re: Review Request 52510: Some tests in TestBucketWriter are flaky

[Attila Simon]

> On Oct. 18, 2016, 4:26 p.m., Attila Simon wrote:
> > Ahh now I see what is going on. BucketWriter instantiate a Clock class 
> > which is in turn used in the file name counter. The test first instantiates 
> > the BucketWriter - which initialize the filename counter -then overrides 
> > the Clock but since Clock is not used in BucketWriter directly - only via 
> > the file name counter at instantiation time so it won't be updated with the 
> > override - the test failes as it checks the file name.
> > So actually a single extra line into the `BucketWriter.setClock(Clock 
> > clock)` method would solve the issue:
> > ```
> >   void setClock(Clock clock) {
> > this.clock = clock;
> > this.fileExtensionCounter.set(clock.currentTimeMillis());
> >   }
> > ```
> > This way the test would update the relevant information - file name counter 
> > - which then in turn can be checked.
> > 
> > Could you please consider simplifying your change?
> 
> Balázs Donát Bessenyei wrote:
> Thank you for the review!
> 
> Even though the change you have suggested does fix the flakiness of the 
> tests (which is awesome!), I am not sure people would expect the caused 
> side-effect in void setClock(Clock clock) -> fileExtensionCounter.
> 
> The change in its current form does eliminate private Clock clock which 
> is "nice". However, it also eliminates void setClock(Clock clock) which helps 
> avoiding violation of the monotony of fileExtensionCounter (which is 
> assumed), thus it improves the integrity of BucketWriter (encapsulation).
> 
> Please, let me know your thoughts.

Clock is only used to initialize file extension counter (so there is 
initialization phase binding between these two). I just made sure this expected 
binding is maintained in the setter as well through the lifetime of a 
BucketWriter. I may suggest setClock should be also tagged with 
@VisibleForTesting (it is already scoped with package auto) to make it clear 
that it is only for tests (which is the case now by checking its usages).


- Attila


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52510/#review153104
---


On Oct. 4, 2016, 11:11 a.m., Balázs Donát Bessenyei wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/52510/
> ---
> 
> (Updated Oct. 4, 2016, 11:11 a.m.)
> 
> 
> Review request for Flume.
> 
> 
> Bugs: FLUME-3002
> https://issues.apache.org/jira/browse/FLUME-3002
> 
> 
> Repository: flume-git
> 
> 
> Description
> ---
> 
> testFileSuffixNotGiven (and probably a few other tests) are flaky
> 
> 
> Diffs
> -
> 
>   
> c/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java
>  b096410 
>   
> c/flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestBucketWriter.java
>  742deb0 
> 
> Diff: https://reviews.apache.org/r/52510/diff/
> 
> 
> Testing
> ---
> 
> mvn clean install runs successfully in flume-ng-sinks
> 
> 
> Thanks,
> 
> Balázs Donát Bessenyei
> 
>

jira reference is missing from git commit messages

[Attila Simon]

Hi,

Not too long ago all the messages in the git log started with the
corresponding jira. I found this habit very useful.
The how to contribute page asks to have a jira first for each change. I
guess with allowing pull requests we have to update the how to contribute
page as well (which only describes attaching patch and using review board).

I would like to ask committers to reestablish the habit of having a jira
for each commit and start the message with that jira.
If we would like to relax the have jira for each change (for pull requests)
then I would suggest putting the request id as the first thing in the
commit message.

instead of:
Author: Mike Percy 
Date:   Wed Oct 12 19:32:13 2016 +0200

Remove test dependencies from binary release artifact

This patch removes some test-specific dependencies from the binary
release artifact. These were introduced by the new
flume-shared-kafka-test module that is intended for sharing test code.
Please see the new comment in bin.xml for more information.

Reviewers: Bessenyei Balázs Donát

Closes #70


I would suggest:
Author: Mike Percy 
Date:   Wed Oct 12 19:32:13 2016 +0200

FLUME- Remove test dependencies from binary release artifact

This patch removes some test-specific dependencies from the binary
release artifact. These were introduced by the new
flume-shared-kafka-test module that is intended for sharing test code.
Please see the new comment in bin.xml for more information.

Reviewers: Bessenyei Balázs Donát

Closes #70

or the relaxed version if it is really needed to drop the jira requirement:
Author: Mike Percy 
Date:   Wed Oct 12 19:32:13 2016 +0200

FLUME-PR70 Remove test dependencies from binary release artifact

This patch removes some test-specific dependencies from the binary
release artifact. These were introduced by the new
flume-shared-kafka-test module that is intended for sharing test code.
Please see the new comment in bin.xml for more information.

Reviewers: Bessenyei Balázs Donát

Closes #70


Cheers,
Attila

Re: Moving Developer Section of cwiki to the git repository

[Attila Simon]

+1
Essentially moving whole site (all web content) to scm would help
contribution.

Cheers,
Attila

On Tuesday, 18 October 2016, Balazs Donat Bessenyei <bes...@cloudera.com>
wrote:

> Hi All,
>
> As it's kind of difficult to get permissions in the wiki to edit pages
> like https://cwiki.apache.org/confluence/display/FLUME/How+to+Contribute
> , I suggest moving contents to the git repository into files like
> CONTRIBUTING.md, etc.
>
> I'd be happy to create the files and move the current texts.
>
> Please, let me know your thoughts.
>
>
> Thank you,
>
> Donat
>


-- 

*Attila Simon*
Software Engineer
Email:   s...@cloudera.com

[image: Cloudera Inc.]

Re: Migrate to Log4j 2

[Attila Simon]

Hi Lior,

Could you please explain a bit what will break?

Cheers,
Attila

On Tuesday, 18 October 2016, Lior Zeno <liorz...@gmail.com> wrote:

> Hi All,
>
> Log4j (v1) has been EOL'ed over a year now (
> https://blogs.apache.org/foundation/entry/apache_logging_services_project_
> announces)
> and is no longer officially supported.
>
> I propose we migrate to Log4j 2. We can begin with using the Log4j 1.x
> bridge, and then incrementally move the whole codebase.
>
> Since this is a breaking change, I think we should schedule this to Flume
> 2.0.0.
>
> Thoughts?
>
> Thanks
>


-- 

*Attila Simon*
Software Engineer
Email:   s...@cloudera.com

[image: Cloudera Inc.]

Re: [ANNOUNCE] Apache Flume 1.7.0 released

[Attila Simon]

Hi,

Congratulations for all of those who made this
possible!

Cheers,
Attila

On Wednesday, 19 October 2016, Hari Shreedharan <hshreedha...@apache.org>
wrote:

> Thanks Donat and Mike! Great work!
>
> On Wed, Oct 19, 2016 at 3:06 AM, Mike Percy <mpe...@apache.org
> <javascript:;>> wrote:
> > Thank you for all your hard work RMing this release Donat and getting it
> to
> > the finish line. It was my pleasure to help out!
> >
> > Best,
> > Mike
> >
> > On Tue, Oct 18, 2016 at 1:32 PM, Balazs Donat Bessenyei <
> bes...@cloudera.com <javascript:;>
> >> wrote:
> >
> >> And thank you, Mike Percy for the mentoring and tremendous amounts of
> >> assistance with the release!
> >>
> >> On Tue, Oct 18, 2016 at 12:46 PM, Balazs Donat Bessenyei
> >> <bes...@cloudera.com <javascript:;>> wrote:
> >> > Thank you all very much who participated and helped with the release!
> >> >
> >> >
> >> > Donat
> >> >
> >> >
> >> > On Tue, Oct 18, 2016 at 12:37 PM, Mike Percy <mpe...@apache.org
> <javascript:;>> wrote:
> >> >> Woot! Congrats everyone!
> >> >>
> >> >> Thanks Donat for working so hard to get this version of Flume out the
> >> door!
> >> >>
> >> >> Best,
> >> >> Mike
> >> >>
> >> >>
> >> >> On Tue, Oct 18, 2016 at 10:09 AM, Bessenyei Balázs Donát <
> >> bes...@apache.org <javascript:;>>
> >> >> wrote:
> >> >>>
> >> >>> The Apache Flume team is pleased to announce the release of Flume
> >> >>> version 1.7.0.
> >> >>>
> >> >>> Flume is a distributed, reliable, and available service for
> efficiently
> >> >>> collecting, aggregating, and moving large amounts of log data.
> >> >>>
> >> >>> This release can be downloaded from the Flume download page at:
> >> >>> http://flume.apache.org/download.html
> >> >>>
> >> >>> The change log and documentation are available on the 1.7.0 release
> >> page:
> >> >>> http://flume.apache.org/releases/1.7.0.html
> >> >>>
> >> >>> Your help and feedback is more than welcome. For more information on
> >> how
> >> >>> to report problems and to get involved, visit the project website at
> >> >>> http://flume.apache.org/
> >> >>>
> >> >>> The Apache Flume Team
> >> >>
> >> >>
> >>
>


-- 

*Attila Simon*
Software Engineer
Email:   s...@cloudera.com

[image: Cloudera Inc.]

Re: Review Request 51244: FLUME-2171: Add Interceptor to remove headers from event

[Attila Simon]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51244/#review153273
---




w/flume-ng-core/src/main/java/org/apache/flume/interceptor/RemoveHeaderInterceptor.java
 (lines 119 - 138)
<https://reviews.apache.org/r/51244/#comment222554>

wasRemoved is not needed here. if removedHeaders list in not empty then 
there were a removal.


Otherwise it looks good to me.

- Attila Simon


On Aug. 24, 2016, 9:08 p.m., Balázs Donát Bessenyei wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51244/
> ---
> 
> (Updated Aug. 24, 2016, 9:08 p.m.)
> 
> 
> Review request for Flume.
> 
> 
> Bugs: FLUME-2171
> https://issues.apache.org/jira/browse/FLUME-2171
> 
> 
> Repository: flume-git
> 
> 
> Description
> ---
> 
> I found Flume OG's decorators to handle event headers useful and some to be 
> missing from Flume NG. More specifically, we could have an interceptor to 
> remove headers from an event.
> 
> 
> Diffs
> -
> 
>   
> c/flume-ng-core/src/main/java/org/apache/flume/interceptor/InterceptorType.java
>  fe341e9 
>   c/flume-ng-doc/sphinx/FlumeUserGuide.rst 5e677c6 
>   
> w/flume-ng-core/src/main/java/org/apache/flume/interceptor/RemoveHeaderInterceptor.java
>  PRE-CREATION 
>   
> w/flume-ng-core/src/test/java/org/apache/flume/interceptor/RemoveHeaderInterceptorTest.java
>  PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/51244/diff/
> 
> 
> Testing
> ---
> 
> All tests (besides the FLUME-2974-related ones) in flume-ng-core run 
> successfully
> 
> I've used this config for manual testing:
> 
> a1.sources = r1
> a1.sources.r1.type = netcat
> a1.sources.r1.bind = 0.0.0.0
> a1.sources.r1.port = 
> a1.sources.r1.channels = c1
> 
> a1.channels = c1
> a1.channels.c1.type = memory
> a1.channels.c1.capacity = 1
> a1.channels.c1.transactionCapacity = 1
> a1.channels.c1.byteCapacityBufferPercentage = 20
> a1.channels.c1.byteCapacity = 80
> 
> a1.channels = c1
> a1.sinks = k1
> a1.sinks.k1.type = logger
> a1.sinks.k1.channel = c1
> 
> 
> a1.sources.r1.interceptors = i1 i2 i3
> a1.sources.r1.interceptors.i1.type = timestamp
> 
> a1.sources.r1.interceptors.i2.type = host
> a1.sources.r1.interceptors.i2.hostHeader = hostname
> 
> a1.sources.r1.interceptors.i3.type = remove_header
> a1.sources.r1.interceptors.i3.with.name = timestamp
> 
> 
> Thanks,
> 
> Balázs Donát Bessenyei
> 
>

Re: Review Request 51244: FLUME-2171: Add Interceptor to remove headers from event

[Attila Simon]

> On Aug. 24, 2016, 4:30 p.m., Attila Simon wrote:
> > w/flume-ng-core/src/main/java/org/apache/flume/interceptor/RemoveHeaderInterceptor.java,
> >  line 139
> > <https://reviews.apache.org/r/51244/diff/6/?file=1482094#file1482094line139>
> >
> > Please don't log out the whole Event for each removed header. Also 
> > please collect all the headers which were removed and print them out as a 
> > single log entry per Event.
> 
> Balázs Donát Bessenyei wrote:
> I think there is a benefit to keeping the event in the log, but I changed 
> the code to collect the headers for logging
> 
> If you think the whole event logging is a blocking issue, please re-open

Fair enough. Then could you please consider wrapping the logging of event with 
a org.apache.flume.conf.LogPrivacyUtil#allowLogRawData() check?


- Attila


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51244/#review146665
---


On Aug. 24, 2016, 9:08 p.m., Balázs Donát Bessenyei wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51244/
> ---
> 
> (Updated Aug. 24, 2016, 9:08 p.m.)
> 
> 
> Review request for Flume.
> 
> 
> Bugs: FLUME-2171
> https://issues.apache.org/jira/browse/FLUME-2171
> 
> 
> Repository: flume-git
> 
> 
> Description
> ---
> 
> I found Flume OG's decorators to handle event headers useful and some to be 
> missing from Flume NG. More specifically, we could have an interceptor to 
> remove headers from an event.
> 
> 
> Diffs
> -
> 
>   
> c/flume-ng-core/src/main/java/org/apache/flume/interceptor/InterceptorType.java
>  fe341e9 
>   c/flume-ng-doc/sphinx/FlumeUserGuide.rst 5e677c6 
>   
> w/flume-ng-core/src/main/java/org/apache/flume/interceptor/RemoveHeaderInterceptor.java
>  PRE-CREATION 
>   
> w/flume-ng-core/src/test/java/org/apache/flume/interceptor/RemoveHeaderInterceptorTest.java
>  PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/51244/diff/
> 
> 
> Testing
> ---
> 
> All tests (besides the FLUME-2974-related ones) in flume-ng-core run 
> successfully
> 
> I've used this config for manual testing:
> 
> a1.sources = r1
> a1.sources.r1.type = netcat
> a1.sources.r1.bind = 0.0.0.0
> a1.sources.r1.port = 
> a1.sources.r1.channels = c1
> 
> a1.channels = c1
> a1.channels.c1.type = memory
> a1.channels.c1.capacity = 1
> a1.channels.c1.transactionCapacity = 1
> a1.channels.c1.byteCapacityBufferPercentage = 20
> a1.channels.c1.byteCapacity = 80
> 
> a1.channels = c1
> a1.sinks = k1
> a1.sinks.k1.type = logger
> a1.sinks.k1.channel = c1
> 
> 
> a1.sources.r1.interceptors = i1 i2 i3
> a1.sources.r1.interceptors.i1.type = timestamp
> 
> a1.sources.r1.interceptors.i2.type = host
> a1.sources.r1.interceptors.i2.hostHeader = hostname
> 
> a1.sources.r1.interceptors.i3.type = remove_header
> a1.sources.r1.interceptors.i3.with.name = timestamp
> 
> 
> Thanks,
> 
> Balázs Donát Bessenyei
> 
>

Re: Review Request 52510: Some tests in TestBucketWriter are flaky

[Attila Simon]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52510/#review153104
---



Ahh now I see what is going on. BucketWriter instantiate a Clock class which is 
in turn used in the file name counter. The test first instantiates the 
BucketWriter - which initialize the filename counter -then overrides the Clock 
but since Clock is not used in BucketWriter directly - only via the file name 
counter at instantiation time so it won't be updated with the override - the 
test failes as it checks the file name.
So actually a single extra line into the `BucketWriter.setClock(Clock clock)` 
method would solve the issue:
```
  void setClock(Clock clock) {
this.clock = clock;
this.fileExtensionCounter.set(clock.currentTimeMillis());
  }
```
This way the test would update the relevant information - file name counter - 
which then in turn can be checked.

Could you please consider simplifying your change?

- Attila Simon


On Oct. 4, 2016, 11:11 a.m., Balázs Donát Bessenyei wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/52510/
> ---
> 
> (Updated Oct. 4, 2016, 11:11 a.m.)
> 
> 
> Review request for Flume.
> 
> 
> Bugs: FLUME-3002
> https://issues.apache.org/jira/browse/FLUME-3002
> 
> 
> Repository: flume-git
> 
> 
> Description
> ---
> 
> testFileSuffixNotGiven (and probably a few other tests) are flaky
> 
> 
> Diffs
> -
> 
>   
> c/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java
>  b096410 
>   
> c/flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestBucketWriter.java
>  742deb0 
> 
> Diff: https://reviews.apache.org/r/52510/diff/
> 
> 
> Testing
> ---
> 
> mvn clean install runs successfully in flume-ng-sinks
> 
> 
> Thanks,
> 
> Balázs Donát Bessenyei
> 
>

Re: Review Request 52510: Some tests in TestBucketWriter are flaky

[Attila Simon]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52510/#review153076
---



Could you please give us some details about the idea of this change? Ie some 
high level description about what was the problem and how you solved it.

- Attila Simon


On Oct. 4, 2016, 11:11 a.m., Balázs Donát Bessenyei wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/52510/
> ---
> 
> (Updated Oct. 4, 2016, 11:11 a.m.)
> 
> 
> Review request for Flume.
> 
> 
> Bugs: FLUME-3002
> https://issues.apache.org/jira/browse/FLUME-3002
> 
> 
> Repository: flume-git
> 
> 
> Description
> ---
> 
> testFileSuffixNotGiven (and probably a few other tests) are flaky
> 
> 
> Diffs
> -
> 
>   
> c/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java
>  b096410 
>   
> c/flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestBucketWriter.java
>  742deb0 
> 
> Diff: https://reviews.apache.org/r/52510/diff/
> 
> 
> Testing
> ---
> 
> mvn clean install runs successfully in flume-ng-sinks
> 
> 
> Thanks,
> 
> Balázs Donát Bessenyei
> 
>

[jira] [Commented] (FLUME-2997) Fix flaky junit test in SpillableMemoryChannel

[Attila Simon (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLUME-2997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15585137#comment-15585137
 ] 

Attila Simon commented on FLUME-2997:
-

I think I missed some description of this change so let me provide it now: 
Originally this test case aimed to verify behaviour about having multiple 
mocked sources putting events on the same SpillableMemoryChannel while multiple 
mocked sinks drains those events. Given the mock implementation didn't do retry 
if sources are faster than the sinks the channel will get full. Since full 
channel doesn't accept new events the mocked sources fail their task by raising 
a runtime exception. This edge case should be avoided thus the increased 
capacity (which makes sure that channel cannot be full). The "channel is full" 
edge case is already covered by another test. 

Those System.out.println-s were replaced by asserts to make the function to 
became a test. It wasn't testing anything before. It only ran to the end 
without checking anything. (The flakiness was caused by a 
ChannelFullException). I guess the original author used those 
System.out.println-s for manual testing (purely looking at the numbers). This 
change also made that manual step automatic.

> Fix flaky junit test in SpillableMemoryChannel
> --
>
> Key: FLUME-2997
> URL: https://issues.apache.org/jira/browse/FLUME-2997
> Project: Flume
>  Issue Type: Test
>Affects Versions: v1.7.0
>Reporter: Attila Simon
>Assignee: Attila Simon
> Fix For: v1.8.0
>
> Attachments: FLUME-2997-1.patch, FLUME-2997.patch
>
>
> testParallelSingleSourceAndSink sometimes trigger an edge case scenario if 
> sinks are slower than sources. In such situations the channel can get full 
> thus uncaught ChannelFullException breaks the test. Since 
> testCapacityWithOverflow was designed to cover such edge-case scenario 
> already we can safely fix the test by increasing the channel capacity to make 
> sure it won't get full.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Flume channel error : Unable to put batch on required channel: FileChannel file-channel-1

[Attila Simon]

Hi,
It might've been my mobile client for gmail but couldn't find the flume
conf attached. Could you please double check?
Cheers,
Attila

On Monday, 17 October 2016, Dheeraj Rokade <dheerajrok...@yahoo.com.invalid>
wrote:

> Hello Team,
>
> My development activity is stuck due to below error in the Apache agent.
>
> *org.apache.flume.ChannelException: Unable to put batch on required
> channel: FileChannel file-channel-1 *
>
> I have the following configuration
> One Source - *Kafka*
> Three channels - *File channel*
> Four Sinks - *HDFS Sinks*
>
> Attached is the flume agent config file and below is the error message
>
> Things tried
> 1) Lowering or increasing the batch size for Kafka source and Hive HDFS
> sink
> 2) Lowering or increasing the TransactionCapacity and Capacity for File
> channel
> 3) With some help from the net, tried keeping File channel
> TransactionCapacity to be 5x of Kafka and HDFS Batch size but in vain.
>
> Please help advise
>
> Error message in detail :
> 16/10/17 18:06:08 ERROR kafka.KafkaSource: KafkaSource EXCEPTION, {}
> org.apache.flume.ChannelException: Unable to put batch on required
> channel: FileChannel file-channel-1 { dataDirs:
> [/home/dheeraj.rokade/flume/.flume/file-channel1/data] }
> at org.apache.flume.channel.ChannelProcessor.processEventBatch(
> ChannelProcessor.java:200)
> at org.apache.flume.source.kafka.KafkaSource.process(
> KafkaSource.java:123)
> at org.apache.flume.source.PollableSourceRunner$PollingRunner.run(
> PollableSourceRunner.java:139)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.NullPointerException
> at org.apache.flume.channel.file.proto.ProtosFactory$
> FlumeEventHeader$Builder.setValue(ProtosFactory.java:7415)
> at org.apache.flume.channel.file.Put.writeProtos(Put.java:85)
> at org.apache.flume.channel.file.TransactionEventRecord.
> toByteBuffer(TransactionEventRecord.java:174)
> at org.apache.flume.channel.file.Log.put(Log.java:642)
> at org.apache.flume.channel.file.FileChannel$
> FileBackedTransaction.doPut(FileChannel.java:468)
> at org.apache.flume.channel.BasicTransactionSemantics.put(
> BasicTransactionSemantics.java:93)
> at org.apache.flume.channel.BasicChannelSemantics.put(
> BasicChannelSemantics.java:80)
> at org.apache.flume.channel.ChannelProcessor.processEventBatch(
> ChannelProcessor.java:189)
> ... 3 more
> 16/10/17 18:06:13 ERROR kafka.KafkaSource: KafkaSource EXCEPTION, {}
> org.apache.flume.ChannelException: Unable to put batch on required
> channel: FileChannel file-channel-1 { dataDirs:
> [/home/dheeraj.rokade/flume/.flume/file-channel1/data] }
> at org.apache.flume.channel.ChannelProcessor.processEventBatch(
> ChannelProcessor.java:200)
> at org.apache.flume.source.kafka.KafkaSource.process(
> KafkaSource.java:123)
> at org.apache.flume.source.PollableSourceRunner$PollingRunner.run(
> PollableSourceRunner.java:139)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.NullPointerException
> at org.apache.flume.channel.file.proto.ProtosFactory$
> FlumeEventHeader$Builder.setValue(ProtosFactory.java:7415)
> at org.apache.flume.channel.file.Put.writeProtos(Put.java:85)
> at org.apache.flume.channel.file.TransactionEventRecord.
> toByteBuffer(TransactionEventRecord.java:174)
> at org.apache.flume.channel.file.Log.put(Log.java:642)
> at org.apache.flume.channel.file.FileChannel$
> FileBackedTransaction.doPut(FileChannel.java:468)
> at org.apache.flume.channel.BasicTransactionSemantics.put(
> BasicTransactionSemantics.java:93)
>     at org.apache.flume.channel.BasicChannelSemantics.put(
> BasicChannelSemantics.java:80)
> at org.apache.flume.channel.ChannelProcessor.processEventBatch(
> ChannelProcessor.java:189)
> ... 3 more
>
>

-- 

*Attila Simon*
Software Engineer
Email:   s...@cloudera.com

[image: Cloudera Inc.]

Re: make junit tests passing again

[Attila Simon]

Hi,

During the last month some "flakiest" tests were identified. I think these
are the ones which are really break a lot. If there is no strong objection
I would create a jira for each of these (if not already exists):

org.apache.flume.source.TestSyslogUdpSource.testSourceCounter
org.apache.flume.channel.jdbc.TestJdbcChannelProvider.
testEventWithSimulatedSourceAndSinks
org.apache.flume.channel.jdbc.TestJdbcChannelProviderNoFK.
testEventWithSimulatedSourceAndSinks
org.apache.flume.sink.hbase.TestAsyncHBaseSink
org.apache.flume.sink.hbase.TestHBaseSink
org.apache.flume.source.thriftLegacy.TestThriftLegacySource
org.apache.flume.channel.TestSpillableMemoryChannel.testParallelMultipleSourcesAndSinks
(sati is already addressing this with
https://issues.apache.org/jira/browse/FLUME-2997)
org.apache.flume.sink.hdfs.TestBucketWriter.testFileSuffixGiven (bessbd is
already addressing this with
https://issues.apache.org/jira/browse/FLUME-3002)
org.apache.flume.sink.hdfs.TestBucketWriter.testFileSuffixNotGiven (bessbd
is already addressing this with
https://issues.apache.org/jira/browse/FLUME-3002)
org.apache.flume.source.TestExecSource.testMonitoredCounterGroup (mpercy is
already addressing this with https://github.com/apache/flume/pull/72)


Cheers,
Attila


On Tue, Jun 14, 2016 at 9:03 PM, Hari Shreedharan <hshreedha...@apache.org>
wrote:

> I completely agree. My best guess is that most of these are flakey tests
> due to the fact that these tests are using Thread.sleep etc. Just FYI,
> fixing all tests will probably take a lot of time.
>
> On Tue, Jun 14, 2016 at 9:40 AM Attila Simon <s...@cloudera.com> wrote:
>
> > Hi All,
> >
> > It happened to me that when I run "mvn test" target it is failing on
> > my vanilla flume repo. Also when I checked an associated jenkins job
> > (https://builds.apache.org/view/All/job/Flume-trunk-hbase-1/) it
> > showed that test wasn't green for a while. Am I doing something
> > completely wrong?
> > I suspect hadoop/hbase libs are required but I couldn't find docs
> > which would tell me what should a new joiner do to have a properly set
> > up environment? I thought maven will resolve all of the dependencies,
> > if not then what would be the recommended way to have that. Could you
> > please point me to the right direction? Every help is greatly
> > appreciated.
> >
> > If this is a known issue I would like to devote some of my time to
> > clean this up as either releasing 1.7.0 would benefit from a stable
> > tests or generally it would increase the healthiness of the whole
> > project.
> >
> > I ran tests with this command after "git clone":
> > mvn test --fail-at-end
> >
> > [INFO] Apache Flume ... SUCCESS [
> > 0.365 s]
> > [INFO] Flume NG SDK ... FAILURE
> [01:00
> > min]
> > [INFO] Flume NG Configuration . SKIPPED
> > [INFO] Flume Auth . SKIPPED
> > [INFO] Flume NG Core .. SKIPPED
> > [INFO] Flume NG Sinks . SUCCESS [
> > 0.010 s]
> > [INFO] Flume NG HDFS Sink . SKIPPED
> > [INFO] Flume NG IRC Sink .. SKIPPED
> > [INFO] Flume NG Channels .. SUCCESS [
> > 0.009 s]
> > [INFO] Flume NG JDBC channel .. SKIPPED
> > [INFO] Flume NG file-based channel  SKIPPED
> > [INFO] Flume NG Spillable Memory channel .. SKIPPED
> > [INFO] Flume NG Node .. SKIPPED
> > [INFO] Flume NG Embedded Agent  SKIPPED
> > [INFO] Flume NG HBase Sink  SKIPPED
> > [INFO] Flume NG ElasticSearch Sink  SKIPPED
> > [INFO] Flume NG Morphline Solr Sink ... SKIPPED
> > [INFO] Flume Kafka Sink ... SKIPPED
> > [INFO] Flume NG Kite Dataset Sink . SKIPPED
> > [INFO] Flume NG Hive Sink . SKIPPED
> > [INFO] Flume Sources .. SUCCESS [
> > 0.010 s]
> > [INFO] Flume Scribe Source  SKIPPED
> > [INFO] Flume JMS Source ... SKIPPED
> > [INFO] Flume Twitter Source ... SKIPPED
> > [INFO] Flume Kafka Source . SKIPPED
> > [INFO] Flume Taildir Source ... SKIPPED
> &g

Re: Enabling Travis-CI on Flume

[Attila Simon]

Denes I'm happy to help you in this endeavor of setting up jenkins job for
verifying pull requests.


*Attila Simon*
Software Engineer
Email:   s...@cloudera.com

[image: Cloudera Inc.]

On Fri, Oct 14, 2016 at 2:47 PM, Denes Arvay <de...@cloudera.com> wrote:

> I'd also vote for Jenkins with github PRs.
> I just checked Mesos and the PRs are checked by Travis, or at least they
> experienced with it, there's a short discussion regarding to Travis at
> https://github.com/apache/mesos/pull/165
>
> As for the jenkins pull request job I'd be happy to set it up or help
> setting it up.
>
> Denes
>
> On Fri, Oct 14, 2016 at 2:15 PM Lior Zeno <liorz...@gmail.com> wrote:
>
> Are we switching to PRs from patches + RB? In Apache Mesos, they have a
>
> review bot that can leave a comment on the patch, we could try and port it
>
> to Flume. I think they use Jenkins too.
>
>
>
> On Fri, Oct 14, 2016 at 3:11 PM, Balazs Donat Bessenyei <
> bes...@cloudera.com
>
> > wrote:
>
>
>
> > If the same function can be achieved with Jenkins and it's easy
>
> > (+quick) to set up, I'm totally happy with that.
>
> >
>
> > What do we have to do to enable Jenkins builds on PR-s?
>
> >
>
> > On Fri, Oct 14, 2016 at 2:05 PM, Lior Zeno <liorz...@gmail.com> wrote:
>
> > > There are ways to do the same with Jenkins, for instance, see this SO
>
> > > thread
>
> > > http://stackoverflow.com/questions/37661602/how-to-set-
>
> > up-a-github-pull-request-build-in-a-jenkinsfile
>
> > >
>
> > > On Fri, Oct 14, 2016 at 11:09 AM, Balazs Donat Bessenyei <
>
> > > bes...@cloudera.com> wrote:
>
> > >
>
> > >> My primary reason for Travis (vs. Jenkins) was that I have experience
>
> > with
>
> > >> it.
>
> > >>
>
> > >> And it leaves these happy little checkmarks:
>
> > >> https://github.com/sebastianbergmann/phpunit/pull/1051/commits on the
>
> > >> commits and messages as seen on
>
> > >> https://github.com/apache/hive/pull/107 .
>
> > >>
>
> > >> Jenkins is probably configurable to achieve similar function. However,
>
> > >> I have no idea how to do such. (And could not find an example when I
>
> > >> did a quick search.)
>
> > >>
>
> > >> Are there any disadvantages of enabling Travis on Flume?
>
> > >>
>
> > >>
>
> > >> Thank you,
>
> > >>
>
> > >> Donat
>
> > >>
>
> > >> On Thu, Oct 13, 2016 at 6:06 PM, Lior Zeno <liorz...@gmail.com>
> wrote:
>
> > >> > Jenkins can do PRs as well. If we can upgrade Jenkins to 2.0, we
> will
>
> > be
>
> > >> > able to define the build step via Jenkinsfile which becomes very
>
> > similar
>
> > >> to
>
> > >> > Travis.
>
> > >> > Is there any reason to prefer Travis over Jenkins in our case?
>
> > >> >
>
> > >> > On Thu, Oct 13, 2016 at 7:01 PM, Balazs Donat Bessenyei <
>
> > >> bes...@cloudera.com
>
> > >> >> wrote:
>
> > >> >
>
> > >> >> Hi All,
>
> > >> >>
>
> > >> >> Having something that checks proposed patches (PR-s especially)
>
> > >> >> automatically would help a lot with the development on Flume.
>
> > >> >>
>
> > >> >> I think, Travis-CI could be an easy solution and (afaik) we'd only
>
> > have
>
> > >> to
>
> > >> >> ask infra to enable it for us.
>
> > >> >>
>
> > >> >> Please, let me know your thoughts.
>
> > >> >>
>
> > >> >> Thank you,
>
> > >> >>
>
> > >> >> Donat
>
> > >> >>
>
> > >>
>
> >
>

[jira] [Commented] (FLUME-2977) Upgrade RAT to 0.12

[Attila Simon (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLUME-2977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15574906#comment-15574906
 ] 

Attila Simon commented on FLUME-2977:
-

Reply from d...@creadur.apache.org:
http://mail-archives.apache.org/mod_mbox/creadur-dev/201609.mbox/%3c5d1d4b7c-cdd3-f5da-e248-ba6308d6f...@gmx.de%3E
http://mail-archives.apache.org/mod_mbox/creadur-dev/201609.mbox/%3ce93f5bac-a80a-c106-18bf-2b0a6228f...@apache.org%3E
http://mail-archives.apache.org/mod_mbox/creadur-dev/201609.mbox/%3ca3ce6b76-5fa8-f6d9-44da-621e2bb13...@gmx.de%3E
And the associated jira to track this issue: 
https://issues.apache.org/jira/browse/RAT-222

> Upgrade RAT to 0.12
> ---
>
> Key: FLUME-2977
> URL: https://issues.apache.org/jira/browse/FLUME-2977
> Project: Flume
>  Issue Type: Improvement
>  Components: Build
>    Reporter: Attila Simon
>Assignee: Attila Simon
>Priority: Minor
> Fix For: v1.8.0
>
> Attachments: FLUME-2977.patch
>
>
> Before RAT check mvn install prints out the following warnings:
> {noformat}
> Warning:  org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser: Property 
> 'http://www.oracle.com/xml/jaxp/properties/entityExpansionLimit' is not 
> recognized.
> Compiler warnings:
>   WARNING:  'org.apache.xerces.jaxp.SAXParserImpl: Property 
> 'http://javax.xml.XMLConstants/property/accessExternalDTD' is not recognized.'
> Warning:  org.apache.xerces.parsers.SAXParser: Feature 
> 'http://javax.xml.XMLConstants/feature/secure-processing' is not recognized.
> Warning:  org.apache.xerces.parsers.SAXParser: Property 
> 'http://javax.xml.XMLConstants/property/accessExternalDTD' is not recognized.
> Warning:  org.apache.xerces.parsers.SAXParser: Property 
> 'http://www.oracle.com/xml/jaxp/properties/entityExpansionLimit' is not 
> recognized.
> [INFO] Rat check: Summary of files. Unapproved: 0 unknown: 0 generated: 0 
> approved: 9 licence.
> {noformat} 
> It doesn't break the build but seems misleading.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: incorrect RAT version on Web Site

[Attila Simon]

Hi All,

Since I'm not on the dev@creadur list let me copy here their answers:

http://mail-archives.apache.org/mod_mbox/creadur-dev/201609.mbox/%3c5d1d4b7c-cdd3-f5da-e248-ba6308d6f...@gmx.de%3E
http://mail-archives.apache.org/mod_mbox/creadur-dev/201609.mbox/%3ce93f5bac-a80a-c106-18bf-2b0a6228f...@apache.org%3E
http://mail-archives.apache.org/mod_mbox/creadur-dev/201609.mbox/%3ca3ce6b76-5fa8-f6d9-44da-621e2bb13...@gmx.de%3E
And the associated jira to track this issue:
https://issues.apache.org/jira/browse/RAT-222

It seems like it is indeed a bug with their site and not with their release.

Cheers,
Attila



*Attila Simon*
Software Engineer
Email:   s...@cloudera.com

[image: Cloudera Inc.]

On Thu, Sep 22, 2016 at 3:36 PM, Attila Simon <s...@cloudera.com> wrote:

> Hi RAT devs,
>
> I saw a related email thread (Re: Creadur Web Site
> <http://mail-archives.apache.org/mod_mbox/creadur-dev/201609.mbox/ajax/%3C236eda00-4edb-1361-c710-df8c19015215%40apache.org%3E>)
> on dev but it seemed detoured from this issue hence raising awareness in a
> fresh thread.
>
> Although Apache RAT latest available version is 0.12 the official site
> mentions 0.13 with broken links to download the artifacts. Could you please
> check whether it is indeed the case and advise on the latest official
> released version?
>
> https://archive.apache.org/dist/creadur/  -> lists 0.12 as latest
> http://creadur.apache.org -> tells that RAT latest release is 0.12 on
> 2016.06.10
> http://creadur.staging.apache.org/rat/download_rat.cgi  -> tells 0.12
> (also available in maven)
> http://creadur.apache.org/rat/download_rat.cgi -> tells 0.13  (maven
> fails to resolve)
>
> Cheers,
> Attila Simon
>
>

Re: Enabling Travis-CI on Flume

[Attila Simon]

+1 on jenkins, and keeping our build infra as simple and intuitive as
possible
If migrating everything - we currently do with Jenkins - to Travis and
abandon Jenkins than I would be fine with that as well.


*Attila Simon*
Software Engineer
Email:   s...@cloudera.com

[image: Cloudera Inc.]

On Fri, Oct 14, 2016 at 10:09 AM, Balazs Donat Bessenyei <
bes...@cloudera.com> wrote:

> My primary reason for Travis (vs. Jenkins) was that I have experience with
> it.
>
> And it leaves these happy little checkmarks:
> https://github.com/sebastianbergmann/phpunit/pull/1051/commits on the
> commits and messages as seen on
> https://github.com/apache/hive/pull/107 .
>
> Jenkins is probably configurable to achieve similar function. However,
> I have no idea how to do such. (And could not find an example when I
> did a quick search.)
>
> Are there any disadvantages of enabling Travis on Flume?
>
>
> Thank you,
>
> Donat
>
> On Thu, Oct 13, 2016 at 6:06 PM, Lior Zeno <liorz...@gmail.com> wrote:
> > Jenkins can do PRs as well. If we can upgrade Jenkins to 2.0, we will be
> > able to define the build step via Jenkinsfile which becomes very similar
> to
> > Travis.
> > Is there any reason to prefer Travis over Jenkins in our case?
> >
> > On Thu, Oct 13, 2016 at 7:01 PM, Balazs Donat Bessenyei <
> bes...@cloudera.com
> >> wrote:
> >
> >> Hi All,
> >>
> >> Having something that checks proposed patches (PR-s especially)
> >> automatically would help a lot with the development on Flume.
> >>
> >> I think, Travis-CI could be an easy solution and (afaik) we'd only have
> to
> >> ask infra to enable it for us.
> >>
> >> Please, let me know your thoughts.
> >>
> >> Thank you,
> >>
> >> Donat
> >>
>

Re: Flume bechmarks

[Attila Simon]

Good idea! What would be required to set up something similar for Flume?
ie initial time cost for setting up the infrastructure and periodic time
cost to add new use-cases.

Cheers,
Attila



On Thu, Oct 13, 2016 at 5:19 PM, Lior Zeno  wrote:

> Hi All,
>
> Monitoring Flume's performance over time is an important step in every
> production-level application.  Benchmarking Flume on a nightly basis has
> the following advantages:
>
> * Better understanding of Flume's bottlenecks.
> * Allow users to compare the performance of different solutions, such as
> Logstash and Fluentd.
> * Better understanding of the influence of recent commits on performance.
>
> Logstash already conducts various performance tests, more details in this
> link:
> http://logstash-benchmarks.elastic.co/
>
> I propose adding a few micro-benchmarks showing Flume's TPS vs date (of
> course, in the ideal case where the input and/or output do not bottleneck
> the system), e.g. using the SeqGen source.
>
> Thoughts?
>
> Thanks
>

Re: [VOTE] Release Apache Flume version 1.7.0 RC2

[Attila Simon]

Hi Folks,

+1 on this Release Candidate

* hashes and signatures match
* build and run environment was Mac OSX El Capitan 10.11.5, Java
HotSpot(TM) 64-Bit Server VM (build 24.79-b02, mixed mode)
* in src tarball
  * Sources of flume-checkstyle module is not included. Rest of the files
are there compared to apache git repo. As discussed it is not a release
blocker (checkstyle was removed intentionally from src tarball).
  * "mvn clean install -DskipTest" builds
  * "mvn test" passed
* in bin tarball
  * verified content of lib dir against the LICENCE file whether "each jar
shipped is included in the LICENCE"  and  "each jar mentioned in the
LICENCE is shipped".
  * sanity checked DEVNOTES, README.md, NOTICE, RELEASE-NOTES,
doap_Flume.rdf and the content of bin, conf, tools dirs
  * in docs dir I verified main page, user guide, dev guide, javadocs
  * executed the "bin/flume-ng  agent -n agent -c conf -f
conf/flume-conf.properties.template" from user guide which is passed (logs
were generated in the logs dir)

Cheers,
Attila



On Thu, Oct 13, 2016 at 3:10 PM, Denes Arvay  wrote:

> Hi,
>
> +1 for the RC2
> - checksums and signatures match
> - source package successfully built (but skipped the tests), rat and
> checkstyle pass
> - I was able to start up flume extracted from the binary package with the
> sample configuration
>
> Denes
>
> On Wed, Oct 12, 2016 at 9:29 PM Balazs Donat Bessenyei <
> bes...@cloudera.com>
> wrote:
>
> Hi All,
>
>
>
> This is the tenth release for Apache Flume as a top-level project,
>
> version 1.7.0. We are voting on release candidate RC2.
>
>
>
> It fixes the following issues:
>
>   https://raw.githubusercontent.com/apache/flume/flume-1.7/CHANGELOG
>
>
>
> *** Please cast your vote within the next 72 hours ***
>
>
>
> The tarball (*.tar.gz), signature (*.asc), and checksums (*.md5, *.sha1)
>
> for the source and binary artifacts can be found here:
>
>   http://people.apache.org/~bessbd/apache-flume-1.7.0-rc2/
>
>
>
> Maven staging repo:
>
>   https://repository.apache.org/content/repositories/orgapacheflume-1020/
>
>
>
> The tag to be voted on:
>
>   https://git-wip-us.apache.org/repos/asf?p=flume.git;a=commit;h=511d868
>
>
>
> Flume's KEYS file containing PGP keys we use to sign the release:
>
>   https://www.apache.org/dist/flume/KEYS
>
>
>
>
>
> Thank you,
>
>
>
> Donat
>

Re: how to make KafkaSource consume the existing messages

[Attila Simon]

Hi,

One more thing. If you switch to the new group.id and would like to
maintain the read from beginning behaviour every time flume restart
then you might try setting enable.auto.commit to false.
Again Kafka normally won't store the events indefinitely.

Cheers,
Attila


On Thu, Oct 13, 2016 at 11:45 AM, Attila Simon <s...@cloudera.com> wrote:
> for the records cc dev@
>
> On Thu, Oct 13, 2016 at 11:43 AM, Attila Simon <s...@cloudera.com> wrote:
>> Hi,
>>
>> auto.offset.reset aim to handle failure scenarios when Flume lost the
>> track of offsets. When Flume is able to successfully consume the
>> messages it also commits the last processed offset. When failure
>> happens and  was set resetting offset would use the last
>> committed value.
>> I don't think that always starting from "zero" offset would be
>> valuable (would result a lot of duplicates). So I assume you would
>> like to have a recovery scenario. What you can do is setting the
>> consumer group.id to something new so if kafka still has the messages
>> - you can check that with command line kafka consumer setting the
>> --from-beginning argument as kafka by default purges them periodically
>> - then flume would reset the offset to the effective beginning since
>> offsets are stored per group.id.
>>
>> Quoted from Kafka docs
>> (http://kafka.apache.org/documentation#newconsumerconfigs):
>> auto.offset.reset - What to do when there is no initial offset in
>> Kafka or if the current offset does not exist any more on the server
>> (e.g. because that data has been deleted):
>>
>> earliest: automatically reset the offset to the earliest offset
>> latest: automatically reset the offset to the latest offset
>> none: throw exception to the consumer if no previous offset is found
>> for the consumer's group
>> anything else: throw exception to the consumer.
>>
>> Cheers,
>> Attila
>>
>>
>> On Thu, Oct 13, 2016 at 10:00 AM, Ping PW Wang <wpw...@cn.ibm.com> wrote:
>>> Hi,
>>> I used KafkaSource to consume the messages from Kafka. I found only new
>>> messages were received while the old existing message not. I tried to use a
>>> new consumer group and update the parameter "auto.offset.reset = latest" to
>>> "earliest", but this does not work.
>>>
>>> tier2.sources.source1.kafka.consumer.group.id = test-consumer-group-new
>>> tier2.sources.source1.kafka.consumer.auto.offset.reset = earliest
>>>
>>> Anyone knows how to make KafkaSource consume the existing messages?
>>> Thanks a lot for any advice!
>>>
>>> Best Regards,
>>>
>>> Wang Ping (王苹)
>>> InfoSphere BigInsights, CDL
>>> Email: wpw...@cn.ibm.com Phone: (8610)82453448 Mobile: (86)17090815725
>>> Address: Ring Bldg.No.28 Building,ZhongGuanCun Software Park,No.8 Dong Bei
>>> Wang West Road, Haidian District Beijing P.R.China 100193
>>> 地址：北京市海淀区东北旺西路8号，中关村软件园28号楼 邮编：100193
>>>

Re: how to make KafkaSource consume the existing messages

[Attila Simon]

for the records cc dev@

On Thu, Oct 13, 2016 at 11:43 AM, Attila Simon <s...@cloudera.com> wrote:
> Hi,
>
> auto.offset.reset aim to handle failure scenarios when Flume lost the
> track of offsets. When Flume is able to successfully consume the
> messages it also commits the last processed offset. When failure
> happens and  was set resetting offset would use the last
> committed value.
> I don't think that always starting from "zero" offset would be
> valuable (would result a lot of duplicates). So I assume you would
> like to have a recovery scenario. What you can do is setting the
> consumer group.id to something new so if kafka still has the messages
> - you can check that with command line kafka consumer setting the
> --from-beginning argument as kafka by default purges them periodically
> - then flume would reset the offset to the effective beginning since
> offsets are stored per group.id.
>
> Quoted from Kafka docs
> (http://kafka.apache.org/documentation#newconsumerconfigs):
> auto.offset.reset - What to do when there is no initial offset in
> Kafka or if the current offset does not exist any more on the server
> (e.g. because that data has been deleted):
>
> earliest: automatically reset the offset to the earliest offset
> latest: automatically reset the offset to the latest offset
> none: throw exception to the consumer if no previous offset is found
> for the consumer's group
> anything else: throw exception to the consumer.
>
> Cheers,
> Attila
>
>
> On Thu, Oct 13, 2016 at 10:00 AM, Ping PW Wang <wpw...@cn.ibm.com> wrote:
>> Hi,
>> I used KafkaSource to consume the messages from Kafka. I found only new
>> messages were received while the old existing message not. I tried to use a
>> new consumer group and update the parameter "auto.offset.reset = latest" to
>> "earliest", but this does not work.
>>
>> tier2.sources.source1.kafka.consumer.group.id = test-consumer-group-new
>> tier2.sources.source1.kafka.consumer.auto.offset.reset = earliest
>>
>> Anyone knows how to make KafkaSource consume the existing messages?
>> Thanks a lot for any advice!
>>
>> Best Regards,
>>
>> Wang Ping (王苹)
>> InfoSphere BigInsights, CDL
>> Email: wpw...@cn.ibm.com Phone: (8610)82453448 Mobile: (86)17090815725
>> Address: Ring Bldg.No.28 Building,ZhongGuanCun Software Park,No.8 Dong Bei
>> Wang West Road, Haidian District Beijing P.R.China 100193
>> 地址：北京市海淀区东北旺西路8号，中关村软件园28号楼 邮编：100193
>>

Re: [VOTE] Release Apache Flume version 1.7.0 RC1

[Attila Simon]

Thanks Mike,

I just created a pull request for this change:
https://github.com/apache/flume/pull/68
Regarding to the jackson-core|mapper-asl: it only has versions up to
1.9.13 as of now. Thus specifying the 1. in the license is confusing
for me. "-asl" is already in the name and that is the important
information.

Cheers,
Attila

On Wed, Oct 12, 2016 at 6:13 PM, Mike Percy <mpe...@apache.org> wrote:
> On Wed, Oct 12, 2016 at 5:07 PM, Attila Simon <s...@cloudera.com> wrote:
>
>> Additional libraries shipped but not mentioned in the licence file:
>> hamcrest-core (BSD License), junit (BSD License), xz (
>> http://git.tukaani.org/?p=xz.git;a=blob;f=COPYING),
>>
>
> Regarding xz it looks like while the main xz program has some weird mixed
> license, the xz-java library (which is what we are using) is in the public
> domain: http://git.tukaani.org/?p=xz-java.git;a=blob_plain;f=COPYING;hb=HEAD
>
> Consulting the official ASF legal guidelines found at
> https://www.apache.org/legal/resolved we can see that public domain works
> are allowed to be included in ASF software releases, but attribution may be
> needed. Simply adding the text of the xz-java COPYING file to the Flume
> LICENSE file verbatim includes copyright and attribution so that should be
> sufficient. Looking at other ASF projects that include public-domain works,
> that seems to be the approach used and they don't additionally include such
> attribution in their NOTICE files.
>
> Libraries which require little correction in the name in the licence file:
>> jackson-core-asl-1..jar
>> jackson-mapper-asl-1..jar
>>
>
> This was done on purpose to make it obvious that we are using the
> asl-licensed versions of jackson-1. jackson-1 is dual-licensed which the
> ASF doesn't like, whereas jackson-2 is ASL 2.0.
>
> Suggested actions:
>>  1) hamcrest-core, junit these are testing libs, I think shouldn't
>> be shipped (only required for tests, I guess they were added by the
>> new .*shared.* module)
>>
>
> +1, the artifacts from kafka-shared-test should not be shipped in the
> binary artifact
>
>
>>  2) kafka-clients, lz4 should be added to our LICENSE file
>>
>
> +1
>
>
>>  3) jackson.* fix the typo in the LICENSE file
>>
>
> I would rather we not change this per my comment above, unless you think
> it's very confusing
>
>  4) zookeeper since we depend on zkclient instead we should remove
>> this from the LICENSE file.
>>
>
> +1 to remove this entry from the LICENSE file now that it's no longer a
> transitive dependency.
>
> Mike

[jira] [Assigned] (FLUME-2924) Flume 1.7.0 release

[Attila Simon (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLUME-2924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Attila Simon reassigned FLUME-2924:
---

Assignee: (was: Attila Simon)

> Flume 1.7.0 release
> ---
>
> Key: FLUME-2924
> URL: https://issues.apache.org/jira/browse/FLUME-2924
> Project: Flume
>  Issue Type: Umbrella
>Affects Versions: v1.7.0
>Reporter: Lior Zeno
> Fix For: v1.7.0
>
>
> Release 1.7.0 umbrella issue



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (FLUME-2929) Update License file for 1.7 release

[Attila Simon (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLUME-2929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Attila Simon reassigned FLUME-2929:
---

Assignee: Attila Simon

> Update License file for 1.7 release
> ---
>
> Key: FLUME-2929
> URL: https://issues.apache.org/jira/browse/FLUME-2929
> Project: Flume
>  Issue Type: Sub-task
>Affects Versions: v1.7.0
>Reporter: Lior Zeno
>Assignee: Attila Simon
> Fix For: v1.7.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)