[jira] [Commented] (FLUME-3057) Build fails due to unsupported snappy-java version on ppc64le
[ https://issues.apache.org/jira/browse/FLUME-3057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16140398#comment-16140398 ] Attila Simon commented on FLUME-3057: - Hi [~pravindsilva], Since this has been committed, could you please close the reviewboard review: https://reviews.apache.org/r/57676/ > Build fails due to unsupported snappy-java version on ppc64le > - > > Key: FLUME-3057 > URL: https://issues.apache.org/jira/browse/FLUME-3057 > Project: Flume > Issue Type: Bug > Components: Build >Affects Versions: 1.8.0 > Environment: $ uname -a > Linux 2f63413ff231 4.4.0-31-generic #50-Ubuntu SMP Wed Jul 13 00:05:18 UTC > 2016 ppc64le ppc64le ppc64le GNU/Linux >Reporter: Pravin Dsilva > Labels: powerpc, ppc64le > Fix For: 1.8.0 > > Attachments: FLUME-3057-1.patch, FLUME-3057.patch > > > Flume has a snappy-java dependency with version 1.1.0. Upon building Flume on > ppc64le architecture, errors such as "[FAILED_TO_LOAD_NATIVE_LIBRARY] no > native library is found for os.name=Linux and os.arch=ppc64le" are seen > Native libraries for ppc64le were added in snappy-java version 1.1.1. Hence > Flume needs to have this higher version of snappy-java. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (FLUME-3057) Build fails due to unsupported snappy-java version on ppc64le
[ https://issues.apache.org/jira/browse/FLUME-3057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16137204#comment-16137204 ] Attila Simon edited comment on FLUME-3057 at 8/22/17 7:01 PM: -- I have the same issue and updating snappy solved it. Unfortunately the patch doesn't apply any more. Since this ticket is unassigned at the moment: to move things forward I created a new patch (FLUME-3057-1.patch) which is ready to be committed on latest trunk (as of now). I cannot add it to your reviewboard review but please feel free to do that. Also you can look at the updated patch as a commit here (I'm happy to create a pull request if that helps resolving this issue): https://github.com/simonati/flume/commit/5dd54881be4a752ea458d54089dc9cc7816568fa was (Author: sati): I have the same issue and updating snappy solved it. Unfortunately the patch doesn't apply any more. Since this ticket is unassigned at the moment: to move things forward I created a new patch which is ready to be committed on latest trunk (as of now). I cannot add it to your reviewboard review but please feel free to do that. Also you can look at the updated patch as a commit here (I'm happy to create a pull request if that helps resolving this issue): https://github.com/simonati/flume/commit/5dd54881be4a752ea458d54089dc9cc7816568fa > Build fails due to unsupported snappy-java version on ppc64le > - > > Key: FLUME-3057 > URL: https://issues.apache.org/jira/browse/FLUME-3057 > Project: Flume > Issue Type: Bug > Components: Build >Affects Versions: 1.8.0 > Environment: $ uname -a > Linux 2f63413ff231 4.4.0-31-generic #50-Ubuntu SMP Wed Jul 13 00:05:18 UTC > 2016 ppc64le ppc64le ppc64le GNU/Linux >Reporter: Pravin Dsilva > Labels: powerpc, ppc64le > Fix For: 1.8.0 > > Attachments: FLUME-3057-1.patch, FLUME-3057.patch > > > Flume has a snappy-java dependency with version 1.1.0. Upon building Flume on > ppc64le architecture, errors such as "[FAILED_TO_LOAD_NATIVE_LIBRARY] no > native library is found for os.name=Linux and os.arch=ppc64le" are seen > Native libraries for ppc64le were added in snappy-java version 1.1.1. Hence > Flume needs to have this higher version of snappy-java. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (FLUME-3057) Build fails due to unsupported snappy-java version on ppc64le
[ https://issues.apache.org/jira/browse/FLUME-3057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16137204#comment-16137204 ] Attila Simon commented on FLUME-3057: - I have the same issue and updating snappy solved it. Unfortunately the patch doesn't apply any more. Since this ticket is unassigned at the moment: to move things forward I created a new patch which is ready to be committed on latest trunk (as of now). I cannot add it to your reviewboard review but please feel free to do that. Also you can look at the updated patch as a commit here (I'm happy to create a pull request if that helps resolving this issue): https://github.com/simonati/flume/commit/5dd54881be4a752ea458d54089dc9cc7816568fa > Build fails due to unsupported snappy-java version on ppc64le > - > > Key: FLUME-3057 > URL: https://issues.apache.org/jira/browse/FLUME-3057 > Project: Flume > Issue Type: Bug > Components: Build >Affects Versions: 1.8.0 > Environment: $ uname -a > Linux 2f63413ff231 4.4.0-31-generic #50-Ubuntu SMP Wed Jul 13 00:05:18 UTC > 2016 ppc64le ppc64le ppc64le GNU/Linux >Reporter: Pravin Dsilva > Labels: powerpc, ppc64le > Fix For: 1.8.0 > > Attachments: FLUME-3057-1.patch, FLUME-3057.patch > > > Flume has a snappy-java dependency with version 1.1.0. Upon building Flume on > ppc64le architecture, errors such as "[FAILED_TO_LOAD_NATIVE_LIBRARY] no > native library is found for os.name=Linux and os.arch=ppc64le" are seen > Native libraries for ppc64le were added in snappy-java version 1.1.1. Hence > Flume needs to have this higher version of snappy-java. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (FLUME-3057) Build fails due to unsupported snappy-java version on ppc64le
[ https://issues.apache.org/jira/browse/FLUME-3057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Simon updated FLUME-3057: Attachment: FLUME-3057-1.patch > Build fails due to unsupported snappy-java version on ppc64le > - > > Key: FLUME-3057 > URL: https://issues.apache.org/jira/browse/FLUME-3057 > Project: Flume > Issue Type: Bug > Components: Build >Affects Versions: 1.8.0 > Environment: $ uname -a > Linux 2f63413ff231 4.4.0-31-generic #50-Ubuntu SMP Wed Jul 13 00:05:18 UTC > 2016 ppc64le ppc64le ppc64le GNU/Linux >Reporter: Pravin Dsilva > Labels: powerpc, ppc64le > Fix For: 1.8.0 > > Attachments: FLUME-3057-1.patch, FLUME-3057.patch > > > Flume has a snappy-java dependency with version 1.1.0. Upon building Flume on > ppc64le architecture, errors such as "[FAILED_TO_LOAD_NATIVE_LIBRARY] no > native library is found for os.name=Linux and os.arch=ppc64le" are seen > Native libraries for ppc64le were added in snappy-java version 1.1.1. Hence > Flume needs to have this higher version of snappy-java. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (FLUME-3143) Fix undeclared and unused dependencies
[
https://issues.apache.org/jira/browse/FLUME-3143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16134964#comment-16134964
]
Attila Simon commented on FLUME-3143:
-
Removing the "unused" slf4j-log4j12 dependency from the flume-ng-sdk module
would result in empty test output files. A logger backend implementation
slf4j-log4j12 is a runtime resolved dependency. AFAK dependency analysis only
checks statically the compiled bytecode thus it cannot be perfect around
dependencies used via runtime/reflection (and causing false positives which
would prevent enforcement).
However I recommended to ignoreNonCompile (since the check cannot be perfect
for those dependencies anyway) but in the end it is implementer's call to use
that recommendation or put the known to be runtime to the "exclude from
analysis" list.
If we don't want to enforce the good state at the end then I would see less
value doing this cleanup.
> Fix undeclared and unused dependencies
> --
>
> Key: FLUME-3143
> URL: https://issues.apache.org/jira/browse/FLUME-3143
> Project: Flume
> Issue Type: Bug
>Affects Versions: 1.7.0
>Reporter: Ferenc Szabo
>Assignee: Ferenc Szabo
> Labels: dependency
> Fix For: 1.8.0
>
>
> Dependency analysis shows some warnings in the project that should be fixed
> {noformat}
> [INFO]
>
> [INFO] Building Flume NG SDK 1.8.0-SNAPSHOT
> [INFO]
>
> [WARNING] Used undeclared dependencies found:
> [WARNING]commons-lang:commons-lang:jar:2.5:compile
> [WARNING] Unused declared dependencies found:
> [WARNING]org.slf4j:slf4j-log4j12:jar:1.6.1:compile
> [INFO]
>
> [INFO]
>
> [INFO] Building Flume NG Configuration 1.8.0-SNAPSHOT
> [INFO]
>
> [WARNING] Unused declared dependencies found:
> [WARNING]org.slf4j:slf4j-log4j12:jar:1.6.1:compile
> [INFO]
>
> [INFO]
>
> [INFO] Building Flume Auth 1.8.0-SNAPSHOT
> [INFO]
>
> [WARNING] Used undeclared dependencies found:
> [WARNING]org.apache.thrift:libthrift:jar:0.9.0:compile
> [WARNING] Unused declared dependencies found:
> [WARNING]org.slf4j:slf4j-log4j12:jar:1.6.1:compile
> [INFO]
>
> [INFO]
>
> [INFO] Building Flume NG Core 1.8.0-SNAPSHOT
> [INFO]
>
> [WARNING] Used undeclared dependencies found:
> [WARNING]com.google.code.findbugs:jsr305:jar:1.3.9:compile
> [WARNING]org.apache.httpcomponents:httpcore:jar:4.2.1:compile
> [WARNING] Unused declared dependencies found:
> [WARNING]org.mortbay.jetty:jetty-util:jar:6.1.26:compile
> [WARNING]log4j:log4j:jar:1.2.17:compile
> [WARNING]org.slf4j:slf4j-log4j12:jar:1.6.1:compile
> [INFO]
>
> [INFO]
>
> [INFO] Building Flume NG HDFS Sink 1.8.0-SNAPSHOT
> [INFO]
>
> [WARNING] Used undeclared dependencies found:
> [WARNING]org.apache.hadoop:hadoop-hdfs:test-jar:tests:2.4.0:test
> [WARNING]org.apache.flume:flume-ng-auth:jar:1.8.0-SNAPSHOT:compile
> [WARNING]org.apache.avro:avro:jar:1.7.4:compile
> [WARNING] Unused declared dependencies found:
> [WARNING]org.apache.hadoop:hadoop-minicluster:jar:2.4.0:test
> [WARNING]org.apache.hadoop:hadoop-auth:jar:2.4.0:compile
> [WARNING]org.slf4j:slf4j-log4j12:jar:1.6.1:test
> [INFO]
>
> [INFO]
>
> [INFO] Building Flume NG IRC Sink 1.8.0-SNAPSHOT
> [INFO]
>
> [WARNING] Used undeclared dependencies found:
> [WARNING]com.google.guava:guava:jar:11.0.2:compile
> [WARNING]commons-io:commons-io:jar:2.1:compile
> [WARNING] Unused declared dependencies found:
> [WARNING]org.slf4j:slf4j-log4j12:jar:1.6.1:test
> [INFO]
>
> [INFO]
[jira] [Commented] (FLUME-3143) Fix undeclared and unused dependencies
[
https://issues.apache.org/jira/browse/FLUME-3143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16133987#comment-16133987
]
Attila Simon commented on FLUME-3143:
-
This output seems partly incorrect since it lists slf4j-log4j12 as unused but
slf4j-log4j12 is used in generating the test log thus it's declaration in the
pom is wrong it should have been a runtime only scope eg test. With that said
{{mvn dependency:analyze -DignoreNonCompile}} might be a better way to run this
report. I don't know whether that would miss some other misconfiguration.
Once the cleanup is complete we may want to enforce the correct state as part
of our build:
https://maven.apache.org/plugins/maven-dependency-plugin/analyze-only-mojo.html
> Fix undeclared and unused dependencies
> --
>
> Key: FLUME-3143
> URL: https://issues.apache.org/jira/browse/FLUME-3143
> Project: Flume
> Issue Type: Bug
>Affects Versions: 1.7.0
>Reporter: Ferenc Szabo
>Assignee: Ferenc Szabo
> Labels: dependency
> Fix For: 1.8.0
>
>
> Dependency analysis shows some warnings in the project that should be fixed
> {noformat}
> [INFO]
>
> [INFO] Building Flume NG SDK 1.8.0-SNAPSHOT
> [INFO]
>
> [WARNING] Used undeclared dependencies found:
> [WARNING]commons-lang:commons-lang:jar:2.5:compile
> [WARNING] Unused declared dependencies found:
> [WARNING]org.slf4j:slf4j-log4j12:jar:1.6.1:compile
> [INFO]
>
> [INFO]
>
> [INFO] Building Flume NG Configuration 1.8.0-SNAPSHOT
> [INFO]
>
> [WARNING] Unused declared dependencies found:
> [WARNING]org.slf4j:slf4j-log4j12:jar:1.6.1:compile
> [INFO]
>
> [INFO]
>
> [INFO] Building Flume Auth 1.8.0-SNAPSHOT
> [INFO]
>
> [WARNING] Used undeclared dependencies found:
> [WARNING]org.apache.thrift:libthrift:jar:0.9.0:compile
> [WARNING] Unused declared dependencies found:
> [WARNING]org.slf4j:slf4j-log4j12:jar:1.6.1:compile
> [INFO]
>
> [INFO]
>
> [INFO] Building Flume NG Core 1.8.0-SNAPSHOT
> [INFO]
>
> [WARNING] Used undeclared dependencies found:
> [WARNING]com.google.code.findbugs:jsr305:jar:1.3.9:compile
> [WARNING]org.apache.httpcomponents:httpcore:jar:4.2.1:compile
> [WARNING] Unused declared dependencies found:
> [WARNING]org.mortbay.jetty:jetty-util:jar:6.1.26:compile
> [WARNING]log4j:log4j:jar:1.2.17:compile
> [WARNING]org.slf4j:slf4j-log4j12:jar:1.6.1:compile
> [INFO]
>
> [INFO]
>
> [INFO] Building Flume NG HDFS Sink 1.8.0-SNAPSHOT
> [INFO]
>
> [WARNING] Used undeclared dependencies found:
> [WARNING]org.apache.hadoop:hadoop-hdfs:test-jar:tests:2.4.0:test
> [WARNING]org.apache.flume:flume-ng-auth:jar:1.8.0-SNAPSHOT:compile
> [WARNING]org.apache.avro:avro:jar:1.7.4:compile
> [WARNING] Unused declared dependencies found:
> [WARNING]org.apache.hadoop:hadoop-minicluster:jar:2.4.0:test
> [WARNING]org.apache.hadoop:hadoop-auth:jar:2.4.0:compile
> [WARNING]org.slf4j:slf4j-log4j12:jar:1.6.1:test
> [INFO]
>
> [INFO]
>
> [INFO] Building Flume NG IRC Sink 1.8.0-SNAPSHOT
> [INFO]
>
> [WARNING] Used undeclared dependencies found:
> [WARNING]com.google.guava:guava:jar:11.0.2:compile
> [WARNING]commons-io:commons-io:jar:2.1:compile
> [WARNING] Unused declared dependencies found:
> [WARNING]org.slf4j:slf4j-log4j12:jar:1.6.1:test
> [INFO]
>
> [INFO]
>
> [INFO] Building Flume NG JDBC channel 1.8.0-SNAPSHOT
> [INFO]
>
[jira] [Commented] (FLUME-3112) Upgrade jackson-core library dependency
[ https://issues.apache.org/jira/browse/FLUME-3112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16128706#comment-16128706 ] Attila Simon commented on FLUME-3112: - I concluded the review will be done on github. If that is indeed the case could you please close the review on reviewboard? > Upgrade jackson-core library dependency > --- > > Key: FLUME-3112 > URL: https://issues.apache.org/jira/browse/FLUME-3112 > Project: Flume > Issue Type: Bug >Affects Versions: 1.7.0 >Reporter: Attila Simon >Assignee: Ferenc Szabo >Priority: Critical > Labels: dependency > Fix For: 1.8.0 > > Attachments: FLUME-3112.patch > > > ||Group||Artifact||Version used||Upgrade target|| > |com.fasterxml.jackson.core|jackson-core|2.3.1|2.8.9| > Security vulnerability: http://www.cvedetails.com/cve/CVE-2016-7051/ > Please do: > - double check the newest version. > - consider to remove a dependency if better alternative is available. > - check whether the lib change would introduce a backward incompatibility (in > which case please add this label `breaking_change` and fix version should be > the next major) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (FLUME-3132) Upgrade tomcat jasper library dependencies
[
https://issues.apache.org/jira/browse/FLUME-3132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16104663#comment-16104663
]
Attila Simon commented on FLUME-3132:
-
Hi [~fszabo],
Thank you for looking into this. Indeed it is included as a transitive
dependency via hadoop-common only. As long as we understand the exact impact
I'm fine with either case. Within our company we ship flume with a slightly
enhanced hadoop-common:2.6.0 so I cannot help you saying that 2.7.0 will be
fine for sure so that has to be heavily verified. By writing this it looks like
that upgrading hadoop-common would be a longer and separate task thus should
deserve a separate jira (and close this one once that has been resolved). I
guess impact of exclusion has a pretty limited scope so sounds easier to me,
but essentially it is your call which way to go.
> Upgrade tomcat jasper library dependencies
> --
>
> Key: FLUME-3132
> URL: https://issues.apache.org/jira/browse/FLUME-3132
> Project: Flume
> Issue Type: Bug
>Affects Versions: 1.7.0
>Reporter: Attila Simon
>Assignee: Ferenc Szabo
>Priority: Critical
> Labels: dependency
> Fix For: 1.8.0
>
>
> ||Group||Artifact||Version used||Upgrade target||
> |tomcat|jasper-compiler|5.5.23|8.5.x|
> |tomcat|jasper-runtime|5.5.23|8.5.x|
> Security vulnerability:
> - https://www.cvedetails.com/cve/CVE-2011-1318/
> -
> http://www.cvedetails.com/vulnerability-list/vendor_id-45/product_id-887/Apache-Tomcat.html
> Maven repositories:
> - https://mvnrepository.com/artifact/org.apache.tomcat/tomcat-jasper
> Note: These artifacts were moved to:
> * New Group org.apache.tomcat
> * New Artifact
> Please do:
> - CVE might be a false alarm or mistake. Please double check.
> - double check the newest version.
> - consider to remove a dependency if better alternative is available.
> - check whether the lib change would introduce a backward incompatibility (in
> which case please add this label `breaking_change` and fix version should be
> the next major)
> Excerpt from mvn dependency:tree
> {noformat}
> org.apache.flume:flume-ng-auth:jar:1.8.0-SNAPSHOT
> +- org.apache.hadoop:hadoop-common:jar:2.4.0:compile
> | +- tomcat:jasper-compiler:jar:5.5.23:runtime
> | +- tomcat:jasper-runtime:jar:5.5.23:runtime
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (FLUME-3093) Groundwork for version changes in root pom
[ https://issues.apache.org/jira/browse/FLUME-3093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Simon updated FLUME-3093: Labels: dependency (was: ) > Groundwork for version changes in root pom > -- > > Key: FLUME-3093 > URL: https://issues.apache.org/jira/browse/FLUME-3093 > Project: Flume > Issue Type: Task >Affects Versions: 1.7.0 >Reporter: Miklos Csanady >Assignee: Miklos Csanady > Labels: dependency > > Flume's root pom should have a parameter block where all the dependency > versions are listed. > This is fundamental to later version changes required to time effectively > overcame 3rd party security vulnerabilities. > This should be done in two steps: first just refactoring with no change, the > second is getting rid of unnecessary different versions. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (FLUME-3131) Upgrade spring framework library dependencies
[
https://issues.apache.org/jira/browse/FLUME-3131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16095935#comment-16095935
]
Attila Simon commented on FLUME-3131:
-
After looking at your patch now it is clear that you wanted to achieve what I
wrote above. Have you considered pulling in the
https://search.maven.org/#artifactdetails%7Cjavax.jms%7Cjms-api%7C1.1-rev-1%7Cjar
instead of the geronimo shaded version?
> Upgrade spring framework library dependencies
> -
>
> Key: FLUME-3131
> URL: https://issues.apache.org/jira/browse/FLUME-3131
> Project: Flume
> Issue Type: Bug
>Affects Versions: 1.7.0
>Reporter: Attila Simon
>Assignee: Ferenc Szabo
>Priority: Critical
> Labels: dependency
> Fix For: 1.8.0
>
> Attachments: FLUME-3131.patch
>
>
> ||Group||Artifact||Version used||Upgrade target||
> |org.springframework|spring-aop|3.0.7.RELEASE|4.3.9.RELEASE,|
> |org.springframework|spring-context|3.0.7.RELEASE|4.3.9.RELEASE,|
> |org.springframework|spring-core|3.0.7.RELEASE|4.3.9.RELEASE,|
> Security vulnerability:
> https://www.cvedetails.com/vulnerability-list/vendor_id-9664/product_id-17274/Springsource-Spring-Framework.html
> Maven repositories:
> - https://mvnrepository.com/artifact/org.springframework/spring-aop
> - https://mvnrepository.com/artifact/org.springframework/spring-context
> - https://mvnrepository.com/artifact/org.springframework/spring-core
> Please do:
> - CVE might be a false alarm or mistake. Please double check.
> - double check the newest version.
> - consider to remove a dependency if better alternative is available.
> - check whether the lib change would introduce a backward incompatibility (in
> which case please add this label `breaking_change` and fix version should be
> the next major)
> Excerpt from mvn dependency:tree
> {noformat}
> org.apache.flume.flume-ng-sources:flume-jms-source:jar:1.8.0-SNAPSHOT
> \- org.apache.activemq:activemq-core:jar:5.7.0:provided
>+- org.springframework:spring-context:jar:3.0.7.RELEASE:provided
>| +- org.springframework:spring-aop:jar:3.0.7.RELEASE:provided
>| +- org.springframework:spring-beans:jar:3.0.7.RELEASE:provided
>| +- org.springframework:spring-core:jar:3.0.7.RELEASE:provided
>| +- org.springframework:spring-expression:jar:3.0.7.RELEASE:provided
>| \- org.springframework:spring-asm:jar:3.0.7.RELEASE:provided
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Comment Edited] (FLUME-3131) Upgrade spring framework library dependencies
[
https://issues.apache.org/jira/browse/FLUME-3131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16095870#comment-16095870
]
Attila Simon edited comment on FLUME-3131 at 7/21/17 6:56 AM:
--
Hi [~fszabo],
In general I'm fine with any approach which getting us closer to the state that
flume is not vulnerable based on our understanding.
Indeed it looks like test only. But having a closer look it seems like that
activemq (parent dependency of spring and also brings in geronimo) also falls
into the same category. I would also consider update the version of the
activemq in case it still passes testing and doesn't bring in undesired
dependencies transitively. (This in turn might help resolving this ticket by
either removing the spring dependency completely or pulling in a "better" one)
{noformat}
⏚ [~/ws/apache/flume] trunk ± ag activemq *
flume-ng-doc/sphinx/FlumeUserGuide.rst
932:application it should work with any JMS provider but has only been tested
with ActiveMQ.
945:**initialContextFactory** -- Inital Context Factory, e.g:
org.apache.activemq.jndi.ActiveMQInitialContextFactory
994: a1.sources.r1.initialContextFactory =
org.apache.activemq.jndi.ActiveMQInitialContextFactory
flume-ng-sources/flume-jms-source/pom.xml
74: org.apache.activemq
75: activemq-core
flume-ng-sources/flume-jms-source/src/test/java/org/apache/flume/source/jms/TestIntegrationActiveMQ.java
37:import org.apache.activemq.ActiveMQConnectionFactory;
38:import org.apache.activemq.broker.BrokerPlugin;
39:import org.apache.activemq.broker.BrokerService;
40:import org.apache.activemq.security.AuthenticationUser;
41:import org.apache.activemq.security.SimpleAuthenticationPlugin;
57:public class TestIntegrationActiveMQ {
60: "org.apache.activemq.jndi.ActiveMQInitialContextFactory";
65: // specific for dynamic queues on ActiveMq
133:ConnectionFactory factory = new ActiveMQConnectionFactory(USERNAME,
154:ConnectionFactory factory = new ActiveMQConnectionFactory(USERNAME,
pom.xml
1081:org.apache.activemq
1082:activemq-core
{noformat}
was (Author: sati):
Hi [~fszabo],
In general I'm fine with any approach which getting us closer to the state that
flume is not vulnerable based on our understanding.
Indeed it looks like test only. But having a closer look it seems like that
activemq (parent dependency of geronimo) also falls into the same category. I
would also consider update the version of the activemq in case it still passes
testing and doesn't bring in undesired dependencies transitively. (This in turn
might help resolving this ticket by either removing the spring dependency
completely or pulling in a "better" one)
{noformat}
⏚ [~/ws/apache/flume] trunk ± ag activemq *
flume-ng-doc/sphinx/FlumeUserGuide.rst
932:application it should work with any JMS provider but has only been tested
with ActiveMQ.
945:**initialContextFactory** -- Inital Context Factory, e.g:
org.apache.activemq.jndi.ActiveMQInitialContextFactory
994: a1.sources.r1.initialContextFactory =
org.apache.activemq.jndi.ActiveMQInitialContextFactory
flume-ng-sources/flume-jms-source/pom.xml
74: org.apache.activemq
75: activemq-core
flume-ng-sources/flume-jms-source/src/test/java/org/apache/flume/source/jms/TestIntegrationActiveMQ.java
37:import org.apache.activemq.ActiveMQConnectionFactory;
38:import org.apache.activemq.broker.BrokerPlugin;
39:import org.apache.activemq.broker.BrokerService;
40:import org.apache.activemq.security.AuthenticationUser;
41:import org.apache.activemq.security.SimpleAuthenticationPlugin;
57:public class TestIntegrationActiveMQ {
60: "org.apache.activemq.jndi.ActiveMQInitialContextFactory";
65: // specific for dynamic queues on ActiveMq
133:ConnectionFactory factory = new ActiveMQConnectionFactory(USERNAME,
154:ConnectionFactory factory = new ActiveMQConnectionFactory(USERNAME,
pom.xml
1081:org.apache.activemq
1082:activemq-core
{noformat}
> Upgrade spring framework library dependencies
> -
>
> Key: FLUME-3131
> URL: https://issues.apache.org/jira/browse/FLUME-3131
> Project: Flume
> Issue Type: Bug
>Affects Versions: 1.7.0
>Reporter: Attila Simon
>Assignee: Ferenc Szabo
>Priority: Critical
> Labels: dependency
> Fix For: 1.8.0
>
> Attachments: FLUME-3131.patch
>
>
> ||Group||Artifact||Version used||Upgrade target||
> |org.springframework|spring-aop|3.0.7.RELEASE|4.3.9.RELEASE,|
> |org.springframework|spring-context|3.0.7.RELEASE|4.3.9.RELEASE,|
> |org.springframework|spring-core|3.0.7.RELEASE|4.3.9.RELEASE,|
> Security vulnerability:
> https://www.cvedetails.com/vulnerability-list/vendor_id-9664/product_id-17274/Springsource-Spring-Framework.html
> Maven
[jira] [Commented] (FLUME-3131) Upgrade spring framework library dependencies
[
https://issues.apache.org/jira/browse/FLUME-3131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16095870#comment-16095870
]
Attila Simon commented on FLUME-3131:
-
Hi [~fszabo],
In general I'm fine with any approach which getting us closer to the state that
flume is not vulnerable based on our understanding.
Indeed it looks like test only. But having a closer look it seems like that
activemq (parent dependency of geronimo) also falls into the same category. I
would also consider update the version of the activemq in case it still passes
testing and doesn't bring in undesired dependencies transitively. (This in turn
might help resolving this ticket by either removing the spring dependency
completely or pulling in a "better" one)
{noformat}
⏚ [~/ws/apache/flume] trunk ± ag activemq *
flume-ng-doc/sphinx/FlumeUserGuide.rst
932:application it should work with any JMS provider but has only been tested
with ActiveMQ.
945:**initialContextFactory** -- Inital Context Factory, e.g:
org.apache.activemq.jndi.ActiveMQInitialContextFactory
994: a1.sources.r1.initialContextFactory =
org.apache.activemq.jndi.ActiveMQInitialContextFactory
flume-ng-sources/flume-jms-source/pom.xml
74: org.apache.activemq
75: activemq-core
flume-ng-sources/flume-jms-source/src/test/java/org/apache/flume/source/jms/TestIntegrationActiveMQ.java
37:import org.apache.activemq.ActiveMQConnectionFactory;
38:import org.apache.activemq.broker.BrokerPlugin;
39:import org.apache.activemq.broker.BrokerService;
40:import org.apache.activemq.security.AuthenticationUser;
41:import org.apache.activemq.security.SimpleAuthenticationPlugin;
57:public class TestIntegrationActiveMQ {
60: "org.apache.activemq.jndi.ActiveMQInitialContextFactory";
65: // specific for dynamic queues on ActiveMq
133:ConnectionFactory factory = new ActiveMQConnectionFactory(USERNAME,
154:ConnectionFactory factory = new ActiveMQConnectionFactory(USERNAME,
pom.xml
1081:org.apache.activemq
1082:activemq-core
{noformat}
> Upgrade spring framework library dependencies
> -
>
> Key: FLUME-3131
> URL: https://issues.apache.org/jira/browse/FLUME-3131
> Project: Flume
> Issue Type: Bug
>Affects Versions: 1.7.0
>Reporter: Attila Simon
>Assignee: Ferenc Szabo
>Priority: Critical
> Labels: dependency
> Fix For: 1.8.0
>
> Attachments: FLUME-3131.patch
>
>
> ||Group||Artifact||Version used||Upgrade target||
> |org.springframework|spring-aop|3.0.7.RELEASE|4.3.9.RELEASE,|
> |org.springframework|spring-context|3.0.7.RELEASE|4.3.9.RELEASE,|
> |org.springframework|spring-core|3.0.7.RELEASE|4.3.9.RELEASE,|
> Security vulnerability:
> https://www.cvedetails.com/vulnerability-list/vendor_id-9664/product_id-17274/Springsource-Spring-Framework.html
> Maven repositories:
> - https://mvnrepository.com/artifact/org.springframework/spring-aop
> - https://mvnrepository.com/artifact/org.springframework/spring-context
> - https://mvnrepository.com/artifact/org.springframework/spring-core
> Please do:
> - CVE might be a false alarm or mistake. Please double check.
> - double check the newest version.
> - consider to remove a dependency if better alternative is available.
> - check whether the lib change would introduce a backward incompatibility (in
> which case please add this label `breaking_change` and fix version should be
> the next major)
> Excerpt from mvn dependency:tree
> {noformat}
> org.apache.flume.flume-ng-sources:flume-jms-source:jar:1.8.0-SNAPSHOT
> \- org.apache.activemq:activemq-core:jar:5.7.0:provided
>+- org.springframework:spring-context:jar:3.0.7.RELEASE:provided
>| +- org.springframework:spring-aop:jar:3.0.7.RELEASE:provided
>| +- org.springframework:spring-beans:jar:3.0.7.RELEASE:provided
>| +- org.springframework:spring-core:jar:3.0.7.RELEASE:provided
>| +- org.springframework:spring-expression:jar:3.0.7.RELEASE:provided
>| \- org.springframework:spring-asm:jar:3.0.7.RELEASE:provided
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Comment Edited] (FLUME-3134) Enforce version of library dependencies during mvn build
[
https://issues.apache.org/jira/browse/FLUME-3134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16094523#comment-16094523
]
Attila Simon edited comment on FLUME-3134 at 7/20/17 10:55 AM:
---
The idea came from a discussion on the dev@ list between
[~ralph.go...@dslextreme.com] and [~mpercy]:
{quote}
> Mike,
>
> In all the projects I actively work on I put a dependencyManagement
> section in the parent pom and try to specify the version of every
> dependency there, whether direct or transitive. I also add the Maven
> enforcer plugin to make sure that subproject aren’t specifying differing
> versions of the same dependency.
>
> If one of the dependencies uses a library that is incompatible with a
> later version then you should create a bug report for that dependency so
> they can get it fixed.
>
> Ralph
>
Hey Ralph,
Nice. I didn't know enforcer could do that. After looking at its usage page
it seems easy to set up, too:
http://maven.apache.org/enforcer/enforcer-rules/dependencyConvergence.html
I would definitely be +1 for harmonizing Flume's dependencies and adding a
dependencyConvergence check to the Maven build. That first step should be
done with some care, though.
Mike
{quote}
was (Author: sati):
The idea came from a discussion on the dev@ list:
{noformat}
> Mike,
>
> In all the projects I actively work on I put a dependencyManagement
> section in the parent pom and try to specify the version of every
> dependency there, whether direct or transitive. I also add the Maven
> enforcer plugin to make sure that subproject aren’t specifying differing
> versions of the same dependency.
>
> If one of the dependencies uses a library that is incompatible with a
> later version then you should create a bug report for that dependency so
> they can get it fixed.
>
> Ralph
>
Hey Ralph,
Nice. I didn't know enforcer could do that. After looking at its usage page
it seems easy to set up, too:
http://maven.apache.org/enforcer/enforcer-rules/dependencyConvergence.html
I would definitely be +1 for harmonizing Flume's dependencies and adding a
dependencyConvergence check to the Maven build. That first step should be
done with some care, though.
Mike
{noformat}
> Enforce version of library dependencies during mvn build
>
>
> Key: FLUME-3134
> URL: https://issues.apache.org/jira/browse/FLUME-3134
> Project: Flume
> Issue Type: Bug
>Affects Versions: 1.7.0
>Reporter: Attila Simon
> Labels: dependency
>
> http://maven.apache.org/enforcer/enforcer-rules/dependencyConvergence.html
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (FLUME-3134) Enforce version of library dependencies during mvn build
[
https://issues.apache.org/jira/browse/FLUME-3134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16094523#comment-16094523
]
Attila Simon commented on FLUME-3134:
-
The idea came from a discussion on the dev@ list:
{noformat}
> Mike,
>
> In all the projects I actively work on I put a dependencyManagement
> section in the parent pom and try to specify the version of every
> dependency there, whether direct or transitive. I also add the Maven
> enforcer plugin to make sure that subproject aren’t specifying differing
> versions of the same dependency.
>
> If one of the dependencies uses a library that is incompatible with a
> later version then you should create a bug report for that dependency so
> they can get it fixed.
>
> Ralph
>
Hey Ralph,
Nice. I didn't know enforcer could do that. After looking at its usage page
it seems easy to set up, too:
http://maven.apache.org/enforcer/enforcer-rules/dependencyConvergence.html
I would definitely be +1 for harmonizing Flume's dependencies and adding a
dependencyConvergence check to the Maven build. That first step should be
done with some care, though.
Mike
{noformat}
> Enforce version of library dependencies during mvn build
>
>
> Key: FLUME-3134
> URL: https://issues.apache.org/jira/browse/FLUME-3134
> Project: Flume
> Issue Type: Bug
>Affects Versions: 1.7.0
>Reporter: Attila Simon
> Labels: dependency
>
> http://maven.apache.org/enforcer/enforcer-rules/dependencyConvergence.html
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (FLUME-3134) Enforce version of library dependencies during mvn build
Attila Simon created FLUME-3134: --- Summary: Enforce version of library dependencies during mvn build Key: FLUME-3134 URL: https://issues.apache.org/jira/browse/FLUME-3134 Project: Flume Issue Type: Bug Affects Versions: 1.7.0 Reporter: Attila Simon http://maven.apache.org/enforcer/enforcer-rules/dependencyConvergence.html -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (FLUME-1520) Timestamp interceptor should support custom headers
[ https://issues.apache.org/jira/browse/FLUME-1520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Simon updated FLUME-1520: Attachment: FLUME-1520-3.patch > Timestamp interceptor should support custom headers > --- > > Key: FLUME-1520 > URL: https://issues.apache.org/jira/browse/FLUME-1520 > Project: Flume > Issue Type: Improvement >Reporter: Hari Shreedharan >Assignee: Hari Shreedharan > Fix For: 1.8.0 > > Attachments: FLUME-1520-2.patch, FLUME-1520-3.patch, FLUME-1520.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (FLUME-1520) Timestamp interceptor should support custom headers
[
https://issues.apache.org/jira/browse/FLUME-1520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16094497#comment-16094497
]
Attila Simon commented on FLUME-1520:
-
Hi [~tmgstev],
The 2nd patch contains an extra file. I guess it was added by mistake:
flume-ng-core/src/main/java/org/apache/flume/interceptor/TimestampInterceptor.java.orig
Unfortunately the patch fails maven build on checkstyle.
Overall the doc, test and functionality looks good to me.
I'm only nitpicking on the naming:
{code}
public class TimestampInterceptor implements Interceptor {
private final String header;
...
public static class Constants {
public static final String CONFIG_PRESERVE = "preserveExisting";
public static final boolean DEFAULT_PRESERVE = false;
public static final String CONFIG_HEADER_NAME = "headerName";
public static final String DEFAULT_HEADER_NAME = "timestamp";
}
{code}
I have this renaming and using final for constants as well as removed some
unused local variables from the tests. I upload a patch which compiles and
passes junit as well. Please feel free to use it (completely or parts) if you
like.
> Timestamp interceptor should support custom headers
> ---
>
> Key: FLUME-1520
> URL: https://issues.apache.org/jira/browse/FLUME-1520
> Project: Flume
> Issue Type: Improvement
>Reporter: Hari Shreedharan
>Assignee: Hari Shreedharan
> Fix For: 1.8.0
>
> Attachments: FLUME-1520-2.patch, FLUME-1520-3.patch, FLUME-1520.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (FLUME-3132) Upgrade tomcat jasper library dependencies
Attila Simon created FLUME-3132:
---
Summary: Upgrade tomcat jasper library dependencies
Key: FLUME-3132
URL: https://issues.apache.org/jira/browse/FLUME-3132
Project: Flume
Issue Type: Bug
Affects Versions: 1.7.0
Reporter: Attila Simon
Priority: Critical
Fix For: 1.8.0
||Group||Artifact||Version used||Upgrade target||
|tomcat|jasper-compiler|5.5.23|8.5.x|
|tomcat|jasper-runtime|5.5.23|8.5.x|
Security vulnerability:
- https://www.cvedetails.com/cve/CVE-2011-1318/
-
http://www.cvedetails.com/vulnerability-list/vendor_id-45/product_id-887/Apache-Tomcat.html
Maven repositories:
- https://mvnrepository.com/artifact/org.apache.tomcat/tomcat-jasper
Note: These artifacts were moved to:
* New Group org.apache.tomcat
* New Artifact
Please do:
- CVE might be a false alarm or mistake. Please double check.
- double check the newest version.
- consider to remove a dependency if better alternative is available.
- check whether the lib change would introduce a backward incompatibility (in
which case please add this label `breaking_change` and fix version should be
the next major)
Excerpt from mvn dependency:tree
{noformat}
org.apache.flume:flume-ng-auth:jar:1.8.0-SNAPSHOT
+- org.apache.hadoop:hadoop-common:jar:2.4.0:compile
| +- tomcat:jasper-compiler:jar:5.5.23:runtime
| +- tomcat:jasper-runtime:jar:5.5.23:runtime
{noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (FLUME-3131) Upgrade spring framework library dependencies
Attila Simon created FLUME-3131:
---
Summary: Upgrade spring framework library dependencies
Key: FLUME-3131
URL: https://issues.apache.org/jira/browse/FLUME-3131
Project: Flume
Issue Type: Bug
Affects Versions: 1.7.0
Reporter: Attila Simon
Priority: Critical
Fix For: 1.8.0
||Group||Artifact||Version used||Upgrade target||
|org.springframework|spring-aop|3.0.7.RELEASE|4.3.9.RELEASE,|
|org.springframework|spring-context|3.0.7.RELEASE|4.3.9.RELEASE,|
|org.springframework|spring-core|3.0.7.RELEASE|4.3.9.RELEASE,|
Security vulnerability:
https://www.cvedetails.com/vulnerability-list/vendor_id-9664/product_id-17274/Springsource-Spring-Framework.html
Maven repositories:
- https://mvnrepository.com/artifact/org.springframework/spring-aop
- https://mvnrepository.com/artifact/org.springframework/spring-context
- https://mvnrepository.com/artifact/org.springframework/spring-core
Please do:
- CVE might be a false alarm or mistake. Please double check.
- double check the newest version.
- consider to remove a dependency if better alternative is available.
- check whether the lib change would introduce a backward incompatibility (in
which case please add this label `breaking_change` and fix version should be
the next major)
Excerpt from mvn dependency:tree
{noformat}
org.apache.flume.flume-ng-sources:flume-jms-source:jar:1.8.0-SNAPSHOT
\- org.apache.activemq:activemq-core:jar:5.7.0:provided
+- org.springframework:spring-context:jar:3.0.7.RELEASE:provided
| +- org.springframework:spring-aop:jar:3.0.7.RELEASE:provided
| +- org.springframework:spring-beans:jar:3.0.7.RELEASE:provided
| +- org.springframework:spring-core:jar:3.0.7.RELEASE:provided
| +- org.springframework:spring-expression:jar:3.0.7.RELEASE:provided
| \- org.springframework:spring-asm:jar:3.0.7.RELEASE:provided
{noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (FLUME-3130) Upgrade restlet library dependency
Attila Simon created FLUME-3130:
---
Summary: Upgrade restlet library dependency
Key: FLUME-3130
URL: https://issues.apache.org/jira/browse/FLUME-3130
Project: Flume
Issue Type: Bug
Affects Versions: 1.7.0
Reporter: Attila Simon
Priority: Critical
Fix For: 1.8.0
||Group||Artifact||Version used||Upgrade target||
|org.restlet.jee|org.restlet|2.1.1|2.3.10|
Security vulnerability:
http://www.cvedetails.com/vulnerability-list/vendor_id-12911/product_id-26316/Restlet-Restlet.html
Maven: https://mvnrepository.com/artifact/org.restlet.jee/org.restlet
Please do:
- CVE might be a false alarm or mistake. Please double check.
- double check the newest version.
- consider to remove a dependency if better alternative is available.
- check whether the lib change would introduce a backward incompatibility (in
which case please add this label `breaking_change` and fix version should be
the next major)
Excerpt from mvn dependency:tree
{noformat}
org.apache.flume.flume-ng-sinks:flume-ng-morphline-solr-sink:jar:1.8.0-SNAPSHOT
+- org.apache.solr:solr-test-framework:jar:4.3.0:test
| +- org.apache.solr:solr-core:jar:4.3.0:compile
| | +- org.restlet.jee:org.restlet:jar:2.1.1:compile
| | +- org.restlet.jee:org.restlet.ext.servlet:jar:2.1.1:compile
{noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (FLUME-2698) Upgrade Jetty Version
[
https://issues.apache.org/jira/browse/FLUME-2698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Attila Simon updated FLUME-2698:
Labels: dependency (was: )
> Upgrade Jetty Version
> -
>
> Key: FLUME-2698
> URL: https://issues.apache.org/jira/browse/FLUME-2698
> Project: Flume
> Issue Type: Bug
> Components: Web
>Affects Versions: 1.6.0, 1.5.1, 1.7.0
>Reporter: Joakim Erdfelt
>Assignee: Tristan Stevens
> Labels: dependency
>
> Flume depends on Jetty 6
> {code:xml}
>
> org.mortbay.jetty
> jetty-util
> 6.1.26
>
> {code}
> Which was EOL (End of Life) back in 2010 and is no longer fit for production
> use (without heavy customizations and modifications like Google does for GAE,
> just to keep it safe and vulnerability free)
> Jetty was moved to Eclipse.org back during the Jetty 7 days.
> http://eclipse.org/jetty/
> Note that [Jetty 7 and Jetty 8 are now also
> EOL|https://dev.eclipse.org/mhonarc/lists/jetty-announce/msg00069.html] (as
> of 2014)
> Jetty 9 is the only stable and supported version of Jetty now.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (FLUME-3129) Upgrade bouncycastle library dependencies
Attila Simon created FLUME-3129:
---
Summary: Upgrade bouncycastle library dependencies
Key: FLUME-3129
URL: https://issues.apache.org/jira/browse/FLUME-3129
Project: Flume
Issue Type: Bug
Affects Versions: 1.7.0
Reporter: Attila Simon
Priority: Critical
Fix For: 1.8.0
||Group||Artifact||Version used||Upgrade target||
|org.bouncycastle|bcprov-jdk15|1.45|1.57|
|org.bouncycastle|bcmail-jdk15|1.45|1.57|
Security vulnerability:
https://www.cvedetails.com/vulnerability-list/vendor_id-7637/Bouncycastle.html
Maven repository:
https://mvnrepository.com/artifact/org.bouncycastle/bcprov-jdk15
Please do:
- CVE might be a false alarm or mistake. Please double check.
- double check the newest version.
- consider to remove a dependency if better alternative is available.
- check whether the lib change would introduce a backward incompatibility (in
which case please add this label `breaking_change` and fix version should be
the next major)
Excerpt from mvn dependency:tree
{noformat}
org.apache.flume.flume-ng-sinks:flume-ng-morphline-solr-sink:jar:1.8.0-SNAPSHOT
+- org.kitesdk:kite-morphlines-all:pom:1.0.0:compile
| +- org.kitesdk:kite-morphlines-solr-cell:jar:1.0.0:compile
| | +- org.apache.tika:tika-xmp:jar:1.5:compile
| | | +- org.apache.tika:tika-parsers:jar:1.5:compile
| | | | +- org.bouncycastle:bcmail-jdk15:jar:1.45:compile
| | | | +- org.bouncycastle:bcprov-jdk15:jar:1.45:compile
{noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (FLUME-3127) Upgrade libfb303 library dependency
Attila Simon created FLUME-3127:
---
Summary: Upgrade libfb303 library dependency
Key: FLUME-3127
URL: https://issues.apache.org/jira/browse/FLUME-3127
Project: Flume
Issue Type: Bug
Affects Versions: 1.7.0
Reporter: Attila Simon
Priority: Critical
Fix For: 1.8.0
||Group||Artifact||Version used||Upgrade target||
|org.apache.thrift|libthrift|0.9.0|0.9.3,0.10.0|
|org.apache.thrift|libfb303|0.9.0|0.9.3|
Security vulnerability: http://www.cvedetails.com/cve/CVE-2015-3254/
Maven repository:
- https://mvnrepository.com/artifact/org.apache.thrift/libthrift
- https://mvnrepository.com/artifact/org.apache.thrift/libfb303
Please do:
- CVE might be a false alarm or mistake. Please double check.
- double check the newest version.
- consider to remove a dependency if better alternative is available.
- check whether the lib change would introduce a backward incompatibility (in
which case please add this label `breaking_change` and fix version should be
the next major)
Excerpt from mvn dependency:tree
{noformat}
org.apache.flume:flume-ng-sdk:jar:1.8.0-SNAPSHOT
\- org.apache.thrift:libthrift:jar:0.9.0:compile
org.apache.flume.flume-ng-sinks:flume-hive-sink:jar:1.8.0-SNAPSHOT
+- org.apache.hive.hcatalog:hive-hcatalog-streaming:jar:1.0.0:provided
| +- org.apache.hive:hive-metastore:jar:1.0.0:provided
| | \- org.apache.thrift:libfb303:jar:0.9.0:provided
{noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (FLUME-3126) Upgrade apache poi library dependencies
Attila Simon created FLUME-3126:
---
Summary: Upgrade apache poi library dependencies
Key: FLUME-3126
URL: https://issues.apache.org/jira/browse/FLUME-3126
Project: Flume
Issue Type: Bug
Affects Versions: 1.7.0
Reporter: Attila Simon
Priority: Critical
Fix For: 1.8.0
||Group||Artifact||Version used||Upgrade target||
|org.apache.poi|poi|3.10-beta2|3.15-beta2|
|org.apache.poi|poi-ooxml|3.10-beta2|3.15-beta2|
|org.apache.poi|poi-scratchpad|3.10-beta2|3.15-beta2|
Security vulnerability:
https://www.cvedetails.com/vulnerability-list/vendor_id-45/product_id-22766/Apache-POI.html
Maven repositories:
- https://mvnrepository.com/artifact/org.apache.poi/poi-ooxml
- https://mvnrepository.com/artifact/org.apache.poi/poi
- https://mvnrepository.com/artifact/org.apache.poi/poi
Please do:
- CVE might be a false alarm or mistake. Please double check.
- double check the newest version.
- consider to remove a dependency if better alternative is available.
- check whether the lib change would introduce a backward incompatibility (in
which case please add this label `breaking_change` and fix version should be
the next major)
Excerpt from mvn dependency:tree
{noformat}
org.apache.flume.flume-ng-sinks:flume-ng-morphline-solr-sink:jar:1.8.0-SNAPSHOT
+- org.kitesdk:kite-morphlines-all:pom:1.0.0:compile
| +- org.kitesdk:kite-morphlines-solr-cell:jar:1.0.0:compile
| | +- org.apache.tika:tika-xmp:jar:1.5:compile
| | | +- org.apache.tika:tika-parsers:jar:1.5:compile
| | | | +- org.apache.poi:poi:jar:3.10-beta2:compile
| | | | +- org.apache.poi:poi-scratchpad:jar:3.10-beta2:compile
| | | | +- org.apache.poi:poi-ooxml:jar:3.10-beta2:compile
{noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (FLUME-3125) Upgrade fontbox library dependency
Attila Simon created FLUME-3125:
---
Summary: Upgrade fontbox library dependency
Key: FLUME-3125
URL: https://issues.apache.org/jira/browse/FLUME-3125
Project: Flume
Issue Type: Bug
Affects Versions: 1.7.0
Reporter: Attila Simon
Priority: Critical
Fix For: 1.8.0
||Group||Artifact||Version used||Upgrade target||
|org.apache.pdfbox|fontbox|1.8.4|2.0.6|
Security vulnerability: http://www.cvedetails.com/cve/CVE-2016-2175/
Maven repository: https://mvnrepository.com/artifact/org.apache.pdfbox/fontbox
Please do:
- CVE might be a false alarm or mistake. Please double check.
- double check the newest version.
- consider to remove a dependency if better alternative is available.
- check whether the lib change would introduce a backward incompatibility (in
which case please add this label `breaking_change` and fix version should be
the next major)
Excerpt from mvn dependency:tree
{noformat}
org.apache.flume.flume-ng-sinks:flume-ng-morphline-solr-sink:jar:1.8.0-SNAPSHOT
+- org.kitesdk:kite-morphlines-all:pom:1.0.0:compile
| +- org.kitesdk:kite-morphlines-solr-cell:jar:1.0.0:compile
| | +- org.apache.tika:tika-xmp:jar:1.5:compile
| | | +- org.apache.tika:tika-parsers:jar:1.5:compile
| | | | +- org.apache.pdfbox:pdfbox:jar:1.8.4:compile
| | | | | +- org.apache.pdfbox:fontbox:jar:1.8.4:compile
{noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (FLUME-3124) Upgrade apache-mime4j-core library dependency
Attila Simon created FLUME-3124:
---
Summary: Upgrade apache-mime4j-core library dependency
Key: FLUME-3124
URL: https://issues.apache.org/jira/browse/FLUME-3124
Project: Flume
Issue Type: Bug
Affects Versions: 1.7.0
Reporter: Attila Simon
Priority: Critical
Fix For: 1.8.0
||Group||Artifact||Version used||Upgrade target||
|org.apache.james|apache-mime4j-core|0.7.2|0.8.1|
Security vulnerability:
https://www.cvedetails.com/vulnerability-list/vendor_id-45/product_id-4526/Apache-James.html
Maven repository:
https://mvnrepository.com/artifact/org.apache.james/apache-mime4j
Please do:
- CVE might be a false alarm or mistake. Please double check.
- double check the newest version.
- consider to remove a dependency if better alternative is available.
- check whether the lib change would introduce a backward incompatibility (in
which case please add this label `breaking_change` and fix version should be
the next major)
Excerpt from mvn dependency:tree
{noformat}
org.apache.flume.flume-ng-sinks:flume-ng-morphline-solr-sink:jar:1.8.0-SNAPSHOT
+- org.kitesdk:kite-morphlines-all:pom:1.0.0:compile
| +- org.kitesdk:kite-morphlines-solr-cell:jar:1.0.0:compile
| | +- org.apache.tika:tika-xmp:jar:1.5:compile
| | | +- org.apache.tika:tika-parsers:jar:1.5:compile
| | | | +- org.apache.james:apache-mime4j-core:jar:0.7.2:compile
| | | | +- org.apache.james:apache-mime4j-dom:jar:0.7.2:compile
{noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (FLUME-3114) Upgrade commons-httpclient library dependency
[
https://issues.apache.org/jira/browse/FLUME-3114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16093050#comment-16093050
]
Attila Simon commented on FLUME-3114:
-
Linking related tickets. Please note that both
commons-httpclient:commons-httpclient and org.apache.httpcomponents:httpclient
(new maven group/artifact name) are loaded into flume classpath. Ideal state
would be to depend only this one:
https://mvnrepository.com/artifact/org.apache.httpcomponents/httpclient
Excerpt from dependecy:tree (it appears multiple places in the dep:tree I
copied only a single location)
{noformat}
org.apache.flume.flume-ng-sinks:flume-ng-elasticsearch-sink:jar:1.8.0-SNAPSHOT
+- org.apache.httpcomponents:httpclient:jar:4.2.1:compile
{noformat}
> Upgrade commons-httpclient library dependency
> -
>
> Key: FLUME-3114
> URL: https://issues.apache.org/jira/browse/FLUME-3114
> Project: Flume
> Issue Type: Bug
>Affects Versions: 1.7.0
>Reporter: Attila Simon
>Priority: Critical
> Labels: dependency
> Fix For: 1.8.0
>
>
> ||Group||Artifact||Version used||Upgrade target||
> |commons-httpclient|commons-httpclient|3.1,3.0.1|4.5.2|
> Note: This artifact was moved to:
> * New Group org.apache.httpcomponents
> * New Artifacthttpclient
> Security vulnerability: https://www.cvedetails.com/cve/CVE-2012-5783/
> Please do:
> - double check the newest version.
> - consider to remove a dependency if better alternative is available.
> - check whether the lib change would introduce a backward incompatibility (in
> which case please add this label `breaking_change` and fix version should be
> the next major)
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (FLUME-3121) Upgrade javax.mail:test dependency
Attila Simon created FLUME-3121:
---
Summary: Upgrade javax.mail:test dependency
Key: FLUME-3121
URL: https://issues.apache.org/jira/browse/FLUME-3121
Project: Flume
Issue Type: Bug
Affects Versions: 1.7.0
Reporter: Attila Simon
Priority: Critical
Fix For: 1.8.0
||Group||Artifact||Version used||Upgrade target||
|javax.mail|mail|1.4.1|1.5.6|
Note: This artifact was moved to:
- New Group javax.mail
- New Artifact javax.mail-api
Security vulnerability:
https://www.cvedetails.com/vulnerability-list/vendor_id-5/product_id-5085/SUN-Javamail.html
Note: this might be a false alarm. Please double check the CVE.
Please do:
- double check the newest version.
- consider to remove a dependency if better alternative is available.
- check whether the lib change would introduce a backward incompatibility (in
which case please add this label `breaking_change` and fix version should be
the next major)
Excerpt from mvn dependency:tree
{noformat}
+- org.apache.hive:hive-cli:jar:1.0.0:test
| +- org.apache.hive:hive-service:jar:1.0.0:test
| | +- org.eclipse.jetty.aggregate:jetty-all:jar:7.6.0.v20120127:test
| | | +- javax.mail:mail:jar:1.4.1:test
{noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (FLUME-2977) Upgrade RAT to 0.12
[
https://issues.apache.org/jira/browse/FLUME-2977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16091741#comment-16091741
]
Attila Simon commented on FLUME-2977:
-
RAT-222 is resolved. 0.12 is available :
http://creadur.apache.org/rat/download_rat.cgi
> Upgrade RAT to 0.12
> ---
>
> Key: FLUME-2977
> URL: https://issues.apache.org/jira/browse/FLUME-2977
> Project: Flume
> Issue Type: Improvement
> Components: Build
>Reporter: Attila Simon
>Assignee: Attila Simon
>Priority: Minor
> Fix For: 1.8.0
>
> Attachments: FLUME-2977.patch
>
>
> Before RAT check mvn install prints out the following warnings:
> {noformat}
> Warning: org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser: Property
> 'http://www.oracle.com/xml/jaxp/properties/entityExpansionLimit' is not
> recognized.
> Compiler warnings:
> WARNING: 'org.apache.xerces.jaxp.SAXParserImpl: Property
> 'http://javax.xml.XMLConstants/property/accessExternalDTD' is not recognized.'
> Warning: org.apache.xerces.parsers.SAXParser: Feature
> 'http://javax.xml.XMLConstants/feature/secure-processing' is not recognized.
> Warning: org.apache.xerces.parsers.SAXParser: Property
> 'http://javax.xml.XMLConstants/property/accessExternalDTD' is not recognized.
> Warning: org.apache.xerces.parsers.SAXParser: Property
> 'http://www.oracle.com/xml/jaxp/properties/entityExpansionLimit' is not
> recognized.
> [INFO] Rat check: Summary of files. Unapproved: 0 unknown: 0 generated: 0
> approved: 9 licence.
> {noformat}
> It doesn't break the build but seems misleading.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (FLUME-3114) Upgrade commons-httpclient library dependency
[
https://issues.apache.org/jira/browse/FLUME-3114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16091725#comment-16091725
]
Attila Simon commented on FLUME-3114:
-
Excerpt from dependency:tree
{noformat}
org.apache.flume:flume-ng-auth:jar:1.8.0-SNAPSHOT
+- org.apache.hadoop:hadoop-common:jar:2.4.0:compile
| +- commons-httpclient:commons-httpclient:jar:3.1:compile
{noformat}
> Upgrade commons-httpclient library dependency
> -
>
> Key: FLUME-3114
> URL: https://issues.apache.org/jira/browse/FLUME-3114
> Project: Flume
> Issue Type: Bug
>Affects Versions: 1.7.0
>Reporter: Attila Simon
>Priority: Critical
> Labels: dependency
> Fix For: 1.8.0
>
>
> ||Group||Artifact||Version used||Upgrade target||
> |commons-httpclient|commons-httpclient|3.1,3.0.1|4.5.2|
> Note: This artifact was moved to:
> * New Group org.apache.httpcomponents
> * New Artifacthttpclient
> Security vulnerability: https://www.cvedetails.com/cve/CVE-2012-5783/
> Please do:
> - double check the newest version.
> - consider to remove a dependency if better alternative is available.
> - check whether the lib change would introduce a backward incompatibility (in
> which case please add this label `breaking_change` and fix version should be
> the next major)
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (FLUME-3113) Upgrade commons-beanutils library dependency
[
https://issues.apache.org/jira/browse/FLUME-3113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16091724#comment-16091724
]
Attila Simon commented on FLUME-3113:
-
Excerpt from dependency:tree
{noformat}
org.apache.flume:flume-ng-auth:jar:1.8.0-SNAPSHOT
+- org.apache.hadoop:hadoop-common:jar:2.4.0:compile
| +- commons-configuration:commons-configuration:jar:1.6:compile
| | +- commons-digester:commons-digester:jar:1.8:compile
| | | \- commons-beanutils:commons-beanutils:jar:1.7.0:compile
| | \- commons-beanutils:commons-beanutils-core:jar:1.8.0:compile
{noformat}
> Upgrade commons-beanutils library dependency
>
>
> Key: FLUME-3113
> URL: https://issues.apache.org/jira/browse/FLUME-3113
> Project: Flume
> Issue Type: Bug
>Affects Versions: 1.7.0
>Reporter: Attila Simon
>Priority: Critical
> Labels: dependency
> Fix For: 1.8.0
>
>
> ||Group||Artifact||Version used||Upgrade target||
> |commons-beanutils|commons-beanutils|1.7.0|1.9.3|
> |commons-beanutils|commons-beanutils-core|1.8.0|1.8.3|
> Security vulnerability: https://www.cvedetails.com/cve/CVE-2014-0114/
> Please do:
> - double check the newest version.
> - consider to remove a dependency if better alternative is available.
> - check whether the lib change would introduce a backward incompatibility (in
> which case please add this label `breaking_change` and fix version should be
> the next major)
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (FLUME-3115) Upgrade netty library dependency
Attila Simon created FLUME-3115: --- Summary: Upgrade netty library dependency Key: FLUME-3115 URL: https://issues.apache.org/jira/browse/FLUME-3115 Project: Flume Issue Type: Bug Affects Versions: 1.7.0 Reporter: Attila Simon Priority: Critical Fix For: 1.8.0 ||Group||Artifact||Version used||Upgrade target|| |io.netty|netty|3.2.2.Final, 3.9.4.Final|4.1.12.Final| Note: This artifact was moved to: - New Group io.netty - New Artifact netty-all Security vulnerability: http://www.cvedetails.com/cve/CVE-2014-3488/ Please do: - double check the newest version. - consider to remove a dependency if better alternative is available. - check whether the lib change would introduce a backward incompatibility (in which case please add this label `breaking_change` and fix version should be the next major) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (FLUME-1732) Build is failing due to netty problems
[
https://issues.apache.org/jira/browse/FLUME-1732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16075044#comment-16075044
]
Attila Simon commented on FLUME-1732:
-
Interestingly it seems like it has been committed:
{noformat}
trunk(964bcf56)$ git log --oneline --grep FLUME-1732
750809c7 FLUME-1732: SpoolableDirectorySource should have configurable support
for deleting files it has already completed instead of renaming
{noformat}
Unfortunately this is just a mistake in commit message. That change belongs to
FLUME-1731 instead.
I followed [~mpercy]'s steps from above but no failure for me. I conclude
everything went back to normal since this ticket was opened. So marking this
ticket as resolved.
{noformat}
trunk(964bcf56)$ mvn -version
Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5;
2015-11-10T17:41:47+01:00)
Maven home: /usr/local/Cellar/maven/3.3.9/libexec
Java version: 1.8.0_101, vendor: Oracle Corporation
Java home: /Library/Java/JavaVirtualMachines/jdk1.8.0_101.jdk/Contents/Home/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "mac os x", version: "10.12.5", arch: "x86_64", family: "mac"
{noformat}
> Build is failing due to netty problems
> --
>
> Key: FLUME-1732
> URL: https://issues.apache.org/jira/browse/FLUME-1732
> Project: Flume
> Issue Type: Bug
>Affects Versions: 1.4.0
>Reporter: Brock Noland
>Assignee: Mike Percy
> Attachments: FLUME-1732-3.patch, FLUME-1732.patch
>
>
> FLUME-1723 changed how we bring in netty and that seems to have broken the
> build https://builds.apache.org/job/flume-trunk/330/#showFailuresLink
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (FLUME-2501) Updating HttpClient lib version to ensure compat with Solr
[
https://issues.apache.org/jira/browse/FLUME-2501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Attila Simon updated FLUME-2501:
Labels: dependency (was: )
> Updating HttpClient lib version to ensure compat with Solr
> --
>
> Key: FLUME-2501
> URL: https://issues.apache.org/jira/browse/FLUME-2501
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
>Affects Versions: 1.5.0.1
>Reporter: Roshan Naik
>Assignee: Roshan Naik
> Labels: dependency
> Attachments: FLUME-2501.patch, FLUME-2501.v2.patch
>
>
> Mismatch in httpclient and http core libs pulled by flume v/s the ones that
> come with Solr causes errors at runtime
> {code}
> 2014-10-13 19:52:32,042 (lifecycleSupervisor-1-1) [DEBUG -
> org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:106)]
> Creating new http client,
> config:maxConnections=128=32=false
> 2014-10-13 19:52:32,225 (lifecycleSupervisor-1-1) [ERROR -
> org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:253)]
> Unable to start SinkRunner: {
> policy:org.apache.flume.sink.DefaultSinkProcessor@4752b854 counterGroup:{
> name:null counters:{} } } - Exception follows.
> java.lang.NoSuchFieldError: DEF_CONTENT_CHARSET
> at
> org.apache.http.impl.client.DefaultHttpClient.setDefaultHttpParams(DefaultHttpClient.java:175)
> at
> org.apache.http.impl.client.DefaultHttpClient.createHttpParams(DefaultHttpClient.java:158)
> at
> org.apache.http.impl.client.AbstractHttpClient.getParams(AbstractHttpClient.java:448)
> at
> org.apache.solr.client.solrj.impl.HttpClientUtil.setFollowRedirects(HttpClientUtil.java:251)
> at
> org.apache.solr.client.solrj.impl.HttpClientConfigurer.configure(HttpClientConfigurer.java:58)
> at
> org.apache.solr.client.solrj.impl.HttpClientUtil.configureClient(HttpClientUtil.java:133)
> at
> org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:109)
> at
> org.apache.solr.client.solrj.impl.HttpSolrServer.(HttpSolrServer.java:161)
> at
> org.apache.solr.client.solrj.impl.HttpSolrServer.(HttpSolrServer.java:138)
> at
> org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer.(ConcurrentUpdateSolrServer.java:122)
> at
> org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer.(ConcurrentUpdateSolrServer.java:114)
> at
> org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer.(ConcurrentUpdateSolrServer.java:104)
> at
> org.kitesdk.morphline.solr.SafeConcurrentUpdateSolrServer.(SafeConcurrentUpdateSolrServer.java:39)
> at
> org.kitesdk.morphline.solr.SafeConcurrentUpdateSolrServer.(SafeConcurrentUpdateSolrServer.java:35)
> at
> org.kitesdk.morphline.solr.SolrLocator.getLoader(SolrLocator.java:116)
> at
> org.kitesdk.morphline.solr.LoadSolrBuilder$LoadSolr.(LoadSolrBuilder.java:70)
> at
> org.kitesdk.morphline.solr.LoadSolrBuilder.build(LoadSolrBuilder.java:52)
> at
> org.kitesdk.morphline.base.AbstractCommand.buildCommand(AbstractCommand.java:303)
> at
> org.kitesdk.morphline.base.AbstractCommand.buildCommandChain(AbstractCommand.java:250)
> at org.kitesdk.morphline.stdlib.Pipe.(Pipe.java:46)
> at org.kitesdk.morphline.stdlib.PipeBuilder.build(PipeBuilder.java:40)
> at org.kitesdk.morphline.base.Compiler.compile(Compiler.java:126)
> at org.kitesdk.morphline.base.Compiler.compile(Compiler.java:55)
> at
> org.apache.flume.sink.solr.morphline.MorphlineHandlerImpl.configure(MorphlineHandlerImpl.java:101)
> at
> org.apache.flume.sink.solr.morphline.MorphlineSink.start(MorphlineSink.java:97)
> at
> org.apache.flume.sink.DefaultSinkProcessor.start(DefaultSinkProcessor.java:46)
> at org.apache.flume.SinkRunner.start(SinkRunner.java:79)
> at
> org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (FLUME-3114) Upgrade commons-httpclient library dependency
[ https://issues.apache.org/jira/browse/FLUME-3114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16074995#comment-16074995 ] Attila Simon commented on FLUME-3114: - We have a very similar jira where patch is ready to upgrade to an older than the currently proposed version. The proposed version there doesn't seem to have any CVE yet. > Upgrade commons-httpclient library dependency > - > > Key: FLUME-3114 > URL: https://issues.apache.org/jira/browse/FLUME-3114 > Project: Flume > Issue Type: Bug >Affects Versions: 1.7.0 >Reporter: Attila Simon >Priority: Critical > Labels: dependency > Fix For: 1.8.0 > > > ||Group||Artifact||Version used||Upgrade target|| > |commons-httpclient|commons-httpclient|3.1,3.0.1|4.5.2| > Note: This artifact was moved to: > * New Group org.apache.httpcomponents > * New Artifacthttpclient > Security vulnerability: https://www.cvedetails.com/cve/CVE-2012-5783/ > Please do: > - double check the newest version. > - consider to remove a dependency if better alternative is available. > - check whether the lib change would introduce a backward incompatibility (in > which case please add this label `breaking_change` and fix version should be > the next major) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (FLUME-3114) Upgrade commons-httpclient library dependency
Attila Simon created FLUME-3114: --- Summary: Upgrade commons-httpclient library dependency Key: FLUME-3114 URL: https://issues.apache.org/jira/browse/FLUME-3114 Project: Flume Issue Type: Bug Affects Versions: 1.7.0 Reporter: Attila Simon Priority: Critical Fix For: 1.8.0 ||Group||Artifact||Version used||Upgrade target|| |commons-httpclient|commons-httpclient|3.1,3.0.1|4.5.2| Note: This artifact was moved to: * New Group org.apache.httpcomponents * New Artifact httpclient Security vulnerability: https://www.cvedetails.com/cve/CVE-2012-5783/ Please do: - double check the newest version. - consider to remove a dependency if better alternative is available. - check whether the lib change would introduce a backward incompatibility (in which case please add this label `breaking_change` and fix version should be the next major) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (FLUME-3113) Upgrade commons-beanutils library dependency
[ https://issues.apache.org/jira/browse/FLUME-3113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Simon updated FLUME-3113: Description: ||Group||Artifact||Version used||Upgrade target|| |commons-beanutils|commons-beanutils|1.7.0|1.9.3| |commons-beanutils|commons-beanutils-core|1.8.0|1.8.3| Security vulnerability: https://www.cvedetails.com/cve/CVE-2014-0114/ Please do: - double check the newest version. - consider to remove a dependency if better alternative is available. - check whether the lib change would introduce a backward incompatibility (in which case please add this label `breaking_change` and fix version should be the next major) was: ||Group||Artifact||Version used||Upgrade target|| |commons-beanutils|commons-beanutils|1.7.0|1.9.3| Security vulnerability: https://www.cvedetails.com/cve/CVE-2014-0114/ Please do: - double check the newest version. - consider to remove a dependency if better alternative is available. - check whether the lib change would introduce a backward incompatibility (in which case please add this label `breaking_change` and fix version should be the next major) > Upgrade commons-beanutils library dependency > > > Key: FLUME-3113 > URL: https://issues.apache.org/jira/browse/FLUME-3113 > Project: Flume > Issue Type: Bug >Affects Versions: 1.7.0 >Reporter: Attila Simon >Priority: Critical > Labels: dependency > Fix For: 1.8.0 > > > ||Group||Artifact||Version used||Upgrade target|| > |commons-beanutils|commons-beanutils|1.7.0|1.9.3| > |commons-beanutils|commons-beanutils-core|1.8.0|1.8.3| > Security vulnerability: https://www.cvedetails.com/cve/CVE-2014-0114/ > Please do: > - double check the newest version. > - consider to remove a dependency if better alternative is available. > - check whether the lib change would introduce a backward incompatibility (in > which case please add this label `breaking_change` and fix version should be > the next major) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (FLUME-3112) Upgrade jackson-core library dependency
[
https://issues.apache.org/jira/browse/FLUME-3112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16074953#comment-16074953
]
Attila Simon commented on FLUME-3112:
-
Excerpted transitive dependency tree from `mvn dependency:tree`
{noformat}
org.apache.flume.flume-ng-sinks:flume-dataset-sink:jar:1.8.0-SNAPSHOT
org.kitesdk:kite-data-core:jar:1.0.0:compile
com.fasterxml.jackson.core:jackson-databind:jar:2.3.1:compile
com.fasterxml.jackson.core:jackson-annotations:jar:2.3.0:compile
com.fasterxml.jackson.core:jackson-core:jar:2.3.1:compile
{noformat}
{noformat}
org.apache.flume.flume-ng-sinks:flume-ng-morphline-solr-sink:jar:1.8.0-SNAPSHOT
org.kitesdk:kite-morphlines-all:pom:1.0.0:compile
org.kitesdk:kite-morphlines-json:jar:1.0.0:compile
com.fasterxml.jackson.core:jackson-databind:jar:2.3.1:compile
com.fasterxml.jackson.core:jackson-annotations:jar:2.3.0:compile
com.fasterxml.jackson.core:jackson-core:jar:2.3.1:compile
{noformat}
> Upgrade jackson-core library dependency
> ---
>
> Key: FLUME-3112
> URL: https://issues.apache.org/jira/browse/FLUME-3112
> Project: Flume
> Issue Type: Bug
>Affects Versions: 1.7.0
>Reporter: Attila Simon
>Priority: Critical
> Labels: dependency
> Fix For: 1.8.0
>
>
> ||Group||Artifact||Version used||Upgrade target||
> |com.fasterxml.jackson.core|jackson-core|2.3.1|2.8.9|
> Security vulnerability: http://www.cvedetails.com/cve/CVE-2016-7051/
> Please do:
> - double check the newest version.
> - consider to remove a dependency if better alternative is available.
> - check whether the lib change would introduce a backward incompatibility (in
> which case please add this label `breaking_change` and fix version should be
> the next major)
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (FLUME-3113) Upgrade commons-beanutils library dependency
Attila Simon created FLUME-3113: --- Summary: Upgrade commons-beanutils library dependency Key: FLUME-3113 URL: https://issues.apache.org/jira/browse/FLUME-3113 Project: Flume Issue Type: Bug Affects Versions: 1.7.0 Reporter: Attila Simon Priority: Critical Fix For: 1.8.0 ||Group||Artifact||Version used||Upgrade target|| |commons-beanutils|commons-beanutils|1.7.0|1.9.3| Security vulnerability: https://www.cvedetails.com/cve/CVE-2014-0114/ Please do: - double check the newest version. - consider to remove a dependency if better alternative is available. - check whether the lib change would introduce a backward incompatibility (in which case please add this label `breaking_change` and fix version should be the next major) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (FLUME-3112) Upgrade jackson-core library dependency
Attila Simon created FLUME-3112: --- Summary: Upgrade jackson-core library dependency Key: FLUME-3112 URL: https://issues.apache.org/jira/browse/FLUME-3112 Project: Flume Issue Type: Bug Affects Versions: 1.7.0 Reporter: Attila Simon Priority: Critical Fix For: 1.8.0 ||Group||Artifact||Version used||Upgrade target|| |com.fasterxml.jackson.core|jackson-core|2.3.1|2.8.9| Security vulnerability: http://www.cvedetails.com/cve/CVE-2016-7051/ Please do: - double check the newest version. - consider to remove a dependency if better alternative is available. - check whether the lib change would introduce a backward incompatibility (in which case please add this label `breaking_change` and fix version should be the next major) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (FLUME-2752) Flume AvroSource will leak the memory and the OOM will be happened.
[
https://issues.apache.org/jira/browse/FLUME-2752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16066342#comment-16066342
]
Attila Simon commented on FLUME-2752:
-
I experienced similar issues on startup. If there is any exception during
startup then the resources won't be released (memory, threadpool, open files,
etc). I have an implementation which conforms with the proposed idea here: call
this.stop() if there were any error during startup and make sure that
this.stop() releases the resources. I plan to open a pull request for this
change soon.
> Flume AvroSource will leak the memory and the OOM will be happened.
> ---
>
> Key: FLUME-2752
> URL: https://issues.apache.org/jira/browse/FLUME-2752
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
>Affects Versions: 1.6.0
>Reporter: yinghua_zh
>Assignee: Attila Simon
>
> If the flume agent config the nonexist IP for the avro source,the exception
> will be happened as follow:
> 2015-07-21 19:57:47,054 | ERROR | [lifecycleSupervisor-1-2] | Unable to
> start EventDrivenSourceRunner: { source:Avro source avro_source_21155: {
> bindAddress: 51.196.27.32, port: 21155 } } - Exception follows. |
> org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:253)
> org.jboss.netty.channel.ChannelException: Failed to bind to:
> /51.196.27.32:21155
> at
> org.jboss.netty.bootstrap.ServerBootstrap.bind(ServerBootstrap.java:297)
> at org.apache.avro.ipc.NettyServer.(NettyServer.java:106)
> at org.apache.flume.source.AvroSource.start(AvroSource.java:294)
> at
> org.apache.flume.source.EventDrivenSourceRunner.start(EventDrivenSourceRunner.java:44)
> at
> org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.net.BindException: Cannot assign requested address
> at sun.nio.ch.Net.bind0(Native Method)
> at sun.nio.ch.Net.bind(Net.java:437)
> at sun.nio.ch.Net.bind(Net.java:429)
> at
> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
> at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
> at
> org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.bind(NioServerSocketPipelineSink.java:140)
> at
> org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.handleServerSocket(NioServerSocketPipelineSink.java:90)
> at
> org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.eventSunk(NioServerSocketPipelineSink.java:64)
> at org.jboss.netty.channel.Channels.bind(Channels.java:569)
> at
> org.jboss.netty.channel.AbstractChannel.bind(AbstractChannel.java:189)
> at
> org.jboss.netty.bootstrap.ServerBootstrap$Binder.channelOpen(ServerBootstrap.java:342)
> at org.jboss.netty.channel.Channels.fireChannelOpen(Channels.java:170)
> at
> org.jboss.netty.channel.socket.nio.NioServerSocketChannel.(NioServerSocketChannel.java:80)
> at
> org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.newChannel(NioServerSocketChannelFactory.java:158)
> at
> org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.newChannel(NioServerSocketChannelFactory.java:86)
> at
> org.jboss.netty.bootstrap.ServerBootstrap.bind(ServerBootstrap.java:276)
> if the above exception happened for 2 hours,and the agent JVM -Xxx is 4G,the
> OutOfMemory will be happened.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (FLUME-2752) Flume AvroSource will leak the memory and the OOM will be happened.
[
https://issues.apache.org/jira/browse/FLUME-2752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Attila Simon updated FLUME-2752:
Summary: Flume AvroSource will leak the memory and the OOM will be
happened. (was: Flume AvroSoucr will leak the memory and the OOM will be
happened.)
> Flume AvroSource will leak the memory and the OOM will be happened.
> ---
>
> Key: FLUME-2752
> URL: https://issues.apache.org/jira/browse/FLUME-2752
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
>Affects Versions: 1.6.0
>Reporter: yinghua_zh
>Assignee: Attila Simon
>
> If the flume agent config the nonexist IP for the avro source,the exception
> will be happened as follow:
> 2015-07-21 19:57:47,054 | ERROR | [lifecycleSupervisor-1-2] | Unable to
> start EventDrivenSourceRunner: { source:Avro source avro_source_21155: {
> bindAddress: 51.196.27.32, port: 21155 } } - Exception follows. |
> org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:253)
> org.jboss.netty.channel.ChannelException: Failed to bind to:
> /51.196.27.32:21155
> at
> org.jboss.netty.bootstrap.ServerBootstrap.bind(ServerBootstrap.java:297)
> at org.apache.avro.ipc.NettyServer.(NettyServer.java:106)
> at org.apache.flume.source.AvroSource.start(AvroSource.java:294)
> at
> org.apache.flume.source.EventDrivenSourceRunner.start(EventDrivenSourceRunner.java:44)
> at
> org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.net.BindException: Cannot assign requested address
> at sun.nio.ch.Net.bind0(Native Method)
> at sun.nio.ch.Net.bind(Net.java:437)
> at sun.nio.ch.Net.bind(Net.java:429)
> at
> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
> at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
> at
> org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.bind(NioServerSocketPipelineSink.java:140)
> at
> org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.handleServerSocket(NioServerSocketPipelineSink.java:90)
> at
> org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.eventSunk(NioServerSocketPipelineSink.java:64)
> at org.jboss.netty.channel.Channels.bind(Channels.java:569)
> at
> org.jboss.netty.channel.AbstractChannel.bind(AbstractChannel.java:189)
> at
> org.jboss.netty.bootstrap.ServerBootstrap$Binder.channelOpen(ServerBootstrap.java:342)
> at org.jboss.netty.channel.Channels.fireChannelOpen(Channels.java:170)
> at
> org.jboss.netty.channel.socket.nio.NioServerSocketChannel.(NioServerSocketChannel.java:80)
> at
> org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.newChannel(NioServerSocketChannelFactory.java:158)
> at
> org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.newChannel(NioServerSocketChannelFactory.java:86)
> at
> org.jboss.netty.bootstrap.ServerBootstrap.bind(ServerBootstrap.java:276)
> if the above exception happened for 2 hours,and the agent JVM -Xxx is 4G,the
> OutOfMemory will be happened.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Assigned] (FLUME-2752) Flume AvroSoucr will leak the memory and the OOM will be happened.
[
https://issues.apache.org/jira/browse/FLUME-2752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Attila Simon reassigned FLUME-2752:
---
Assignee: Attila Simon
> Flume AvroSoucr will leak the memory and the OOM will be happened.
> --
>
> Key: FLUME-2752
> URL: https://issues.apache.org/jira/browse/FLUME-2752
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
>Affects Versions: 1.6.0
>Reporter: yinghua_zh
>Assignee: Attila Simon
>
> If the flume agent config the nonexist IP for the avro source,the exception
> will be happened as follow:
> 2015-07-21 19:57:47,054 | ERROR | [lifecycleSupervisor-1-2] | Unable to
> start EventDrivenSourceRunner: { source:Avro source avro_source_21155: {
> bindAddress: 51.196.27.32, port: 21155 } } - Exception follows. |
> org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:253)
> org.jboss.netty.channel.ChannelException: Failed to bind to:
> /51.196.27.32:21155
> at
> org.jboss.netty.bootstrap.ServerBootstrap.bind(ServerBootstrap.java:297)
> at org.apache.avro.ipc.NettyServer.(NettyServer.java:106)
> at org.apache.flume.source.AvroSource.start(AvroSource.java:294)
> at
> org.apache.flume.source.EventDrivenSourceRunner.start(EventDrivenSourceRunner.java:44)
> at
> org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.net.BindException: Cannot assign requested address
> at sun.nio.ch.Net.bind0(Native Method)
> at sun.nio.ch.Net.bind(Net.java:437)
> at sun.nio.ch.Net.bind(Net.java:429)
> at
> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
> at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
> at
> org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.bind(NioServerSocketPipelineSink.java:140)
> at
> org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.handleServerSocket(NioServerSocketPipelineSink.java:90)
> at
> org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.eventSunk(NioServerSocketPipelineSink.java:64)
> at org.jboss.netty.channel.Channels.bind(Channels.java:569)
> at
> org.jboss.netty.channel.AbstractChannel.bind(AbstractChannel.java:189)
> at
> org.jboss.netty.bootstrap.ServerBootstrap$Binder.channelOpen(ServerBootstrap.java:342)
> at org.jboss.netty.channel.Channels.fireChannelOpen(Channels.java:170)
> at
> org.jboss.netty.channel.socket.nio.NioServerSocketChannel.(NioServerSocketChannel.java:80)
> at
> org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.newChannel(NioServerSocketChannelFactory.java:158)
> at
> org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.newChannel(NioServerSocketChannelFactory.java:86)
> at
> org.jboss.netty.bootstrap.ServerBootstrap.bind(ServerBootstrap.java:276)
> if the above exception happened for 2 hours,and the agent JVM -Xxx is 4G,the
> OutOfMemory will be happened.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (FLUME-2905) NetcatSource - Socket not closed when an exception is encountered during start() leading to file descriptor leaks
[
https://issues.apache.org/jira/browse/FLUME-2905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16059612#comment-16059612
]
Attila Simon commented on FLUME-2905:
-
It looks good, thanks! You can check others eg:
https://reviews.apache.org/r/48161/
I also left some comments there (you can upload your corrections there, I guess
attaching the final version of the patch here would be enough)
> NetcatSource - Socket not closed when an exception is encountered during
> start() leading to file descriptor leaks
> -
>
> Key: FLUME-2905
> URL: https://issues.apache.org/jira/browse/FLUME-2905
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
>Affects Versions: 1.6.0
>Reporter: Siddharth Ahuja
>Assignee: Siddharth Ahuja
> Attachments: FLUME-2905-0.patch, FLUME-2905-1.patch,
> FLUME-2905-2.patch, FLUME-2905-3.patch, FLUME-2905-4.patch, FLUME-2905-5.patch
>
>
> During the flume agent start-up, the flume configuration containing the
> NetcatSource is parsed and the source's start() is called. If there is an
> issue while binding the channel's socket to a local address to configure the
> socket to listen for connections following exception is thrown but the socket
> open just before is not closed.
> {code}
> 2016-05-01 03:04:37,273 ERROR org.apache.flume.lifecycle.LifecycleSupervisor:
> Unable to start EventDrivenSourceRunner: {
> source:org.apache.flume.source.NetcatSource{name:src-1,state:IDLE} } -
> Exception follows.
> org.apache.flume.FlumeException: java.net.BindException: Address already in
> use
> at org.apache.flume.source.NetcatSource.start(NetcatSource.java:173)
> at
> org.apache.flume.source.EventDrivenSourceRunner.start(EventDrivenSourceRunner.java:44)
> at
> org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.net.BindException: Address already in use
> at sun.nio.ch.Net.bind0(Native Method)
> at sun.nio.ch.Net.bind(Net.java:444)
> at sun.nio.ch.Net.bind(Net.java:436)
> at
> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)
> at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
> at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:67)
> at org.apache.flume.source.NetcatSource.start(NetcatSource.java:167)
> ... 9 more
> {code}
> The source's start() is then called again leading to another socket being
> opened but not closed and so on. This leads to file descriptor (socket) leaks.
> This can be easily reproduced as follows:
> 1. Set Netcat as the source in flume agent configuration.
> 2. Set the bind port for the netcat source to a port which is already in use.
> e.g. in my case I used 50010 which is the port for DataNode's XCeiver
> Protocol in use by the HDFS service.
> 3. Start flume agent and perform "lsof -p | wc -l". Notice
> the file descriptors keep on growing due to socket leaks with errors like:
> "can't identify protocol".
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (FLUME-2905) NetcatSource - Socket not closed when an exception is encountered during start() leading to file descriptor leaks
[
https://issues.apache.org/jira/browse/FLUME-2905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16058287#comment-16058287
]
Attila Simon commented on FLUME-2905:
-
Hi [~sahuja],
Junit passed for me but hasn't run exhaustively yet. I checked the
FLUME-2905-4.patch and would have some minor suggestions. Would you mind
uploading to https://reviews.apache.org/ for a code review? Please let me know
if you have trouble with it (I'm also happy to upload it there for you).
In a nutshell:
* I would recommend calling stop() after writing out the exception.
* I would recommend removing the return statement from that part of the stop
function which catches an Exception from Socket close. That currently prevents
the rest of the stop() to be executed including super.stop() which then can't
set the Lifecycle state.
* Test currently checks that stop was executed properly expecting the source to
be in LifecycleState.STOP state. I think that is fine since it is internal to
the NetcatSource that it opens a ServerSocket. What we have to check whether
the Source is stopped on error. And this is indeed checked by the test.
As a side note: It seems like that startSource() doesn't behave as expected
(should pick a new port on error). This is a separate issue so I guess it
should be fixed with a separate jira/PR.
> NetcatSource - Socket not closed when an exception is encountered during
> start() leading to file descriptor leaks
> -
>
> Key: FLUME-2905
> URL: https://issues.apache.org/jira/browse/FLUME-2905
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
>Affects Versions: 1.6.0
>Reporter: Siddharth Ahuja
>Assignee: Siddharth Ahuja
> Attachments: FLUME-2905-0.patch, FLUME-2905-1.patch,
> FLUME-2905-2.patch, FLUME-2905-3.patch, FLUME-2905-4.patch
>
>
> During the flume agent start-up, the flume configuration containing the
> NetcatSource is parsed and the source's start() is called. If there is an
> issue while binding the channel's socket to a local address to configure the
> socket to listen for connections following exception is thrown but the socket
> open just before is not closed.
> {code}
> 2016-05-01 03:04:37,273 ERROR org.apache.flume.lifecycle.LifecycleSupervisor:
> Unable to start EventDrivenSourceRunner: {
> source:org.apache.flume.source.NetcatSource{name:src-1,state:IDLE} } -
> Exception follows.
> org.apache.flume.FlumeException: java.net.BindException: Address already in
> use
> at org.apache.flume.source.NetcatSource.start(NetcatSource.java:173)
> at
> org.apache.flume.source.EventDrivenSourceRunner.start(EventDrivenSourceRunner.java:44)
> at
> org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.net.BindException: Address already in use
> at sun.nio.ch.Net.bind0(Native Method)
> at sun.nio.ch.Net.bind(Net.java:444)
> at sun.nio.ch.Net.bind(Net.java:436)
> at
> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)
> at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
> at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:67)
> at org.apache.flume.source.NetcatSource.start(NetcatSource.java:167)
> ... 9 more
> {code}
> The source's start() is then called again leading to another socket being
> opened but not closed and so on. This leads to file descriptor (socket) leaks.
> This can be easily reproduced as follows:
> 1. Set Netcat as the source in flume agent configuration.
> 2. Set the bind port for the netcat source to a port which is already in use.
> e.g. in my case I used 50010 which is the port for DataNode's XCeiver
> Protocol in use by the HDFS service.
> 3. Start flume agent and perform "lsof -p | wc -l". Notice
> the file descriptors keep on growing due to socket leaks with errors like:
> "can't identify protocol".
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (FLUME-3050) add error stats to monitor URL
[
https://issues.apache.org/jira/browse/FLUME-3050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15892039#comment-15892039
]
Attila Simon commented on FLUME-3050:
-
Hi [~yuvalif],
Thanks for your comments. Most of these metrics indeed look useful. I hope it
will get picked soon. I think however picks it up can discuss further which
ones are possible/makes sense and do the implementation based on that agreement.
> add error stats to monitor URL
> --
>
> Key: FLUME-3050
> URL: https://issues.apache.org/jira/browse/FLUME-3050
> Project: Flume
> Issue Type: Improvement
> Components: Channel, Shell, Sinks+Sources
>Affects Versions: v1.7.0
>Reporter: Yuval Lifshitz
> Labels: features
>
> currently error counters are not present when getting stats. for example:
> {code}
> > curl http://my-flume-host:4/metrics
> {"SINK.k1":{"ConnectionCreatedCount":"1","ConnectionClosedCount":"0","Type":"SINK","BatchCompleteCount":"0","BatchEmptyCount":"4","EventDrainAttemptCount":"10","StartTime":"1485348138992","EventDrainSuccessCount":"10","BatchUnderflowCount":"1","StopTime":"0","ConnectionFailedCount":"0"},"CHANNEL.c1":{"ChannelCapacity":"100","ChannelFillPercentage":"0.0","Type":"CHANNEL","ChannelSize":"0","EventTakeSuccessCount":"10","EventTakeAttemptCount":"15","StartTime":"1485348138990","EventPutAttemptCount":"10","EventPutSuccessCount":"10","StopTime":"0"},"SOURCE.r1":{"EventReceivedCount":"10","AppendBatchAcceptedCount":"0","Type":"SOURCE","AppendReceivedCount":"0","EventAcceptedCount":"10","StartTime":"1485348138993","AppendAcceptedCount":"0","OpenConnectionCount":"0","AppendBatchReceivedCount":"0","StopTime":"0"}}
> {code}
> return only "good" stats for source, channel and sink.
> to get error you need to look into the log file. this makes it hard to
> integrate flume into automatic monitoring systems, NMS etc.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (FLUME-3050) add error stats to monitor URL
[
https://issues.apache.org/jira/browse/FLUME-3050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15878576#comment-15878576
]
Attila Simon commented on FLUME-3050:
-
Hi [~yuvalif],
Could you please mention some example counters (or the full list would be even
more helpful) you expect to see?
> add error stats to monitor URL
> --
>
> Key: FLUME-3050
> URL: https://issues.apache.org/jira/browse/FLUME-3050
> Project: Flume
> Issue Type: Improvement
> Components: Channel, Shell, Sinks+Sources
>Affects Versions: v1.7.0
>Reporter: Yuval Lifshitz
> Labels: features
>
> currently error counters are not present when getting stats. for example:
> {code}
> > curl http://my-flume-host:4/metrics
> {"SINK.k1":{"ConnectionCreatedCount":"1","ConnectionClosedCount":"0","Type":"SINK","BatchCompleteCount":"0","BatchEmptyCount":"4","EventDrainAttemptCount":"10","StartTime":"1485348138992","EventDrainSuccessCount":"10","BatchUnderflowCount":"1","StopTime":"0","ConnectionFailedCount":"0"},"CHANNEL.c1":{"ChannelCapacity":"100","ChannelFillPercentage":"0.0","Type":"CHANNEL","ChannelSize":"0","EventTakeSuccessCount":"10","EventTakeAttemptCount":"15","StartTime":"1485348138990","EventPutAttemptCount":"10","EventPutSuccessCount":"10","StopTime":"0"},"SOURCE.r1":{"EventReceivedCount":"10","AppendBatchAcceptedCount":"0","Type":"SOURCE","AppendReceivedCount":"0","EventAcceptedCount":"10","StartTime":"1485348138993","AppendAcceptedCount":"0","OpenConnectionCount":"0","AppendBatchReceivedCount":"0","StopTime":"0"}}
> {code}
> return only "good" stats for source, channel and sink.
> to get error you need to look into the log file. this makes it hard to
> integrate flume into automatic monitoring systems, NMS etc.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (FLUME-3032) taildir source sleeps frequently.
[
https://issues.apache.org/jira/browse/FLUME-3032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15767508#comment-15767508
]
Attila Simon commented on FLUME-3032:
-
I would refactor {{tailFileProcess()}} to a separate class which would get its
dependencies via constructor and define this function as part of its public
API. That class could be tested as a unit by mocking its constructor params and
see whether all the events was passed to the channel just by invoking the
{{tailFileProcess()}} once. Mock which will be needed:
{{getChannelProcessor().processEventBatch()}} should also modify the number of
{{events}} beside count how many arrived (to imitate an interceptor which
dropped one and also to verify that all events arrived). I think mocking the
{{reader}} to produce {{events}} simply would be a convenient choice over
setting up temporal files. I know this would be a relatively bigger change
compared to your existing patch but would make an improvement on code quality.
If you think you need help with it I'm happy to answer your questions or
provide more details.
Also another option would be to test this functionality via the public
{{process()}} function by mocking all the relevant objects as written above
plus the ones required by {{process()}} itself to work.
There might be other options I guess every idea is welcomed.
Just double checked and you missed a space before '=' here: {{long
receivedSize= 0;}}
and missed a semicolon here: {{receivedSize = events.size()}}
> taildir source sleeps frequently.
> -
>
> Key: FLUME-3032
> URL: https://issues.apache.org/jira/browse/FLUME-3032
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
>Affects Versions: v1.7.0
> Environment: CentOS Linux release 7.2.1511 (Core)
> java version "1.7.0_80"
>Reporter: Liu Tianhao
> Labels: newbie
> Attachments: FLUME-3032.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> Test configuration.
> source - taildir
> interceptor - The custom interceptor drops some events
> channel - anyone
> sink - none
> I found that taildir source sleeps frequently.
> The tailFileProcess() function in TaildirSource.java break the loop by
> (events.size() < batchSize), but interceptor may change events.size().
> I think the events.size() should be used before interceptor processing.
> Avoid unnecessary sleep.
> {code:title=TaildirSource.java|borderStyle=solid}
> private void tailFileProcess(TailFile tf, boolean backoffWithoutNL)
> throws IOException, InterruptedException {
> long receivedSize = 0;
> while (true) {
> reader.setCurrentFile(tf);
> List events = reader.readEvents(batchSize,
> backoffWithoutNL);
> if (events.isEmpty()) {
> break;
> }
> receivedSize = events.size();
> sourceCounter.addToEventReceivedCount(receivedSize);
> sourceCounter.incrementAppendBatchReceivedCount();
> try {
> getChannelProcessor().processEventBatch(events);
> reader.commit();
> } catch (ChannelException ex) {
> logger.warn("The channel is full or unexpected failure. "
> + "The source will try again after " + retryInterval
> + " ms");
> TimeUnit.MILLISECONDS.sleep(retryInterval);
> retryInterval = retryInterval << 1;
> retryInterval = Math.min(retryInterval, maxRetryInterval);
> continue;
> }
> retryInterval = 1000;
> sourceCounter.addToEventAcceptedCount(events.size());
> sourceCounter.incrementAppendBatchAcceptedCount();
> if (receivedSize < batchSize) {
> break;
> }
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLUME-2988) Kafka Sink metrics missing eventDrainAttemptCount
[
https://issues.apache.org/jira/browse/FLUME-2988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15726404#comment-15726404
]
Attila Simon commented on FLUME-2988:
-
Hi [~udai],
Sorry for not getting back to you earlier. The change looks good with two small
corrections:
- as [~denes] pointed out Long.valueOf is not needed
- I would recommend adding this to closer to the top of process() function eg:
where we know for sure that we have a reference to an event from the
channel.take() (== where flume first attempt to drain an event and knows that
the channel was not empty). More specifically in an else path at
https://github.com/apache/flume/blob/trunk/flume-ng-sinks/flume-ng-kafka-sink/src/main/java/org/apache/flume/sink/kafka/KafkaSink.java#L169
I think if you open a pull request for this on https://github.com/apache/flume
would help the reviewing process.
> Kafka Sink metrics missing eventDrainAttemptCount
> -
>
> Key: FLUME-2988
> URL: https://issues.apache.org/jira/browse/FLUME-2988
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
>Reporter: Denes Arvay
>Assignee: Udai Kiran Potluri
>Priority: Minor
> Attachments: FLUME-2988.1.patch
>
>
> {{eventDrainAttemptCount}} doesn't get incremented in Kafka Sink, only the
> {{eventDrainSuccessCount}}
> (https://github.com/apache/flume/blob/trunk/flume-ng-sinks/flume-ng-kafka-sink/src/main/java/org/apache/flume/sink/kafka/KafkaSink.java#L210)
> resulting in misleading statistics (e.g. when calculating {{attemptCount -
> successCount}})
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLUME-3032) taildir source sleeps frequently.
[
https://issues.apache.org/jira/browse/FLUME-3032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15699547#comment-15699547
]
Attila Simon commented on FLUME-3032:
-
Nice finding and the change also looks good to me. Would you consider adding a
junit test to cover this change?
> taildir source sleeps frequently.
> -
>
> Key: FLUME-3032
> URL: https://issues.apache.org/jira/browse/FLUME-3032
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
>Affects Versions: v1.7.0
> Environment: CentOS Linux release 7.2.1511 (Core)
> java version "1.7.0_80"
>Reporter: Liu Tianhao
> Labels: newbie
> Attachments: FLUME-3032.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> Test configuration.
> source - taildir
> interceptor - The custom interceptor drops some events
> channel - anyone
> sink - none
> I found that taildir source sleeps frequently.
> The tailFileProcess() function in TaildirSource.java break the loop by
> (events.size() < batchSize), but interceptor may change events.size().
> I think the events.size() should be used before interceptor processing.
> Avoid unnecessary sleep.
> {code:title=TaildirSource.java|borderStyle=solid}
> private void tailFileProcess(TailFile tf, boolean backoffWithoutNL)
> throws IOException, InterruptedException {
> long receivedSize = 0;
> while (true) {
> reader.setCurrentFile(tf);
> List events = reader.readEvents(batchSize,
> backoffWithoutNL);
> if (events.isEmpty()) {
> break;
> }
> receivedSize = events.size();
> sourceCounter.addToEventReceivedCount(receivedSize);
> sourceCounter.incrementAppendBatchReceivedCount();
> try {
> getChannelProcessor().processEventBatch(events);
> reader.commit();
> } catch (ChannelException ex) {
> logger.warn("The channel is full or unexpected failure. "
> + "The source will try again after " + retryInterval
> + " ms");
> TimeUnit.MILLISECONDS.sleep(retryInterval);
> retryInterval = retryInterval << 1;
> retryInterval = Math.min(retryInterval, maxRetryInterval);
> continue;
> }
> retryInterval = 1000;
> sourceCounter.addToEventAcceptedCount(events.size());
> sourceCounter.incrementAppendBatchAcceptedCount();
> if (receivedSize < batchSize) {
> break;
> }
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (FLUME-3031) sequence source should reset its counter for event body on channel exception
Attila Simon created FLUME-3031: --- Summary: sequence source should reset its counter for event body on channel exception Key: FLUME-3031 URL: https://issues.apache.org/jira/browse/FLUME-3031 Project: Flume Issue Type: Bug Components: Sinks+Sources, Test Affects Versions: v1.7.0 Reporter: Attila Simon Assignee: Attila Simon SequenceGeneratorSource uses a counter to construct the body of the generated Events which counter is not reseted when writing event to channel failed. This can lead to a situation that the total number of unique events at destination (if deduplication relies on msg body) is bigger than the totalEvents configuration parameter due to the retries. Fix should make sure that number of events at destination after filtering out the duplicates is equal to the configured value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (FLUME-3015) Fix flaky org.apache.flume.channel.jdbc.TestJdbcChannelProviderNoFK.testEventWithSimulatedSourceAndSinks
[
https://issues.apache.org/jira/browse/FLUME-3015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15595616#comment-15595616
]
Attila Simon edited comment on FLUME-3015 at 10/21/16 4:54 PM:
---
TestJdbcChannelProvider.testEventWithSimulatedSourceAndSinks might also be
flaky. If you found the root cause for this ticket please double check whether
it also applicable to
TestJdbcChannelProvider.testEventWithSimulatedSourceAndSinks. Stack traces and
exception is the similar to NoFK.
was (Author: sati):
TestJdbcChannelProvider.testEventWithSimulatedSourceAndSinks might also be
flaky. If you found the root cause for this ticket please double check whether
it also applicable to
TestJdbcChannelProvider.testEventWithSimulatedSourceAndSinks
> Fix flaky
> org.apache.flume.channel.jdbc.TestJdbcChannelProviderNoFK.testEventWithSimulatedSourceAndSinks
>
>
> Key: FLUME-3015
> URL: https://issues.apache.org/jira/browse/FLUME-3015
> Project: Flume
> Issue Type: Bug
> Components: Test
>Affects Versions: v1.7.0
>Reporter: Attila Simon
>
> testEventWithSimulatedSourceAndSinks(org.apache.flume.channel.jdbc.TestJdbcChannelProviderNoFK):
> org.apache.flume.channel.jdbc.JdbcChannelException: Failed to persist event
> {noformat}
>
> testEventWithSimulatedSourceAndSinks(org.apache.flume.channel.jdbc.TestJdbcChannelProviderNoFK)
> Time elapsed: 310978 sec <<< ERROR!
> java.util.concurrent.ExecutionException:
> org.apache.flume.channel.jdbc.JdbcChannelException: Failed to persist event
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:188)
> at
> org.apache.flume.channel.jdbc.BaseJdbcChannelProviderTest.testEventWithSimulatedSourceAndSinks(BaseJdbcChannelProviderTest.java:202)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
> at
> org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
> at
> org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)
> Caused by: org.apache.flume.channel.jdbc.JdbcChannelException: Failed to
> persist event
> at
>
[jira] [Created] (FLUME-3017) Fix flaky org.apache.flume.sink.hbase.TestHBaseSink
Attila Simon created FLUME-3017:
---
Summary: Fix flaky org.apache.flume.sink.hbase.TestHBaseSink
Key: FLUME-3017
URL: https://issues.apache.org/jira/browse/FLUME-3017
Project: Flume
Issue Type: Bug
Components: Test
Affects Versions: v1.7.0
Reporter: Attila Simon
{noformat}
ERROR [main] hbase.MiniHBaseCluster (MiniHBaseCluster.java:init(230)) - Error
starting cluster
java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/procedure2/Procedure
at java.lang.Class.getDeclaredConstructors0(Native Method)
at java.lang.Class.privateGetDeclaredConstructors(Class.java:2671)
at java.lang.Class.getConstructor0(Class.java:3075)
at java.lang.Class.getConstructor(Class.java:1825)
at
org.apache.hadoop.hbase.util.JVMClusterUtil.createMasterThread(JVMClusterUtil.java:139)
at
org.apache.hadoop.hbase.LocalHBaseCluster.addMaster(LocalHBaseCluster.java:219)
at
org.apache.hadoop.hbase.LocalHBaseCluster.(LocalHBaseCluster.java:155)
at
org.apache.hadoop.hbase.MiniHBaseCluster.init(MiniHBaseCluster.java:214)
at
org.apache.hadoop.hbase.MiniHBaseCluster.(MiniHBaseCluster.java:94)
at
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniHBaseCluster(HBaseTestingUtility.java:1036)
at
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:996)
at
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:868)
at
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:862)
at
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:806)
at
org.apache.flume.sink.hbase.TestHBaseSink.setUpOnce(TestHBaseSink.java:81)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:27)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30)
at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
at
org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
at
org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
at
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)
at
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)
Caused by: java.lang.ClassNotFoundException:
org.apache.hadoop.hbase.procedure2.Procedure
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 37 more
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (FLUME-3016) Fix flaky org.apache.flume.sink.hbase.TestAsyncHBaseSink
Attila Simon created FLUME-3016:
---
Summary: Fix flaky org.apache.flume.sink.hbase.TestAsyncHBaseSink
Key: FLUME-3016
URL: https://issues.apache.org/jira/browse/FLUME-3016
Project: Flume
Issue Type: Bug
Components: Test
Affects Versions: v1.7.0
Reporter: Attila Simon
{noformat}
2016-09-26 05:11:33,321 ERROR [main] hbase.MiniHBaseCluster
(MiniHBaseCluster.java:init(230)) - Error starting cluster
java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/procedure2/Procedure
at java.lang.Class.getDeclaredConstructors0(Native Method)
at java.lang.Class.privateGetDeclaredConstructors(Class.java:2671)
at java.lang.Class.getConstructor0(Class.java:3075)
at java.lang.Class.getConstructor(Class.java:1825)
at
org.apache.hadoop.hbase.util.JVMClusterUtil.createMasterThread(JVMClusterUtil.java:139)
at
org.apache.hadoop.hbase.LocalHBaseCluster.addMaster(LocalHBaseCluster.java:219)
at
org.apache.hadoop.hbase.LocalHBaseCluster.(LocalHBaseCluster.java:155)
at
org.apache.hadoop.hbase.MiniHBaseCluster.init(MiniHBaseCluster.java:214)
at
org.apache.hadoop.hbase.MiniHBaseCluster.(MiniHBaseCluster.java:94)
at
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniHBaseCluster(HBaseTestingUtility.java:1036)
at
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:996)
at
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:868)
at
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:862)
at
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:806)
at
org.apache.flume.sink.hbase.TestAsyncHBaseSink.setUp(TestAsyncHBaseSink.java:73)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:27)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30)
at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
at
org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
at
org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
at
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)
at
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)
Caused by: java.lang.ClassNotFoundException:
org.apache.hadoop.hbase.procedure2.Procedure
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLUME-3015) Fix flaky org.apache.flume.channel.jdbc.TestJdbcChannelProviderNoFK.testEventWithSimulatedSourceAndSinks
[
https://issues.apache.org/jira/browse/FLUME-3015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15595616#comment-15595616
]
Attila Simon commented on FLUME-3015:
-
TestJdbcChannelProvider.testEventWithSimulatedSourceAndSinks might also be
flaky. If you found the root cause for this ticket please double check whether
it also applicable to
TestJdbcChannelProvider.testEventWithSimulatedSourceAndSinks
> Fix flaky
> org.apache.flume.channel.jdbc.TestJdbcChannelProviderNoFK.testEventWithSimulatedSourceAndSinks
>
>
> Key: FLUME-3015
> URL: https://issues.apache.org/jira/browse/FLUME-3015
> Project: Flume
> Issue Type: Bug
> Components: Test
>Affects Versions: v1.7.0
>Reporter: Attila Simon
>
> testEventWithSimulatedSourceAndSinks(org.apache.flume.channel.jdbc.TestJdbcChannelProviderNoFK):
> org.apache.flume.channel.jdbc.JdbcChannelException: Failed to persist event
> {noformat}
>
> testEventWithSimulatedSourceAndSinks(org.apache.flume.channel.jdbc.TestJdbcChannelProviderNoFK)
> Time elapsed: 310978 sec <<< ERROR!
> java.util.concurrent.ExecutionException:
> org.apache.flume.channel.jdbc.JdbcChannelException: Failed to persist event
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:188)
> at
> org.apache.flume.channel.jdbc.BaseJdbcChannelProviderTest.testEventWithSimulatedSourceAndSinks(BaseJdbcChannelProviderTest.java:202)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
> at
> org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
> at
> org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)
> Caused by: org.apache.flume.channel.jdbc.JdbcChannelException: Failed to
> persist event
> at
> org.apache.flume.channel.jdbc.impl.JdbcChannelProviderImpl.persistEvent(JdbcChannelProviderImpl.java:312)
> at
> org.apache.flume.channel.jdbc.BaseJdbcChannelProviderTest$MockSource.call(BaseJdbcChannelProviderTest.java:368)
> at
> org.apache.flume.channel.jdbc.BaseJdbcChannelProviderTest$MockSource.call(BaseJdbcChannelProviderTest.java:345)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
>
[jira] [Created] (FLUME-3015) Fix flaky org.apache.flume.channel.jdbc.TestJdbcChannelProviderNoFK.testEventWithSimulatedSourceAndSinks
Attila Simon created FLUME-3015:
---
Summary: Fix flaky
org.apache.flume.channel.jdbc.TestJdbcChannelProviderNoFK.testEventWithSimulatedSourceAndSinks
Key: FLUME-3015
URL: https://issues.apache.org/jira/browse/FLUME-3015
Project: Flume
Issue Type: Bug
Components: Test
Affects Versions: v1.7.0
Reporter: Attila Simon
testEventWithSimulatedSourceAndSinks(org.apache.flume.channel.jdbc.TestJdbcChannelProviderNoFK):
org.apache.flume.channel.jdbc.JdbcChannelException: Failed to persist event
{noformat}
testEventWithSimulatedSourceAndSinks(org.apache.flume.channel.jdbc.TestJdbcChannelProviderNoFK)
Time elapsed: 310978 sec <<< ERROR!
java.util.concurrent.ExecutionException:
org.apache.flume.channel.jdbc.JdbcChannelException: Failed to persist event
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:188)
at
org.apache.flume.channel.jdbc.BaseJdbcChannelProviderTest.testEventWithSimulatedSourceAndSinks(BaseJdbcChannelProviderTest.java:202)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
at
org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
at
org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
at
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)
at
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)
Caused by: org.apache.flume.channel.jdbc.JdbcChannelException: Failed to
persist event
at
org.apache.flume.channel.jdbc.impl.JdbcChannelProviderImpl.persistEvent(JdbcChannelProviderImpl.java:312)
at
org.apache.flume.channel.jdbc.BaseJdbcChannelProviderTest$MockSource.call(BaseJdbcChannelProviderTest.java:368)
at
org.apache.flume.channel.jdbc.BaseJdbcChannelProviderTest$MockSource.call(BaseJdbcChannelProviderTest.java:345)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.flume.channel.jdbc.JdbcChannelException: Unable to
persist event: org.apache.flume.channel.jdbc.impl.PersistableEvent@4fbafc05
at
org.apache.flume.channel.jdbc.impl.DerbySchemaHandler.storeEvent(DerbySchemaHandler.java:733)
at
org.apache.flume.channel.jdbc.impl.JdbcChannelProviderImpl.persistEvent(JdbcChannelProviderImpl.java:305)
[jira] [Created] (FLUME-3014) fix flaky test org.apache.flume.source.TestSyslogUdpSource.testSourceCounter
Attila Simon created FLUME-3014:
---
Summary: fix flaky test
org.apache.flume.source.TestSyslogUdpSource.testSourceCounter
Key: FLUME-3014
URL: https://issues.apache.org/jira/browse/FLUME-3014
Project: Flume
Issue Type: Bug
Components: Test
Affects Versions: v1.7.0
Reporter: Attila Simon
{noformat}
java.lang.AssertionError: expected:<1> but was:<0>
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.failNotEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:128)
at org.junit.Assert.assertEquals(Assert.java:472)
at org.junit.Assert.assertEquals(Assert.java:456)
at
org.apache.flume.source.TestSyslogUdpSource.testSourceCounter(TestSyslogUdpSource.java:205)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
at
org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
at
org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
at
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)
at
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (FLUME-3013) Fix flaky testShutdown(org.apache.flume.source.TestExecSource)
Attila Simon created FLUME-3013:
---
Summary: Fix flaky
testShutdown(org.apache.flume.source.TestExecSource)
Key: FLUME-3013
URL: https://issues.apache.org/jira/browse/FLUME-3013
Project: Flume
Issue Type: Bug
Components: Test
Affects Versions: v1.7.0
Reporter: Attila Simon
{noformat}
java.lang.IllegalStateException: Command [ps -ef] exited with 143
at org.apache.flume.source.TestExecSource.exec(TestExecSource.java:460)
at
org.apache.flume.source.TestExecSource.testShutdown(TestExecSource.java:406)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
at
org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
at
org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
at
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)
testShutdown(org.apache.flume.source.TestExecSource) Time elapsed: 621 sec
<<< ERROR!
java.lang.NullPointerException
at org.apache.flume.source.ExecSource.stop(ExecSource.java:202)
at org.apache.flume.source.TestExecSource.tearDown(TestExecSource.java:83)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:36)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
[jira] [Commented] (FLUME-2997) Fix flaky junit test in SpillableMemoryChannel
[ https://issues.apache.org/jira/browse/FLUME-2997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15585137#comment-15585137 ] Attila Simon commented on FLUME-2997: - I think I missed some description of this change so let me provide it now: Originally this test case aimed to verify behaviour about having multiple mocked sources putting events on the same SpillableMemoryChannel while multiple mocked sinks drains those events. Given the mock implementation didn't do retry if sources are faster than the sinks the channel will get full. Since full channel doesn't accept new events the mocked sources fail their task by raising a runtime exception. This edge case should be avoided thus the increased capacity (which makes sure that channel cannot be full). The "channel is full" edge case is already covered by another test. Those System.out.println-s were replaced by asserts to make the function to became a test. It wasn't testing anything before. It only ran to the end without checking anything. (The flakiness was caused by a ChannelFullException). I guess the original author used those System.out.println-s for manual testing (purely looking at the numbers). This change also made that manual step automatic. > Fix flaky junit test in SpillableMemoryChannel > -- > > Key: FLUME-2997 > URL: https://issues.apache.org/jira/browse/FLUME-2997 > Project: Flume > Issue Type: Test >Affects Versions: v1.7.0 >Reporter: Attila Simon >Assignee: Attila Simon > Fix For: v1.8.0 > > Attachments: FLUME-2997-1.patch, FLUME-2997.patch > > > testParallelSingleSourceAndSink sometimes trigger an edge case scenario if > sinks are slower than sources. In such situations the channel can get full > thus uncaught ChannelFullException breaks the test. Since > testCapacityWithOverflow was designed to cover such edge-case scenario > already we can safely fix the test by increasing the channel capacity to make > sure it won't get full. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2977) Upgrade RAT to 0.12
[
https://issues.apache.org/jira/browse/FLUME-2977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15574906#comment-15574906
]
Attila Simon commented on FLUME-2977:
-
Reply from d...@creadur.apache.org:
http://mail-archives.apache.org/mod_mbox/creadur-dev/201609.mbox/%3c5d1d4b7c-cdd3-f5da-e248-ba6308d6f...@gmx.de%3E
http://mail-archives.apache.org/mod_mbox/creadur-dev/201609.mbox/%3ce93f5bac-a80a-c106-18bf-2b0a6228f...@apache.org%3E
http://mail-archives.apache.org/mod_mbox/creadur-dev/201609.mbox/%3ca3ce6b76-5fa8-f6d9-44da-621e2bb13...@gmx.de%3E
And the associated jira to track this issue:
https://issues.apache.org/jira/browse/RAT-222
> Upgrade RAT to 0.12
> ---
>
> Key: FLUME-2977
> URL: https://issues.apache.org/jira/browse/FLUME-2977
> Project: Flume
> Issue Type: Improvement
> Components: Build
>Reporter: Attila Simon
>Assignee: Attila Simon
>Priority: Minor
> Fix For: v1.8.0
>
> Attachments: FLUME-2977.patch
>
>
> Before RAT check mvn install prints out the following warnings:
> {noformat}
> Warning: org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser: Property
> 'http://www.oracle.com/xml/jaxp/properties/entityExpansionLimit' is not
> recognized.
> Compiler warnings:
> WARNING: 'org.apache.xerces.jaxp.SAXParserImpl: Property
> 'http://javax.xml.XMLConstants/property/accessExternalDTD' is not recognized.'
> Warning: org.apache.xerces.parsers.SAXParser: Feature
> 'http://javax.xml.XMLConstants/feature/secure-processing' is not recognized.
> Warning: org.apache.xerces.parsers.SAXParser: Property
> 'http://javax.xml.XMLConstants/property/accessExternalDTD' is not recognized.
> Warning: org.apache.xerces.parsers.SAXParser: Property
> 'http://www.oracle.com/xml/jaxp/properties/entityExpansionLimit' is not
> recognized.
> [INFO] Rat check: Summary of files. Unapproved: 0 unknown: 0 generated: 0
> approved: 9 licence.
> {noformat}
> It doesn't break the build but seems misleading.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (FLUME-2924) Flume 1.7.0 release
[ https://issues.apache.org/jira/browse/FLUME-2924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Simon reassigned FLUME-2924: --- Assignee: (was: Attila Simon) > Flume 1.7.0 release > --- > > Key: FLUME-2924 > URL: https://issues.apache.org/jira/browse/FLUME-2924 > Project: Flume > Issue Type: Umbrella >Affects Versions: v1.7.0 >Reporter: Lior Zeno > Fix For: v1.7.0 > > > Release 1.7.0 umbrella issue -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (FLUME-2929) Update License file for 1.7 release
[ https://issues.apache.org/jira/browse/FLUME-2929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Simon reassigned FLUME-2929: --- Assignee: Attila Simon > Update License file for 1.7 release > --- > > Key: FLUME-2929 > URL: https://issues.apache.org/jira/browse/FLUME-2929 > Project: Flume > Issue Type: Sub-task >Affects Versions: v1.7.0 >Reporter: Lior Zeno >Assignee: Attila Simon > Fix For: v1.7.0 > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (FLUME-2924) Flume 1.7.0 release
[ https://issues.apache.org/jira/browse/FLUME-2924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Simon reassigned FLUME-2924: --- Assignee: Attila Simon > Flume 1.7.0 release > --- > > Key: FLUME-2924 > URL: https://issues.apache.org/jira/browse/FLUME-2924 > Project: Flume > Issue Type: Umbrella >Affects Versions: v1.7.0 >Reporter: Lior Zeno >Assignee: Attila Simon > Fix For: v1.7.0 > > > Release 1.7.0 umbrella issue -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLUME-2971) Document secure Kafka Sink/Source/Channel setup
[ https://issues.apache.org/jira/browse/FLUME-2971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Simon updated FLUME-2971: Attachment: FLUME-2971-1.patch > Document secure Kafka Sink/Source/Channel setup > --- > > Key: FLUME-2971 > URL: https://issues.apache.org/jira/browse/FLUME-2971 > Project: Flume > Issue Type: Documentation > Components: Channel, Sinks+Sources >Affects Versions: v1.7.0 >Reporter: Tristan Stevens >Assignee: Attila Simon > Attachments: FLUME-2971-0.patch, FLUME-2971-1.patch > > > As per FLUME-2901 but for Kafka Source and Kafka Sink -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLUME-2971) Document secure Kafka Sink/Source/Channel setup
[ https://issues.apache.org/jira/browse/FLUME-2971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Simon updated FLUME-2971: Component/s: Channel > Document secure Kafka Sink/Source/Channel setup > --- > > Key: FLUME-2971 > URL: https://issues.apache.org/jira/browse/FLUME-2971 > Project: Flume > Issue Type: Documentation > Components: Channel, Sinks+Sources >Affects Versions: v1.7.0 >Reporter: Tristan Stevens >Assignee: Attila Simon > Attachments: FLUME-2971-0.patch > > > As per FLUME-2901 but for Kafka Source and Kafka Sink -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLUME-2971) Document secure Kafka Sink/Source/Channel setup
[ https://issues.apache.org/jira/browse/FLUME-2971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Simon updated FLUME-2971: Issue Type: Documentation (was: Bug) > Document secure Kafka Sink/Source/Channel setup > --- > > Key: FLUME-2971 > URL: https://issues.apache.org/jira/browse/FLUME-2971 > Project: Flume > Issue Type: Documentation > Components: Channel, Sinks+Sources >Affects Versions: v1.7.0 >Reporter: Tristan Stevens >Assignee: Attila Simon > Attachments: FLUME-2971-0.patch > > > As per FLUME-2901 but for Kafka Source and Kafka Sink -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLUME-2971) Document secure Kafka Sink/Source/Channel setup
[ https://issues.apache.org/jira/browse/FLUME-2971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Simon updated FLUME-2971: Summary: Document secure Kafka Sink/Source/Channel setup (was: Document Kerberos set-up for Kafka Sink and Kafka Source) > Document secure Kafka Sink/Source/Channel setup > --- > > Key: FLUME-2971 > URL: https://issues.apache.org/jira/browse/FLUME-2971 > Project: Flume > Issue Type: Bug > Components: Sinks+Sources >Affects Versions: v1.7.0 >Reporter: Tristan Stevens >Assignee: Attila Simon > Attachments: FLUME-2971-0.patch > > > As per FLUME-2901 but for Kafka Source and Kafka Sink -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2971) Document Kerberos set-up for Kafka Sink and Kafka Source
[ https://issues.apache.org/jira/browse/FLUME-2971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15554728#comment-15554728 ] Attila Simon commented on FLUME-2971: - The patch aims to extend the existing documentation of secure Kafka channel with describing SSL+Plaintext setup as well as providing the whole package (SSL+Kerberos+Plain) for KafkaSource and KafkaSink. > Document Kerberos set-up for Kafka Sink and Kafka Source > > > Key: FLUME-2971 > URL: https://issues.apache.org/jira/browse/FLUME-2971 > Project: Flume > Issue Type: Bug > Components: Sinks+Sources >Affects Versions: v1.7.0 >Reporter: Tristan Stevens >Assignee: Attila Simon > Attachments: FLUME-2971-0.patch > > > As per FLUME-2901 but for Kafka Source and Kafka Sink -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLUME-2971) Document Kerberos set-up for Kafka Sink and Kafka Source
[ https://issues.apache.org/jira/browse/FLUME-2971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Simon updated FLUME-2971: Attachment: FLUME-2971-0.patch > Document Kerberos set-up for Kafka Sink and Kafka Source > > > Key: FLUME-2971 > URL: https://issues.apache.org/jira/browse/FLUME-2971 > Project: Flume > Issue Type: Bug > Components: Sinks+Sources >Affects Versions: v1.7.0 >Reporter: Tristan Stevens >Assignee: Attila Simon > Attachments: FLUME-2971-0.patch > > > As per FLUME-2901 but for Kafka Source and Kafka Sink -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2716) File Channel cannot handle capacity Integer.MAX_VALUE
[
https://issues.apache.org/jira/browse/FLUME-2716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15551480#comment-15551480
]
Attila Simon commented on FLUME-2716:
-
My previous comment might have been confusing so let me clarify:
Following the source code in EventQueueBackingStoreFile#86-103 :
{noformat}
long totalBytes = (capacity + HEADER_SIZE) * Serialization.SIZE_OF_LONG; //the
formula I wrote above
...
allocate(checkpointFile, totalBytes); //allocate the disk space, essentially
create a file with size of totalBytes, no upper limit
...
mappedBuffer = checkpointFileHandle.getChannel().map(MapMode.READ_WRITE, 0,
checkpointFile.length()); // checkpointFile.length() == totalBytes and
map function has an Integer.MAX_VALUE limit on the size of mapped content
{noformat}
>From that limitation we have a constraint:
Integer.MAX_VALUE >= checkpointFile.length()
Integer.MAX_VALUE >= totalBytes
Integer.MAX_VALUE >= (capacity + HEADER_SIZE) * Serialization.SIZE_OF_LONG
floor(Integer.MAX_VALUE/Serialization.SIZE_OF_LONG) - HEADER_SIZE >= capacity
268434426 >= capacity
I recommend to add an extra(additional to the long cast) check to
EventQueueBackingStoreFile before the allocation which checks the totalsize <
integer.max_value and if fails then print out the formula similarly to the
message constructed on lines EventQueueBackingStoreFile#96-100 as well as
throwing an exception to prevent channel startup. It has to be before the
allocation as otherwise flume would allocate a system resource(file) which
couldn't be used afterwards thus resulting a resource leak.
To relax this limitation the way how memory mapping of the checkpoint file was
requested from java.nio.channels.FileChannel has to be reconsidered, eg mapping
only different parts of same checkpoint file by multiple map calls or split the
checkpoints to multiple files are alternative solutions to this problem.
> File Channel cannot handle capacity Integer.MAX_VALUE
> -
>
> Key: FLUME-2716
> URL: https://issues.apache.org/jira/browse/FLUME-2716
> Project: Flume
> Issue Type: Bug
> Components: Channel, File Channel
>Affects Versions: v1.6.0, v1.7.0
>Reporter: Dong Zhao
> Labels: unit-test-missing
> Fix For: v1.7.0
>
> Attachments: FLUME-2716.patch
>
>
> if capacity is set to Integer.MAX_VALUE(2147483647), checkpoint file size is
> calculated wrongly to 8224. The calculation should first cast int to long,
> then calculate the totalBytes. See the patch for details. Thanks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLUME-2716) File Channel cannot handle capacity Integer.MAX_VALUE
[ https://issues.apache.org/jira/browse/FLUME-2716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15545567#comment-15545567 ] Attila Simon commented on FLUME-2716: - As I can see max capacity is 268434426. Checkpoint file is memory mapped as a whole and that cannot exceed Integer.MAX_VALUE : https://docs.oracle.com/javase/7/docs/api/java/nio/channels/FileChannel.html#map(java.nio.channels.FileChannel.MapMode,%20long,%20long) Checkpoint file size = ( + ) * So basically capacity is less than Integer.MAX_VALUE by the header part. I see no issue with splitting up the memory mapping in some next release eg mapping only different parts of same checkpoint file by multiple map calls or split the checkpoints to multiple files. > File Channel cannot handle capacity Integer.MAX_VALUE > - > > Key: FLUME-2716 > URL: https://issues.apache.org/jira/browse/FLUME-2716 > Project: Flume > Issue Type: Bug > Components: Channel, File Channel >Affects Versions: v1.6.0, v1.7.0 >Reporter: Dong Zhao > Labels: unit-test-missing > Fix For: v1.7.0 > > Attachments: FLUME-2716.patch > > > if capacity is set to Integer.MAX_VALUE(2147483647), checkpoint file size is > calculated wrongly to 8224. The calculation should first cast int to long, > then calculate the totalBytes. See the patch for details. Thanks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (FLUME-2971) Document Kerberos set-up for Kafka Sink and Kafka Source
[ https://issues.apache.org/jira/browse/FLUME-2971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Simon reassigned FLUME-2971: --- Assignee: Attila Simon > Document Kerberos set-up for Kafka Sink and Kafka Source > > > Key: FLUME-2971 > URL: https://issues.apache.org/jira/browse/FLUME-2971 > Project: Flume > Issue Type: Bug > Components: Sinks+Sources >Affects Versions: v1.7.0 >Reporter: Tristan Stevens >Assignee: Attila Simon > > As per FLUME-2901 but for Kafka Source and Kafka Sink -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2998) Add missing configuration parameter to SequenceSource docs
[ https://issues.apache.org/jira/browse/FLUME-2998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15532922#comment-15532922 ] Attila Simon commented on FLUME-2998: - Long.MAX_VALUE is 9223372036854775807. Although Long.MAX_VALUE is not a valid expression in the config file I would doubt that the exact number is useful in the docs. What I saw in other places is "see description". If it really needed to be changed from Long.MAX_VALUE in the "default value" column then we may use that trick and put it into the description column. > Add missing configuration parameter to SequenceSource docs > -- > > Key: FLUME-2998 > URL: https://issues.apache.org/jira/browse/FLUME-2998 > Project: Flume > Issue Type: Documentation > Components: Docs >Affects Versions: v1.7.0 >Reporter: Attila Simon >Assignee: Attila Simon > Fix For: v1.7.0 > > Attachments: FLUME-2998.patch > > > totalEvents can be specified -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLUME-2998) Add missing configuration parameter to SequenceSource docs
[ https://issues.apache.org/jira/browse/FLUME-2998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Simon updated FLUME-2998: Attachment: (was: FLUME-2998.patch) > Add missing configuration parameter to SequenceSource docs > -- > > Key: FLUME-2998 > URL: https://issues.apache.org/jira/browse/FLUME-2998 > Project: Flume > Issue Type: Documentation > Components: Docs >Affects Versions: v1.7.0 >Reporter: Attila Simon >Assignee: Attila Simon > Fix For: v1.7.0 > > Attachments: FLUME-2998.patch > > > totalEvents can be specified -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLUME-2998) Add missing configuration parameter to SequenceSource docs
[ https://issues.apache.org/jira/browse/FLUME-2998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Simon updated FLUME-2998: Attachment: FLUME-2998.patch > Add missing configuration parameter to SequenceSource docs > -- > > Key: FLUME-2998 > URL: https://issues.apache.org/jira/browse/FLUME-2998 > Project: Flume > Issue Type: Documentation > Components: Docs >Affects Versions: v1.7.0 >Reporter: Attila Simon >Assignee: Attila Simon > Fix For: v1.7.0 > > Attachments: FLUME-2998.patch > > > totalEvents can be specified -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLUME-2997) Fix flaky junit test in SpillableMemoryChannel
[ https://issues.apache.org/jira/browse/FLUME-2997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Simon updated FLUME-2997: Attachment: FLUME-2997-1.patch missed the asserts from prev version, mvn clean install builds the project, unit test passes for the module > Fix flaky junit test in SpillableMemoryChannel > -- > > Key: FLUME-2997 > URL: https://issues.apache.org/jira/browse/FLUME-2997 > Project: Flume > Issue Type: Test >Affects Versions: v1.7.0 >Reporter: Attila Simon >Assignee: Attila Simon > Fix For: v1.7.0 > > Attachments: FLUME-2997-1.patch, FLUME-2997.patch > > > testParallelSingleSourceAndSink sometimes trigger an edge case scenario if > sinks are slower than sources. In such situations the channel can get full > thus uncaught ChannelFullException breaks the test. Since > testCapacityWithOverflow was designed to cover such edge-case scenario > already we can safely fix the test by increasing the channel capacity to make > sure it won't get full. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLUME-2998) Add missing configuration parameter to SequenceSource docs
[ https://issues.apache.org/jira/browse/FLUME-2998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Simon updated FLUME-2998: Attachment: FLUME-2998.patch > Add missing configuration parameter to SequenceSource docs > -- > > Key: FLUME-2998 > URL: https://issues.apache.org/jira/browse/FLUME-2998 > Project: Flume > Issue Type: Documentation > Components: Docs >Affects Versions: v1.7.0 >Reporter: Attila Simon >Assignee: Attila Simon > Fix For: v1.7.0 > > Attachments: FLUME-2998.patch > > > totalEvents can be specified -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLUME-2998) Add missing configuration parameter to SequenceSource docs
Attila Simon created FLUME-2998: --- Summary: Add missing configuration parameter to SequenceSource docs Key: FLUME-2998 URL: https://issues.apache.org/jira/browse/FLUME-2998 Project: Flume Issue Type: Documentation Components: Docs Affects Versions: v1.7.0 Reporter: Attila Simon Assignee: Attila Simon Fix For: v1.7.0 totalEvents can be specified -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2997) Fix flaky junit test in SpillableMemoryChannel
[ https://issues.apache.org/jira/browse/FLUME-2997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15523123#comment-15523123 ] Attila Simon commented on FLUME-2997: - Code change in the attached patch sets up a channel in the test with a capacity enough to store the produced events. > Fix flaky junit test in SpillableMemoryChannel > -- > > Key: FLUME-2997 > URL: https://issues.apache.org/jira/browse/FLUME-2997 > Project: Flume > Issue Type: Test >Affects Versions: v1.7.0 >Reporter: Attila Simon >Assignee: Attila Simon > Fix For: v1.7.0 > > Attachments: FLUME-2997.patch > > > testParallelSingleSourceAndSink sometimes trigger an edge case scenario if > sinks are slower than sources. In such situations the channel can get full > thus uncaught ChannelFullException breaks the test. Since > testCapacityWithOverflow was designed to cover such edge-case scenario > already we can safely fix the test by increasing the channel capacity to make > sure it won't get full. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLUME-2997) Fix flaky junit test in SpillableMemoryChannel
[ https://issues.apache.org/jira/browse/FLUME-2997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Simon updated FLUME-2997: Attachment: FLUME-2997.patch > Fix flaky junit test in SpillableMemoryChannel > -- > > Key: FLUME-2997 > URL: https://issues.apache.org/jira/browse/FLUME-2997 > Project: Flume > Issue Type: Test >Affects Versions: v1.7.0 >Reporter: Attila Simon >Assignee: Attila Simon > Fix For: v1.7.0 > > Attachments: FLUME-2997.patch > > > testParallelSingleSourceAndSink sometimes trigger an edge case scenario if > sinks are slower than sources. In such situations the channel can get full > thus uncaught ChannelFullException breaks the test. Since > testCapacityWithOverflow was designed to cover such edge-case scenario > already we can safely fix the test by increasing the channel capacity to make > sure it won't get full. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLUME-2997) Fix flaky junit test in SpillableMemoryChannel
Attila Simon created FLUME-2997: --- Summary: Fix flaky junit test in SpillableMemoryChannel Key: FLUME-2997 URL: https://issues.apache.org/jira/browse/FLUME-2997 Project: Flume Issue Type: Test Affects Versions: v1.7.0 Reporter: Attila Simon Assignee: Attila Simon Fix For: v1.7.0 testParallelSingleSourceAndSink sometimes trigger an edge case scenario if sinks are slower than sources. In such situations the channel can get full thus uncaught ChannelFullException breaks the test. Since testCapacityWithOverflow was designed to cover such edge-case scenario already we can safely fix the test by increasing the channel capacity to make sure it won't get full. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2966) NULL text in a TextMessage from a JMS source in Flume can lead to NPE
[
https://issues.apache.org/jira/browse/FLUME-2966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15522921#comment-15522921
]
Attila Simon commented on FLUME-2966:
-
After applying FLUME-2966-1.patch: flume builds and unittest passed (within the
impacted module). the change looks good to me. +1
> NULL text in a TextMessage from a JMS source in Flume can lead to NPE
> -
>
> Key: FLUME-2966
> URL: https://issues.apache.org/jira/browse/FLUME-2966
> Project: Flume
> Issue Type: Bug
>Affects Versions: v1.5.0
>Reporter: Siddharth Ahuja
>Assignee: Siddharth Ahuja
> Attachments: App.java, FLUME-2966-0.patch, FLUME-2966-1.patch
>
>
> Code at
> https://github.com/apache/flume/blob/trunk/flume-ng-sources/flume-jms-source/src/main/java/org/apache/flume/source/jms/DefaultJMSMessageConverter.java#L103
> does not check for a NULL text in a TextMessage from a Flume JMS source.
> This can lead to a NullPointerException here:
> {code}textMessage.getText().getBytes(charset){code} while trying to
> de-reference a null text from the textmessage.
> We should probably skip these like the NULL Objects in the ObjectMessage just
> below at:
> https://github.com/apache/flume/blob/trunk/flume-ng-sources/flume-jms-source/src/main/java/org/apache/flume/source/jms/DefaultJMSMessageConverter.java#L107.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLUME-2977) Upgrade RAT to 0.12
[
https://issues.apache.org/jira/browse/FLUME-2977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15513324#comment-15513324
]
Attila Simon commented on FLUME-2977:
-
I feel disturbance in the force related to RAT versions so sent a mail to RAT
dev and cc-ed dev@flume. On the main site (https://creadur.apache.org/) though
0.12 is now displayed correctly as the latest available release:
Version history
2016-06-10 — Apache Rat 0.12 released (release notes)
2014-08-25 — Apache Rat 0.11 released (release notes)
2013-09-05 — Apache Rat 0.10 released (release notes)
2013-06-13 — Apache Rat 0.9 released (release notes)
> Upgrade RAT to 0.12
> ---
>
> Key: FLUME-2977
> URL: https://issues.apache.org/jira/browse/FLUME-2977
> Project: Flume
> Issue Type: Improvement
> Components: Build
>Reporter: Attila Simon
>Assignee: Attila Simon
>Priority: Minor
> Fix For: v1.7.0
>
> Attachments: FLUME-2977.patch
>
>
> Before RAT check mvn install prints out the following warnings:
> {noformat}
> Warning: org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser: Property
> 'http://www.oracle.com/xml/jaxp/properties/entityExpansionLimit' is not
> recognized.
> Compiler warnings:
> WARNING: 'org.apache.xerces.jaxp.SAXParserImpl: Property
> 'http://javax.xml.XMLConstants/property/accessExternalDTD' is not recognized.'
> Warning: org.apache.xerces.parsers.SAXParser: Feature
> 'http://javax.xml.XMLConstants/feature/secure-processing' is not recognized.
> Warning: org.apache.xerces.parsers.SAXParser: Property
> 'http://javax.xml.XMLConstants/property/accessExternalDTD' is not recognized.
> Warning: org.apache.xerces.parsers.SAXParser: Property
> 'http://www.oracle.com/xml/jaxp/properties/entityExpansionLimit' is not
> recognized.
> [INFO] Rat check: Summary of files. Unapproved: 0 unknown: 0 generated: 0
> approved: 9 licence.
> {noformat}
> It doesn't break the build but seems misleading.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLUME-2977) fix compile warnings from SAXParser
[
https://issues.apache.org/jira/browse/FLUME-2977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15464818#comment-15464818
]
Attila Simon commented on FLUME-2977:
-
Although there is no track of 0.12 in docs, it is available in maven:
http://search.maven.org/#artifactdetails%7Corg.apache.rat%7Capache-rat%7C0.12%7Cjar
> fix compile warnings from SAXParser
> ---
>
> Key: FLUME-2977
> URL: https://issues.apache.org/jira/browse/FLUME-2977
> Project: Flume
> Issue Type: Improvement
> Components: Build
>Reporter: Attila Simon
>Assignee: Attila Simon
>Priority: Minor
> Fix For: v1.7.0
>
> Attachments: FLUME-2977.patch
>
>
> Before RAT check mvn install prints out the following warnings:
> {noformat}
> Warning: org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser: Property
> 'http://www.oracle.com/xml/jaxp/properties/entityExpansionLimit' is not
> recognized.
> Compiler warnings:
> WARNING: 'org.apache.xerces.jaxp.SAXParserImpl: Property
> 'http://javax.xml.XMLConstants/property/accessExternalDTD' is not recognized.'
> Warning: org.apache.xerces.parsers.SAXParser: Feature
> 'http://javax.xml.XMLConstants/feature/secure-processing' is not recognized.
> Warning: org.apache.xerces.parsers.SAXParser: Property
> 'http://javax.xml.XMLConstants/property/accessExternalDTD' is not recognized.
> Warning: org.apache.xerces.parsers.SAXParser: Property
> 'http://www.oracle.com/xml/jaxp/properties/entityExpansionLimit' is not
> recognized.
> [INFO] Rat check: Summary of files. Unapproved: 0 unknown: 0 generated: 0
> approved: 9 licence.
> {noformat}
> It doesn't break the build but seems misleading.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLUME-2979) File descriptor leaks in TaildirSource
[
https://issues.apache.org/jira/browse/FLUME-2979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15445823#comment-15445823
]
Attila Simon commented on FLUME-2979:
-
Thanks [~buaalk]! I resolved the other one.
> File descriptor leaks in TaildirSource
> --
>
> Key: FLUME-2979
> URL: https://issues.apache.org/jira/browse/FLUME-2979
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
>Affects Versions: v1.7.0
>Reporter: Kun Liu
>
> TaildirSource creates ReliableTaildirEventReader object in start() method,
> and the constructor of ReliableTaildirEventReader will try to open matched
> files in updateTailFiles(). However, If the flume process has no permission
> to some matched files, then the updateTailFiles() method throws IOException
> and interrupts the creation of ReliableTaildirEventReader object. The
> following logs show the opening of some matched files and the exception
> thrown when try to open a file with no permission.
> logs--
> 25 Aug 2016 12:50:28,337 INFO [lifecycleSupervisor-1-0]
> (org.apache.flume.source.taildir.ReliableTaildirEventReader.openFile:294) -
> Opening file: /home/work/quota/test.2016-06-27-121958, inode: 91668538, pos: 0
> 25 Aug 2016 12:50:28,337 INFO [lifecycleSupervisor-1-0]
> (org.apache.flume.source.taildir.ReliableTaildirEventReader.openFile:294) -
> Opening file: /home/work/quota/test.2016-06-27-111458, inode: 91668558, pos: 0
> 25 Aug 2016 12:50:28,337 INFO [lifecycleSupervisor-1-0]
> (org.apache.flume.source.taildir.ReliableTaildirEventReader.openFile:294) -
> Opening file: /home/work/quota/test.2016-06-27-080458, inode: 91668536, pos: 0
> 25 Aug 2016 12:50:28,338 INFO [lifecycleSupervisor-1-0]
> (org.apache.flume.source.taildir.ReliableTaildirEventReader.openFile:294) -
> Opening file: /home/work/quota/test.2016-06-27-140958, inode: 91668655, pos: 0
> 25 Aug 2016 12:50:28,338 ERROR [lifecycleSupervisor-1-0]
> (org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run:253) -
> Unable to start PollableSourceRunner: { source:Taildir source: {
> positionFile: /home/work/taildir_position.json, skipToEnd: false,
> byteOffsetHeader: false, idleTimeout: 12, writePosInterval: 3000 }
> counterGroup:{ name:null counters:{} } } - Exception follows.
> org.apache.flume.FlumeException: Failed opening file:
> /home/work/quota/test.2016-06-27-140958
> at
> org.apache.flume.source.taildir.ReliableTaildirEventReader.openFile(ReliableTaildirEventReader.java:297)
> at
> org.apache.flume.source.taildir.ReliableTaildirEventReader.updateTailFiles(ReliableTaildirEventReader.java:260)
> at
> org.apache.flume.source.taildir.ReliableTaildirEventReader.(ReliableTaildirEventReader.java:94)
> at
> org.apache.flume.source.taildir.ReliableTaildirEventReader.(ReliableTaildirEventReader.java:48)
> at
> org.apache.flume.source.taildir.ReliableTaildirEventReader$Builder.build(ReliableTaildirEventReader.java:361)
> at
> org.apache.flume.source.taildir.TaildirSource.start(TaildirSource.java:100)
> at
> org.apache.flume.source.PollableSourceRunner.start(PollableSourceRunner.java:72)
> at
> org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
> at java.util.concurrent.FutureTask.runAndReset(Unknown Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Caused by: java.io.FileNotFoundException:
> /home/work/quota/test.2016-06-27-140958 (Permission denied)
> at java.io.RandomAccessFile.open(Native Method)
> at java.io.RandomAccessFile.(Unknown Source)
> at org.apache.flume.source.taildir.TailFile.(TailFile.java:60)
> at
> org.apache.flume.source.taildir.ReliableTaildirEventReader.openFile(ReliableTaildirEventReader.java:295)
> ... 14 more
> logs--
> As the creation of ReliableTaildirEventReader is interrupted, the
> TaildirSource catches the IOException and throws FlumeException in start()
> method, but the opened files are not closed, which leads to the leak of file
> descriptors. As time goes by, the PollableSourceRunner keeps starting
> TaildirSource, and the number of opened files in flume process exceeds the
> upper limit, then the 'Too many open files' error occurs.
--
This message was sent by
[jira] [Commented] (FLUME-2526) Build flume by jdk 7 in default
[ https://issues.apache.org/jira/browse/FLUME-2526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15444203#comment-15444203 ] Attila Simon commented on FLUME-2526: - If no violent objections received I'm going to resolve this in a week since we already build on jdk1.7 by default. > Build flume by jdk 7 in default > --- > > Key: FLUME-2526 > URL: https://issues.apache.org/jira/browse/FLUME-2526 > Project: Flume > Issue Type: Improvement > Components: Build >Reporter: Guo Ruijing > > Flume may use jdk 7 in default. > Existing pom code: > > 1.6 > 1.6 > > **/generated-sources/** > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2979) File descriptor leaks in TaildirSource
[
https://issues.apache.org/jira/browse/FLUME-2979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15436665#comment-15436665
]
Attila Simon commented on FLUME-2979:
-
Hi [~buaalk],
This ticket is a duplication of FLUME-2978, isn't it? could you please tell me
which one should survive? (the other should be closed as duplicate)
> File descriptor leaks in TaildirSource
> --
>
> Key: FLUME-2979
> URL: https://issues.apache.org/jira/browse/FLUME-2979
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
>Affects Versions: v1.7.0
>Reporter: Kun Liu
>
> TaildirSource creates ReliableTaildirEventReader object in start() method,
> and the constructor of ReliableTaildirEventReader will try to open matched
> files in updateTailFiles(). However, If the flume process has no permission
> to some matched files, then the updateTailFiles() method throws IOException
> and interrupts the creation of ReliableTaildirEventReader object. The
> following logs show the opening of some matched files and the exception
> thrown when try to open a file with no permission.
> logs--
> 25 Aug 2016 12:50:28,337 INFO [lifecycleSupervisor-1-0]
> (org.apache.flume.source.taildir.ReliableTaildirEventReader.openFile:294) -
> Opening file: /home/work/quota/test.2016-06-27-121958, inode: 91668538, pos: 0
> 25 Aug 2016 12:50:28,337 INFO [lifecycleSupervisor-1-0]
> (org.apache.flume.source.taildir.ReliableTaildirEventReader.openFile:294) -
> Opening file: /home/work/quota/test.2016-06-27-111458, inode: 91668558, pos: 0
> 25 Aug 2016 12:50:28,337 INFO [lifecycleSupervisor-1-0]
> (org.apache.flume.source.taildir.ReliableTaildirEventReader.openFile:294) -
> Opening file: /home/work/quota/test.2016-06-27-080458, inode: 91668536, pos: 0
> 25 Aug 2016 12:50:28,338 INFO [lifecycleSupervisor-1-0]
> (org.apache.flume.source.taildir.ReliableTaildirEventReader.openFile:294) -
> Opening file: /home/work/quota/test.2016-06-27-140958, inode: 91668655, pos: 0
> 25 Aug 2016 12:50:28,338 ERROR [lifecycleSupervisor-1-0]
> (org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run:253) -
> Unable to start PollableSourceRunner: { source:Taildir source: {
> positionFile: /home/work/taildir_position.json, skipToEnd: false,
> byteOffsetHeader: false, idleTimeout: 12, writePosInterval: 3000 }
> counterGroup:{ name:null counters:{} } } - Exception follows.
> org.apache.flume.FlumeException: Failed opening file:
> /home/work/quota/test.2016-06-27-140958
> at
> org.apache.flume.source.taildir.ReliableTaildirEventReader.openFile(ReliableTaildirEventReader.java:297)
> at
> org.apache.flume.source.taildir.ReliableTaildirEventReader.updateTailFiles(ReliableTaildirEventReader.java:260)
> at
> org.apache.flume.source.taildir.ReliableTaildirEventReader.(ReliableTaildirEventReader.java:94)
> at
> org.apache.flume.source.taildir.ReliableTaildirEventReader.(ReliableTaildirEventReader.java:48)
> at
> org.apache.flume.source.taildir.ReliableTaildirEventReader$Builder.build(ReliableTaildirEventReader.java:361)
> at
> org.apache.flume.source.taildir.TaildirSource.start(TaildirSource.java:100)
> at
> org.apache.flume.source.PollableSourceRunner.start(PollableSourceRunner.java:72)
> at
> org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
> at java.util.concurrent.FutureTask.runAndReset(Unknown Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Caused by: java.io.FileNotFoundException:
> /home/work/quota/test.2016-06-27-140958 (Permission denied)
> at java.io.RandomAccessFile.open(Native Method)
> at java.io.RandomAccessFile.(Unknown Source)
> at org.apache.flume.source.taildir.TailFile.(TailFile.java:60)
> at
> org.apache.flume.source.taildir.ReliableTaildirEventReader.openFile(ReliableTaildirEventReader.java:295)
> ... 14 more
> logs--
> As the creation of ReliableTaildirEventReader is interrupted, the
> TaildirSource catches the IOException and throws FlumeException in start()
> method, but the opened files are not closed, which leads to the leak of file
> descriptors. As time goes by, the PollableSourceRunner keeps starting
> TaildirSource, and the number of opened files in
[jira] [Updated] (FLUME-2977) fix compile warnings from SAXParser
[
https://issues.apache.org/jira/browse/FLUME-2977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Attila Simon updated FLUME-2977:
Priority: Minor (was: Major)
> fix compile warnings from SAXParser
> ---
>
> Key: FLUME-2977
> URL: https://issues.apache.org/jira/browse/FLUME-2977
> Project: Flume
> Issue Type: Improvement
> Components: Build
>Reporter: Attila Simon
>Assignee: Attila Simon
>Priority: Minor
> Fix For: v1.7.0
>
> Attachments: FLUME-2977.patch
>
>
> Before RAT check mvn install prints out the following warnings:
> {noformat}
> Warning: org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser: Property
> 'http://www.oracle.com/xml/jaxp/properties/entityExpansionLimit' is not
> recognized.
> Compiler warnings:
> WARNING: 'org.apache.xerces.jaxp.SAXParserImpl: Property
> 'http://javax.xml.XMLConstants/property/accessExternalDTD' is not recognized.'
> Warning: org.apache.xerces.parsers.SAXParser: Feature
> 'http://javax.xml.XMLConstants/feature/secure-processing' is not recognized.
> Warning: org.apache.xerces.parsers.SAXParser: Property
> 'http://javax.xml.XMLConstants/property/accessExternalDTD' is not recognized.
> Warning: org.apache.xerces.parsers.SAXParser: Property
> 'http://www.oracle.com/xml/jaxp/properties/entityExpansionLimit' is not
> recognized.
> [INFO] Rat check: Summary of files. Unapproved: 0 unknown: 0 generated: 0
> approved: 9 licence.
> {noformat}
> It doesn't break the build but seems misleading.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (FLUME-2977) fix compile warnings from SAXParser
[
https://issues.apache.org/jira/browse/FLUME-2977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Attila Simon updated FLUME-2977:
Attachment: FLUME-2977.patch
> fix compile warnings from SAXParser
> ---
>
> Key: FLUME-2977
> URL: https://issues.apache.org/jira/browse/FLUME-2977
> Project: Flume
> Issue Type: Improvement
> Components: Build
>Reporter: Attila Simon
>Assignee: Attila Simon
> Fix For: v1.7.0
>
> Attachments: FLUME-2977.patch
>
>
> Before RAT check mvn install prints out the following warnings:
> {noformat}
> Warning: org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser: Property
> 'http://www.oracle.com/xml/jaxp/properties/entityExpansionLimit' is not
> recognized.
> Compiler warnings:
> WARNING: 'org.apache.xerces.jaxp.SAXParserImpl: Property
> 'http://javax.xml.XMLConstants/property/accessExternalDTD' is not recognized.'
> Warning: org.apache.xerces.parsers.SAXParser: Feature
> 'http://javax.xml.XMLConstants/feature/secure-processing' is not recognized.
> Warning: org.apache.xerces.parsers.SAXParser: Property
> 'http://javax.xml.XMLConstants/property/accessExternalDTD' is not recognized.
> Warning: org.apache.xerces.parsers.SAXParser: Property
> 'http://www.oracle.com/xml/jaxp/properties/entityExpansionLimit' is not
> recognized.
> [INFO] Rat check: Summary of files. Unapproved: 0 unknown: 0 generated: 0
> approved: 9 licence.
> {noformat}
> It doesn't break the build but seems misleading.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLUME-2977) fix compile warnings from SAXParser
[
https://issues.apache.org/jira/browse/FLUME-2977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15435003#comment-15435003
]
Attila Simon commented on FLUME-2977:
-
Seems like it is related to RAT->Doxia->Xerces dependency.
https://issues.apache.org/jira/browse/RAT-158
Bumping RAT to 0.12 solves the issue. Interestingly 0.12 is available in maven
only the official wiki doesn't have any info about that release.
> fix compile warnings from SAXParser
> ---
>
> Key: FLUME-2977
> URL: https://issues.apache.org/jira/browse/FLUME-2977
> Project: Flume
> Issue Type: Improvement
> Components: Build
>Reporter: Attila Simon
>Assignee: Attila Simon
> Fix For: v1.7.0
>
>
> Before RAT check mvn install prints out the following warnings:
> {noformat}
> Warning: org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser: Property
> 'http://www.oracle.com/xml/jaxp/properties/entityExpansionLimit' is not
> recognized.
> Compiler warnings:
> WARNING: 'org.apache.xerces.jaxp.SAXParserImpl: Property
> 'http://javax.xml.XMLConstants/property/accessExternalDTD' is not recognized.'
> Warning: org.apache.xerces.parsers.SAXParser: Feature
> 'http://javax.xml.XMLConstants/feature/secure-processing' is not recognized.
> Warning: org.apache.xerces.parsers.SAXParser: Property
> 'http://javax.xml.XMLConstants/property/accessExternalDTD' is not recognized.
> Warning: org.apache.xerces.parsers.SAXParser: Property
> 'http://www.oracle.com/xml/jaxp/properties/entityExpansionLimit' is not
> recognized.
> [INFO] Rat check: Summary of files. Unapproved: 0 unknown: 0 generated: 0
> approved: 9 licence.
> {noformat}
> It doesn't break the build but seems misleading.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (FLUME-2977) fix compile warnings from SAXParser
Attila Simon created FLUME-2977:
---
Summary: fix compile warnings from SAXParser
Key: FLUME-2977
URL: https://issues.apache.org/jira/browse/FLUME-2977
Project: Flume
Issue Type: Improvement
Components: Build
Reporter: Attila Simon
Assignee: Attila Simon
Fix For: v1.7.0
Before RAT check mvn install prints out the following warnings:
{noformat}
Warning: org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser: Property
'http://www.oracle.com/xml/jaxp/properties/entityExpansionLimit' is not
recognized.
Compiler warnings:
WARNING: 'org.apache.xerces.jaxp.SAXParserImpl: Property
'http://javax.xml.XMLConstants/property/accessExternalDTD' is not recognized.'
Warning: org.apache.xerces.parsers.SAXParser: Feature
'http://javax.xml.XMLConstants/feature/secure-processing' is not recognized.
Warning: org.apache.xerces.parsers.SAXParser: Property
'http://javax.xml.XMLConstants/property/accessExternalDTD' is not recognized.
Warning: org.apache.xerces.parsers.SAXParser: Property
'http://www.oracle.com/xml/jaxp/properties/entityExpansionLimit' is not
recognized.
[INFO] Rat check: Summary of files. Unapproved: 0 unknown: 0 generated: 0
approved: 9 licence.
{noformat}
It doesn't break the build but seems misleading.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (FLUME-2954) make raw data appearing in log messages explicit
[ https://issues.apache.org/jira/browse/FLUME-2954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Simon updated FLUME-2954: Attachment: (was: FLUME-2954-2.patch) > make raw data appearing in log messages explicit > > > Key: FLUME-2954 > URL: https://issues.apache.org/jira/browse/FLUME-2954 > Project: Flume > Issue Type: Improvement > Components: Channel, Configuration, Sinks+Sources >Affects Versions: v1.6.0 >Reporter: Attila Simon >Assignee: Attila Simon >Priority: Critical > Fix For: v1.7.0 > > Attachments: FLUME-2954-4.patch > > > Flume has built in functionality to log out data flowing through > mainly for debugging purposes. This functionality appears in several > places of the codebase. I think such functionality rise security > concerns in production environments where sensitive information might > be ingested so it is crucial that enabling such functionality has to > be as explicit as possible (avoid implicit side effect setup). > Eg: setting the level of root logger to debug/trace cause that every > other logger will start logging at debug/trace including the ones > logging raw data. > In this jira I would like to provide a patch capturing how I imagined solving > this issue. It can be refined iteratively or used as a basis for a broader > discussion. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLUME-2954) make raw data appearing in log messages explicit
[ https://issues.apache.org/jira/browse/FLUME-2954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Simon updated FLUME-2954: Attachment: (was: FLUME-2954.patch) > make raw data appearing in log messages explicit > > > Key: FLUME-2954 > URL: https://issues.apache.org/jira/browse/FLUME-2954 > Project: Flume > Issue Type: Improvement > Components: Channel, Configuration, Sinks+Sources >Affects Versions: v1.6.0 >Reporter: Attila Simon >Assignee: Attila Simon >Priority: Critical > Fix For: v1.7.0 > > Attachments: FLUME-2954-4.patch > > > Flume has built in functionality to log out data flowing through > mainly for debugging purposes. This functionality appears in several > places of the codebase. I think such functionality rise security > concerns in production environments where sensitive information might > be ingested so it is crucial that enabling such functionality has to > be as explicit as possible (avoid implicit side effect setup). > Eg: setting the level of root logger to debug/trace cause that every > other logger will start logging at debug/trace including the ones > logging raw data. > In this jira I would like to provide a patch capturing how I imagined solving > this issue. It can be refined iteratively or used as a basis for a broader > discussion. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLUME-2954) make raw data appearing in log messages explicit
[ https://issues.apache.org/jira/browse/FLUME-2954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Simon updated FLUME-2954: Attachment: (was: FLUME-2954-1.patch) > make raw data appearing in log messages explicit > > > Key: FLUME-2954 > URL: https://issues.apache.org/jira/browse/FLUME-2954 > Project: Flume > Issue Type: Improvement > Components: Channel, Configuration, Sinks+Sources >Affects Versions: v1.6.0 >Reporter: Attila Simon >Assignee: Attila Simon >Priority: Critical > Fix For: v1.7.0 > > Attachments: FLUME-2954-4.patch > > > Flume has built in functionality to log out data flowing through > mainly for debugging purposes. This functionality appears in several > places of the codebase. I think such functionality rise security > concerns in production environments where sensitive information might > be ingested so it is crucial that enabling such functionality has to > be as explicit as possible (avoid implicit side effect setup). > Eg: setting the level of root logger to debug/trace cause that every > other logger will start logging at debug/trace including the ones > logging raw data. > In this jira I would like to provide a patch capturing how I imagined solving > this issue. It can be refined iteratively or used as a basis for a broader > discussion. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLUME-2954) make raw data appearing in log messages explicit
[ https://issues.apache.org/jira/browse/FLUME-2954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Simon updated FLUME-2954: Attachment: FLUME-2954-4.patch > make raw data appearing in log messages explicit > > > Key: FLUME-2954 > URL: https://issues.apache.org/jira/browse/FLUME-2954 > Project: Flume > Issue Type: Improvement > Components: Channel, Configuration, Sinks+Sources >Affects Versions: v1.6.0 >Reporter: Attila Simon >Assignee: Attila Simon >Priority: Critical > Fix For: v1.7.0 > > Attachments: FLUME-2954-1.patch, FLUME-2954-2.patch, > FLUME-2954-4.patch, FLUME-2954.patch > > > Flume has built in functionality to log out data flowing through > mainly for debugging purposes. This functionality appears in several > places of the codebase. I think such functionality rise security > concerns in production environments where sensitive information might > be ingested so it is crucial that enabling such functionality has to > be as explicit as possible (avoid implicit side effect setup). > Eg: setting the level of root logger to debug/trace cause that every > other logger will start logging at debug/trace including the ones > logging raw data. > In this jira I would like to provide a patch capturing how I imagined solving > this issue. It can be refined iteratively or used as a basis for a broader > discussion. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2954) make raw data appearing in log messages explicit
[ https://issues.apache.org/jira/browse/FLUME-2954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15424794#comment-15424794 ] Attila Simon commented on FLUME-2954: - Follow up items: JdbcChannelProviderImpl#261 #431 jdbc url might include password. Since JDBC Channel allows to specify username and password as a separate configuration parameter I guess it is considered safe to remain there. In flume-hive-sink module it is common to log out the HiveEndPoint instances (provided by an external lib). The problem with that it contains a URI string to the Hive meta store. I'm not an expert of Hive and after some search I couldn't find whether security aware configuration can be achieved by not putting secrets into that URI. If yes than it can remain there. If not then I would tackle this in a separate jira. > make raw data appearing in log messages explicit > > > Key: FLUME-2954 > URL: https://issues.apache.org/jira/browse/FLUME-2954 > Project: Flume > Issue Type: Improvement > Components: Channel, Configuration, Sinks+Sources >Affects Versions: v1.6.0 >Reporter: Attila Simon >Assignee: Attila Simon >Priority: Critical > Fix For: v1.7.0 > > Attachments: FLUME-2954-1.patch, FLUME-2954-2.patch, FLUME-2954.patch > > > Flume has built in functionality to log out data flowing through > mainly for debugging purposes. This functionality appears in several > places of the codebase. I think such functionality rise security > concerns in production environments where sensitive information might > be ingested so it is crucial that enabling such functionality has to > be as explicit as possible (avoid implicit side effect setup). > Eg: setting the level of root logger to debug/trace cause that every > other logger will start logging at debug/trace including the ones > logging raw data. > In this jira I would like to provide a patch capturing how I imagined solving > this issue. It can be refined iteratively or used as a basis for a broader > discussion. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2954) make raw data appearing in log messages explicit
[ https://issues.apache.org/jira/browse/FLUME-2954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15424747#comment-15424747 ] Attila Simon commented on FLUME-2954: - Changes made in the spirit of the discussed: flume-ng-channel --- flume-jdbc-channel --- JdbcChannelProviderImpl#98<- fail properties JdbcChannelProviderImpl#261 #431 <- fail properties: jdbc url might include password flume-kafka-channel --- KafkaChannel#230 #253 <- fail properties flume-ng-configuration--- FlumeConfiguration#315 #372 <- fail properties flume-ng-core --- SyslogAvroEventSerializer#150 <- fail data: SyslogEvent.message gets logged GangliaServer#224 #245 <- safe data: only flume component metrics data LoggerSink#95 <- fail data: on purpose AvroSource#347 <- fail data: log whole message MultiportSyslogTCPSource#360<- fail data: log whole message BLOBHandler#70 <- fail data: logs http request headers ---q- flume-ng-embedded-agent --- EmbeddedAgent#155 <- fail properties: printing all config flume-ng-sinks--- flume-hive-sink --- HiveEndPoint has an URI field.<- fail properties It may contain private data (URI string may contain password) as it is excessively logged within this module. Appears in HiveSink#298 #342 #400 #403 #428, HiveWriter#210 #319 #330 #337 #353 #365 #368 #407...) HiveEndPoint is also attached to exception logs as well flume-ng-hbase-sink --- AsyncHBaseSink#641<- safe data: error details gets logged in case of failure flume-ng-kafka-sink --- KafkaSink#179 <- fail data: log whole message KafkaSink#304 <- fail properties flume-ng-morphline-solr-sink--- BlobHandler#98 #113 <- fail data: log http request headers MorphlineSink#139 <- fail data: logs event flume-ng-sources --- flume-kafka-source --- KafkaSource#247 <- fail data: log whole flume-twitter-source--- TwitterSource#110-113 <- fail properties > make raw data appearing in log messages explicit > > > Key: FLUME-2954 > URL: https://issues.apache.org/jira/browse/FLUME-2954 > Project: Flume > Issue Type: Improvement > Components: Channel, Configuration, Sinks+Sources >Affects Versions: v1.6.0 >Reporter: Attila Simon >Assignee: Attila Simon >Priority: Critical > Fix For: v1.7.0 > > Attachments: FLUME-2954-1.patch, FLUME-2954-2.patch, FLUME-2954.patch > > > Flume has built in functionality to log out data flowing through > mainly for debugging purposes. This functionality appears in several > places of the codebase. I think such functionality rise security > concerns in production environments where sensitive information might > be ingested so it is crucial that enabling such functionality has to > be as explicit as possible (avoid implicit side effect setup). > Eg: setting the level of root logger to debug/trace cause that every > other logger will start logging at debug/trace including the ones > logging raw data. > In this jira I would like to provide a patch capturing how I imagined solving > this issue. It can be refined iteratively or used as a basis for a broader > discussion. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (FLUME-2954) make raw data appearing in log messages explicit
[
https://issues.apache.org/jira/browse/FLUME-2954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15424747#comment-15424747
]
Attila Simon edited comment on FLUME-2954 at 8/17/16 3:48 PM:
--
Changes made in the spirit of the discussed:
{noformat}
flume-ng-channel ---
flume-jdbc-channel ---
JdbcChannelProviderImpl#98<- fail properties
JdbcChannelProviderImpl#261 #431 <- fail properties: jdbc url
might include password
flume-kafka-channel ---
KafkaChannel#230 #253 <- fail properties
flume-ng-configuration---
FlumeConfiguration#315 #372 <- fail properties
flume-ng-core ---
SyslogAvroEventSerializer#150 <- fail data: SyslogEvent.message
gets logged
GangliaServer#224 #245 <- safe data: only flume
component metrics data
LoggerSink#95 <- fail data: on purpose
AvroSource#347 <- fail data: log whole message
MultiportSyslogTCPSource#360<- fail data: log whole message
BLOBHandler#70 <- fail data: logs http request
headers
---q-
flume-ng-embedded-agent ---
EmbeddedAgent#155 <- fail properties: printing all
config
flume-ng-sinks---
flume-hive-sink ---
HiveEndPoint has an URI field.<- fail properties
It may contain private data
(URI string may contain password) as it is
excessively logged within this module.
Appears in HiveSink#298 #342 #400 #403 #428,
HiveWriter#210 #319 #330 #337 #353 #365 #368 #407...)
HiveEndPoint is also attached to exception logs as well
flume-ng-hbase-sink ---
AsyncHBaseSink#641<- safe data: error details gets
logged in case of failure
flume-ng-kafka-sink ---
KafkaSink#179 <- fail data: log whole message
KafkaSink#304 <- fail properties
flume-ng-morphline-solr-sink---
BlobHandler#98 #113 <- fail data: log http request
headers
MorphlineSink#139 <- fail data: logs event
flume-ng-sources ---
flume-kafka-source ---
KafkaSource#247 <- fail data: log whole
flume-twitter-source---
TwitterSource#110-113 <- fail properties
{noformat}
was (Author: sati):
Changes made in the spirit of the discussed:
flume-ng-channel ---
flume-jdbc-channel ---
JdbcChannelProviderImpl#98<- fail properties
JdbcChannelProviderImpl#261 #431 <- fail properties: jdbc url
might include password
flume-kafka-channel ---
KafkaChannel#230 #253 <- fail properties
flume-ng-configuration---
FlumeConfiguration#315 #372 <- fail properties
flume-ng-core ---
SyslogAvroEventSerializer#150 <- fail data: SyslogEvent.message
gets logged
GangliaServer#224 #245 <- safe data: only flume
component metrics data
LoggerSink#95 <- fail data: on purpose
AvroSource#347 <- fail data: log whole message
MultiportSyslogTCPSource#360<- fail data: log whole message
BLOBHandler#70 <- fail data: logs http request
headers
---q-
flume-ng-embedded-agent ---
EmbeddedAgent#155 <- fail
[jira] [Updated] (FLUME-2954) make raw data appearing in log messages explicit
[ https://issues.apache.org/jira/browse/FLUME-2954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Simon updated FLUME-2954: Attachment: FLUME-2954-2.patch remove garbage files from diff > make raw data appearing in log messages explicit > > > Key: FLUME-2954 > URL: https://issues.apache.org/jira/browse/FLUME-2954 > Project: Flume > Issue Type: Improvement > Components: Channel, Configuration, Sinks+Sources >Affects Versions: v1.6.0 >Reporter: Attila Simon >Assignee: Attila Simon >Priority: Critical > Fix For: v1.7.0 > > Attachments: FLUME-2954-1.patch, FLUME-2954-2.patch, FLUME-2954.patch > > > Flume has built in functionality to log out data flowing through > mainly for debugging purposes. This functionality appears in several > places of the codebase. I think such functionality rise security > concerns in production environments where sensitive information might > be ingested so it is crucial that enabling such functionality has to > be as explicit as possible (avoid implicit side effect setup). > Eg: setting the level of root logger to debug/trace cause that every > other logger will start logging at debug/trace including the ones > logging raw data. > In this jira I would like to provide a patch capturing how I imagined solving > this issue. It can be refined iteratively or used as a basis for a broader > discussion. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLUME-2954) make raw data appearing in log messages explicit
[ https://issues.apache.org/jira/browse/FLUME-2954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Simon updated FLUME-2954: Attachment: FLUME-2954-1.patch corrected style issues, added docs, compiles, site builds, all unit test passes, distribution target handles the system properties as expected > make raw data appearing in log messages explicit > > > Key: FLUME-2954 > URL: https://issues.apache.org/jira/browse/FLUME-2954 > Project: Flume > Issue Type: Improvement > Components: Channel, Configuration, Sinks+Sources >Affects Versions: v1.6.0 >Reporter: Attila Simon >Assignee: Attila Simon >Priority: Critical > Attachments: FLUME-2954-1.patch, FLUME-2954.patch > > > Flume has built in functionality to log out data flowing through > mainly for debugging purposes. This functionality appears in several > places of the codebase. I think such functionality rise security > concerns in production environments where sensitive information might > be ingested so it is crucial that enabling such functionality has to > be as explicit as possible (avoid implicit side effect setup). > Eg: setting the level of root logger to debug/trace cause that every > other logger will start logging at debug/trace including the ones > logging raw data. > In this jira I would like to provide a patch capturing how I imagined solving > this issue. It can be refined iteratively or used as a basis for a broader > discussion. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
