[jira] [Commented] (FLUME-3115) Upgrade netty library dependency
[ https://issues.apache.org/jira/browse/FLUME-3115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16283893#comment-16283893 ] Hudson commented on FLUME-3115: --- SUCCESS: Integrated in Jenkins build Flume-trunk-hbase-1 #338 (See [https://builds.apache.org/job/Flume-trunk-hbase-1/338/]) FLUME-3115. Update netty library (denes: [http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git=commit=ffc5554438aceb24eabc9815511285f069084968]) * (edit) pom.xml > Upgrade netty library dependency > > > Key: FLUME-3115 > URL: https://issues.apache.org/jira/browse/FLUME-3115 > Project: Flume > Issue Type: Bug >Affects Versions: 1.7.0 >Reporter: Attila Simon >Assignee: Ferenc Szabo >Priority: Critical > Labels: dependency > Fix For: 1.8.1, 1.9.0 > > > ||Group||Artifact||Version used||Upgrade target|| > |io.netty|netty|3.2.2.Final, 3.9.4.Final|4.1.12.Final| > Note: This artifact was moved to: > - New Group io.netty > - New Artifactnetty-all > Security vulnerability: http://www.cvedetails.com/cve/CVE-2014-3488/ > Please do: > - double check the newest version. > - consider to remove a dependency if better alternative is available. > - check whether the lib change would introduce a backward incompatibility (in > which case please add this label `breaking_change` and fix version should be > the next major) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (FLUME-3201) Fix SyslogUtil to handle RFC3164 format in december correctly
[ https://issues.apache.org/jira/browse/FLUME-3201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16279943#comment-16279943 ] Hudson commented on FLUME-3201: --- SUCCESS: Integrated in Jenkins build Flume-trunk-hbase-1 #337 (See [https://builds.apache.org/job/Flume-trunk-hbase-1/337/]) FLUME-3201. Fix SyslogUtil to handle RFC3164 format in December (denes: [http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git=commit=8466766538ed7e8325abf81392008b1d9de8a1f6]) * (edit) flume-ng-core/src/test/java/org/apache/flume/source/TestSyslogUtils.java * (edit) flume-ng-core/src/main/java/org/apache/flume/source/SyslogUtils.java > Fix SyslogUtil to handle RFC3164 format in december correctly > - > > Key: FLUME-3201 > URL: https://issues.apache.org/jira/browse/FLUME-3201 > Project: Flume > Issue Type: Bug >Reporter: Ferenc Szabo >Assignee: Ferenc Szabo > Fix For: 1.9.0 > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (FLUME-3194) upgrade derby to the latest (1.14.1.0) version
[ https://issues.apache.org/jira/browse/FLUME-3194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16278643#comment-16278643 ] Hudson commented on FLUME-3194: --- UNSTABLE: Integrated in Jenkins build Flume-trunk-hbase-1 #336 (See [https://builds.apache.org/job/Flume-trunk-hbase-1/336/]) FLUME-3194 upgrade derby to the latest version (szaboferee: [http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git=commit=8324af363216a45603830909c19a3b26e6625205]) * (edit) pom.xml > upgrade derby to the latest (1.14.1.0) version > -- > > Key: FLUME-3194 > URL: https://issues.apache.org/jira/browse/FLUME-3194 > Project: Flume > Issue Type: Dependency upgrade >Affects Versions: 1.7.0, 1.8.0 >Reporter: Miklos Csanady >Assignee: Miklos Csanady > > Flume uses org.apache.derby:derby 1.11.1.1 > This is affected by this vulnerability: > https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2015-1832 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (FLUME-2050) Upgrade to log4j2 (when GA)
[ https://issues.apache.org/jira/browse/FLUME-2050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16265539#comment-16265539 ] Hudson commented on FLUME-2050: --- SUCCESS: Integrated in Jenkins build Flume-trunk-hbase-1 #335 (See [https://builds.apache.org/job/Flume-trunk-hbase-1/335/]) FLUME-2050 Upgrade to Log4j 2 (rgoers: [http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git=commit=140ea5d2808b7f6fce496e5e2d175821f302f25b]) * (add) flume-ng-sinks/flume-hive-sink/metastore_db/seg0/c771.dat * (add) flume-ng-sinks/flume-hive-sink/metastore_db/seg0/cb1.dat * (add) flume-ng-sinks/flume-hive-sink/metastore_db/seg0/c6f1.dat * (add) flume-ng-sinks/flume-hive-sink/metastore_db/seg0/c2a1.dat * (delete) flume-ng-channels/flume-kafka-channel/src/test/resources/log4j.properties * (add) flume-ng-sinks/flume-hive-sink/metastore_db/seg0/d3390.dat * (add) flume-ng-sinks/flume-hive-sink/metastore_db/seg0/c271.dat * (add) flume-ng-sinks/flume-hive-sink/metastore_db/seg0/c6c0.dat * (add) flume-ng-sinks/flume-hive-sink/metastore_db/service.properties * (add) flume-ng-sinks/flume-hive-sink/metastore_db/seg0/cc40.dat * (add) flume-ng-sinks/flume-hive-sink/metastore_db/seg0/c3a1.dat * (add) flume-ng-sinks/flume-hive-sink/metastore_db/seg0/c121.dat * (add) flume-ng-sinks/flume-hive-sink/metastore_db/seg0/c180.dat * (add) flume-ng-sinks/flume-hive-sink/metastore_db/seg0/d2f71.dat * (add) flume-ng-sinks/flume-hive-sink/metastore_db/seg0/c7c0.dat * (add) flume-ng-sinks/flume-hive-sink/metastore_db/seg0/cb41.dat * (edit) flume-ng-sources/flume-kafka-source/pom.xml * (add) flume-ng-sinks/flume-hive-sink/metastore_db/seg0/c981.dat * (edit) pom.xml * (delete) flume-ng-sinks/flume-http-sink/src/test/resources/log4j.properties * (add) flume-ng-sinks/flume-hive-sink/metastore_db/seg0/c481.dat * (add) flume-ng-sinks/flume-hive-sink/metastore_db/seg0/d3511.dat * (add) flume-ng-sinks/flume-hive-sink/metastore_db/seg0/c10.dat * (add) flume-ng-sinks/flume-hive-sink/metastore_db/seg0/d3521.dat * (add) flume-ng-sinks/flume-hive-sink/metastore_db/seg0/d34d1.dat * (add) flume-ng-sinks/flume-hive-sink/metastore_db/seg0/c251.dat * (add) conf/log4j2.xml * (add) flume-ng-sinks/flume-hive-sink/metastore_db/seg0/d31f0.dat * (add) flume-ng-sinks/flume-hive-sink/metastore_db/seg0/c371.dat * (add) flume-ng-sinks/flume-hive-sink/metastore_db/dbex.lck * (add) flume-ng-sinks/flume-hive-sink/metastore_db/seg0/ca71.dat * (edit) flume-ng-sinks/flume-http-sink/src/main/java/org/apache/flume/sink/http/HttpSink.java * (add) flume-ng-sinks/flume-hive-sink/metastore_db/seg0/d33c0.dat * (edit) flume-ng-embedded-agent/pom.xml * (add) flume-ng-sinks/flume-hive-sink/metastore_db/seg0/c3b1.dat * (add) flume-ng-sinks/flume-hive-sink/metastore_db/seg0/ca60.dat * (edit) flume-ng-configuration/pom.xml * (add) flume-ng-sinks/flume-hive-sink/metastore_db/seg0/c3e1.dat * (add) flume-ng-sinks/flume-hive-sink/metastore_db/seg0/d34c0.dat * (add) flume-ng-sinks/flume-hive-sink/metastore_db/seg0/c101.dat * (edit) flume-ng-sinks/flume-ng-morphline-solr-sink/pom.xml * (delete) flume-ng-sources/flume-kafka-source/src/test/resources/log4j.properties * (add) flume-ng-sinks/flume-hive-sink/metastore_db/seg0/cbc1.dat * (add) flume-ng-sinks/flume-hive-sink/metastore_db/seg0/d3441.dat * (add) flume-ng-sinks/flume-hive-sink/metastore_db/seg0/c2d0.dat * (edit) flume-ng-sinks/flume-hive-sink/pom.xml * (edit) flume-ng-legacy-sources/flume-thrift-source/pom.xml * (add) flume-ng-sinks/flume-hive-sink/metastore_db/seg0/d3320.dat * (add) flume-ng-sinks/flume-hive-sink/metastore_db/seg0/d32a1.dat * (add) flume-ng-sinks/flume-hive-sink/metastore_db/seg0/d2fd1.dat * (add) flume-ng-sinks/flume-hive-sink/metastore_db/seg0/d3340.dat * (add) flume-ng-sinks/flume-hive-sink/metastore_db/seg0/d3200.dat * (edit) flume-ng-sinks/flume-ng-hbase-sink/pom.xml * (add) flume-ng-sinks/flume-hive-sink/metastore_db/seg0/d33f0.dat * (add) flume-ng-sinks/flume-hive-sink/metastore_db/seg0/cae1.dat * (add) flume-ng-sinks/flume-hive-sink/metastore_db/seg0/d32b1.dat * (add) flume-ng-sinks/flume-hive-sink/metastore_db/seg0/c680.dat * (add) flume-ng-sinks/flume-hive-sink/metastore_db/seg0/c6b1.dat * (add) flume-ng-sinks/flume-hive-sink/metastore_db/seg0/d3311.dat * (add) flume-ng-sinks/flume-hive-sink/metastore_db/seg0/c8c1.dat * (add) flume-ng-sinks/flume-hive-sink/metastore_db/seg0/cab0.dat * (add) flume-ng-sinks/flume-hive-sink/metastore_db/seg0/cbd1.dat * (add) flume-ng-sinks/flume-hive-sink/metastore_db/seg0/ca81.dat * (add) flume-ng-sinks/flume-hive-sink/metastore_db/seg0/c8b1.dat * (add) flume-ng-sinks/flume-hive-sink/metastore_db/seg0/c351.dat * (add) flume-ng-sinks/flume-hive-sink/metastore_db/seg0/c451.dat * (add) flume-ng-sinks/flume-hive-sink/metastore_db/seg0/c361.dat * (add) flume-ng-sinks/flume-hive-sink/metastore_db/seg0/d2f01.dat * (add) flume-ng-sdk/src/test/resources/log4j2.xml * (add)
[jira] [Commented] (FLUME-3158) Upgrade surefire version and config
[ https://issues.apache.org/jira/browse/FLUME-3158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16260608#comment-16260608 ] Hudson commented on FLUME-3158: --- SUCCESS: Integrated in Jenkins build Flume-trunk-hbase-1 #334 (See [https://builds.apache.org/job/Flume-trunk-hbase-1/334/]) FLUME-3158: Upgrade surefire version and config (szaboferee: [http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git=commit=d116f9707d5b5da501b1bb9a5fca812db7ce849b]) * (edit) pom.xml * (edit) flume-ng-sinks/flume-ng-morphline-solr-sink/pom.xml > Upgrade surefire version and config > --- > > Key: FLUME-3158 > URL: https://issues.apache.org/jira/browse/FLUME-3158 > Project: Flume > Issue Type: Sub-task >Reporter: Ferenc Szabo >Assignee: Miklos Csanady > Fix For: 1.8.1, 1.9.0 > > > set surefire version to the latest and configure it to rerun failed tests. > This way the flaky tests will not brake the build as often -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (FLUME-3195) Split up the TestKafkaChannel class
[ https://issues.apache.org/jira/browse/FLUME-3195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16255279#comment-16255279 ] Hudson commented on FLUME-3195: --- FAILURE: Integrated in Jenkins build Flume-trunk-hbase-1 #332 (See [https://builds.apache.org/job/Flume-trunk-hbase-1/332/]) FLUME-3195. Split the KafkaChannelTest to avoid timeouts (fszabo: [http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git=commit=f3c4a40ed708bc1b2dd7630df103e01a5ebc4482]) * (add) flume-ng-channels/flume-kafka-channel/src/test/java/org/apache/flume/channel/kafka/TestParseAsFlumeEvent.java * (add) flume-ng-channels/flume-kafka-channel/src/test/java/org/apache/flume/channel/kafka/TestBasicFunctionality.java * (delete) flume-ng-channels/flume-kafka-channel/src/test/java/org/apache/flume/channel/kafka/TestKafkaChannel.java * (add) flume-ng-channels/flume-kafka-channel/src/test/java/org/apache/flume/channel/kafka/TestKafkaChannelBase.java * (add) flume-ng-channels/flume-kafka-channel/src/test/java/org/apache/flume/channel/kafka/TestOffsetsAndMigration.java * (add) flume-ng-channels/flume-kafka-channel/src/test/java/org/apache/flume/channel/kafka/TestPartitions.java * (add) flume-ng-channels/flume-kafka-channel/src/test/java/org/apache/flume/channel/kafka/TestRollback.java > Split up the TestKafkaChannel class > --- > > Key: FLUME-3195 > URL: https://issues.apache.org/jira/browse/FLUME-3195 > Project: Flume > Issue Type: Test > Components: Kafka Channel, Test >Reporter: Viktor Somogyi >Assignee: Viktor Somogyi >Priority: Minor > Fix For: 1.9.0 > > > The TestKafkaChannel class is big (27 tests in 1 class) and sometimes times > out. Splitting it up to smaller classes mitigates the problem. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (FLUME-2698) Upgrade Jetty Version
[
https://issues.apache.org/jira/browse/FLUME-2698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16220049#comment-16220049
]
Hudson commented on FLUME-2698:
---
SUCCESS: Integrated in Jenkins build Flume-trunk-hbase-1 #330 (See
[https://builds.apache.org/job/Flume-trunk-hbase-1/330/])
FLUME-2698. Upgrade Jetty Version (denes:
[http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git=commit=580f781341b42672f234af069b9190ff1cc00dca])
* (add)
flume-ng-core/src/main/java/org/apache/flume/tools/FlumeBeanConfigurator.java
* (edit)
flume-ng-core/src/test/java/org/apache/flume/source/http/TestHTTPSource.java
* (edit) flume-ng-core/pom.xml
* (add)
flume-ng-core/src/test/java/org/apache/flume/tools/TestFlumeConfigurator.java
* (edit) flume-ng-legacy-sources/flume-thrift-source/pom.xml
* (edit) flume-ng-sources/flume-scribe-source/pom.xml
* (edit)
flume-ng-core/src/test/java/org/apache/flume/instrumentation/http/TestHTTPMetricsServer.java
* (edit)
flume-ng-sinks/flume-ng-morphline-solr-sink/src/test/java/org/apache/flume/sink/solr/morphline/FlumeHttpServletRequestWrapper.java
* (edit)
flume-ng-core/src/main/java/org/apache/flume/source/http/HTTPSourceConfigurationConstants.java
* (edit)
flume-ng-core/src/main/java/org/apache/flume/tools/HTTPServerConstraintUtil.java
* (edit) flume-ng-legacy-sources/flume-avro-source/pom.xml
* (edit)
flume-ng-core/src/test/java/org/apache/flume/source/http/TestBLOBHandler.java
* (edit) flume-ng-doc/sphinx/FlumeUserGuide.rst
* (edit) pom.xml
* (edit)
flume-ng-core/src/main/java/org/apache/flume/instrumentation/http/HTTPMetricsServer.java
* (edit)
flume-ng-core/src/main/java/org/apache/flume/source/http/HTTPSource.java
* (edit)
flume-ng-core/src/test/java/org/apache/flume/source/http/FlumeHttpServletRequestWrapper.java
* (edit) flume-ng-sinks/flume-http-sink/pom.xml
> Upgrade Jetty Version
> -
>
> Key: FLUME-2698
> URL: https://issues.apache.org/jira/browse/FLUME-2698
> Project: Flume
> Issue Type: Bug
> Components: Web
>Affects Versions: 1.6.0, 1.5.1, 1.7.0
>Reporter: Joakim Erdfelt
>Assignee: Tristan Stevens
> Labels: dependency
> Fix For: 1.9.0
>
> Attachments: HttpSourceLoadTest.jmx, jetty-6-http-test.csv,
> jetty-6-https-test.csv, jetty-9-http-test.csv, jetty-9-https-test.csv
>
>
> Flume depends on Jetty 6
> {code:xml}
>
> org.mortbay.jetty
> jetty-util
> 6.1.26
>
> {code}
> Which was EOL (End of Life) back in 2010 and is no longer fit for production
> use (without heavy customizations and modifications like Google does for GAE,
> just to keep it safe and vulnerability free)
> Jetty was moved to Eclipse.org back during the Jetty 7 days.
> http://eclipse.org/jetty/
> Note that [Jetty 7 and Jetty 8 are now also
> EOL|https://dev.eclipse.org/mhonarc/lists/jetty-announce/msg00069.html] (as
> of 2014)
> Jetty 9 is the only stable and supported version of Jetty now.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (FLUME-3186) Make asyncHbaseClient configuration parameters available from flume config
[ https://issues.apache.org/jira/browse/FLUME-3186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16211481#comment-16211481 ] Hudson commented on FLUME-3186: --- UNSTABLE: Integrated in Jenkins build Flume-trunk-hbase-1 #329 (See [https://builds.apache.org/job/Flume-trunk-hbase-1/329/]) FLUME-3186. Make asyncHbaseClient config parameters available from Flume (denes: [http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git=commit=ed288acba39bfd611c10b338e36224c1415c2c4c]) * (edit) flume-ng-sinks/flume-ng-hbase-sink/src/main/java/org/apache/flume/sink/hbase/HBaseSinkConfigurationConstants.java * (edit) flume-ng-doc/sphinx/FlumeUserGuide.rst * (add) flume-ng-sinks/flume-ng-hbase-sink/src/test/java/org/apache/flume/sink/hbase/TestAsyncHBaseSinkConfiguration.java * (edit) flume-ng-sinks/flume-ng-hbase-sink/src/main/java/org/apache/flume/sink/hbase/AsyncHBaseSink.java > Make asyncHbaseClient configuration parameters available from flume config > -- > > Key: FLUME-3186 > URL: https://issues.apache.org/jira/browse/FLUME-3186 > Project: Flume > Issue Type: Bug > Components: Sinks+Sources >Affects Versions: 1.9.0 >Reporter: Miklos Csanady >Assignee: Miklos Csanady > > In 1.7 version AsyncHbaseClient it is possible to pass all configuration to > the constructor. > This makes it possible to pass all the parameters to this library with a > specific prefix. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (FLUME-3159) make jdbc channel test use in-memory database with random name to make the test faster
[ https://issues.apache.org/jira/browse/FLUME-3159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16193000#comment-16193000 ] Hudson commented on FLUME-3159: --- SUCCESS: Integrated in Jenkins build Flume-trunk-hbase-1 #328 (See [https://builds.apache.org/job/Flume-trunk-hbase-1/328/]) FLUME-3159. Setting JdbcChannel test connection to in-memory derby (denes: [http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git=commit=c7b95a746ba8ffd0337540e3233032d2b071c648]) * (edit) flume-ng-channels/flume-jdbc-channel/src/test/java/org/apache/flume/channel/jdbc/BaseJdbcChannelProviderTest.java > make jdbc channel test use in-memory database with random name to make the > test faster > -- > > Key: FLUME-3159 > URL: https://issues.apache.org/jira/browse/FLUME-3159 > Project: Flume > Issue Type: Sub-task >Reporter: Ferenc Szabo >Assignee: Ferenc Szabo >Priority: Minor > Fix For: 1.8.0 > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (FLUME-3164) Update pom.xml version on trunk
[ https://issues.apache.org/jira/browse/FLUME-3164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16165006#comment-16165006 ] Hudson commented on FLUME-3164: --- UNSTABLE: Integrated in Jenkins build Flume-trunk-hbase-1 #326 (See [https://builds.apache.org/job/Flume-trunk-hbase-1/326/]) FLUME-3164. Update version to 1.9.0-SNAPSHOT in pom.xmls and (denes: [http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git=commit=82e9f4d4845934ea77c26be491dc5a856970c7bd]) * (edit) flume-ng-sinks/flume-ng-kafka-sink/pom.xml * (edit) flume-ng-sources/flume-jms-source/pom.xml * (edit) flume-ng-doc/sphinx/FlumeDeveloperGuide.rst * (edit) flume-ng-tests/pom.xml * (edit) flume-ng-doc/sphinx/FlumeUserGuide.rst * (edit) flume-ng-legacy-sources/pom.xml * (edit) flume-ng-sinks/flume-dataset-sink/pom.xml * (edit) flume-ng-sinks/flume-http-sink/pom.xml * (edit) pom.xml * (edit) flume-ng-sources/flume-taildir-source/pom.xml * (edit) flume-shared/pom.xml * (edit) flume-ng-channels/flume-kafka-channel/pom.xml * (edit) flume-ng-sources/flume-kafka-source/pom.xml * (edit) flume-ng-sinks/flume-hdfs-sink/pom.xml * (edit) flume-ng-sinks/flume-ng-morphline-solr-sink/pom.xml * (edit) flume-ng-channels/flume-jdbc-channel/pom.xml * (edit) flume-ng-legacy-sources/flume-thrift-source/pom.xml * (edit) flume-checkstyle/pom.xml * (edit) flume-tools/pom.xml * (edit) flume-ng-channels/pom.xml * (edit) flume-ng-legacy-sources/flume-avro-source/pom.xml * (edit) flume-ng-sources/pom.xml * (edit) flume-ng-channels/flume-spillable-memory-channel/pom.xml * (edit) flume-ng-sinks/flume-ng-elasticsearch-sink/pom.xml * (edit) flume-ng-node/pom.xml * (edit) flume-ng-channels/flume-file-channel/pom.xml * (edit) flume-ng-clients/pom.xml * (edit) flume-ng-sources/flume-scribe-source/pom.xml * (edit) flume-ng-auth/pom.xml * (edit) flume-ng-embedded-agent/pom.xml * (edit) flume-ng-dist/pom.xml * (edit) flume-ng-clients/flume-ng-log4jappender/pom.xml * (edit) flume-ng-core/pom.xml * (edit) flume-ng-sinks/flume-ng-hbase-sink/pom.xml * (edit) flume-ng-sources/flume-twitter-source/pom.xml * (edit) flume-ng-sinks/pom.xml * (edit) flume-shared/flume-shared-kafka-test/pom.xml * (edit) flume-ng-sdk/pom.xml * (edit) flume-ng-configuration/pom.xml * (edit) flume-ng-sinks/flume-irc-sink/pom.xml * (edit) flume-ng-doc/sphinx/index.rst * (edit) flume-ng-sinks/flume-hive-sink/pom.xml > Update pom.xml version on trunk > --- > > Key: FLUME-3164 > URL: https://issues.apache.org/jira/browse/FLUME-3164 > Project: Flume > Issue Type: Sub-task >Reporter: Ferenc Szabo > Fix For: 1.8.0 > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (FLUME-3168) Update ChangeLog for 1.8 Release
[ https://issues.apache.org/jira/browse/FLUME-3168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16165003#comment-16165003 ] Hudson commented on FLUME-3168: --- UNSTABLE: Integrated in Jenkins build Flume-trunk-hbase-1 #326 (See [https://builds.apache.org/job/Flume-trunk-hbase-1/326/]) FLUME-3168. Update CHANGELOG for 1.8.0 (denes: [http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git=commit=0c4260801c6dfee0003e437b0a7ca5a6cc7391f3]) * (edit) CHANGELOG > Update ChangeLog for 1.8 Release > > > Key: FLUME-3168 > URL: https://issues.apache.org/jira/browse/FLUME-3168 > Project: Flume > Issue Type: Sub-task >Reporter: Ferenc Szabo >Assignee: Denes Arvay > Fix For: 1.8.0 > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (FLUME-3178) Update ReleaseNotes for 1.8 Release
[ https://issues.apache.org/jira/browse/FLUME-3178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16165004#comment-16165004 ] Hudson commented on FLUME-3178: --- UNSTABLE: Integrated in Jenkins build Flume-trunk-hbase-1 #326 (See [https://builds.apache.org/job/Flume-trunk-hbase-1/326/]) FLUME-3178. Update RELEASE-NOTES for 1.8.0 (denes: [http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git=commit=2bdb8323eb1abb28d443471b19a3a5bdae58f0bb]) * (edit) RELEASE-NOTES > Update ReleaseNotes for 1.8 Release > --- > > Key: FLUME-3178 > URL: https://issues.apache.org/jira/browse/FLUME-3178 > Project: Flume > Issue Type: Sub-task >Reporter: Denes Arvay >Assignee: Denes Arvay > Fix For: 1.8.0 > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (FLUME-3166) Update License file for 1.8 release
[ https://issues.apache.org/jira/browse/FLUME-3166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16165005#comment-16165005 ] Hudson commented on FLUME-3166: --- UNSTABLE: Integrated in Jenkins build Flume-trunk-hbase-1 #326 (See [https://builds.apache.org/job/Flume-trunk-hbase-1/326/]) FLUME-3166. Update LICENSE (denes: [http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git=commit=7e6c3f1f4122648b2ed9ca5a77f57b7139a369cc]) * (edit) LICENSE > Update License file for 1.8 release > --- > > Key: FLUME-3166 > URL: https://issues.apache.org/jira/browse/FLUME-3166 > Project: Flume > Issue Type: Sub-task >Reporter: Ferenc Szabo >Assignee: Denes Arvay > Fix For: 1.8.0 > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (FLUME-3175) Javadoc generation fails due to Java8's strict doclint
[ https://issues.apache.org/jira/browse/FLUME-3175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16164544#comment-16164544 ] Hudson commented on FLUME-3175: --- UNSTABLE: Integrated in Jenkins build Flume-trunk-hbase-1 #324 (See [https://builds.apache.org/job/Flume-trunk-hbase-1/324/]) FLUME-3175. Turn off doclint for javadoc generation (denes: [http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git=commit=1541f089a5dec9b21549f6a6e961ef835a8b68bb]) * (edit) pom.xml > Javadoc generation fails due to Java8's strict doclint > -- > > Key: FLUME-3175 > URL: https://issues.apache.org/jira/browse/FLUME-3175 > Project: Flume > Issue Type: Bug >Affects Versions: 1.7.0 >Reporter: Denes Arvay >Assignee: Denes Arvay > Fix For: 1.8.0 > > > I'd suggest to turn it off for now, but in the long run the javadoc errors > should be fixed. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (FLUME-3046) Kafka Sink and Source Configuration Improvements
[ https://issues.apache.org/jira/browse/FLUME-3046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163646#comment-16163646 ] Hudson commented on FLUME-3046: --- FAILURE: Integrated in Jenkins build Flume-trunk-hbase-1 #323 (See [https://builds.apache.org/job/Flume-trunk-hbase-1/323/]) FLUME-3046. Kafka Sink and Source Configuration Improvements (denes: [http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git=commit=54e2728a8e141ee63704018c4497bbe083c0f75f]) * (edit) flume-ng-doc/sphinx/FlumeUserGuide.rst * (edit) flume-ng-sinks/flume-ng-kafka-sink/src/main/java/org/apache/flume/sink/kafka/KafkaSinkConstants.java * (edit) flume-ng-sources/flume-kafka-source/src/test/java/org/apache/flume/source/kafka/TestKafkaSource.java * (edit) flume-ng-sources/flume-kafka-source/src/main/java/org/apache/flume/source/kafka/KafkaSource.java * (edit) flume-ng-sources/flume-kafka-source/src/main/java/org/apache/flume/source/kafka/KafkaSourceConstants.java * (edit) flume-ng-sinks/flume-ng-kafka-sink/src/main/java/org/apache/flume/sink/kafka/KafkaSink.java * (edit) flume-ng-sinks/flume-ng-kafka-sink/src/test/java/org/apache/flume/sink/kafka/TestKafkaSink.java > Kafka Sink and Source Configuration Improvements > > > Key: FLUME-3046 > URL: https://issues.apache.org/jira/browse/FLUME-3046 > Project: Flume > Issue Type: Bug >Affects Versions: 1.7.0 >Reporter: Jeff Holoman >Assignee: Tristan Stevens > Fix For: 1.8.0 > > > Currently the Kafka Source sets the header for the topic. The sink reads this > value, rather than the statically defined topic value. We should fix this so > that you can either change the topic header that is used, or just choose to > prefer the statically defined topic in the sink. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (FLUME-3173) Upgrade joda-time
[
https://issues.apache.org/jira/browse/FLUME-3173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163031#comment-16163031
]
Hudson commented on FLUME-3173:
---
FAILURE: Integrated in Jenkins build Flume-trunk-hbase-1 #320 (See
[https://builds.apache.org/job/Flume-trunk-hbase-1/320/])
FLUME-3173. Upgrade joda-time to 2.9.9 (denes:
[http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git=commit=d434d23dadc411e5d7486447316172c495d70f22])
* (edit) pom.xml
> Upgrade joda-time
> -
>
> Key: FLUME-3173
> URL: https://issues.apache.org/jira/browse/FLUME-3173
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
>Affects Versions: 1.7.0
>Reporter: Azat Nizametdinov
>Assignee: Miklos Csanady
> Fix For: 1.8.0
>
>
> Flume 1.7 depends on joda-time version 2.1 which uses outdated tz database.
> For example following code
> {code}
> new org.joda.time.DateTime(
> org.joda.time.DateTimeZone.forID("Europe/Moscow")
> ).toString()
> {code}
> returns time with offset {{+04:00}}, but Moscow timezone is UTC+3 since 2014.
> Furthermore this version of joda-time does not allow to specify custom tz
> databse folder in contrast to newer versions.
> It affects {{RegexExtractorInterceptorMillisSerializer}}. Test to reproduce
> the bug:
> {code}
> public void testMoscowTimezone() throws Exception {
> TimeZone.setDefault(TimeZone.getTimeZone("Europe/Moscow"));
> String pattern = "-MM-dd HH:mm:ss";
> SimpleDateFormat format = new SimpleDateFormat(pattern);
> String dateStr = "2017-09-10 10:00:00";
> Date expectedDate = format.parse(dateStr);
> RegexExtractorInterceptorMillisSerializer sut = new
> RegexExtractorInterceptorMillisSerializer();
> Context context = new Context();
> context.put("pattern", pattern);
> sut.configure(context);
> assertEquals(String.valueOf(expectedDate.getTime()),
> sut.serialize(dateStr));
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (FLUME-1520) Timestamp interceptor should support custom headers
[ https://issues.apache.org/jira/browse/FLUME-1520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16161964#comment-16161964 ] Hudson commented on FLUME-1520: --- FAILURE: Integrated in Jenkins build Flume-trunk-hbase-1 #319 (See [https://builds.apache.org/job/Flume-trunk-hbase-1/319/]) FLUME-1520. Timestamp interceptor should support custom headers (denes: [http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git=commit=13771c905316052d3e94aeb3b4a0d49a27c0f852]) * (edit) flume-ng-core/src/test/java/org/apache/flume/interceptor/TestTimestampInterceptor.java * (edit) flume-ng-core/src/main/java/org/apache/flume/interceptor/TimestampInterceptor.java * (edit) flume-ng-doc/sphinx/FlumeUserGuide.rst > Timestamp interceptor should support custom headers > --- > > Key: FLUME-1520 > URL: https://issues.apache.org/jira/browse/FLUME-1520 > Project: Flume > Issue Type: Improvement >Reporter: Hari Shreedharan >Assignee: Hari Shreedharan > Fix For: 1.8.0 > > Attachments: FLUME-1520-2.patch, FLUME-1520-3.patch, FLUME-1520.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (FLUME-3127) Upgrade libfb303 library dependency
[
https://issues.apache.org/jira/browse/FLUME-3127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16161311#comment-16161311
]
Hudson commented on FLUME-3127:
---
FAILURE: Integrated in Jenkins build Flume-trunk-hbase-1 #317 (See
[https://builds.apache.org/job/Flume-trunk-hbase-1/317/])
FLUME-3127. Upgrade libfb303 library dependency (denes:
[http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git=commit=773555c5c7d09650f8f4173ced232954b1a3a361])
* (edit)
flume-ng-legacy-sources/flume-thrift-source/src/main/java/com/cloudera/flume/handlers/thrift/Priority.java
* (edit)
flume-ng-sdk/src/main/java/org/apache/flume/thrift/ThriftSourceProtocol.java
* (edit)
flume-ng-sources/flume-scribe-source/src/main/java/org/apache/flume/source/scribe/LogEntry.java
* (edit) pom.xml
* (edit)
flume-ng-legacy-sources/flume-thrift-source/src/main/java/com/cloudera/flume/handlers/thrift/EventStatus.java
* (edit)
flume-ng-sources/flume-scribe-source/src/main/java/org/apache/flume/source/scribe/Scribe.java
* (edit)
flume-ng-sdk/src/main/java/org/apache/flume/thrift/ThriftFlumeEvent.java
* (edit)
flume-ng-sources/flume-scribe-source/src/main/java/org/apache/flume/source/scribe/ScribeSource.java
* (edit)
flume-ng-legacy-sources/flume-thrift-source/src/main/java/com/cloudera/flume/handlers/thrift/ThriftFlumeEvent.java
* (edit)
flume-ng-legacy-sources/flume-thrift-source/src/main/java/com/cloudera/flume/handlers/thrift/ThriftFlumeEventServer.java
* (edit)
flume-ng-sources/flume-scribe-source/src/main/java/org/apache/flume/source/scribe/ResultCode.java
* (edit) flume-ng-sdk/src/main/java/org/apache/flume/thrift/Status.java
> Upgrade libfb303 library dependency
> ---
>
> Key: FLUME-3127
> URL: https://issues.apache.org/jira/browse/FLUME-3127
> Project: Flume
> Issue Type: Bug
>Affects Versions: 1.7.0
>Reporter: Attila Simon
>Assignee: Ferenc Szabo
>Priority: Critical
> Labels: dependency
> Fix For: 1.8.0
>
>
> ||Group||Artifact||Version used||Upgrade target||
> |org.apache.thrift|libthrift|0.9.0|0.9.3,0.10.0|
> |org.apache.thrift|libfb303|0.9.0|0.9.3|
> Security vulnerability: http://www.cvedetails.com/cve/CVE-2015-3254/
> Maven repository:
> - https://mvnrepository.com/artifact/org.apache.thrift/libthrift
> - https://mvnrepository.com/artifact/org.apache.thrift/libfb303
> Please do:
> - CVE might be a false alarm or mistake. Please double check.
> - double check the newest version.
> - consider to remove a dependency if better alternative is available.
> - check whether the lib change would introduce a backward incompatibility (in
> which case please add this label `breaking_change` and fix version should be
> the next major)
> Excerpt from mvn dependency:tree
> {noformat}
> org.apache.flume:flume-ng-sdk:jar:1.8.0-SNAPSHOT
> \- org.apache.thrift:libthrift:jar:0.9.0:compile
> org.apache.flume.flume-ng-sinks:flume-hive-sink:jar:1.8.0-SNAPSHOT
> +- org.apache.hive.hcatalog:hive-hcatalog-streaming:jar:1.0.0:provided
> | +- org.apache.hive:hive-metastore:jar:1.0.0:provided
> | | \- org.apache.thrift:libfb303:jar:0.9.0:provided
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (FLUME-3156) UserGuide document questions
[ https://issues.apache.org/jira/browse/FLUME-3156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16152718#comment-16152718 ] Hudson commented on FLUME-3156: --- SUCCESS: Integrated in Jenkins build Flume-trunk-hbase-1 #316 (See [https://builds.apache.org/job/Flume-trunk-hbase-1/316/]) FLUME-3156. Update example configuration of Kafka Sink (denes: [http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git=commit=d3552627869b45956181149f4b7ecf0e344080b7]) * (edit) flume-ng-doc/sphinx/FlumeUserGuide.rst > UserGuide document questions > > > Key: FLUME-3156 > URL: https://issues.apache.org/jira/browse/FLUME-3156 > Project: Flume > Issue Type: Documentation > Components: Docs >Affects Versions: 1.7.0 >Reporter: Robin Wang >Priority: Trivial > > Hello Team, > I found there is a mistake in the User Guide : > http://flume.apache.org/FlumeUserGuide.html > which > a1.sinks.ki.kafka.producer.compression.type = snappy > ki should be k1 here. it might misleading the junior user. > Just wish flume more and more powerful. > thanks ! -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (FLUME-3157) Refactor TestHDFSEventSinkOnMiniCluster to not use LeaseManager private API
[ https://issues.apache.org/jira/browse/FLUME-3157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16152482#comment-16152482 ] Hudson commented on FLUME-3157: --- UNSTABLE: Integrated in Jenkins build Flume-trunk-hbase-1 #314 (See [https://builds.apache.org/job/Flume-trunk-hbase-1/314/]) FLUME-3157. Refactor TestHDFSEventSinkOnMiniCluster to not use (denes: [http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git=commit=32c167dccd711f163221b374bf6a4af68e66f0df]) * (edit) flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSinkOnMiniCluster.java > Refactor TestHDFSEventSinkOnMiniCluster to not use LeaseManager private API > --- > > Key: FLUME-3157 > URL: https://issues.apache.org/jira/browse/FLUME-3157 > Project: Flume > Issue Type: Bug >Affects Versions: 1.7.0 >Reporter: Ferenc Szabo >Assignee: Ferenc Szabo > Fix For: 1.8.0 > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (FLUME-3154) Add HBase client version check to AsyncHBaseSink and HBaseSink
[ https://issues.apache.org/jira/browse/FLUME-3154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16147392#comment-16147392 ] Hudson commented on FLUME-3154: --- SUCCESS: Integrated in Jenkins build Flume-trunk-hbase-1 #313 (See [https://builds.apache.org/job/Flume-trunk-hbase-1/313/]) FLUME-3154. Add HBase client version check to AsyncHBaseSink and (denes: [http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git=commit=a7d2a289a67f057a5cdb28dabbd3e651613dfa3e]) * (edit) flume-ng-sinks/flume-ng-hbase-sink/src/main/java/org/apache/flume/sink/hbase/AsyncHBaseSink.java * (edit) flume-ng-sinks/flume-ng-hbase-sink/src/main/java/org/apache/flume/sink/hbase/HBaseSink.java * (add) flume-ng-sinks/flume-ng-hbase-sink/src/main/java/org/apache/flume/sink/hbase/HBaseVersionCheck.java > Add HBase client version check to AsyncHBaseSink and HBaseSink > -- > > Key: FLUME-3154 > URL: https://issues.apache.org/jira/browse/FLUME-3154 > Project: Flume > Issue Type: Sub-task >Reporter: Miklos Csanady >Assignee: Miklos Csanady >Priority: Minor > Fix For: 1.8.0 > > > The current AsyncHBase sink should run only with HBase < 2 > If it finds an incompatible HBase client on the classpath it should fail with > some meaningful error message > The same check should be performed with hbasesink -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (FLUME-3155) Use batch mode in mvn to fix Travis CI error
[ https://issues.apache.org/jira/browse/FLUME-3155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16143826#comment-16143826 ] Hudson commented on FLUME-3155: --- SUCCESS: Integrated in Jenkins build Flume-trunk-hbase-1 #312 (See [https://builds.apache.org/job/Flume-trunk-hbase-1/312/]) FLUME-3155. Use batch mode in mvn to fix Travis CI error (denes: [http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git=commit=be25c5e1142a8d8b637df18de62f1a0a8759ea82]) * (edit) .travis.yml > Use batch mode in mvn to fix Travis CI error > > > Key: FLUME-3155 > URL: https://issues.apache.org/jira/browse/FLUME-3155 > Project: Flume > Issue Type: Bug >Reporter: Miklos Csanady >Assignee: Miklos Csanady > > The Travis test failing with this error: > The log length has exceeded the limit of 4 MB (this usually means that the > test suite is raising the same exception over and over). > The job has been terminated > The mvn -B switch should be used to reduce output verbosity. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (FLUME-3132) Upgrade tomcat jasper library dependencies
[
https://issues.apache.org/jira/browse/FLUME-3132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16140525#comment-16140525
]
Hudson commented on FLUME-3132:
---
UNSTABLE: Integrated in Jenkins build Flume-trunk-hbase-1 #311 (See
[https://builds.apache.org/job/Flume-trunk-hbase-1/311/])
FLUME-3132. Upgrade tomcat jasper library dependencies (denes:
[http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git=commit=0cc7542ddc18239754befb19408dc3aca4337a66])
* (edit) pom.xml
> Upgrade tomcat jasper library dependencies
> --
>
> Key: FLUME-3132
> URL: https://issues.apache.org/jira/browse/FLUME-3132
> Project: Flume
> Issue Type: Bug
>Affects Versions: 1.7.0
>Reporter: Attila Simon
>Assignee: Ferenc Szabo
>Priority: Critical
> Labels: dependency
> Fix For: 1.8.0
>
>
> ||Group||Artifact||Version used||Upgrade target||
> |tomcat|jasper-compiler|5.5.23|8.5.x|
> |tomcat|jasper-runtime|5.5.23|8.5.x|
> Security vulnerability:
> - https://www.cvedetails.com/cve/CVE-2011-1318/
> -
> http://www.cvedetails.com/vulnerability-list/vendor_id-45/product_id-887/Apache-Tomcat.html
> Maven repositories:
> - https://mvnrepository.com/artifact/org.apache.tomcat/tomcat-jasper
> Note: These artifacts were moved to:
> * New Group org.apache.tomcat
> * New Artifact
> Please do:
> - CVE might be a false alarm or mistake. Please double check.
> - double check the newest version.
> - consider to remove a dependency if better alternative is available.
> - check whether the lib change would introduce a backward incompatibility (in
> which case please add this label `breaking_change` and fix version should be
> the next major)
> Excerpt from mvn dependency:tree
> {noformat}
> org.apache.flume:flume-ng-auth:jar:1.8.0-SNAPSHOT
> +- org.apache.hadoop:hadoop-common:jar:2.4.0:compile
> | +- tomcat:jasper-compiler:jar:5.5.23:runtime
> | +- tomcat:jasper-runtime:jar:5.5.23:runtime
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (FLUME-3131) Upgrade spring framework library dependencies
[
https://issues.apache.org/jira/browse/FLUME-3131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16140380#comment-16140380
]
Hudson commented on FLUME-3131:
---
FAILURE: Integrated in Jenkins build Flume-trunk-hbase-1 #310 (See
[https://builds.apache.org/job/Flume-trunk-hbase-1/310/])
FLUME-3131. Upgrade Spring Framework library dependencies (denes:
[http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git=commit=aa1aea07b7e2bd25e28efdc262239ec501fbf086])
* (edit) pom.xml
* (edit) flume-ng-sources/flume-jms-source/pom.xml
> Upgrade spring framework library dependencies
> -
>
> Key: FLUME-3131
> URL: https://issues.apache.org/jira/browse/FLUME-3131
> Project: Flume
> Issue Type: Bug
>Affects Versions: 1.7.0
>Reporter: Attila Simon
>Assignee: Ferenc Szabo
>Priority: Critical
> Labels: dependency
> Fix For: 1.8.0
>
> Attachments: FLUME-3131-1.patch, FLUME-3131.patch
>
>
> ||Group||Artifact||Version used||Upgrade target||
> |org.springframework|spring-aop|3.0.7.RELEASE|4.3.9.RELEASE,|
> |org.springframework|spring-context|3.0.7.RELEASE|4.3.9.RELEASE,|
> |org.springframework|spring-core|3.0.7.RELEASE|4.3.9.RELEASE,|
> Security vulnerability:
> https://www.cvedetails.com/vulnerability-list/vendor_id-9664/product_id-17274/Springsource-Spring-Framework.html
> Maven repositories:
> - https://mvnrepository.com/artifact/org.springframework/spring-aop
> - https://mvnrepository.com/artifact/org.springframework/spring-context
> - https://mvnrepository.com/artifact/org.springframework/spring-core
> Please do:
> - CVE might be a false alarm or mistake. Please double check.
> - double check the newest version.
> - consider to remove a dependency if better alternative is available.
> - check whether the lib change would introduce a backward incompatibility (in
> which case please add this label `breaking_change` and fix version should be
> the next major)
> Excerpt from mvn dependency:tree
> {noformat}
> org.apache.flume.flume-ng-sources:flume-jms-source:jar:1.8.0-SNAPSHOT
> \- org.apache.activemq:activemq-core:jar:5.7.0:provided
>+- org.springframework:spring-context:jar:3.0.7.RELEASE:provided
>| +- org.springframework:spring-aop:jar:3.0.7.RELEASE:provided
>| +- org.springframework:spring-beans:jar:3.0.7.RELEASE:provided
>| +- org.springframework:spring-core:jar:3.0.7.RELEASE:provided
>| +- org.springframework:spring-expression:jar:3.0.7.RELEASE:provided
>| \- org.springframework:spring-asm:jar:3.0.7.RELEASE:provided
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (FLUME-3112) Upgrade jackson-core library dependency
[ https://issues.apache.org/jira/browse/FLUME-3112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16140381#comment-16140381 ] Hudson commented on FLUME-3112: --- FAILURE: Integrated in Jenkins build Flume-trunk-hbase-1 #310 (See [https://builds.apache.org/job/Flume-trunk-hbase-1/310/]) FLUME-3112. Upgrade jackson-core library dependency (denes: [http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git=commit=3cd1b331308d8d312fae258d94cd64a69ceb1644]) * (edit) flume-ng-node/pom.xml * (edit) pom.xml > Upgrade jackson-core library dependency > --- > > Key: FLUME-3112 > URL: https://issues.apache.org/jira/browse/FLUME-3112 > Project: Flume > Issue Type: Bug >Affects Versions: 1.7.0 >Reporter: Attila Simon >Assignee: Ferenc Szabo >Priority: Critical > Labels: dependency > Fix For: 1.8.0 > > Attachments: FLUME-3112.patch > > > ||Group||Artifact||Version used||Upgrade target|| > |com.fasterxml.jackson.core|jackson-core|2.3.1|2.8.9| > Security vulnerability: http://www.cvedetails.com/cve/CVE-2016-7051/ > Please do: > - double check the newest version. > - consider to remove a dependency if better alternative is available. > - check whether the lib change would introduce a backward incompatibility (in > which case please add this label `breaking_change` and fix version should be > the next major) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (FLUME-3057) Build fails due to unsupported snappy-java version on ppc64le
[ https://issues.apache.org/jira/browse/FLUME-3057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16140247#comment-16140247 ] Hudson commented on FLUME-3057: --- UNSTABLE: Integrated in Jenkins build Flume-trunk-hbase-1 #309 (See [https://builds.apache.org/job/Flume-trunk-hbase-1/309/]) FLUME-3057. Update snappy-java version (denes: [http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git=commit=a51ca18b77625055051e310a021c38a002ffe89c]) * (edit) pom.xml > Build fails due to unsupported snappy-java version on ppc64le > - > > Key: FLUME-3057 > URL: https://issues.apache.org/jira/browse/FLUME-3057 > Project: Flume > Issue Type: Bug > Components: Build >Affects Versions: 1.8.0 > Environment: $ uname -a > Linux 2f63413ff231 4.4.0-31-generic #50-Ubuntu SMP Wed Jul 13 00:05:18 UTC > 2016 ppc64le ppc64le ppc64le GNU/Linux >Reporter: Pravin Dsilva > Labels: powerpc, ppc64le > Fix For: 1.8.0 > > Attachments: FLUME-3057-1.patch, FLUME-3057.patch > > > Flume has a snappy-java dependency with version 1.1.0. Upon building Flume on > ppc64le architecture, errors such as "[FAILED_TO_LOAD_NATIVE_LIBRARY] no > native library is found for os.name=Linux and os.arch=ppc64le" are seen > Native libraries for ppc64le were added in snappy-java version 1.1.1. Hence > Flume needs to have this higher version of snappy-java. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (FLUME-3152) Add Flume Metric for Backup Checkpoint Errors
[ https://issues.apache.org/jira/browse/FLUME-3152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16138663#comment-16138663 ] Hudson commented on FLUME-3152: --- SUCCESS: Integrated in Jenkins build Flume-trunk-hbase-1 #308 (See [https://builds.apache.org/job/Flume-trunk-hbase-1/308/]) FLUME-3152 Add Flume Metric for Backup Checkpoint Errors (denes: [http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git=commit=4d79aa003aa02e8d513a1ae1406795d758143397]) * (edit) flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/EventQueueBackingStoreFileV3.java * (edit) flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/instrumentation/FileChannelCounter.java * (edit) flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/instrumentation/FileChannelCounterMBean.java * (edit) flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java * (edit) flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestEventQueueBackingStoreFactory.java * (edit) flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannelErrorMetrics.java * (edit) flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/CheckpointRebuilder.java * (edit) flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/EventQueueBackingStoreFile.java * (edit) flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestCheckpointRebuilder.java * (edit) flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/EventQueueBackingStoreFactory.java * (edit) flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestCheckpoint.java * (edit) flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/EventQueueBackingStoreFileV2.java * (edit) flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventQueue.java > Add Flume Metric for Backup Checkpoint Errors > - > > Key: FLUME-3152 > URL: https://issues.apache.org/jira/browse/FLUME-3152 > Project: Flume > Issue Type: Bug > Components: File Channel >Affects Versions: 1.7.0 >Reporter: Ferenc Szabo >Assignee: Ferenc Szabo > Fix For: 1.8.0 > > > File channel metrics should contain a counter for checkpoint backup errors -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (FLUME-3144) Improve Log4jAppender's performance by allowing logging collection of messages
[
https://issues.apache.org/jira/browse/FLUME-3144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16135032#comment-16135032
]
Hudson commented on FLUME-3144:
---
UNSTABLE: Integrated in Jenkins build Flume-trunk-hbase-1 #307 (See
[https://builds.apache.org/job/Flume-trunk-hbase-1/307/])
FLUME-3144. Improve Log4jAppender's performance by allowing logging (denes:
[http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git=commit=66327aa20207ca62cd7c7a4cf9e24724b5ee0dfc])
* (edit)
flume-ng-clients/flume-ng-log4jappender/src/test/java/org/apache/flume/clients/log4jappender/TestLog4jAppenderWithAvro.java
* (edit)
flume-ng-clients/flume-ng-log4jappender/src/main/java/org/apache/flume/clients/log4jappender/Log4jAppender.java
* (edit) flume-ng-clients/flume-ng-log4jappender/pom.xml
* (edit)
flume-ng-clients/flume-ng-log4jappender/src/test/java/org/apache/flume/clients/log4jappender/TestLog4jAppender.java
> Improve Log4jAppender's performance by allowing logging collection of messages
> --
>
> Key: FLUME-3144
> URL: https://issues.apache.org/jira/browse/FLUME-3144
> Project: Flume
> Issue Type: Improvement
>Affects Versions: 1.7.0
>Reporter: Denes Arvay
>Assignee: Denes Arvay
>
> Currently it's only possible to send events one by one with the Log4j
> appender. (Except if the events are wrapped in an avro record but it's quite
> cumbersome and might need special handling on the receiving side)
> As the Log4j methods can handle any {{Object}} I'd suggest to improve the
> Log4j appender to treat {{Collection}} event as a special case and send its
> content to Flume with one {{rpcClient.appendBatch()}} call.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (FLUME-3141) Small typo found in RegexHbaseEventSerializer.java
[ https://issues.apache.org/jira/browse/FLUME-3141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16119919#comment-16119919 ] Hudson commented on FLUME-3141: --- UNSTABLE: Integrated in Jenkins build Flume-trunk-hbase-1 #306 (See [https://builds.apache.org/job/Flume-trunk-hbase-1/306/]) FLUME-3141. Small typo found in RegexHbaseEventSerializer.java (denes: [http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git=commit=1fcef1a3bef16bd0fe67cb7984c67e5fd600b520]) * (edit) flume-ng-sinks/flume-ng-hbase-sink/src/main/java/org/apache/flume/sink/hbase/RegexHbaseEventSerializer.java > Small typo found in RegexHbaseEventSerializer.java > -- > > Key: FLUME-3141 > URL: https://issues.apache.org/jira/browse/FLUME-3141 > Project: Flume > Issue Type: Bug >Affects Versions: 1.7.0 >Reporter: Miklos Csanady >Assignee: Miklos Csanady >Priority: Trivial > Labels: trivial > > Trivial typo found in RegexHbaseEventSerializer.java -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (FLUME-3135) property logger in org.apache.flume.interceptor.RegexFilteringInterceptor confused
[
https://issues.apache.org/jira/browse/FLUME-3135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16102604#comment-16102604
]
Hudson commented on FLUME-3135:
---
ABORTED: Integrated in Jenkins build Flume-trunk-hbase-1 #304 (See
[https://builds.apache.org/job/Flume-trunk-hbase-1/304/])
FLUME-3135. Remove the references to StaticInterceptor from (denes:
[http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git=commit=685b79a8718ea983b451ce88555248510efe84ea])
* (edit)
flume-ng-core/src/main/java/org/apache/flume/interceptor/RegexFilteringInterceptor.java
> property logger in org.apache.flume.interceptor.RegexFilteringInterceptor
> confused
> --
>
> Key: FLUME-3135
> URL: https://issues.apache.org/jira/browse/FLUME-3135
> Project: Flume
> Issue Type: Bug
>Affects Versions: 1.7.0
>Reporter: Peter Chen
>Priority: Minor
> Labels: easyfix
>
> *+org.apache.flume.interceptor.RegexFilteringInterceptor.java+*
> * *line 72-75:*
> the parameter to the getLogger method should be
> RegexFilteringInterceptor.class
> {code:java}
> public class RegexFilteringInterceptor implements Interceptor {
> private static final Logger logger = LoggerFactory
> .getLogger(StaticInterceptor.class);
> {code}
> * *line 141-143:*
> Javadoc
> {code:java}
> /**
>* Builder which builds new instance of the StaticInterceptor.
>*/
> {code}
> :D
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (FLUME-2620) File channel throws NullPointerException if a header value is null
[
https://issues.apache.org/jira/browse/FLUME-2620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16102603#comment-16102603
]
Hudson commented on FLUME-2620:
---
ABORTED: Integrated in Jenkins build Flume-trunk-hbase-1 #304 (See
[https://builds.apache.org/job/Flume-trunk-hbase-1/304/])
FLUME-2620. File Channel to support empty values in headers (denes:
[http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git=commit=1e69fc7c29f104a2117a62de11cba9b2a2c740e1])
* (edit)
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Put.java
* (edit)
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/proto/ProtosFactory.java
* (edit) flume-ng-channels/flume-file-channel/src/main/proto/filechannel.proto
* (edit)
flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java
* (edit)
flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannel.java
> File channel throws NullPointerException if a header value is null
> --
>
> Key: FLUME-2620
> URL: https://issues.apache.org/jira/browse/FLUME-2620
> Project: Flume
> Issue Type: Bug
> Components: File Channel
>Reporter: Santiago M. Mola
>Assignee: Marcell Hegedus
> Attachments: FLUME-2620-0.patch, FLUME-2620-1.patch,
> FLUME-2620-2.patch, FLUME-2620-3.patch, FLUME-2620-4.patch,
> FLUME-2620-5.patch, FLUME-2620.patch, FLUME-2620.patch
>
>
> File channel throws NullPointerException if a header value is null.
> If this is intended, it should be reported correctly in the logs.
> Sample trace:
> org.apache.flume.ChannelException: Unable to put batch on required channel:
> FileChannel chan { dataDirs: [/var/lib/ingestion-csv/chan/data] }
> at
> org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:200)
> at
> org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(SpoolDirectorySource.java:236)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.NullPointerException
> at
> org.apache.flume.channel.file.proto.ProtosFactory$FlumeEventHeader$Builder.setValue(ProtosFactory.java:7415)
> at org.apache.flume.channel.file.Put.writeProtos(Put.java:85)
> at
> org.apache.flume.channel.file.TransactionEventRecord.toByteBuffer(TransactionEventRecord.java:174)
> at org.apache.flume.channel.file.Log.put(Log.java:622)
> at
> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:469)
> at
> org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)
> at
> org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)
> at
> org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:189)
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (FLUME-2620) File channel throws NullPointerException if a header value is null
[
https://issues.apache.org/jira/browse/FLUME-2620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16100649#comment-16100649
]
Hudson commented on FLUME-2620:
---
FAILURE: Integrated in Jenkins build Flume-trunk-hbase-1 #261 (See
[https://builds.apache.org/job/Flume-trunk-hbase-1/261/])
FLUME-2620. File Channel to support empty values in headers (denes:
[http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git=commit=1e69fc7c29f104a2117a62de11cba9b2a2c740e1])
* (edit)
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Put.java
* (edit) flume-ng-channels/flume-file-channel/src/main/proto/filechannel.proto
* (edit)
flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannel.java
* (edit)
flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java
* (edit)
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/proto/ProtosFactory.java
> File channel throws NullPointerException if a header value is null
> --
>
> Key: FLUME-2620
> URL: https://issues.apache.org/jira/browse/FLUME-2620
> Project: Flume
> Issue Type: Bug
> Components: File Channel
>Reporter: Santiago M. Mola
>Assignee: Marcell Hegedus
> Attachments: FLUME-2620-0.patch, FLUME-2620-1.patch,
> FLUME-2620-2.patch, FLUME-2620-3.patch, FLUME-2620-4.patch,
> FLUME-2620-5.patch, FLUME-2620.patch, FLUME-2620.patch
>
>
> File channel throws NullPointerException if a header value is null.
> If this is intended, it should be reported correctly in the logs.
> Sample trace:
> org.apache.flume.ChannelException: Unable to put batch on required channel:
> FileChannel chan { dataDirs: [/var/lib/ingestion-csv/chan/data] }
> at
> org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:200)
> at
> org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(SpoolDirectorySource.java:236)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.NullPointerException
> at
> org.apache.flume.channel.file.proto.ProtosFactory$FlumeEventHeader$Builder.setValue(ProtosFactory.java:7415)
> at org.apache.flume.channel.file.Put.writeProtos(Put.java:85)
> at
> org.apache.flume.channel.file.TransactionEventRecord.toByteBuffer(TransactionEventRecord.java:174)
> at org.apache.flume.channel.file.Log.put(Log.java:622)
> at
> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:469)
> at
> org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)
> at
> org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)
> at
> org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:189)
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (FLUME-2175) Update Developer Guide with notes on how to upgrade Protocol Buffer version
[ https://issues.apache.org/jira/browse/FLUME-2175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16073466#comment-16073466 ] Hudson commented on FLUME-2175: --- UNSTABLE: Integrated in Jenkins build Flume-trunk-hbase-1 #258 (See [https://builds.apache.org/job/Flume-trunk-hbase-1/258/]) FLUME-2175. Document Protocol Buffer upgrade steps (denes: [http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git=commit=b54f085ffa89b74c9e91e478e864424bfbe0a7a2]) * (edit) flume-ng-doc/sphinx/FlumeDeveloperGuide.rst > Update Developer Guide with notes on how to upgrade Protocol Buffer version > --- > > Key: FLUME-2175 > URL: https://issues.apache.org/jira/browse/FLUME-2175 > Project: Flume > Issue Type: Documentation > Components: Docs >Affects Versions: 1.4.0 >Reporter: Roshan Naik >Assignee: Roshan Naik > Labels: build, doc > Fix For: 1.8.0 > > Attachments: FLUME-2175.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (FLUME-2917) Provide netcat UDP source as alternative to TCP
[ https://issues.apache.org/jira/browse/FLUME-2917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16071798#comment-16071798 ] Hudson commented on FLUME-2917: --- FAILURE: Integrated in Jenkins build Flume-trunk-hbase-1 #255 (See [https://builds.apache.org/job/Flume-trunk-hbase-1/255/]) FLUME-2917. Provide netcat UDP source as alternative to TCP (bessbd: [http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git=commit=ef54f9fd6372143adc8c6a66f174feba381fec95]) * (add) flume-ng-core/src/test/java/org/apache/flume/source/TestNetcatUdpSource.java * (edit) flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceConfiguration.java * (edit) flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceType.java * (edit) flume-ng-core/src/test/java/org/apache/flume/source/TestDefaultSourceFactory.java * (add) flume-ng-core/src/main/java/org/apache/flume/source/NetcatUdpSource.java * (edit) flume-ng-doc/sphinx/FlumeUserGuide.rst > Provide netcat UDP source as alternative to TCP > --- > > Key: FLUME-2917 > URL: https://issues.apache.org/jira/browse/FLUME-2917 > Project: Flume > Issue Type: New Feature > Components: Sinks+Sources >Affects Versions: 1.6.0 >Reporter: Tristan Stevens >Assignee: Tristan Stevens >Priority: Minor > Fix For: 1.8.0 > > Attachments: FLUME-2917-7.patch, FLUME-2917.patch > > > Currently Flume provides a Netcat TCP source, however Netcat is often used > with UDP. There is an implementation of a UDP client in the SyslogUDP source, > this request takes this implementation and strips out the Syslog parts, thus > forming a Netcat UDP source - where each datagram is recorded as a Flume > event. > The implementation is provided for this at > https://github.com/tmgstevens/FlumeNetcatUDPSource and also provided as an > attached patch for inclusion. > N.B. Unit tests are provided for this. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (FLUME-2905) NetcatSource - Socket not closed when an exception is encountered during start() leading to file descriptor leaks
[
https://issues.apache.org/jira/browse/FLUME-2905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16070036#comment-16070036
]
Hudson commented on FLUME-2905:
---
FAILURE: Integrated in Jenkins build Flume-trunk-hbase-1 #254 (See
[https://builds.apache.org/job/Flume-trunk-hbase-1/254/])
FLUME-2905. Fix NetcatSource file descriptor leak if startup fails (denes:
[http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git=commit=5e9cfef2b26f1960601d08d571e4c85c269503af])
* (edit)
flume-ng-core/src/test/java/org/apache/flume/source/TestNetcatSource.java
* (edit) flume-ng-core/src/main/java/org/apache/flume/source/NetcatSource.java
> NetcatSource - Socket not closed when an exception is encountered during
> start() leading to file descriptor leaks
> -
>
> Key: FLUME-2905
> URL: https://issues.apache.org/jira/browse/FLUME-2905
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
>Affects Versions: 1.6.0
>Reporter: Siddharth Ahuja
>Assignee: Siddharth Ahuja
> Attachments: FLUME-2905-0.patch, FLUME-2905-1.patch,
> FLUME-2905-2.patch, FLUME-2905-3.patch, FLUME-2905-4.patch,
> FLUME-2905-5.patch, FLUME-2905-6.patch
>
>
> During the flume agent start-up, the flume configuration containing the
> NetcatSource is parsed and the source's start() is called. If there is an
> issue while binding the channel's socket to a local address to configure the
> socket to listen for connections following exception is thrown but the socket
> open just before is not closed.
> {code}
> 2016-05-01 03:04:37,273 ERROR org.apache.flume.lifecycle.LifecycleSupervisor:
> Unable to start EventDrivenSourceRunner: {
> source:org.apache.flume.source.NetcatSource{name:src-1,state:IDLE} } -
> Exception follows.
> org.apache.flume.FlumeException: java.net.BindException: Address already in
> use
> at org.apache.flume.source.NetcatSource.start(NetcatSource.java:173)
> at
> org.apache.flume.source.EventDrivenSourceRunner.start(EventDrivenSourceRunner.java:44)
> at
> org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.net.BindException: Address already in use
> at sun.nio.ch.Net.bind0(Native Method)
> at sun.nio.ch.Net.bind(Net.java:444)
> at sun.nio.ch.Net.bind(Net.java:436)
> at
> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)
> at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
> at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:67)
> at org.apache.flume.source.NetcatSource.start(NetcatSource.java:167)
> ... 9 more
> {code}
> The source's start() is then called again leading to another socket being
> opened but not closed and so on. This leads to file descriptor (socket) leaks.
> This can be easily reproduced as follows:
> 1. Set Netcat as the source in flume agent configuration.
> 2. Set the bind port for the netcat source to a port which is already in use.
> e.g. in my case I used 50010 which is the port for DataNode's XCeiver
> Protocol in use by the HDFS service.
> 3. Start flume agent and perform "lsof -p | wc -l". Notice
> the file descriptors keep on growing due to socket leaks with errors like:
> "can't identify protocol".
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (FLUME-2752) Flume AvroSource will leak the memory and the OOM will be happened.
[
https://issues.apache.org/jira/browse/FLUME-2752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16069985#comment-16069985
]
Hudson commented on FLUME-2752:
---
FAILURE: Integrated in Jenkins build Flume-trunk-hbase-1 #253 (See
[https://builds.apache.org/job/Flume-trunk-hbase-1/253/])
FLUME-2752. Fix AvroSource startup resource leaks (denes:
[http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git=commit=b5e5ba50f4333272b9e2f2be2b32027e667f32e2])
* (edit) flume-ng-core/src/test/java/org/apache/flume/source/TestAvroSource.java
* (edit) flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java
> Flume AvroSource will leak the memory and the OOM will be happened.
> ---
>
> Key: FLUME-2752
> URL: https://issues.apache.org/jira/browse/FLUME-2752
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
>Affects Versions: 1.6.0
>Reporter: yinghua_zh
>Assignee: Attila Simon
>
> If the flume agent config the nonexist IP for the avro source,the exception
> will be happened as follow:
> 2015-07-21 19:57:47,054 | ERROR | [lifecycleSupervisor-1-2] | Unable to
> start EventDrivenSourceRunner: { source:Avro source avro_source_21155: {
> bindAddress: 51.196.27.32, port: 21155 } } - Exception follows. |
> org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:253)
> org.jboss.netty.channel.ChannelException: Failed to bind to:
> /51.196.27.32:21155
> at
> org.jboss.netty.bootstrap.ServerBootstrap.bind(ServerBootstrap.java:297)
> at org.apache.avro.ipc.NettyServer.(NettyServer.java:106)
> at org.apache.flume.source.AvroSource.start(AvroSource.java:294)
> at
> org.apache.flume.source.EventDrivenSourceRunner.start(EventDrivenSourceRunner.java:44)
> at
> org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.net.BindException: Cannot assign requested address
> at sun.nio.ch.Net.bind0(Native Method)
> at sun.nio.ch.Net.bind(Net.java:437)
> at sun.nio.ch.Net.bind(Net.java:429)
> at
> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
> at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
> at
> org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.bind(NioServerSocketPipelineSink.java:140)
> at
> org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.handleServerSocket(NioServerSocketPipelineSink.java:90)
> at
> org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.eventSunk(NioServerSocketPipelineSink.java:64)
> at org.jboss.netty.channel.Channels.bind(Channels.java:569)
> at
> org.jboss.netty.channel.AbstractChannel.bind(AbstractChannel.java:189)
> at
> org.jboss.netty.bootstrap.ServerBootstrap$Binder.channelOpen(ServerBootstrap.java:342)
> at org.jboss.netty.channel.Channels.fireChannelOpen(Channels.java:170)
> at
> org.jboss.netty.channel.socket.nio.NioServerSocketChannel.(NioServerSocketChannel.java:80)
> at
> org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.newChannel(NioServerSocketChannelFactory.java:158)
> at
> org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.newChannel(NioServerSocketChannelFactory.java:86)
> at
> org.jboss.netty.bootstrap.ServerBootstrap.bind(ServerBootstrap.java:276)
> if the above exception happened for 2 hours,and the agent JVM -Xxx is 4G,the
> OutOfMemory will be happened.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (FLUME-2579) JMS source support durable subscriptions and message listening
[ https://issues.apache.org/jira/browse/FLUME-2579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16057625#comment-16057625 ] Hudson commented on FLUME-2579: --- FAILURE: Integrated in Jenkins build Flume-trunk-hbase-1 #251 (See [https://builds.apache.org/job/Flume-trunk-hbase-1/251/]) FLUME-2579 Support durable subscription in JMSSource (denes: [http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git=commit=857df3fe9b211a0e34f17cc6d776a1296951134a]) * (edit) flume-ng-sources/flume-jms-source/src/test/java/org/apache/flume/source/jms/JMSMessageConsumerTestBase.java * (delete) flume-ng-sources/flume-jms-source/src/main/java/org/apache/flume/source/jms/JMSMessageConsumerFactory.java * (edit) flume-ng-sources/flume-jms-source/src/test/java/org/apache/flume/source/jms/TestIntegrationActiveMQ.java * (edit) flume-ng-sources/flume-jms-source/src/test/java/org/apache/flume/source/jms/TestJMSMessageConsumer.java * (edit) flume-ng-sources/flume-jms-source/src/main/java/org/apache/flume/source/jms/JMSMessageConsumer.java * (edit) flume-ng-sources/flume-jms-source/src/test/java/org/apache/flume/source/jms/TestJMSSource.java * (edit) flume-ng-doc/sphinx/FlumeUserGuide.rst * (edit) flume-ng-sources/flume-jms-source/src/main/java/org/apache/flume/source/jms/JMSSource.java * (edit) flume-ng-sources/flume-jms-source/src/main/java/org/apache/flume/source/jms/JMSSourceConfiguration.java > JMS source support durable subscriptions and message listening > -- > > Key: FLUME-2579 > URL: https://issues.apache.org/jira/browse/FLUME-2579 > Project: Flume > Issue Type: Wish > Components: Sinks+Sources >Affects Versions: 1.5.2 >Reporter: zhangshengbing >Assignee: Andras Beni > Fix For: 1.8.0 > > > best wishes -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (FLUME-3072) Add IP address to headers in flume log4j appender
[ https://issues.apache.org/jira/browse/FLUME-3072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16057624#comment-16057624 ] Hudson commented on FLUME-3072: --- FAILURE: Integrated in Jenkins build Flume-trunk-hbase-1 #251 (See [https://builds.apache.org/job/Flume-trunk-hbase-1/251/]) FLUME-3072. Add IP address to headers in Flume Log4j appender (denes: [http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git=commit=f2996cca55bbb852dc10512ca3369ae7fd56c44f]) * (edit) flume-ng-clients/flume-ng-log4jappender/src/test/java/org/apache/flume/clients/log4jappender/TestLoadBalancingLog4jAppender.java * (edit) flume-ng-clients/flume-ng-log4jappender/src/main/java/org/apache/flume/clients/log4jappender/LoadBalancingLog4jAppender.java * (edit) flume-ng-clients/flume-ng-log4jappender/src/main/java/org/apache/flume/clients/log4jappender/Log4jAppender.java * (edit) flume-ng-clients/flume-ng-log4jappender/src/main/java/org/apache/flume/clients/log4jappender/Log4jAvroHeaders.java * (edit) flume-ng-clients/flume-ng-log4jappender/src/test/java/org/apache/flume/clients/log4jappender/TestLog4jAppender.java > Add IP address to headers in flume log4j appender > - > > Key: FLUME-3072 > URL: https://issues.apache.org/jira/browse/FLUME-3072 > Project: Flume > Issue Type: Improvement > Components: Sinks+Sources >Affects Versions: 1.8.0 >Reporter: Andras Beni >Assignee: Andras Beni > Fix For: 1.8.0 > > > Add client's host address to log event headers in Flume NG Log4j Appender. > By receiving such events through AvroSource, one can route events based on > origin host in the receiving Flume agent. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (FLUME-2945) Bump java target version to 1.8
[ https://issues.apache.org/jira/browse/FLUME-2945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16055754#comment-16055754 ] Hudson commented on FLUME-2945: --- FAILURE: Integrated in Jenkins build Flume-trunk-hbase-1 #250 (See [https://builds.apache.org/job/Flume-trunk-hbase-1/250/]) FLUME-2945. Bump java target version to 1.8 (denes: [http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git=commit=33d19046f0db66c3093f68716e4f6100301d3b49]) * (edit) flume-ng-doc/sphinx/FlumeUserGuide.rst * (edit) README.md * (edit) DEVNOTES * (edit) conf/flume-env.sh.template * (edit) flume-ng-sources/flume-taildir-source/pom.xml * (edit) pom.xml * (edit) .travis.yml > Bump java target version to 1.8 > --- > > Key: FLUME-2945 > URL: https://issues.apache.org/jira/browse/FLUME-2945 > Project: Flume > Issue Type: Improvement >Affects Versions: 1.8.0 >Reporter: Lior Zeno >Assignee: Lior Zeno > Fix For: 1.8.0 > > Attachments: FLUME-2945-0.patch > > > We should move to Java 8 as a minimum requirement. > # Java 7 is EOL'ed http://www.oracle.com/technetwork/java/eol-135779.html. > # Many projects are adopting java 8 as a minimum requirement, for instance: > * Solr 6: https://issues.apache.org/jira/browse/LUCENE-6722 > * Hbase 2: https://issues.apache.org/jira/browse/HBASE-15624 > * elasticsearch 5: > https://www.elastic.co/guide/en/elasticsearch/reference/master/setup.html > The attached patch was tested on Ubuntu 16.04 with openjdk version "1.8.0_91". > All unit tests, except for the known flaky ones, successfully passed. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (FLUME-3100) Support arbitrary header substitution for topic of Kafka
[
https://issues.apache.org/jira/browse/FLUME-3100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16047736#comment-16047736
]
Hudson commented on FLUME-3100:
---
SUCCESS: Integrated in Jenkins build Flume-trunk-hbase-1 #249 (See
[https://builds.apache.org/job/Flume-trunk-hbase-1/249/])
FLUME-3100. Support arbitrary header substitution for topic of Kafka (denes:
[http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git=commit=83e25691dc5f32d020b122d679b6f124162e4aef])
* (edit)
flume-ng-sinks/flume-ng-kafka-sink/src/main/java/org/apache/flume/sink/kafka/KafkaSink.java
* (edit)
flume-ng-sinks/flume-ng-kafka-sink/src/test/java/org/apache/flume/sink/kafka/TestConstants.java
* (edit) flume-ng-doc/sphinx/FlumeUserGuide.rst
* (edit)
flume-ng-sinks/flume-ng-kafka-sink/src/test/java/org/apache/flume/sink/kafka/TestKafkaSink.java
> Support arbitrary header substitution for topic of Kafka
>
>
> Key: FLUME-3100
> URL: https://issues.apache.org/jira/browse/FLUME-3100
> Project: Flume
> Issue Type: Improvement
> Components: Sinks+Sources
>Reporter: Takafumi Saito
>Priority: Minor
> Fix For: 1.8.0
>
>
> Kafka Sink supports arbitrary header substitution which replace a substring
> ${A} in "kafka.topic" property with a value of header named “A”.
> Using the substitution become possible to dynamically select the topic by
> combining one or more values of event headers.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (FLUME-3092) Extend the FileChannel's monitoring metrics
[
https://issues.apache.org/jira/browse/FLUME-3092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16013092#comment-16013092
]
Hudson commented on FLUME-3092:
---
UNSTABLE: Integrated in Jenkins build Flume-trunk-hbase-1 #248 (See
[https://builds.apache.org/job/Flume-trunk-hbase-1/248/])
FLUME-3092. Extend the FileChannel's monitoring metrics (mpercy:
[http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git=commit=fdc53f338931b96addf06f3f2be59da128656ec0])
* (edit)
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java
* (edit)
flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannelBase.java
* (edit)
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java
* (edit)
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/instrumentation/FileChannelCounterMBean.java
* (add)
flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannelErrorMetrics.java
* (edit)
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/instrumentation/FileChannelCounter.java
* (edit)
flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLog.java
> Extend the FileChannel's monitoring metrics
> ---
>
> Key: FLUME-3092
> URL: https://issues.apache.org/jira/browse/FLUME-3092
> Project: Flume
> Issue Type: Improvement
> Components: File Channel
>Affects Versions: 1.7.0
>Reporter: Denes Arvay
>Assignee: Denes Arvay
> Fix For: 1.8.0
>
>
> There are already several generic metrics (e.g. {{eventPutAttemptCount}} and
> {{eventPutSuccessCount}}) which can be used to create compound metrics for
> monitoring the FileChannel's health.
> Some monitoring system's aren't capable to calculate such derived metrics,
> though, so I recommend to add the following extra counters to represent if a
> channel operation failed or the channel is in an unhealthy state.
> - {{eventPutErrorCount}}: incremented if an {{IOException}} occurs during
> {{put}} operation.
> - {{eventTakeErrorCount}}: incremented if an {{IOException}} or
> {{CorruptEventException}} occurs during {{take}} operation.
> - {{checkpointWriteErrorCount}}: incremented if an exception occurs during
> checkpoint write.
> - {{unhealthy}}: this flag represents whether the channel has started
> successfully (i.e. the replay ran without any problem). This is similar to
> the already existing {{open}} flag except that the latter is initially false
> and is set to {{true}} if the initialization (including the log replay) is
> successfully done. The {{unhealthy}}, in contrary, is {{false}} by default
> and is set to {{true}} if there is an error during startup.
> Beside these flags I'd also introduce a {{closed}} flag which is the numeric
> representation (1: closed, 0: open) of the negated (already existing)
> {{open}} flag.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (FLUME-3085) HDFS Sink can skip flushing some BucketWriters, might lead to data loss
[
https://issues.apache.org/jira/browse/FLUME-3085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16001644#comment-16001644
]
Hudson commented on FLUME-3085:
---
UNSTABLE: Integrated in Jenkins build Flume-trunk-hbase-1 #247 (See
[https://builds.apache.org/job/Flume-trunk-hbase-1/247/])
FLUME-3085. HDFS Sink can skip flushing some BucketWriters, might lead (mpercy:
[http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git=commit=ed433ae1b12d40117ca3dca2ffd57389984ede72])
* (edit)
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java
* (edit)
flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java
> HDFS Sink can skip flushing some BucketWriters, might lead to data loss
> ---
>
> Key: FLUME-3085
> URL: https://issues.apache.org/jira/browse/FLUME-3085
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
>Affects Versions: 1.7.0
>Reporter: Denes Arvay
>Assignee: Denes Arvay
>Priority: Critical
> Fix For: 1.8.0
>
>
> The {{HDFSEventSink.process()}} is already prepared for a rare race
> condition, namely when the BucketWriter acquired in [line
> 389|https://github.com/apache/flume/blob/trunk/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java#L389]
> gets closed by an other thread (e.g. because the {{idleTimeout}} or the
> {{rollInterval}}) before the {{append()}} is called in [line
> 406|https://github.com/apache/flume/blob/trunk/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java#L406].
> If this is the case the {{BucketWriter.append()}} call throws a
> {{BucketClosedException}} and the sink creates a new {{BucketWriter}}
> instance and appends to it.
> But this newly created instance won't be added to the {{writers}} list, which
> means that it won't be flushed after the processing loop finished:
> https://github.com/apache/flume/blob/trunk/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java#L429
> This has multiple consequences:
> - unflushed data might get lost
> - the {{BucketWriter}}'s {{idleAction}} won't be scheduled
> (https://github.com/apache/flume/blob/trunk/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java#L450),
> which means that it won't be closed nor renamed if the idle timeout is the
> only trigger for closing the file.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (FLUME-3083) Taildir source can miss events if file updated in same second as file close
[
https://issues.apache.org/jira/browse/FLUME-3083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15987636#comment-15987636
]
Hudson commented on FLUME-3083:
---
FAILURE: Integrated in Jenkins build Flume-trunk-hbase-1 #246 (See
[https://builds.apache.org/job/Flume-trunk-hbase-1/246/])
FLUME-3083. Check byte position of file in update condition of Taildir (bessbd:
[http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git=commit=dfa0627573b9a75a25dc7149a7d63c9bac953ff4])
* (edit)
flume-ng-sources/flume-taildir-source/src/main/java/org/apache/flume/source/taildir/ReliableTaildirEventReader.java
* (edit)
flume-ng-sources/flume-taildir-source/src/test/java/org/apache/flume/source/taildir/TestTaildirEventReader.java
> Taildir source can miss events if file updated in same second as file close
> ---
>
> Key: FLUME-3083
> URL: https://issues.apache.org/jira/browse/FLUME-3083
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
>Affects Versions: 1.7.0
>Reporter: eskrm
>Assignee: eskrm
> Fix For: 1.8.0
>
> Attachments: FLUME-3083-0.patch, FLUME-3083-1.patch
>
>
> I suspect there is a scenario where the taildir source can miss reading
> events from a log file due to how the source determines whether a file has
> been updated. In ReliableTaildirEventReader:
> {code}
> boolean updated = tf.getLastUpdated() < f.lastModified()
> ...
> tf.setNeedTail(updated);
> {code}
> Consider this sequence of events from TaildirSource.process(). Assume they
> all happen within the same second and there is just a single log file.
> # Call ReliableTaildirEventReader.updateTailFiles()
> #* This call will set ReliableTaildirEventReader.updateTime to current time
> in milliseconds
> #* Assume the underlying file has not been updated within the last
> idleTimeout milliseconds
> # Due to idleness, the tail file's inode is added to TaildirSource.idleInodes
> in idleFileCheckerRunnable
> # tf.needTail is false. Skip reading file.
> # Underlying file is updated with events E1
> # TaildirSource.closeTailFiles()
> #* Call TaildirSource.tailFileProcess() before close to read any pending
> events
> #* Events E1 are read and processed
> #* Since events were read, call ReliableTaildirEventReader.commit() which
> updates the tail file's position and sets its last updated time to
> ReliableTaildirEventReader.updateTime from 1.a
> #* Close file
> # Events E2 are written to underlying file. File's modification time is in
> the same second as the tail file's last updated time.
> # Since the time returned by File.lastModified() is the mtime in seconds
> converted to milliseconds the file's last modified time is less than the tail
> file's last updated time and taildir won't reopen the file to read E2.
> #* This behaviour of File.lastModified() may be platform/jvm specific. I
> confirmed the behavior using OpenJDK 8 on Ubuntu precise.
> Can someone confirm this?
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (FLUME-3043) KafkaSink SinkCallback throws NullPointerException when Log4J level is debug
[
https://issues.apache.org/jira/browse/FLUME-3043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15970538#comment-15970538
]
Hudson commented on FLUME-3043:
---
SUCCESS: Integrated in Jenkins build Flume-trunk-hbase-1 #245 (See
[https://builds.apache.org/job/Flume-trunk-hbase-1/245/])
FLUME-3043. Fix NPE in Kafka Sink and Channel (bessbd:
[http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git=commit=c718dae09d10db640cb9eb59f8abb11bd385a799])
* (edit)
flume-ng-channels/flume-kafka-channel/src/main/java/org/apache/flume/channel/kafka/KafkaChannel.java
* (edit)
flume-ng-sinks/flume-ng-kafka-sink/src/main/java/org/apache/flume/sink/kafka/KafkaSink.java
> KafkaSink SinkCallback throws NullPointerException when Log4J level is debug
>
>
> Key: FLUME-3043
> URL: https://issues.apache.org/jira/browse/FLUME-3043
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
>Affects Versions: 1.7.0
>Reporter: dengkai
>Assignee: dengkai
> Fix For: 1.8.0
>
> Attachments: FLUME-3043.patch
>
>
> When we send a event with SinkCallback to kafka under DEBUG level of log4j,
> if kafka reponses a result with exception but not a RecordMetadata. Kafka
> Sink will throw NPE.
> code in SinkCallback:
> {code}
> if (logger.isDebugEnabled()) {
> long eventElapsedTime = System.currentTimeMillis() - startTime;
> logger.debug("Acked message partition:{} ofset:{}", metadata.partition(),
> metadata.offset());
> logger.debug("Elapsed time for send: {}", eventElapsedTime);
> }
> {code}
> code in Kafka Producer:
> {code}
> if (exception == null) {
> RecordMetadata metadata = new RecordMetadata(...);
> thunk.callback.onCompletion(metadata, null);
> } else {
> thunk.callback.onCompletion(null, exception);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (FLUME-2817) Sink for multi-agent flow example in user guide is set up incorrectly
[
https://issues.apache.org/jira/browse/FLUME-2817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15962154#comment-15962154
]
Hudson commented on FLUME-2817:
---
UNSTABLE: Integrated in Jenkins build Flume-trunk-hbase-1 #244 (See
[https://builds.apache.org/job/Flume-trunk-hbase-1/244/])
FLUME-2817. docs: Fix multi-agent flow example (bessbd:
[http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git=commit=03c8357dfeff9615d1590a5c980eb4cfe035f3be])
* (edit) flume-ng-doc/sphinx/FlumeUserGuide.rst
> Sink for multi-agent flow example in user guide is set up incorrectly
> -
>
> Key: FLUME-2817
> URL: https://issues.apache.org/jira/browse/FLUME-2817
> Project: Flume
> Issue Type: Documentation
> Components: Docs
>Affects Versions: 1.6.0
>Reporter: Sam Thompson
>Assignee: Marcell Hegedus
>Priority: Trivial
> Labels: documentation, easyfix
> Fix For: 1.8.0
>
>
> The [multi-agent flow
> example|https://flume.apache.org/FlumeUserGuide.html#configuring-a-multi-agent-flow]
> tries to set up an avro sink like this:
> {noformat}
> # avro sink properties
> agent_foo.sources.avro-forward-sink.type = avro
> agent_foo.sources.avro-forward-sink.hostname = 10.1.1.100
> agent_foo.sources.avro-forward-sink.port = 1
> {noformat}
> But shouldn't it be like this?
> {noformat}
> # avro sink properties
> agent_foo.sinks.avro-forward-sink.type = avro
> agent_foo.sinks.avro-forward-sink.hostname = 10.1.1.100
> agent_foo.sinks.avro-forward-sink.port = 1
> {noformat}
> I tried the original way just to make sure, but it doesn't seem to work.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (FLUME-3080) Close failure in HDFS Sink might cause data loss
[
https://issues.apache.org/jira/browse/FLUME-3080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15962102#comment-15962102
]
Hudson commented on FLUME-3080:
---
UNSTABLE: Integrated in Jenkins build Flume-trunk-hbase-1 #243 (See
[https://builds.apache.org/job/Flume-trunk-hbase-1/243/])
FLUME-3080. Close failure in HDFS Sink might cause data loss (bessbd:
[http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git=commit=e5c3e6aa76cf2b0bb0838ff6dcd3853656bff704])
* (edit)
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java
* (edit)
flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSinkOnMiniCluster.java
> Close failure in HDFS Sink might cause data loss
>
>
> Key: FLUME-3080
> URL: https://issues.apache.org/jira/browse/FLUME-3080
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
>Affects Versions: 1.7.0
>Reporter: Denes Arvay
>Assignee: Denes Arvay
>Priority: Blocker
> Fix For: 1.8.0
>
>
> If the HDFS Sink tries to close a file but it fails (e.g. due to timeout) the
> last block might not end up in COMPLETE state. In this case block recovery
> should happen but as the lease is still held by Flume the NameNode will start
> the recovery process only after the hard limit of 1 hour expires.
> The lease recovery can be started manually by the {{hdfs debug recoverLease}}
> command.
> For reproduction I removed the close call from the {{BucketWriter}}
> (https://github.com/apache/flume/blob/trunk/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java#L380)
> to simulate the failure and used the following config:
> {noformat}
> agent.sources = testsrc
> agent.sinks = testsink
> agent.channels = testch
> agent.sources.testsrc.type = netcat
> agent.sources.testsrc.bind = localhost
> agent.sources.testsrc.port = 9494
> agent.sources.testsrc.channels = testch
> agent.sinks.testsink.type = hdfs
> agent.sinks.testsink.hdfs.path = /user/flume/test
> agent.sinks.testsink.hdfs.rollInterval = 0
> agent.sinks.testsink.hdfs.rollCount = 3
> agent.sinks.testsink.serializer = avro_event
> agent.sinks.testsink.serializer.compressionCodec = snappy
> agent.sinks.testsink.hdfs.fileSuffix = .avro
> agent.sinks.testsink.hdfs.fileType = DataStream
> agent.sinks.testsink.hdfs.batchSize = 2
> agent.sinks.testsink.hdfs.writeFormat = Text
> agent.sinks.testsink.hdfs.idleTimeout=20
> agent.sinks.testsink.channel = testch
> agent.channels.testch.type = memory
> {noformat}
> After ingesting 6 events ("a" - "f") 2 files were created on HDFS, as
> expected. But there are missing events when listing the contents in Spark
> shell:
> {noformat}
> scala>
> sqlContext.read.avro("/user/flume/test/FlumeData.14908867127*.avro").collect().map(a
> => new String(a(1).asInstanceOf[Array[Byte]])).foreach(println)
> a
> b
> d
> {noformat}
> {{hdfs fsck}} also confirms that the blocks are still in
> {{UNDER_CONSTRUCTION}} state:
> {noformat}
> $ hdfs fsck /user/flume/test/ -openforwrite -files -blocks
> FSCK started by root (auth:SIMPLE) from /172.31.114.3 for path
> /user/flume/test/ at Thu Mar 30 08:23:56 PDT 2017
> /user/flume/test/
> /user/flume/test/FlumeData.1490887185312.avro 310 bytes, 1 block(s),
> OPENFORWRITE: MISSING 1 blocks of total size 310 B
> 0.
> BP-1285398861-172.31.114.3-1489845696835:blk_1073761923_21128{blockUCState=UNDER_CONSTRUCTION,
> primaryNodeIndex=-1,
> replicas=[ReplicaUnderConstruction[[DISK]DS-e0d04bef-a861-40b0-99dd-27bfb2871ecd:NORMAL:172.31.114.27:20002|RBW],
>
> ReplicaUnderConstruction[[DISK]DS-d1979e0c-db81-4790-b225-ae8a4cf42dd8:NORMAL:172.31.114.32:20002|RBW],
>
> ReplicaUnderConstruction[[DISK]DS-ca00550d-702e-4892-a54a-7105af0c19ee:NORMAL:172.31.114.24:20002|RBW]]}
> len=310 MISSING!
> /user/flume/test/FlumeData.1490887185313.avro 292 bytes, 1 block(s),
> OPENFORWRITE: MISSING 1 blocks of total size 292 B
> 0.
> BP-1285398861-172.31.114.3-1489845696835:blk_1073761924_21129{blockUCState=UNDER_CONSTRUCTION,
> primaryNodeIndex=-1,
> replicas=[ReplicaUnderConstruction[[DISK]DS-ca00550d-702e-4892-a54a-7105af0c19ee:NORMAL:172.31.114.24:20002|RBW],
>
> ReplicaUnderConstruction[[DISK]DS-e0d04bef-a861-40b0-99dd-27bfb2871ecd:NORMAL:172.31.114.27:20002|RBW],
>
> ReplicaUnderConstruction[[DISK]DS-d1979e0c-db81-4790-b225-ae8a4cf42dd8:NORMAL:172.31.114.32:20002|RBW]]}
> len=292 MISSING!
> {noformat}
> These blocks need to be recovered by starting a lease recovery process on the
> NameNode (which will then run the block recovery as well). This can be
> triggered programmatically via the DFSClient.
> Adding this call if the close fails solves the issue.
> cc [~jojochuang]
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (FLUME-2427) java.lang.NoSuchMethodException and warning on HDFS (S3) sink
[ https://issues.apache.org/jira/browse/FLUME-2427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15871049#comment-15871049 ] Hudson commented on FLUME-2427: --- UNSTABLE: Integrated in Jenkins build Flume-trunk-hbase-1 #240 (See [https://builds.apache.org/job/Flume-trunk-hbase-1/240/]) FLUME-2427. Be less verbose in isFileClosed() notification (mpercy: [http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git=commit=630083228364f6cd5dc18839fbd4b167cb615b38]) * (edit) flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java > java.lang.NoSuchMethodException and warning on HDFS (S3) sink > -- > > Key: FLUME-2427 > URL: https://issues.apache.org/jira/browse/FLUME-2427 > Project: Flume > Issue Type: Question >Reporter: Bijith Kumar >Assignee: Ping Wang >Priority: Minor > Fix For: v1.8.0 > > Attachments: FLUME-2427-0.patch > > > The below warning and Exception is thrown every time a file is written to S3 > using HDFS sink. Looks like a jar mismatch to me. Tried latest hadoop and > jets3 jars but didn't work > 17 Jul 2014 23:30:18,159 INFO [hdfs-s3sink-engagements-call-runner-6] > (org.apache.flume.sink.hdfs.AbstractHDFSWriter.reflectGetNumCurrentReplicas:184) > - FileSystem's output stream doesn't support getNumCurrentReplicas; > --HDFS-826 not available; > fsOut=org.apache.hadoop.fs.s3native.NativeS3FileSystem$NativeS3FsOutputStream; > err=java.lang.NoSuchMethodException: > org.apache.hadoop.fs.s3native.NativeS3FileSystem$NativeS3FsOutputStream.getNumCurrentReplicas() > 17 Jul 2014 23:30:18,160 WARN > [SinkRunner-PollingRunner-DefaultSinkProcessor] > (org.apache.flume.sink.hdfs.BucketWriter.getRefIsClosed:210) - isFileClosed > is not available in the version of HDFS being used. Flume will not attempt to > close files if the close fails on the first attempt > java.lang.NoSuchMethodException: > org.apache.hadoop.fs.s3native.NativeS3FileSystem.isFileClosed(org.apache.hadoop.fs.Path) > at java.lang.Class.getMethod(Class.java:1665) > at > org.apache.flume.sink.hdfs.BucketWriter.getRefIsClosed(BucketWriter.java:207) > at org.apache.flume.sink.hdfs.BucketWriter.open(BucketWriter.java:295) > at org.apache.flume.sink.hdfs.BucketWriter.append(BucketWriter.java:554) > at > org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:426) > at > org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68) > at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147) > at java.lang.Thread.run(Thread.java:744) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (FLUME-3049) Wrapping the exception into SecurityException in UGIExecutor.execute hides the original one
[
https://issues.apache.org/jira/browse/FLUME-3049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15848850#comment-15848850
]
Hudson commented on FLUME-3049:
---
ABORTED: Integrated in Jenkins build Flume-trunk-hbase-1 #239 (See
[https://builds.apache.org/job/Flume-trunk-hbase-1/239/])
FLUME-3049. Make HDFS sink rotate more reliably in secure mode (mpercy:
[http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git=commit=f215374bdf9a08b16fa7ccd3b40098024afe8677])
* (edit)
flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestBucketWriter.java
* (edit) flume-ng-auth/src/main/java/org/apache/flume/auth/UGIExecutor.java
* (edit)
flume-ng-auth/src/test/java/org/apache/flume/auth/TestFlumeAuthenticator.java
> Wrapping the exception into SecurityException in UGIExecutor.execute hides
> the original one
> ---
>
> Key: FLUME-3049
> URL: https://issues.apache.org/jira/browse/FLUME-3049
> Project: Flume
> Issue Type: Bug
>Reporter: Denes Arvay
>Assignee: Denes Arvay
> Fix For: v1.8.0
>
>
> see:
> https://github.com/apache/flume/blob/trunk/flume-ng-auth/src/main/java/org/apache/flume/auth/UGIExecutor.java#L49
> This has unexpected side effects as the callers try to catch the wrapped
> exception, for example in {{BucketWriter.append()}}:
> https://github.com/apache/flume/blob/trunk/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java#L563
> Here IOException is considered as non-transient failure thus the {{close()}}
> is called, but when the original exception is wrapped into
> {{SecurityException}} it doesn't trigger the close of the file.
> Similarly in {{HDFSEventSink.process()}} method the `IOException` is handled
> in a different way than any other exception. It might come from
> {{BucketWriter.append()}} or {{BucketWriter.flush()}} for example, and both
> of them invoke the hdfs call via a {{PrivilegedExecutor}} instance which
> might be the problematic {{UGIExecutor}}.
> The bottom line is that the contract in {{PrivilegedExecutor.execute()}} is
> that they shouldn't change the exception thrown in the business logic - at
> least it's not indicated in its signature in any way. The default
> implementation ({{SimpleAuthenticator}}) behaves according to this.
> I don't know the original intend behind this wrapping, [~jrufus] or
> [~hshreedharan], do you happen to remember? (You were involved in the
> original implementation in FLUME-2631)
> Right now I don't see any problem in removing this and letting the original
> exception to propagate as the {{org.apache.flume.auth.SecurityException}}
> doesn't appear anywhere in the public interface.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (FLUME-2993) Support environment variables in configuration files
[ https://issues.apache.org/jira/browse/FLUME-2993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15805528#comment-15805528 ] Hudson commented on FLUME-2993: --- ABORTED: Integrated in Jenkins build Flume-trunk-hbase-1 #236 (See [https://builds.apache.org/job/Flume-trunk-hbase-1/236/]) FLUME-2993. Add support for environment variables in configuration files (bessbd: [http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git=commit=5e52ac4ad18a7776f4c74499c1314fbfd3513160]) * (edit) flume-ng-node/src/main/java/org/apache/flume/node/PropertiesFileConfigurationProvider.java * (add) flume-ng-node/src/main/java/org/apache/flume/node/EnvVarResolverProperties.java * (edit) flume-ng-doc/sphinx/FlumeUserGuide.rst * (edit) flume-ng-node/pom.xml * (add) flume-ng-node/src/test/java/org/apache/flume/node/TestEnvVarResolverProperties.java * (add) flume-ng-node/src/test/resources/flume-conf-with-envvars.properties > Support environment variables in configuration files > > > Key: FLUME-2993 > URL: https://issues.apache.org/jira/browse/FLUME-2993 > Project: Flume > Issue Type: New Feature > Components: Configuration >Reporter: Bessenyei Balázs Donát >Assignee: Bessenyei Balázs Donát > Fix For: v1.8.0 > > > Flume does not currently support environment variable interpolation in the > properties file configuration. > Enabling it would help with > * removing security credentials from config files > * help with copy-pastes in configuration files with multiple agents defined > It is arguably a best practice to store (some) config in the environment > variables: https://12factor.net/config -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-3031) sequence source should reset its counter for event body on channel exception
[ https://issues.apache.org/jira/browse/FLUME-3031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15726265#comment-15726265 ] Hudson commented on FLUME-3031: --- UNSTABLE: Integrated in Jenkins build Flume-trunk-hbase-1 #235 (See [https://builds.apache.org/job/Flume-trunk-hbase-1/235/]) FLUME-3031. Change sequence source to reset its counter for event body (bessbd: [http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git=commit=2ff2dbbd13db5de747c654ef132c98941cdd45dc]) * (edit) flume-ng-core/src/main/java/org/apache/flume/source/SequenceGeneratorSource.java * (edit) flume-ng-core/src/test/java/org/apache/flume/source/TestSequenceGeneratorSource.java * (edit) flume-ng-doc/sphinx/FlumeUserGuide.rst > sequence source should reset its counter for event body on channel exception > > > Key: FLUME-3031 > URL: https://issues.apache.org/jira/browse/FLUME-3031 > Project: Flume > Issue Type: Bug > Components: Sinks+Sources, Test >Affects Versions: v1.7.0 >Reporter: Attila Simon >Assignee: Attila Simon > Fix For: v1.8.0 > > > SequenceGeneratorSource uses a counter to construct the body of the generated > Events which counter is not reseted when writing event to channel failed. > This can lead to a situation that the total number of unique events at > destination (if deduplication relies on msg body) is bigger than the > totalEvents configuration parameter due to the retries. > Fix should make sure that number of events at destination after filtering out > the duplicates is equal to the configured value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-3027) Kafka Channel should clear offsets map after commit
[ https://issues.apache.org/jira/browse/FLUME-3027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15724784#comment-15724784 ] Hudson commented on FLUME-3027: --- UNSTABLE: Integrated in Jenkins build Flume-trunk-hbase-1 #234 (See [https://builds.apache.org/job/Flume-trunk-hbase-1/234/]) FLUME-3027. Change Kafka Channel to clear offsets map after commit (bessbd: [http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git=commit=fa1ee05af38bcf08ed18ff36d4284e68836a9054]) * (edit) flume-ng-channels/flume-kafka-channel/src/main/java/org/apache/flume/channel/kafka/KafkaChannel.java > Kafka Channel should clear offsets map after commit > --- > > Key: FLUME-3027 > URL: https://issues.apache.org/jira/browse/FLUME-3027 > Project: Flume > Issue Type: Bug >Reporter: Jeff Holoman >Assignee: Jeff Holoman > Fix For: v1.8.0 > > > Currently the channel doesn't clear the TopicPartition / OffsetsAndMetadata > map after a commit. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-3025) Expose FileChannel.open on JMX
[
https://issues.apache.org/jira/browse/FLUME-3025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15684972#comment-15684972
]
Hudson commented on FLUME-3025:
---
SUCCESS: Integrated in Jenkins build Flume-trunk-hbase-1 #230 (See
[https://builds.apache.org/job/Flume-trunk-hbase-1/230/])
FLUME-3025. Expose FileChannel.open on JMX (bessbd:
[http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git=commit=21a64db1b0ebcb812e5c510668b580e84ff37f35])
* (edit)
flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java
* (edit)
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java
* (add)
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/instrumentation/FileChannelCounter.java
* (add)
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/instrumentation/FileChannelCounterMBean.java
> Expose FileChannel.open on JMX
> --
>
> Key: FLUME-3025
> URL: https://issues.apache.org/jira/browse/FLUME-3025
> Project: Flume
> Issue Type: Improvement
>Reporter: Denes Arvay
>Assignee: Denes Arvay
> Fix For: v1.8.0
>
>
> We've encountered an issue several times when the {{FileChannel}} wasn't able
> to start because of some startup-time exception.
> In this case {{FileChannel.open}} stays false {{false}}, exposing this field
> on JMX would make it easier to detect when the file channel hasn't been
> started / or has been stopped because of some reason.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLUME-2171) Add Interceptor to remove headers from event
[ https://issues.apache.org/jira/browse/FLUME-2171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15682438#comment-15682438 ] Hudson commented on FLUME-2171: --- SUCCESS: Integrated in Jenkins build Flume-trunk-hbase-1 #229 (See [https://builds.apache.org/job/Flume-trunk-hbase-1/229/]) FLUME-2171. Add Interceptor to remove headers from event (jholoman: [http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git=commit=de941e7b85ed55bfbd804880f421be3889187d31]) * (edit) flume-ng-doc/sphinx/FlumeUserGuide.rst * (edit) flume-ng-core/src/main/java/org/apache/flume/interceptor/InterceptorType.java * (add) flume-ng-core/src/main/java/org/apache/flume/interceptor/RemoveHeaderInterceptor.java * (add) flume-ng-core/src/test/java/org/apache/flume/interceptor/RemoveHeaderInterceptorTest.java > Add Interceptor to remove headers from event > > > Key: FLUME-2171 > URL: https://issues.apache.org/jira/browse/FLUME-2171 > Project: Flume > Issue Type: New Feature > Components: Easy >Affects Versions: v1.4.0 >Reporter: Gabriel Commeau >Assignee: Bessenyei Balázs Donát > Labels: reviewboard-missing > Attachments: FLUME-2171-0.patch > > > I found Flume OG's decorators to handle event headers useful and some to be > missing from Flume NG. More specifically, we could have an interceptor to > remove headers from an event. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2812) Exception in thread "SinkRunner-PollingRunner-DefaultSinkProcessor" java.lang.Error: Maximum permit count exceeded
[ https://issues.apache.org/jira/browse/FLUME-2812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15638843#comment-15638843 ] Hudson commented on FLUME-2812: --- FAILURE: Integrated in Jenkins build Flume-trunk-hbase-1 #226 (See [https://builds.apache.org/job/Flume-trunk-hbase-1/226/]) FLUME-2812. Fix semaphore leak causing java.lang.Error: Maximum permit (bessbd: [http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git=commit=f581f6ed0e820da370c4a83a4ff7b05278d09fc3]) * (edit) flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannel.java * (edit) flume-ng-core/src/main/java/org/apache/flume/channel/MemoryChannel.java > Exception in thread "SinkRunner-PollingRunner-DefaultSinkProcessor" > java.lang.Error: Maximum permit count exceeded > -- > > Key: FLUME-2812 > URL: https://issues.apache.org/jira/browse/FLUME-2812 > Project: Flume > Issue Type: Bug > Components: Channel, Sinks+Sources >Affects Versions: v1.6.0 > Environment: **OS INFO** > CentOS release 6.6 (Final) > Kernel \r on an \m > **JAVA INFO** > java version "1.8.0_40" > Java(TM) SE Runtime Environment (build 1.8.0_40-b26) > Java HotSpot(TM) 64-Bit Server VM (build 25.40-b25, mixed mode) >Reporter: Rollin Crittendon >Assignee: Denes Arvay >Priority: Critical > Fix For: v1.8.0 > > > We are finding that around after an hour or so of heavy processing of Flume > data in an agent we are getting the following exception. This is after > processing about 5-7 k lines/second during that time. > The configuration of this agent is using a Kafka source, the one that comes > with 1.6.0. > It is also using a Memory channel, and a Thrift sink. > === > Exception in thread "SinkRunner-PollingRunner-DefaultSinkProcessor" > java.lang.Error: Maximum permit count exceeded > at > java.util.concurrent.Semaphore$Sync.tryReleaseShared(Semaphore.java:192) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.releaseShared(AbstractQueuedSynchronizer.java:1341) > at java.util.concurrent.Semaphore.release(Semaphore.java:609) > at > org.apache.flume.channel.MemoryChannel$MemoryTransaction.doCommit(MemoryChannel.java:147) > at > org.apache.flume.channel.BasicTransactionSemantics.commit(BasicTransactionSemantics.java:151) > at > org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:379) > at > org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68) > at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147) > at java.lang.Thread.run(Thread.java:745) > === > The above error is from standard error when running the Flume agent. The > effect is that the "SinkRunner-PollingRunner-DefaultSinkProcessor" thread > disappears from the agent, this can be seen on a JMX console. > For us, this means that the Flume agent needs to get restarted. It is an > error that is terminal in that instance of the Java process due to the thread > disappearing as a result. > It sounds like something in JDK 7+ got stricter?! -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2857) Kafka Source/Channel/Sink does not restore default values when live update config
[
https://issues.apache.org/jira/browse/FLUME-2857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15615973#comment-15615973
]
Hudson commented on FLUME-2857:
---
UNSTABLE: Integrated in Jenkins build Flume-trunk-hbase-1 #225 (See
[https://builds.apache.org/job/Flume-trunk-hbase-1/225/])
FLUME-2857. Make Kafka Source/Channel/Sink restore default values when (bessbd:
[http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git=commit=14fb4d84fd0e100253ca947bc96810c242e7a82b])
* (edit)
flume-ng-channels/flume-kafka-channel/src/test/java/org/apache/flume/channel/kafka/TestKafkaChannel.java
* (edit)
flume-ng-sources/flume-kafka-source/src/test/java/org/apache/flume/source/kafka/TestKafkaSource.java
* (edit)
flume-ng-sources/flume-kafka-source/src/main/java/org/apache/flume/source/kafka/KafkaSource.java
* (edit)
flume-ng-sinks/flume-ng-kafka-sink/src/test/java/org/apache/flume/sink/kafka/TestKafkaSink.java
* (edit)
flume-ng-channels/flume-kafka-channel/src/main/java/org/apache/flume/channel/kafka/KafkaChannel.java
* (edit)
flume-ng-sinks/flume-ng-kafka-sink/src/main/java/org/apache/flume/sink/kafka/KafkaSink.java
> Kafka Source/Channel/Sink does not restore default values when live update
> config
> -
>
> Key: FLUME-2857
> URL: https://issues.apache.org/jira/browse/FLUME-2857
> Project: Flume
> Issue Type: Bug
> Components: Channel
>Reporter: Tristan Stevens
>Assignee: Tristan Stevens
> Fix For: v1.8.0
>
> Attachments: FLUME-2857.patch
>
>
> Been using the following config:
> {noformat}
> tier1.channels.channel1.type = org.apache.flume.channel.kafka.KafkaChannel
> tier1.channels.channel1.capacity = 1
> tier1.channels.channel1.transactionCapacity = 1
> tier1.channels.channel1.brokerList =
> 10.0.0.64:9092,10.0.0.65:9092,10.0.0.66:9092
> tier1.channels.channel1.topic = flume.aggregator.channel
> tier1.channels.channel1.zookeeperConnect = 10.0.0.64:2181
> tier1.channels.channel1.kafka.producer.type=async
> tier1.channels.channel1.kafka.batch.num.messages=200
> {noformat}
> If I remove the producer.type and batch.num.messages (i.e. restore them to
> default values) when doing a live update config the new values do not take
> effect.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLUME-2997) Fix flaky junit test in SpillableMemoryChannel
[ https://issues.apache.org/jira/browse/FLUME-2997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15611692#comment-15611692 ] Hudson commented on FLUME-2997: --- FAILURE: Integrated in Jenkins build Flume-trunk-hbase-1 #222 (See [https://builds.apache.org/job/Flume-trunk-hbase-1/222/]) FLUME-2997. Fix flaky test in SpillableMemoryChannel (bessbd: [http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git=commit=7c4b2fd3e88f038dea3bdae01eb864547770ad48]) * (edit) flume-ng-channels/flume-spillable-memory-channel/src/test/java/org/apache/flume/channel/TestSpillableMemoryChannel.java > Fix flaky junit test in SpillableMemoryChannel > -- > > Key: FLUME-2997 > URL: https://issues.apache.org/jira/browse/FLUME-2997 > Project: Flume > Issue Type: Test >Affects Versions: v1.7.0 >Reporter: Attila Simon >Assignee: Attila Simon > Fix For: v1.8.0 > > Attachments: FLUME-2997-1.patch, FLUME-2997.patch > > > testParallelSingleSourceAndSink sometimes trigger an edge case scenario if > sinks are slower than sources. In such situations the channel can get full > thus uncaught ChannelFullException breaks the test. Since > testCapacityWithOverflow was designed to cover such edge-case scenario > already we can safely fix the test by increasing the channel capacity to make > sure it won't get full. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-3002) Some tests in TestBucketWriter are flaky
[ https://issues.apache.org/jira/browse/FLUME-3002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15602231#comment-15602231 ] Hudson commented on FLUME-3002: --- UNSTABLE: Integrated in Jenkins build Flume-trunk-hbase-1 #220 (See [https://builds.apache.org/job/Flume-trunk-hbase-1/220/]) FLUME-3002. Fix tests in TestBucketWriter (bessbd: [http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git=commit=2399329ee2ca2d9fc4ec0ec8fc5d16fb213795b2]) * (edit) flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestBucketWriter.java * (edit) flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java > Some tests in TestBucketWriter are flaky > > > Key: FLUME-3002 > URL: https://issues.apache.org/jira/browse/FLUME-3002 > Project: Flume > Issue Type: Test > Components: Sinks+Sources >Affects Versions: v1.6.0 >Reporter: Bessenyei Balázs Donát >Assignee: Bessenyei Balázs Donát > Fix For: v1.8.0 > > > testFileSuffixNotGiven (and probably a few other tests) are flaky -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2924) Flume 1.7.0 release
[ https://issues.apache.org/jira/browse/FLUME-2924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15563531#comment-15563531 ] Hudson commented on FLUME-2924: --- UNSTABLE: Integrated in Jenkins build Flume-trunk-hbase-1 #210 (See [https://builds.apache.org/job/Flume-trunk-hbase-1/210/]) FLUME-2924. Update POM versions to 1.8.0-SNAPSHOT (bessbd: [http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git=commit=bb0eba0b2000bbd0ec6602f6221279412cc208b4]) * (edit) flume-ng-node/pom.xml * (edit) flume-shared/pom.xml * (edit) flume-ng-configuration/pom.xml * (edit) flume-ng-embedded-agent/pom.xml * (edit) flume-ng-sdk/pom.xml * (edit) flume-ng-sinks/flume-irc-sink/pom.xml * (edit) flume-ng-sources/flume-jms-source/pom.xml * (edit) flume-ng-sinks/flume-ng-elasticsearch-sink/pom.xml * (edit) flume-ng-tests/pom.xml * (edit) flume-shared/flume-shared-kafka-test/pom.xml * (edit) flume-ng-sinks/flume-ng-morphline-solr-sink/pom.xml * (edit) flume-ng-doc/sphinx/index.rst * (edit) flume-ng-legacy-sources/pom.xml * (edit) flume-ng-channels/pom.xml * (edit) flume-ng-dist/pom.xml * (edit) flume-ng-auth/pom.xml * (edit) flume-ng-sources/flume-scribe-source/pom.xml * (edit) flume-ng-sinks/flume-ng-hbase-sink/pom.xml * (edit) flume-ng-sinks/flume-dataset-sink/pom.xml * (edit) pom.xml * (edit) flume-ng-sinks/flume-hdfs-sink/pom.xml * (edit) flume-ng-sinks/flume-hive-sink/pom.xml * (edit) flume-ng-channels/flume-kafka-channel/pom.xml * (edit) flume-ng-channels/flume-jdbc-channel/pom.xml * (edit) flume-checkstyle/pom.xml * (edit) flume-ng-sinks/flume-ng-kafka-sink/pom.xml * (edit) flume-ng-sources/flume-taildir-source/pom.xml * (edit) flume-ng-clients/pom.xml * (edit) flume-ng-doc/sphinx/FlumeUserGuide.rst * (edit) flume-ng-legacy-sources/flume-avro-source/pom.xml * (edit) flume-ng-legacy-sources/flume-thrift-source/pom.xml * (edit) flume-ng-sources/pom.xml * (edit) flume-ng-sinks/pom.xml * (edit) flume-ng-channels/flume-file-channel/pom.xml * (edit) flume-ng-sources/flume-kafka-source/pom.xml * (edit) flume-ng-sources/flume-twitter-source/pom.xml * (edit) flume-tools/pom.xml * (edit) flume-ng-doc/sphinx/FlumeDeveloperGuide.rst * (edit) flume-ng-clients/flume-ng-log4jappender/pom.xml * (edit) flume-ng-core/pom.xml * (edit) flume-ng-channels/flume-spillable-memory-channel/pom.xml > Flume 1.7.0 release > --- > > Key: FLUME-2924 > URL: https://issues.apache.org/jira/browse/FLUME-2924 > Project: Flume > Issue Type: Umbrella >Affects Versions: v1.7.0 >Reporter: Lior Zeno > Fix For: v1.7.0 > > > Release 1.7.0 umbrella issue -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2999) Kafka channel and sink should enable statically assigned partition per event via header
[ https://issues.apache.org/jira/browse/FLUME-2999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15563288#comment-15563288 ] Hudson commented on FLUME-2999: --- FAILURE: Integrated in Jenkins build Flume-trunk-hbase-1 #209 (See [https://builds.apache.org/job/Flume-trunk-hbase-1/209/]) FLUME-2999. Kafka channel and sink should enable statically assigned (mpercy: [http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git=commit=585c4c92ed65c529871765ab64a5b3637fbdb07d]) * (edit) flume-ng-doc/sphinx/FlumeUserGuide.rst * (add) flume-shared/flume-shared-kafka-test/src/main/java/org/apache/flume/shared/kafka/test/PartitionTestScenario.java * (add) flume-shared/pom.xml * (edit) flume-ng-sources/flume-kafka-source/pom.xml * (edit) flume-ng-channels/flume-kafka-channel/src/main/java/org/apache/flume/channel/kafka/KafkaChannel.java * (edit) pom.xml * (edit) flume-ng-channels/flume-kafka-channel/src/main/java/org/apache/flume/channel/kafka/KafkaChannelConfiguration.java * (edit) flume-ng-channels/flume-kafka-channel/src/test/java/org/apache/flume/channel/kafka/TestKafkaChannel.java * (edit) flume-ng-sinks/flume-ng-kafka-sink/src/main/java/org/apache/flume/sink/kafka/KafkaSink.java * (add) flume-shared/flume-shared-kafka-test/src/main/java/org/apache/flume/shared/kafka/test/KafkaPartitionTestUtil.java * (add) flume-shared/flume-shared-kafka-test/src/main/java/org/apache/flume/shared/kafka/test/PartitionOption.java * (add) flume-shared/flume-shared-kafka-test/pom.xml * (edit) flume-ng-sinks/flume-ng-kafka-sink/pom.xml * (edit) flume-ng-sinks/flume-ng-kafka-sink/src/main/java/org/apache/flume/sink/kafka/KafkaSinkConstants.java * (edit) flume-ng-channels/flume-kafka-channel/pom.xml * (edit) flume-ng-sinks/flume-ng-kafka-sink/src/test/java/org/apache/flume/sink/kafka/TestKafkaSink.java > Kafka channel and sink should enable statically assigned partition per event > via header > --- > > Key: FLUME-2999 > URL: https://issues.apache.org/jira/browse/FLUME-2999 > Project: Flume > Issue Type: Improvement > Components: Channel, Sinks+Sources >Affects Versions: v1.7.0 >Reporter: Tristan Stevens >Assignee: Tristan Stevens >Priority: Minor > Fix For: v1.7.0 > > Attachments: FLUME-2999-4.patch > > > This feature is useful for anyone who needs greater control of which > partitions are being written to - normally in a situation where multiple > Flume agents are being deployed in order to horizontally scale, or > alternatively if there is a scenario where there is a skew in data that might > lead to one or more partitions hotspotting. > We also have the ability to specify custom partitions on to the Kafka > Producer itself using the kafka.* configuration properties. > The Kafka Producer provides the ability to set the partition ID using the > following constructor > (https://kafka.apache.org/090/javadoc/org/apache/kafka/clients/producer/ProducerRecord.html#ProducerRecord(java.lang.String,%20java.lang.Integer,%20K,%20V%29 > ), this is just a matter of providing the option to use this constructor. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2911) Add includePattern option in SpoolDirectorySource configuration
[
https://issues.apache.org/jira/browse/FLUME-2911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15562829#comment-15562829
]
Hudson commented on FLUME-2911:
---
FAILURE: Integrated in Jenkins build Flume-trunk-hbase-1 #208 (See
[https://builds.apache.org/job/Flume-trunk-hbase-1/208/])
FLUME-2911. Add include pattern option in SpoolDir source (mpercy:
[http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git=commit=7d5ceacac49f5d15bf8f75e0209592c5524a3dda])
* (edit) flume-ng-doc/sphinx/FlumeUserGuide.rst
* (edit)
flume-ng-core/src/main/java/org/apache/flume/client/avro/ReliableSpoolingFileEventReader.java
* (edit)
flume-ng-core/src/main/java/org/apache/flume/source/SpoolDirectorySource.java
* (edit)
flume-ng-core/src/test/java/org/apache/flume/client/avro/TestReliableSpoolingFileEventReader.java
* (edit)
flume-ng-core/src/main/java/org/apache/flume/source/SpoolDirectorySourceConfigurationConstants.java
> Add includePattern option in SpoolDirectorySource configuration
> ---
>
> Key: FLUME-2911
> URL: https://issues.apache.org/jira/browse/FLUME-2911
> Project: Flume
> Issue Type: Improvement
> Components: Sinks+Sources
>Affects Versions: v1.6.0, v1.7.0
>Reporter: Andrea Rota
>Assignee: Andrea Rota
> Labels: features
> Fix For: v1.7.0
>
> Attachments: FLUME-2911.patch
>
>
> Current implementation of SpoolDirectorySource does not allow users to
> specify a regex pattern to select which files should be monitored. Instead,
> the current implementation allows users to specify which should *not*
> monitored, via the ignorePattern parameter.
> I implemented the feature, allowing users to specify the include pattern as
> {{a1.sources.src-1.includePattern=^foo.*$}} (includes all the files that
> starts in "foo").
> By default, the includePattern regex is set to {{^.*$}} (all files). Include
> and exclude patterns can be used at same time and the results are combined.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLUME-3003) testSourceCounter in TestSyslogUdpSource is flaky
[ https://issues.apache.org/jira/browse/FLUME-3003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1043#comment-1043 ] Hudson commented on FLUME-3003: --- FAILURE: Integrated in Jenkins build Flume-trunk-hbase-1 #207 (See [https://builds.apache.org/job/Flume-trunk-hbase-1/207/]) FLUME-3003. Fix flaky testSourceCounter in TestSyslogUdpSource (bessbd: [http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git=commit=27c725802b16915acaa3ea05cdac0236972af1f0]) * (edit) flume-ng-core/src/test/java/org/apache/flume/source/TestSyslogUdpSource.java > testSourceCounter in TestSyslogUdpSource is flaky > - > > Key: FLUME-3003 > URL: https://issues.apache.org/jira/browse/FLUME-3003 > Project: Flume > Issue Type: Test > Components: Sinks+Sources >Affects Versions: v1.6.0 >Reporter: Denes Arvay >Assignee: Denes Arvay > Fix For: v1.7.0 > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2998) Add missing configuration parameter to SequenceSource docs
[ https://issues.apache.org/jira/browse/FLUME-2998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15535822#comment-15535822 ] Hudson commented on FLUME-2998: --- ABORTED: Integrated in Jenkins build Flume-trunk-hbase-1 #206 (See [https://builds.apache.org/job/Flume-trunk-hbase-1/206/]) FLUME-2998. Add a missing configuration parameter to SequenceSource docs (bessbd: [http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git=commit=fca8c5b970cf0bbae61bae08f2084dd4f3bd1143]) * (edit) flume-ng-doc/sphinx/FlumeUserGuide.rst > Add missing configuration parameter to SequenceSource docs > -- > > Key: FLUME-2998 > URL: https://issues.apache.org/jira/browse/FLUME-2998 > Project: Flume > Issue Type: Documentation > Components: Docs >Affects Versions: v1.7.0 >Reporter: Attila Simon >Assignee: Attila Simon > Fix For: v1.7.0 > > Attachments: FLUME-2998.patch > > > totalEvents can be specified -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2966) NULL text in a TextMessage from a JMS source in Flume can lead to NPE
[
https://issues.apache.org/jira/browse/FLUME-2966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15523310#comment-15523310
]
Hudson commented on FLUME-2966:
---
UNSTABLE: Integrated in Jenkins build Flume-trunk-hbase-1 #205 (See
[https://builds.apache.org/job/Flume-trunk-hbase-1/205/])
FLUME-2966. Fix NPE in JMS Source (bessbd:
[http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git=commit=591b138321280cea1e7d61efcaf625a67202cb3d])
* (edit)
flume-ng-sources/flume-jms-source/src/test/java/org/apache/flume/source/jms/TestDefaultJMSMessageConverter.java
* (edit)
flume-ng-sources/flume-jms-source/src/main/java/org/apache/flume/source/jms/DefaultJMSMessageConverter.java
> NULL text in a TextMessage from a JMS source in Flume can lead to NPE
> -
>
> Key: FLUME-2966
> URL: https://issues.apache.org/jira/browse/FLUME-2966
> Project: Flume
> Issue Type: Bug
>Affects Versions: v1.5.0
>Reporter: Siddharth Ahuja
>Assignee: Siddharth Ahuja
> Attachments: App.java, FLUME-2966-0.patch, FLUME-2966-1.patch
>
>
> Code at
> https://github.com/apache/flume/blob/trunk/flume-ng-sources/flume-jms-source/src/main/java/org/apache/flume/source/jms/DefaultJMSMessageConverter.java#L103
> does not check for a NULL text in a TextMessage from a Flume JMS source.
> This can lead to a NullPointerException here:
> {code}textMessage.getText().getBytes(charset){code} while trying to
> de-reference a null text from the textmessage.
> We should probably skip these like the NULL Objects in the ObjectMessage just
> below at:
> https://github.com/apache/flume/blob/trunk/flume-ng-sources/flume-jms-source/src/main/java/org/apache/flume/source/jms/DefaultJMSMessageConverter.java#L107.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLUME-2797) SyslogTcpSource uses Deprecated Class + Deprecate SyslogTcpSource
[ https://issues.apache.org/jira/browse/FLUME-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15474629#comment-15474629 ] Hudson commented on FLUME-2797: --- SUCCESS: Integrated in Jenkins build Flume-trunk-hbase-1 #204 (See [https://builds.apache.org/job/Flume-trunk-hbase-1/204/]) FLUME-2797. Use SourceCounter for SyslogTcpSource (mpercy: [http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git=commit=d9c9a7dd9a6889ecf6b9dc88fb8e02ccc1cd5167]) * (edit) flume-ng-core/src/test/java/org/apache/flume/source/TestSyslogUdpSource.java * (edit) flume-ng-core/src/main/java/org/apache/flume/source/SyslogTcpSource.java * (edit) flume-ng-core/src/main/java/org/apache/flume/source/SyslogUDPSource.java * (edit) flume-ng-core/src/test/java/org/apache/flume/source/TestSyslogTcpSource.java > SyslogTcpSource uses Deprecated Class + Deprecate SyslogTcpSource > - > > Key: FLUME-2797 > URL: https://issues.apache.org/jira/browse/FLUME-2797 > Project: Flume > Issue Type: Bug >Reporter: Otis Gospodnetic >Assignee: Denes Arvay >Priority: Minor > Fix For: v1.7.0 > > Attachments: FLUME-2797-0.patch, FLUME-2797-2.patch, > FLUME-2797-3.patch, FLUME-2797-4.patch, FLUME-2797-5.patch, FLUME-2797-6.patch > > > From the mailing list: > From Ashish: > Source uses an deprecated class. Can you please file a JIRA for this? > The fix is simple. In SyslogTcpSource Line#61, replace the > CounterGroup usage with SourceCounter and make related changes in > code. You can refer other Sources for details. Same is applicable for > SyslogUDPSource. > From Hari: > I think the Syslog TCP source should be deprecated in favor of the Multiport > Syslog Source - that is more stable and gives better performance too. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2982) Add localhost escape sequence to HDFS sink
[ https://issues.apache.org/jira/browse/FLUME-2982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15454273#comment-15454273 ] Hudson commented on FLUME-2982: --- FAILURE: Integrated in Jenkins build Flume-trunk-hbase-1 #203 (See [https://builds.apache.org/job/Flume-trunk-hbase-1/203/]) FLUME-2982. Add localhost escape sequence to HDFS sink (mpercy: [http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git=commit=a0a50849d5ad33e6e6903316a3bd3dbba8547843]) * (edit) flume-ng-core/src/main/java/org/apache/flume/formatter/output/BucketPath.java * (edit) flume-ng-doc/sphinx/FlumeUserGuide.rst * (edit) flume-ng-core/src/test/java/org/apache/flume/formatter/output/TestBucketPath.java > Add localhost escape sequence to HDFS sink > -- > > Key: FLUME-2982 > URL: https://issues.apache.org/jira/browse/FLUME-2982 > Project: Flume > Issue Type: Improvement >Reporter: Jeff Holoman >Assignee: Jeff Holoman > Fix For: v1.7.0 > > Attachments: FLUME-2982.patch.4 > > > It would be useful to be able to just use %localhost% escape sequence instead > of having to pass in a header / use the host interceptor. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2954) make raw data appearing in log messages explicit
[ https://issues.apache.org/jira/browse/FLUME-2954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15446933#comment-15446933 ] Hudson commented on FLUME-2954: --- FAILURE: Integrated in Jenkins build Flume-trunk-hbase-1 #200 (See [https://builds.apache.org/job/Flume-trunk-hbase-1/200/]) FLUME-2954. Make raw data appearing in log messages explicit (mpercy: [http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git=commit=25e4bc6d80cf475862a1686fb2c3c97fcea27278]) * (edit) flume-ng-channels/flume-kafka-channel/src/main/java/org/apache/flume/channel/kafka/KafkaChannel.java * (edit) flume-ng-doc/sphinx/FlumeUserGuide.rst * (add) flume-ng-configuration/src/main/java/org/apache/flume/conf/LogPrivacyUtil.java * (edit) flume-ng-sinks/flume-ng-morphline-solr-sink/src/main/java/org/apache/flume/sink/solr/morphline/BlobHandler.java * (edit) flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java * (edit) flume-ng-sources/flume-kafka-source/src/main/java/org/apache/flume/source/kafka/KafkaSource.java * (edit) conf/flume-env.ps1.template * (edit) flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/JdbcChannelProviderImpl.java * (edit) flume-ng-sinks/flume-ng-morphline-solr-sink/src/main/java/org/apache/flume/sink/solr/morphline/MorphlineSink.java * (edit) flume-ng-core/src/main/java/org/apache/flume/source/MultiportSyslogTCPSource.java * (edit) flume-ng-sources/flume-twitter-source/src/main/java/org/apache/flume/source/twitter/TwitterSource.java * (edit) flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfiguration.java * (edit) flume-ng-core/src/main/java/org/apache/flume/source/http/BLOBHandler.java * (edit) conf/flume-env.sh.template * (edit) flume-ng-embedded-agent/src/main/java/org/apache/flume/agent/embedded/EmbeddedAgent.java * (edit) flume-ng-core/src/test/java/org/apache/flume/serialization/SyslogAvroEventSerializer.java * (edit) flume-ng-sinks/flume-ng-kafka-sink/src/main/java/org/apache/flume/sink/kafka/KafkaSink.java > make raw data appearing in log messages explicit > > > Key: FLUME-2954 > URL: https://issues.apache.org/jira/browse/FLUME-2954 > Project: Flume > Issue Type: Improvement > Components: Channel, Configuration, Sinks+Sources >Affects Versions: v1.6.0 >Reporter: Attila Simon >Assignee: Attila Simon >Priority: Critical > Fix For: v1.7.0 > > Attachments: FLUME-2954-4.patch > > > Flume has built in functionality to log out data flowing through > mainly for debugging purposes. This functionality appears in several > places of the codebase. I think such functionality rise security > concerns in production environments where sensitive information might > be ingested so it is crucial that enabling such functionality has to > be as explicit as possible (avoid implicit side effect setup). > Eg: setting the level of root logger to debug/trace cause that every > other logger will start logging at debug/trace including the ones > logging raw data. > In this jira I would like to provide a patch capturing how I imagined solving > this issue. It can be refined iteratively or used as a basis for a broader > discussion. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2972) Handle offset migration in the new Kafka Channel
[ https://issues.apache.org/jira/browse/FLUME-2972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15438282#comment-15438282 ] Hudson commented on FLUME-2972: --- FAILURE: Integrated in Jenkins build Flume-trunk-hbase-1 #199 (See [https://builds.apache.org/job/Flume-trunk-hbase-1/199/]) FLUME-2972. Handle offset migration in the new Kafka Channel (mpercy: [http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git=commit=1e8f2651dacf5daef55d75c7b9b12492962e7921]) * (edit) flume-ng-channels/flume-kafka-channel/src/main/java/org/apache/flume/channel/kafka/KafkaChannel.java * (edit) flume-ng-channels/flume-kafka-channel/src/main/java/org/apache/flume/channel/kafka/KafkaChannelConfiguration.java * (edit) flume-ng-channels/flume-kafka-channel/src/test/java/org/apache/flume/channel/kafka/TestKafkaChannel.java * (edit) flume-ng-doc/sphinx/FlumeUserGuide.rst * (edit) flume-ng-channels/flume-kafka-channel/pom.xml > Handle offset migration in the new Kafka Channel > > > Key: FLUME-2972 > URL: https://issues.apache.org/jira/browse/FLUME-2972 > Project: Flume > Issue Type: Bug > Components: Channel >Affects Versions: v1.7.0 >Reporter: Grant Henke >Assignee: Grant Henke > Fix For: v1.7.0 > > Attachments: FLUME-2972-1.patch, FLUME-2972-2.patch, FLUME-2972.patch > > > Offsets tracking the position in Kafka consumers change from using zookeeper > for offset storage to Kafka when moving from 0.8.x to 0.9.x. > FLUME-2823 makes the client change in the Kafka Channel but does not ensure > existing offsets get migrated in order to continue consuming where it left > off. > Flume should have some automated logic on startup to check if Kafka offsets > exist, if not and migration is enabled (by default) then copy the offsets > from Zookeeper and commit them to Kafka. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2974) Some tests are broken in TestReliableSpoolingFileEventReader and TestSpoolingFileLineReader
[ https://issues.apache.org/jira/browse/FLUME-2974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15431685#comment-15431685 ] Hudson commented on FLUME-2974: --- UNSTABLE: Integrated in Jenkins build Flume-trunk-hbase-1 #198 (See [https://builds.apache.org/job/Flume-trunk-hbase-1/198/]) FLUME-2974. Fix some broken SpoolDirSource unit tests (mpercy: [http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git=commit=9eb92dab0d24813e9a3871f0c7038920797c0466]) * (edit) flume-ng-core/src/test/java/org/apache/flume/client/avro/TestSpoolingFileLineReader.java * (edit) flume-ng-core/src/test/java/org/apache/flume/client/avro/TestReliableSpoolingFileEventReader.java > Some tests are broken in TestReliableSpoolingFileEventReader and > TestSpoolingFileLineReader > --- > > Key: FLUME-2974 > URL: https://issues.apache.org/jira/browse/FLUME-2974 > Project: Flume > Issue Type: Bug >Reporter: Bessenyei Balázs Donát >Assignee: Bessenyei Balázs Donát > Fix For: v1.7.0 > > > Changes in FLUME-2318 broke some tests that have empty files -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2975) Minor mistake in NetCat Source example in documentation
[ https://issues.apache.org/jira/browse/FLUME-2975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15429887#comment-15429887 ] Hudson commented on FLUME-2975: --- UNSTABLE: Integrated in Jenkins build Flume-trunk-hbase-1 #197 (See [https://builds.apache.org/job/Flume-trunk-hbase-1/197/]) FLUME-2975. docs: Fix NetcatSource example (mpercy: [http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git=commit=e0deb7281fce0f6937f20da305b58d52d9a67a94]) * (edit) flume-ng-doc/sphinx/FlumeUserGuide.rst > Minor mistake in NetCat Source example in documentation > --- > > Key: FLUME-2975 > URL: https://issues.apache.org/jira/browse/FLUME-2975 > Project: Flume > Issue Type: Documentation >Affects Versions: v1.6.0 >Reporter: Bessenyei Balázs Donát >Assignee: Bessenyei Balázs Donát >Priority: Trivial > Fix For: v1.7.0 > > Attachments: FLUME-2975.patch > > > Second "a1.sources.r1.bind" should be "a1.sources.r1.port" -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2844) ChannelCounter of SpillableMemoryChannel doesn't register actually.
[ https://issues.apache.org/jira/browse/FLUME-2844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15425620#comment-15425620 ] Hudson commented on FLUME-2844: --- ABORTED: Integrated in Jenkins build Flume-trunk-hbase-1 #196 (See [https://builds.apache.org/job/Flume-trunk-hbase-1/196/]) FLUME-2844. SpillableMemoryChannel must start ChannelCounter (mpercy: [http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git=commit=118d5a4bcf5371fb215e0f7393fadf26073643f8]) * (edit) flume-ng-channels/flume-spillable-memory-channel/src/test/java/org/apache/flume/channel/TestSpillableMemoryChannel.java * (edit) flume-ng-channels/flume-spillable-memory-channel/src/main/java/org/apache/flume/channel/SpillableMemoryChannel.java > ChannelCounter of SpillableMemoryChannel doesn't register actually. > --- > > Key: FLUME-2844 > URL: https://issues.apache.org/jira/browse/FLUME-2844 > Project: Flume > Issue Type: Bug > Components: Channel >Affects Versions: v1.6.0 >Reporter: Daniel Wen >Assignee: Denes Arvay > Fix For: v1.7.0 > > Attachments: FLUME-2844-2.patch, FLUME-2844..patch > > > When using SpillableMemoryChannel, the values of all metrics of channel > component int monitoring system is zero.It's caused by a bug belowed. > SpillableMemoryChannel extends FileChannel,and each of them have a > ChannelCounter-Type field named of 'channelCounter'. FileChannel's > channelCounter filed is private, so it can not be override by > SpillableMemoryChannel. SpillableMemoryChannel's channelCounter is > initialized in configure method(Line 600 at SpillableMemoryChannel.java), > then super.configure() is called(Line 727 at SpillableMemoryChannel.java). So > FileChannel's channelCounter is also initialized(Line 267 at > SpillableMemoryChannel.java). > When SpillableMemoryChannel start(call start() method), it should and want to > register ChannelCounter. In start() method, SpillableMemoryChannel calls > super.start() (Line 768 at SpillableMemoryChannel.java), then FileChannel > calls channelCounter.start() to register channelCounter. But this > channelCounter is the one of FileChannle, not the one of > SpillableMemoryChannel! > This bug results that when using SpillableMemoryChannel, the values of all > metrics of channel component is zero. > Maybe the qualifier of FileChannel's channelCounter filed should be changed > to prtected, and remove the channelCounter filed of SpillableMemoryChannel. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2318) SpoolingDirectory is unable to handle empty files
[
https://issues.apache.org/jira/browse/FLUME-2318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15424989#comment-15424989
]
Hudson commented on FLUME-2318:
---
FAILURE: Integrated in Jenkins build Flume-trunk-hbase-1 #194 (See
[https://builds.apache.org/job/Flume-trunk-hbase-1/194/])
FLUME-2318: Make SpoolingDirectorySource able to handle empty files (mpercy:
[http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git=commit=a9a775a9c8324e0724ed4720ae4f383896ea8c96])
* (edit)
flume-ng-core/src/main/java/org/apache/flume/client/avro/ReliableSpoolingFileEventReader.java
* (edit)
flume-ng-core/src/test/java/org/apache/flume/source/TestSpoolDirectorySource.java
> SpoolingDirectory is unable to handle empty files
> -
>
> Key: FLUME-2318
> URL: https://issues.apache.org/jira/browse/FLUME-2318
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
>Affects Versions: v1.4.0
>Reporter: Muhammad Ehsan ul Haque
>Assignee: Bessenyei Balázs Donát
>Priority: Minor
> Labels: easytest, patch
> Fix For: v1.7.0
>
> Attachments: FLUME-2318-0.patch, FLUME-2318-1.patch,
> FLUME-2318-2.patch
>
>
> Empty files should be returned as an empty event instead of no event.
> h4. Scenario
> From the start consume files in this order
> # f1: File with data or empty file
> # f2: Empty File
> # No file in spooling directory
> h4. Expected Outcome
> # channel.take() should return event with f1 data.
> # channel.take() should return event with f2 data (empty data).
> # channel.take() should return null.
> h4. What happens
> # channel.take() returns event with f1 data.
> # channel.take() returns null.
> # Exception is raised when the SpoolDirectorySource thread tries to read
> events from the ReliableSpoolingFileEventReader. Snippet of trace is
> 2014-02-09 15:46:35,832 (pool-1-thread-1) [INFO -
> org.apache.flume.client.avro.ReliableSpoolingFileEventReader.rollCurrentFile(ReliableSpoolingFileEventReader.java:346)]
> Preparing to move file /tmp/1391957195572-0/file1 to
> /tmp/1391957195572-0/file1.COMPLETED
> 2014-02-09 15:46:36,334 (pool-1-thread-1) [INFO -
> org.apache.flume.client.avro.ReliableSpoolingFileEventReader.readEvents(ReliableSpoolingFileEventReader.java:228)]
> Last read was never committed - resetting mark position.
> 2014-02-09 15:46:36,335 (pool-1-thread-1) [INFO -
> org.apache.flume.client.avro.ReliableSpoolingFileEventReader.rollCurrentFile(ReliableSpoolingFileEventReader.java:346)]
> Preparing to move file /tmp/1391957195572-0/file2 to
> /tmp/1391957195572-0/file2.COMPLETED
> 2014-02-09 15:46:36,839 (pool-1-thread-1) [ERROR -
> org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(SpoolDirectorySource.java:252)]
> FATAL: Spool Directory source null: { spoolDir: /tmp/1391957195572-0 }:
> Uncaught exception in SpoolDirectorySource thread. Restart or reconfigure
> Flume to continue processing.
> java.lang.IllegalStateException: File should not roll when commit is
> outstanding.
> at
> org.apache.flume.client.avro.ReliableSpoolingFileEventReader.readEvents(ReliableSpoolingFileEventReader.java:225)
> at
> org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(SpoolDirectorySource.java:224)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at
> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:722)
> h4. Unit Test
> In TestSpoolDirectorySource
> {code}
> @Test
> public void testWithEmptyFile2()
> throws InterruptedException, IOException {
> Context context = new Context();
> File f1 = new File(tmpDir.getAbsolutePath() + "/file1");
> Files.write("some data".getBytes(), f1);
> File f2 = new File(tmpDir.getAbsolutePath() + "/file2");
> Files.write(new byte[0], f2);
> context.put(SpoolDirectorySourceConfigurationConstants.SPOOL_DIRECTORY,
> tmpDir.getAbsolutePath());
> Configurables.configure(source, context);
> source.start();
> Thread.sleep(10);
> for (int i=0; i<2; i++) {
> Transaction txn = channel.getTransaction();
> txn.begin();
> Event e = channel.take();
> txn.commit();
> txn.close();
> }
[jira] [Commented] (FLUME-2619) Spooldir source does not log channel exceptions
[ https://issues.apache.org/jira/browse/FLUME-2619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15412857#comment-15412857 ] Hudson commented on FLUME-2619: --- FAILURE: Integrated in Flume-trunk-hbase-1 #192 (See [https://builds.apache.org/job/Flume-trunk-hbase-1/192/]) FLUME-2619. Spooldir source should log channel exceptions (mpercy: [http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git=commit=1422f733007dbb78caae7e5135bc33470e88502a]) * flume-ng-core/src/main/java/org/apache/flume/channel/ChannelProcessor.java * flume-ng-core/src/main/java/org/apache/flume/source/SpoolDirectorySource.java * flume-ng-core/src/test/java/org/apache/flume/source/TestSpoolDirectorySource.java > Spooldir source does not log channel exceptions > --- > > Key: FLUME-2619 > URL: https://issues.apache.org/jira/browse/FLUME-2619 > Project: Flume > Issue Type: Bug > Components: Sinks+Sources >Reporter: Santiago M. Mola >Assignee: Bessenyei Balázs Donát > Fix For: v1.7.0 > > Attachments: FLUME-2619-1.patch, FLUME-2619.patch > > > Spooldir assumes that any ChannelException means that the channel is full and > it does not log the exception message. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2963) FlumeUserGuide - error in Kafka Source properties table
[
https://issues.apache.org/jira/browse/FLUME-2963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15400233#comment-15400233
]
Hudson commented on FLUME-2963:
---
SUCCESS: Integrated in Flume-trunk-hbase-1 #190 (See
[https://builds.apache.org/job/Flume-trunk-hbase-1/190/])
FLUME-2963. FlumeUserGuide: Fix error in Kafka Source properties table (mpercy:
[http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git=commit=4b74aa286b16916900854ccea95cf906d3fdda5d])
* flume-ng-doc/sphinx/FlumeUserGuide.rst
> FlumeUserGuide - error in Kafka Source properties table
> ---
>
> Key: FLUME-2963
> URL: https://issues.apache.org/jira/browse/FLUME-2963
> Project: Flume
> Issue Type: Documentation
> Components: Docs
>Reporter: Denes Arvay
>Assignee: Denes Arvay
>Priority: Trivial
> Fix For: v1.7.0
>
> Attachments: FLUME-2963.patch
>
>
> {noformat}
> System Message: ERROR/3 (FlumeUserGuide.rst, line 1208)
> Malformed table. Column span alignment problem at line offset 27.
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLUME-2959) Fix issues with flume-checkstyle module
[ https://issues.apache.org/jira/browse/FLUME-2959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396171#comment-15396171 ] Hudson commented on FLUME-2959: --- FAILURE: Integrated in Flume-trunk-hbase-1 #186 (See [https://builds.apache.org/job/Flume-trunk-hbase-1/186/]) FLUME-2959. Fix issues with flume-checkstyle module (mpercy: [http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git=commit=988ede948ffaf6526c226323a6808922f38b625c]) * flume-checkstyle/pom.xml * flume-checkstyle/src/main/resources/flume/checkstyle-suppressions.xml * flume-checkstyle/src/main/resources/flume/checkstyle.xml > Fix issues with flume-checkstyle module > --- > > Key: FLUME-2959 > URL: https://issues.apache.org/jira/browse/FLUME-2959 > Project: Flume > Issue Type: Improvement >Reporter: Lior Zeno >Assignee: Lior Zeno >Priority: Minor > Fix For: v1.7.0 > > Attachments: FLUME-2959-0.patch, FLUME-2959-1.patch > > > In FLUME-2937 we added the flume-checkstyle module to Flume. > There are three minor issues with it: > # It does not follow our modules naming convention. It should be > flume-ng-checkstyle. > # It does not declare a parent module, resulting in build warnings (the > source encoding is not explicitly set to UTF-8) > # Resource files don't include the ASF licensing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2761) Move Hive sink out of preview mode
[ https://issues.apache.org/jira/browse/FLUME-2761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15387084#comment-15387084 ] Hudson commented on FLUME-2761: --- SUCCESS: Integrated in Flume-trunk-hbase-1 #184 (See [https://builds.apache.org/job/Flume-trunk-hbase-1/184/]) FLUME-2761. Move Hive sink out of preview mode (mpercy: [http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git=commit=10639e80cc83c14213dcc78692ab8f78945ddf8e]) * flume-ng-doc/sphinx/FlumeUserGuide.rst > Move Hive sink out of preview mode > -- > > Key: FLUME-2761 > URL: https://issues.apache.org/jira/browse/FLUME-2761 > Project: Flume > Issue Type: Bug > Components: Docs >Affects Versions: v1.6.0 >Reporter: Roshan Naik >Assignee: Roshan Naik > Fix For: v1.7.0 > > Attachments: FLUME-2761.patch > > > Been through one release. Move out of 'preview' mode. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2955) Add file path to the header in TaildirSource
[ https://issues.apache.org/jira/browse/FLUME-2955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15386672#comment-15386672 ] Hudson commented on FLUME-2955: --- UNSTABLE: Integrated in Flume-trunk-hbase-1 #183 (See [https://builds.apache.org/job/Flume-trunk-hbase-1/183/]) FLUME-2955. Add file path to the header in TaildirSource (mpercy: [http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git=commit=1ca0765aae795a41a43e39324f5f1c8bae57b751]) * flume-ng-sources/flume-taildir-source/src/test/java/org/apache/flume/source/taildir/TestTaildirSource.java * flume-ng-sources/flume-taildir-source/src/main/java/org/apache/flume/source/taildir/ReliableTaildirEventReader.java * flume-ng-sources/flume-taildir-source/src/main/java/org/apache/flume/source/taildir/TaildirSourceConfigurationConstants.java * flume-ng-doc/sphinx/FlumeUserGuide.rst * flume-ng-sources/flume-taildir-source/src/main/java/org/apache/flume/source/taildir/TaildirSource.java > Add file path to the header in TaildirSource > > > Key: FLUME-2955 > URL: https://issues.apache.org/jira/browse/FLUME-2955 > Project: Flume > Issue Type: Improvement > Components: Sinks+Sources >Affects Versions: v1.7.0 >Reporter: tinawenqiao >Assignee: tinawenqiao > Labels: file, path, taildirsource > Fix For: v1.7.0 > > Attachments: FLUME-2955.patch, FLUME-2955_2.patch, > FLUME-2955_3.patch, FLUME-2955_4.patch > > Original Estimate: 168h > Remaining Estimate: 168h > > After using TaildirSource to collect log data, I found that there is no file > path in the flume event. > Though TaildirSource supports add customed header value like this : > a1.sources.r1.headers.f1.headerKey1 = value1, > we can't add file path to the header when the filegroup path contains regex > expression.File path is necessary to locate the log data. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2514) Some TestFileChannelRestart tests are extremely slow
[ https://issues.apache.org/jira/browse/FLUME-2514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15386674#comment-15386674 ] Hudson commented on FLUME-2514: --- UNSTABLE: Integrated in Flume-trunk-hbase-1 #183 (See [https://builds.apache.org/job/Flume-trunk-hbase-1/183/]) FLUME-2514. Speed up TestFileChannelRestart tests (mpercy: [http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git=commit=c554ba3d9b8234b37ba0b5dde3657034d11acb0b]) * flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannelRestart.java > Some TestFileChannelRestart tests are extremely slow > > > Key: FLUME-2514 > URL: https://issues.apache.org/jira/browse/FLUME-2514 > Project: Flume > Issue Type: Bug > Components: File Channel >Reporter: Santiago M. Mola >Assignee: Santiago M. Mola > Fix For: v1.7.0 > > Attachments: FLUME-2215-0.patch, FLUME-2215-1.patch > > > TestFileChannelRestart tests are really slow. For example, > testToggleCheckpointCompressionFromFalseToTrue and > testToggleCheckpointCompressionFromTrueToFalse take ~4 minutes each. > Some of them could be made faster by using channels with lower capacity. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2810) Add static Schema URL to AvroEventSerializer configuration
[ https://issues.apache.org/jira/browse/FLUME-2810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15385230#comment-15385230 ] Hudson commented on FLUME-2810: --- SUCCESS: Integrated in Flume-trunk-hbase-1 #181 (See [https://builds.apache.org/job/Flume-trunk-hbase-1/181/]) FLUME-2810. Add static Schema URL to AvroEventSerializer configuration (mpercy: [http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git=commit=dbf2e989744a6b2921076be017102f75323a69f4]) * flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestAvroEventSerializer.java * flume-ng-core/src/main/java/org/apache/flume/serialization/AvroEventSerializerConfigurationConstants.java * flume-ng-doc/sphinx/FlumeUserGuide.rst * flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/AvroEventSerializer.java > Add static Schema URL to AvroEventSerializer configuration > -- > > Key: FLUME-2810 > URL: https://issues.apache.org/jira/browse/FLUME-2810 > Project: Flume > Issue Type: Improvement >Reporter: Jeff Holoman >Assignee: Jeff Holoman >Priority: Minor > Fix For: v1.7.0 > > Attachments: FLUME-2810v7.patch > > > Currently the only way to pass a schema to the avro event serializer is via > header. This would introduce an option to specify the location directly in > the configuration. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2952) SyslogAgent possible NPE on stop()
[ https://issues.apache.org/jira/browse/FLUME-2952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15376202#comment-15376202 ] Hudson commented on FLUME-2952: --- SUCCESS: Integrated in Flume-trunk-hbase-1 #180 (See [https://builds.apache.org/job/Flume-trunk-hbase-1/180/]) FLUME-2952. SyslogAgent: Avoid NPE on stop() (mpercy: [http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git=commit=c7de4ba5ca70c42099a2f9bf4ad2de3fdcbb7d30]) * flume-ng-tests/src/test/java/org/apache/flume/test/util/SyslogAgent.java > SyslogAgent possible NPE on stop() > -- > > Key: FLUME-2952 > URL: https://issues.apache.org/jira/browse/FLUME-2952 > Project: Flume > Issue Type: Bug >Affects Versions: v1.6.0 >Reporter: Mike Percy >Assignee: Mike Percy > Fix For: v1.7.0 > > Attachments: FLUME-2952.patch > > > If the SyslogAgent fails to start, calling stop() will result in an NPE. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2725) HDFS Sink does not use configured timezone for rounding
[ https://issues.apache.org/jira/browse/FLUME-2725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15375769#comment-15375769 ] Hudson commented on FLUME-2725: --- FAILURE: Integrated in Flume-trunk-hbase-1 #179 (See [https://builds.apache.org/job/Flume-trunk-hbase-1/179/]) FLUME-2725. HDFS Sink does not use configured timezone for rounding (mpercy: [http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git=commit=ec28b66246f1f165ccaf01abf7fb27adebc9e4bb]) * flume-ng-core/src/test/java/org/apache/flume/tools/TestTimestampRoundDownUtil.java * flume-ng-core/src/main/java/org/apache/flume/tools/TimestampRoundDownUtil.java * flume-ng-core/src/test/java/org/apache/flume/formatter/output/TestBucketPath.java * flume-ng-core/src/main/java/org/apache/flume/formatter/output/BucketPath.java > HDFS Sink does not use configured timezone for rounding > --- > > Key: FLUME-2725 > URL: https://issues.apache.org/jira/browse/FLUME-2725 > Project: Flume > Issue Type: Bug > Components: Sinks+Sources >Reporter: Eric Czech >Assignee: Denes Arvay >Priority: Minor > Fix For: v1.7.0 > > Attachments: FLUME-2725-2.patch, FLUME-2725-3.patch, FLUME-2725.patch > > > When a BucketPath used by an HDFS sink is configured to run with some > roundUnit and roundValue > 1 (e.g. 6 hours), the "roundDown" function used by > BucketPath does not actually round the date correctly. > That function calls TimestampRoundDownUtil which creates a Calendar instance > using the *local* timezone to truncate a unix timestamp rather than the > TimeZone that the sink was configured to convert dates to paths with (and > that timezone is already available in the BucketPath class but it just isn't > passed to TimestampRoundDownUtil). > The net effect of this is that if a flume jvm is running on a system with an > EST clock while trying to write, say, 6 hour directories in UTC time, the > directories are written with the hours 04, 10, 16, 22 rather than 00, 06, 12, > 18 like you would expect. > I found a workaround for this by passing > "-Duser.timezone=" as a system property, but I wanted to > create a ticket for this since it seems like it would be very minimal effort > to carry that configured timezone down into the rounding utility as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2950) ReliableSpoolingFileEventReader.rollCurrentFile is broken
[ https://issues.apache.org/jira/browse/FLUME-2950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15370012#comment-15370012 ] Hudson commented on FLUME-2950: --- FAILURE: Integrated in Flume-trunk-hbase-1 #176 (See [https://builds.apache.org/job/Flume-trunk-hbase-1/176/]) FLUME-2950. ReliableSpoolingFileEventReader.rollCurrentFile is broken (mpercy: [http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git=commit=ba64b12670f7b2c9ce079f8898f4601689a51def]) * flume-ng-core/src/main/java/org/apache/flume/client/avro/ReliableSpoolingFileEventReader.java > ReliableSpoolingFileEventReader.rollCurrentFile is broken > - > > Key: FLUME-2950 > URL: https://issues.apache.org/jira/browse/FLUME-2950 > Project: Flume > Issue Type: Bug >Affects Versions: v1.7.0 >Reporter: Lior Zeno >Assignee: Lior Zeno > Fix For: v1.7.0 > > Attachments: FLUME-2950-0.patch > > > FLUME-2939 introduced Java 7 API (nio.file) to > ReliableSpoolingFileEventReader. However, the committed patch breaks > rollCurrentFile on Windows machines. > File rename might be not atomic, in such cases, and under the assumption that > the rolled file and the current file store the same data, Flume only logs a > warning and does not assume that the user broke the contract that the > spooling directory source demands. > In order to determine whether the files store the same data, we used the > com.google.common.io.Files.equal method. However, FLUME-2939 uses > java.nio.file.Files.isSameFile which has different semantics. > Consequently, this relaxation does not hold anymore. In addition, > org.apache.flume.client.avro.TestSpoolingFileLineReader.testDestinationExistsAndSameFileWindows > fails. > I attach a patch that brings back the old check for Files.equal. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2936) KafkaSource tests arbitrarily fail
[
https://issues.apache.org/jira/browse/FLUME-2936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15369371#comment-15369371
]
Hudson commented on FLUME-2936:
---
UNSTABLE: Integrated in Flume-trunk-hbase-1 #175 (See
[https://builds.apache.org/job/Flume-trunk-hbase-1/175/])
FLUME-2936. KafkaSource tests: Use loopback instead of host address (mpercy:
[http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git=commit=209c1e2548080525919fe78a1a61cc8fc7ab306f])
*
flume-ng-sources/flume-kafka-source/src/test/java/org/apache/flume/source/kafka/KafkaSourceEmbeddedKafka.java
> KafkaSource tests arbitrarily fail
> ---
>
> Key: FLUME-2936
> URL: https://issues.apache.org/jira/browse/FLUME-2936
> Project: Flume
> Issue Type: Bug
> Components: Test
>Affects Versions: v1.7.0
>Reporter: Lior Zeno
>Assignee: Lior Zeno
> Fix For: v1.7.0
>
> Attachments: FLUME-2936-0.patch
>
>
> KafkaSource tests are flaky, and tend to fail. I investigated this issue and
> discovered the following issues:
> * KafkaSource hangs in doStart at line 413:
> {code:title=KafkaSourceEmbeddedKafka.java|borderStyle=solid}
> it = consumer.poll(1000).iterator();
> {code}
> I fixed that by simply changing HOST in KafkaSourceEmbeddedKafka to localhost.
> * Out of memory exception. I fixed that by properly configuring MAVEN_OPTS,
> as stated in the documentation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLUME-2949) Flume fails to build on Windows
[ https://issues.apache.org/jira/browse/FLUME-2949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15369353#comment-15369353 ] Hudson commented on FLUME-2949: --- FAILURE: Integrated in Flume-trunk-hbase-1 #174 (See [https://builds.apache.org/job/Flume-trunk-hbase-1/174/]) FLUME-2949. checkstyle: Make suppressions compatible with Windows (mpercy: [http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git=commit=188c3104ab6030c40d652595a2274527a4ad4105]) * flume-checkstyle/src/main/resources/flume/checkstyle-suppressions.xml > Flume fails to build on Windows > --- > > Key: FLUME-2949 > URL: https://issues.apache.org/jira/browse/FLUME-2949 > Project: Flume > Issue Type: Bug >Affects Versions: v1.7.0 >Reporter: Lior Zeno >Assignee: Lior Zeno > Fix For: v1.7.0 > > Attachments: FLUME-2949-0.patch > > > Now that style violations fail Flume build process, Flume fails to build on > Windows. > The reason for that is that file paths in checkstyle-suppressions.xml are > written with non-portable separators. Switching to "[/\\]" as a path > separator, should fix this issue. > References: > * http://checkstyle.sourceforge.net/config_filters.html#SuppressionFilter > * > http://checkstyle.2069334.n4.nabble.com/suppression-filter-regex-td2070198.html -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2948) Docs: Fixed parameters on Replicating Channel Selector documentation example
[ https://issues.apache.org/jira/browse/FLUME-2948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15368879#comment-15368879 ] Hudson commented on FLUME-2948: --- FAILURE: Integrated in Flume-trunk-hbase-1 #173 (See [https://builds.apache.org/job/Flume-trunk-hbase-1/173/]) FLUME-2948. docs: Fix parameters on Replicating Channel Selector example (mpercy: [http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git=commit=19664c3c559c7d0360f7bb5a64eff88b8f25a80d]) * flume-ng-doc/sphinx/FlumeUserGuide.rst > Docs: Fixed parameters on Replicating Channel Selector documentation example > > > Key: FLUME-2948 > URL: https://issues.apache.org/jira/browse/FLUME-2948 > Project: Flume > Issue Type: Bug > Components: Docs > Environment: Documentation page >Reporter: niccolo becchi >Assignee: niccolo becchi >Priority: Trivial > Fix For: v1.7.0 > > > On the documentation page, in the section: > Flume Channel Selectors => Replicating Channel Selector > (https://flume.apache.org/FlumeUserGuide.html#replicating-channel-selector-default) > some of the parameters have a typo and, as it is, it doesn't work. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2939) Upgrade recursive SpoolDir to use Java7 features
[ https://issues.apache.org/jira/browse/FLUME-2939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15368880#comment-15368880 ] Hudson commented on FLUME-2939: --- FAILURE: Integrated in Flume-trunk-hbase-1 #173 (See [https://builds.apache.org/job/Flume-trunk-hbase-1/173/]) FLUME-2939. Update recursive SpoolDir source to use Java 7 APIs (mpercy: [http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git=commit=7013708baddc8ed7d861797d1fd8280a94b6025c]) * flume-ng-core/src/main/java/org/apache/flume/client/avro/ReliableSpoolingFileEventReader.java > Upgrade recursive SpoolDir to use Java7 features > > > Key: FLUME-2939 > URL: https://issues.apache.org/jira/browse/FLUME-2939 > Project: Flume > Issue Type: Bug > Components: Sinks+Sources >Reporter: Bessenyei Balázs Donát >Assignee: Bessenyei Balázs Donát >Priority: Minor > Fix For: v1.7.0 > > > FLUME-1899 was committed without the DirectoryStream API > This ticket is for "upgrading" the patch provided to use Java 7 features as > the Java version for Flume has been recently bumped. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2941) Integrate checkstyle for test classes
[ https://issues.apache.org/jira/browse/FLUME-2941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15368809#comment-15368809 ] Hudson commented on FLUME-2941: --- FAILURE: Integrated in Flume-trunk-hbase-1 #172 (See [https://builds.apache.org/job/Flume-trunk-hbase-1/172/]) FLUME-2941. Integrate checkstyle for test classes (mpercy: [http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git=commit=cfbf1156858af9ae26975fefc94594d91c8cd3f4]) * flume-ng-sdk/src/test/java/org/apache/flume/api/RpcTestUtils.java * flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelConcurrency.java * flume-ng-core/src/test/java/org/apache/flume/serialization/TestDurablePositionTracker.java * flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelTransaction.java * flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java * flume-checkstyle/src/main/resources/flume/checkstyle.xml * flume-ng-sinks/flume-ng-elasticsearch-sink/src/test/java/org/apache/flume/sink/elasticsearch/client/RoundRobinListTest.java * flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/CountingSinkRunner.java * flume-ng-sources/flume-kafka-source/src/test/java/org/apache/flume/source/kafka/TestKafkaSource.java * flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/encryption/CipherProviderTestSuite.java * flume-ng-core/src/test/java/org/apache/flume/source/TestSyslogUtils.java * flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestEventQueueBackingStoreFactory.java * flume-tools/src/test/java/org/apache/flume/tools/TestFileChannelIntegrityTool.java * flume-ng-sinks/flume-ng-elasticsearch-sink/src/test/java/org/apache/flume/sink/elasticsearch/TestElasticSearchLogStashEventSerializer.java * flume-ng-channels/flume-jdbc-channel/src/test/java/org/apache/flume/channel/jdbc/TestDerbySchemaHandlerQueries.java * flume-ng-auth/src/test/java/org/apache/flume/auth/TestFlumeAuthenticator.java * flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/encryption/TestAESCTRNoPaddingProvider.java * flume-ng-core/src/test/java/org/apache/flume/channel/TestChannelProcessor.java * flume-ng-core/src/test/java/org/apache/flume/source/TestExecSource.java * flume-ng-sinks/flume-ng-kafka-sink/src/test/java/org/apache/flume/sink/kafka/util/ZooKeeperLocal.java * flume-ng-embedded-agent/src/test/java/org/apache/flume/agent/embedded/TestEmbeddedAgent.java * flume-ng-sdk/src/test/java/org/apache/flume/api/TestNettyAvroRpcClient.java * flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannelRestart.java * flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/encryption/TestJCEFileKeyProvider.java * flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestUtils.java * flume-ng-sdk/src/test/java/org/apache/flume/api/TestLoadBalancingRpcClient.java * flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannelRollback.java * flume-ng-core/src/test/java/org/apache/flume/interceptor/TestRegexExtractorInterceptorPassThroughSerializer.java * flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannelFormatRegression.java * flume-ng-core/src/test/java/org/apache/flume/tools/TestVersionInfo.java * flume-ng-node/src/test/java/org/apache/flume/node/TestPropertiesFileConfigurationProvider.java * flume-ng-sinks/flume-ng-elasticsearch-sink/src/test/java/org/apache/flume/sink/elasticsearch/TestElasticSearchSink.java * flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/encryption/TestFileChannelEncryption.java * flume-ng-node/src/test/java/org/apache/flume/node/TestAbstractZooKeeperConfigurationProvider.java * flume-ng-channels/flume-spillable-memory-channel/src/test/java/org/apache/flume/channel/TestSpillableMemoryChannel.java * flume-ng-sources/flume-taildir-source/src/test/java/org/apache/flume/source/taildir/TestTaildirEventReader.java * flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestIntegration.java * flume-ng-tests/src/test/java/org/apache/flume/test/agent/TestFileChannel.java * flume-ng-core/src/test/java/org/apache/flume/serialization/TestFlumeEventAvroEventSerializer.java * flume-ng-channels/flume-jdbc-channel/src/test/java/org/apache/flume/channel/jdbc/MockEventUtils.java * flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/MockDataStream.java * flume-ng-core/src/test/java/org/apache/flume/formatter/output/TestBucketPath.java * flume-ng-sinks/flume-hive-sink/src/test/java/org/apache/flume/sink/hive/TestUtil.java * flume-ng-sdk/src/test/java/org/apache/flume/api/TestThriftRpcClient.java *
[jira] [Commented] (FLUME-2937) Integrate checkstyle for non-test classes
[ https://issues.apache.org/jira/browse/FLUME-2937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15356482#comment-15356482 ] Hudson commented on FLUME-2937: --- FAILURE: Integrated in Flume-trunk-hbase-1 #170 (See [https://builds.apache.org/job/Flume-trunk-hbase-1/170/]) FLUME-2937. Integrate checkstyle for non-test classes (mpercy: [http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git=commit=2252fb1938a4fd578f88c64eb444c74777c46212]) * flume-ng-sdk/src/main/java/org/apache/flume/api/FailoverRpcClient.java * flume-ng-core/src/main/java/org/apache/flume/instrumentation/ChannelCounter.java * flume-ng-auth/src/main/java/org/apache/flume/auth/UGIExecutor.java * flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/TransactionEventRecord.java * flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Pair.java * flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/ElasticSearchIndexRequestBuilderFactory.java * flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/EventQueueBackingStoreFileV2.java * flume-ng-sources/flume-taildir-source/src/main/java/org/apache/flume/source/taildir/TaildirSourceConfigurationConstants.java * flume-ng-core/src/main/java/org/apache/flume/sink/DefaultSinkProcessor.java * flume-ng-sinks/flume-ng-hbase-sink/src/main/java/org/apache/flume/sink/hbase/AsyncHBaseSink.java * flume-ng-core/src/main/java/org/apache/flume/channel/MultiplexingChannelSelector.java * flume-ng-core/src/main/java/org/apache/flume/sink/LoggerSink.java * flume-ng-core/src/main/java/org/apache/flume/Clock.java * flume-ng-sinks/flume-hive-sink/src/main/java/org/apache/flume/sink/hive/HiveEventSerializer.java * flume-ng-core/src/main/java/org/apache/flume/sink/AbstractSinkProcessor.java * flume-ng-core/src/main/java/org/apache/flume/channel/AbstractChannelSelector.java * flume-ng-core/src/main/java/org/apache/flume/source/AbstractSource.java * flume-ng-sinks/flume-dataset-sink/src/main/java/org/apache/flume/sink/kite/parser/EntityParserFactory.java * flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/DerbySchemaHandler.java * flume-ng-core/src/main/java/org/apache/flume/lifecycle/LifecycleSupervisor.java * flume-ng-core/src/main/java/org/apache/flume/source/DefaultSourceFactory.java * flume-ng-core/src/main/java/org/apache/flume/source/SyslogTcpSource.java * flume-ng-sinks/flume-hive-sink/src/main/java/org/apache/flume/sink/hive/HiveSink.java * flume-ng-core/src/main/java/org/apache/flume/instrumentation/MonitoringType.java * flume-ng-sources/flume-jms-source/src/main/java/org/apache/flume/source/jms/JMSMessageConsumerFactory.java * flume-ng-sinks/flume-ng-morphline-solr-sink/src/main/java/org/apache/flume/sink/solr/morphline/MorphlineInterceptor.java * flume-ng-core/src/main/java/org/apache/flume/source/StressSource.java * flume-tools/src/main/java/org/apache/flume/tools/FileChannelIntegrityTool.java * flume-ng-core/src/main/java/org/apache/flume/interceptor/RegexFilteringInterceptor.java * flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java * flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/encryption/AESCTRNoPaddingProvider.java * flume-ng-sources/flume-kafka-source/src/main/java/org/apache/flume/source/kafka/KafkaSourceConstants.java * flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/AbstractHDFSWriter.java * flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogUtils.java * flume-ng-core/src/main/java/org/apache/flume/instrumentation/http/HTTPMetricsServer.java * flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java * flume-ng-sinks/flume-ng-hbase-sink/src/main/java/org/apache/flume/sink/hbase/SimpleRowKeyGenerator.java * flume-ng-core/src/main/java/org/apache/flume/annotations/InterfaceStability.java * flume-ng-core/src/main/java/org/apache/flume/formatter/output/DefaultPathManager.java * flume-ng-clients/flume-ng-log4jappender/src/main/java/org/apache/flume/clients/log4jappender/Log4jAvroHeaders.java * flume-ng-core/src/main/java/org/apache/flume/SourceFactory.java * flume-ng-sdk/src/main/java/org/apache/flume/event/EventBuilder.java * flume-ng-core/src/main/java/org/apache/flume/source/MultiportSyslogTCPSource.java * flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Commit.java * flume-ng-sdk/src/main/java/org/apache/flume/api/AbstractRpcClient.java * flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/WritableUtils.java * flume-ng-embedded-agent/src/main/java/org/apache/flume/agent/embedded/EmbeddedAgent.java * flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/ReplayHandler.java *
[jira] [Commented] (FLUME-1899) Make SpoolDir work with Sub-Directories
[ https://issues.apache.org/jira/browse/FLUME-1899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15353802#comment-15353802 ] Hudson commented on FLUME-1899: --- FAILURE: Integrated in Flume-trunk-hbase-1 #169 (See [https://builds.apache.org/job/Flume-trunk-hbase-1/169/]) FLUME-1899. Make SpoolDir work with subdirectories (mpercy: [http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git=commit=2fe393898f420d100117ca277cce198858e2c24f]) * flume-ng-core/src/main/java/org/apache/flume/client/avro/ReliableSpoolingFileEventReader.java * flume-ng-core/src/main/java/org/apache/flume/source/SpoolDirectorySourceConfigurationConstants.java * flume-ng-core/src/main/java/org/apache/flume/source/SpoolDirectorySource.java * flume-ng-core/src/test/java/org/apache/flume/source/TestSpoolDirectorySource.java * flume-ng-doc/sphinx/FlumeUserGuide.rst > Make SpoolDir work with Sub-Directories > --- > > Key: FLUME-1899 > URL: https://issues.apache.org/jira/browse/FLUME-1899 > Project: Flume > Issue Type: Improvement > Components: Sinks+Sources >Affects Versions: v1.4.0, v1.3.1 >Reporter: Tzur Turkenitz >Assignee: Bessenyei Balázs Donát >Priority: Minor > Labels: Sources, SpoolDir > Fix For: v1.7.0 > > Attachments: FLUM-1899-1.patch, FLUME-1899-2.patch, FLUME-1899-3.patch > > > SpoolrDir currently monitors a directory and can not handle sub-directories. > This JIRA is to make SpoolDir able to walk down a source directory and > monitor new files. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2567) Remove unneeded repository declarations in pom.xml
[ https://issues.apache.org/jira/browse/FLUME-2567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15352031#comment-15352031 ] Hudson commented on FLUME-2567: --- FAILURE: Integrated in Flume-trunk-hbase-1 #168 (See [https://builds.apache.org/job/Flume-trunk-hbase-1/168/]) FLUME-2567. Remove external Maven repositories (mpercy: [http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git=commit=4eacba1930c3abfddf4fb50c5a272798708df811]) * flume-ng-sinks/flume-ng-morphline-solr-sink/pom.xml * pom.xml * flume-ng-sinks/flume-dataset-sink/pom.xml > Remove unneeded repository declarations in pom.xml > -- > > Key: FLUME-2567 > URL: https://issues.apache.org/jira/browse/FLUME-2567 > Project: Flume > Issue Type: Bug >Reporter: Santiago M. Mola >Assignee: Santiago M. Mola >Priority: Minor > Fix For: v1.7.0 > > Attachments: FLUME-2567-0.patch > > > pom.xml contains multiple repository declarations that do not seem to be > needed (jboss, maven central, cloudera). Actually, it seems to me that none > of these is needed. > Can we remove all repository declarations and let it use just the default > (Maven Central)? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2935) Bump java target version to 1.7
[ https://issues.apache.org/jira/browse/FLUME-2935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15350271#comment-15350271 ] Hudson commented on FLUME-2935: --- FAILURE: Integrated in Flume-trunk-hbase-1 #166 (See [https://builds.apache.org/job/Flume-trunk-hbase-1/166/]) FLUME-2935. Bump java target version to 1.7 (mpercy: [http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git=commit=2f731f0aaeb490ab72400d8c83074daac840ac44]) * flume-ng-doc/sphinx/FlumeUserGuide.rst * README * flume-ng-sinks/flume-ng-elasticsearch-sink/pom.xml * flume-ng-sinks/flume-ng-kafka-sink/src/test/java/org/apache/flume/sink/kafka/util/KafkaLocal.java * pom.xml * README.textile > Bump java target version to 1.7 > --- > > Key: FLUME-2935 > URL: https://issues.apache.org/jira/browse/FLUME-2935 > Project: Flume > Issue Type: Task >Affects Versions: v1.7.0 >Reporter: Lior Zeno >Assignee: Lior Zeno > Fix For: v1.7.0 > > Attachments: FLUME-2395-0.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2934) Document new cachePatternMatching option for TaildirSource
[ https://issues.apache.org/jira/browse/FLUME-2934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15348972#comment-15348972 ] Hudson commented on FLUME-2934: --- ABORTED: Integrated in Flume-trunk-hbase-1 #165 (See [https://builds.apache.org/job/Flume-trunk-hbase-1/165/]) FLUME-2934. Document new cachePatternMatching option for TaildirSource (mpercy: [http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git=commit=4a7db4a363a7ed7391e11685bb6a0f47cbfa2328]) * flume-ng-doc/sphinx/FlumeUserGuide.rst > Document new cachePatternMatching option for TaildirSource > -- > > Key: FLUME-2934 > URL: https://issues.apache.org/jira/browse/FLUME-2934 > Project: Flume > Issue Type: Documentation > Components: Docs >Affects Versions: v1.6.0 >Reporter: Attila Simon >Assignee: Attila Simon > Labels: documentation > Fix For: v1.7.0 > > Attachments: FLUME-2934.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2918) TaildirSource is underperforming with huge parent directories
[ https://issues.apache.org/jira/browse/FLUME-2918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15339297#comment-15339297 ] Hudson commented on FLUME-2918: --- FAILURE: Integrated in Flume-trunk-hbase-1 #164 (See [https://builds.apache.org/job/Flume-trunk-hbase-1/164/]) FLUME-2918. Speed up TaildirSource on directories with many files (mpercy: [http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git=commit=7d1e683fbd7d261fff9fcf17ad78fd8469c64905]) * flume-ng-sources/flume-taildir-source/src/main/java/org/apache/flume/source/taildir/TailFile.java * flume-ng-sources/flume-taildir-source/src/test/java/org/apache/flume/source/taildir/TestTaildirMatcher.java * flume-ng-sources/flume-taildir-source/src/main/java/org/apache/flume/source/taildir/TaildirMatcher.java * flume-ng-sources/flume-taildir-source/src/main/java/org/apache/flume/source/taildir/TaildirSourceConfigurationConstants.java * flume-ng-sources/flume-taildir-source/src/main/java/org/apache/flume/source/taildir/ReliableTaildirEventReader.java * flume-ng-sources/flume-taildir-source/src/main/java/org/apache/flume/source/taildir/TaildirSource.java * flume-ng-sources/flume-taildir-source/src/test/java/org/apache/flume/source/taildir/TestTaildirSource.java > TaildirSource is underperforming with huge parent directories > - > > Key: FLUME-2918 > URL: https://issues.apache.org/jira/browse/FLUME-2918 > Project: Flume > Issue Type: Improvement > Components: Sinks+Sources >Affects Versions: v1.6.0 >Reporter: Attila Simon >Assignee: Attila Simon > Labels: performance > Fix For: v1.7.0 > > Attachments: FLUME-2918-2.patch, PerfHugeDir.java, perftest.png, > profiling_after.png, profiling_before.png, test.csv > > > TailDir source cause high cpu utilization, when large amount of file is > sitting in the target directory. File pattern matches only a single file, but > the parent directory contains about 50,000 other file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2923) Bump AsyncHBase version
[ https://issues.apache.org/jira/browse/FLUME-2923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15337256#comment-15337256 ] Hudson commented on FLUME-2923: --- FAILURE: Integrated in Flume-trunk-hbase-1 #163 (See [https://builds.apache.org/job/Flume-trunk-hbase-1/163/]) FLUME-2923. Bump asynchbase version to 1.7.0 (mpercy: [http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git=commit=1b9e589152f0093b6ef7cea580c00ff52944d2cf]) * pom.xml > Bump AsyncHBase version > --- > > Key: FLUME-2923 > URL: https://issues.apache.org/jira/browse/FLUME-2923 > Project: Flume > Issue Type: Bug > Components: Sinks+Sources >Affects Versions: v1.7.0 >Reporter: Bessenyei Balázs Donát >Assignee: Bessenyei Balázs Donát >Priority: Minor > Fix For: v1.7.0 > > Attachments: FLUME-2923-2.patch, FLUME-2923-3.patch, FLUME-2923.patch > > > The current version of AsyncHBase ( http://opentsdb.github.io/asynchbase/ ) > is outdated. > Reason for the upgrade would be many bug fixes and some new public APIs. (See > https://github.com/OpenTSDB/asynchbase/blob/master/NEWS ) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2910) AsyncHBaseSink - Failure callbacks should log the exception that caused them
[ https://issues.apache.org/jira/browse/FLUME-2910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15325277#comment-15325277 ] Hudson commented on FLUME-2910: --- FAILURE: Integrated in Flume-trunk-hbase-1 #161 (See [https://builds.apache.org/job/Flume-trunk-hbase-1/161/]) FLUME-2910. AsyncHBaseSink: Failure callbacks should log the exception (mpercy: [http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git=commit=ee4999bc23f42bc300ed87b0d46fd96418d6a185]) * flume-ng-sinks/flume-ng-hbase-sink/src/main/java/org/apache/flume/sink/hbase/AsyncHBaseSink.java > AsyncHBaseSink - Failure callbacks should log the exception that caused them > > > Key: FLUME-2910 > URL: https://issues.apache.org/jira/browse/FLUME-2910 > Project: Flume > Issue Type: Improvement > Components: Sinks+Sources >Affects Versions: v1.6.0 >Reporter: Abraham Fine >Assignee: Abraham Fine > Fix For: v1.7.0 > > Attachments: FLUME-2910.patch > > > Failure callbacks in the AsyncHBaseSink currently do not log the exception > that causes them to be called, this should be fixed -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2915) The kafka channel using new APIs will be stuck when the sink is avro sink
[
https://issues.apache.org/jira/browse/FLUME-2915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15324600#comment-15324600
]
Hudson commented on FLUME-2915:
---
FAILURE: Integrated in Flume-trunk-hbase-1 #160 (See
[https://builds.apache.org/job/Flume-trunk-hbase-1/160/])
FLUME-2915: The kafka channel using new APIs will be stuck when the (jarcec:
[http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git=commit=a619cc15748d76357f84bd3d0fa0a3e496ee61a1])
*
flume-ng-channels/flume-kafka-channel/src/main/java/org/apache/flume/channel/kafka/KafkaChannel.java
*
flume-ng-channels/flume-kafka-channel/src/test/java/org/apache/flume/channel/kafka/TestKafkaChannel.java
> The kafka channel using new APIs will be stuck when the sink is avro sink
> -
>
> Key: FLUME-2915
> URL: https://issues.apache.org/jira/browse/FLUME-2915
> Project: Flume
> Issue Type: Bug
> Components: Channel
>Affects Versions: v1.7.0
>Reporter: Julian Zhang
>Assignee: Jeff Holoman
> Fix For: v1.7.0
>
> Attachments: FLUME-2915.patch, FLUME-2915v2.patch
>
>
> The avro sink was stuck when I using the kafka channel which using the new
> APIs.
> After couple of hours I found the issue at KafkaChannel.java#L384:
> e.getHeaders().put(KEY_HEADER, record.key());
> and change it to:
> if (record.key() != null) {
> e.getHeaders().put(KEY_HEADER, record.key());
> }
> The reason is:
> record.key() could be null if the user didn't set it. And the avro serialize
> the event will throw a NullPointerException.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLUME-2913) Flume classpath too long
[ https://issues.apache.org/jira/browse/FLUME-2913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15304762#comment-15304762 ] Hudson commented on FLUME-2913: --- FAILURE: Integrated in Flume-trunk-hbase-1 #159 (See [https://builds.apache.org/job/Flume-trunk-hbase-1/159/]) FLUME-2913. Don't strip SLF4J from imported classpaths (mpercy: [http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git=commit=f4f15bee1e4ba109c6a21f639668911a13241014]) * bin/flume-ng > Flume classpath too long > > > Key: FLUME-2913 > URL: https://issues.apache.org/jira/browse/FLUME-2913 > Project: Flume > Issue Type: Bug >Reporter: Bessenyei Balázs Donát >Assignee: Bessenyei Balázs Donát > Fix For: v1.7.0 > > Attachments: FLUME-2913.patch > > > Starting Flume generates the FLUME_CLASSPATH variable dynamically, adding > jars one by one. > In some cases, the classpath can grow too long, sometimes exceeding ARG_MAX, > maximum length of arguments for a new process ( > http://www.in-ulm.de/~mascheck/various/argmax/ ) > Thus, being unable to start Flume: > flume-ng/bin/flume-ng: line 239: /usr/java/jdk1.7.0_67/bin/java: Argument > list too long -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2909) Bump Rat version
[ https://issues.apache.org/jira/browse/FLUME-2909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15289628#comment-15289628 ] Hudson commented on FLUME-2909: --- FAILURE: Integrated in Flume-trunk-hbase-1 #158 (See [https://builds.apache.org/job/Flume-trunk-hbase-1/158/]) FLUME-2909. Upgrade RAT to 0.11 (mpercy: [http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git=commit=37ddd1b27258f2c3095094b79cbd8852478de33d]) * pom.xml > Bump Rat version > > > Key: FLUME-2909 > URL: https://issues.apache.org/jira/browse/FLUME-2909 > Project: Flume > Issue Type: Improvement > Components: Build >Affects Versions: v1.7.0 >Reporter: Bessenyei Balázs Donát >Assignee: Bessenyei Balázs Donát >Priority: Minor > Fix For: v1.7.0 > > Attachments: FLUME-2909.patch > > > The current version of Rat ( http://creadur.apache.org/rat/index.html ) is > outdated. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2908) NetcatSource - SocketChannel not closed when session is broken
[ https://issues.apache.org/jira/browse/FLUME-2908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15272747#comment-15272747 ] Hudson commented on FLUME-2908: --- FAILURE: Integrated in Flume-trunk-hbase-1 #157 (See [https://builds.apache.org/job/Flume-trunk-hbase-1/157/]) FLUME-2908: NetcatSource - SocketChannel not closed when session is (jarcec: [http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git=commit=46ae5accd03e5b8b18d3e1fb52761c1df4e51e5a]) * flume-ng-core/src/main/java/org/apache/flume/source/NetcatSource.java > NetcatSource - SocketChannel not closed when session is broken > -- > > Key: FLUME-2908 > URL: https://issues.apache.org/jira/browse/FLUME-2908 > Project: Flume > Issue Type: Bug > Components: Sinks+Sources >Affects Versions: v1.3.0 >Reporter: Grant Henke >Assignee: Grant Henke > Fix For: v1.7.0 > > Attachments: FLUME-2908-1.patch, FLUME-2908.patch > > > When the NetcatSource socketChannel receives an IOException due to an > abruptly closed session the exception is caught but the socketChannel is > never closed. This results in leaked sockets and ever growing file > descriptors. > Thanks to [~pdvorak] for finding the leak. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2852) Kafka Source/Sink should optionally read/write Flume records
[ https://issues.apache.org/jira/browse/FLUME-2852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15252876#comment-15252876 ] Hudson commented on FLUME-2852: --- FAILURE: Integrated in Flume-trunk-hbase-1 #155 (See [https://builds.apache.org/job/Flume-trunk-hbase-1/155/]) FLUME-2852: Kafka Source/Sink should optionally read/write Flume records (jarcec: [http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git=commit=4eb2a3bb510671fe92cec0ebb61d7e78adc8f526]) * flume-ng-sources/flume-kafka-source/src/main/java/org/apache/flume/source/kafka/KafkaSource.java * flume-ng-sinks/flume-ng-kafka-sink/src/main/java/org/apache/flume/sink/kafka/KafkaSink.java * flume-ng-doc/sphinx/FlumeUserGuide.rst * flume-ng-sinks/flume-ng-kafka-sink/src/test/java/org/apache/flume/sink/kafka/TestConstants.java * flume-ng-sinks/flume-ng-kafka-sink/src/test/java/org/apache/flume/sink/kafka/TestKafkaSink.java * flume-ng-sinks/flume-ng-kafka-sink/src/main/java/org/apache/flume/sink/kafka/KafkaSinkConstants.java * flume-ng-sources/flume-kafka-source/src/main/java/org/apache/flume/source/kafka/KafkaSourceConstants.java * flume-ng-sources/flume-kafka-source/src/test/java/org/apache/flume/source/kafka/TestKafkaSource.java * flume-ng-sources/flume-kafka-source/src/test/java/org/apache/flume/source/kafka/KafkaSourceEmbeddedKafka.java > Kafka Source/Sink should optionally read/write Flume records > > > Key: FLUME-2852 > URL: https://issues.apache.org/jira/browse/FLUME-2852 > Project: Flume > Issue Type: Improvement > Components: Sinks+Sources >Affects Versions: v1.6.0 >Reporter: Tristan Stevens >Assignee: Tristan Stevens > Fix For: v1.7.0 > > Attachments: FLUME-2852.patch, FLUME-2852v4.patch, FLUME-2852v5.patch > > > Currently the Kafka Sink only writes the event body to Kafka rather than an > Avro Datum. This works fine when being used with a Kafka Source, or when > being used with Kafka Channel, however it does mean that any Flume headers > are lost when transported via Kafka. > Request is to implement an equivalent of the Kafka Channel's > parseAsFlumeEvent parameter to the sink/source so that optionally Avro Datums > can be read from and written to Kafka. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2822) Flume-Kafka-Sink with new Producer
[ https://issues.apache.org/jira/browse/FLUME-2822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15216675#comment-15216675 ] Hudson commented on FLUME-2822: --- FAILURE: Integrated in Flume-trunk-hbase-1 #154 (See [https://builds.apache.org/job/Flume-trunk-hbase-1/154/]) FLUME-2822: Flume-Kafka-Sink with new Producer (jarcec: [http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git=commit=7f588e6a158f5d108e39f50a92f8d1d108b12c24]) * flume-ng-sinks/flume-ng-kafka-sink/src/main/java/org/apache/flume/sink/kafka/KafkaSink.java * flume-ng-sinks/flume-ng-kafka-sink/src/test/java/org/apache/flume/sink/kafka/TestKafkaSink.java * flume-ng-sinks/flume-ng-kafka-sink/src/test/java/org/apache/flume/sink/kafka/KafkaSinkUtilTest.java * flume-ng-sinks/flume-ng-kafka-sink/src/main/java/org/apache/flume/sink/kafka/KafkaSinkConstants.java * flume-ng-sinks/flume-ng-kafka-sink/pom.xml * flume-ng-doc/sphinx/FlumeUserGuide.rst * flume-ng-sinks/flume-ng-kafka-sink/src/main/java/org/apache/flume/sink/kafka/KafkaSinkUtil.java > Flume-Kafka-Sink with new Producer > -- > > Key: FLUME-2822 > URL: https://issues.apache.org/jira/browse/FLUME-2822 > Project: Flume > Issue Type: Sub-task >Reporter: Jeff Holoman >Assignee: Jeff Holoman > Fix For: v1.7.0 > > Attachments: FLUME-2822.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
