Jenkins build is unstable: Flume-trunk-hbase-1 #225

2016-10-28 Thread Apache Jenkins Server 
See

[jira] [Commented] (FLUME-2857) Kafka Source/Channel/Sink does not restore default values when live update config

2016-10-28 Thread Hudson (JIRA) 

[ 
https://issues.apache.org/jira/browse/FLUME-2857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15615973#comment-15615973
 ] 

Hudson commented on FLUME-2857:
---

UNSTABLE: Integrated in Jenkins build Flume-trunk-hbase-1 #225 (See 
[https://builds.apache.org/job/Flume-trunk-hbase-1/225/])
FLUME-2857. Make Kafka Source/Channel/Sink restore default values when (bessbd: 
[http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git=commit=14fb4d84fd0e100253ca947bc96810c242e7a82b])
* (edit) 
flume-ng-channels/flume-kafka-channel/src/test/java/org/apache/flume/channel/kafka/TestKafkaChannel.java
* (edit) 
flume-ng-sources/flume-kafka-source/src/test/java/org/apache/flume/source/kafka/TestKafkaSource.java
* (edit) 
flume-ng-sources/flume-kafka-source/src/main/java/org/apache/flume/source/kafka/KafkaSource.java
* (edit) 
flume-ng-sinks/flume-ng-kafka-sink/src/test/java/org/apache/flume/sink/kafka/TestKafkaSink.java
* (edit) 
flume-ng-channels/flume-kafka-channel/src/main/java/org/apache/flume/channel/kafka/KafkaChannel.java
* (edit) 
flume-ng-sinks/flume-ng-kafka-sink/src/main/java/org/apache/flume/sink/kafka/KafkaSink.java


> Kafka Source/Channel/Sink does not restore default values when live update 
> config
> -
>
> Key: FLUME-2857
> URL: https://issues.apache.org/jira/browse/FLUME-2857
> Project: Flume
>  Issue Type: Bug
>  Components: Channel
>Reporter: Tristan Stevens
>Assignee: Tristan Stevens
> Fix For: v1.8.0
>
> Attachments: FLUME-2857.patch
>
>
> Been using the following config:
> {noformat}
> tier1.channels.channel1.type = org.apache.flume.channel.kafka.KafkaChannel
> tier1.channels.channel1.capacity = 1
> tier1.channels.channel1.transactionCapacity = 1
> tier1.channels.channel1.brokerList = 
> 10.0.0.64:9092,10.0.0.65:9092,10.0.0.66:9092
> tier1.channels.channel1.topic = flume.aggregator.channel
> tier1.channels.channel1.zookeeperConnect = 10.0.0.64:2181
> tier1.channels.channel1.kafka.producer.type=async
> tier1.channels.channel1.kafka.batch.num.messages=200
> {noformat}
> If I remove the producer.type and batch.num.messages (i.e. restore them to 
> default values) when doing a live update config the new values do not take 
> effect.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flume pull request #83: FLUME-2812: Exception in thread "SinkRunner-PollingR...

2016-10-28 Thread adenes 
GitHub user adenes opened a pull request:

https://github.com/apache/flume/pull/83

FLUME-2812: Exception in thread 
"SinkRunner-PollingRunner-DefaultSinkProcessor" java.lang.Error: Maximum permit 
count exceeded

Fix semaphore leak causing `java.lang.Error: Maximum permit count exceeded` 
in `MemoryChannel`

```java
Exception in thread "SinkRunner-PollingRunner-DefaultSinkProcessor" 
java.lang.Error: Maximum permit count exceeded
at 
java.util.concurrent.Semaphore$Sync.tryReleaseShared(Semaphore.java:192)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.releaseShared(AbstractQueuedSynchronizer.java:1341)
at java.util.concurrent.Semaphore.release(Semaphore.java:609)
at 
org.apache.flume.channel.MemoryChannel$MemoryTransaction.doCommit(MemoryChannel.java:147)
at 
org.apache.flume.channel.BasicTransactionSemantics.commit(BasicTransactionSemantics.java:151)
at 
org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:379)
at 
org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
at java.lang.Thread.run(Thread.java:745)
```

for more details and explanation see my comment on 
[FLUME-2812](https://issues.apache.org/jira/browse/FLUME-2812)

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/adenes/flume FLUME-2812

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flume/pull/83.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #83


commit 538837d5c035461c9b3073a2ce89da375aa139f3
Author: Denes Arvay 
Date:   2016-10-28T15:38:33Z

FLUME-2812: Exception in thread 
"SinkRunner-PollingRunner-DefaultSinkProcessor" java.lang.Error: Maximum permit 
count exceeded

Change-Id: Ib6761364abefca91ca50d80c073637766bfd93ca




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLUME-2812) Exception in thread "SinkRunner-PollingRunner-DefaultSinkProcessor" java.lang.Error: Maximum permit count exceeded

2016-10-28 Thread Denes Arvay (JIRA) 

[ 
https://issues.apache.org/jira/browse/FLUME-2812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15615773#comment-15615773
 ] 

Denes Arvay commented on FLUME-2812:


I have investigated this issue and the cause is that 
{{bytesRemaining.release(putByteCounter)}} gets called in 
{{MemoryTransaction.doRollback}} 
(https://github.com/apache/flume/blob/trunk/flume-ng-core/src/main/java/org/apache/flume/channel/MemoryChannel.java#L174)
 while acquire is called only in {{doCommit}}. This results in semaphore leak 
and the number of permits in the semaphore eventually exceeds 
{{Integer.MAX_VALUE}} and {{Semaphore.release()}} throws this error.
I think this bug was introduced unintentionally when the {{bytesRemaining}} 
semaphore handling was refactored and moved to {{doCommit}} from {{doPut}} in 
FLUME-2233.
If acquire is called in the {{doPut}} then release should happen in 
{{doRollback}} but since it has been moved to {{doCommit}} the release should 
have been deleted.
GitHub Pull Request for this ticket: https://github.com/apache/flume/pull/83

> Exception in thread "SinkRunner-PollingRunner-DefaultSinkProcessor" 
> java.lang.Error: Maximum permit count exceeded
> --
>
> Key: FLUME-2812
> URL: https://issues.apache.org/jira/browse/FLUME-2812
> Project: Flume
>  Issue Type: Bug
>  Components: Channel, Sinks+Sources
>Affects Versions: v1.6.0
> Environment: **OS INFO**
> CentOS release 6.6 (Final)
> Kernel \r on an \m
> **JAVA INFO**
> java version "1.8.0_40"
> Java(TM) SE Runtime Environment (build 1.8.0_40-b26)
> Java HotSpot(TM) 64-Bit Server VM (build 25.40-b25, mixed mode)
>Reporter: Rollin Crittendon
>Assignee: Denes Arvay
>Priority: Critical
>
> We are finding that around after an hour or so of heavy processing of Flume 
> data in an agent we are getting the following exception.  This is after 
> processing about 5-7 k lines/second during that time.
> The configuration of this agent is using a Kafka source, the one that comes 
> with 1.6.0. 
> It is also using a Memory channel, and a Thrift sink.
> ===
> Exception in thread "SinkRunner-PollingRunner-DefaultSinkProcessor" 
> java.lang.Error: Maximum permit count exceeded
>   at 
> java.util.concurrent.Semaphore$Sync.tryReleaseShared(Semaphore.java:192)
>   at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer.releaseShared(AbstractQueuedSynchronizer.java:1341)
>   at java.util.concurrent.Semaphore.release(Semaphore.java:609)
>   at 
> org.apache.flume.channel.MemoryChannel$MemoryTransaction.doCommit(MemoryChannel.java:147)
>   at 
> org.apache.flume.channel.BasicTransactionSemantics.commit(BasicTransactionSemantics.java:151)
>   at 
> org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:379)
>   at 
> org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
>   at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>   at java.lang.Thread.run(Thread.java:745)
> ===
> The above error is from standard error when running the Flume agent.  The 
> effect is that the "SinkRunner-PollingRunner-DefaultSinkProcessor" thread 
> disappears from the agent, this can be seen on a JMX console.
> For us, this means that the Flume agent needs to get restarted.  It is an 
> error that is terminal in that instance of the Java process due to the thread 
> disappearing as a result.
> It sounds like something in JDK 7+ got stricter?!



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLUME-2812) Exception in thread "SinkRunner-PollingRunner-DefaultSinkProcessor" java.lang.Error: Maximum permit count exceeded

2016-10-28 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/FLUME-2812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15615772#comment-15615772
 ] 

ASF GitHub Bot commented on FLUME-2812:
---

GitHub user adenes opened a pull request:

https://github.com/apache/flume/pull/83

FLUME-2812: Exception in thread 
"SinkRunner-PollingRunner-DefaultSinkProcessor" java.lang.Error: Maximum permit 
count exceeded

Fix semaphore leak causing `java.lang.Error: Maximum permit count exceeded` 
in `MemoryChannel`

```java
Exception in thread "SinkRunner-PollingRunner-DefaultSinkProcessor" 
java.lang.Error: Maximum permit count exceeded
at 
java.util.concurrent.Semaphore$Sync.tryReleaseShared(Semaphore.java:192)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.releaseShared(AbstractQueuedSynchronizer.java:1341)
at java.util.concurrent.Semaphore.release(Semaphore.java:609)
at 
org.apache.flume.channel.MemoryChannel$MemoryTransaction.doCommit(MemoryChannel.java:147)
at 
org.apache.flume.channel.BasicTransactionSemantics.commit(BasicTransactionSemantics.java:151)
at 
org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:379)
at 
org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
at java.lang.Thread.run(Thread.java:745)
```

for more details and explanation see my comment on 
[FLUME-2812](https://issues.apache.org/jira/browse/FLUME-2812)

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/adenes/flume FLUME-2812

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flume/pull/83.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #83


commit 538837d5c035461c9b3073a2ce89da375aa139f3
Author: Denes Arvay 
Date:   2016-10-28T15:38:33Z

FLUME-2812: Exception in thread 
"SinkRunner-PollingRunner-DefaultSinkProcessor" java.lang.Error: Maximum permit 
count exceeded

Change-Id: Ib6761364abefca91ca50d80c073637766bfd93ca




> Exception in thread "SinkRunner-PollingRunner-DefaultSinkProcessor" 
> java.lang.Error: Maximum permit count exceeded
> --
>
> Key: FLUME-2812
> URL: https://issues.apache.org/jira/browse/FLUME-2812
> Project: Flume
>  Issue Type: Bug
>  Components: Channel, Sinks+Sources
>Affects Versions: v1.6.0
> Environment: **OS INFO**
> CentOS release 6.6 (Final)
> Kernel \r on an \m
> **JAVA INFO**
> java version "1.8.0_40"
> Java(TM) SE Runtime Environment (build 1.8.0_40-b26)
> Java HotSpot(TM) 64-Bit Server VM (build 25.40-b25, mixed mode)
>Reporter: Rollin Crittendon
>Assignee: Denes Arvay
>Priority: Critical
>
> We are finding that around after an hour or so of heavy processing of Flume 
> data in an agent we are getting the following exception.  This is after 
> processing about 5-7 k lines/second during that time.
> The configuration of this agent is using a Kafka source, the one that comes 
> with 1.6.0. 
> It is also using a Memory channel, and a Thrift sink.
> ===
> Exception in thread "SinkRunner-PollingRunner-DefaultSinkProcessor" 
> java.lang.Error: Maximum permit count exceeded
>   at 
> java.util.concurrent.Semaphore$Sync.tryReleaseShared(Semaphore.java:192)
>   at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer.releaseShared(AbstractQueuedSynchronizer.java:1341)
>   at java.util.concurrent.Semaphore.release(Semaphore.java:609)
>   at 
> org.apache.flume.channel.MemoryChannel$MemoryTransaction.doCommit(MemoryChannel.java:147)
>   at 
> org.apache.flume.channel.BasicTransactionSemantics.commit(BasicTransactionSemantics.java:151)
>   at 
> org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:379)
>   at 
> org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
>   at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>   at java.lang.Thread.run(Thread.java:745)
> ===
> The above error is from standard error when running the Flume agent.  The 
> effect is that the "SinkRunner-PollingRunner-DefaultSinkProcessor" thread 
> disappears from the agent, this can be seen on a JMX console.
> For us, this means that the Flume agent needs to get restarted.  It is an 
> error that is terminal in that instance of the Java process due to the thread 
> disappearing as a result.
> It sounds like something in JDK 7+ got stricter?!



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (FLUME-2812) Exception in thread "SinkRunner-PollingRunner-DefaultSinkProcessor" java.lang.Error: Maximum permit count exceeded

2016-10-28 Thread Denes Arvay (JIRA) 

 [ 
https://issues.apache.org/jira/browse/FLUME-2812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Denes Arvay reassigned FLUME-2812:
--

Assignee: Denes Arvay

> Exception in thread "SinkRunner-PollingRunner-DefaultSinkProcessor" 
> java.lang.Error: Maximum permit count exceeded
> --
>
> Key: FLUME-2812
> URL: https://issues.apache.org/jira/browse/FLUME-2812
> Project: Flume
>  Issue Type: Bug
>  Components: Channel, Sinks+Sources
>Affects Versions: v1.6.0
> Environment: **OS INFO**
> CentOS release 6.6 (Final)
> Kernel \r on an \m
> **JAVA INFO**
> java version "1.8.0_40"
> Java(TM) SE Runtime Environment (build 1.8.0_40-b26)
> Java HotSpot(TM) 64-Bit Server VM (build 25.40-b25, mixed mode)
>Reporter: Rollin Crittendon
>Assignee: Denes Arvay
>Priority: Critical
>
> We are finding that around after an hour or so of heavy processing of Flume 
> data in an agent we are getting the following exception.  This is after 
> processing about 5-7 k lines/second during that time.
> The configuration of this agent is using a Kafka source, the one that comes 
> with 1.6.0. 
> It is also using a Memory channel, and a Thrift sink.
> ===
> Exception in thread "SinkRunner-PollingRunner-DefaultSinkProcessor" 
> java.lang.Error: Maximum permit count exceeded
>   at 
> java.util.concurrent.Semaphore$Sync.tryReleaseShared(Semaphore.java:192)
>   at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer.releaseShared(AbstractQueuedSynchronizer.java:1341)
>   at java.util.concurrent.Semaphore.release(Semaphore.java:609)
>   at 
> org.apache.flume.channel.MemoryChannel$MemoryTransaction.doCommit(MemoryChannel.java:147)
>   at 
> org.apache.flume.channel.BasicTransactionSemantics.commit(BasicTransactionSemantics.java:151)
>   at 
> org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:379)
>   at 
> org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
>   at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>   at java.lang.Thread.run(Thread.java:745)
> ===
> The above error is from standard error when running the Flume agent.  The 
> effect is that the "SinkRunner-PollingRunner-DefaultSinkProcessor" thread 
> disappears from the agent, this can be seen on a JMX console.
> For us, this means that the Flume agent needs to get restarted.  It is an 
> error that is terminal in that instance of the Java process due to the thread 
> disappearing as a result.
> It sounds like something in JDK 7+ got stricter?!



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flume pull request #81: Add Developer Section / Developers Quick Hack Sheet ...

2016-10-28 Thread asfgit 
Github user asfgit closed the pull request at:

https://github.com/apache/flume/pull/81


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flume pull request #78: Add e-mail templates to dev-docs/HowToRelease.md

2016-10-28 Thread asfgit 
Github user asfgit closed the pull request at:

https://github.com/apache/flume/pull/78


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Build failed in Jenkins: Flume-trunk-hbase-1 #224

2016-10-28 Thread Apache Jenkins Server 
See 

Changes:

[bessbd] Add Developer Section / How to Commit from cwiki.apache.org

[bessbd] Add Developer Section / Developers Quick Hack Sheet from

--
[...truncated 6914 lines...]
  TestSyslogSource.tearDown:63 »  Failed to locate tar-ball distribution. 
Please...
  TestSyslogSource.testKeepFields:72 »  Failed to locate tar-ball distribution. 
...
  TestSyslogSource.tearDown:63 »  Failed to locate tar-ball distribution. 
Please...
  TestSyslogSource.testRemoveFields:82 »  Failed to locate tar-ball 
distribution...
  TestSyslogSource.tearDown:63 »  Failed to locate tar-ball distribution. 
Please...
  TestSyslogSource.testKeepTimestampAndHostname:92 »  Failed to locate tar-ball 
...
  TestSyslogSource.tearDown:63 »  Failed to locate tar-ball distribution. 
Please...
  TestRpcClient.setUp:36 »  Failed to locate tar-ball distribution. Please 
speci...
  TestRpcClient.tearDown:42 »  Failed to locate tar-ball distribution. Please 
sp...
  TestFileChannel.testInOut:117 »  Failed to locate tar-ball distribution. 
Pleas...
  TestFileChannel.tearDown:95 »  Failed to locate tar-ball distribution. Please 
...
  TestRpcClientCommunicationFailure.testFailure:61 »  Failed to locate tar-ball 
...

Tests run: 19, Failures: 0, Errors: 19, Skipped: 0

[ERROR] There are test failures.

Please refer to 

 for the individual test results.
[JENKINS] Recording test results
[INFO] 
[INFO] --- maven-jar-plugin:3.0.0:jar (default-jar) @ flume-ng-tests ---
[INFO] Building jar: 

[INFO] 
[INFO] --- maven-site-plugin:3.3:attach-descriptor (attach-descriptor) @ 
flume-ng-tests ---
[INFO] 
[INFO] --- apache-rat-plugin:0.11:check (verify.rat) @ flume-ng-tests ---
[INFO] 51 implicit excludes (use -debug for more details).
[INFO] Exclude: **/.idea/
[INFO] Exclude: **/*.iml
[INFO] Exclude: **/nb-configuration.xml
[INFO] Exclude: .git/
[INFO] Exclude: patchprocess/
[INFO] Exclude: .gitignore
[INFO] Exclude: .repository/
[INFO] Exclude: **/*.diff
[INFO] Exclude: **/*.patch
[INFO] Exclude: **/*.avsc
[INFO] Exclude: **/*.avro
[INFO] Exclude: **/docs/**
[INFO] Exclude: **/test/resources/**
[INFO] Exclude: **/.settings/*
[INFO] Exclude: **/.classpath
[INFO] Exclude: **/.project
[INFO] Exclude: **/target/**
[INFO] Exclude: **/derby.log
[INFO] Exclude: **/metastore_db/
[INFO] 9 resources included (use -debug for more details)
[INFO] Rat check: Summary of files. Unapproved: 0 unknown: 0 generated: 0 
approved: 9 licence.
[INFO] 
[INFO] --- maven-checkstyle-plugin:2.17:check (verify) @ flume-ng-tests ---
[INFO] 
[INFO] --- maven-install-plugin:2.5.2:install (default-install) @ 
flume-ng-tests ---
[INFO] Installing 

 to 

[INFO] Installing 
 
to 

[INFO] 
[INFO] Reactor Summary:
[INFO] 
[INFO] Flume checkstyle project ... SUCCESS [ 19.937 s]
[INFO] Apache Flume ... SUCCESS [ 13.708 s]
[INFO] Flume NG SDK ... SUCCESS [02:42 min]
[INFO] Flume NG Configuration . SUCCESS [  9.948 s]
[INFO] Flume Auth . SUCCESS [ 24.212 s]
[INFO] Flume NG Core .. SUCCESS [09:13 min]
[INFO] Flume NG Sinks . SUCCESS [  3.967 s]
[INFO] Flume NG HDFS Sink . SUCCESS [02:18 min]
[INFO] Flume NG IRC Sink .. SUCCESS [ 10.415 s]
[INFO] Flume NG Channels .. SUCCESS [  4.739 s]
[INFO] Flume NG JDBC channel .. SUCCESS [ 54.403 s]
[INFO] Flume NG file-based channel  SUCCESS [06:13 min]
[INFO] Flume NG Spillable Memory channel .. SUCCESS [02:06 min]
[INFO] Flume NG Node .. SUCCESS [ 47.795 s]
[INFO] Flume NG Embedded Agent  SUCCESS [ 38.328 s]
[INFO] Flume NG HBase Sink  SUCCESS [05:15 min]
[INFO] Flume NG ElasticSearch Sink  SUCCESS [01:22 min]
[INFO] Flume NG Morphline Solr Sink ... SUCCESS [ 52.625 s]

[jira] [Commented] (FLUME-2857) Kafka Source/Channel/Sink does not restore default values when live update config

2016-10-28 Thread JIRA 

[ 
https://issues.apache.org/jira/browse/FLUME-2857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15615260#comment-15615260
 ] 

Bessenyei Balázs Donát commented on FLUME-2857:
---

[~tmgstev]: thank you for the patch!

[~denes], [~sati]: thank you for the reviews!

> Kafka Source/Channel/Sink does not restore default values when live update 
> config
> -
>
> Key: FLUME-2857
> URL: https://issues.apache.org/jira/browse/FLUME-2857
> Project: Flume
>  Issue Type: Bug
>  Components: Channel
>Reporter: Tristan Stevens
>Assignee: Tristan Stevens
> Fix For: v1.8.0
>
> Attachments: FLUME-2857.patch
>
>
> Been using the following config:
> {noformat}
> tier1.channels.channel1.type = org.apache.flume.channel.kafka.KafkaChannel
> tier1.channels.channel1.capacity = 1
> tier1.channels.channel1.transactionCapacity = 1
> tier1.channels.channel1.brokerList = 
> 10.0.0.64:9092,10.0.0.65:9092,10.0.0.66:9092
> tier1.channels.channel1.topic = flume.aggregator.channel
> tier1.channels.channel1.zookeeperConnect = 10.0.0.64:2181
> tier1.channels.channel1.kafka.producer.type=async
> tier1.channels.channel1.kafka.batch.num.messages=200
> {noformat}
> If I remove the producer.type and batch.num.messages (i.e. restore them to 
> default values) when doing a live update config the new values do not take 
> effect.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (FLUME-2857) Kafka Source/Channel/Sink does not restore default values when live update config

2016-10-28 Thread JIRA 

 [ 
https://issues.apache.org/jira/browse/FLUME-2857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bessenyei Balázs Donát resolved FLUME-2857.
---
   Resolution: Fixed
Fix Version/s: v1.8.0

> Kafka Source/Channel/Sink does not restore default values when live update 
> config
> -
>
> Key: FLUME-2857
> URL: https://issues.apache.org/jira/browse/FLUME-2857
> Project: Flume
>  Issue Type: Bug
>  Components: Channel
>Reporter: Tristan Stevens
>Assignee: Tristan Stevens
> Fix For: v1.8.0
>
> Attachments: FLUME-2857.patch
>
>
> Been using the following config:
> {noformat}
> tier1.channels.channel1.type = org.apache.flume.channel.kafka.KafkaChannel
> tier1.channels.channel1.capacity = 1
> tier1.channels.channel1.transactionCapacity = 1
> tier1.channels.channel1.brokerList = 
> 10.0.0.64:9092,10.0.0.65:9092,10.0.0.66:9092
> tier1.channels.channel1.topic = flume.aggregator.channel
> tier1.channels.channel1.zookeeperConnect = 10.0.0.64:2181
> tier1.channels.channel1.kafka.producer.type=async
> tier1.channels.channel1.kafka.batch.num.messages=200
> {noformat}
> If I remove the producer.type and batch.num.messages (i.e. restore them to 
> default values) when doing a live update config the new values do not take 
> effect.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLUME-2857) Kafka Source/Channel/Sink does not restore default values when live update config

2016-10-28 Thread ASF subversion and git services (JIRA) 

[ 
https://issues.apache.org/jira/browse/FLUME-2857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15615255#comment-15615255
 ] 

ASF subversion and git services commented on FLUME-2857:


Commit 14fb4d84fd0e100253ca947bc96810c242e7a82b in flume's branch 
refs/heads/trunk from [~tmgstev]
[ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=14fb4d8 ]

FLUME-2857. Make Kafka Source/Channel/Sink restore default values when live 
updating config

This commit changes Kafka Channel, Sink and Source to fix an error where
sub-configurations aren't tolerant of the configure() method being called more
than once (as happens when a Live Config Update happens).

Reviewers: Denes Arvay, Attila Simon, Bessenyei Balázs Donát

(Tristan Stevens via Bessenyei Balázs Donát)


> Kafka Source/Channel/Sink does not restore default values when live update 
> config
> -
>
> Key: FLUME-2857
> URL: https://issues.apache.org/jira/browse/FLUME-2857
> Project: Flume
>  Issue Type: Bug
>  Components: Channel
>Reporter: Tristan Stevens
>Assignee: Tristan Stevens
> Attachments: FLUME-2857.patch
>
>
> Been using the following config:
> {noformat}
> tier1.channels.channel1.type = org.apache.flume.channel.kafka.KafkaChannel
> tier1.channels.channel1.capacity = 1
> tier1.channels.channel1.transactionCapacity = 1
> tier1.channels.channel1.brokerList = 
> 10.0.0.64:9092,10.0.0.65:9092,10.0.0.66:9092
> tier1.channels.channel1.topic = flume.aggregator.channel
> tier1.channels.channel1.zookeeperConnect = 10.0.0.64:2181
> tier1.channels.channel1.kafka.producer.type=async
> tier1.channels.channel1.kafka.batch.num.messages=200
> {noformat}
> If I remove the producer.type and batch.num.messages (i.e. restore them to 
> default values) when doing a live update config the new values do not take 
> effect.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLUME-2857) Kafka Source/Channel/Sink does not restore default values when live update config

2016-10-28 Thread JIRA 

[ 
https://issues.apache.org/jira/browse/FLUME-2857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15615187#comment-15615187
 ] 

Bessenyei Balázs Donát commented on FLUME-2857:
---

LGTM, +1

I'm about to commit this

> Kafka Source/Channel/Sink does not restore default values when live update 
> config
> -
>
> Key: FLUME-2857
> URL: https://issues.apache.org/jira/browse/FLUME-2857
> Project: Flume
>  Issue Type: Bug
>  Components: Channel
>Reporter: Tristan Stevens
>Assignee: Tristan Stevens
> Attachments: FLUME-2857.patch
>
>
> Been using the following config:
> {noformat}
> tier1.channels.channel1.type = org.apache.flume.channel.kafka.KafkaChannel
> tier1.channels.channel1.capacity = 1
> tier1.channels.channel1.transactionCapacity = 1
> tier1.channels.channel1.brokerList = 
> 10.0.0.64:9092,10.0.0.65:9092,10.0.0.66:9092
> tier1.channels.channel1.topic = flume.aggregator.channel
> tier1.channels.channel1.zookeeperConnect = 10.0.0.64:2181
> tier1.channels.channel1.kafka.producer.type=async
> tier1.channels.channel1.kafka.batch.num.messages=200
> {noformat}
> If I remove the producer.type and batch.num.messages (i.e. restore them to 
> default values) when doing a live update config the new values do not take 
> effect.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 53231: FLUME-2857 Kafka Source/Channel/Sink does not restore default values when live update config

2016-10-28 Thread Balázs Donát Bessenyei 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53231/#review154116
---


Ship it!




Ship It!

- Balázs Donát Bessenyei


On Oct. 28, 2016, 10:24 a.m., Tristan Stevens wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/53231/
> ---
> 
> (Updated Oct. 28, 2016, 10:24 a.m.)
> 
> 
> Review request for Flume.
> 
> 
> Repository: flume-git
> 
> 
> Description
> ---
> 
> Change to fix error where Kafka Channel, Sink and Source sub-configurations 
> aren't tolerant of the configure() method being called more than once (as 
> happens when a Live Config Update happens)
> 
> 
> Diffs
> -
> 
>   
> flume-ng-channels/flume-kafka-channel/src/main/java/org/apache/flume/channel/kafka/KafkaChannel.java
>  47c0634 
>   
> flume-ng-channels/flume-kafka-channel/src/test/java/org/apache/flume/channel/kafka/TestKafkaChannel.java
>  276fee1 
>   
> flume-ng-sinks/flume-ng-kafka-sink/src/main/java/org/apache/flume/sink/kafka/KafkaSink.java
>  dd40224 
>   
> flume-ng-sinks/flume-ng-kafka-sink/src/test/java/org/apache/flume/sink/kafka/TestKafkaSink.java
>  7eccf76 
>   
> flume-ng-sources/flume-kafka-source/src/main/java/org/apache/flume/source/kafka/KafkaSource.java
>  195eca3 
>   
> flume-ng-sources/flume-kafka-source/src/test/java/org/apache/flume/source/kafka/TestKafkaSource.java
>  9554201 
> 
> Diff: https://reviews.apache.org/r/53231/diff/
> 
> 
> Testing
> ---
> 
> Added unit tests which all would have failed and now pass.
> Also tested manually for Channel and Sink (Producer settings only), Consumer 
> settings are a little harder as you don't get the verbose logs, however unit 
> tests pass.
> 
> 
> Thanks,
> 
> Tristan Stevens
> 
>

[GitHub] flume pull request #80: Add Developer Section / How to Commit from cwiki.apa...

2016-10-28 Thread asfgit 
Github user asfgit closed the pull request at:

https://github.com/apache/flume/pull/80


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Review Request 53231: FLUME-2857 Kafka Source/Channel/Sink does not restore default values when live update config

2016-10-28 Thread Attila Simon 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53231/#review154111
---


Ship it!




LGTM

- Attila Simon


On Oct. 28, 2016, 10:24 a.m., Tristan Stevens wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/53231/
> ---
> 
> (Updated Oct. 28, 2016, 10:24 a.m.)
> 
> 
> Review request for Flume.
> 
> 
> Repository: flume-git
> 
> 
> Description
> ---
> 
> Change to fix error where Kafka Channel, Sink and Source sub-configurations 
> aren't tolerant of the configure() method being called more than once (as 
> happens when a Live Config Update happens)
> 
> 
> Diffs
> -
> 
>   
> flume-ng-channels/flume-kafka-channel/src/main/java/org/apache/flume/channel/kafka/KafkaChannel.java
>  47c0634 
>   
> flume-ng-channels/flume-kafka-channel/src/test/java/org/apache/flume/channel/kafka/TestKafkaChannel.java
>  276fee1 
>   
> flume-ng-sinks/flume-ng-kafka-sink/src/main/java/org/apache/flume/sink/kafka/KafkaSink.java
>  dd40224 
>   
> flume-ng-sinks/flume-ng-kafka-sink/src/test/java/org/apache/flume/sink/kafka/TestKafkaSink.java
>  7eccf76 
>   
> flume-ng-sources/flume-kafka-source/src/main/java/org/apache/flume/source/kafka/KafkaSource.java
>  195eca3 
>   
> flume-ng-sources/flume-kafka-source/src/test/java/org/apache/flume/source/kafka/TestKafkaSource.java
>  9554201 
> 
> Diff: https://reviews.apache.org/r/53231/diff/
> 
> 
> Testing
> ---
> 
> Added unit tests which all would have failed and now pass.
> Also tested manually for Channel and Sink (Producer settings only), Consumer 
> settings are a little harder as you don't get the verbose logs, however unit 
> tests pass.
> 
> 
> Thanks,
> 
> Tristan Stevens
> 
>

Re: Review Request 53231: FLUME-2857 Kafka Source/Channel/Sink does not restore default values when live update config

2016-10-28 Thread Tristan Stevens 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53231/
---

(Updated Oct. 28, 2016, 10:24 a.m.)


Review request for Flume.


Changes
---

Issues fixed, thanks @Denes


Repository: flume-git


Description
---

Change to fix error where Kafka Channel, Sink and Source sub-configurations 
aren't tolerant of the configure() method being called more than once (as 
happens when a Live Config Update happens)


Diffs (updated)
-

  
flume-ng-channels/flume-kafka-channel/src/main/java/org/apache/flume/channel/kafka/KafkaChannel.java
 47c0634 
  
flume-ng-channels/flume-kafka-channel/src/test/java/org/apache/flume/channel/kafka/TestKafkaChannel.java
 276fee1 
  
flume-ng-sinks/flume-ng-kafka-sink/src/main/java/org/apache/flume/sink/kafka/KafkaSink.java
 dd40224 
  
flume-ng-sinks/flume-ng-kafka-sink/src/test/java/org/apache/flume/sink/kafka/TestKafkaSink.java
 7eccf76 
  
flume-ng-sources/flume-kafka-source/src/main/java/org/apache/flume/source/kafka/KafkaSource.java
 195eca3 
  
flume-ng-sources/flume-kafka-source/src/test/java/org/apache/flume/source/kafka/TestKafkaSource.java
 9554201 

Diff: https://reviews.apache.org/r/53231/diff/


Testing
---

Added unit tests which all would have failed and now pass.
Also tested manually for Channel and Sink (Producer settings only), Consumer 
settings are a little harder as you don't get the verbose logs, however unit 
tests pass.


Thanks,

Tristan Stevens