Re: Review Request 51918: FLUME-2993: Support environment variables in configuration files

2016-09-24 Thread Balázs Donát Bessenyei 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51918/
---

(Updated Sept. 24, 2016, 8:26 p.m.)


Review request for Flume.


Bugs: FLUME-2993
https://issues.apache.org/jira/browse/FLUME-2993


Repository: flume-git


Description (updated)
---

Flume does not currently support environment variable interpolation in the 
properties file configuration.

Enabling it would help with:
* removing security credentials from config files
* help with copy-pastes in configuration files with multiple agents defined

It is arguably a best practice to store (some) config in the environment 
variables: https://12factor.net/config


Diffs
-

  c/flume-ng-doc/sphinx/FlumeUserGuide.rst 0fecee6 
  c/flume-ng-node/pom.xml e33b566 
  
c/flume-ng-node/src/main/java/org/apache/flume/node/PropertiesFileConfigurationProvider.java
 b428c9e 
  
w/flume-ng-node/src/main/java/org/apache/flume/node/EnvVarResolverProperties.java
 PRE-CREATION 
  
w/flume-ng-node/src/test/java/org/apache/flume/node/TestEnvVarResolverProperties.java
 PRE-CREATION 
  w/flume-ng-node/src/test/resources/flume-conf-with-envvars.properties 
PRE-CREATION 

Diff: https://reviews.apache.org/r/51918/diff/


Testing
---

mvn clean install runs successfully

new unit tests were added


Thanks,

Balázs Donát Bessenyei

Re: Review Request 51918: FLUME-2993: Support environment variables in configuration files

2016-09-24 Thread Balázs Donát Bessenyei 


> On Sept. 24, 2016, 12:52 p.m., Lior Zeno wrote:
> > LGTM, should we enable this by default?

Thank you for the review!

I'm not against enabling it by default. I'd like to have more people's opinion 
about it before making the change.


- Balázs Donát


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51918/#review150300
---


On Sept. 15, 2016, 5:01 p.m., Balázs Donát Bessenyei wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51918/
> ---
> 
> (Updated Sept. 15, 2016, 5:01 p.m.)
> 
> 
> Review request for Flume.
> 
> 
> Bugs: FLUME-2993
> https://issues.apache.org/jira/browse/FLUME-2993
> 
> 
> Repository: flume-git
> 
> 
> Description
> ---
> 
> Flume does not currently support environment variable interpolation in the 
> properties file configuration.
> 
> Enabling it would help with
> * removing security credentials from config files
> * help with copy-pastes in configuration files with multiple agents defined
> 
> It is arguably a best practice to store (some) config in the environment 
> variables: https://12factor.net/config
> 
> 
> Diffs
> -
> 
>   c/flume-ng-doc/sphinx/FlumeUserGuide.rst 0fecee6 
>   c/flume-ng-node/pom.xml e33b566 
>   
> c/flume-ng-node/src/main/java/org/apache/flume/node/PropertiesFileConfigurationProvider.java
>  b428c9e 
>   
> w/flume-ng-node/src/main/java/org/apache/flume/node/EnvVarResolverProperties.java
>  PRE-CREATION 
>   
> w/flume-ng-node/src/test/java/org/apache/flume/node/TestEnvVarResolverProperties.java
>  PRE-CREATION 
>   w/flume-ng-node/src/test/resources/flume-conf-with-envvars.properties 
> PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/51918/diff/
> 
> 
> Testing
> ---
> 
> mvn clean install runs successfully
> 
> new unit tests were added
> 
> 
> Thanks,
> 
> Balázs Donát Bessenyei
> 
>

[jira] [Assigned] (FLUME-2857) Kafka Channel does not restore default values when live update config

2016-09-24 Thread JIRA 

 [ 
https://issues.apache.org/jira/browse/FLUME-2857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bessenyei Balázs Donát reassigned FLUME-2857:
-

Assignee: Bessenyei Balázs Donát

> Kafka Channel does not restore default values when live update config
> -
>
> Key: FLUME-2857
> URL: https://issues.apache.org/jira/browse/FLUME-2857
> Project: Flume
>  Issue Type: Bug
>  Components: Channel
>Reporter: Tristan Stevens
>Assignee: Bessenyei Balázs Donát
>
> Been using the following config:
> {noformat}
> tier1.channels.channel1.type = org.apache.flume.channel.kafka.KafkaChannel
> tier1.channels.channel1.capacity = 1
> tier1.channels.channel1.transactionCapacity = 1
> tier1.channels.channel1.brokerList = 
> 10.0.0.64:9092,10.0.0.65:9092,10.0.0.66:9092
> tier1.channels.channel1.topic = flume.aggregator.channel
> tier1.channels.channel1.zookeeperConnect = 10.0.0.64:2181
> tier1.channels.channel1.kafka.producer.type=async
> tier1.channels.channel1.kafka.batch.num.messages=200
> {noformat}
> If I remove the producer.type and batch.num.messages (i.e. restore them to 
> default values) when doing a live update config the new values do not take 
> effect.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLUME-2917) Provide netcat UDP source as alternative to TCP

2016-09-24 Thread JIRA 

[ 
https://issues.apache.org/jira/browse/FLUME-2917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15519572#comment-15519572
 ] 

Bessenyei Balázs Donát commented on FLUME-2917:
---

[~tmgstev]: thank you for the patch!

I think, this could be a useful change

I just don't know if it's easier to maintain the feature in a separate module 
or we should merge it with Netcat Source and add an extra parameter to decide 
if it's TCP or UDP? What is your opinion?

Also, could you please create a review request on https://reviews.apache.org ?

> Provide netcat UDP source as alternative to TCP
> ---
>
> Key: FLUME-2917
> URL: https://issues.apache.org/jira/browse/FLUME-2917
> Project: Flume
>  Issue Type: New Feature
>  Components: Sinks+Sources
>Affects Versions: v1.6.0
>Reporter: Tristan Stevens
>Priority: Minor
> Attachments: FLUME-2917.patch
>
>
> Currently Flume provides a Netcat TCP source, however Netcat is often used 
> with UDP. There is an implementation of a UDP client in the SyslogUDP source, 
> this request takes this implementation and strips out the Syslog parts, thus 
> forming a Netcat UDP source - where each datagram is recorded as a Flume 
> event.
> The implementation is provided for this at 
> https://github.com/tmgstevens/FlumeNetcatUDPSource and also provided as an 
> attached patch for inclusion.
> N.B. Unit tests are provided for this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Issue Comment Deleted] (FLUME-2981) Upgrade the Solr version to 5.5.2

2016-09-24 Thread Tristan Stevens (JIRA) 

 [ 
https://issues.apache.org/jira/browse/FLUME-2981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tristan Stevens updated FLUME-2981:
---
Comment: was deleted

(was: Should this not be Solr 6 for Flume 1.7?)

> Upgrade the Solr version to 5.5.2
> -
>
> Key: FLUME-2981
> URL: https://issues.apache.org/jira/browse/FLUME-2981
> Project: Flume
>  Issue Type: Improvement
>  Components: Sinks+Sources
>Affects Versions: v1.6.0
>Reporter: Minoru Osuka
>Priority: Minor
> Fix For: v1.7.0
>
> Attachments: FLUME-2981-0.patch
>
>
> Currently flume-ng-morphline-solr-sink is using Solr 4.3.0. I propose to 
> upgrade to Solr 5.5.2 for Flume 1.7.0. Also Solr 5.5.2 requires Java 1.7 same 
> as Flume 1.7.0. 
> Solr 5.5.2 includes guava 14 but it also works with guava 11. There were no 
> problems when I confirmed it with guava 11.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLUME-2981) Upgrade the Solr version to 5.5.2

2016-09-24 Thread Tristan Stevens (JIRA) 

[ 
https://issues.apache.org/jira/browse/FLUME-2981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15519538#comment-15519538
 ] 

Tristan Stevens commented on FLUME-2981:


Should this not be Solr 6 for Flume 1.7?

> Upgrade the Solr version to 5.5.2
> -
>
> Key: FLUME-2981
> URL: https://issues.apache.org/jira/browse/FLUME-2981
> Project: Flume
>  Issue Type: Improvement
>  Components: Sinks+Sources
>Affects Versions: v1.6.0
>Reporter: Minoru Osuka
>Priority: Minor
> Fix For: v1.7.0
>
> Attachments: FLUME-2981-0.patch
>
>
> Currently flume-ng-morphline-solr-sink is using Solr 4.3.0. I propose to 
> upgrade to Solr 5.5.2 for Flume 1.7.0. Also Solr 5.5.2 requires Java 1.7 same 
> as Flume 1.7.0. 
> Solr 5.5.2 includes guava 14 but it also works with guava 11. There were no 
> problems when I confirmed it with guava 11.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLUME-2917) Provide netcat UDP source as alternative to TCP

2016-09-24 Thread Tristan Stevens (JIRA) 

[ 
https://issues.apache.org/jira/browse/FLUME-2917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15519524#comment-15519524
 ] 

Tristan Stevens commented on FLUME-2917:


[~bessbd] Do you think this source has value? I'd need to add a section to the 
docs, but I've seen many cases where UDP would be preferable to TCP.

> Provide netcat UDP source as alternative to TCP
> ---
>
> Key: FLUME-2917
> URL: https://issues.apache.org/jira/browse/FLUME-2917
> Project: Flume
>  Issue Type: New Feature
>  Components: Sinks+Sources
>Affects Versions: v1.6.0
>Reporter: Tristan Stevens
>Priority: Minor
> Attachments: FLUME-2917.patch
>
>
> Currently Flume provides a Netcat TCP source, however Netcat is often used 
> with UDP. There is an implementation of a UDP client in the SyslogUDP source, 
> this request takes this implementation and strips out the Syslog parts, thus 
> forming a Netcat UDP source - where each datagram is recorded as a Flume 
> event.
> The implementation is provided for this at 
> https://github.com/tmgstevens/FlumeNetcatUDPSource and also provided as an 
> attached patch for inclusion.
> N.B. Unit tests are provided for this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLUME-2857) Kafka Channel does not restore default values when live update config

2016-09-24 Thread Tristan Stevens (JIRA) 

[ 
https://issues.apache.org/jira/browse/FLUME-2857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15519510#comment-15519510
 ] 

Tristan Stevens commented on FLUME-2857:


[~bessbd] Please take this one - hopefully it's a simple fix :)

> Kafka Channel does not restore default values when live update config
> -
>
> Key: FLUME-2857
> URL: https://issues.apache.org/jira/browse/FLUME-2857
> Project: Flume
>  Issue Type: Bug
>  Components: Channel
>Reporter: Tristan Stevens
>
> Been using the following config:
> {noformat}
> tier1.channels.channel1.type = org.apache.flume.channel.kafka.KafkaChannel
> tier1.channels.channel1.capacity = 1
> tier1.channels.channel1.transactionCapacity = 1
> tier1.channels.channel1.brokerList = 
> 10.0.0.64:9092,10.0.0.65:9092,10.0.0.66:9092
> tier1.channels.channel1.topic = flume.aggregator.channel
> tier1.channels.channel1.zookeeperConnect = 10.0.0.64:2181
> tier1.channels.channel1.kafka.producer.type=async
> tier1.channels.channel1.kafka.batch.num.messages=200
> {noformat}
> If I remove the producer.type and batch.num.messages (i.e. restore them to 
> default values) when doing a live update config the new values do not take 
> effect.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 51918: FLUME-2993: Support environment variables in configuration files

2016-09-24 Thread Lior Zeno 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51918/#review150300
---



LGTM, should we enable this by default?

- Lior Zeno


On Sept. 15, 2016, 5:01 p.m., Balázs Donát Bessenyei wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51918/
> ---
> 
> (Updated Sept. 15, 2016, 5:01 p.m.)
> 
> 
> Review request for Flume.
> 
> 
> Bugs: FLUME-2993
> https://issues.apache.org/jira/browse/FLUME-2993
> 
> 
> Repository: flume-git
> 
> 
> Description
> ---
> 
> Flume does not currently support environment variable interpolation in the 
> properties file configuration.
> 
> Enabling it would help with
> * removing security credentials from config files
> * help with copy-pastes in configuration files with multiple agents defined
> 
> It is arguably a best practice to store (some) config in the environment 
> variables: https://12factor.net/config
> 
> 
> Diffs
> -
> 
>   c/flume-ng-doc/sphinx/FlumeUserGuide.rst 0fecee6 
>   c/flume-ng-node/pom.xml e33b566 
>   
> c/flume-ng-node/src/main/java/org/apache/flume/node/PropertiesFileConfigurationProvider.java
>  b428c9e 
>   
> w/flume-ng-node/src/main/java/org/apache/flume/node/EnvVarResolverProperties.java
>  PRE-CREATION 
>   
> w/flume-ng-node/src/test/java/org/apache/flume/node/TestEnvVarResolverProperties.java
>  PRE-CREATION 
>   w/flume-ng-node/src/test/resources/flume-conf-with-envvars.properties 
> PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/51918/diff/
> 
> 
> Testing
> ---
> 
> mvn clean install runs successfully
> 
> new unit tests were added
> 
> 
> Thanks,
> 
> Balázs Donát Bessenyei
> 
>

Re: Review Request 52216: Patch for FLUME-2647

2016-09-24 Thread Lior Zeno 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52216/#review150299
---


Fix it, then Ship it!





flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java
 (line 650)


style nit: can you please join the first two strings?



flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java
 (line 656)


same here


- Lior Zeno


On Sept. 24, 2016, 12:23 p.m., Umesh Chaudhary wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/52216/
> ---
> 
> (Updated Sept. 24, 2016, 12:23 p.m.)
> 
> 
> Review request for Flume.
> 
> 
> Repository: flume-git
> 
> 
> Description
> ---
> 
> Added two new booleans to identify the reason for closed channel: 1) 
> isClosedOnStart 2) isClosedNormally
> And checked them to print the appropriate reason for closed channel
> 
> 
> Diffs
> -
> 
>   
> flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java
>  9d82e43 
>   
> flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java
>  bfc2d0d 
> 
> Diff: https://reviews.apache.org/r/52216/diff/
> 
> 
> Testing
> ---
> 
> Yes
> 
> 
> Thanks,
> 
> Umesh Chaudhary
> 
>

Re: Review Request 52216: Patch for FLUME-2647

2016-09-24 Thread Umesh Chaudhary 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52216/
---

(Updated Sept. 24, 2016, 12:23 p.m.)


Review request for Flume.


Changes
---

Incorporated Balázs's comments and added unit tests.


Repository: flume-git


Description
---

Added two new booleans to identify the reason for closed channel: 1) 
isClosedOnStart 2) isClosedNormally
And checked them to print the appropriate reason for closed channel


Diffs (updated)
-

  
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java
 9d82e43 
  
flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java
 bfc2d0d 

Diff: https://reviews.apache.org/r/52216/diff/


Testing
---

Yes


Thanks,

Umesh Chaudhary

[jira] [Comment Edited] (FLUME-2647) File Channel must print the reason for the channel being closed in channelClosed checks

2016-09-24 Thread Umesh Chaudhary (JIRA) 

[ 
https://issues.apache.org/jira/browse/FLUME-2647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15518851#comment-15518851
 ] 

Umesh Chaudhary edited comment on FLUME-2647 at 9/24/16 10:44 AM:
--

Hi @bessbd, 
Thanks for reviewing the patch.
Sure, will submit the updated patch soon.


was (Author: umesh9...@gmail.com):
Hi @bessbd, 
Sure, will submit the updated patch soon.

> File Channel must print the reason for the channel being closed in 
> channelClosed checks
> ---
>
> Key: FLUME-2647
> URL: https://issues.apache.org/jira/browse/FLUME-2647
> Project: Flume
>  Issue Type: Bug
>Reporter: Hari Shreedharan
>Assignee: Umesh Chaudhary
> Attachments: FLUME-2647-0.patch
>
>
> Before each operation we check if the channel is open, but if it is closed, 
> we don't say how it was closed. We must print this exception to the logs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLUME-2647) File Channel must print the reason for the channel being closed in channelClosed checks

2016-09-24 Thread Umesh Chaudhary (JIRA) 

[ 
https://issues.apache.org/jira/browse/FLUME-2647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15518851#comment-15518851
 ] 

Umesh Chaudhary commented on FLUME-2647:


Hi @bessbd, 
Sure, will submit the updated patch soon.

> File Channel must print the reason for the channel being closed in 
> channelClosed checks
> ---
>
> Key: FLUME-2647
> URL: https://issues.apache.org/jira/browse/FLUME-2647
> Project: Flume
>  Issue Type: Bug
>Reporter: Hari Shreedharan
>Assignee: Umesh Chaudhary
> Attachments: FLUME-2647-0.patch
>
>
> Before each operation we check if the channel is open, but if it is closed, 
> we don't say how it was closed. We must print this exception to the logs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (FLUME-2996) Flume process shutdown automaticly

2016-09-24 Thread hailin (JIRA) 

[ 
https://issues.apache.org/jira/browse/FLUME-2996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15518810#comment-15518810
 ] 

hailin edited comment on FLUME-2996 at 9/24/16 10:14 AM:
-

hi bellow is the configure file content:

a1.sources = r1
a1.sinks = sink_etl_product sink_kinesis_stream
a1.channels = channel_etl_product channel_kinesis_stream

#source setting
#a1.sources.r1.type = http
a1.sources.r1.type = flume.source.HTTPSource
a1.sources.r1.port = 6558
a1.sources.r1.dataSize = 1
a1.sources.r1.selector.type = multiplexing
a1.sources.r1.selector.header = event_type
a1.sources.r1.selector.mapping.etl_product = channel_etl_product 
channel_kinesis_stream

#handler setting
a1.sources.r1.handler = flume.source.BlobExtHandler
a1.sources.r1.handler.gzip = true
a1.sources.r1.handler.maxBlobLength = 10

a1.channels.channel_etl_product.type = file
a1.channels.channel_etl_product.checkpointDir = ./checkpoint/product
a1.channels.channel_etl_product.dataDirs = ./data/product
a1.channels.channel_etl_product.checkpointInterval = 10

#Kinesis Firehose
a1.sinks.sink_etl_mix.type = flume.sink.KinesisFirehoseSink
a1.sinks.sink_etl_mix.channel = channel_etl_product
a1.sinks.sink_etl_mix.streamName = product_flumeFirehoseStream
a1.sinks.sink_etl_mix.endpoint = {firehose_endpoint}


flume.source.HTTPSource,flume.source.BlobExtHandler,flume.sink.KinesisFirehoseSink,
 these 3 Java class contains our bussiness logic. 
what I want is that the flume will transfer data into AWS Kinesis Firehose.
The application works fine on AWS EC2 linux instance. What I meet is flume 
shutdown automaticly after several hours , the process disappear when run "ps 
-ef | grep flume".

I check the flume log, no errors found, seems it shutdown normally:
24 Sep 2016 05:53:59,391 INFO  [agent-shutdown-hook] 
(org.apache.flume.lifecycle.LifecycleSupervisor.stop:79)  - Stopping lifecycle 
supervisor 10
24 Sep 2016 05:53:59,394 INFO  [agent-shutdown-hook] 
(org.apache.flume.channel.file.FileChannel.stop:321)  - Stopping FileChannel 
channel_etl_product { dataDirs: [./data/product] }...
24 Sep 2016 05:53:59,395 INFO  [agent-shutdown-hook] 
(org.apache.flume.channel.file.EventQueueBackingStoreFile.beginCheckpoint:230)  
- Start checkpoint for ./checkpoint/product/checkpoint, elements to sync = 0
24 Sep 2016 05:53:59,401 INFO  [agent-shutdown-hook] 
(org.apache.flume.channel.file.EventQueueBackingStoreFile.checkpoint:255)  - 
Updating checkpoint metadata: logWriteOrderID: 1474685940834, queueSize: 0, 
queueHead: 0
24 Sep 2016 05:53:59,407 INFO  [agent-shutdown-hook] 
(org.apache.flume.channel.file.Log.writeCheckpoint:1034)  - Updated checkpoint 
for file: ./data/product/log-3 position: 6674 logWriteOrderID: 1474685940834
24 Sep 2016 05:53:59,407 INFO  [agent-shutdown-hook] 
(org.apache.flume.channel.file.Log.shutdownWorker:857)  - Attempting to 
shutdown background worker.
24 Sep 2016 05:53:59,409 INFO  [agent-shutdown-hook] 
(org.apache.flume.channel.file.LogFile$Writer.close:370)  - Closing 
./data/product/log-3
24 Sep 2016 05:53:59,409 INFO  [agent-shutdown-hook] 
(org.apache.flume.channel.file.LogFile$RandomReader.close:504)  - Closing 
RandomReader ./data/product/log-2
24 Sep 2016 05:53:59,414 INFO  [agent-shutdown-hook] 
(org.apache.flume.channel.file.LogFile$RandomReader.close:504)  - Closing 
RandomReader ./data/product/log-3
24 Sep 2016 05:53:59,420 INFO  [agent-shutdown-hook] 
(org.apache.flume.instrumentation.MonitoredCounterGroup.stop:150)  - Component 
type: CHANNEL, name: channel_etl_product stopped

please let me know if need more information.


thanks


was (Author: hehailin):
hi bellow is the configure file content:

a1.sources = r1
a1.sinks = sink_etl_product sink_kinesis_stream
a1.channels = channel_etl_product channel_kinesis_stream

#source setting
#a1.sources.r1.type = http
a1.sources.r1.type = flume.source.HTTPSource
a1.sources.r1.port = 6558
a1.sources.r1.dataSize = 1
a1.sources.r1.selector.type = multiplexing
a1.sources.r1.selector.header = event_type
a1.sources.r1.selector.mapping.etl_product = channel_etl_product 
channel_kinesis_stream

#handler setting
a1.sources.r1.handler = flume.source.BlobExtHandler
a1.sources.r1.handler.gzip = true
a1.sources.r1.handler.maxBlobLength = 10

a1.channels.channel_etl_product.type = file
a1.channels.channel_etl_product.checkpointDir = ./checkpoint/product
a1.channels.channel_etl_product.dataDirs = ./data/product
a1.channels.channel_etl_product.checkpointInterval = 10

#Kinesis Firehose
a1.sinks.sink_etl_mix.type = flume.sink.KinesisFirehoseSink
a1.sinks.sink_etl_mix.channel = channel_etl_product
a1.sinks.sink_etl_mix.streamName = product_flumeFirehoseStream
a1.sinks.sink_etl_mix.endpoint = {firehose_endpoint}


flume.source.HTTPSource,flume.source.BlobExtHandler,flume.sink.KinesisFirehoseSink,
 these 3 Java class contains our bussiness logic. 
what I want is that the flume will transfer data into

[jira] [Commented] (FLUME-2647) File Channel must print the reason for the channel being closed in channelClosed checks

2016-09-24 Thread JIRA 

[ 
https://issues.apache.org/jira/browse/FLUME-2647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15518813#comment-15518813
 ] 

Bessenyei Balázs Donát commented on FLUME-2647:
---

Hi [~umesh9...@gmail.com],

Thank you for the patch.

I have left some notes on the review request. 

> File Channel must print the reason for the channel being closed in 
> channelClosed checks
> ---
>
> Key: FLUME-2647
> URL: https://issues.apache.org/jira/browse/FLUME-2647
> Project: Flume
>  Issue Type: Bug
>Reporter: Hari Shreedharan
>Assignee: Umesh Chaudhary
> Attachments: FLUME-2647-0.patch
>
>
> Before each operation we check if the channel is open, but if it is closed, 
> we don't say how it was closed. We must print this exception to the logs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLUME-2996) Flume process shutdown automaticly

2016-09-24 Thread hailin (JIRA) 

[ 
https://issues.apache.org/jira/browse/FLUME-2996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15518810#comment-15518810
 ] 

hailin commented on FLUME-2996:
---

hi bellow is the configure file content:

a1.sources = r1
a1.sinks = sink_etl_product sink_kinesis_stream
a1.channels = channel_etl_product channel_kinesis_stream

#source setting
#a1.sources.r1.type = http
a1.sources.r1.type = flume.source.HTTPSource
a1.sources.r1.port = 6558
a1.sources.r1.dataSize = 1
a1.sources.r1.selector.type = multiplexing
a1.sources.r1.selector.header = event_type
a1.sources.r1.selector.mapping.etl_product = channel_etl_product 
channel_kinesis_stream

#handler setting
a1.sources.r1.handler = flume.source.BlobExtHandler
a1.sources.r1.handler.gzip = true
a1.sources.r1.handler.maxBlobLength = 10

a1.channels.channel_etl_product.type = file
a1.channels.channel_etl_product.checkpointDir = ./checkpoint/product
a1.channels.channel_etl_product.dataDirs = ./data/product
a1.channels.channel_etl_product.checkpointInterval = 10

#Kinesis Firehose
a1.sinks.sink_etl_mix.type = flume.sink.KinesisFirehoseSink
a1.sinks.sink_etl_mix.channel = channel_etl_product
a1.sinks.sink_etl_mix.streamName = product_flumeFirehoseStream
a1.sinks.sink_etl_mix.endpoint = {firehose_endpoint}


flume.source.HTTPSource,flume.source.BlobExtHandler,flume.sink.KinesisFirehoseSink,
 these 3 Java class contains our bussiness logic. 
what I want is that the flume will transfer data into AWS Kinesis Firehose.
The application works fine on AWS EC2 linux instance. What I meet is flume 
shutdown automaticly after several hours , the process disappear when run "ps 
-ef | grep flume".

please let me know if need more information.


thanks

> Flume process shutdown automaticly
> --
>
> Key: FLUME-2996
> URL: https://issues.apache.org/jira/browse/FLUME-2996
> Project: Flume
>  Issue Type: Bug
> Environment: Amazon Linux
>Reporter: hailin
>
> hi guys,
> I install flume from apache-flume-1.6.0-bin.tar.gz, and startup flume with 
> some my custom configure file with cmd line like this: 
> bin/flume-ng agent -Dflume.monitoring.type=http -Dflume.monitoring.port=41414 
> -c conf -f conf/flume.conf --name a1 -Dflume.root.logger=INFO,DAILY 
> it works fine without any issue, but after several hours when I check the 
> process, I found it was shutdown automaticly. Bellow is the flume log 
> information:
> .
> 24 Sep 2016 05:53:59,391 INFO  [agent-shutdown-hook] 
> (org.apache.flume.lifecycle.LifecycleSupervisor.stop:79)  - Stopping 
> lifecycle supervisor 10
> 24 Sep 2016 05:53:59,394 INFO  [agent-shutdown-hook] 
> (org.apache.flume.instrumentation.MonitoredCounterGroup.stop:150)  - 
> Component type: SINK, name: sink_kinesis_stream stopped
> 24 Sep 2016 05:53:59,394 INFO  [agent-shutdown-hook] 
> (org.apache.flume.instrumentation.MonitoredCounterGroup.stop:156)  - Shutdown 
> Metric for type: SINK, name: sink_kinesis_stream. sink.start.time == 
> 1474685921906
> 
> Can anyone help me fix it? How can I do to stop this issue as our bussiness 
> already block because of this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 52216: Patch for FLUME-2647

2016-09-24 Thread Balázs Donát Bessenyei 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52216/#review150298
---



Do you think you could create a unit test for this fix?
Like triggering an error and checking if the message is correct


flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java
 (line 107)


This field seems to be never accessed.
Maybe it should be removed.


- Balázs Donát Bessenyei


On Sept. 24, 2016, 9:54 a.m., Umesh Chaudhary wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/52216/
> ---
> 
> (Updated Sept. 24, 2016, 9:54 a.m.)
> 
> 
> Review request for Flume.
> 
> 
> Repository: flume-git
> 
> 
> Description
> ---
> 
> Added two new booleans to identify the reason for closed channel: 1) 
> isClosedOnStart 2) isClosedNormally
> And checked them to print the appropriate reason for closed channel
> 
> 
> Diffs
> -
> 
>   
> flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java
>  9d82e43 
> 
> Diff: https://reviews.apache.org/r/52216/diff/
> 
> 
> Testing
> ---
> 
> Yes
> 
> 
> Thanks,
> 
> Umesh Chaudhary
> 
>

[jira] [Commented] (FLUME-2996) Flume process shutdown automaticly

2016-09-24 Thread JIRA 

[ 
https://issues.apache.org/jira/browse/FLUME-2996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15518792#comment-15518792
 ] 

Bessenyei Balázs Donát commented on FLUME-2996:
---

Hi [~hehailin],

Thank you for the bug report.

Can you please provide a little more information to help? (Such as your config 
file you are using, the environment you are trying to run Flume in, etc.)

Probably a few more lines from the log could be useful. (Any errors maybe?)

> Flume process shutdown automaticly
> --
>
> Key: FLUME-2996
> URL: https://issues.apache.org/jira/browse/FLUME-2996
> Project: Flume
>  Issue Type: Bug
> Environment: Amazon Linux
>Reporter: hailin
>
> hi guys,
> I install flume from apache-flume-1.6.0-bin.tar.gz, and startup flume with 
> some my custom configure file with cmd line like this: 
> bin/flume-ng agent -Dflume.monitoring.type=http -Dflume.monitoring.port=41414 
> -c conf -f conf/flume.conf --name a1 -Dflume.root.logger=INFO,DAILY 
> it works fine without any issue, but after several hours when I check the 
> process, I found it was shutdown automaticly. Bellow is the flume log 
> information:
> .
> 24 Sep 2016 05:53:59,391 INFO  [agent-shutdown-hook] 
> (org.apache.flume.lifecycle.LifecycleSupervisor.stop:79)  - Stopping 
> lifecycle supervisor 10
> 24 Sep 2016 05:53:59,394 INFO  [agent-shutdown-hook] 
> (org.apache.flume.instrumentation.MonitoredCounterGroup.stop:150)  - 
> Component type: SINK, name: sink_kinesis_stream stopped
> 24 Sep 2016 05:53:59,394 INFO  [agent-shutdown-hook] 
> (org.apache.flume.instrumentation.MonitoredCounterGroup.stop:156)  - Shutdown 
> Metric for type: SINK, name: sink_kinesis_stream. sink.start.time == 
> 1474685921906
> 
> Can anyone help me fix it? How can I do to stop this issue as our bussiness 
> already block because of this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 52216: Patch for FLUME-2647

2016-09-24 Thread Umesh Chaudhary 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52216/
---

(Updated Sept. 24, 2016, 9:54 a.m.)


Review request for Flume.


Changes
---

Incorporated Jeff's comments.


Repository: flume-git


Description
---

Added two new booleans to identify the reason for closed channel: 1) 
isClosedOnStart 2) isClosedNormally
And checked them to print the appropriate reason for closed channel


Diffs (updated)
-

  
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java
 9d82e43 

Diff: https://reviews.apache.org/r/52216/diff/


Testing
---

Yes


Thanks,

Umesh Chaudhary

[jira] [Commented] (FLUME-2811) Taildir source doesn't call stop() on graceful shutdown

2016-09-24 Thread Lior Zeno (JIRA) 

[ 
https://issues.apache.org/jira/browse/FLUME-2811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15518721#comment-15518721
 ] 

Lior Zeno commented on FLUME-2811:
--

Thanks [~umesh9...@gmail.com], go ahead and assign this issue to yourself.

> Taildir source doesn't call stop() on graceful shutdown
> ---
>
> Key: FLUME-2811
> URL: https://issues.apache.org/jira/browse/FLUME-2811
> Project: Flume
>  Issue Type: Bug
>  Components: Sinks+Sources
>Affects Versions: v1.7.0
>Reporter: Jun Seok Hong
>Priority: Critical
>  Labels: newbie
> Fix For: v1.7.0
>
>
> Taildir source doesn't call stop() on graceful shutdown.
> Test configuration.
> source - taildir
> channel - PseudoTxnMemoryChannel / flume-kafka-channel
> sink - none
> I found that flume sometimes doesn't terminate with Taildir source. 
> I had to kill the process to terminate it.
> tailFileProcess() function in TaildirSource.java has a infinite loop.
> When the process interrupted, ChannelException will happen, but it can't 
> breaks the infinite loop.
> I think that's the reason why Taildir can't call stop() function.
> {code:title=TaildirSource.java|borderStyle=solid}
>  private void tailFileProcess(TailFile tf, boolean backoffWithoutNL)
>   throws IOException, InterruptedException {
> while (true) {
>   reader.setCurrentFile(tf);
>   List events = reader.readEvents(batchSize, backoffWithoutNL);
>   if (events.isEmpty()) {
> break;
>   }
>   sourceCounter.addToEventReceivedCount(events.size());
>   sourceCounter.incrementAppendBatchReceivedCount();
>   try {
> getChannelProcessor().processEventBatch(events);
> reader.commit();
>   } catch (ChannelException ex) {
> logger.warn("The channel is full or unexpected failure. " +
>   "The source will try again after " + retryInterval + " ms");
> TimeUnit.MILLISECONDS.sleep(retryInterval);
> retryInterval = retryInterval << 1;
> retryInterval = Math.min(retryInterval, maxRetryInterval);
> continue;
>   }
>   retryInterval = 1000;
>   sourceCounter.addToEventAcceptedCount(events.size());
>   sourceCounter.incrementAppendBatchAcceptedCount();
>   if (events.size() < batchSize) {
> break;
>   }
> }
>   }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLUME-2996) Flume process shutdown automaticly

2016-09-24 Thread hailin (JIRA) 
hailin created FLUME-2996:
-

 Summary: Flume process shutdown automaticly
 Key: FLUME-2996
 URL: https://issues.apache.org/jira/browse/FLUME-2996
 Project: Flume
  Issue Type: Bug
 Environment: Amazon Linux
Reporter: hailin


hi guys,

I install flume from apache-flume-1.6.0-bin.tar.gz, and startup flume with some 
my custom configure file with cmd line like this: 
bin/flume-ng agent -Dflume.monitoring.type=http -Dflume.monitoring.port=41414 
-c conf -f conf/flume.conf --name a1 -Dflume.root.logger=INFO,DAILY 

it works fine without any issue, but after several hours when I check the 
process, I found it was shutdown automaticly. Bellow is the flume log 
information:

.
24 Sep 2016 05:53:59,391 INFO  [agent-shutdown-hook] 
(org.apache.flume.lifecycle.LifecycleSupervisor.stop:79)  - Stopping lifecycle 
supervisor 10
24 Sep 2016 05:53:59,394 INFO  [agent-shutdown-hook] 
(org.apache.flume.instrumentation.MonitoredCounterGroup.stop:150)  - Component 
type: SINK, name: sink_kinesis_stream stopped
24 Sep 2016 05:53:59,394 INFO  [agent-shutdown-hook] 
(org.apache.flume.instrumentation.MonitoredCounterGroup.stop:156)  - Shutdown 
Metric for type: SINK, name: sink_kinesis_stream. sink.start.time == 
1474685921906


Can anyone help me fix it? How can I do to stop this issue as our bussiness 
already block because of this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)