[jira] [Commented] (FLUME-2905) NetcatSource - Socket not closed when an exception is encountered during start() leading to file descriptor leaks

2016-05-12 Thread Siddharth Ahuja (JIRA) 

[ 
https://issues.apache.org/jira/browse/FLUME-2905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15282411#comment-15282411
 ] 

Siddharth Ahuja commented on FLUME-2905:


Thanks [~jarcec], attached a new patch, hopefully this should be enough:)

> NetcatSource - Socket not closed when an exception is encountered during 
> start() leading to file descriptor leaks
> -
>
> Key: FLUME-2905
> URL: https://issues.apache.org/jira/browse/FLUME-2905
> Project: Flume
>  Issue Type: Bug
>  Components: Sinks+Sources
>Affects Versions: v1.6.0
>Reporter: Siddharth Ahuja
>Assignee: Siddharth Ahuja
> Attachments: FLUME-2905-0.patch, FLUME-2905-1.patch
>
>
> During the flume agent start-up, the flume configuration containing the 
> NetcatSource is parsed and the source's start() is called. If there is an 
> issue while binding the channel's socket to a local address to configure the 
> socket to listen for connections following exception is thrown but the socket 
> open just before is not closed. 
> {code}
> 2016-05-01 03:04:37,273 ERROR org.apache.flume.lifecycle.LifecycleSupervisor: 
> Unable to start EventDrivenSourceRunner: { 
> source:org.apache.flume.source.NetcatSource{name:src-1,state:IDLE} } - 
> Exception follows.
> org.apache.flume.FlumeException: java.net.BindException: Address already in 
> use
> at org.apache.flume.source.NetcatSource.start(NetcatSource.java:173)
> at 
> org.apache.flume.source.EventDrivenSourceRunner.start(EventDrivenSourceRunner.java:44)
> at 
> org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
> at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
> at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.net.BindException: Address already in use
> at sun.nio.ch.Net.bind0(Native Method)
> at sun.nio.ch.Net.bind(Net.java:444)
> at sun.nio.ch.Net.bind(Net.java:436)
> at 
> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)
> at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
> at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:67)
> at org.apache.flume.source.NetcatSource.start(NetcatSource.java:167)
> ... 9 more
> {code}
> The source's start() is then called again leading to another socket being 
> opened but not closed and so on. This leads to file descriptor (socket) leaks.
> This can be easily reproduced as follows:
> 1. Set Netcat as the source in flume agent configuration.
> 2. Set the bind port for the netcat source to a port which is already in use. 
> e.g. in my case I used 50010 which is the port for DataNode's XCeiver 
> Protocol in use by the HDFS service.
> 3. Start flume agent and perform "lsof -p  | wc -l". Notice 
> the file descriptors keep on growing due to socket leaks with errors like: 
> "can't identify protocol".



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLUME-2905) NetcatSource - Socket not closed when an exception is encountered during start() leading to file descriptor leaks

2016-05-12 Thread Siddharth Ahuja (JIRA) 

 [ 
https://issues.apache.org/jira/browse/FLUME-2905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Siddharth Ahuja updated FLUME-2905:
---
Attachment: FLUME-2905-1.patch

> NetcatSource - Socket not closed when an exception is encountered during 
> start() leading to file descriptor leaks
> -
>
> Key: FLUME-2905
> URL: https://issues.apache.org/jira/browse/FLUME-2905
> Project: Flume
>  Issue Type: Bug
>  Components: Sinks+Sources
>Affects Versions: v1.6.0
>Reporter: Siddharth Ahuja
>Assignee: Siddharth Ahuja
> Attachments: FLUME-2905-0.patch, FLUME-2905-1.patch
>
>
> During the flume agent start-up, the flume configuration containing the 
> NetcatSource is parsed and the source's start() is called. If there is an 
> issue while binding the channel's socket to a local address to configure the 
> socket to listen for connections following exception is thrown but the socket 
> open just before is not closed. 
> {code}
> 2016-05-01 03:04:37,273 ERROR org.apache.flume.lifecycle.LifecycleSupervisor: 
> Unable to start EventDrivenSourceRunner: { 
> source:org.apache.flume.source.NetcatSource{name:src-1,state:IDLE} } - 
> Exception follows.
> org.apache.flume.FlumeException: java.net.BindException: Address already in 
> use
> at org.apache.flume.source.NetcatSource.start(NetcatSource.java:173)
> at 
> org.apache.flume.source.EventDrivenSourceRunner.start(EventDrivenSourceRunner.java:44)
> at 
> org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
> at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
> at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.net.BindException: Address already in use
> at sun.nio.ch.Net.bind0(Native Method)
> at sun.nio.ch.Net.bind(Net.java:444)
> at sun.nio.ch.Net.bind(Net.java:436)
> at 
> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)
> at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
> at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:67)
> at org.apache.flume.source.NetcatSource.start(NetcatSource.java:167)
> ... 9 more
> {code}
> The source's start() is then called again leading to another socket being 
> opened but not closed and so on. This leads to file descriptor (socket) leaks.
> This can be easily reproduced as follows:
> 1. Set Netcat as the source in flume agent configuration.
> 2. Set the bind port for the netcat source to a port which is already in use. 
> e.g. in my case I used 50010 which is the port for DataNode's XCeiver 
> Protocol in use by the HDFS service.
> 3. Start flume agent and perform "lsof -p  | wc -l". Notice 
> the file descriptors keep on growing due to socket leaks with errors like: 
> "can't identify protocol".



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLUME-2620) File channel throws NullPointerException if a header value is null

2016-05-12 Thread Neerja Khattar (JIRA) 

 [ 
https://issues.apache.org/jira/browse/FLUME-2620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Neerja Khattar updated FLUME-2620:
--
Attachment: FLUME-2620-3.patch

updated patch  attached

> File channel throws NullPointerException if a header value is null
> --
>
> Key: FLUME-2620
> URL: https://issues.apache.org/jira/browse/FLUME-2620
> Project: Flume
>  Issue Type: Bug
>  Components: File Channel
>Reporter: Santiago M. Mola
>Assignee: Neerja Khattar
> Attachments: FLUME-2620-0.patch, FLUME-2620-1.patch, 
> FLUME-2620-2.patch, FLUME-2620-3.patch, FLUME-2620.patch, FLUME-2620.patch
>
>
> File channel throws NullPointerException if a header value is null.
> If this is intended, it should be reported correctly in the logs.
> Sample trace:
> org.apache.flume.ChannelException: Unable to put batch on required channel: 
> FileChannel chan { dataDirs: [/var/lib/ingestion-csv/chan/data] }
>   at 
> org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:200)
>   at 
> org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(SpoolDirectorySource.java:236)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>   at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
>   at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
>   at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.NullPointerException
>   at 
> org.apache.flume.channel.file.proto.ProtosFactory$FlumeEventHeader$Builder.setValue(ProtosFactory.java:7415)
>   at org.apache.flume.channel.file.Put.writeProtos(Put.java:85)
>   at 
> org.apache.flume.channel.file.TransactionEventRecord.toByteBuffer(TransactionEventRecord.java:174)
>   at org.apache.flume.channel.file.Log.put(Log.java:622)
>   at 
> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:469)
>   at 
> org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)
>   at 
> org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)
>   at 
> org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:189)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 47098: FLUME-2620 File channel throws NullPointerException if a header value is null

2016-05-12 Thread neerja khattar 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47098/
---

(Updated May 13, 2016, 4:43 a.m.)


Review request for Flume.


Changes
---

fixed all the changes.


Repository: flume-git


Description
---

The issue is when the header value is null it throws null pointer exception and 
flume stops processing further events.
For example:
[{
  "headers" : {
 "timestamp" : "434324343",
 "host" : null
 },
  "body" : "random_body"
  }]
  
  The solution to fix this is:
  
  1. If the header has a null value in the json, flume will replace it with a 
replacement string.
  2. The default value for a replacement string is an empty string.
  3. To overwrite default string, set "handler.nullReplacementHeader" property 
in flume config.


Diffs (updated)
-

  flume-ng-core/src/main/java/org/apache/flume/source/http/HTTPSource.java 
b520b03 
  
flume-ng-core/src/main/java/org/apache/flume/source/http/HTTPSourceConfigurationConstants.java
 86caf7d 
  flume-ng-core/src/main/java/org/apache/flume/source/http/JSONHandler.java 
197f66a 
  
flume-ng-core/src/main/java/org/apache/flume/source/http/NullHeaderReplacement.java
 PRE-CREATION 
  flume-ng-core/src/test/java/org/apache/flume/source/http/TestJSONHandler.java 
455781c 

Diff: https://reviews.apache.org/r/47098/diff/


Testing
---

The following are the test cases:

1. Header has null value in json and handler.nullReplacementHeader is not set 
in flume config. The default value will be used to replace null.  

[{
  "headers" : {
 "timestamp" : "434324343",
 "host" : null
 },
  "body" : "random_body"
  }]
  
  Output in hdfs : {timestamp=434324343, host=} random_body  
  
  2. Header is not null in json and handler.nullReplacementHeader is not set in 
flume config. The replacement implementation doesnt come in to consideration.
   
  [{
  "headers" : {
 "timestamp" : "434324343",
 "host" : 1
 },
  "body" : "random_body"
  }]
  Output in hdfs : {timestamp=434324343, host=1} random_body 
  
  3. Header has null value in json and handler.nullReplacementHeader=abc is set 
in flume config. The null value in header will be replaced by abc.

  
  [{
  "headers" : {
 "timestamp" : "434324343",
 "host" : null
 },
  "body" : "random_body"
  }]
  
 
  Output in hdfs {timestamp=434324343, host=abc} random_body 
  
  4. Header has null value in json and handler.nullReplacementHeader=1 is set 
in flume config. The null value in header will be replaced by 1 as a string .
  
  [{
  "headers" : {
 "timestamp" : "434324343",
 "host" : null
 },
  "body" : "random_body"
  }]
  
 
  Output in hdfs: {timestamp=434324343, host=1} random_body
  
  5. Header is not null in json and handler.nullReplacementHeader is also set 
in flume config. The replacement implementation doesnt come in to consideration.
   
  [{
  "headers" : {
 "timestamp" : "434324343",
 "host" : 1
 },
  "body" : "random_body"
  }]
  Output in hdfs : {timestamp=434324343, host=1} random_body


File Attachments


flume-2620
  
https://reviews.apache.org/media/uploaded/files/2016/05/09/0eff1d56-caf3-4d36-bb45-6b9e7fd6a1ff__FLUME-2620-1.patch


Thanks,

neerja khattar

Re: Review Request 47098: FLUME-2620 File channel throws NullPointerException if a header value is null

2016-05-12 Thread Jarek Cecho 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47098/#review132968
---




flume-ng-core/src/main/java/org/apache/flume/source/http/HTTPSource.java (line 
153)


if(handler isinstanceof NullHeaderReplacement), right?



flume-ng-core/src/main/java/org/apache/flume/source/http/JSONHandler.java (line 
98)


Nit: Trailing whitespace.



flume-ng-core/src/main/java/org/apache/flume/source/http/NullHeaderReplacement.java
 (line 1)


Missing licence header.



flume-ng-core/src/test/java/org/apache/flume/source/http/TestJSONHandler.java 
(lines 161 - 163)


I believe that in the test we can be sure that we have JSONHandler, right?



flume-ng-core/src/test/java/org/apache/flume/source/http/TestJSONHandler.java 
(lines 174 - 176)


I believe that in the test we can be sure that we have JSONHandler, right?



flume-ng-core/src/test/java/org/apache/flume/source/http/TestJSONHandler.java 
(lines 214 - 216)


I believe that in the test we can be sure that we have JSONHandler, right?


- Jarek Cecho


On May 12, 2016, 1:03 a.m., neerja khattar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47098/
> ---
> 
> (Updated May 12, 2016, 1:03 a.m.)
> 
> 
> Review request for Flume.
> 
> 
> Repository: flume-git
> 
> 
> Description
> ---
> 
> The issue is when the header value is null it throws null pointer exception 
> and flume stops processing further events.
> For example:
> [{
>   "headers" : {
>  "timestamp" : "434324343",
>  "host" : null
>  },
>   "body" : "random_body"
>   }]
>   
>   The solution to fix this is:
>   
>   1. If the header has a null value in the json, flume will replace it with a 
> replacement string.
>   2. The default value for a replacement string is an empty string.
>   3. To overwrite default string, set "handler.nullReplacementHeader" 
> property in flume config.
> 
> 
> Diffs
> -
> 
>   flume-ng-core/src/main/java/org/apache/flume/source/http/HTTPSource.java 
> b520b03 
>   
> flume-ng-core/src/main/java/org/apache/flume/source/http/HTTPSourceConfigurationConstants.java
>  86caf7d 
>   flume-ng-core/src/main/java/org/apache/flume/source/http/JSONHandler.java 
> 197f66a 
>   
> flume-ng-core/src/main/java/org/apache/flume/source/http/NullHeaderReplacement.java
>  PRE-CREATION 
>   
> flume-ng-core/src/test/java/org/apache/flume/source/http/TestJSONHandler.java 
> 455781c 
> 
> Diff: https://reviews.apache.org/r/47098/diff/
> 
> 
> Testing
> ---
> 
> The following are the test cases:
> 
> 1. Header has null value in json and handler.nullReplacementHeader is not set 
> in flume config. The default value will be used to replace null.  
> 
> [{
>   "headers" : {
>  "timestamp" : "434324343",
>  "host" : null
>  },
>   "body" : "random_body"
>   }]
>   
>   Output in hdfs : {timestamp=434324343, host=} random_body  
>   
>   2. Header is not null in json and handler.nullReplacementHeader is not set 
> in flume config. The replacement implementation doesnt come in to 
> consideration.
>
>   [{
>   "headers" : {
>  "timestamp" : "434324343",
>  "host" : 1
>  },
>   "body" : "random_body"
>   }]
>   Output in hdfs : {timestamp=434324343, host=1} random_body 
>   
>   3. Header has null value in json and handler.nullReplacementHeader=abc is 
> set in flume config. The null value in header will be replaced by abc.
> 
>   
>   [{
>   "headers" : {
>  "timestamp" : "434324343",
>  "host" : null
>  },
>   "body" : "random_body"
>   }]
>   
>  
>   Output in hdfs {timestamp=434324343, host=abc} random_body 
>   
>   4. Header has null value in json and handler.nullReplacementHeader=1 is set 
> in flume config. The null value in header will be replaced by 1 as a string .
>   
>   [{
>   "headers" : {
>  "timestamp" : "434324343",
>  "host" : null
>  },
>   "body" : "random_body"
>   }]
>   
>  
>   Output in hdfs: {timestamp=434324343, host=1} random_body
>   
>   5. Header is not null in json and handler.nullReplacementHeader is also set 
> in flume config. The replacement implementation doesnt come in to 
> consideration.
>
>   [{
>   "headers" : {
>  "timestamp" : "434324343",
>  "host" : 1
>  },
>   "body" : "random_body"
>

[jira] [Commented] (FLUME-2905) NetcatSource - Socket not closed when an exception is encountered during start() leading to file descriptor leaks

2016-05-12 Thread Jarek Jarcec Cecho (JIRA) 

[ 
https://issues.apache.org/jira/browse/FLUME-2905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15281953#comment-15281953
 ] 

Jarek Jarcec Cecho commented on FLUME-2905:
---

Sure, my pleasure.  Quickly looking at the patch, can you please remove the 
trailing white space warnings [~sahuja]?

{code}
jarcec@arlene flume % git apply FLUME-2905-0.patch  


   
[trunk ✗] (1.8.0_77-b03) (ruby-2.2.3) [12:54:00]
FLUME-2905-0.patch:25: space before tab in indent.
.setNameFormat("netcat-handler-%d").build());
FLUME-2905-0.patch:50: trailing whitespace.

FLUME-2905-0.patch:52: trailing whitespace.
   * Tests that the source is stopped when an exception is thrown
FLUME-2905-0.patch:54: trailing whitespace.
   * clean up the sockets opened during source.start().
FLUME-2905-0.patch:55: trailing whitespace.
   *
warning: squelched 5 whitespace errors
warning: 10 lines add whitespace errors.
{code}

> NetcatSource - Socket not closed when an exception is encountered during 
> start() leading to file descriptor leaks
> -
>
> Key: FLUME-2905
> URL: https://issues.apache.org/jira/browse/FLUME-2905
> Project: Flume
>  Issue Type: Bug
>  Components: Sinks+Sources
>Affects Versions: v1.6.0
>Reporter: Siddharth Ahuja
>Assignee: Siddharth Ahuja
> Attachments: FLUME-2905-0.patch
>
>
> During the flume agent start-up, the flume configuration containing the 
> NetcatSource is parsed and the source's start() is called. If there is an 
> issue while binding the channel's socket to a local address to configure the 
> socket to listen for connections following exception is thrown but the socket 
> open just before is not closed. 
> {code}
> 2016-05-01 03:04:37,273 ERROR org.apache.flume.lifecycle.LifecycleSupervisor: 
> Unable to start EventDrivenSourceRunner: { 
> source:org.apache.flume.source.NetcatSource{name:src-1,state:IDLE} } - 
> Exception follows.
> org.apache.flume.FlumeException: java.net.BindException: Address already in 
> use
> at org.apache.flume.source.NetcatSource.start(NetcatSource.java:173)
> at 
> org.apache.flume.source.EventDrivenSourceRunner.start(EventDrivenSourceRunner.java:44)
> at 
> org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
> at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
> at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.net.BindException: Address already in use
> at sun.nio.ch.Net.bind0(Native Method)
> at sun.nio.ch.Net.bind(Net.java:444)
> at sun.nio.ch.Net.bind(Net.java:436)
> at 
> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)
> at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
> at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:67)
> at org.apache.flume.source.NetcatSource.start(NetcatSource.java:167)
> ... 9 more
> {code}
> The source's start() is then called again leading to another socket being 
> opened but not closed and so on. This leads to file descriptor (socket) leaks.
> This can be easily reproduced as follows:
> 1. Set Netcat as the source in flume agent configuration.
> 2. Set the bind port for the netcat source to a port which is already in use. 
> e.g. in my case I used 50010 which is the port for DataNode's XCeiver 
> Protocol in use by the HDFS service.
> 3. Start flume agent and perform "lsof -p  | wc -l". Notice 
> the file descriptors keep on growing due to socket leaks with errors like: 
> "can't identify protocol".



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLUME-2905) NetcatSource - Socket not closed when an exception is encountered during start() leading to file descriptor leaks

2016-05-12 Thread Grant Henke (JIRA) 

[ 
https://issues.apache.org/jira/browse/FLUME-2905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15281789#comment-15281789
 ] 

Grant Henke commented on FLUME-2905:


[~jarcec] Would you have time to review this? 

> NetcatSource - Socket not closed when an exception is encountered during 
> start() leading to file descriptor leaks
> -
>
> Key: FLUME-2905
> URL: https://issues.apache.org/jira/browse/FLUME-2905
> Project: Flume
>  Issue Type: Bug
>  Components: Sinks+Sources
>Affects Versions: v1.6.0
>Reporter: Siddharth Ahuja
>Assignee: Siddharth Ahuja
> Attachments: FLUME-2905-0.patch
>
>
> During the flume agent start-up, the flume configuration containing the 
> NetcatSource is parsed and the source's start() is called. If there is an 
> issue while binding the channel's socket to a local address to configure the 
> socket to listen for connections following exception is thrown but the socket 
> open just before is not closed. 
> {code}
> 2016-05-01 03:04:37,273 ERROR org.apache.flume.lifecycle.LifecycleSupervisor: 
> Unable to start EventDrivenSourceRunner: { 
> source:org.apache.flume.source.NetcatSource{name:src-1,state:IDLE} } - 
> Exception follows.
> org.apache.flume.FlumeException: java.net.BindException: Address already in 
> use
> at org.apache.flume.source.NetcatSource.start(NetcatSource.java:173)
> at 
> org.apache.flume.source.EventDrivenSourceRunner.start(EventDrivenSourceRunner.java:44)
> at 
> org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
> at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
> at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.net.BindException: Address already in use
> at sun.nio.ch.Net.bind0(Native Method)
> at sun.nio.ch.Net.bind(Net.java:444)
> at sun.nio.ch.Net.bind(Net.java:436)
> at 
> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)
> at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
> at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:67)
> at org.apache.flume.source.NetcatSource.start(NetcatSource.java:167)
> ... 9 more
> {code}
> The source's start() is then called again leading to another socket being 
> opened but not closed and so on. This leads to file descriptor (socket) leaks.
> This can be easily reproduced as follows:
> 1. Set Netcat as the source in flume agent configuration.
> 2. Set the bind port for the netcat source to a port which is already in use. 
> e.g. in my case I used 50010 which is the port for DataNode's XCeiver 
> Protocol in use by the HDFS service.
> 3. Start flume agent and perform "lsof -p  | wc -l". Notice 
> the file descriptors keep on growing due to socket leaks with errors like: 
> "can't identify protocol".



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: How to contribute?

2016-05-12 Thread Hari Shreedharan 
We don't really need a vote, I think. We just need someone to work on
getting merge scripts in place (we could copy it from one of the
existing projects, like Spark).

On Thu, May 12, 2016 at 7:06 AM, Jeff Holoman  wrote:
> Speaking of this, didn't we decide to moe to PR's?
>
> Do we need a formal vote for this?
>
> Thanks
>
> Jeff
>
> On Wed, May 11, 2016 at 5:16 PM, Hari Shreedharan 
> wrote:
>
>> Hi Lior,
>>
>> Flume does not use PRs yet for reviews. Please attach the patch to the
>> jira and someone will get to it. Optionally, you can also submit the
>> patch on review board for review (though you must attach the patch on
>> the jira as well).
>>
>> See this as well:
>> https://cwiki.apache.org/confluence/display/FLUME/How+to+Contribute
>>
>> HTH.
>>
>>
>>
>> On Wed, May 11, 2016 at 8:48 AM, Lior Zeno  wrote:
>> > Hi All,
>> >
>> > I would like to make code contributions to the project, however I'm not
>> > sure how.
>> > I have sent a PR (FLUME-2726) on github a few days ago, but never got a
>> > reply.
>> >
>> > Please let me know how to properly get started.
>> >
>> > Thanks
>>
>
>
>
> --
> Jeff Holoman

Re: How to contribute?

2016-05-12 Thread Jeff Holoman 
Speaking of this, didn't we decide to moe to PR's?

Do we need a formal vote for this?

Thanks

Jeff

On Wed, May 11, 2016 at 5:16 PM, Hari Shreedharan 
wrote:

> Hi Lior,
>
> Flume does not use PRs yet for reviews. Please attach the patch to the
> jira and someone will get to it. Optionally, you can also submit the
> patch on review board for review (though you must attach the patch on
> the jira as well).
>
> See this as well:
> https://cwiki.apache.org/confluence/display/FLUME/How+to+Contribute
>
> HTH.
>
>
>
> On Wed, May 11, 2016 at 8:48 AM, Lior Zeno  wrote:
> > Hi All,
> >
> > I would like to make code contributions to the project, however I'm not
> > sure how.
> > I have sent a PR (FLUME-2726) on github a few days ago, but never got a
> > reply.
> >
> > Please let me know how to properly get started.
> >
> > Thanks
>



-- 
Jeff Holoman

Re: How to contribute?

2016-05-12 Thread Lior Zeno 
Thank you. I attached a patch to the relevant JIRA issue.

How can I contribute to the next upcoming release? Most of the issues that
are tagged with "Fix version: 1.7.0" are in a "Patch available" status but
are still open.

On Thu, May 12, 2016 at 12:16 AM, Hari Shreedharan 
wrote:

> Hi Lior,
>
> Flume does not use PRs yet for reviews. Please attach the patch to the
> jira and someone will get to it. Optionally, you can also submit the
> patch on review board for review (though you must attach the patch on
> the jira as well).
>
> See this as well:
> https://cwiki.apache.org/confluence/display/FLUME/How+to+Contribute
>
> HTH.
>
>
>
> On Wed, May 11, 2016 at 8:48 AM, Lior Zeno  wrote:
> > Hi All,
> >
> > I would like to make code contributions to the project, however I'm not
> > sure how.
> > I have sent a PR (FLUME-2726) on github a few days ago, but never got a
> > reply.
> >
> > Please let me know how to properly get started.
> >
> > Thanks
>

[jira] [Updated] (FLUME-2726) Remove * imports from Kafka Channel

2016-05-12 Thread Lior Zeno (JIRA) 

 [ 
https://issues.apache.org/jira/browse/FLUME-2726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lior Zeno updated FLUME-2726:
-
Attachment: FLUME-2726-0.patch

> Remove * imports from Kafka Channel
> ---
>
> Key: FLUME-2726
> URL: https://issues.apache.org/jira/browse/FLUME-2726
> Project: Flume
>  Issue Type: Bug
>  Components: Channel
>Affects Versions: v1.6.0
>Reporter: Ashish Paliwal
>Priority: Trivial
> Attachments: FLUME-2726-0.patch
>
>
> As per Flume dev guide, we prefer explicit imports. Need to remove * imports 
> and replace with explicit imports



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)