simonchi created FLUME-2688:
-------------------------------
Summary: the avro sink in sinkgroup will never close the
connection when process failed
Key: FLUME-2688
URL: https://issues.apache.org/jira/browse/FLUME-2688
Project: Flume
Issue Type: Bug
Components: Sinks+Sources
Affects Versions: v1.5.2
Environment: flume 1.5.2 two flume client and two flume
ha(load_balance-->round_bin)
Reporter: simonchi
169,177 two app server with flume agent to collect log and send them to remote
ha flume by avro sink to connect with avro source.
84,88 two ha flume to receive the log from 169,177 by avro source to connect
now when the time going on ,the connection between app and ha flume become more
and more,it should be two connection in every ha flume ,ri ght ?
but actually,it is not, as follows:
[root@usercenter ~]# netstat -ano | grep 9520
tcp 0 0 :::9520 :::*
LISTEN off (0.00/0/0)
tcp 0 0 ::ffff:172.30.129.84:9520 ::ffff:172.30.128.169:26371
ESTABLISHED off (0.00/0/0)
tcp 0 0 ::ffff:172.30.129.84:9520 ::ffff:172.30.128.177:43354
ESTABLISHED off (0.00/0/0)
tcp 0 0 ::ffff:172.30.129.84:9520 ::ffff:172.30.128.177:44055
ESTABLISHED off (0.00/0/0)
tcp 0 0 ::ffff:172.30.129.84:9520 ::ffff:172.30.128.177:18355
ESTABLISHED off (0.00/0/0)
tcp 0 0 ::ffff:172.30.129.84:9520 ::ffff:172.30.128.177:19968
ESTABLISHED off (0.00/0/0)
tcp 0 0 ::ffff:172.30.129.84:9520 ::ffff:172.30.128.169:21389
ESTABLISHED off (0.00/0/0)
tcp 0 0 ::ffff:172.30.129.84:9520 ::ffff:172.30.128.169:39635
ESTABLISHED off (0.00/0/0)
tcp 0 0 ::ffff:172.30.129.84:9520 ::ffff:172.30.128.169:56417
ESTABLISHED off (0.00/0/0)
tcp 0 0 ::ffff:172.30.129.84:9520 ::ffff:172.30.128.169:26416
ESTABLISHED off (0.00/0/0)
tcp 0 0 ::ffff:172.30.129.84:9520 ::ffff:172.30.128.169:63654
ESTABLISHED off (0.00/0/0)
tcp 0 0 ::ffff:172.30.129.84:9520 ::ffff:172.30.128.169:28258
ESTABLISHED off (0.00/0/0)
tcp 0 0 ::ffff:172.30.129.84:9520 ::ffff:172.30.128.177:29226
ESTABLISHED off (0.00/0/0)
tcp 0 0 ::ffff:172.30.129.84:9520 ::ffff:172.30.128.169:51033
ESTABLISHED off (0.00/0/0)
tcp 0 0 ::ffff:172.30.129.84:9520 ::ffff:172.30.128.177:21432
ESTABLISHED off (0.00/0/0)
tcp 0 0 ::ffff:172.30.129.84:9520 ::ffff:172.30.128.169:25367
ESTABLISHED off (0.00/0/0)
tcp 0 0 ::ffff:172.30.129.84:9520 ::ffff:172.30.128.177:16682
ESTABLISHED off (0.00/0/0)
tcp 0 0 ::ffff:172.30.129.84:9520 ::ffff:172.30.128.177:22288
ESTABLISHED off (0.00/0/0)
tcp 0 0 ::ffff:172.30.129.84:9520 ::ffff:172.30.128.169:23637
ESTABLISHED off (0.00/0/0)
tcp 0 0 ::ffff:172.30.129.84:9520 ::ffff:172.30.128.177:12397
ESTABLISHED off (0.00/0/0)
tcp 0 0 ::ffff:172.30.129.84:9520 ::ffff:172.30.128.169:40406
ESTABLISHED off (0.00/0/0)
tcp 0 0 ::ffff:172.30.129.84:9520 ::ffff:172.30.128.177:48131
ESTABLISHED off (0.00/0/0)
tcp 0 0 ::ffff:172.30.129.84:9520 ::ffff:172.30.128.177:39718
ESTABLISHED off (0.00/0/0)
tcp 0 0 ::ffff:172.30.129.84:9520 ::ffff:172.30.128.177:26650
ESTABLISHED off (0.00/0/0)
tcp 0 0 ::ffff:172.30.129.84:9520 ::ffff:172.30.128.169:32709
ESTABLISHED off (0.00/0/0)
tcp 0 0 ::ffff:172.30.129.84:9520 ::ffff:172.30.128.177:13135
ESTABLISHED off (0.00/0/0)
tcp 0 0 ::ffff:172.30.129.84:9520 ::ffff:172.30.128.177:28231
ESTABLISHED off (0.00/0/0)
tcp 0 0 ::ffff:172.30.129.84:9520 ::ffff:172.30.128.169:14285
ESTABLISHED off (0.00/0/0)
tcp 0 0 ::ffff:172.30.129.84:9520 ::ffff:172.30.128.177:19528
ESTABLISHED off (0.00/0/0)
tcp 0 0 ::ffff:172.30.129.84:9520 ::ffff:172.30.128.177:29502
ESTABLISHED off (0.00/0/0)
tcp 0 0 ::ffff:172.30.129.84:9520 ::ffff:172.30.128.169:46403
ESTABLISHED off (0.00/0/0)
tcp 0 0 ::ffff:172.30.129.84:9520 ::ffff:172.30.128.177:55529
ESTABLISHED off (0.00/0/0)
tcp 0 0 ::ffff:172.30.129.84:9520 ::ffff:172.30.128.169:32832
ESTABLISHED off (0.00/0/0)
tcp 0 0 ::ffff:172.30.129.84:9520 ::ffff:172.30.128.169:14920
ESTABLISHED off (0.00/0/0)
tcp 0 0 ::ffff:172.30.129.84:9520 ::ffff:172.30.128.169:30913
ESTABLISHED off (0.00/0/0)
tcp 0 0 ::ffff:172.30.129.84:9520 ::ffff:172.30.128.177:55971
ESTABLISHED off (0.00/0/0)
tcp 0 0 ::ffff:172.30.129.84:9520 ::ffff:172.30.128.177:46014
ESTABLISHED off (0.00/0/0)
tcp 0 0 ::ffff:172.30.129.84:9520 ::ffff:172.30.128.169:29924
ESTABLISHED off (0.00/0/0)
tcp 0 0 ::ffff:172.30.129.84:9520 ::ffff:172.30.128.169:60584
ESTABLISHED off (0.00/0/0)
tcp 0 0 ::ffff:172.30.129.84:9520 ::ffff:172.30.128.177:51096
ESTABLISHED off (0.00/0/0)
tcp 0 0 ::ffff:172.30.129.84:9520 ::ffff:172.30.128.169:14947
ESTABLISHED off (0.00/0/0)
tcp 0 0 ::ffff:172.30.129.84:9520 ::ffff:172.30.128.177:27936
ESTABLISHED off (0.00/0/0)
tcp 0 0 ::ffff:172.30.129.84:9520 ::ffff:172.30.128.177:33280
ESTABLISHED off (0.00/0/0)
tcp 0 0 ::ffff:172.30.129.84:9520 ::ffff:172.30.128.177:20794
ESTABLISHED off (0.00/0/0)
tcp 0 0 ::ffff:172.30.129.84:9520 ::ffff:172.30.128.169:16101
ESTABLISHED off (0.00/0/0)
tcp 0 0 ::ffff:172.30.129.84:9520 ::ffff:172.30.128.169:49969
ESTABLISHED off (0.00/0/0)
tcp 0 0 ::ffff:172.30.129.84:9520 ::ffff:172.30.128.177:45663
ESTABLISHED off (0.00/0/0)
tcp 0 0 ::ffff:172.30.129.84:9520 ::ffff:172.30.128.169:18501
ESTABLISHED off (0.00/0/0)
tcp 0 0 ::ffff:172.30.129.84:9520 ::ffff:172.30.128.169:63662
ESTABLISHED off (0.00/0/0)
tcp 0 0 ::ffff:172.30.129.84:9520 ::ffff:172.30.128.177:18487
ESTABLISHED off (0.00/0/0)
tcp 0 0 ::ffff:172.30.129.84:9520 ::ffff:172.30.128.177:34131
ESTABLISHED off (0.00/0/0)
tcp 0 0 ::ffff:172.30.129.84:9520 ::ffff:172.30.128.177:49777
ESTABLISHED off (0.00/0/0)
tcp 0 0 ::ffff:172.30.129.84:9520 ::ffff:172.30.128.169:26972
ESTABLISHED off (0.00/0/0)
tcp 0 0 ::ffff:172.30.129.84:9520 ::ffff:172.30.128.169:24546
ESTABLISHED off (0.00/0/0)
tcp 0 0 ::ffff:172.30.129.84:9520 ::ffff:172.30.128.177:48143
ESTABLISHED off (0.00/0/0)
tcp 0 0 ::ffff:172.30.129.84:9520 ::ffff:172.30.128.177:59868
ESTABLISHED off (0.00/0/0)
tcp 0 0 ::ffff:172.30.129.84:9520 ::ffff:172.30.128.177:26420
ESTABLISHED off (0.00/0/0)
tcp 0 0 ::ffff:172.30.129.84:9520 ::ffff:172.30.128.169:39200
ESTABLISHED off (0.00/0/0)
tcp 0 0 ::ffff:172.30.129.84:9520 ::ffff:172.30.128.169:21346
ESTABLISHED off (0.00/0/0)
tcp 0 0 ::ffff:172.30.129.84:9520 ::ffff:172.30.128.169:29591
ESTABLISHED off (0.00/0/0)
tcp 0 0 ::ffff:172.30.129.84:9520 ::ffff:172.30.128.177:35161
ESTABLISHED off (0.00/0/0)
so ,I think ,in the source code
while (sinkIterator.hasNext()) {
Sink sink = sinkIterator.next();
try {
status = sink.process();
break;
} catch (Exception ex) {
selector.informSinkFailed(sink);
LOGGER.warn("Sink failed to consume event. "
+ "Attempting next sink if available.", ex);
}
}
in catch block,you should add a logic :
if the sink is avro sink,you should close the old connection when it create a
new connection.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
