[
https://issues.apache.org/jira/browse/AVRO-1538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Platon Vai updated AVRO-1538:
-----------------------------
Description:
NettyTransceiver created without channel factory results in file descriptors
leak while connecting with IOException (java.io.IOException: Error connecting
to, NettyTransceiver.java:280).
It seems very similar to https://issues.apache.org/jira/browse/FLUME-2221. But
I am not flume user.
To reproduce - run the test below (IP:9999 was nobody listened.)
In my enviroment (run from IDEA, openSuse 13.1 64bit, all system settings are
by default) stable fails after 146 cycles (count = 146).
{noformat}
@Test
public void testAvroNettyTest() {
for (int count = 0; count < 4500; count++) {
try {
NettyTransceiver apiClient = new NettyTransceiver(new
InetSocketAddress("<IP>", 9999), 5000L);
} catch (Exception exc) {
System.out.println(String.format("===========================
%d ===========================", count));
exc.printStackTrace();
if ( exc.getMessage().contains("Too many open files")
|| exc.getCause().getMessage().contains("Too many open
files") ) {
System.out.println("Found [Too many open files].");
break;
}
}
}
}
{noformat}
And test that OK
{noformat}
@Test
public void testAvroCustomNettyTest() throws Exception {
for (int count = 0; count < 4500; count++) {
NioClientSocketChannelFactory channelFactory = null;
ExecutorService bossExecutor = null, workerExecutor = null;
try {
bossExecutor = Executors.newCachedThreadPool(new
NettyTransceiverThreadFactory(
"Avro " + NettyTransceiver.class.getSimpleName() + "
Boss"));
workerExecutor = Executors.newCachedThreadPool(new
NettyTransceiverThreadFactory(
"Avro " + NettyTransceiver.class.getSimpleName() + "
I/O Worker"));
channelFactory = new NioClientSocketChannelFactory(
bossExecutor,
workerExecutor
);
NettyTransceiver apiClient = new NettyTransceiver(
new InetSocketAddress("<IP>", 9999),
channelFactory,
5000L);
} catch (Exception exc) {
System.out.println(String.format("===========================
%d ===========================", count));
exc.printStackTrace();
if ( channelFactory != null ) {
channelFactory.shutdown();
}
if ( exc.getMessage().contains("Too many open files")
|| exc.getCause().getMessage().contains("Too many open
files") ) {
System.out.println("Found [Too many open files].");
break;
}
}
}
}
{noformat}
So workaround exists it would be cool if NettyTransceiver in next release will
be more "friendly" (release all internal resources if initialization failed).
was:
NettyTransceiver created without channel factory results in file descriptors
leak while connecting with IOException (java.io.IOException: Error connecting
to, NettyTransceiver.java:280).
It seems very similar to https://issues.apache.org/jira/browse/FLUME-2221. But
I am not flume user.
To reproduce - run the test below (IP:9999 was nobody listened.)
In my enviroment (run from IDEA, openSuse 13.1 64bit, all system settings are
by default) stable fails after 146 cycles (count = 146).
<code>
@Test
public void testAvroNettyTest() {
for (int count = 0; count < 4500; count++) {
try {
NettyTransceiver apiClient = new NettyTransceiver(new
InetSocketAddress("<IP>", 9999), 5000L);
} catch (Exception exc) {
System.out.println(String.format("===========================
%d ===========================", count));
exc.printStackTrace();
if ( exc.getMessage().contains("Too many open files")
|| exc.getCause().getMessage().contains("Too many open
files") ) {
System.out.println("Found [Too many open files].");
break;
}
}
}
}
</code>
And test that OK
<code>
@Test
public void testAvroCustomNettyTest() throws Exception {
for (int count = 0; count < 4500; count++) {
NioClientSocketChannelFactory channelFactory = null;
ExecutorService bossExecutor = null, workerExecutor = null;
try {
bossExecutor = Executors.newCachedThreadPool(new
NettyTransceiverThreadFactory(
"Avro " + NettyTransceiver.class.getSimpleName() + "
Boss"));
</code>
workerExecutor = Executors.newCachedThreadPool(new
NettyTransceiverThreadFactory(
"Avro " + NettyTransceiver.class.getSimpleName() + "
I/O Worker"));
channelFactory = new NioClientSocketChannelFactory(
bossExecutor,
workerExecutor
);
NettyTransceiver apiClient = new NettyTransceiver(
new InetSocketAddress("<IP>", 9999),
channelFactory,
5000L);
} catch (Exception exc) {
System.out.println(String.format("===========================
%d ===========================", count));
exc.printStackTrace();
if ( channelFactory != null ) {
channelFactory.shutdown();
}
if ( exc.getMessage().contains("Too many open files")
|| exc.getCause().getMessage().contains("Too many open
files") ) {
System.out.println("Found [Too many open files].");
break;
}
}
}
}
So workaround exists it would be cool if NettyTransceiver in next release will
be more "friendly" (release all internal resources if initialization failed).
> NettyTransceiver descriptor leaks
> ---------------------------------
>
> Key: AVRO-1538
> URL: https://issues.apache.org/jira/browse/AVRO-1538
> Project: Avro
> Issue Type: Bug
> Components: java
> Affects Versions: 1.7.5, 1.7.6
> Environment: openSuse 13.1 64 bit, jdk1.7.0_51 64 bit
> Reporter: Platon Vai
>
> NettyTransceiver created without channel factory results in file descriptors
> leak while connecting with IOException (java.io.IOException: Error connecting
> to, NettyTransceiver.java:280).
> It seems very similar to https://issues.apache.org/jira/browse/FLUME-2221.
> But I am not flume user.
> To reproduce - run the test below (IP:9999 was nobody listened.)
> In my enviroment (run from IDEA, openSuse 13.1 64bit, all system settings are
> by default) stable fails after 146 cycles (count = 146).
> {noformat}
> @Test
> public void testAvroNettyTest() {
> for (int count = 0; count < 4500; count++) {
> try {
> NettyTransceiver apiClient = new NettyTransceiver(new
> InetSocketAddress("<IP>", 9999), 5000L);
> } catch (Exception exc) {
> System.out.println(String.format("===========================
> %d ===========================", count));
> exc.printStackTrace();
> if ( exc.getMessage().contains("Too many open files")
> || exc.getCause().getMessage().contains("Too many
> open files") ) {
> System.out.println("Found [Too many open files].");
> break;
> }
> }
> }
> }
> {noformat}
> And test that OK
> {noformat}
> @Test
> public void testAvroCustomNettyTest() throws Exception {
> for (int count = 0; count < 4500; count++) {
> NioClientSocketChannelFactory channelFactory = null;
> ExecutorService bossExecutor = null, workerExecutor = null;
> try {
> bossExecutor = Executors.newCachedThreadPool(new
> NettyTransceiverThreadFactory(
> "Avro " + NettyTransceiver.class.getSimpleName() + "
> Boss"));
> workerExecutor = Executors.newCachedThreadPool(new
> NettyTransceiverThreadFactory(
> "Avro " + NettyTransceiver.class.getSimpleName() + "
> I/O Worker"));
> channelFactory = new NioClientSocketChannelFactory(
> bossExecutor,
> workerExecutor
> );
> NettyTransceiver apiClient = new NettyTransceiver(
> new InetSocketAddress("<IP>", 9999),
> channelFactory,
> 5000L);
> } catch (Exception exc) {
> System.out.println(String.format("===========================
> %d ===========================", count));
> exc.printStackTrace();
> if ( channelFactory != null ) {
> channelFactory.shutdown();
> }
> if ( exc.getMessage().contains("Too many open files")
> || exc.getCause().getMessage().contains("Too many
> open files") ) {
> System.out.println("Found [Too many open files].");
> break;
> }
> }
> }
> }
> {noformat}
> So workaround exists it would be cool if NettyTransceiver in next release
> will be more "friendly" (release all internal resources if initialization
> failed).
--
This message was sent by Atlassian JIRA
(v6.2#6252)
