GitHub user uce opened a pull request:
https://github.com/apache/incubator-flink/pull/76
[FLINK-998] Close network connections when idle
1. **Close idle TCP connections**
Tasks enqueue envelopes to be dispatched via NettyConnectionManager. The
first task to enqueue an envelope triggers the establishment of a new
TCP connection to the target task manager. Usually, there are multiple
connections between the same task managers in order to avoid deadlocks.
This commit addresses the following problem: idle connections were never
closed. This meant that the number of TCP connections was monotonically
increasing over time. Idling connections are now closed after a
configured amount of time, see `taskmanager.net.closeAfterIdleForMs`.
2. **Enqueue envelopes directly**
TCP connections are handled by Netty's IO loop. Every enqueue operation
needs to ensure that the envelope to be enqueued is handed over to the
respective IO thread in a thread-safe manner. Previously, this was done
via Netty's user events.
This commit addresses the following problem: every enqueue operation
resulted in the creation of an IO event loop task to hand over the
envelope. Envelopes are now directly added to a `ConcurrentLinkedQueue`,
which improves the throughput of enqueue operations. The
user event is now only fired to trigger the processing if the queue was
empty.
3. **Added unit tests for OutboundConnectionQueue and adapted existing
tests to test concurrent enqueue and close**
4. **Removed unused arguments to NettyConnectionManager (low and high
water mark)**
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/uce/incubator-flink
FLINK-998-close_connections
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-flink/pull/76.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #76
----
commit 3636a450b47e93182e4fb710b3e9757723a3fc69
Author: uce <[email protected]>
Date: 2014-07-17T14:32:06Z
[FLINK-998] Close network connections when idle
1. Close idle TCP connections
Tasks enqueue envelopes to be dispatched via NettyConnectionManager. The
first task to enqueue an envelope triggers the establishment of a new
TCP connection to the target task manager. Usually, there are multiple
connections between the same task managers in order to avoid deadlocks.
This commit addresses the following problem: idle connections were never
closed. This meant that the number of TCP connections was monotonically
increasing over time. Idling connections are now closed after a
configured amount of time, see 'taskmanager.net.closeAfterIdleForMs'.
2. Enqueue envelopes directly
TCP connections are handled by Netty's IO loop. Every enqueue operation
needs to ensure that the envelope to be enqueued is handed over to the
respective IO thread in a thread-safe manner. Previously, this was done
via Netty's user events.
This commit addresses the following problem: every enqueue operation
resulted in the creation of an IO event loop task to hand over the
envelope. Envelopes are now directly added to a ConcurrentLinkedQueue,
which drastically improves the throughput of enqueue operations. The
user event is now only fired to trigger the processing if the queue was
empty.
3. Added unit tests for OutboundConnectionQueue and adapted existing
tests to test concurrent enqueue and close
4. Removed unused arguments to NettyConnectionManager (low and high
water mark)
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
