[
https://issues.apache.org/jira/browse/TS-1075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13270075#comment-13270075
]
B Wyatt commented on TS-1075:
-----------------------------
After lots of investigation, i think I can safely say this is "not our fault"
(I'll come back to this) nor is it really a TPROXY issue (though TPROXY is one
of the more "normal" ways of exercising the bug case).
You can replicate the error using a regular ip alias in linux attached to any
device. If you bind a socket to every available local port on the ip alias you
can no longer bind/connect/listen to ports on your original IP address. You
have effectively poisoned your bind table.
I can easily replicate a scenario where several thousand explicitly bound (with
auto-port) outgoing sockets will make an unbound ::connect on a different IP
fail as it does for traffic_cop.
I cannot _easily_ replicate a scenario where several thousand explicitly bound
(with auto-port) outgoing sockets with REUSEADDR set (as we do in TS) will
cause one of their own to fail. In fact, as long as you explicitly bind every
socket and use the REUSEADDR flag, then you should effectively "workaround" the
issue. The problem is that the bind table is a shared resource and other
processes that don't use the REUSEADDR flag can poison buckets until _all_
sockets with a given port, regardless of IP, close/expire. It is possible to
keep a bind bucket alive infinitely in the right circumstances, and an
extremely high traffic system would be a fine example when considering socket
TIME WAIT. This would allow a relatively innocuous and low frequency
non-REUSEADDR ::bind or ::connect to eventually poison the whole thing and
create a DoS for any auto-port ::binds regardless of IP space.
FWIW, it gets better in linux 3.3, when at the very least it semi-correctly
determine that there is no conflict for explicit ::binds with different IPs but
without universal REUSEADDR usage. However, my read on the 3.3 kernel is that
you can still muck up the table to the point where an unbound ::connect can
fail even if there are plenty of ports available for the route-determined
source IP.
... where does this leave us?
The patch is an "effective" workaround because the traffic cop no longer has an
unbound ::connect; as long is there no bind conflict it should succeed.
The random explicit ports is a workaround for poisoned bind buckets that
effectively "live forever" in a high traffic system and have been poisoned by a
non REUSEADDR ::bind somewhere along the way. I will probably end up not using
the time of day in favor of something cheaper like a multiply+carry RNG.
Alternatively, you could run linux 3.3+ (I'm only half kidding) and this half
of the bug would "fix" itself.
At least until 3.3+ becomes the stable norm (read: 10 years from now) we will
need some TS-level workaround. I may investigate what it would take to fix the
lingering ::connect bug in linux 3.3+, which would remove the need for
traffic_cop to explicitly bind (eventually).
yeesh
> Port range bottleneck in transparent proxy mode
> -----------------------------------------------
>
> Key: TS-1075
> URL: https://issues.apache.org/jira/browse/TS-1075
> Project: Traffic Server
> Issue Type: Bug
> Components: Core
> Affects Versions: 3.0.1
> Environment: Centos 5.6, kernel 2.6.39.2 compiled with TPROXY support
> ATS compiled as: ./configure --enable-tproxy
> Reporter: Danny Shporer
> Assignee: B Wyatt
> Fix For: 3.3.0
>
> Attachments: ports.patch
>
>
> The Linux TPROXY stack only takes into account the local addresses when using
> dynamic bind (bind without specifying a specific port). This limits the port
> range to only the local range (around 30K by default and can be extended to
> around 64K) - this together with the TIME-WAIT Linux method of releasing
> ports causes a bottleneck).
> One symptom of this is that traffic_cop cannot open a connection to the
> server to monitor it (it gets error 99 - address already in use) and kills
> it.
> Another issue is when opening the connection to the server.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
