Re: [iccrg] Bufferbloat and Congestion Control

[Tom Herbert]

On Fri, Mar 31, 2017 at 3:18 PM, Joe Touch  wrote:
>
>
> On 3/31/2017 7:47 AM, Jim Gettys wrote:
>> One full size packet @ 1mbps == 13 milliseconds.
> That sort of blocking is its own problem and the only solution is to use
> smaller packets.
>
> It can be solved at TCP by pushing the MSS down, but for UDP that's
> another good reason for considering UDP options as a TSVWG WG item!
>
I'm not sure how much having MSS for UDP solves. MSS doesn't give us
path MTU, we still need some PMTU discovery in TCP and presumably need
this for connected UDP.

Tom

> Joe
>

Re: [Int-area] Middleboxes to aid the deployment of MPTCP

[Tom Herbert]

On Tue, Jul 18, 2017 at 3:31 PM, Joe Touch  wrote:
> Hi, all,
>
> I've noted this before, but to share with other areas:
>
> Although I'm not averse to middleboxes as optional optimizations, I find
> the proposed mechanisms aren't quite optional -- they inject option
> information into the SYN data. That information would poison a
> connection to a legacy receiver if (more to the point, when) that info
> isn't removed by a proxy upstream of the receiver.
>
> TCP must be E2E and fall back to legacy endpoints without a reconnection
> attempt, as required by RFC793.
>
> These aren't generic solutions; they're attacks on a TCP connection, IMO.
>
I agree. This seems be akin to stateful firewalls model that impose
artificial requirements on networking (like every TCP packet for a
connection must got through some middlebox or the connection is
dropped). We need to move things back to E2E semantics for transport
protocols-- nodes that try to maintain transport state in the network
should be considered the problem not the solution!

Tom

> Joe
>
>
> On 7/18/2017 3:22 PM, Olivier Bonaventure wrote:
>> The working group has discussed this issue several times and today we
>> have presented a new design that supports the creation of such
>> functions to convert MPTCP connections into TCP connections and vice
>> versa. The design was done with MPTCP in mind, but the proposed
>> solution could be more generic and applied to other use cases than
>> MPTCP. The draft that describes the new design is available via:
>>
>> https://www.ietf.org/internet-drafts/draft-bonaventure-mptcp-converters-01.txt
>>
>>
>> Mirja Kuehlewind suggested to send the document on the int-area and
>> tsv-area mailing lists to see whether other working groups could be
>> interested by this approach.
>
> ___
> Int-area mailing list
> int-a...@ietf.org
> https://www.ietf.org/mailman/listinfo/int-area

Re: [Int-area] Middleboxes to aid the deployment of MPTCP

[Tom Herbert]

On Wed, Jul 19, 2017 at 1:46 AM,   wrote:
> Hi Erik,
>
> That's the intuitive approach to follow but unfortunately the situation is 
> not that obvious to get into.
>
I can give a little background on the Linux situation. There have been
several attempts to get MPTCP into the stack over the past few years.
Each time the patches were rejected primary because of complexity and
invasiveness. I believe in the initial attempt there were something
like 7,000 LOC change to tcp and that was whittled down to 5K on the
next attempt IIRC. It is not impossible to get the code in the stack,
but it's going to take more work to minimize code change and convince
the maintainer the complexity is justified.

> Network providers do not have a control on the servers and the terminals that 
> are enabled by customers behind the CPE. So making use of MPTCP to grab more 
> resources (when available) or to provide better resiliency (when a network 
> attachment is lost) will require both endpoints to be MPTCP-capable.
>
Network providers don't control CPE is either. I know iOS has support
for MPTCP, AFAIK Android does not (Erik is that correct?). If Android
were shipping it that would be a strong datapoint towards getting
support in Linux.

Tom

Re: [Int-area] Middleboxes to aid the deployment of MPTCP

[Tom Herbert]

On Thu, Jul 20, 2017 at 8:26 AM,   wrote:
> Re-,
>
> Please see inline.
>
> Cheers,
> Med
>
>> -Message d'origine-
>> De : Joe Touch [mailto:to...@isi.edu]
>> Envoyé : jeudi 20 juillet 2017 16:37
>> À : BOUCADAIR Mohamed IMT/OLN; Olivier Bonaventure; Internet Area; tsv-
>> a...@ietf.org
>> Objet : Re: [Int-area] Middleboxes to aid the deployment of MPTCP
>>
>>
>>
>> On 7/19/2017 10:39 PM, mohamed.boucad...@orange.com wrote:
>> > Hi Joe,
>> >
>> > The text can always be worked out. This is not an IETF LC :)
>> Agreed - I was explaining that the current text - and many of the
>> responses in this chain, both from you and others, continues to be
>> confusing.
>>
> [Med] Calling out those confusing points would help.
>
>> > The main point is that we are following your suggestion to define the
>> solution as an application proxy using a dedicated port number.
>> As feedback for the doc, that could be made more explicit, including in
>> the title (this is an application proxy, not typically referred to as
>> middleboxes).
>>
>
> [Med] Looks like a good idea to me.
>
That use of the term middleboxes as opposed to proxy was also
confusing to me. I'd also suggest not to call this a netwotk function,
that has other connotations that don't apply here.

Tom

>> That also includes limiting the doc to using TCP app-layer APIs and
>> describing any behavior that *might* happen at the segment level as just
>> that - hypothetical, perhaps desired, but NOT the mechanism being
>> proposed.
>>
>> FWIW.
>>
>> Joe
>
> ___
> Int-area mailing list
> int-a...@ietf.org
> https://www.ietf.org/mailman/listinfo/int-area

Re: [Int-area] Middleboxes to aid the deployment of MPTCP

[Tom Herbert]

On Tue, Jul 18, 2017 at 11:37 PM,  <mohamed.boucad...@orange.com> wrote:
> Hi Tom,
>
> Please see inline.
>
> Cheers,
> Med
>
>> -Message d'origine-
>> De : Int-area [mailto:int-area-boun...@ietf.org] De la part de Tom Herbert
>> Envoyé : mercredi 19 juillet 2017 00:43
>> À : Joe Touch
>> Cc : tsv-area@ietf.org; Internet Area
>> Objet : Re: [Int-area] Middleboxes to aid the deployment of MPTCP
>>
>> On Tue, Jul 18, 2017 at 3:31 PM, Joe Touch <to...@isi.edu> wrote:
>> > Hi, all,
>> >
>> > I've noted this before, but to share with other areas:
>> >
>> > Although I'm not averse to middleboxes as optional optimizations, I find
>> > the proposed mechanisms aren't quite optional -- they inject option
>> > information into the SYN data. That information would poison a
>> > connection to a legacy receiver if (more to the point, when) that info
>> > isn't removed by a proxy upstream of the receiver.
>> >
>> > TCP must be E2E and fall back to legacy endpoints without a reconnection
>> > attempt, as required by RFC793.
>> >
>> > These aren't generic solutions; they're attacks on a TCP connection,
>> IMO.
>> >
>> I agree. This seems be akin to stateful firewalls model that impose
>> artificial requirements on networking (like every TCP packet for a
>> connection must got through some middlebox or the connection is
>> dropped). We need to move things back to E2E semantics for transport
>> protocols-- nodes that try to maintain transport state in the network
>> should be considered the problem not the solution!
>>
>
> [Med] We would love to avoid requiring adding a function in the network to 
> assist devices to make use of their multiple interfaces/paths. Nevertheless, 
> the situation is that servers do not support MPTCP.
>
> The proposed solution allows to:
> * elect some flows to benefit from network-assisted MPTCP while avoiding 
> session failures.
> * Policies are configured to identify traffic that won't be forwarded via a 
> proxy
> * 0-RTT proxying
> * MPTCP can be used e2e if both end points are MPTCP-capable (that is, the 
> proxy is withdrawn from the path).
> * No interference with TCP options to be negotiated between endpoints.
>
> Do you have in mind such use cases when you referred to the "stateful 
> firewalls model"?

One of the problems with stateful firewalls (as well as transparent
proxies) is that they require all packets for a flow to consistently
traverse the same middlebox device. The middlebox is not addressed by
the packet, so the assumed requirement is that packets for the flow go
though the same network node by virtue of routing. For a firewall in a
single-homed site to the Internet this works because the firewall is
the only egress/ingress point to the network. If a site is
multi-homed, then the hope is that routing of packets in both
directions will go through the same point. But there is absolutely no
requirement that network layer packets for a flow are always routed
the same way, and if routing does change and packets need to flow
through different points the session breaks.

If I'm reading the draft correctly, then it has the same property of
maintaining transport state in the network so it implies the same
requirement that all packets for a flow follow the same path.
Personally,  would find it ironic that a a protocol to allow muli-path
transport would require the network layer to be single path.

Tom

>
>> Tom
>>
>> > Joe
>> >
>> >
>> > On 7/18/2017 3:22 PM, Olivier Bonaventure wrote:
>> >> The working group has discussed this issue several times and today we
>> >> have presented a new design that supports the creation of such
>> >> functions to convert MPTCP connections into TCP connections and vice
>> >> versa. The design was done with MPTCP in mind, but the proposed
>> >> solution could be more generic and applied to other use cases than
>> >> MPTCP. The draft that describes the new design is available via:
>> >>
>> >> https://www.ietf.org/internet-drafts/draft-bonaventure-mptcp-
>> converters-01.txt
>> >>
>> >>
>> >> Mirja Kuehlewind suggested to send the document on the int-area and
>> >> tsv-area mailing lists to see whether other working groups could be
>> >> interested by this approach.
>> >
>> > ___
>> > Int-area mailing list
>> > int-a...@ietf.org
>> > https://www.ietf.org/mailman/listinfo/int-area
>>
>> ___
>> Int-area mailing list
>> int-a...@ietf.org
>> https://www.ietf.org/mailman/listinfo/int-area

Re: Progressing the MPTCP proxy work

[Tom Herbert]

On Wed, Sep 13, 2017 at 8:01 AM,   wrote:
> Hi,
>
> At the IETF we discussed (in MPTCP WG) a new proposal on MPTCP proxy, “0-RTT
> TCP converters”
> https://tools.ietf.org/html/draft-bonaventure-mptcp-converters  This seemed
> to get a good reception, and had taken on board the feedback about the
> “plain mode” draft.
>
> We believe the next steps are:
>
> • WGs – TCPM & MPTCP - please send any further initial comments,
> especially if you believe it hasn’t dealt with technical concerns that were
> raised about the “plain mode” draft, or if you have fresh concerns
>
Hello,

I have some concerns about both the motivation and solution described
in the draft.

The primary motivation stated for this protocol seems to be that
"deploying those extensions on both a wide range of clients and
servers remains difficult". That's a pretty general and subjective
rationalization. The argument that things take too long to deploy can
be applied to nearly any protocol; for instance, a similar argument
was recently given in 6man to justify the need for IPv10 since IPv6 is
taking too long to deploy. The typical problems with this argument are
three fold: 1) it presumes that development and deployment of an
interim solution will take less time than to deploy the protocol being
fixed 2) it doesn't address the root cause of why a protocol is taking
long to deploy in the first place 3) the interim solution likely
creates new problems and incompatibilities that need to be dealt with.
For this draft I think these should be considered in the specific
context of MPTCP.

For #1 the assumption, the key assertion in the draft is "There are
some situations where the transport stack used on clients (resp.
servers) can be upgraded at a faster pace than the transport stack
running on servers (resp. clients)."-- I think the reality of this is
quite debatable. Major content provider providers that make up the
bulk Internet traffic, such as Google and Facebook, upgrade their
front end web server software at a much faster rate than clients
typically do. There a good examples, TFO for instance, where there was
support for new TCP features on servers long before clients. The other
part of this is that the solution assumes client support of MPTCP.
AFAIK Android (which represents a huge number of clients) does not
ship with an MPTCP implementation even though its been part of IOS
since 2013. So, the real question that should be asked is why isn't
MPTCP being deployed which leads to problem #2.

I suspect a major reason that MPTCP is not deployed in Android or
servers is that fact that upstream Linux has not adopted the
implementation. There was an effort a while back to get it into the
stack, however there was pushback because of the invasiveness of the
code (it was 1000's of lines of core TCP code change). Unfortunately,
the developers didn't followup and continue working on reducing the
complexity of implementation. I think with enough persistence in
addressing the feedback the code would eventually make it in. Even so,
acceptance into Linux is not a requirement for deployment. Google and
Facebook could be running MPTCP on their servers, and Android could
shipped with MPTCP support. I suspect the reason they're not doing
that is motivation. If they were convinced that MPTCP is a great value
to users, they can make deployment happen quickly.

For #3, the problems of the interim solution need to be elaborated.
TCP is an end to end protocol, and we know that whenever a middlebox
attempts to transparently process TCP some things will break. As
section 5 states "The Converter protocol was designed to be used in
networks that do not contain middleboxes that interfere with TCP." --
the irony of this statement is that the converter itself becomes a
middlebox that interferes with TCP. The converter has many of the same
issues of other middlebox solutions (proxies, firewalls) in the
network that are invasive at the transport layer. For instance E2E
security (e.g. TCP-AO) is likely broken. Also, I have to wonder about
the case where the client and server support an option that is unknown
to the converter-- say for instance that an experimental option is
being used for a new feature such as we did with TFO. In this case I
don't see how the converter can deal with the option, passing it
through between MPTCP and non-MPTCP may or may not be correct. So
probably the only reasonable behavior is to drop unknown options. But,
then that means the endpoints would be bound to using only TCP options
that the converter supports, so ultimately the converter is an
obstacle not a facilitator for deploying new options. Perhaps the
biggest irony of this proposal is that the converter is stateful for a
connection, so all packets must flow through a single point in the
network. Like stateful firewalls, this inevitably becomes a bottleneck
and a single point of failure. I believe that is nearly the exact
opposite of what MPTCP endeavors to 

Re: [multipathtcp] Progressing the MPTCP proxy work

[Tom Herbert]

On Mon, Sep 18, 2017 at 2:18 AM, Olivier Bonaventure
 wrote:
> Tom,
>>
>>
>> I disagree, discussions about deployment and implementation are in
>> scope. The primary argument for necessity that this draft makes is
>> that MPTCP is being deployed too slowly.
>
>
> The main argument of the draft is that deployment on client and servers does
> not go at the same pace. This is the motivation for having a converter.
> There *is* significant deployement of Multipath TCP today on clients and in
> some specific use cases.
>
> If we ignore the very large providers like Google or Facebook, deployment of

Olivier,

Why ignore the large content providers? If you want MPTCP to be a
success, these (e.g. FANG) are exactly who you should be engaging.
Once they're on board that covers a lot of user experience and others
are likely to follow their lead.

> new TCP features on servers takes more time. This does not depend only on
> the availability of a particular feature in the mainline Linux kernel but on
> many other factors. Server administrators are usually rather conservative
> and they favor stability and only deploy new features when they are strictly
> required. Looking at operating systems, they also tend to deploy stable
> releases that have been well tested and rarely deploy cutting edge features.
> Table 2 in
>
> https://irtf.org/anrw/2017/anrw17-final16.pdf
>
> shows that TFO has similar deployment issues than MPTCP. It has been enabled
> on client devices (Linux, iOS/Macos, soon Windows), but Brian Trammel and
> his colleagues could only find 578 servers (IP address) in Oct 2016 and 866
> in Jan 2017 that negotiated TFO. Most of them where in a single AS.
>
That says nothing about MPTCP. I think you're making broad
generalizations about deployment based on a few select data points.
Not all TCP features take a long time to get significant deployment
(e.g., ICW=10, congestion avoidance improvements, retrans.
improvements). And more than that, QUIC is entirely new transport
protocol that has gotten significant deployment in a relatively short
amount of time. The particulars of each protocol or feature need to be
considered if a perceived deployment issue is to be addressed.

TFO and MPTCP differ in some critical ways. TFO has good kernel
support, but the server applications need to be fixed to support it.
This dependency makes deployment longer. Also, TFO is a nice feature,
but is only relevant at the beginning of a connection, and until
zero-RTT TLS gets deployed it is of limited value. MPTCP does not
currently have kernel support (i.e. not accepted by Linux), however
shouldn't require server application layer changes-- the latter
characteristic is an important simplification. This means if there
were good kernel support then when the servers are updated (in cycles
of 2-3 yrs. for a major content provider) they will get support and
the value of MPTCP without additional work or action. Honestly, had
the support gotten into Linux fours years ago when that was proposed
we probably wouldn't be having this conversation!

>
>> If we are to consider that
>> argument then we need to understand _why_ deployment of MPTCP is slow,
>
>
> This is true for any TCP extension. Client and servers migrate at their own
> pace and it takes many years to deploy any TCP extension. MPTCP is not
> different from RFC1323, SACK or TFO
>
As I said above, the deployment and implementation characteristics of
all of these is different and need to be considered for each feature.

>> so the details about current deployment state and implementation are
>> pertinent. Also, while there is significant engineering needed to get
>> MPTCP into Linux or other systems, we cannot ignore the significant
>> (possibly more) engineering effort to define, standardize, develop and
>> deploy an interim solution on clients and converters.
>
>
> This cost will be on the devices that will benefit from the extension. The
> deployment of MPTCP in Korea to bond WiFi and LTE shows that there is a
> benefit for this type of service.
>
Please elaborate on the cost of this solution. Specifically, please
explain why the cost of doing this interim solution is less than the
cost of figuring out how to get servers to support MPTCP. Also, the
cost analysis should take into account any negative effects on nodes
outside of the devices being touched-- this solution is not
transparent to the outside world (similar to how NAT isn't really
transparent to end hosts).

Thanks,
Tom

Re: [multipathtcp] Progressing the MPTCP proxy work

[Tom Herbert]

> From the viewpoint of the enduser, there is a benefit in using MPTCP on her
> smartphone because she can combine LTE and WiFi to achieve higher bandwidth
> or have seamless handovers. From the viewpoint of the network operator, this
> is an added value service that they provide to their customers. If all
> Internet servers supported MPTCP, they would not have had to deploy and
> support the SOCKS servers in their network.
>
Olivier,

The benefits of using MPTCP are understood. The question is cost of
this solution. As you mention above MPTCP requires client and server
cooperation, but this solution requires cooperation between clients
and middleboxes, and middlebox and server. The number of cooperating
parties is not be reduced, and effectively turns TCP connection
negotiation into a three-way negotiation. The server/middlebox
interaction may be mostly transparent, but not the former. The clients
and middlebox support requires new protocol which means new software
needs to be deployed on clients and middleboxes. This takes time and
engineering effort. And there are a lot more network carriers than
there are significant content providers or client vendor, so that is
more parties to deal with in deployment. Given this, please provide
the details on why this path is going to be faster than working
directly with the content providers to start support MPTCP. WIth the
right motivation and effort, I believe that most major content
providers could have good support for MPTCP within 3 years. What is
your time and effort estimate for deploying this solution across all
clients and bringing up the converter in a significant number of
networks?

Tom

>> Also, the
>> cost analysis should take into account any negative effects on nodes
>> outside of the devices being touched-- this solution is not
>> transparent to the outside world (similar to how NAT isn't really
>> transparent to end hosts).
>
>
> The negative effect is that the smartphone needs to include MPTCP and the
> SOCKS client. The SOCKS server is a single-point of failure inside the ISP
> network. It is not totally transparent since the smartphone needs to be
> configured with a SOCKS server, but there are clear benefits since this kind
> of solution is deployed by several ISPs in several countries.
>
>
> Olivier

Re: [multipathtcp] Progressing the MPTCP proxy work

[Tom Herbert]

On Mon, Sep 18, 2017 at 1:06 PM, Yoshifumi Nishida
 wrote:
> Hi Tom,
>
> Only a few companies can control both client and server sides.
> However, ISPs might be able to control the STB at the client side and the
> middleboxes in their networks.
> This may be a relatively easy way to deploy MPTCP technology rather than
> updating clients or servers.

Yoshi,

I think you're focusing too much on the benefits of this solution and
not considering the cost. We've seen time and time again that when
middleboxes get involved in transport layer operations they break the
end to end nature of TCP and that leads to problems. Middlebox
involvement in TCP is one of the major source of protocol ossification
on the Internet. MPTCP is just one feature of TCP that we might want
do deploy there are many others. If this solution hampers use and
deployment of those, then I don't believe this is a reasonable
tradeoff regardless of what the benefits are.

Tom

Re: [multipathtcp] Progressing the MPTCP proxy work

[Tom Herbert]

On Sun, Sep 17, 2017 at 1:24 PM, Olivier Bonaventure
 wrote:
> Tom,
>
> Thanks for your comments.
>
>> For #1 the assumption, the key assertion in the draft is "There are
>> some situations where the transport stack used on clients (resp.
>> servers) can be upgraded at a faster pace than the transport stack
>> running on servers (resp. clients)."-- I think the reality of this is
>> quite debatable. Major content provider providers that make up the
>> bulk Internet traffic, such as Google and Facebook, upgrade their
>> front end web server software at a much faster rate than clients
>> typically do.
>
>
> Major content providers are one part of the Internet, but not the entire
> Internet. Many companies are still using old servers running Linux 2.x
>
>> There a good examples, TFO for instance, where there was
>> support for new TCP features on servers long before clients.
>
>
> Looking at TFO servers for example, a recent paper showed that TFO is
> enabled on only 0.1% of the web sites
>
> https://irtf.org/anrw/2017/anrw17-final16.pdf
>
> and 80% of these sites are part of Google's AS.
>
>> The other
>> part of this is that the solution assumes client support of MPTCP.
>> AFAIK Android (which represents a huge number of clients) does not
>> ship with an MPTCP implementation even though its been part of IOS
>> since 2013. So, the real question that should be asked is why isn't
>> MPTCP being deployed which leads to problem #2.
>
>
> In Korea, Samsung and LG have deployed MPTCP on various types of Android
> smartphones.
>
>> I suspect a major reason that MPTCP is not deployed in Android or
>> servers is that fact that upstream Linux has not adopted the
>> implementation. There was an effort a while back to get it into the
>> stack, however there was pushback because of the invasiveness of the
>> code (it was 1000's of lines of core TCP code change). Unfortunately,
>> the developers didn't followup and continue working on reducing the
>> complexity of implementation. I think with enough persistence in
>> addressing the feedback the code would eventually make it in.
>
>
> I agree, there is engineering effort to bring MPTCP in the mainline Linux
> kernel, but this is outside the scope of IETF.
>
Oliver,

I disagree, discussions about deployment and implementation are in
scope. The primary argument for necessity that this draft makes is
that MPTCP is being deployed too slowly. If we are to consider that
argument then we need to understand _why_ deployment of MPTCP is slow,
so the details about current deployment state and implementation are
pertinent. Also, while there is significant engineering needed to get
MPTCP into Linux or other systems, we cannot ignore the significant
(possibly more) engineering effort to define, standardize, develop and
deploy an interim solution on clients and converters. In the end, if
the problem can be solved without creating a new complex and invasive
protocol that is the better path forward.

Tom

>> Even so,
>> acceptance into Linux is not a requirement for deployment. Google and
>> Facebook could be running MPTCP on their servers, and Android could
>> shipped with MPTCP support. I suspect the reason they're not doing
>> that is motivation. If they were convinced that MPTCP is a great value
>> to users, they can make deployment happen quickly.
>
>
> Apple is using MPTCP on both their clients and their servers. Starting with
> iOS11, all applications will be able to use MPTCP. This is a reasonable
> deployment AFAIK. Korea is another example.
>>
>>
>> For #3, the problems of the interim solution need to be elaborated.
>> TCP is an end to end protocol, and we know that whenever a middlebox
>> attempts to transparently process TCP some things will break. As
>> section 5 states "The Converter protocol was designed to be used in
>> networks that do not contain middleboxes that interfere with TCP." --
>> the irony of this statement is that the converter itself becomes a
>> middlebox that interferes with TCP. The converter has many of the same
>> issues of other middlebox solutions (proxies, firewalls) in the
>> network that are invasive at the transport layer.
>
>
> Agreed, but note that the converter was designed to enable the client to
> detect when the server supports the required extension (e.g. MPTCP) so that
> it can stop using the converter to reach this particular server. This means
> that as a particular extension gets deployed, the utilisation of the
> converter will diminish.
>
>> For instance E2E
>> security (e.g. TCP-AO) is likely broken.
>
>
> If a client requests TCP-AO, then it's clear that this TCP connection should
> not go through a converter.
>
>> Also, I have to wonder about
>> the case where the client and server support an option that is unknown
>> to the converter-- say for instance that an experimental option is
>> being used for a new feature such as we did with TFO. In this case I
>> don't see how the converter can deal with 

Re: statement regarding keepalives

[Tom Herbert]

On Wed, Aug 15, 2018, 1:56 PM Kent Watsen  wrote:

>
> Hi Tom,
>
> I recall you're mentioning NAT before.  It fell into a crack and I
> lost sight of it.
>
> You bring up an interesting point, it goes to the motivation for
> wanting to do keepalives in the first place.  The text doesn't
> yet mention maintain flow state as a motivation.
>
> The first paragraph of the "keepalives" section says:
>
>   When the initiator of a networking session needs to maintain a
>   long-lived connection, it is necessary for it to periodically test
>   the aliveness of the remote device.
>
> Would it make sense to adjust it to say the following?
>
>   When the initiator of a networking session needs to maintain a
>   long-lived connection, it is necessary for it to periodically
>   ensure network accessibility to and test the aliveness of the
>   remote device.  For instance, without keepalive, an intermediate
>   NAT or firewalls may evict the flow state for quiet connections
>   due to a timeout or least recently used policy.  Similarly, the
>   remote application process, while accessible, may be hung, thus
>   accounting for the reason why the connection is quiet.
>
>
>
> Regarding your other comment, that the discussion should "include
> considerations on the frequency of keepalives and their cost", it
> seems that you almost wrote the paragraph below.  Would you be
> willing to proffer some formal text we could paste in, maybe to
> the end of the "keepalives" section or another section?  If not,
> I can try to take a stab at it.
>

Kent, I'm not sure what the context of formal text is. Is this write up
going to be in an I-D, or is it intended to be published by some other
mechanism?

Tom


>
> Thanks,
> Kent
>
>
>
> = original message =
>
> I think the statement is missing a primary purpose of keepalives,
> maybe the most important one, which to maintain flow state in NAT and
> firewalls and prevent eviction by timeout or LRU.
>
> Also, any meaningful discussion or statement about keepalives should
> include considerations on the frequency of keepalives and their cost.
>
> Keepalives themselves carry no meaningful end user data, they are
> purely management overhead. The higher the frequency of keepalives,
> the higher the overhead and hence the more network resources they
> consume. At some point they can become a source of congestion,
> especially when keepalive timers become synchronized across a network
> as I previously pointed out. Unfortunately, there is no standard for
> how NAT state eviction is done and no standard NAT timeout, so the
> frequency of keepalives to prevent NAT state eviction is probably
> higher than it should be (hence more network overhead).
>
> In terms of cost, consider the effects of waking up the transmitter on
> a smart phone periodically just for the purpose of keeping connections
> up. With a high enough frequency this will drain the battery quickly.
> In fact, one of the touted benefits of IPv6 was supposed to be that
> NAT isn't present so there is no need for periodic keepalives to
> maintain NAT state and hence this would conserve power on mobile
> devices. Use of keepalives in power constrained devices is a real
> issue.
>
> Tom
>
> >
>
>
>

Re: statement regarding keepalives

[Tom Herbert]

On Wed, Aug 15, 2018 at 2:24 PM, Kent Watsen  wrote:
> Hi Tom,
>
>
>
>> Kent, I'm not sure what the context of formal text is. Is this write up
>> going
>
>> to be in an I-D, or is it intended to be published by some other
>> mechanism?
>
>
>
> That is a good question.  At first, we were thinking that is might be an
> AD-level
>
> statement, but I think Spencer last suggested it being put into an I-D that
> might
>
> be published by TVGWG, if the chairs were to support that idea.
>
Kent,

There's little cost for individuals to create and post
Internet-Drafts, and you can always ask later for adoption as a
working group item. I think this will be a lot easier to work with
once it's in I-D form. Also, that exercise should help to clarify
exactly what the intent of this work is. To me, it seems like the
intent is to make recommendations on protocol design concerning
keepalives, so I believe that is reasonable to target an informational
RFC or possibly even a BCP.

Tom

>
>
> For now, I'm just pecking away at the text.  I figure that how to publish it
>
> will make more sense the more we know what "it" is.  Does that work for
>
> you?
>
Thanks. I think this will be easier to work with as draft. I suspect
there's also some additional background material on keepalives
>
>
> Kent

Re: statement regarding keepalives

[Tom Herbert]

On Thu, Aug 16, 2018 at 8:01 AM, Mikael Abrahamsson  wrote:
> On Thu, 16 Aug 2018, Tom Herbert wrote:
>
>> They are already on, TCP has a default keepalive for 2 hrs. The issue
>
>
> http://tldp.org/HOWTO/TCP-Keepalive-HOWTO/usingkeepalive.html says:
>
> "Remember that keepalive support, even if configured in the kernel, is not
> the default behavior in Linux."
>
> Which OSes default to this on?
>
Linux, but I misread the code. The default keepalive timeout is 2
hrs., but it needs to be enabled per socket.

Tom

>
> --
> Mikael Abrahamssonemail: swm...@swm.pp.se

Re: statement regarding keepalives

[Tom Herbert]

On Thu, Aug 16, 2018 at 12:44 AM, Olle E. Johansson  wrote:
>
>
> On 16 Aug 2018, at 09:28, Mikael Abrahamsson  wrote:
>
> On Wed, 15 Aug 2018, Kent Watsen wrote:
>
> You bring up an interesting point, it goes to the motivation for wanting to
> do keepalives in the first place.  The text doesn't yet mention maintain
> flow state as a motivation.
>
>
> It's not only to maintain flow state, it's also to close the connection when
> the network goes down and doesn't work anymore, and "give up" on connections
> that doesn't work anymore (for some definition of "anymore").
>
> I have operationally been in the situation where a server/client application
> was implemented so that the server could only handle 256 connections (some
> filedescriptor limit). Every time the firewall was rebooted, lost state, the
> connection hung around forever. So the server administrators had to go in
> and restart the process to clear these connections, otherwise there were 256
> hung connections and no new connections could be established.
>
> Sometimes the other endpoint goes down, and doesn't come back. We will for
> instance deploy home gateways probably keeping netconf-call-home sessions to
> an NMS, and we want them to be around forever, as long as they work. TCP
> level keepalives would solve this, as if the customer just powers off the
> device, after a while the session will be cleared. Using TCP keepalives here
> means you get this kind of behaviour even if the upper-layer application
> doesn't support it (netconf might have been a bad example here). It's a
> single socket option to set, so it's very easy to do.
>
> From knowing approximately what settings people have in their NAT44 and
>
> firewalls etc, I'd say the recommendation should be that keepalives are set
> to around 60-300 second interval, and then kill the connection if no traffic
> has passed in 3-5 of these intervals, kill the connection. Otherwise TCP
> will have backed off so far anyway, that it's probably faster to just re-try
> the connection instead of waiting for TCP to re-send the packet.
>
> I have seen so many times in my 20 years working in networking where lack of
> keepalives have caused all kinds of problems. I wish everybody would turn it
> on and keep it on.
>
Olle,

They are already on, TCP has a default keepalive for 2 hrs. The issue
that is inevitably raised is that 2 hrs. is much too long a period for
maintaining NAT state (NAT timeouts are usuallu far less time). But,
as I pointed out already, sending keepalives at a higher frequency is
not devoid of cost nor problems.

Tom

>
> As more and more connections flow over mobile networks, it seems more and
> more important, even for flows you did not expect. I have to send keepalives
> over IPv6 connections - not for NAT as on IPv4. but for middlebox devices
> that has an interesting approach and attitude towards connection management.
> ;-)
>
> The SIP Outbound RFC has a lot of reasoning behind keep-alives for
> connection failover and may be good input here.
>
> https://tools.ietf.org/html/rfc5626
>
> /O

Re: statement regarding keepalives

[Tom Herbert]

On Fri, Aug 17, 2018 at 10:27 AM, Joe Touch  wrote:
>
>
>
>
> On 2018-08-17 09:05, Tom Herbert wrote:
>
> On Fri, Aug 17, 2018 at 7:40 AM, Joe Touch  wrote:
>
>
> ...
> It's not subtle. There's no way to know whether keepalives at a higher level
> have any desired affect at the lower level at all - except using Wireshark
> to trace the packets sent.
>
> I don't think that's necessarily true. RFC1122 states:
>
> "Keep-alive packets MUST only be sent when no data or acknowledgement
> packets have been received for the connection within an interval."
>
>
>
> That's Sec 4.2.3.6. and it's talking about what TCP does inside TCP.
>
> It's not talking about actions by layers above TCP. For all TCP knows, a
> user might have tried to send data that's been hung up in the OS. There's
> simply no specific way to know that anything above TCP causes TCP to do
> anything per se; even if an upper layer protocol does a TCP_SEND() directly,
> TCP might stall that data because of other things going on.
>
>
> So if an application is performing keepalives by sending and receiving
> keepalive messages over the connection then that is enough to supress
> TCP keepalives.
>
>
>
> That may or may not be true, but it's for TCP to decide for itself. If the
> data isn't getting down to TCP in a way that causes TCP to send data before
> a TCP keepalive timer expires, TCP will - and should - send a keepalive. If
> the data does cause that timer to be reset, then that's for TCP to know.
>
>
> For instance, if the period of application sending
> keepalives on a connection is less then the one for TCP keepalives,
> then there should be no TCP keepalives ever sent on the connection (if
> Wireshark is showing otherwise then that might be a bug in the
> implementation).
>
>
> Consider an app that writes 1GB to TCP every day. If TCP sends that out
> slowly (for whatever reason), it's possible no TCP keepalives will ever be
> sent. An app that thinks it's doing TCP a favor by sending an app keepalive
> every 1.9 hrs (just under the 2 hour default config) would simply be causing
> TCP to do unnecessary work.
>
The purpose of an application keep alive is not to do favors for TCP,
it's to verify the end to end liveness between application end points.
This is at a much higher layer, verifying liveness of the TCP
connection is a side effect.

> However, if that 1GB goes out in 10 seconds, then TCP would have sent its
> own keepalives just fine. It didn't need the app's help.
>
> So the app didn't help at all; at best, it does nothing and at worst it
> hurts.

Consider that someone sets an application keepalive to 35 second
interval and the TCP keepalive timer is 30 seconds. When the
connection goes idle TCP keepalive will fire at thirty seconds, and
five seconds later the application keepalive fires. So every
thirty-five seconds two keepalives are done at two layers. This is not
good as it wastes network resources and power. In this case, the
application keepalive is sufficient and the TCP keepalive shouldn't be
used. This is an example of the problems in running two control loops
at different layers with overlapping functionality, if the
ramifications of doing aren't understood it can lead to undesirable
interactions and behavior.

Tom

>
> Joe
>
>

Re: statement regarding keepalives

[Tom Herbert]

On Fri, Aug 17, 2018 at 7:40 AM, Joe Touch  wrote:
>
>
>> On Aug 16, 2018, at 3:57 PM, Benjamin Kaduk  wrote:
>>
>> On Thu, Aug 16, 2018 at 03:52:54PM -0700, Joe Touch wrote
>
>>>
>>> On Aug 16, 2018, at 3:10 PM, Benjamin Kaduk  wrote:
>>>
> Keepalives at a layer SHOULD NOT be interpreted as implying state at
> any other layer.

 What's going on here in the last sentence is probably a bit subtle -- a
 keeaplive both does not indicate "real" protocol activity but also can
 serve to exercise the lower protocol layers (and, even, per the previous
 sentence, suppresses their keepalives).
>>>
>>> That may be intended but is never actually known. Lower layers can 
>>> compress, cache, merge, and otherwise change the effect a transmission st 
>>> one layer has on any other.
>>
>> Right, that's why it's subtle :)
>
> It’s not subtle. There’s no way to know whether keepalives at a higher level 
> have any desired affect at the lower level at all - except using Wireshark to 
> trace the packets sent.
>
I don't think that's necessarily true. RFC1122 states:

"Keep-alive packets MUST only be sent when no data or acknowledgement
packets have been received for the connection within an interval."

So if an application is performing keepalives by sending and receiving
keepalive messages over the connection then that is enough to supress
TCP keepalives. For instance, if the period of application sending
keepalives on a connection is less then the one for TCP keepalives,
then there should be no TCP keepalives ever sent on the connection (if
Wireshark is showing otherwise then that might be a bug in the
implementation).

Tom

> That’s why users SHOULD NOT try to affect lower level keepalives using higher 
> level ones.  (it’s not MUST NOT because there’s no strict harm, except that 
> it simply can’t be known whether it achieved its desired effect).
>
> Joe

Re: statement regarding keepalives

[Tom Herbert]

On Fri, Aug 17, 2018 at 4:06 PM, Joe Touch  wrote:
>
>
>
> On 2018-08-17 14:13, Tom Herbert wrote:
>
> On Fri, Aug 17, 2018 at 1:31 PM, Joe Touch  wrote:
>
>
>
> If you KNOW that the app keepalive will cause the TCP transmission, sure -
> but how do you KNOW that? You don't and can't. Even if you write to the TCP
> socket, all you know when the socket returns is that the data was copied to
> the kernel. You don't know for sure that you've triggered a TCP packet.
>
> Actually, you do know that information. Application keepalives are
> request/response messages sent in TCP data. When a response is
> received to keepalive request over the TCP connection that is proof
> that the keepalive was sent.
>
>
> Yes, but the keepalive itself is a guarantee of nothing. It is the keepalive
> ACK that matters.
>
>
>
> If the application keepalive was sent on
> the socket, and no response is received before the application timer
> expires, then the application declares the the connection dead and
> most likely will just close the socket and try to reconnect. The fact
> that an application keepalive request, or its response, might be stuck
> in a TCP send buffer (e.g. peer rcv window is zero) versus the peer
> host completely disappeared is irrelevant. To the application it's all
> the same, a connection to a peer application has failed and action
> needs to be taken.
>
>
> In that case it never mattered whether TCP had a keepalive or whether the
> app action interacted with that TCP keepalive.
>
> What mattered was that the app-app communication was maintained.
>
> Again, this reiterates my point - run keepalives at the layer that matter to
> you. Ignore how they affect other layers; they will (and should) take care
> of themselves.

Joe,

I agree to the extent that keepalives are run only at one layer with
one keepalive control loop. If they are simultaneously done at
multiple layers without consideration that can be problematic. It's
also probably a good general recommendation to do keepalives only at
the application (highest layer) if at all possible. Most modern
application protocols support them (HTTP, RPC protocols, etc). There
are still protocols like telnet ssh would need TCP keepalives. TCP
keepalives are a weaker signal and have become increasingly prone to
false positives. For instance, if a connection goes through a
transparent proxy, a TCP keepalive would only verify the liveness of a
connection to the proxy, not TCP all the way to the peer unbeknownst
to the user.

Tom

>
> Joe
>
>
>
>

Re: statement regarding keepalives

[Tom Herbert]

On Fri, Aug 17, 2018 at 1:31 PM, Joe Touch  wrote:
>
>
>
>
> On 2018-08-17 11:43, Tom Herbert wrote:The purpose of an application keep
> alive is not to do favors for TCP,
>
> it's to verify the end to end liveness between application end points.
> This is at a much higher layer, verifying liveness of the TCP
> connection is a side effect.
>
>
> Sure - that's fine and not what I'm concerned about.
>
> I don't want the text to say that higher level protocols or apps should try
> to do favors to keepalive lower level protocols - because it doesn't
> necessarily work.
>
>
>
>
> However, if that 1GB goes out in 10 seconds, then TCP would have sent its
> own keepalives just fine. It didn't need the app's help.
>
> So the app didn't help at all; at best, it does nothing and at worst it
> hurts.
>
>
> Consider that someone sets an application keepalive to 35 second
> interval and the TCP keepalive timer is 30 seconds. When the
> connection goes idle TCP keepalive will fire at thirty seconds, and
> five seconds later the application keepalive fires. So every
> thirty-five seconds two keepalives are done at two layers. This is not
> good as it wastes network resources and power.
>
>
> Agreed.
>
>
> In this case, the
> application keepalive is sufficient
>
>
> In this *implementation* it *might* be sufficient, in others, it might not.
> There's simply no way for the layers to know.
>
>
> and the TCP keepalive shouldn't be
> used.
>
>
> If you KNOW that the app keepalive will cause the TCP transmission, sure -
> but how do you KNOW that? You don't and can't. Even if you write to the TCP
> socket, all you know when the socket returns is that the data was copied to
> the kernel. You don't know for sure that you've triggered a TCP packet.
>
Actually, you do know that information. Application keepalives are
request/response messages sent in TCP data. When a response is
received to keepalive request over the TCP connection that is proof
that the keepalive was sent. If the application keepalive was sent on
the socket, and no response is received before the application timer
expires, then the application declares the the connection dead and
most likely will just close the socket and try to reconnect. The fact
that an application keepalive request, or its response, might be stuck
in a TCP send buffer (e.g. peer rcv window is zero) versus the peer
host completely disappeared is irrelevant. To the application it's all
the same, a connection to a peer application has failed and action
needs to be taken.

Tom

> Besides, your "keepalives" might end up causing TCP to send packets it never
> needed to send in the first place - even IF you think you're doing it a
> favor.
>
>
>
> This is an example of the problems in running two control loops
> at different layers with overlapping functionality,
>
>
>
> The problem is trying to infer overlap in functionality. If you realize that
> these are independent control loops *and leave them alone* you're fine.
>
Independence of control loops does not mean they can't conflict.
Mulitple layers performing keepalives is just one example, and
probably one with lesser insidious behavoir. Look at
http://qurinet.ucdavis.edu/pubs/conf/wicon2008.pdf for good example
how link layer retransmissions can conflict with TCP algorithms to
produce really bad results.

Tom

> It's only in trying to optimize them as overlapping that a problem is
> created.
>
>
>
>
> if the
> ramifications of doing aren't understood it can lead to undesirable
> interactions and behavior.
>
>
> Agreed - so don't. Admit that there are inefficiencies *regardless of how
> hard you try to do otherwise* and leave them alone, IMO.
>
>
> If the app needs an app-level keepalive, do it.
>
> If the app wants TCP to be kept alive, let IT do it and leave it alone.
>
> Don't try to couple the two because you can't, and whatever you think you
> might gain you could easily lose. Leaving the two alone and separate is
> sufficient and robust.
>
> Joe

Re: statement regarding keepalives

[Tom Herbert]

On Fri, Jul 20, 2018, 7:40 AM Spencer Dawkins at IETF <
spencerdawkins.i...@gmail.com> wrote:

> Hi, Mikael,
>
> On Fri, Jul 20, 2018 at 6:48 AM Mikael Abrahamsson 
> wrote:
>
>>
>> Hi,
>>
>> While I agree with the sentiment here, I have personally been in
>> positions
>> where application programmers were unable to (in a timely manner) modify
>> whatever was running, to implement a keepalive protocol. In that case,
>> turning on TCP keepalives was a very easy thing to do that immediately
>> would yield operational benefits.
>>
>> So I'd like to see in the text that we recommend to do it as "high up" in
>> the stack as possible, but still don't put off people turning on TCP
>> keepalives "because the IETF doesn't recommend that", and thus they do
>> nothing at all and the problem just persists.
>>
>> Also, should we talk about recommendations for what these timers should
>> be? In my experience, it's typically in tens of seconds up to 5-10
>> minutes
>> that makes sense for Internet use. Shorter than that might interrupt the
>> connection prematurely, longer than that causes things to take too long
>> to
>> detect a problem. Of course it's up to the application/environment to
>> choose the best value for each use-case, but some text on this might be
>> worthwhile to have there?
>>
>
> This is exactly the kind of feedback I'd like to reflect in whatever
> guidance we give, in whatever form. Thank you for that.
>
> Other thoughts?
>

Spencer,

When I saw the subject I was wondering if this was going to be the proposal
to deprecate keepalives! With vast numbers of connections performing them,
they at some point start to create congestion. This is especially
problematic if somehow keepalives timers become sychronized. IIRC Google
calendar was brought down at one point a while back because of a keepalive
avalanche. I would assume that keepalives are primarily needed to maintain
NAT state, so if NAT were deprecated (like conceptually promised by  IPv6)
then maybe keepalives could go away.

In any case, I think the text here probably would be good in the draft that
examines current state of keepalives and makes recommendations.

Tom


> Spencer
>
> (And, yes, there's a tension between "why would I tear down a perfectly
> good idle connection because it might not work the next time I try to use
> it? for real" and "OMG, I need to send a message to the other side, but my
> idle connection is now timing out, so I have to set up a new connection,
> secure it, and do whatever else I need to do with the other side before I
> can send a message!!!". That's a good thing to include in our advice)
>

Re: Draft agenda uploaded

[Tom Herbert]

On Wed, Mar 7, 2018 at 9:04 AM, Mirja Kuehlewind (IETF)
<i...@kuehlewind.net> wrote:
> Hi Tom,
>
> no, this is more a topic that we as the AD would like to discuss with the 
> community as we see more and more work that adds TCP encapsulation to all 
> kind of protocols for middlebox traversal issues.
).

So this is encapsulating protocols over TCP then (as opposed
encapsulating TCP and other transports over UDP like in
draft-herbert-transports-over-udp-01

Does this discussion have anything to do with STT
(https://datatracker.ietf.org/doc/draft-davie-stt/)?

Thanks,
Tom

>
> Mirja
>
>> Am 07.03.2018 um 17:56 schrieb Tom Herbert <t...@herbertland.com>:
>>
>>
>>
>> On Mar 7, 2018 8:16 AM, "Mirja Kuehlewind (IETF)" <i...@kuehlewind.net> 
>> wrote:
>> Hi all,
>>
>> we plan to have a short (1h) TSV area meeting in London and are scheduled for
>>
>> tsvarea Session 1 (1:00:00)
>>Monday, Afternoon Session III 1740-1840
>>Room Name: Sandringham size: 300
>>
>> We just uploaded the planned agenda:
>>
>> https://datatracker.ietf.org/meeting/101/materials/agenda-101-tsvarea
>>
>> Besides the usual updates and status report, we are planning for a 
>> discussion about TCP encapsulation with a corresponding presentation about 
>> experience with TCP encapsulation for IKE.
>>
>> Mirja,
>>
>> Is there a draft or RFC for TCP encapsulation that is being discussed?
>>
>> Tom
>>
>>
>> See you all in London!
>>
>> Mirja and Spencer
>>
>

Re: Draft agenda uploaded

[Tom Herbert]

On Mar 7, 2018 8:16 AM, "Mirja Kuehlewind (IETF)" 
wrote:

Hi all,

we plan to have a short (1h) TSV area meeting in London and are scheduled
for

tsvarea Session 1 (1:00:00)
   Monday, Afternoon Session III 1740-1840
   Room Name: Sandringham size: 300

We just uploaded the planned agenda:

https://datatracker.ietf.org/meeting/101/materials/agenda-101-tsvarea

Besides the usual updates and status report, we are planning for a
discussion about TCP encapsulation with a corresponding presentation about
experience with TCP encapsulation for IKE.


Mirja,

Is there a draft or RFC for TCP encapsulation that is being discussed?

Tom


See you all in London!

Mirja and Spencer