Bob,
Thank you very much for reviewing the draft and provided
in-depth comments. I am very sorry for the delayed response
due to traveling.
Replies to your comments are inserted below marked by [Linda]:
-----Original Message-----
From: Bob Briscoe [mailto:[email protected]]
Sent: Monday, September 03, 2018 9:45 PM
To: [email protected] <mailto:[email protected]>
Cc: [email protected] <mailto:[email protected]>; [email protected]
<mailto:[email protected]>; [email protected]
<mailto:[email protected]>
Subject: Tsvart last call review of draft-ietf-nvo3-vmm-04
Reviewer: Bob Briscoe
Review result: Not Ready
I have been selected as the Transport Directorate reviewer
for this draft. The Transport Directorate seeks to review
all transport or transport-related drafts as they pass
through IETF last call and IESG review, and sometimes on
special request. The purpose of the review is to provide
assistance to the Transport ADs. For more information about
the Transport Directorate Reviews and the Transport Area
Review Team, please see
https://trac..ietf.org/trac/tsv/wiki/TSV-Directorate-Reviews
<https://trac.ietf.org/trac/tsv/wiki/TSV-Directorate-Reviews>
In this case, very very few of the review comments relate to
transport issues, although the greatest issue concerns a
desire that the network could pause or stop connections
during L3 VM Mobility, which is certainly a transport issue.
[Linda] There is “Hot Migration” with transport service
continuing, and there is a “Cold Migration”, which is a
common practice in many data centers, which stop the task
running on the old place and move to the new place before
restart as described in the Task Migration.
Is it helpful to add this description to the draft?
==Summary==
The technical aspects of the draft concerning L2 VM mobility
(within a subnet) seem sound. However, this is only part of
the draft, which has the following
issues:
#. The introduction does not say what the purpose of
publishing this draft is.
It seems that, rather than describing a specific protocol or
protocols, it intends to describe the overall system
procedure that would typically be used in DCs for VM
mobility. It is tagged as a BCP, but it does not say who
needs this BCP, why it is useful for the IETF to publish
this BCP, how wide the authors' knowledge is of current
practice (given DCs are private), or why this is a BCP
rather than a protocol spec.
[Linda] The first paragraph on Page 3 has the description
why VM Mobility is needed. Is it helpful to move this
paragraph to the beginning of the Introduction Section?
/“//Virtualization which is being used in almost all of
today’s data/
/centers enables many virtual machines to run on a single
physical/
/computer or compute server. Virtual machines (VM) need
hypervisor/
/running on the physical compute server to provide them shared/
/processor/memory/storage. Network connectivity is provided
by the/
/network virtualization edge (NVE) [RFC8014]. Being able to
move VMs/
/dynamically, or live migration, from one server to another
allows for/
/dynamic load balancing or work distribution and thus it is
a highly/
/desirable feature [RFC7364].//”/
The draft starts out (S.3) as if it intends to say what a
good VM Mobility protocol should or shouldn't do, but the
rest of the document doesn't give any reasoning for these
recommendations, it just asserts what appears to be one view
of how a whole VM Mobility system works, sometimes referring
to one example protocol RFC for a component part, but more
often with no references or details.
[Linda] Is it helpful to move the paragraph above to the
beginning of the Introduction Section? So that audience is
aware of why VM Mobility is needed. And then follow up with
what a good VM Mobility protocol should or shouldn't do?
#. It does not seem as if the NVO WG has discussed the
purpose of using normative text in this draft. See detailed
comments.
[Linda] The “Intended status” of the draft is “Best Current
Practice”. So all the text are not “normative”. Is it Okay?
#. The draft silently slips back and forth between VM
mobility and VM redundancy, without recognizing the
differences. See detailed comments.
[Linda] There is only one usage of “redundancy” in the
entire document, used under the context of “Hot standby
option”, indicating the “redundancy” of “the VMs in both
primary and secondary domains have identical information and
can provide services simultaneously as in load-share mode of
operation” being expensive.
#. Please adopt different terminology than "source NVE" and
"destination NVE", which are really poor choices of terms
for an intermediate node. See detailed comments. Why not use
"old NVE" and "new NVE", which is what you mean?
[Linda] Thanks for the suggestion. We will change to “Old
NVE”, and “new NVE”.
#. Applicability is fairly clearly outlined, but it is not
clear whether hosts corresponding with the mobile VMs are
part of the same controlled environment or on the
uncontrolled public Internet. See detailed comments.
[Linda] “Hosts” are the App running on the VM. It is the
under the same controlled environment. Not on uncontrolled
public internet.
#. Section 4.2.1 on L3 VM mobility reads like some potential
half-thought-through ideas on how to solve L3 mobility,
rather than current practice, let alone best current
practice. Either current practice should be described
instead, or the scope of the draft should be narrowed solely
to L2 VM mobility. See detailed comments.
[Linda] This is refereeing to “Cold Migration”, which is a
common practice in many data centers.
# The VM's file system is described as state that moves with
the VM (S.6), but VM mobility solutions often move the VM
but stitch it back to its (unmoved) storage. Conversely, the
storage can also move independent of the VM.
[Linda] It depends. When a VM move to a different zone, the
storage/file can becomes inaccessible.
#. The draft omits some of the security, transport and
management aspects of VM mobility. See detailed comments.
[Linda] Can you provide some text?
#. The draft reads as if different sections have been
written by different authors and no-one has edited the whole
to give it a coherent structure, or to ensure consistency
(both technical and editorial) between the parts. See
detailed comments.
[Linda] we can improve.
#. The quality of the English grammar does not allow a
reviewer to concentrate on the technical aspects rather than
the English. It would have been useful if one of the
English-speaking co-authors had improved the English before
submission for review. See detailed comments.
[Linda] can you help? Becoming a co-author to improve?
==Detailed Comments==
===#. Normative statements===
In the body of the document, there is just one occurrence of
normative text (actually two "MUST"s, but both state a
common requirement - just written separately for IPv4 and
IPv6). This merely serves to imply that everything else the
document says is less important or optional, which was
probably not the intention.
[Linda] The goal is to indicate any solution in moving the
VM “MUST” follow this rule. They make sense, aren’t they?
At the start there is a requirements section, which states
what a VM Mobility protocol "SHOULD" or "SHOULD NOT" do. I
think this is intended as a set of goals for the rest of the
document. If so, these "SHOULDs" are not intended to apply
to implementations, so they ought not to be capitalized.
[Linda] okay, will change.
The first requirement, "Data center network SHOULD support
virtual machine mobility in IPv6", is written as a
requirement on all DC networks, not on implementations. I
assume this was intended to read as "Data center network
virtual machine mobility protocols SHOULD support IPv6".
Even then, it doesn't really add anything to say VM mobility
should support v6 and it should support v4. A L2 solution
won't. While undoubtedly, a L3 solution will at least
support one of them.
[Linda]Agree. Will change it to “Data center that support
IPv6 address should …”
I'm not sure that 'protocol' is the right word anyway; I
think 'VM Mobility procedure' would be a better phrase,
because it includes steps such as suspending the VM, which
is more than a protocol.
[Linda] yes. Will change to “Procedure”.
The requirement "Virtual machine mobility protocol MAY
support host routes to accomplish virtualization", is not
followed up at all in the rest of the draft.
Even if this requirement stays, the last 3 words should be
deleted.
[Linda] will change to “Host Route can be used to support
the Virtual Machine Mobility Procedure.”
By the end of the draft, the solution falls far short of the
most relevant "Requirements" anyway, so one assumes the
title of the section ought to have been "Goals".
Specifically, even in the simpler case of L2 VM mobility,
S.4.1 says that triangular routing and tunnelling persist
"until a neighbour cache entry times out". A cache timeout
is about 10 orders of magnitude longer than the requirement
to only persist "while handling packets in flight", which
would be a few milliseconds at most (the time for packets to
clear the network that were already launched into flight
when the old VM stopped).
Whatever, it would be preferable for the draft to give
rationale for these requirements, rather than just assert
them. This would help to shed light on the merits of the
different trade offs that solutions choose.
[Linda] Agree, will add.
===#. Mobility vs. Redundancy===
Redundancy and mobility have a lot of similarities, but they
have different goals. With mobility, it is necessary to know
the exact instant when one set of state is identical to the
other so it can hand over. With redundancy, the aim is to
keep two (or more) sets of state evolving through the same
sequence of changes, but there is no need to know the point
at which one is the same as the other was at a certain point.
[Linda] Agree with what you said. There is only one usage of
“redundancy” in the entire document, used under the context
of “Hot standby option”, indicating the “redundancy” of
“the VMs in both primary and secondary domains have
identical information and can provide services
simultaneously as in load-share mode of operation” being
expensive.
The draft slips from mobility to resilience in the following
places:
* S.2. Terminology: Warm VM Mobility is defined without any
ending, as if it is permanent replication. * S.7. "Handling
of Hot, Warm and Cold Virtual Machine Mobility" is actually
all about redundancy, and doesn't address mobility explicitly.
[Linda] Will add the definition “Hot Migration”, “cold
migration”, and “warm migration”.
===#. Terminology===
Packets run from the source at A to the destination at B via
NVE1, then via NVE2. Please don't call NVE1 and NVE2 the
source NVE and the destination NVE.
In future, no-one will thank you for the apparent
contradictions when they continually stumble over phrases
like this one in S.4.1: "...send their packets to the source
NVE"..
The term "packets in flight" is used incorrectly to refer to
all the packets sent to the old NVE after the VM has moved,
even if they were launched into flight long after the old VM
stopped receiving packets.
[Linda] thank for the comments. Will change.
BTW, I think s/before/after/ in: "that have old ARP or
neighbor cache entry before VM or task migration".
I think: s/IP-based VM mobility/L3 VM mobility/ throughout,
because "based"
sounds (to me) like the mobility control protocol is over
(i.e. based on) IP.
===#. Applicability===
In section 4.2 it says that the protocol mostly used as the
IP based task migration protocol is ILA. This implies that
all hosts corresponding with the mobile VMs are either part
of the same controlled environment, or they are proxied via
nodes that are part of the same controlled environment (I
only have passing knowledge of ILA, but I understand that it
depends on ILA routers on the path). If I am correct, this
aspect of scope needs to be made clear from the start.
Also under the heading of applicabiliy, the sentence "Since
migrations should be relatively rare events" appears very
late in the document (S.4.2.1). The assumed level of churn
ought to be stated nearer the start.
[Linda] yes, under the same controlled environment.
===#. L3 Mobility===
L2 VM mobility is independent of the application, because
resolution of L2 mappings is delegated to the stack. In
contrast, L3 VM mobility is only feasible under certain
conditions, because an application needs an IP address to
open a socket (resolution of DNS names is not delegated to
the stack, and apps can use IP addresses directly anyway).
Examples of the 'certain conditions':
a) /All/ applications used in the whole DC load balancing
scheme contain IP address migration logic for /all/ their
connections; b) VMs running solely applications that support
IP address migration register this fact with the NVA, and it
only select such VMs for mobility. c) An abstraction is
layered over /all/ the IP addresses exposed to applications
(at both ends) so that the IP addresses that applications
use are solely identifiers (e.g. ILA, LISP, HIP), not also
locators.
The introduction says the draft is about VM mobility in a
multi-tenant DC, so the DC admin will not know the range of
applications being used. This excludes condition (a) above.
When the draft says "...if all applications running are
known to handle this gracefully...", it doesn't quantify
just how restrictive this condition is, and it gives no
explanation of how this knowledge might be 'known' or which
function within the system 'knows' it.
S.4.2.1 contains what seems like plenty of arm-waving.
* "TCP connections could be automatically closed in the
network stack during a migration event."
o There is no TCP connection state in the network stack.
o Even if the network starts to drop every packet,
the TCP connection
state persists in the end-points for a duration of
the order of 30-90
minutes (OS-dependent) before TCP deems the
connection is broken. o
Other transport protocols have similar designs
(including the app-layer
of protocols over UDP).
* "More involved approach to connection migration":
o pausing the connection [does this refer to an
actual feature of any
L4 protocol?] o packaging connection state and
sending to target [does
this assume logic written into the application, or
is this assuming the
stack handles this and the app is restricted to
using some form of
separate identifier/locator addresses?] o
instantiating connection
state in the peer stack [ditto?].
There's some arm-waving in S.7 too:
"Cold Virtual Machine mobility is facilitated by the VM
initially
sending an ARP or Neighbor Discovery message at the
destination NVE
but the source NVE not receiving any packets inflight."
[How is it arranged for the source NVE not to receive any
packets in flight?]
And in S.7:
"In hot
standby option, regarding TCP connections, one option is
to start
with and maintain TCP connections to two different VMs at
the same
time."
[This sounds like resilience logic has been written into
the application,
which would be a special case but not something VM
mobility infrastructure
could depend on.]
[Linda] will add.
===#. Gaps===
#. Security Considerations: repeats issues in other drafts
that are not specific to mobility, but it does not mention
any security issues specifically due to VM mobility. It says
that address spoofing may arise in a DC (sort-of implying it
is worse than in non-DC environments, but not saying why).
The handshake at the start of a connection (e.g. TCP, SCTP,
QUIC) checks for source address spoofing. So L3 VM mobility
would be more vulnerable to source address spoofing in cases
where the mobile VM was the connection initiator and there
was not a new handshake after the move. However, this draft
does not contain any detailed mobility protocols, so it is
not possible to identify any specific security flaws.
#. Transport Issues: Effect of delay on the transport: Cold
mobility introduces significant delay, and other forms less,
but still some delay. It should be pointed out that some
applications (e.g. real-time) will therefore not be useful
if subjected to VM mobility. Similarly, even a short period
of delay will drive most congestion controls to severely
reduce throughput. These points might be self-evident, but
perhaps they should be stated explicitly.
BTW, in the L3 VM mobility case, the draft often refers to
TCP connections, but the address bindings of any transport
protocols would have to be migrated due to VM mobility (e.g.
SCTP; sequences of datagrams over UDP; streams over UDP such
as with RTP, QUIC).
#. Management Issues: perhaps the draft ought to recommend
statistics gathering (e.g. time taken, amount of duplicate
data) to aid a DC's future decisions on the cost-benefit of
moving a VM. The OPSDIR review says a BCP does not /have/ to
describe management issues, but this document seems to
describe a whole system procedure, not just a protocol,
which then surely includes the management plane.
[Linda] can you become a co-author and add those in?
===#. Incoherent Structure===
S.4.1. happens to talk about VMs moving, while S.4.2.
happens to talk about tasks moving, but this is not the
distinguishing aspect of these two sections (anyway, S.2.
says "the draft uses task and VM interchangeably"): * "4.1
VM Migration" is about "L2 VM Mobility" so this ought to be
the section heading, *
"4.2 Task Migration" is about "L3 VM Mobility" so this ought
to be the section heading. It would also help not to switch
from VM to task across these sections
- it's just a distraction.
S.4.1 needs better signposting of where each sub-case ends
(Subsections might be useful to solve this): * IPv4 *
end-user client * 2 paras starting "All NVEs communicating
with this virtual machine..." [Not clear that the end-user
case has ended and we have returned to the general IPv4
case?] * IPv6 [Strictly, it still hasn't said whether the
end-user client case has ended.] [Also, it doesn't explain
why there is no need for an end-user client case under
IPv6?] Sections 5 & 6 seem to be about either L2 or L3
mobility, whereas Sections 7 &
8 seem to be restricted to L2.
The draft vacillates over what to do with packets arriving
at the old NVE in the L3 case (see also L3 mobility above):
* S4.2 first says packets are dropped, possibly with an ICMP
error message;
o then later it says they are silently dropped;
o then in the very next sentence it says either silently
drop them or forward
them to the new location
* S.5 says they should not be lost, but instead delivered to
the destination hypervisor
o then it describes how they are tunnelled (which is not
the same as
"forwarding").
The order in which all the stages of mobilty are given is
jumbled up across sections that also appear in arbitrary
order: * S.5 prepares, establishes uses then stops a tunnel,
but it doesn't say where the other stages fit between these
steps
o When tunneling packets, it talks about the
*migrating* VM not the
*migrated* VM, which implies tunnelling has started
before the new VM
is running. Does this imply there is a huge buffer?
o It says "Stop
Tunneling Packets - When source NVE stops receiving
packets destined
to..." but it is never clear when a source has
stopped sending packets
to a destination, unless it explicitly closes the
connection (e.g. with
a FIN in the case of TCP). Often there are long gaps
between packets,
because many flows are 'thin' (meaning the
application frequently has
nothing to send). These gaps can last for
milliseconds, hours or even
days without any implication that the connection has
ended.
* Then S.6. describes moving state, but doesn't say that
this is not after the previous tunnelling steps (or where it
fits within those steps). * Then S.7 describes hot, warm and
cold mobility, but doesn't lay out the tunnelling or steps
to move state in each case. * Then S.8 says it's about VM
life-cycle, but just gives the very first 3 steps for
allocation of resources to a VM, then abruptly ends, without
even starting the VM, let alone getting to move it.
S.5 exhibits another inconsistency by talking about the
hypervisor, not the NVE.
==#. Nits==
Nits with the English are too numerous to mention them all.
Below are pointers to general problems as well as some
individual instances.
S.4
"Layer 2 and Layer 3 protocols are described next. In the
following
sections, we examine more advanced features."
s/following/subsequent/
S.4.1
Expand WSC, MSC and NVA on first use.
s/the VM moves in the same link/the VM moves in the same subnet/
"i.e. end-user clients ask for the same MAC address upon
migration. [...] to ensure that the same IPv4 address is
assigned to the VM." I think s/IPv4/MAC/ was intended?
" All NVEs communicating with this virtual machine uses the
old ARP
entry. If any VM in those NVEs need to talk to the new
VM in the
destination NVE, it uses the old ARP entry."
Repetition: these 2 sentences say the same. (The mistake is
also repeated when these 2 sentences are repeated for IPv6).
S.4.2.1
s/Push the new mapping to hosts./Push the new mapping to
communicating hosts./
S.5.
The IPv4/IPv6 pairs of paras for "tunnel estabilshment" and
"tunneling packets"
only differ in the words "IPv4"/"IPv6". So in each case a
single para could be given for IP (irrespective of whether
v4 or v6).
Thank you very much.
Linda Dunbar
_______________________________________________
Tsv-art mailing list
[email protected] <mailto:[email protected]>
https://www.ietf.org/mailman/listinfo/tsv-art
<https://www..ietf.org/mailman/listinfo/tsv-art>
