Bob,
Thank you very much for reviewing the draft and provided in-depth
comments. I am very sorry for the delayed response due to traveling.
Replies to your comments are inserted below marked by [Linda]:
-----Original Message-----
From: Bob Briscoe [mailto:[email protected]]
Sent: Monday, September 03, 2018 9:45 PM
To: [email protected] <mailto:[email protected]>
Cc: [email protected] <mailto:[email protected]>; [email protected]
<mailto:[email protected]>; [email protected]
<mailto:[email protected]>
Subject: Tsvart last call review of draft-ietf-nvo3-vmm-04
Reviewer: Bob Briscoe
Review result: Not Ready
I have been selected as the Transport Directorate reviewer for
this draft. The Transport Directorate seeks to review all
transport or transport-related drafts as they pass through IETF
last call and IESG review, and sometimes on special request. The
purpose of the review is to provide assistance to the Transport
ADs. For more information about the Transport Directorate Reviews
and the Transport Area Review Team, please see
https://trac.ietf.org/trac/tsv/wiki/TSV-Directorate-Reviews
In this case, very very few of the review comments relate to
transport issues, although the greatest issue concerns a desire
that the network could pause or stop connections during L3 VM
Mobility, which is certainly a transport issue.
[Linda] There is “Hot Migration” with transport service
continuing, and there is a “Cold Migration”, which is a common
practice in many data centers, which stop the task running on the
old place and move to the new place before restart as described
in the Task Migration.
Is it helpful to add this description to the draft?
==Summary==
The technical aspects of the draft concerning L2 VM mobility
(within a subnet) seem sound. However, this is only part of the
draft, which has the following
issues:
#. The introduction does not say what the purpose of publishing
this draft is.
It seems that, rather than describing a specific protocol or
protocols, it intends to describe the overall system procedure
that would typically be used in DCs for VM mobility. It is tagged
as a BCP, but it does not say who needs this BCP, why it is
useful for the IETF to publish this BCP, how wide the authors'
knowledge is of current practice (given DCs are private), or why
this is a BCP rather than a protocol spec.
[Linda] The first paragraph on Page 3 has the description why VM
Mobility is needed. Is it helpful to move this paragraph to the
beginning of the Introduction Section?
/“//Virtualization which is being used in almost all of today’s data/
/centers enables many virtual machines to run on a single physical/
/computer or compute server. Virtual machines (VM) need hypervisor/
/running on the physical compute server to provide them shared/
/processor/memory/storage. Network connectivity is provided by the/
/network virtualization edge (NVE) [RFC8014]. Being able to move VMs/
/dynamically, or live migration, from one server to another
allows for/
/dynamic load balancing or work distribution and thus it is a highly/
/desirable feature [RFC7364].//”/
The draft starts out (S.3) as if it intends to say what a good VM
Mobility protocol should or shouldn't do, but the rest of the
document doesn't give any reasoning for these recommendations, it
just asserts what appears to be one view of how a whole VM
Mobility system works, sometimes referring to one example
protocol RFC for a component part, but more often with no
references or details.
[Linda] Is it helpful to move the paragraph above to the
beginning of the Introduction Section? So that audience is aware
of why VM Mobility is needed. And then follow up with what a good
VM Mobility protocol should or shouldn't do?
#. It does not seem as if the NVO WG has discussed the purpose of
using normative text in this draft. See detailed comments.
[Linda] The “Intended status” of the draft is “Best Current
Practice”. So all the text are not “normative”. Is it Okay?
#. The draft silently slips back and forth between VM mobility
and VM redundancy, without recognizing the differences. See
detailed comments.
[Linda] There is only one usage of “redundancy” in the entire
document, used under the context of “Hot standby option”,
indicating the “redundancy” of “the VMs in both primary and
secondary domains have identical information and can provide
services simultaneously as in load-share mode of operation” being
expensive.
#. Please adopt different terminology than "source NVE" and
"destination NVE", which are really poor choices of terms for an
intermediate node. See detailed comments. Why not use "old NVE"
and "new NVE", which is what you mean?
[Linda] Thanks for the suggestion. We will change to “Old NVE”,
and “new NVE”.
#. Applicability is fairly clearly outlined, but it is not clear
whether hosts corresponding with the mobile VMs are part of the
same controlled environment or on the uncontrolled public
Internet. See detailed comments.
[Linda] “Hosts” are the App running on the VM. It is the under
the same controlled environment. Not on uncontrolled public
internet.
#. Section 4.2.1 on L3 VM mobility reads like some potential
half-thought-through ideas on how to solve L3 mobility, rather
than current practice, let alone best current practice. Either
current practice should be described instead, or the scope of the
draft should be narrowed solely to L2 VM mobility. See detailed
comments.
[Linda] This is refereeing to “Cold Migration”, which is a common
practice in many data centers.
# The VM's file system is described as state that moves with the
VM (S.6), but VM mobility solutions often move the VM but stitch
it back to its (unmoved) storage. Conversely, the storage can
also move independent of the VM.
[Linda] It depends. When a VM move to a different zone, the
storage/file can becomes inaccessible.
#. The draft omits some of the security, transport and management
aspects of VM mobility. See detailed comments.
[Linda] Can you provide some text?
#. The draft reads as if different sections have been written by
different authors and no-one has edited the whole to give it a
coherent structure, or to ensure consistency (both technical and
editorial) between the parts. See detailed comments.
[Linda] we can improve.
#. The quality of the English grammar does not allow a reviewer
to concentrate on the technical aspects rather than the English.
It would have been useful if one of the English-speaking
co-authors had improved the English before submission for review.
See detailed comments.
[Linda] can you help? Becoming a co-author to improve?
==Detailed Comments==
===#. Normative statements===
In the body of the document, there is just one occurrence of
normative text (actually two "MUST"s, but both state a common
requirement - just written separately for IPv4 and IPv6). This
merely serves to imply that everything else the document says is
less important or optional, which was probably not the intention.
[Linda] The goal is to indicate any solution in moving the VM
“MUST” follow this rule. They make sense, aren’t they?
At the start there is a requirements section, which states what a
VM Mobility protocol "SHOULD" or "SHOULD NOT" do. I think this is
intended as a set of goals for the rest of the document. If so,
these "SHOULDs" are not intended to apply to implementations, so
they ought not to be capitalized.
[Linda] okay, will change.
The first requirement, "Data center network SHOULD support
virtual machine mobility in IPv6", is written as a requirement on
all DC networks, not on implementations. I assume this was
intended to read as "Data center network virtual machine mobility
protocols SHOULD support IPv6". Even then, it doesn't really add
anything to say VM mobility should support v6 and it should
support v4. A L2 solution won't. While undoubtedly, a L3 solution
will at least support one of them.
[Linda]Agree. Will change it to “Data center that support IPv6
address should …”
I'm not sure that 'protocol' is the right word anyway; I think
'VM Mobility procedure' would be a better phrase, because it
includes steps such as suspending the VM, which is more than a
protocol.
[Linda] yes. Will change to “Procedure”.
The requirement "Virtual machine mobility protocol MAY support
host routes to accomplish virtualization", is not followed up at
all in the rest of the draft.
Even if this requirement stays, the last 3 words should be deleted.
[Linda] will change to “Host Route can be used to support the
Virtual Machine Mobility Procedure.”
By the end of the draft, the solution falls far short of the most
relevant "Requirements" anyway, so one assumes the title of the
section ought to have been "Goals". Specifically, even in the
simpler case of L2 VM mobility, S.4.1 says that triangular
routing and tunnelling persist "until a neighbour cache entry
times out". A cache timeout is about 10 orders of magnitude
longer than the requirement to only persist "while handling
packets in flight", which would be a few milliseconds at most
(the time for packets to clear the network that were already
launched into flight when the old VM stopped).
Whatever, it would be preferable for the draft to give rationale
for these requirements, rather than just assert them. This would
help to shed light on the merits of the different trade offs that
solutions choose.
[Linda] Agree, will add.
===#. Mobility vs. Redundancy===
Redundancy and mobility have a lot of similarities, but they have
different goals. With mobility, it is necessary to know the exact
instant when one set of state is identical to the other so it can
hand over. With redundancy, the aim is to keep two (or more) sets
of state evolving through the same sequence of changes, but there
is no need to know the point at which one is the same as the
other was at a certain point.
[Linda] Agree with what you said. There is only one usage of
“redundancy” in the entire document, used under the context of
“Hot standby option”, indicating the “redundancy” of “the VMs
in both primary and secondary domains have identical information
and can provide services simultaneously as in load-share mode of
operation” being expensive.
The draft slips from mobility to resilience in the following places:
* S.2. Terminology: Warm VM Mobility is defined without any
ending, as if it is permanent replication. * S.7. "Handling of
Hot, Warm and Cold Virtual Machine Mobility" is actually all
about redundancy, and doesn't address mobility explicitly.
[Linda] Will add the definition “Hot Migration”, “cold
migration”, and “warm migration”.
===#. Terminology===
Packets run from the source at A to the destination at B via
NVE1, then via NVE2. Please don't call NVE1 and NVE2 the source
NVE and the destination NVE.
In future, no-one will thank you for the apparent contradictions
when they continually stumble over phrases like this one in
S.4.1: "...send their packets to the source NVE".
The term "packets in flight" is used incorrectly to refer to all
the packets sent to the old NVE after the VM has moved, even if
they were launched into flight long after the old VM stopped
receiving packets.
[Linda] thank for the comments. Will change.
BTW, I think s/before/after/ in: "that have old ARP or neighbor
cache entry before VM or task migration".
I think: s/IP-based VM mobility/L3 VM mobility/ throughout,
because "based"
sounds (to me) like the mobility control protocol is over (i.e.
based on) IP.
===#. Applicability===
In section 4.2 it says that the protocol mostly used as the IP
based task migration protocol is ILA. This implies that all hosts
corresponding with the mobile VMs are either part of the same
controlled environment, or they are proxied via nodes that are
part of the same controlled environment (I only have passing
knowledge of ILA, but I understand that it depends on ILA routers
on the path). If I am correct, this aspect of scope needs to be
made clear from the start.
Also under the heading of applicabiliy, the sentence "Since
migrations should be relatively rare events" appears very late in
the document (S.4.2.1). The assumed level of churn ought to be
stated nearer the start.
[Linda] yes, under the same controlled environment.
===#. L3 Mobility===
L2 VM mobility is independent of the application, because
resolution of L2 mappings is delegated to the stack. In contrast,
L3 VM mobility is only feasible under certain conditions, because
an application needs an IP address to open a socket (resolution
of DNS names is not delegated to the stack, and apps can use IP
addresses directly anyway).
Examples of the 'certain conditions':
a) /All/ applications used in the whole DC load balancing scheme
contain IP address migration logic for /all/ their connections;
b) VMs running solely applications that support IP address
migration register this fact with the NVA, and it only select
such VMs for mobility. c) An abstraction is layered over /all/
the IP addresses exposed to applications (at both ends) so that
the IP addresses that applications use are solely identifiers
(e.g. ILA, LISP, HIP), not also locators.
The introduction says the draft is about VM mobility in a
multi-tenant DC, so the DC admin will not know the range of
applications being used. This excludes condition (a) above. When
the draft says "...if all applications running are known to
handle this gracefully...", it doesn't quantify just how
restrictive this condition is, and it gives no explanation of how
this knowledge might be 'known' or which function within the
system 'knows' it.
S.4.2.1 contains what seems like plenty of arm-waving.
* "TCP connections could be automatically closed in the network
stack during a migration event."
o There is no TCP connection state in the network stack.
o Even if the network starts to drop every packet, the
TCP connection
state persists in the end-points for a duration of the
order of 30-90
minutes (OS-dependent) before TCP deems the connection is
broken. o
Other transport protocols have similar designs (including
the app-layer
of protocols over UDP).
* "More involved approach to connection migration":
o pausing the connection [does this refer to an actual
feature of any
L4 protocol?] o packaging connection state and sending to
target [does
this assume logic written into the application, or is
this assuming the
stack handles this and the app is restricted to using
some form of
separate identifier/locator addresses?] o instantiating
connection
state in the peer stack [ditto?].
There's some arm-waving in S.7 too:
"Cold Virtual Machine mobility is facilitated by the VM initially
sending an ARP or Neighbor Discovery message at the
destination NVE
but the source NVE not receiving any packets inflight."
[How is it arranged for the source NVE not to receive any
packets in flight?]
And in S.7:
"In hot
standby option, regarding TCP connections, one option is to start
with and maintain TCP connections to two different VMs at the same
time."
[This sounds like resilience logic has been written into the
application,
which would be a special case but not something VM mobility
infrastructure
could depend on.]
[Linda] will add.
===#. Gaps===
#. Security Considerations: repeats issues in other drafts that
are not specific to mobility, but it does not mention any
security issues specifically due to VM mobility. It says that
address spoofing may arise in a DC (sort-of implying it is worse
than in non-DC environments, but not saying why). The handshake
at the start of a connection (e.g. TCP, SCTP, QUIC) checks for
source address spoofing. So L3 VM mobility would be more
vulnerable to source address spoofing in cases where the mobile
VM was the connection initiator and there was not a new handshake
after the move. However, this draft does not contain any detailed
mobility protocols, so it is not possible to identify any
specific security flaws.
#. Transport Issues: Effect of delay on the transport: Cold
mobility introduces significant delay, and other forms less, but
still some delay. It should be pointed out that some applications
(e.g. real-time) will therefore not be useful if subjected to VM
mobility. Similarly, even a short period of delay will drive most
congestion controls to severely reduce throughput. These points
might be self-evident, but perhaps they should be stated explicitly.
BTW, in the L3 VM mobility case, the draft often refers to TCP
connections, but the address bindings of any transport protocols
would have to be migrated due to VM mobility (e.g. SCTP;
sequences of datagrams over UDP; streams over UDP such as with
RTP, QUIC).
#. Management Issues: perhaps the draft ought to recommend
statistics gathering (e.g. time taken, amount of duplicate data)
to aid a DC's future decisions on the cost-benefit of moving a
VM. The OPSDIR review says a BCP does not /have/ to describe
management issues, but this document seems to describe a whole
system procedure, not just a protocol, which then surely includes
the management plane.
[Linda] can you become a co-author and add those in?
===#. Incoherent Structure===
S.4.1. happens to talk about VMs moving, while S.4.2. happens to
talk about tasks moving, but this is not the distinguishing
aspect of these two sections (anyway, S.2. says "the draft uses
task and VM interchangeably"): * "4.1 VM Migration" is about "L2
VM Mobility" so this ought to be the section heading, *
"4.2 Task Migration" is about "L3 VM Mobility" so this ought to
be the section heading. It would also help not to switch from VM
to task across these sections
- it's just a distraction.
S.4.1 needs better signposting of where each sub-case ends
(Subsections might be useful to solve this): * IPv4 * end-user
client * 2 paras starting "All NVEs communicating with this
virtual machine..." [Not clear that the end-user case has ended
and we have returned to the general IPv4 case?] * IPv6 [Strictly,
it still hasn't said whether the end-user client case has ended.]
[Also, it doesn't explain why there is no need for an end-user
client case under IPv6?] Sections 5 & 6 seem to be about either
L2 or L3 mobility, whereas Sections 7 &
8 seem to be restricted to L2.
The draft vacillates over what to do with packets arriving at the
old NVE in the L3 case (see also L3 mobility above): * S4.2 first
says packets are dropped, possibly with an ICMP error message;
o then later it says they are silently dropped;
o then in the very next sentence it says either silently drop
them or forward
them to the new location
* S.5 says they should not be lost, but instead delivered to the
destination hypervisor
o then it describes how they are tunnelled (which is not the
same as
"forwarding").
The order in which all the stages of mobilty are given is jumbled
up across sections that also appear in arbitrary order: * S.5
prepares, establishes uses then stops a tunnel, but it doesn't
say where the other stages fit between these steps
o When tunneling packets, it talks about the *migrating*
VM not the
*migrated* VM, which implies tunnelling has started
before the new VM
is running. Does this imply there is a huge buffer? o It
says "Stop
Tunneling Packets - When source NVE stops receiving
packets destined
to..." but it is never clear when a source has stopped
sending packets
to a destination, unless it explicitly closes the
connection (e.g. with
a FIN in the case of TCP). Often there are long gaps
between packets,
because many flows are 'thin' (meaning the application
frequently has
nothing to send). These gaps can last for milliseconds,
hours or even
days without any implication that the connection has ended.
* Then S.6. describes moving state, but doesn't say that this is
not after the previous tunnelling steps (or where it fits within
those steps). * Then S.7 describes hot, warm and cold mobility,
but doesn't lay out the tunnelling or steps to move state in each
case. * Then S.8 says it's about VM life-cycle, but just gives
the very first 3 steps for allocation of resources to a VM, then
abruptly ends, without even starting the VM, let alone getting to
move it.
S.5 exhibits another inconsistency by talking about the
hypervisor, not the NVE.
==#. Nits==
Nits with the English are too numerous to mention them all. Below
are pointers to general problems as well as some individual
instances.
S.4
"Layer 2 and Layer 3 protocols are described next. In the
following
sections, we examine more advanced features."
s/following/subsequent/
S.4.1
Expand WSC, MSC and NVA on first use.
s/the VM moves in the same link/the VM moves in the same subnet/
"i.e. end-user clients ask for the same MAC address upon
migration. [...] to ensure that the same IPv4 address is assigned
to the VM." I think s/IPv4/MAC/ was intended?
" All NVEs communicating with this virtual machine uses the old ARP
entry. If any VM in those NVEs need to talk to the new VM in the
destination NVE, it uses the old ARP entry."
Repetition: these 2 sentences say the same. (The mistake is also
repeated when these 2 sentences are repeated for IPv6).
S.4.2.1
s/Push the new mapping to hosts./Push the new mapping to
communicating hosts./
S.5.
The IPv4/IPv6 pairs of paras for "tunnel estabilshment" and
"tunneling packets"
only differ in the words "IPv4"/"IPv6". So in each case a single
para could be given for IP (irrespective of whether v4 or v6).
Thank you very much.
Linda Dunbar
_______________________________________________
Tsv-art mailing list
[email protected] <mailto:[email protected]>
https://www.ietf.org/mailman/listinfo/tsv-art
<https://www..ietf.org/mailman/listinfo/tsv-art>
