BTW, I felt it is better to be called as an agency, rather than a proxy. Any comments from native speaker?
Thx Eddie > Hi, all > > We are planning to implement colo-proxy in qemu to cache and compare > packets. > This module is one of the important component of COLO project and now it is > still in early stage, so any comments and feedback are warmly welcomed, thanks > in advance. > > ## Background > COLO FT/HA (COarse-grain LOck-stepping Virtual Machines for Non-stop Service) > project is a high availability solution. Both Primary VM (PVM) and Secondary > VM > (SVM) run in parallel. They receive the same request from client, and generate > responses in parallel too. If the response packets from PVM and SVM are > identical, they are released immediately. Otherwise, a VM checkpoint (on > demand) is conducted. > Paper: > http://www.socc2013.org/home/program/a3-dong.pdf?attredirects=0 > COLO on Xen: > http://wiki.xen.org/wiki/COLO_-_Coarse_Grain_Lock_Stepping > COLO on Qemu/KVM: > http://wiki.qemu.org/Features/COLO > > By the needs of capturing response packets from PVM and SVM and finding out > whether they are identical, we introduce a new module to qemu networking > called colo-proxy. > > This document describes the design of the colo-proxy module > > ## Glossary > PVM - Primary VM, which provides services to clients. > SVM - Secondary VM, a hot standby and replication of PVM. > PN - Primary Node, the host which PVM runs on > SN - Secondary Node, the host which SVM runs on > > ## Workflow ## > The following image show the qemu networking packet datapath between > guest's NIC and qemu's backend in colo-proxy. > > +---+ +---+ > |PN | |SN | > +---+--------------------------+ > +---+--------------------------+ +------------------------------+ > | +-------+ | | +-------+ | > +--------+ |chkpoint<--------[socket]------->chkpoint +--------+ > |PVM | +---^---+ | | +---+---+ |SVM | > | | +proxy--v--------+ | | | | | > | | | | | | | | | > | +---+ | | +TCP/IP stack+ | | | +-----v-------proxy | +---+ | > +-|NIC|--+ | | | | | | | | +-|NIC|--+ > | +^-++ | | +--------+ | | | | | +TCP/IP stack-+ | +^--+ | > | | +------> | | compare| | <-[socket]-forward- | +--------+ | | | | > | | | | +---+----+ | | | | | | |seq&ack | | <----+ | > | | | +-----|------+ | | | | | |adjust | | | | > | | | | | | | | | +--------+ | | | > | +-----------<+>-----copy&forward-[socket]---> +-------------+ | | > | +---|---|--------+ | | +------------^----+ | > | | | | | | | > | | | | | x | > | +--+---v----+ | | +-v---------+ | > | QEMU | backend | | | QEMU | backend | | > +------------+ (tap) +-----+ +------------+ (tap) +-----+ > +-----------+ +-----------+ > > ## Our Idea ## > > ### Net filter > In current QEMU, a packet is transported between networking backend(tap) > and qemu network adapter(NIC) directly. Backend and adapter is linked by > NetClientState->peer in qemu as following > +----------------------------------------+ > v | > +NetClientState+ +------->+NetClientState+ | > |info->type=TAP| | |info->type=NIC| | > +--------------+ | +--------------+ | > | *peer +---+ | *peer +----+ > +--------------+ +--------------+ > |name="tap0" | |name="e1000" | > +--------------+ +--------------+ > | ... | | ... | > +--------------+ +--------------+ > > In COLO QEMU, we insert a net filter named colo-proxy between backend and > adapter like below: > typedef struct COLOState { > NetClientState nc; > NetClientState *peer; > } COLOState; > +------->+NetClientState+ +NetClientState+<--------+ > | |info->type=TAP| |info->type=NIC| | > | +--------------+ +--------------+ | > +-----------+ *peer | | *peer +------------+ > | | +--------------+ +--------------+ | | > | | |name="tap0" | |name="e1000" | | | > | | +--------------+ +--------------+ | | > | | | ... | | ... | | | > | | +--------------+ +--------------+ | | > | | | | > | | +-COLOState------------+ +-COLOState------------+ | | > +--------->+NetClientState+<- - + +---->+NetClientState+<---------+ > | | |info->type=COLO | | | | |info->type=COLO | | > | | +--------------+ | | | | +--------------+ | | > +-------+ *peer | | | | | | *peer +-------+ > | +--------------+ | | | | +--------------+ | > | |name="colo1" | | | | | |name="colo2" | | > | +--------------+ | | | | +--------------+ | > | | | | | | > | +--------------+---------+ | +--------------+ | > | | *peer | | | | | *peer | | > | +--------------+ | +---------+--------------+ | > +----------------------+ +----------------------+ > > After we insert colo-proxy filter, all packets will pass by this filter and > more > important thing is that we can analysis packet by ourselves. > > ### QEMU space TCP/IP stack(re-use SLIRP) ### We need a QEMU space TCP/IP > stack to help us to analysis packet. After looking into QEMU, we found that > SLIRP > http://wiki.qemu.org/Documentation/Networking#User_Networking_.28SLIRP. > 29 > is a good choice for us. SLIRP proivdes a full TCP/IP stack within QEMU, it > can > help use to handle the packet written to/read from backend(tap) device which > is > just like a link layer(L2) packet. > > ### packet enqueue and compare ### > Together with QEMU space TCP/IP stack, we enqueue all packets sent by PVM > and SVM on Primary QEMU, and then compare the packet payload for each > connection. > > ### Net filter Usage ### > On both Primary/Secondary host, invoke QEMU with the following parameters > to insert a net filter(colo-proxy): > "-netdev tap,id=hn0 -device e1000,netdev=hn0 \ > -netdev colo,id=colo,backend=hn0" >
