Summary of performance assessments of "pure routed RPC" in Helium Plugin
Details of test and test environment :
==============================
Set-up :
=======
- 3 standard ODL VMs constituting ODL cluster + 1 VM running a network
simulator (this simulator emulates minimalistic "openflow message listener" -
ie. connect to controller, perform basic handshake and just stay idle and
listen for OFMessages and record reception-rate of each type of message
received from ODL - simulator is NOT even a OF Agent let alone could be thought
as a "switch").
- Each VM is of 10 vCPU + 8GB RAM all running on single physical node
with single bridge connecting all VMs
- OF-HA capability - NOT used (ie. 1 emulated "switch" of network
emulator connects to only 1 controller cluster-node)
ODL Code Changes done for testing :
=============================
- Completely disable statistics polling and also disable Oper DS update
for flows (Only Node and NodeConnector updates in Oper DS allowed) provisioning
of flows in OpenflowPlugin (helium plugin is used). Therefore the term "pure
Routed RPC".
- Created a stub application which invokes OpenflowPlugin for
flow-programming (ie. invoke add-flow RPC) continuously (without waiting for
RPC result) with 1 thread per switch
Test Scenario 1 (Pure Local RPC) :
===========================
- Emulate 5 switches using above network simulator and connect the same
to *Node 1* of cluster
- Pump flows using RPC from same *Node (ie Node 1)*
Observed flow-programming rate on network simulator - upto 23K flows/second.
Did not try more switches because this type of traffic would only constitute
1/3 when equal number of switches are connected.
Test Scenario 2 (Remote RPC) :
=========================
- Emulate 5 switches using above network simulator and connect the same
to *Node 2* of cluster
- Pump flows using RPC from *Node 1 (objective is to stress akka RPC
cluster with as much routed-RPC traffic as possible)*
Observed flow-programming rate on network simulator - upto 60 (Sixty)
flows/second.
Additional observations for scenario 2:
=================================
1. For scenario 2, RPC messages become so overwhelming that Akka starts
hitting UNREACHABLE <-> REACHABLE flapping across nodes sending RPC and node
receiving RPC. This transition fills the log evry extensively
2. Above makes shards to jump into fray so that they start their
activities (which may further impact RPC traffic), all standard akka reactions
like ignoring gossip messages (akka's own gossip NOT RPC gossip) get triggered
which I think amplifies the effect and brings down flow-rate
3. Looks like Akka is extremely stressed and its *EndpointWriter on
controller - from where flows are pushed* , complains following :
2016-01-25 02:39:03,526 | WARN | lt-dispatcher-19 | EndpointWriter
| 129 - com.typesafe.akka.slf4j - 2.3.14 | [233271] buffered messages in
EndpointWriter for [akka.tcp://[email protected]:2550].
*You should probably implement flow control to avoid flooding the remote
connection.*
Clarifications :
===========
1. Has anybody benchmarked this earlier and had seen different numbers ?
2. We assume that inclusion of statistics and Oper DS update in
Openflowplugin is only going to make the above test still worse (may be even
make cluster unstable) . So we disabled them to assess "pure Routed RPC". Has
anybody observed contrary ?
3. Is this the max Routed RPC rate which could be expected even without
datastore updates ?
If required, I can raise a bug with collected logs.
Regards
Muthu
_______________________________________________
openflowplugin-dev mailing list
[email protected]
https://lists.opendaylight.org/mailman/listinfo/openflowplugin-dev
