Hi Luis
I am calculating config_datastore_write_time , config_datastore_write_rate and
Mininet_write_time as below. I tested those in 15-30-60 DPN in a low end setup.
The script is stable, however once you approve the method of calculation then I
can run those scripts in a high end server whenever it is available early next
week. I am not sure how to calculate oper-DS time yet, need suggestion for it.
Add Bulk Flow In Node
[Arguments] ${json_body_add} ${controller_index} ${timeout}
[Documentation] Add Bulk Flow in member ${controller_index} and wait
until operation is completed.
Add Bulk Flow ${json_body_add} ${controller_index}
${datastore_write_start_time}= Get Current Date
result_format=timestamp
Wait Until Write Finishes ${controller_index} ${timeout}
${datastore_write_end_time}= Get Current Date result_format=timestamp
${config_datastore_write_time}= Subtract Date From Date
${datastore_write_end_time} ${datastore_write_start_time}
Set Suite Variable ${datastore_write_start_time}
Set Suite Variable ${config_datastore_write_time}
Log To Console The config_datastore_write_time is
${config_datastore_write_time} seconds
Add Bulk Flow From Follower
[Documentation] 100020 Flows (3334 Flows per DPN) in 30 DPN added via
Follower Node1 and verify it gets applied in all instances.
BulkomaticKeywords.Add Bulk Flow In Node ${temp_json_config_add}
${Follower_Node_1} ${operation_timeout}
${config_datastore_write_time}= Convert To Number
${config_datastore_write_time}
${total_flows}= Convert To Integer ${flow_count_after_add}
${config_datastore_write_rate}= Evaluate ${flow_count_after_add} /
${config_datastore_write_time}
Log To Console The config_datastore_write_rate is
${config_datastore_write_rate}
Verify Flows In Switch Before Cluster Restart
[Documentation] Verify flows are installed in switch before cluster
restart.
${Mininet_write_start_time}= Get Current Date result_format=timestamp
Log To Console The Mininet_write_start_time is
${Mininet_write_start_time} seconds
MininetKeywords.Verify Aggregate Flows From Mininet Sessions
${mininet_conn_list} ${switch_count_per_controller}
${flow_count_per_switch} ${operation_timeout}
${Mininet_write_end_time}= Get Current Date result_format=timestamp
Log To Console The Mininet_write_end_time is ${Mininet_write_end_time}
seconds
${Mininet_write_time}= Subtract Date From Date
${Mininet_write_end_time} ${datastore_write_start_time}
Log To Console The Mininet_write_time is ${Mininet_write_time} seconds
==============================================================================
100K 15DPN 3Node Cluster Scale Up Test :: Test suite for Cluster with Bulk ...
==============================================================================
Check Shards Status And Initialize Variables :: Check Status for a... | PASS |
------------------------------------------------------------------------------
Get Inventory Follower Before Cluster Restart :: Find a follower i... | PASS |
------------------------------------------------------------------------------
Start Mininet Connect To Follower Node1 :: Start mininet with conn... | PASS |
------------------------------------------------------------------------------
Add Bulk Flow From Follower :: 100020 Flows (6668 Flows per DPN) i... The
config_datastore_write_time is 107.781 seconds
....The config_datastore_write_rate is 927.99287444
Add Bulk Flow From Follower :: 100020 Flows (6668 Flows per DPN) i... | PASS |
------------------------------------------------------------------------------
Verify Flows In Switch Before Cluster Restart :: Verify flows are ... .The
Mininet_write_start_time is 2016-07-29 05:51:27.084 seconds
...The Mininet_write_end_time is 2016-07-29 05:51:27.653 seconds
..The Mininet_write_time is 108.388 seconds
Verify Flows In Switch Before Cluster Restart :: Verify flows are ... | PASS |
------------------------------------------------------------------------------
Stop Mininet Connected To Follower Node1 and Exit :: Stop mininet ... | PASS |
------------------------------------------------------------------------------
Delete All Flows From Follower Node1 :: 100020 Flows deleted via F... | PASS |
------------------------------------------------------------------------------
==============================================================================
100K 30DPN 3Node Cluster Scale Up Test :: Test suite for Cluster with Bulk ...
==============================================================================
Check Shards Status And Initialize Variables :: Check Status for a... | PASS |
------------------------------------------------------------------------------
Get Inventory Follower Before Cluster Restart :: Find a follower i... | PASS |
------------------------------------------------------------------------------
Start Mininet Connect To Follower Node1 :: Start mininet with conn... | PASS |
------------------------------------------------------------------------------
Add Bulk Flow From Follower :: 100020 Flows (3334 Flows per DPN) i... The
config_datastore_write_time is 166.36 seconds
....The config_datastore_write_rate is 601.226256312
Add Bulk Flow From Follower :: 100020 Flows (3334 Flows per DPN) i... | PASS |
------------------------------------------------------------------------------
Verify Flows In Switch Before Cluster Restart :: Verify flows are ... .The
Mininet_write_start_time is 2016-07-29 07:32:38.237 seconds
...The Mininet_write_end_time is 2016-07-29 07:32:39.431 seconds
..The Mininet_write_time is 167.615 seconds
Verify Flows In Switch Before Cluster Restart :: Verify flows are ... | PASS |
------------------------------------------------------------------------------
Stop Mininet Connected To Follower Node1 and Exit :: Stop mininet ... | PASS |
------------------------------------------------------------------------------
Delete All Flows From Follower Node1 :: 100020 Flows deleted via F... | PASS |
------------------------------------------------------------------------------
==============================================================================
100K 60DPN 3Node Cluster Scale Up Test :: Test suite for Cluster with Bulk ...
==============================================================================
Check Shards Status And Initialize Variables :: Check Status for a... | PASS |
------------------------------------------------------------------------------
Get Inventory Follower Before Cluster Restart :: Find a follower i... | PASS |
------------------------------------------------------------------------------
Start Mininet Connect To Follower Node1 :: Start mininet with conn... | PASS |
------------------------------------------------------------------------------
Add Bulk Flow From Follower :: 100020 Flows (1667 Flows per DPN) i... The
config_datastore_write_time is 211.392 seconds
....The config_datastore_write_rate is 473.149409628
Add Bulk Flow From Follower :: 100020 Flows (1667 Flows per DPN) i... | PASS |
------------------------------------------------------------------------------
Verify Flows In Switch Before Cluster Restart :: Verify flows are ... .The
Mininet_write_start_time is 2016-07-29 08:00:58.648 seconds
...The Mininet_write_end_time is 2016-07-29 08:01:00.518 seconds
..The Mininet_write_time is 213.384 seconds
Verify Flows In Switch Before Cluster Restart :: Verify flows are ... | PASS |
------------------------------------------------------------------------------
Stop Mininet Connected To Follower Node1 and Exit :: Stop mininet ... | PASS |
------------------------------------------------------------------------------
Delete All Flows From Follower Node1 :: 100020 Flows deleted via F... | PASS |
------------------------------------------------------------------------------
Thanks
Sanjib
-----Original Message-----
From: Luis Gomez [mailto:[email protected]]
Sent: 29 July 2016 02:27
To: Sanjib Mohapatra
Cc: Abhijit Kumbhare; [email protected];
openflowplugin-dev; Jin Gan; BUCI DUAC R&D INDIA PDG SDNC XFT6
Subject: Re: [integration-dev] OpenFlow tests for next ODL perf paper
> On Jul 28, 2016, at 12:24 PM, Sanjib Mohapatra
> <[email protected]> wrote:
>
> Hi Luis
>
> I would like to have one clarification, By “100K flows on an OF network of
> different size: 16-32-64-128 switches”, does it mean same 100K flows for
> 16-32-64 switches ?
The goal is to run the perf test on different OF network sizes so we know what
is the impact (if any) of this.
>
> I have some local scripts to run 150K-15 DPN, 330K-33DPN etc, I run it with
> following configuration optimisation in controller nodes. The DPNs are
> equally spread across controller nodes.
If you increase the number of flows from one network to another, you are
running some sort of scalability test mixed with the performance test that can
confuse the result, like the perf degradation is because of the number of
switches or because the number of flows that has to be stored and programmed…
For this reason I would prefer to run the test with the same condition of fix
100K flows divided among the switches that are available in the network. It is
also that testing this way we can easily compare one network result to another.
>
> I. fsync = off in akka.conf
What is this setting?
> II. changing root logging to ERROR from INFO
> III. Set JAVA_MIN_MEM=512M JAVA_MAX_MEM=8192M in karaf
> IV. <skip-table-features>true</skip-table-features> in
> 42-openflowplugin-He.xml
>
> I can rerun those scripts by tweaking it to the current requirement . Do let
> me know if any other configuration optimisation is required. Also could you
> please provide path to Boron distribution. I ran below scripts in stable
> Beryllium.
Master distribution is in nexus:
https://nexus.opendaylight.org/content/repositories/opendaylight.snapshot/org/opendaylight/integration/distribution-karaf/0.5.0-SNAPSHOT/
Does your test automation provide all the perf numbers required in this TC:
controller programming time, switch programming time, and flow confirmation
delay? If not, that is the next thing to work on.
>
>
> root@mininet-vm:/home/mininet/integration/test/csit/suites/openflowplu
> gin/Clustering_Bulkomatic# pybot -L TRACE -v MININET_USER:mininet -v
> USER_HOME:/home/mininet -v CONTROLLER:10.183.181.51 -v
> CONTROLLER1:10.183.181.52 -v CONTROLLER2:10.183.181.53 -v USER:root -v
> PASSWORD:rootroot -v WORKSPACE:/home/mininet -v
> BUNDLEFOLDER:/controller-Be/deploy/current/odl -v
> DEFAULT_LINUX_PROMPT:\# -v NUM_ODL_SYSTEM:3 -v
> MININET_PASSWORD:rootroot -v
> OVS_SWITCH_FILE:Multi_Switch_Medium_Config.py
> 150K__Cluster_Reconcilliation_Multi_DPN.robot
> ======================================================================
> ======== Cluster Reconcilliation Multi DPN :: Test suite for Cluster
> with Bulk Flows...
> ======================================================================
> ======== Check Shards Status And Initialize Variables :: Check Status
> for a... | PASS |
> ----------------------------------------------------------------------
> -------- Get Inventory Follower Before Cluster Restart :: Find a
> follower i... | PASS |
> ----------------------------------------------------------------------
> -------- Start Mininet Connect To Follower Node1 :: Start mininet with
> conn... | PASS |
> ----------------------------------------------------------------------
> -------- Add Bulk Flow From Follower :: 10000 Flows (10K per DPN) in
> 15 DPN... | PASS |
> ------------------------------------------------------------------------------
> Verify Flows In Switch :: Verify 150K flows are installed. | PASS |
> ----------------------------------------------------------------------
> -------- Stop Mininet Connected To Follower Node1 and Exit :: Stop
> mininet ... | PASS |
> ----------------------------------------------------------------------
> -------- Delete All Flows From Follower Node1 :: 150000 Flows deleted
> via F... | PASS |
> ----------------------------------------------------------------------
> -------- Cluster Reconcilliation Multi DPN :: Test suite for Cluster
> with B... | PASS |
> 7 critical tests, 7 passed, 0 failed
> 7 tests total, 7 passed, 0 failed
> ======================================================================
> ========
>
>
> root@mininet-vm:/home/mininet/integration/test/csit/suites/openflowplu
> gin/Clustering_Bulkomatic# pybot -L TRACE -v MININET_USER:mininet -v
> USER_HOME:/home/mininet -v CONTROLLER:10.183.181.51 -v
> CONTROLLER1:10.183.181.52 -v CONTROLLER2:10.183.181.53 -v USER:root -v
> PASSWORD:rootroot -v WORKSPACE:/home/mininet -v
> BUNDLEFOLDER:/controller-Be/deploy/current/odl -v
> DEFAULT_LINUX_PROMPT:\# -v NUM_ODL_SYSTEM:3 -v
> MININET_PASSWORD:rootroot -v
> OVS_SWITCH_FILE:Multi_11Switch_Medium_Config.py
> 330K__Cluster_Reconcilliation_Multi_DPN.robot
> ======================================================================
> ======== Cluster Reconcilliation Multi DPN :: Test suite for Cluster
> with Bulk Flows...
> ======================================================================
> ======== Check Shards Status And Initialize Variables :: Check Status
> for a... | PASS |
> ----------------------------------------------------------------------
> -------- Get Inventory Follower Before Cluster Restart :: Find a
> follower i... | PASS |
> ----------------------------------------------------------------------
> -------- Start Mininet Connect To Follower Node1 :: Start mininet with
> conn... | PASS |
> ----------------------------------------------------------------------
> -------- Add Bulk Flow From Follower :: 10000 Flows (10K per DPN) in
> 33 DPN... | PASS |
> ------------------------------------------------------------------------------
> Verify Flows In Switch :: Verify 330K flows are installed. | PASS |
> ----------------------------------------------------------------------
> -------- Stop Mininet Connected To Follower Node1 and Exit :: Stop
> mininet ... | PASS |
> ----------------------------------------------------------------------
> -------- Delete All Flows From Follower Node1 :: 330000 Flows deleted
> via F... | PASS |
> ----------------------------------------------------------------------
> -------- Cluster Reconcilliation Multi DPN :: Test suite for Cluster
> with B... | PASS |
> 7 critical tests, 7 passed, 0 failed
> 7 tests total, 7 passed, 0 failed
> ======================================================================
> ========
> Output:
> /home/mininet/integration/test/csit/suites/openflowplugin/Clustering_Bulkomatic/output.xml
> Log:
> /home/mininet/integration/test/csit/suites/openflowplugin/Clustering_Bulkomatic/log.html
> Report:
> /home/mininet/integration/test/csit/suites/openflowplugin/Clustering_B
> ulkomatic/report.html
> root@mininet-vm:/home/mininet/integration/test/csit/suites/openflowplu
> gin/Clustering_Bulkomatic#
>
> Thanks
> Sanjib
>
> From: Luis Gomez [mailto:[email protected]]
> Sent: 28 July 2016 05:17
> To: Abhijit Kumbhare
> Cc: [email protected]; openflowplugin-dev; Jin
> Gan; Sanjib Mohapatra
> Subject: Re: [integration-dev] OpenFlow tests for next ODL perf paper
>
> Hi Abhijit,
>
> We can definitely leverage any automation your team will prepare to test all
> the combinations below, but just to be clear the ODL perf paper we will only
> include out-of-the-box Boron release.
>
> BR/Luis
>
> On Jul 27, 2016, at 4:23 PM, Abhijit Kumbhare <[email protected]> wrote:
>
> Hi Luis,
>
> I was discussing with our OpenFlow team (Manohar/Muthu/Shuva/Vinayak) today
> about the tests - and one of the things which would be very interesting would
> be the bulk-o-matic with small/medium/large configs (same as your test 2) and
> the following combinations:
>
> 1. Lithium design + the old FRM
> 2. Lithium design + the new FRM => are there improvements?
> 3. He design => bulk-o-matic may not have been used for the He/Li
>
> They were also planning to discuss this with Sanjib in their daytime today if
> this has been already done.
>
> Thanks,
> Abhijit
>
> On Wed, Jul 27, 2016 at 3:13 PM, Luis Gomez <[email protected]> wrote:
> Hi all,
>
> I got the action point from last S3P call to start some discussion around
> OpenFlow tests we can do for next ODL perf paper (Boron).
>
> I am not sure we will have time for all the below but ideally I was thinking
> in 4 tests:
>
> 1) REST programming performance:
>
> - Goal: Measure OF programming rate using NB REST interface
> - Methodology: Use test REST scripts (flows in datastore) to program 100K
> flows on an OF network of different size: 16-32-64-128 switches, etc...
> - Test variations: Use single flow/REST request and multiple flows/REST
> (bulk) request.
> - Collected data: controller programming time (from first to last flow) from
> REST script, switch programming time (from first to last flow) polling the
> OVS, flow confirmation delay (time after T1) polling the operational DS.
>
> 2) Java programming performance:
>
> - Goal: Measure OF programming rate using internal Java interface
> - Methodology: Use bluk-o-matic application (flows in datastore or rpc) to
> program 100K flows on an OF network of different size: 16-32-64-128 switches,
> etc...
> - Test variations: Use single flow/REST request and multiple flows/REST
> (bulk) request.
> - Collected data: controller programming time (from first to last flow) from
> bulk-o-matic, switch programming time (from first to last flow) polling the
> OVS, flow confirmation delay (time after T1) polling the operational DS.
>
> 3) Network message processing latency:
>
> - Goal: Measure OF packet message processing time
> - Methodology: Use some OF public tool (Cbench, MT-Cbench, SDN-blaster) to
> generate OF packets and measure the delay (latency mode) of the received
> controller flows on an OF network of different size: 16-32-64-128 switches,
> etc...
> - Test variations: Use controller drop-test application in DS and RPC mode.
> - Collected data: packet processing rate (latency=1/rate)
>
> 4) Topology scalability:
>
> - Goal: Scale OF network and measure learning time.
> - Methodology: Use some OF public tool (Mininet, Multinet) to generate
> different sizes of large topologies: 1000, 2000, 3000 switches, etc...
> - Collected data: Time for the controller to learn about the topology.
>
> In addition the same tests (or a subset) should run in a cluster environment
> (3 node cluster).
>
> The main problem we have today for running and automating the above is people
> resources, so far Jin and Sanjib offered to help but more help would be
> appreciated.
>
> BR/Luis
>
> _______________________________________________
> integration-dev mailing list
> [email protected]
> https://lists.opendaylight.org/mailman/listinfo/integration-dev
_______________________________________________
openflowplugin-dev mailing list
[email protected]
https://lists.opendaylight.org/mailman/listinfo/openflowplugin-dev
