Hi Luis and all, After thinking about the Collected Data for Test Case 1 (similar for 2), I thought that drawing examples of these measured times might help to clarify where/how they are measured and what they mean. I uploaded a 2 page pdf file which includes a small test setup and associates the time measured with various components (NB intfc, etc.). [0]
I took a guess the “T1” is the start of all REST/NB activity, since it is a reference time for the data collected by polling. One of the times measured will represent the overall programming time (but not sure which one). The second page looks at alternative time metrics which might be collected with wireshark captures of NB and SB interfaces (as we have discussed on cperf list). I’ll be glad to get comments and corrections on this, I’ll certainly learn something. regards, Al [0] https://wiki.opendaylight.org/images/8/88/ODL_tests_times_r0.pdf From: [email protected] [mailto:[email protected]] On Behalf Of Shuva Jyoti Kar Sent: Tuesday, August 02, 2016 10:11 PM To: Luis Gomez Cc: Jin Gan; [email protected]; openflowplugin-dev Subject: Re: [integration-dev] OpenFlow tests for next ODL perf paper Sure Luis I will. The flag for table features for the li-plugin design is currently under review https://git.opendaylight.org/gerrit/#/c/42821/ Disabling stats-collection for the perf measurement is also another option that I am trying out. Thanks Shuva From: Luis Gomez [mailto:[email protected]] Sent: Wednesday, August 03, 2016 6:36 AM To: Shuva Jyoti Kar Cc: Vratko Polak -X (vrpolak - PANTHEON TECHNOLOGIES at Cisco); openflowplugin-dev; Jin Gan; [email protected]<mailto:[email protected]> Subject: Re: [integration-dev] OpenFlow tests for next ODL perf paper Hi Shuva, I started this: https://wiki.opendaylight.org/view/Integration/Test/S3P/OpenFlow Can you please update information on how to disable table features or enable new FRS for testing? Feel free to add any other thing. Thanks/Luis On Aug 2, 2016, at 4:21 AM, Shuva Jyoti Kar <[email protected]<mailto:[email protected]>> wrote: I strongly agree to it. Do we have any wiki /doc to update ? -----Original Message----- From: Vratko Polak -X (vrpolak - PANTHEON TECHNOLOGIES at Cisco) [mailto:[email protected]] Sent: Tuesday, August 02, 2016 4:47 PM To: Luis Gomez; Shuva Jyoti Kar Cc: openflowplugin-dev; Jin Gan; [email protected]<mailto:[email protected]>; BUCI DUAC R&D INDIA PDG SDNC XFT6 Subject: RE: [integration-dev] OpenFlow tests for next ODL perf paper I recommend to keep track of every such improvement. We can do tests for "optimized" configuration, but we should also have few suites/runs with "default" configuration, just to show how big difference it makes. Especially if only Robot suites (as opposed to configuration steps) would be automated. Vratko. -----Original Message----- From: [email protected]<mailto:[email protected]> [mailto:[email protected]] On Behalf Of Luis Gomez Sent: 29 July, 2016 08:45 To: Shuva Jyoti Kar <[email protected]<mailto:[email protected]>> Cc: openflowplugin-dev <[email protected]<mailto:[email protected]>>; Jin Gan <[email protected]<mailto:[email protected]>>; [email protected]<mailto:[email protected]>; BUCI DUAC R&D INDIA PDG SDNC XFT6 <[email protected]<mailto:[email protected]>> Subject: Re: [integration-dev] OpenFlow tests for next ODL perf paper Yes, we set up that too in some tests to avoid too many logs. On Jul 28, 2016, at 11:43 PM, Shuva Jyoti Kar <[email protected]<mailto:[email protected]>> wrote: Also it would be great if we could the test with the logging in ERROR mode only -----Original Message----- From: Luis Gomez [mailto:[email protected]] Sent: Friday, July 29, 2016 12:11 PM To: Shuva Jyoti Kar Cc: Sanjib Mohapatra; Abhijit Kumbhare; [email protected]<mailto:[email protected]>; openflowplugin-dev; Jin Gan; BUCI DUAC R&D INDIA PDG SDNC XFT6 Subject: Re: [integration-dev] OpenFlow tests for next ODL perf paper Marcus has a setup with SSD in Intel Lab that we used for 1st paper and also plan to use for the second. Disabling persistence is something we can do in the ODL infra for example. BR/Luis On Jul 28, 2016, at 11:34 PM, Shuva Jyoti Kar <[email protected]<mailto:[email protected]>> wrote: Ahhh....can we have 2 sets - one with SDD and the other with persistence off ? -----Original Message----- From: Luis Gomez [mailto:[email protected]] Sent: Friday, July 29, 2016 12:04 PM To: Shuva Jyoti Kar Cc: Sanjib Mohapatra; Abhijit Kumbhare; [email protected]<mailto:[email protected]>; openflowplugin-dev; Jin Gan; BUCI DUAC R&D INDIA PDG SDNC XFT6 Subject: Re: [integration-dev] OpenFlow tests for next ODL perf paper Thanks Shuva, we will not use it then. On the other hand, in general for performance test we either use fast disk (SSD) or disable the datastore persistence [1] so that controller does not need to write into physical HDD every datastore transaction. [1] https://wiki.opendaylight.org/view/Integration/Distribution/Cluster_S c ripts On Jul 28, 2016, at 11:28 PM, Shuva Jyoti Kar <[email protected]<mailto:[email protected]>> wrote: It was only experimental. There's some level of testing(triggering scenarios like kernel panic) to be done before actually formalizing it. So would not recommend formal testing using it now. Thanks Shuva -----Original Message----- From: Sanjib Mohapatra Sent: Friday, July 29, 2016 11:42 AM To: Luis Gomez Cc: Abhijit Kumbhare; [email protected]<mailto:[email protected]>; openflowplugin-dev; Jin Gan; BUCI DUAC R&D INDIA PDG SDNC XFT6 Subject: RE: [integration-dev] OpenFlow tests for next ODL perf paper Hi Luis Got a recommendation from dev to use fsync = off in akka.conf under "odl-cluster-data" It was tested at their end and they found performance to be increased 2x. persistence { journal { leveldb { # Set native = off to use a Java-only implementation of leveldb. # Note that the Java-only version is not currently considered by Akka to be production quality. fsync = off } } } Thanks Sanjib -----Original Message----- From: Luis Gomez [mailto:[email protected]] Sent: 29 July 2016 02:27 To: Sanjib Mohapatra Cc: Abhijit Kumbhare; [email protected]<mailto:[email protected]>; openflowplugin-dev; Jin Gan; BUCI DUAC R&D INDIA PDG SDNC XFT6 Subject: Re: [integration-dev] OpenFlow tests for next ODL perf paper On Jul 28, 2016, at 12:24 PM, Sanjib Mohapatra <[email protected]<mailto:[email protected]>> wrote: Hi Luis I would like to have one clarification, By “100K flows on an OF network of different size: 16-32-64-128 switches”, does it mean same 100K flows for 16-32-64 switches ? The goal is to run the perf test on different OF network sizes so we know what is the impact (if any) of this. I have some local scripts to run 150K-15 DPN, 330K-33DPN etc, I run it with following configuration optimisation in controller nodes. The DPNs are equally spread across controller nodes. If you increase the number of flows from one network to another, you are running some sort of scalability test mixed with the performance test that can confuse the result, like the perf degradation is because of the number of switches or because the number of flows that has to be stored and programmed… For this reason I would prefer to run the test with the same condition of fix 100K flows divided among the switches that are available in the network. It is also that testing this way we can easily compare one network result to another. I. fsync = off in akka.conf What is this setting? II. changing root logging to ERROR from INFO III. Set JAVA_MIN_MEM=512M JAVA_MAX_MEM=8192M in karaf IV. <skip-table-features>true</skip-table-features> in 42-openflowplugin-He.xml I can rerun those scripts by tweaking it to the current requirement . Do let me know if any other configuration optimisation is required. Also could you please provide path to Boron distribution. I ran below scripts in stable Beryllium. Master distribution is in nexus: https://nexus.opendaylight.org/content/repositories/opendaylight.sna p s hot/org/opendaylight/integration/distribution-karaf/0.5.0-SNAPSHOT/ Does your test automation provide all the perf numbers required in this TC: controller programming time, switch programming time, and flow confirmation delay? If not, that is the next thing to work on. root@mininet-vm:/home/mininet/integration/test/csit/suites/openflow p l u gin/Clustering_Bulkomatic# pybot -L TRACE -v MININET_USER:mininet -v USER_HOME:/home/mininet -v CONTROLLER:10.183.181.51 -v CONTROLLER1:10.183.181.52 -v CONTROLLER2:10.183.181.53 -v USER:root -v PASSWORD:rootroot -v WORKSPACE:/home/mininet -v BUNDLEFOLDER:/controller-Be/deploy/current/odl -v DEFAULT_LINUX_PROMPT:\# -v NUM_ODL_SYSTEM:3 -v MININET_PASSWORD:rootroot -v OVS_SWITCH_FILE:Multi_Switch_Medium_Config.py 150K__Cluster_Reconcilliation_Multi_DPN.robot =================================================================== = = = ======== Cluster Reconcilliation Multi DPN :: Test suite for Cluster with Bulk Flows... =================================================================== = = = ======== Check Shards Status And Initialize Variables :: Check Status for a... | PASS | ------------------------------------------------------------------- - - - -------- Get Inventory Follower Before Cluster Restart :: Find a follower i... | PASS | ------------------------------------------------------------------- - - - -------- Start Mininet Connect To Follower Node1 :: Start mininet with conn... | PASS | ------------------------------------------------------------------- - - - -------- Add Bulk Flow From Follower :: 10000 Flows (10K per DPN) in 15 DPN... | PASS | ------------------------------------------------------------------------------ Verify Flows In Switch :: Verify 150K flows are installed. | PASS | ------------------------------------------------------------------- - - - -------- Stop Mininet Connected To Follower Node1 and Exit :: Stop mininet ... | PASS | ------------------------------------------------------------------- - - - -------- Delete All Flows From Follower Node1 :: 150000 Flows deleted via F... | PASS | ------------------------------------------------------------------- - - - -------- Cluster Reconcilliation Multi DPN :: Test suite for Cluster with B... | PASS | 7 critical tests, 7 passed, 0 failed 7 tests total, 7 passed, 0 failed =================================================================== = = = ======== root@mininet-vm:/home/mininet/integration/test/csit/suites/openflow p l u gin/Clustering_Bulkomatic# pybot -L TRACE -v MININET_USER:mininet -v USER_HOME:/home/mininet -v CONTROLLER:10.183.181.51 -v CONTROLLER1:10.183.181.52 -v CONTROLLER2:10.183.181.53 -v USER:root -v PASSWORD:rootroot -v WORKSPACE:/home/mininet -v BUNDLEFOLDER:/controller-Be/deploy/current/odl -v DEFAULT_LINUX_PROMPT:\# -v NUM_ODL_SYSTEM:3 -v MININET_PASSWORD:rootroot -v OVS_SWITCH_FILE:Multi_11Switch_Medium_Config.py 330K__Cluster_Reconcilliation_Multi_DPN.robot =================================================================== = = = ======== Cluster Reconcilliation Multi DPN :: Test suite for Cluster with Bulk Flows... =================================================================== = = = ======== Check Shards Status And Initialize Variables :: Check Status for a... | PASS | ------------------------------------------------------------------- - - - -------- Get Inventory Follower Before Cluster Restart :: Find a follower i... | PASS | ------------------------------------------------------------------- - - - -------- Start Mininet Connect To Follower Node1 :: Start mininet with conn... | PASS | ------------------------------------------------------------------- - - - -------- Add Bulk Flow From Follower :: 10000 Flows (10K per DPN) in 33 DPN... | PASS | ------------------------------------------------------------------------------ Verify Flows In Switch :: Verify 330K flows are installed. | PASS | ------------------------------------------------------------------- - - - -------- Stop Mininet Connected To Follower Node1 and Exit :: Stop mininet ... | PASS | ------------------------------------------------------------------- - - - -------- Delete All Flows From Follower Node1 :: 330000 Flows deleted via F... | PASS | ------------------------------------------------------------------- - - - -------- Cluster Reconcilliation Multi DPN :: Test suite for Cluster with B... | PASS | 7 critical tests, 7 passed, 0 failed 7 tests total, 7 passed, 0 failed =================================================================== = = = ======== Output: /home/mininet/integration/test/csit/suites/openflowplugin/Clustering_Bulkomatic/output.xml Log: /home/mininet/integration/test/csit/suites/openflowplugin/Clustering_Bulkomatic/log.html Report: /home/mininet/integration/test/csit/suites/openflowplugin/Clusterin g _ B ulkomatic/report.html root@mininet-vm:/home/mininet/integration/test/csit/suites/openflow p l u gin/Clustering_Bulkomatic# Thanks Sanjib From: Luis Gomez [mailto:[email protected]] Sent: 28 July 2016 05:17 To: Abhijit Kumbhare Cc: [email protected]<mailto:[email protected]>; openflowplugin-dev; Jin Gan; Sanjib Mohapatra Subject: Re: [integration-dev] OpenFlow tests for next ODL perf paper Hi Abhijit, We can definitely leverage any automation your team will prepare to test all the combinations below, but just to be clear the ODL perf paper we will only include out-of-the-box Boron release. BR/Luis On Jul 27, 2016, at 4:23 PM, Abhijit Kumbhare <[email protected]<mailto:[email protected]>> wrote: Hi Luis, I was discussing with our OpenFlow team (Manohar/Muthu/Shuva/Vinayak) today about the tests - and one of the things which would be very interesting would be the bulk-o-matic with small/medium/large configs (same as your test 2) and the following combinations: 1. Lithium design + the old FRM 2. Lithium design + the new FRM => are there improvements? 3. He design => bulk-o-matic may not have been used for the He/Li They were also planning to discuss this with Sanjib in their daytime today if this has been already done. Thanks, Abhijit On Wed, Jul 27, 2016 at 3:13 PM, Luis Gomez <[email protected]<mailto:[email protected]>> wrote: Hi all, I got the action point from last S3P call to start some discussion around OpenFlow tests we can do for next ODL perf paper (Boron). I am not sure we will have time for all the below but ideally I was thinking in 4 tests: 1) REST programming performance: - Goal: Measure OF programming rate using NB REST interface - Methodology: Use test REST scripts (flows in datastore) to program 100K flows on an OF network of different size: 16-32-64-128 switches, etc... - Test variations: Use single flow/REST request and multiple flows/REST (bulk) request. - Collected data: controller programming time (from first to last flow) from REST script, switch programming time (from first to last flow) polling the OVS, flow confirmation delay (time after T1) polling the operational DS. 2) Java programming performance: - Goal: Measure OF programming rate using internal Java interface - Methodology: Use bluk-o-matic application (flows in datastore or rpc) to program 100K flows on an OF network of different size: 16-32-64-128 switches, etc... - Test variations: Use single flow/REST request and multiple flows/REST (bulk) request. - Collected data: controller programming time (from first to last flow) from bulk-o-matic, switch programming time (from first to last flow) polling the OVS, flow confirmation delay (time after T1) polling the operational DS. 3) Network message processing latency: - Goal: Measure OF packet message processing time - Methodology: Use some OF public tool (Cbench, MT-Cbench, SDN-blaster) to generate OF packets and measure the delay (latency mode) of the received controller flows on an OF network of different size: 16-32-64-128 switches, etc... - Test variations: Use controller drop-test application in DS and RPC mode. - Collected data: packet processing rate (latency=1/rate) 4) Topology scalability: - Goal: Scale OF network and measure learning time. - Methodology: Use some OF public tool (Mininet, Multinet) to generate different sizes of large topologies: 1000, 2000, 3000 switches, etc... - Collected data: Time for the controller to learn about the topology. In addition the same tests (or a subset) should run in a cluster environment (3 node cluster). The main problem we have today for running and automating the above is people resources, so far Jin and Sanjib offered to help but more help would be appreciated. BR/Luis _______________________________________________ integration-dev mailing list [email protected]<mailto:[email protected]> https://lists.opendaylight.org/mailman/listinfo/integration-dev _______________________________________________ integration-dev mailing list [email protected]<mailto:[email protected]> https://lists.opendaylight.org/mailman/listinfo/integration-dev
_______________________________________________ openflowplugin-dev mailing list [email protected] https://lists.opendaylight.org/mailman/listinfo/openflowplugin-dev
