OK, we will take a look at that too, specially when we test with the OVS. > On Jul 29, 2016, at 12:02 AM, Shuva Jyoti Kar <[email protected]> > wrote: > > Hmmm...i am not seeing so...:( > > -----Original Message----- > From: Luis Gomez [mailto:[email protected]] > Sent: Friday, July 29, 2016 12:32 PM > To: Shuva Jyoti Kar > Cc: openflowplugin-dev; Jin Gan; [email protected]; > Sanjib Mohapatra; Muthukumaran K; Abhijit Kumbhare > Subject: Re: [openflowplugin-dev] [integration-dev] OpenFlow tests for next > ODL perf paper > > That is the default behavior in Boron right? > >> On Jul 28, 2016, at 11:55 PM, Shuva Jyoti Kar <[email protected]> >> wrote: >> >> And for scaling out on the number of switches, can we have table-features >> turned off ? >> >> -----Original Message----- >> From: Muthukumaran K >> Sent: Friday, July 29, 2016 12:23 PM >> To: Luis Gomez; Shuva Jyoti Kar >> Cc: openflowplugin-dev; Jin Gan; >> [email protected]; Sanjib Mohapatra; BUCI DUAC >> R&D INDIA PDG SDNC XFT6 >> Subject: RE: [openflowplugin-dev] [integration-dev] OpenFlow tests for >> next ODL perf paper >> >> With logs set to ERROR it could be almost like persistence-off++ :-) >> >> -----Original Message----- >> From: [email protected] >> [mailto:[email protected]] On Behalf >> Of Luis Gomez >> Sent: Friday, July 29, 2016 12:15 PM >> To: Shuva Jyoti Kar >> Cc: openflowplugin-dev; Jin Gan; >> [email protected]; Sanjib Mohapatra; BUCI DUAC >> R&D INDIA PDG SDNC XFT6 >> Subject: Re: [openflowplugin-dev] [integration-dev] OpenFlow tests for >> next ODL perf paper >> >> Yes, we set up that too in some tests to avoid too many logs. >> >>> On Jul 28, 2016, at 11:43 PM, Shuva Jyoti Kar >>> <[email protected]> wrote: >>> >>> Also it would be great if we could the test with the logging in ERROR >>> mode only >>> >>> -----Original Message----- >>> From: Luis Gomez [mailto:[email protected]] >>> Sent: Friday, July 29, 2016 12:11 PM >>> To: Shuva Jyoti Kar >>> Cc: Sanjib Mohapatra; Abhijit Kumbhare; >>> [email protected]; openflowplugin-dev; Jin Gan; >>> BUCI DUAC R&D INDIA PDG SDNC XFT6 >>> Subject: Re: [integration-dev] OpenFlow tests for next ODL perf paper >>> >>> Marcus has a setup with SSD in Intel Lab that we used for 1st paper and >>> also plan to use for the second. Disabling persistence is something we can >>> do in the ODL infra for example. >>> >>> BR/Luis >>> >>> >>>> On Jul 28, 2016, at 11:34 PM, Shuva Jyoti Kar >>>> <[email protected]> wrote: >>>> >>>> Ahhh....can we have 2 sets - one with SDD and the other with persistence >>>> off ? >>>> >>>> -----Original Message----- >>>> From: Luis Gomez [mailto:[email protected]] >>>> Sent: Friday, July 29, 2016 12:04 PM >>>> To: Shuva Jyoti Kar >>>> Cc: Sanjib Mohapatra; Abhijit Kumbhare; >>>> [email protected]; openflowplugin-dev; Jin Gan; >>>> BUCI DUAC R&D INDIA PDG SDNC XFT6 >>>> Subject: Re: [integration-dev] OpenFlow tests for next ODL perf >>>> paper >>>> >>>> Thanks Shuva, we will not use it then. >>>> >>>> On the other hand, in general for performance test we either use fast disk >>>> (SSD) or disable the datastore persistence [1] so that controller does not >>>> need to write into physical HDD every datastore transaction. >>>> >>>> [1] >>>> https://wiki.opendaylight.org/view/Integration/Distribution/Cluster_ >>>> S >>>> c >>>> ripts >>>> >>>> >>>>> On Jul 28, 2016, at 11:28 PM, Shuva Jyoti Kar >>>>> <[email protected]> wrote: >>>>> >>>>> It was only experimental. There's some level of testing(triggering >>>>> scenarios like kernel panic) to be done before actually formalizing it. >>>>> So would not recommend formal testing using it now. >>>>> >>>>> >>>>> Thanks >>>>> Shuva >>>>> -----Original Message----- >>>>> From: Sanjib Mohapatra >>>>> Sent: Friday, July 29, 2016 11:42 AM >>>>> To: Luis Gomez >>>>> Cc: Abhijit Kumbhare; [email protected]; >>>>> openflowplugin-dev; Jin Gan; BUCI DUAC R&D INDIA PDG SDNC XFT6 >>>>> Subject: RE: [integration-dev] OpenFlow tests for next ODL perf >>>>> paper >>>>> >>>>> Hi Luis >>>>> >>>>> Got a recommendation from dev to use fsync = off in akka.conf under >>>>> "odl-cluster-data" >>>>> It was tested at their end and they found performance to be increased 2x. >>>>> >>>>> persistence { >>>>> journal { >>>>> leveldb { >>>>> # Set native = off to use a Java-only implementation of leveldb. >>>>> # Note that the Java-only version is not currently considered by Akka >>>>> to be production quality. >>>>> >>>>> fsync = off >>>>> } >>>>> } >>>>> } >>>>> >>>>> Thanks >>>>> Sanjib >>>>> -----Original Message----- >>>>> From: Luis Gomez [mailto:[email protected]] >>>>> Sent: 29 July 2016 02:27 >>>>> To: Sanjib Mohapatra >>>>> Cc: Abhijit Kumbhare; [email protected]; >>>>> openflowplugin-dev; Jin Gan; BUCI DUAC R&D INDIA PDG SDNC XFT6 >>>>> Subject: Re: [integration-dev] OpenFlow tests for next ODL perf >>>>> paper >>>>> >>>>> >>>>>> On Jul 28, 2016, at 12:24 PM, Sanjib Mohapatra >>>>>> <[email protected]> wrote: >>>>>> >>>>>> Hi Luis >>>>>> >>>>>> I would like to have one clarification, By “100K flows on an OF network >>>>>> of different size: 16-32-64-128 switches”, does it mean same 100K flows >>>>>> for 16-32-64 switches ? >>>>> >>>>> The goal is to run the perf test on different OF network sizes so we know >>>>> what is the impact (if any) of this. >>>>> >>>>>> >>>>>> I have some local scripts to run 150K-15 DPN, 330K-33DPN etc, I run it >>>>>> with following configuration optimisation in controller nodes. The DPNs >>>>>> are equally spread across controller nodes. >>>>> >>>>> If you increase the number of flows from one network to another, you are >>>>> running some sort of scalability test mixed with the performance test >>>>> that can confuse the result, like the perf degradation is because of the >>>>> number of switches or because the number of flows that has to be stored >>>>> and programmed… For this reason I would prefer to run the test with the >>>>> same condition of fix 100K flows divided among the switches that are >>>>> available in the network. It is also that testing this way we can easily >>>>> compare one network result to another. >>>>> >>>>>> >>>>>> I. fsync = off in akka.conf >>>>> >>>>> What is this setting? >>>>> >>>>>> II. changing root logging to ERROR from INFO >>>>>> III. Set JAVA_MIN_MEM=512M JAVA_MAX_MEM=8192M in karaf >>>>>> IV. <skip-table-features>true</skip-table-features> in >>>>>> 42-openflowplugin-He.xml >>>>>> >>>>>> I can rerun those scripts by tweaking it to the current requirement . Do >>>>>> let me know if any other configuration optimisation is required. Also >>>>>> could you please provide path to Boron distribution. I ran below scripts >>>>>> in stable Beryllium. >>>>> >>>>> Master distribution is in nexus: >>>>> https://nexus.opendaylight.org/content/repositories/opendaylight.sn >>>>> a >>>>> p s >>>>> hot/org/opendaylight/integration/distribution-karaf/0.5.0-SNAPSHOT/ >>>>> >>>>> Does your test automation provide all the perf numbers required in this >>>>> TC: controller programming time, switch programming time, and flow >>>>> confirmation delay? If not, that is the next thing to work on. >>>>> >>>>>> >>>>>> >>>>>> root@mininet-vm:/home/mininet/integration/test/csit/suites/openflo >>>>>> w p l u gin/Clustering_Bulkomatic# pybot -L TRACE -v >>>>>> MININET_USER:mininet -v USER_HOME:/home/mininet -v >>>>>> CONTROLLER:10.183.181.51 -v >>>>>> CONTROLLER1:10.183.181.52 -v CONTROLLER2:10.183.181.53 -v >>>>>> USER:root -v PASSWORD:rootroot -v WORKSPACE:/home/mininet -v >>>>>> BUNDLEFOLDER:/controller-Be/deploy/current/odl -v >>>>>> DEFAULT_LINUX_PROMPT:\# -v NUM_ODL_SYSTEM:3 -v >>>>>> MININET_PASSWORD:rootroot -v >>>>>> OVS_SWITCH_FILE:Multi_Switch_Medium_Config.py >>>>>> 150K__Cluster_Reconcilliation_Multi_DPN.robot >>>>>> ================================================================== >>>>>> = = = = ======== Cluster Reconcilliation Multi DPN :: Test suite >>>>>> for Cluster with Bulk Flows... >>>>>> ================================================================== >>>>>> = = = = ======== Check Shards Status And Initialize Variables :: >>>>>> Check Status for a... | PASS | >>>>>> ------------------------------------------------------------------ >>>>>> - >>>>>> - >>>>>> - >>>>>> - >>>>>> -------- Get Inventory Follower Before Cluster Restart :: Find a >>>>>> follower i... | PASS | >>>>>> ------------------------------------------------------------------ >>>>>> - >>>>>> - >>>>>> - >>>>>> - >>>>>> -------- Start Mininet Connect To Follower Node1 :: Start mininet >>>>>> with conn... | PASS | >>>>>> ------------------------------------------------------------------ >>>>>> - >>>>>> - >>>>>> - >>>>>> - >>>>>> -------- Add Bulk Flow From Follower :: 10000 Flows (10K per DPN) >>>>>> in >>>>>> 15 DPN... | PASS | >>>>>> ------------------------------------------------------------------------------ >>>>>> Verify Flows In Switch :: Verify 150K flows are installed. | >>>>>> PASS | >>>>>> ------------------------------------------------------------------ >>>>>> - >>>>>> - >>>>>> - >>>>>> - >>>>>> -------- Stop Mininet Connected To Follower Node1 and Exit :: Stop >>>>>> mininet ... | PASS | >>>>>> ------------------------------------------------------------------ >>>>>> - >>>>>> - >>>>>> - >>>>>> - >>>>>> -------- Delete All Flows From Follower Node1 :: 150000 Flows >>>>>> deleted via F... | PASS | >>>>>> ------------------------------------------------------------------ >>>>>> - >>>>>> - >>>>>> - >>>>>> - >>>>>> -------- Cluster Reconcilliation Multi DPN :: Test suite for >>>>>> Cluster with B... | PASS | >>>>>> 7 critical tests, 7 passed, 0 failed >>>>>> 7 tests total, 7 passed, 0 failed >>>>>> ================================================================== >>>>>> = >>>>>> = >>>>>> = >>>>>> = >>>>>> ======== >>>>>> >>>>>> >>>>>> root@mininet-vm:/home/mininet/integration/test/csit/suites/openflo >>>>>> w p l u gin/Clustering_Bulkomatic# pybot -L TRACE -v >>>>>> MININET_USER:mininet -v USER_HOME:/home/mininet -v >>>>>> CONTROLLER:10.183.181.51 -v >>>>>> CONTROLLER1:10.183.181.52 -v CONTROLLER2:10.183.181.53 -v >>>>>> USER:root -v PASSWORD:rootroot -v WORKSPACE:/home/mininet -v >>>>>> BUNDLEFOLDER:/controller-Be/deploy/current/odl -v >>>>>> DEFAULT_LINUX_PROMPT:\# -v NUM_ODL_SYSTEM:3 -v >>>>>> MININET_PASSWORD:rootroot -v >>>>>> OVS_SWITCH_FILE:Multi_11Switch_Medium_Config.py >>>>>> 330K__Cluster_Reconcilliation_Multi_DPN.robot >>>>>> ================================================================== >>>>>> = = = = ======== Cluster Reconcilliation Multi DPN :: Test suite >>>>>> for Cluster with Bulk Flows... >>>>>> ================================================================== >>>>>> = = = = ======== Check Shards Status And Initialize Variables :: >>>>>> Check Status for a... | PASS | >>>>>> ------------------------------------------------------------------ >>>>>> - >>>>>> - >>>>>> - >>>>>> - >>>>>> -------- Get Inventory Follower Before Cluster Restart :: Find a >>>>>> follower i... | PASS | >>>>>> ------------------------------------------------------------------ >>>>>> - >>>>>> - >>>>>> - >>>>>> - >>>>>> -------- Start Mininet Connect To Follower Node1 :: Start mininet >>>>>> with conn... | PASS | >>>>>> ------------------------------------------------------------------ >>>>>> - >>>>>> - >>>>>> - >>>>>> - >>>>>> -------- Add Bulk Flow From Follower :: 10000 Flows (10K per DPN) >>>>>> in >>>>>> 33 DPN... | PASS | >>>>>> ------------------------------------------------------------------------------ >>>>>> Verify Flows In Switch :: Verify 330K flows are installed. | >>>>>> PASS | >>>>>> ------------------------------------------------------------------ >>>>>> - >>>>>> - >>>>>> - >>>>>> - >>>>>> -------- Stop Mininet Connected To Follower Node1 and Exit :: Stop >>>>>> mininet ... | PASS | >>>>>> ------------------------------------------------------------------ >>>>>> - >>>>>> - >>>>>> - >>>>>> - >>>>>> -------- Delete All Flows From Follower Node1 :: 330000 Flows >>>>>> deleted via F... | PASS | >>>>>> ------------------------------------------------------------------ >>>>>> - >>>>>> - >>>>>> - >>>>>> - >>>>>> -------- Cluster Reconcilliation Multi DPN :: Test suite for >>>>>> Cluster with B... | PASS | >>>>>> 7 critical tests, 7 passed, 0 failed >>>>>> 7 tests total, 7 passed, 0 failed >>>>>> ================================================================== >>>>>> = >>>>>> = >>>>>> = >>>>>> = >>>>>> ======== >>>>>> Output: >>>>>> /home/mininet/integration/test/csit/suites/openflowplugin/Clustering_Bulkomatic/output.xml >>>>>> Log: >>>>>> /home/mininet/integration/test/csit/suites/openflowplugin/Clustering_Bulkomatic/log.html >>>>>> Report: >>>>>> /home/mininet/integration/test/csit/suites/openflowplugin/Clusteri >>>>>> n >>>>>> g >>>>>> _ >>>>>> B >>>>>> ulkomatic/report.html >>>>>> root@mininet-vm:/home/mininet/integration/test/csit/suites/openflo >>>>>> w >>>>>> p >>>>>> l >>>>>> u >>>>>> gin/Clustering_Bulkomatic# >>>>>> >>>>>> Thanks >>>>>> Sanjib >>>>>> >>>>>> From: Luis Gomez [mailto:[email protected]] >>>>>> Sent: 28 July 2016 05:17 >>>>>> To: Abhijit Kumbhare >>>>>> Cc: [email protected]; openflowplugin-dev; >>>>>> Jin Gan; Sanjib Mohapatra >>>>>> Subject: Re: [integration-dev] OpenFlow tests for next ODL perf >>>>>> paper >>>>>> >>>>>> Hi Abhijit, >>>>>> >>>>>> We can definitely leverage any automation your team will prepare to test >>>>>> all the combinations below, but just to be clear the ODL perf paper we >>>>>> will only include out-of-the-box Boron release. >>>>>> >>>>>> BR/Luis >>>>>> >>>>>> On Jul 27, 2016, at 4:23 PM, Abhijit Kumbhare <[email protected]> >>>>>> wrote: >>>>>> >>>>>> Hi Luis, >>>>>> >>>>>> I was discussing with our OpenFlow team (Manohar/Muthu/Shuva/Vinayak) >>>>>> today about the tests - and one of the things which would be very >>>>>> interesting would be the bulk-o-matic with small/medium/large configs >>>>>> (same as your test 2) and the following combinations: >>>>>> >>>>>> 1. Lithium design + the old FRM >>>>>> 2. Lithium design + the new FRM => are there improvements? >>>>>> 3. He design => bulk-o-matic may not have been used for the He/Li >>>>>> >>>>>> They were also planning to discuss this with Sanjib in their daytime >>>>>> today if this has been already done. >>>>>> >>>>>> Thanks, >>>>>> Abhijit >>>>>> >>>>>> On Wed, Jul 27, 2016 at 3:13 PM, Luis Gomez <[email protected]> wrote: >>>>>> Hi all, >>>>>> >>>>>> I got the action point from last S3P call to start some discussion >>>>>> around OpenFlow tests we can do for next ODL perf paper (Boron). >>>>>> >>>>>> I am not sure we will have time for all the below but ideally I was >>>>>> thinking in 4 tests: >>>>>> >>>>>> 1) REST programming performance: >>>>>> >>>>>> - Goal: Measure OF programming rate using NB REST interface >>>>>> - Methodology: Use test REST scripts (flows in datastore) to program >>>>>> 100K flows on an OF network of different size: 16-32-64-128 switches, >>>>>> etc... >>>>>> - Test variations: Use single flow/REST request and multiple flows/REST >>>>>> (bulk) request. >>>>>> - Collected data: controller programming time (from first to last flow) >>>>>> from REST script, switch programming time (from first to last flow) >>>>>> polling the OVS, flow confirmation delay (time after T1) polling the >>>>>> operational DS. >>>>>> >>>>>> 2) Java programming performance: >>>>>> >>>>>> - Goal: Measure OF programming rate using internal Java interface >>>>>> - Methodology: Use bluk-o-matic application (flows in datastore or rpc) >>>>>> to program 100K flows on an OF network of different size: 16-32-64-128 >>>>>> switches, etc... >>>>>> - Test variations: Use single flow/REST request and multiple flows/REST >>>>>> (bulk) request. >>>>>> - Collected data: controller programming time (from first to last flow) >>>>>> from bulk-o-matic, switch programming time (from first to last flow) >>>>>> polling the OVS, flow confirmation delay (time after T1) polling the >>>>>> operational DS. >>>>>> >>>>>> 3) Network message processing latency: >>>>>> >>>>>> - Goal: Measure OF packet message processing time >>>>>> - Methodology: Use some OF public tool (Cbench, MT-Cbench, SDN-blaster) >>>>>> to generate OF packets and measure the delay (latency mode) of the >>>>>> received controller flows on an OF network of different size: >>>>>> 16-32-64-128 switches, etc... >>>>>> - Test variations: Use controller drop-test application in DS and RPC >>>>>> mode. >>>>>> - Collected data: packet processing rate (latency=1/rate) >>>>>> >>>>>> 4) Topology scalability: >>>>>> >>>>>> - Goal: Scale OF network and measure learning time. >>>>>> - Methodology: Use some OF public tool (Mininet, Multinet) to generate >>>>>> different sizes of large topologies: 1000, 2000, 3000 switches, etc... >>>>>> - Collected data: Time for the controller to learn about the topology. >>>>>> >>>>>> In addition the same tests (or a subset) should run in a cluster >>>>>> environment (3 node cluster). >>>>>> >>>>>> The main problem we have today for running and automating the above is >>>>>> people resources, so far Jin and Sanjib offered to help but more help >>>>>> would be appreciated. >>>>>> >>>>>> BR/Luis >>>>>> >>>>>> _______________________________________________ >>>>>> integration-dev mailing list >>>>>> [email protected] >>>>>> https://lists.opendaylight.org/mailman/listinfo/integration-dev >>>>> >>>> >>> >> >> _______________________________________________ >> openflowplugin-dev mailing list >> [email protected] >> https://lists.opendaylight.org/mailman/listinfo/openflowplugin-dev >
_______________________________________________ openflowplugin-dev mailing list [email protected] https://lists.opendaylight.org/mailman/listinfo/openflowplugin-dev
