Hi All, I agree with David on LOC matter. I too feel that LOC is not a very good metric for comparing controllers. I also agree with Rob that a lot of other metrics can be brought on to the table to compare the controllers. One thing that we are considering doing is that we may be testing these controllers with actual openflow switches or multiple OVSs so that a more realistic scenario may be observed. And I strongly believe that standard I/O is one of the most important metrics for a controller as it really enables you to decide what controller one should use in a real time environment so that one can be sure of the stability of the openflow network. Personally I would also like to see some improvement in the Cbench as at the moment it does quite synthetic comparison of the controllers. So I would break down the controllers in two main parts: 1- From developers point of view the controller that provides the best ease of use is to be preferred. 2- From a real time network stability point of view the controller with the best and most stable I/O is to be preferred. As we are currently trying to evaluate different controllers I would really like some feedback from you guys like what other metrics can we introduce in our experiments to make the comparison as comprehensive as possible.
Thanks. :) On Tue, Feb 14, 2012 at 1:00 AM, < openflow-discuss-requ...@lists.stanford.edu> wrote: > Send openflow-discuss mailing list submissions to > openflow-discuss@lists.stanford.edu > > To subscribe or unsubscribe via the World Wide Web, visit > https://mailman.stanford.edu/mailman/listinfo/openflow-discuss > or, via email, send a message with subject or body 'help' to > openflow-discuss-requ...@lists.stanford.edu > > You can reach the person managing the list at > openflow-discuss-ow...@lists.stanford.edu > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of openflow-discuss digest..." > > > Today's Topics: > > 1. Re: openflow controllers (Abdulah Shah) (Syed Abdullah Shah) > 2. Re: [trema-dev] Testing trema apps with an external CBENCH > TOOL (HIDEyuki Shimonishi) > 3. Re: [trema-dev] Testing trema apps with an external CBENCH > TOOL (David Erickson) > 4. Re: [trema-dev] Testing trema apps with an external CBENCH > TOOL (HIDEyuki Shimonishi) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Mon, 13 Feb 2012 02:58:30 -0500 > From: Syed Abdullah Shah <08beeabs...@seecs.edu.pk> > To: Rob Sherwood <rob.sherw...@bigswitch.com> > Cc: openflow-discuss@lists.stanford.edu, Ali Khayam > <ali.kha...@seecs.nust.edu.pk> > Subject: Re: [openflow-discuss] openflow controllers (Abdulah Shah) > Message-ID: > <CADV1p+n=S4+YEeD=MO_ym3XMGhD2C=-xxnm1el6a4462u_a...@mail.gmail.com > > > Content-Type: text/plain; charset="iso-8859-1" > > We understand your appreciation Rob and Thank you for that :) > > On Mon, Feb 13, 2012 at 2:56 AM, Rob Sherwood <rob.sherw...@bigswitch.com > >wrote: > > > On Sun, Feb 12, 2012 at 11:50 PM, Rob Sherwood > > <rob.sherw...@bigswitch.com> wrote: > > > Not that all of this performance work isn't great > > > > Syed -- I really apologize ... I meant to type "all of this > > performance work *is* great" .. not "*isn't* great" -- I hope that it > > was obvious and that I really did appreciate your work. Apologies for > > the late night (for me at least) typos :-) > > > > - Rob > > . > > > > > > -- > Syed Abdullah Shah, > Undergraduate Student,NUST SEECS, > BEE 5 > 08beeabs...@seecs.edu.pk > abdullah.s...@hotmail.com > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: < > http://mailman.stanford.edu/pipermail/openflow-discuss/attachments/20120213/c780ac4a/attachment-0001.html > > > > ------------------------------ > > Message: 2 > Date: Mon, 13 Feb 2012 21:21:16 +0900 > From: "HIDEyuki Shimonishi" <h-shimoni...@cd.jp.nec.com> > To: "'Rob Sherwood'" <rob.sherw...@bigswitch.com> > Cc: openflow-discuss@lists.stanford.edu, trema-...@googlegroups.com > Subject: Re: [openflow-discuss] [trema-dev] Testing trema apps with an > external CBENCH TOOL > Message-ID: <046901ccea49$fbe43150$f3ac93f0$@jp.nec.com> > Content-Type: text/plain; charset="iso-8859-1" > > Hi Rob, > > Sorry for the crosspost. Same discussions are going on different mailing > lists, so I guess it'd be beneficial. > > > From: openflow-discuss-boun...@lists.stanford.edu > ... > > Subject: Re: [openflow-discuss] openflow controllers (Abdulah Shah) > ... > > company for all the effort), but IMHO most developers will probably > > want to choose a controller based on ease of use, documentation, and > > preferred language -- all of which are very subjective. > > I agree that these subjective metrics are important, maybe rather than > simple "I/O bench marking". > At least in the use of academic research in SDN, productivity using the > controllers should be one of the most important criteria. > > One simple method to QUANTITATIVELY compare the productivity of controllers > would be comparing line counts needed for typical operations, functions, or > applications. > For example, many controllers provide simple learning switch functions as > their sample code, so we can count the lines of codes for this function > among different controllers. > > The four metrics you propose below would be nice too. They also measure > performance of controller applications in addition to controller itself. > So, > maybe we should write purpose-built controller applications for > benchmarking, rather than using the applications they provide for other > purpose. Then, we can also compare the line counts. > > One more thing is that, performance benchmarking would be so meaningful if > the controllers have SAME functionality. > For example, Trema assumes that different applications run concurrently on > top of Trema, so it maintains and manages flow cookies so that cookies from > different applications do not collide, which slow-downs Trema. > This is just one example, but different controllers may provide different > functionality even in such bottom layer, so comparing simple "I/O > benchmarking" might not so easy. > > HIDE > > > > > > -----Original Message----- > > From: Rob Sherwood [mailto:rob.sherw...@bigswitch.com] > > Sent: Monday, February 13, 2012 4:54 PM > > To: HIDEyuki Shimonishi > > Cc: trema-...@googlegroups.com > > Subject: Re: [trema-dev] Testing trema apps with an external CBENCH TOOL > > > > Hello all, > > > > Thank you for the patch to cbench -- I can apply it ASAP to the code, > > but I would like to know who wrote the patch so I can give proper > > credit. > > > > Shimonishi-san: you ask a very good question. cbench does a very > > limited benchmark only for I/O performance, which is ultimately not > > very interesting. I have a number of ideas for how one might want to > > benchmark controllers, but they all involve a tool more complicated > > than cbench. > > > > You can imagine: > > > > 1) time to react to a new host > > 2) time to react to a host move > > 3) time to recover from a link down > > > > 4) traffic distribution : stretch vs. stress > > > > and lots of other metrics. > > > > Do others have ideas in this space? > > > > Thanks again, > > > > - Rob > > . > > > > On Sat, Feb 11, 2012 at 7:41 AM, HIDEyuki Shimonishi > > <h-shimoni...@cd.jp.nec.com> wrote: > > > Hi, > > > > > > > > > > > > Maybe NEC gays staying at Stanford will do ? > > > > > > > > > > > > BTW, cbench is something like measuring an ?I/O performance? of a > > > controller. What kind of performance metrics we?ll need to measure > > > ?performance? of a controller ? > > > > > > > > > > > > > > > > > > > > > > > > From: trema-...@googlegroups.com [mailto:trema-...@googlegroups.com] > > On > > > Behalf Of kk yap > > > Sent: Saturday, February 11, 2012 3:39 AM > > > To: trema-...@googlegroups.com > > > Cc: Rob Sherwood > > > > > > > > > Subject: Re: [trema-dev] Testing trema apps with an external CBENCH > TOOL > > > > > > > > > > > > Hi, > > > > > > > > > > > > I just checked with Rob and everything should work with OpenFlow 1.0. > ?We > > > have people who have used it against 1.0 compliant controllers. > > > > > > > > > > > > As such, can Lafaiet or someone grab me a tcpdump of the interaction > between > > > cbench and Trema? ?Putting together a setup on my side would take a > long > > > time. > > > > > > > > > > > > Thanks! > > > > > > > > > > > > Regards > > > > > > KK > > > > > > > > > > > > On 7 February 2012 16:55, CHIBA Yasunobu <y...@dsl.gr.jp> wrote: > > > > > > Hi, > > > > > > Here is a patch to comply with OpenFlow 1.0.0. Hope this helps. > > > > > > Regards, > > > CHIBA Yasunobu > > > > > > 2012/2/8 kk yap <yap...@stanford.edu>: > > > > > >> Hi, > > >> > > >> Hmm... nope, I wasn't listening in until now. > > >> > > >> Can someone grab a tcpdump of the controller channel? ?In an attempt > > to > > >> minimize work, does Chiba have a patch we can apply or do I need to go > > and > > >> whip the thing into shape? > > >> > > >> Regards > > >> KK > > >> > > >> On 7 February 2012 04:57, HIDEyuki Shimonishi > > <h-shimoni...@cd.jp.nec.com> > > >> wrote: > > >>> > > >>> Hi, > > >>> > > >>> Is KK hearing ? Do you think our comment is correct ? If so, I guess > > >>> you'd > > >>> better to fix cbench. > > >>> > > >>> Actually, I told with Yasunobu and he was saying the cbench tool is > > >>> designed > > >>> for OpenFlow v0.8.9 and not speaking OpenFlow v1.0. The cbench > > >>> distribution > > >>> that comes with Trema is modified for v1.0 actually. > > >>> So, if you connect original cbench, Trema parser correctly finds > > protocol > > >>> error and disconnects the session. > > >>> > > >>> Cheers, > > >>> > > >>> HIDE > > >>> > > >>> > > >>> > -----Original Message----- > > >>> > From: trema-...@googlegroups.com > > [mailto:trema-...@googlegroups.com] On > > >>> > Behalf Of CHIBA Yasunobu > > >>> > Sent: Sunday, February 05, 2012 10:57 AM > > >>> > To: trema-...@googlegroups.com > > >>> > Subject: Re: [trema-dev] Testing trema apps with an external CBENCH > > >>> > TOOL > > >>> > > > >>> > Hi, > > >>> > > > >>> > > I actually tried to run the tool pointing to my running > controller > > >>> > > (with ruby learning switch app) but cbench is giving me a > "connection > > >>> > > closed error". > > >>> > > > >>> > AFAIK, cbench distributed under > > git://gitosis.stanford.edu/oflops.git > > >>> > is not compliant with OpenFlow 1.0.0. Thus, it does not work with > > >>> > controllers developed on top of Trema. > > >>> > > > >>> > Regards, > > >>> > CHIBA Yasunobu > > >>> > > >> > > > > > > > > > > ------------------------------ > > Message: 3 > Date: Mon, 13 Feb 2012 10:26:43 -0800 > From: David Erickson <deric...@stanford.edu> > To: HIDEyuki Shimonishi <h-shimoni...@cd.jp.nec.com> > Cc: openflow-disc...@mailman.stanford.edu > Subject: Re: [openflow-discuss] [trema-dev] Testing trema apps with an > external CBENCH TOOL > Message-ID: <4f3955e3.6010...@stanford.edu> > Content-Type: text/plain; charset=ISO-8859-1; format=flowed > > On 2/13/2012 4:21 AM, HIDEyuki Shimonishi wrote: > >> company for all the effort), but IMHO most developers will probably > >> want to choose a controller based on ease of use, documentation, and > >> preferred language -- all of which are very subjective. > > ... > > > > One simple method to QUANTITATIVELY compare the productivity of > controllers > > would be comparing line counts needed for typical operations, functions, > or > > applications. > > For example, many controllers provide simple learning switch functions as > > their sample code, so we can count the lines of codes for this function > > among different controllers. > > > > > > Hi Hide- > Personally I don't find LOC very enlightening other than as a possible > warning sign if the size is vastly larger than other software with > similar functionality. One can relatively easily decrease LOC for some > app by abstracting away functionality that app needs into a few function > calls behind the controller's API, but it doesn't say much about what > happens when someone writes a different app that needs to do things that > aren't available behind those few minimized function calls. If LOC were > a driving motivator we would probably all be writing 1 line applications > written in Perl, which I don't think anyone wants. > > -D > > > ------------------------------ > > Message: 4 > Date: Tue, 14 Feb 2012 04:03:30 +0900 > From: "HIDEyuki Shimonishi" <h-shimoni...@cd.jp.nec.com> > To: "'David Erickson'" <deric...@stanford.edu> > Cc: openflow-disc...@mailman.stanford.edu > Subject: Re: [openflow-discuss] [trema-dev] Testing trema apps with an > external CBENCH TOOL > Message-ID: <04dc01ccea82$2cba0ba0$862e22e0$@jp.nec.com> > Content-Type: text/plain; charset="us-ascii" > > Hi David, > > LOC is just one metric and is not a perfect one, of cause. And it differs > by > the language a controller supports. It would as useful as, and as useless > as, I/O benchmarking. > As you pointed out, one can reduce LOC by defining application specific > APIs, but it could result in tons of APIs in the long run and that would > spoil maintenanceability of the controller. > If controller A and B has the same level of API sets and if you need twice > LOCs for controller A in writing typical applications like learning switch, > then LOC can say something in this case. > Well, it's really difficult to measure a controller. It might be good to > read sample codes of these controllers, but it is difficult to share with > others... > > HIDE > > > > > -----Original Message----- > > From: David Erickson [mailto:deric...@stanford.edu] > > Sent: Tuesday, February 14, 2012 3:27 AM > > To: HIDEyuki Shimonishi > > Cc: openflow-disc...@mailman.stanford.edu > > Subject: Re: [openflow-discuss] [trema-dev] Testing trema apps with an > > external CBENCH TOOL > > > > On 2/13/2012 4:21 AM, HIDEyuki Shimonishi wrote: > > >> company for all the effort), but IMHO most developers will probably > > >> want to choose a controller based on ease of use, documentation, and > > >> preferred language -- all of which are very subjective. > > > ... > > > > > > One simple method to QUANTITATIVELY compare the productivity of > > controllers > > > would be comparing line counts needed for typical operations, > functions, > > or > > > applications. > > > For example, many controllers provide simple learning switch functions > > as > > > their sample code, so we can count the lines of codes for this function > > > among different controllers. > > > > > > > > > > Hi Hide- > > Personally I don't find LOC very enlightening other than as a possible > > warning sign if the size is vastly larger than other software with > > similar functionality. One can relatively easily decrease LOC for some > > app by abstracting away functionality that app needs into a few function > > calls behind the controller's API, but it doesn't say much about what > > happens when someone writes a different app that needs to do things that > > aren't available behind those few minimized function calls. If LOC were > > a driving motivator we would probably all be writing 1 line applications > > written in Perl, which I don't think anyone wants. > > > > -D > > > > ------------------------------ > > _______________________________________________ > openflow-discuss mailing list > openflow-discuss@lists.stanford.edu > https://mailman.stanford.edu/mailman/listinfo/openflow-discuss > > > End of openflow-discuss Digest, Vol 40, Issue 12 > ************************************************ > -- Syed Abdullah Shah, Undergraduate Student,NUST SEECS, BEE 5 08beeabs...@seecs.edu.pk abdullah.s...@hotmail.com
_______________________________________________ openflow-discuss mailing list openflow-discuss@lists.stanford.edu https://mailman.stanford.edu/mailman/listinfo/openflow-discuss
