I got some issue: I'm using Cbench in throughput mode but after a few seconds of running Cbench I get the following error messages: 146439|event-dispatcher|ERR:Event ofp_packet_in processing leaked an exception: boost::asio::streambuf too long 146440|event-dispatcher|ERR:Extra information: Throw location unknown (consider using BOOST_THROW_EXCEPTION) Dynamic exception type:boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<std::length_error> > std::exception::what: boost::asio::streambuf too long
and i got poor results! 2013/1/20 David Erickson <deric...@stanford.edu> > Hi Ralf- > > > On 1/20/2013 7:50 AM, Ralf Leichter wrote: > >> Hi David, >> >> thanks for the hints, beacon and floodlight are running (and >> benchmarking) without an issue now. During testing i ran into some >> questions related, more later. >> I still have the issue with the NOX controller generating the >> "boost::asio::streambuf too long" messages - i ran some tests and the issue >> arises as soon as i run cbench with 9 or more simulated Switches. I >> verified this behaviour on 4 machines so i must be me having a conceptual >> issue somewhere. >> >> - I tried to replace libboost 1.42 with 1.46 without effect >> - I ran it single threaded to rule out ressource conflict without any >> effect. >> - I increased the TCP buffer settings on the machine without effect >> >> i am invoking nox with the following command: "screen -d -m taskset -c >> 0-$i nox_core -i ptcp:0.0.0.0:6633 switch" >> The version i am using is 0.9.2 beta compiled from git, checked out via >> "git clone http://noxrepo.org/git/nox.git**"; >> my cbench call is "taskset -c 7 ../../oflops/cbench/./cbench -c localhost >> -p 6633 -m 10000 -l 10 -s $j -M 1000000 -t >> results-nox-$k.txt" where $k >> is the number of threads the controller is currently running on ($i+1), $j >> increasing the switchcount. >> So, any hints are welcome what could cause such an issue - i can also >> rule out low memory issues as i tested it on a machine with 20gig, 32gig >> and 60gig of ram - experiencing the same problem. >> >> During my tests with beacon and floodlight i found that when running the >> controller with 10 threads or more, cbench apparently becomes the >> bottleneck as the cbench process utilizes one core 100%, the controller >> threads not being maxed out at all. Is there a way to run cbench >> multithreaded and to get meaningful results of it? >> > > I've run into the same issue with cbench not being able to generate > sufficient traffic, and my (quick) solution was to simultaneously launch > two instances of it, but it would be preferable to have a truly > multithreaded version. > > > For the high threadcount tests i've been using an Amazon EC2 instance >> with 32 threads if that makes any difference. >> > > I assume you are using the cluster instances, just be aware half of the > presented cores in the VM are hyperthreaded cores, which should be avoided > for benchmarking. > > -D > > > >> -- Ralf >> >> >> >> Am 15.01.2013 03:34, schrieb David Erickson: >> >>> Hi Ralf- >>> >>> On 1/14/2013 5:43 PM, Ralf Leichter wrote: >>> >>>> Hi David, >>>> sorry for the cc - must have pressed the reply instead of the reply to >>>> all button - i ran the unset command and at it seems to be more stable with >>>> this setting - but still beacon is crashing - i compiled the relevant parts >>>> of the errorlog here (logfile is 169MB): >>>> >>>> http://pastebin.com/f2T7ZbtV >>>> >>>> if i interpret this correctly there is a timer disconnecting the >>>> switches? >>>> >>> >>> It looks like you are including extra modules in your Beacon launch. >>> Have you had a look at the Benchmarking Guidelines page on Beacon's >>> wiki >>> (https://openflow.stanford.**edu/display/Beacon/**Benchmarking<https://openflow.stanford.edu/display/Beacon/Benchmarking> >>> )? >>> Also are you using the latest release of Beacon? I recommend >>> downloading a binary package if you are just testing >>> (https://openflow.stanford.**edu/display/Beacon/Releases<https://openflow.stanford.edu/display/Beacon/Releases>). >>> Follow the >>> directions on that wiki page and let me know if it clears up the >>> issues you are seeing. >>> >>> -David >>> >>> >>>> Floodlight seems to run now as of my first 10 rounds of cbench. >>>> >>>> nox still produces an exobitant amount of the "boost::asio::streambuf >>>> too long" errors. (~100MB/Thread and Round) >>>> >>>> regards, >>>> Ralf >>>> >>>> >>>> Am 15.01.2013 01:08, schrieb David Erickson: >>>> >>>>> Hi Ralf- >>>>> Please keep openflow-discuss in the CC incase this is useful to others. >>>>> >>>>> Can you run "unset LD_PRELOAD" in both your NOX and cbench terminal >>>>> and rerun it? I've found that most of the time NOX does not work >>>>> properly with tcmalloc. >>>>> >>>>> You mentioned you also tried Beacon and Floodlight, what happened >>>>> when you ran them? >>>>> >>>>> -David >>>>> >>>>> On 1/14/2013 3:11 PM, ralf_leich...@gaming-31337.de wrote: >>>>> >>>>>> Hi David, >>>>>> first of all - thanks for the quick reply! I am running controller(s) >>>>>> and cbench on the same machine - i have tought also that this might be >>>>>> due >>>>>> to a local ressource conflict so i already tried to separate the two - >>>>>> running the controller on the server mentioned, running cbench from my >>>>>> Ubuntu 13.04 Macbook over a direct attached gigabit link.... with the >>>>>> same >>>>>> result. >>>>>> >>>>>> I have checked port 6633 everytime before running the controller >>>>>> using "lsof | grep :6633" which returns nothing. I have checked out the >>>>>> latest version of oflops from github and compiled it against the github >>>>>> version of Openflow, which mentions version 1.0.0 in the makefile >>>>>> (commands >>>>>> used to checkout and install): >>>>>> >>>>>> git clone >>>>>> git://gitosis.stanford.edu/**openflow.git<http://gitosis.stanford.edu/openflow.git> >>>>>> cd openflow; git checkout -b buildbranch origin/release/1.0.0 >>>>>> git clone >>>>>> git://gitosis.stanford.edu/**oflops.git<http://gitosis.stanford.edu/oflops.git> >>>>>> cd oflops >>>>>> ./boot.sh >>>>>> ./configure --with-openflow-src-dir=/home/**openflow/openflow >>>>>> make >>>>>> sudo make install >>>>>> >>>>>> NOX Startup (only 1 line ;)) >>>>>> NOX 0.9.2~core~beta (nox_core), compiled Jan 13 2013 20:25:57 >>>>>> >>>>>> Nox errorlog exported via "taskset -c 0-1 ./nox_core -i ptcp:6633 >>>>>> switch -t 2 2> noxerrors" (shortened on duplicate parts, logfile has >>>>>> 489MB) >>>>>> >>>>>> http://pastebin.com/DnK4B5Vj >>>>>> >>>>>> For starting NOX (with 2 threads) I have used the following >>>>>> commandline (following the controler performance page): >>>>>> >>>>>> LD_PRELOAD=/usr/lib/**libtcmalloc_minimal.so.0 >>>>>> taskset -c 0-1 ./nox_core -i ptcp:6633 switch -t 2 >>>>>> >>>>>> cbench ist started with >>>>>> >>>>>> LD_PRELOAD=/usr/lib/**libtcmalloc_minimal.so.0 >>>>>> taskset -c 7 ./cbench -c localhost -p 6633 -m 10000 -l 10 -s 32 -M >>>>>> 1000000 -t >>>>>> >>>>>> regards, >>>>>> Ralf Leichter >>>>>> >>>>>> >>>>>> Am 14.01.2013 22:15, schrieb David Erickson: >>>>>> >>>>>>> Hi Ralf- >>>>>>> Are you running both the controller(s) and cbench locally on the same >>>>>>> machine? Have you checked to ensure nothing else is running on the >>>>>>> controller port before launching the controller under test? Did you >>>>>>> check out the latest oflops from git, and compile it against the >>>>>>> correct OpenFlow version (1.0)? What command line are you using for >>>>>>> both the controller(s) and cbench? And can you attach logs from >>>>>>> both? >>>>>>> >>>>>>> -David >>>>>>> >>>>>>> On 1/14/2013 12:27 PM, ralf_leich...@gaming-31337.de wrote: >>>>>>> >>>>>>>> Hello Openflow Mailinglist :), >>>>>>>> i am relatively new to openflow so if this question has already >>>>>>>> been asked some times i am sorry, but i couldnt find any further info >>>>>>>> on >>>>>>>> this issue on the net than this link (which has no solution and is nox >>>>>>>> specific -> >>>>>>>> https://github.com/noxrepo/**nox/issues/5<https://github.com/noxrepo/nox/issues/5> >>>>>>>> ) >>>>>>>> >>>>>>>> I have setup a small openflow testbed using a server installed with >>>>>>>> Ubuntu 11.10, I set up the beacon, floodlight and nox controller >>>>>>>> (compiled >>>>>>>> from source) to get an idea on starting with openflow. My first goal >>>>>>>> was to >>>>>>>> get to a point where i can do my own tests for applications - as a >>>>>>>> basic >>>>>>>> idea i started with the benchmarks pointed out at >>>>>>>> http://www.openflow.org/wk/**index.php/Controller_** >>>>>>>> Performance_Comparisons<http://www.openflow.org/wk/index.php/Controller_Performance_Comparisons>. >>>>>>>> I followed the instructions and settings outlined on the page and >>>>>>>> all controllers are initially running, but upon running cbench i am >>>>>>>> getting >>>>>>>> output similar to the following after a few seconds: >>>>>>>> >>>>>>>> controller msgbuf_read() = -1: msgbuf_read: Connection reset by >>>>>>>> peer >>>>>>>> >>>>>>>> the controller is still receiving messages at this point and ends >>>>>>>> with disconnecting the switches still running the controller, cbench >>>>>>>> terminates. >>>>>>>> >>>>>>>> At first i thought this related to memory issues on the controller >>>>>>>> side so i increased this memory, same effect. >>>>>>>> same relates to setting up the nox controller - cbench terminates >>>>>>>> with the similar message after a few seconds, on the controllerside a >>>>>>>> "boost::asio::streambuf too long" exception is thrown. (i am using the >>>>>>>> default switch implementation as outlined on the performance comparison >>>>>>>> page) >>>>>>>> floodlight also has the same behaviour as beacon >>>>>>>> >>>>>>>> I also tried to set up a debian sid installation on the same >>>>>>>> hardware and ran into the same issues. Also a virtual setup using >>>>>>>> ubuntu on >>>>>>>> a virtualbox install has the same issue >>>>>>>> >>>>>>>> Testbench Server: >>>>>>>> HP DL380G5 >>>>>>>> 2x Xeon 2.84Ghz Quadcore >>>>>>>> 20GB DDR2 >>>>>>>> running Ubuntu 11.10 x64 >>>>>>>> >>>>>>>> what am i doing wrong? any hint is appreciated. >>>>>>>> >>>>>>>> regards, >>>>>>>> Ralf Leichter >>>>>>>> ______________________________**_________________ >>>>>>>> openflow-discuss mailing list >>>>>>>> openflow-discuss@lists.**stanford.edu<openflow-discuss@lists.stanford.edu> >>>>>>>> https://mailman.stanford.edu/**mailman/listinfo/openflow-**discuss<https://mailman.stanford.edu/mailman/listinfo/openflow-discuss> >>>>>>>> >>>>>>>> >>>>>> >>>>>> >>>> >>>> >> >> > ______________________________**_________________ > openflow-discuss mailing list > openflow-discuss@lists.**stanford.edu<openflow-discuss@lists.stanford.edu> > https://mailman.stanford.edu/**mailman/listinfo/openflow-**discuss<https://mailman.stanford.edu/mailman/listinfo/openflow-discuss> >
_______________________________________________ openflow-discuss mailing list openflow-discuss@lists.stanford.edu https://mailman.stanford.edu/mailman/listinfo/openflow-discuss
