Hi Charlie, did you have the chance to simulate your rebased n23-5 branch using Shadow?
If not, maybe I can help by simulating it once and writing down more specific instructions for you? Thanks! Karsten On 8/8/13 10:36 AM, Karsten Loesing wrote: > (Moving this discussion back to tor-dev@ with Charlie's permission.) > > On 8/6/13 4:10 AM, Charlie Belmer wrote: >> Hey Karsten, >> >> I've been reading up on Shadow & Scallion to do some performance testing on >> N23. Before I go too far down that road, I wanted to check in with you. >> >> Are there existing blueprints or best practices I should be following? > > Thanks for looking into this! > > So, I wouldn't call the following notes blueprints or best practices, > but they are a description of what I would do to run Shadow simulations > of your branch. > > First of all, be sure to read the Shadow wiki which has contains lots of > interesting stuff: https://github.com/shadow/shadow/wiki > > The next step would be to get a vanilla Tor master (that your branch is > based on) running in a minimal or tiny Shadow network. It's probably > easiest to use a local Ubuntu VM for this. Or you can start an m1.large > EC2 instance for the tiny network, or an even smaller instance for the > minimal network. > > Once that simulation succeeds, you'll want to run another simulation > using your branch, extract performance metrics using Shadow's analyze > script, produce a PDF with graphs also using the analyze script, and > then wonder if things are faster than before. > > The minimal or tiny networks are mostly there to learn that things are > working as expected. You'll probably want larger networks for real > performance results, but starting small to get everything working is > probably easier. > > Sounds doable, I guess. However, here's where things start to get > tricky: in order to simulate Tor master (or a branch based on it) in > Shadow, you'll have to perform quite a few extra steps like compiling > your own CMake and Clang/LLVM. Not really hard, but you need to get all > pieces together before Shadow does what you want. > > I'm pasting detailed instructions that I used to simulate Steven's uTP > branch in Shadow below as an example. Not everything applies to your > branch, but it should give you the general idea. > > If something remains unclear, please feel free to ask! > >> Depending how it goes, I'd also look at re-trying adaptive 23 or tuning the >> existing implementation. > > Neat! We should probably include Mashael in that discussion once you > have a working and simulated n23-5 branch. Maybe she has ideas what > parts need tweaking. I'm cc'ing her in this mail so she knows what > you're working on. > > In any case, having a working rebased n23-5 branch is already a big step > forward. > > Thanks! > Karsten > > > === START OF INSTRUCTIONS === > > Launch new EC2 instance running Ubuntu Server 13.04 64 bit > > Set ulimit -n to 25000, see Shadow wiki for details > https://github.com/shadow/shadow/wiki/Preparing-your-machine#system-configs-and-limits > > $ sudo apt-get update > $ sudo apt-get upgrade > > Clone and test-compile modified libutp > > $ sudo apt-get install build-essential git automake libssl-dev libevent-dev > $ mkdir src > $ cd src/ > $ git clone https://github.com/sjmurdoch/libutp > $ cd libutp/ > $ make > $ cd ../ > > Clone and test-compile Tor branch utp > > $ git clone https://git.torproject.org/tor.git > $ cd tor/ > $ git remote add sjm217 https://git.torproject.org/sjm217/tor.git > $ git fetch sjm217 > $ git checkout -b utp sjm217/utp > $ ./autogen.sh > $ LDFLAGS="-L/home/ubuntu/src/libutp" CFLAGS="-I/home/ubuntu/src/libutp" > LIBS="-lutp -lrt" ./configure --disable-asciidoc > $ make > $ make distclean > $ cd ../ > > Modify Tor to use uTP for all links (or "0 &&" for not using uTP for any > link) > > diff --git a/src/or/channeltls.c b/src/or/channeltls.c > index 0551b73..b7b36e1 100644 > --- a/src/or/channeltls.c > +++ b/src/or/channeltls.c > @@ -418,7 +418,7 @@ channel_tls_connect(const tor_addr_t *addr, uint16_t > port, > /* Create a uTP connection */ > tor_addr_to_sockaddr(addr, port, (struct sockaddr*)&sin, sizeof(sin)); > tor_addr_to_str(addr_str, addr, sizeof(addr_str), 0); > - if (!strncmp(addr_str, "128.232.10.129", sizeof(addr_str))) { > + if (1 || !strncmp(addr_str, "128.232.10.129", sizeof(addr_str))) { > log_info(LD_CHANNEL, > "Trying uTP connection to %s", addr_str); > tlschan->utp = UTP_Create(tor_UTPSendToProc, tlschan, (const struct > sockaddr*)&sin, > > Install CMake 2.8.10 > > $ wget http://www.cmake.org/files/v2.8/cmake-2.8.10.2.tar.gz > $ tar xf cmake-2.8.10.2.tar.gz > $ cd cmake-2.8.10.2/ > $ ./configure > $ make > $ export PATH=~/src/cmake-2.8.10.2/bin:$PATH > $ cd ../ > $ which cmake # result: /home/ubuntu/src/cmake-2.8.10.2/bin/cmake > > Build Clang and LLVM from source > > $ wget http://llvm.org/releases/3.2/llvm-3.2.src.tar.gz > $ wget http://llvm.org/releases/3.2/clang-3.2.src.tar.gz > $ tar xaf llvm-3.2.src.tar.gz > $ tar xaf clang-3.2.src.tar.gz > $ cp -R clang-3.2.src llvm-3.2.src/tools/clang > $ cd llvm-3.2.src/ > $ mkdir build > $ cd build/ > $ cmake -DCMAKE_INSTALL_PREFIX=/home/ubuntu/.local ../. > $ make > $ make install > $ export PATH=~/.local/bin/:$PATH > $ cd ../../ > $ which clang # result: /home/ubuntu/.local/bin//clang > > Install Shadow > > $ sudo apt-get install gcc xz-utils make automake autoconf cmake tidy > libtidy-dev libglib2.0 libglib2.0-dev dstat pdftk python2.7 > python-matplotlib python-numpy python-scipy > $ git clone https://github.com/shadow/shadow.git > $ cd shadow > $ ./setup dependencies > $ ./setup build -g -i /home/ubuntu/src/libutp -i > /usr/include/x86_64-linux-gnu/c++/4.7 -l /home/ubuntu/src/libutp > --tor-prefix /home/ubuntu/src/tor --tor-lib utp > $ ./setup install > $ export PATH=~/.shadow/bin/:$PATH > > Simulate tiny network in Shadow > > $ cd resource/examples/scallion/ > $ tar xf tiny-4GB-m1.large.tar.xz > $ cd tiny-4GB-m1.large/ > $ scallion > > Graph simulation results > > Add one line to beginning of analyze.py: > > import matplotlib; matplotlib.use('Agg') > > For additional traffic simulations: > > $ grep -v "\[traffic-" scallion.log > scallion-notraffic.log > > $ cd ~/src/ > $ mkdir results > $ cd results/ > $ python ~/src/shadow/contrib/analyze.py parse --output uTP-1 > ~/src/shadow/resource/examples/scallion/tiny-4GB-m1.large/data-utp-1/scallion.log > $ python ~/src/shadow/contrib/analyze.py parse --output uTP-0 > ~/src/shadow/resource/examples/scallion/tiny-4GB-m1.large/data-utp-0/scallion.log > $ python ~/src/shadow/contrib/analyze.py parse --output vanilla > ~/src/shadow/resource/examples/scallion/tiny-4GB-m1.large/data-vanilla/scallion.log > $ python ~/src/shadow/contrib/analyze.py plot --title "sjm217's utp > branch (#9166)" --prefix "sjm217-utp" --data vanilla/ "vanilla > 0.2.4.4-alpha" --data uTP-1/ "uTP for all links" --data uTP-0/ "uTP for > none of the links" > > === END OF INSTRUCTIONS === > > _______________________________________________ tor-dev mailing list [email protected] https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
