------------------------------------------------------------ revno: 4004 committer: bchareyre <bruno.chare...@grenoble-inp.fr> timestamp: Thu 2017-02-23 12:21:46 +0100 message: Robert Caulk's guide to cloud computing inserted in doc added: doc/sphinx/amazonEC2.rst doc/sphinx/fig/connect.png doc/sphinx/fig/gnuplot.png doc/sphinx/fig/instancetype.png doc/sphinx/fig/launchinstance.png doc/sphinx/fig/publicdns.png doc/sphinx/fig/ubunut.png modified: doc/sphinx/index-toctree.rst doc/sphinx/installation.rst
-- lp:yade https://code.launchpad.net/~yade-pkg/yade/git-trunk Your team Yade developers is subscribed to branch lp:yade. To unsubscribe from this branch go to https://code.launchpad.net/~yade-pkg/yade/git-trunk/+edit-subscription
=== added file 'doc/sphinx/amazonEC2.rst' --- doc/sphinx/amazonEC2.rst 1970-01-01 00:00:00 +0000 +++ doc/sphinx/amazonEC2.rst 2017-02-23 11:21:46 +0000 @@ -0,0 +1,214 @@ +.. _CloudComputing: + +============================================= +Using YADE with cloud computing on Amazon EC2 +============================================= + +.. role:: math(raw) + :format: html latex + +(Note: we thank Robert Caulk for preparing and sharing this guide) + +Summary +======= + +This guide is intended to help YADE users migrate their simulations to +Amazon Web Service (AWS) EC2. Two of the most notable benefits of using +scalable cloud computing for YADE include decreased upfront cost and +increased productivity. The entire process, from launching an instance, +to installing YADE, to running a YADE simulation on the cloud can be +executed in under 5 minutes. Once the EC2 instance is running, you can +submit YADE scripts the same way you would submit jobs on a local +workstation. + +Launching an EC2 instance +========================= + +.. _fig-console: +.. figure:: fig/launchinstance.* + :scale: 60 % + :align: center + + Amazon Web Services (AWS) Console + + +Start by signing into the console on `Amazon +EC2 <https://aws.amazon.com/?nc2=h_lgl>`__. This will require an +existing or new Amazon account. Once you’ve signed in, you should find +the EC2 console by clicking on ‘services’ in the upper left hand corner +of the AWS homepage. Start by clicking on the \`\`launch an instance“ +blue button (Fig. :ref:`fig-console`). Select the Amazon +Machine Image (AMI): \`\`Ubuntu Server 16.04 LTS\`\` (Fig. +:ref:`fig-ubuntu`). + +.. _fig-ubuntu: +.. figure:: fig/ubunut.* + :scale: 60 % + :align: center + + Select Ubuntu server 16.04 LTS AMI + +You will now select the instance type. It is worth looking at the +`specifications for each of the +instances <https://aws.amazon.com/ec2/instance-types/>`__ so you can +properly select the power you need for you YADE simulation. This +document will not go into detail in the selection of size, but you can +find plenty of `YADE specific performance +reports <https://yade-dem.org/publi/1stWorkshop/booklet.pdf>`__ that +will help you decide. However, the instance type is an important +selection. The \`\`Compute Optimized" instances are necessary for most +YADE simulations because they provide access to high performing +processors and guaranteed computing power. The C3.2xlarge (Fig. +:ref:`fig-type`) is equivalent to an 8 core 2.8ghz Xeon E5 +with 25 mb of cache, which is likely the best option for medium-large +scale YADE simulations. + +.. _fig-type: +.. figure:: fig/instancetype.* + :scale: 60 % + :align: center + + Compute optimized (C3) instance tier + + +Before launching, you will be asked to \`\`select an existing key pair +or create a new key pair". Create a new one, download it, and place it +in a folder that you know the path to. Modify the permissions on the +file by navigating to the same directory in the terminal and typing: + +:: + + chmod 400 KeyPair.pem + +Now the instance is launched, you will need to connect to it via SSH. On +unix systems this is as easy as typing: + +:: + + ssh -i path/to/KeyPair.pem ubu...@ec2-xx-xxx-xx-xx.us-west-2.compute.amazon.com + +into the terminal. There are other options such as using PuTTY, or even +a java based terminal on the AWS website. You can find the necessary +information by navigating to \`\`Instances" in the left menu of the AWS +console. Right click on the instance as shown in Fig. +:ref:`fig-connect` and click connect. + + +.. _fig-connect: +.. figure:: fig/connect.* + :scale: 60 % + :align: center + + Connecting to the instance + +You will be presented with the public DNS, which should look something +like Fig. :ref:`fig-dns`. + + +.. _fig-dns: +.. figure:: fig/publicdns.* + :scale: 60 % + :align: center + + Public DNS + + +Installing YADE and managing files +================================== + +After you’ve connected to the instance through SSH, you will need to +install YADE. The following commands should be issued to install +yadedaily, python, and some other useful tools: + +:: + + #install yadedaily + sudo bash -c 'echo "deb http://www.yade-dem.org/packages/ xenial/" >> /etc/apt/sources.list' + wget -O - http://www.yade-dem.org/packages/yadedev_pub.gpg | sudo apt-key add - + sudo apt-get update + sudo apt-get install -y yadedaily + + # install python + sudo apt-get -y install python + sudo apt-get -y install python-pip python-dev build-essential + + # install htop + sudo apt-get -y install htop + +| Note that \`\`..packages/ xenial/" should match the Ubuntu +distribution. 16.04 LTS is Xenial, but if you chose to start Ubuntu +14.04, you will need to change ‘xenial’ to ‘trusty’. + +Finally, you will need to upload the necessary YADE files. If you have a +folder with the contents of your simulation titled \`\`yadeSimulation" +you can upload the folder and its contents by issuing the following +command: + +:: + + scp -r -i path/to/KeyYADEbox.pem path/to/yadeSimulation ubu...@ec2-xx-xxx-xx-xx.us-west-2.compute.amazonaws.com:~/yadeSimulation + +You should now be able to run your simulation by changing to the proper +directory and typing: + +:: + + yadedaily nameOfSimulation.py + +In order to retrieve the output files (folder titled ‘out’ below) for +post processing purposes, you will use the same command that you used to +upload the folder, but the remote and local file destinations should be +reversed: + +:: + + scp -r -i path/to/KeyYADEbox.pem ubu...@ec2-xx-xxx-xx-xx.us-west-2.compute.amazonaws.com:~/yadeSimulation/out/ path/to/yadeSimulation/out + +Plotting output in the terminal +=============================== + +One of the main issues encountered with cloud computing is the lack of +graphical feedback. There is an easy solution for graphically checking +the status of your simulations which makes use of gnuplot’s wonderful +‘terminal dumb’ feature. Any data can be easily plotted by navigating to +the subfolder where the simulation is saving its output and typing: + +:: + + gnuplot + set terminal dumb + plot ``data.txt" using 1:2 with lines + +Where ‘1:2’ refers to the columns in data.txt that you wish to plot +against one another. Your output should look something like this: + + +.. _fig-gnuplot: +.. figure:: fig/gnuplot.* + :scale: 60 % + :align: center + + Gnuplot output + +Comments +======== + +- Amazon AWS allows you to stop your instance and restart it again + later with the same files and package installations. If you wish to + create several instances that all contain the same installation and + file directory you can create a snapshot of your default image which + you will be able to use to create various volumes that you can attach + to new instances. These actions are all performed very easily and + graphically through the EC2 console + +- You can use Spot Instances, which are a special type of instance that + allow you to bid on unused servers. The price is heavily discounted + and worth looking into for any YADE user that wishes to run hundreds + of hours of simulations. + +- For most simulations, your computational efficiency will decrease if + you use :math:`\textgreater` 8 cores per simulation. It is preferable + to use yadedaily-batch to distribute your cores accordingly so that + you always dedicate 8 cores to each simulation and ensure 100% of the + processor is running. + === added file 'doc/sphinx/fig/connect.png' Binary files doc/sphinx/fig/connect.png 1970-01-01 00:00:00 +0000 and doc/sphinx/fig/connect.png 2017-02-23 11:21:46 +0000 differ === added file 'doc/sphinx/fig/gnuplot.png' Binary files doc/sphinx/fig/gnuplot.png 1970-01-01 00:00:00 +0000 and doc/sphinx/fig/gnuplot.png 2017-02-23 11:21:46 +0000 differ === added file 'doc/sphinx/fig/instancetype.png' Binary files doc/sphinx/fig/instancetype.png 1970-01-01 00:00:00 +0000 and doc/sphinx/fig/instancetype.png 2017-02-23 11:21:46 +0000 differ === added file 'doc/sphinx/fig/launchinstance.png' Binary files doc/sphinx/fig/launchinstance.png 1970-01-01 00:00:00 +0000 and doc/sphinx/fig/launchinstance.png 2017-02-23 11:21:46 +0000 differ === added file 'doc/sphinx/fig/publicdns.png' Binary files doc/sphinx/fig/publicdns.png 1970-01-01 00:00:00 +0000 and doc/sphinx/fig/publicdns.png 2017-02-23 11:21:46 +0000 differ === added file 'doc/sphinx/fig/ubunut.png' Binary files doc/sphinx/fig/ubunut.png 1970-01-01 00:00:00 +0000 and doc/sphinx/fig/ubunut.png 2017-02-23 11:21:46 +0000 differ === modified file 'doc/sphinx/index-toctree.rst' --- doc/sphinx/index-toctree.rst 2015-03-09 09:26:53 +0000 +++ doc/sphinx/index-toctree.rst 2017-02-23 11:21:46 +0000 @@ -20,6 +20,8 @@ installation.rst + amazonEC2.rst + github.rst formulation.rst === modified file 'doc/sphinx/installation.rst' --- doc/sphinx/installation.rst 2017-01-16 19:55:04 +0000 +++ doc/sphinx/installation.rst 2017-02-23 11:21:46 +0000 @@ -315,3 +315,8 @@ More informations about this alternative are available `here <http://people.3sr-grenoble.fr/users/bchareyre/pubs/yubuntu/>`_ (see the README file first). + +Cloud Computing +---------------- + +It is possible to exploit cloud computing services to run Yade. The combo Yade/Amazon Web Service has been found to work well, namely. Detailed instructions for migrating to amazon can be found in the section :ref:`CloudComputing`. \ No newline at end of file
_______________________________________________ Mailing list: https://launchpad.net/~yade-dev Post to : yade-dev@lists.launchpad.net Unsubscribe : https://launchpad.net/~yade-dev More help : https://help.launchpad.net/ListHelp
