Modified: aurora/site/publish/documentation/latest/hooks/index.html URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/latest/hooks/index.html?rev=1719617&r1=1719616&r2=1719617&view=diff ============================================================================== --- aurora/site/publish/documentation/latest/hooks/index.html (original) +++ aurora/site/publish/documentation/latest/hooks/index.html Sat Dec 12 01:46:48 2015 @@ -21,12 +21,11 @@ </script> </head> <body> - <div class="container-fluid section-header"> <div class="container"> <div class="nav nav-bar"> <a href="/"><img src="/assets/img/aurora_logo_dkbkg.svg" width="300" alt="Transparent Apache Aurora logo with dark background"/></a> - <ul class="nav navbar-nav navbar-right"> + <ul class="nav navbar-nav navbar-right"> <li><a href="/documentation/latest/">Documentation</a></li> <li><a href="/community/">Community</a></li> <li><a href="/downloads/">Downloads</a></li> @@ -34,7 +33,8 @@ </ul> </div> </div> -</div> +</div> + <div class="container-fluid"> <div class="container content"> <div class="col-md-12 documentation"> @@ -81,7 +81,7 @@ return <code>False</code>. Designers/con consider whether or not to error-trap them. You can error trap at the highest level very generally and always pass the <code>pre_</code> hook by returning <code>True</code>. For example:</p> -<pre class="highlight python"><span class="k">def</span> <span class="nf">pre_create</span><span class="p">(</span><span class="o">...</span><span class="p">):</span> +<pre class="highlight python"><code><span class="k">def</span> <span class="nf">pre_create</span><span class="p">(</span><span class="o">...</span><span class="p">):</span> <span class="n">do_something</span><span class="p">()</span> <span class="c"># if do_something fails with an exception, the create_job is not attempted!</span> <span class="k">return</span> <span class="bp">True</span> @@ -89,10 +89,11 @@ returning <code>True</code>. For example <span class="k">def</span> <span class="nf">pre_create</span><span class="p">(</span><span class="o">...</span><span class="p">):</span> <span class="k">try</span><span class="p">:</span> <span class="n">do_something</span><span class="p">()</span> <span class="c"># may cause exception</span> - <span class="k">except</span> <span class="n">Exception</span><span class="p">:</span> <span class="c"># generic error trap will catch it</span> + <span class="k">except</span> <span class="nb">Exception</span><span class="p">:</span> <span class="c"># generic error trap will catch it</span> <span class="k">pass</span> <span class="c"># and ignore the exception</span> <span class="k">return</span> <span class="bp">True</span> <span class="c"># create_job will run in any case!</span> -</pre> +</code></pre> + <p><code>post_<method_name></code>: A <code>post_</code> hook executes after its associated method successfully finishes running. If it fails, the already executed method is unaffected. A <code>post_</code> hook’s error is trapped, and any later operations are unaffected.</p> <p><code>err_<method_name></code>: Executes only when its associated method returns a status other than OK or throws an exception. If an <code>err_</code> hook fails, the already executed method is unaffected. An <code>err_</code> hook’s error is trapped, and any later operations are unaffected.</p> @@ -187,11 +188,12 @@ returning <code>True</code>. For example <p>By default, hooks are inactive. If you do not want to use hooks, you do not need to make any changes to your code. If you do want to use hooks, you will need to alter your <code>.aurora</code> config file to activate them both for the configuration as a whole as well as for individual <code>Job</code>s. And, of course, you will need to define in your config file what happens when a particular hook executes.</p> -<h2 id="-aurora-config-file-settings">.aurora Config File Settings</h2> +<h2 id="aurora-config-file-settings">.aurora Config File Settings</h2> <p>You can define a top-level <code>hooks</code> variable in any <code>.aurora</code> config file. <code>hooks</code> is a list of all objects that define hooks used by <code>Job</code>s defined in that config file. If you do not want to define any hooks for a configuration, <code>hooks</code> is optional.</p> -<pre class="highlight text">hooks = [Object_with_defined_hooks1, Object_with_defined_hooks2] -</pre> +<pre class="highlight plaintext"><code>hooks = [Object_with_defined_hooks1, Object_with_defined_hooks2] +</code></pre> + <p>Be careful when assembling a config file using <code>include</code> on multiple smaller config files. If there are multiple files that assign a value to <code>hooks</code>, only the last assignment made will stick. For example, if <code>x.aurora</code> has <code>hooks = [a, b, c]</code> and <code>y.aurora</code> has <code>hooks = [d, e, f]</code> and <code>z.aurora</code> has, in this order, <code>include x.aurora</code> and <code>include y.aurora</code>, the <code>hooks</code> value will be <code>[d, e, f]</code>.</p> <p>Also, for any <code>Job</code> that you want to use hooks with, its <code>Job</code> definition in the <code>.aurora</code> config file must set an <code>enable_hooks</code> flag to <code>True</code> (it defaults to <code>False</code>). By default, hooks are disabled and you must enable them for <code>Job</code>s of your choice.</p> @@ -199,21 +201,24 @@ returning <code>True</code>. For example <p>To summarize, to use hooks for a particular job, you must both activate hooks for your config file as a whole, and for that job. Activating hooks only for individual jobs won’t work, nor will only activating hooks for your config file as a whole. You must also specify the hooks’ defining object in the <code>hooks</code> variable.</p> <p>Recall that <code>.aurora</code> config files are written in Pystachio. So the following turns on hooks for production jobs at cluster1 and cluster2, but leaves them off for similar jobs with a defined user role. Of course, you also need to list the objects that define the hooks in your config file’s <code>hooks</code> variable.</p> -<pre class="highlight python"><span class="n">jobs</span> <span class="o">=</span> <span class="p">[</span> - <span class="n">Job</span><span class="p">(</span><span class="n">enable_hooks</span> <span class="o">=</span> <span class="bp">True</span><span class="p">,</span> <span class="n">cluster</span> <span class="o">=</span> <span class="n">c</span><span class="p">,</span> <span class="n">env</span> <span class="o">=</span> <span class="s">'prod'</span><span class="p">)</span> <span class="k">for</span> <span class="n">c</span> <span class="ow">in</span> <span class="p">(</span><span class="s">'cluster1'</span><span class="p">,</span> <span class="s">'cluster2'</span><span class="p">)</span> +<pre class="highlight python"><code><span class="n">jobs</span> <span class="o">=</span> <span class="p">[</span> + <span class="n">Job</span><span class="p">(</span><span class="n">enable_hooks</span> <span class="o">=</span> <span class="bp">True</span><span class="p">,</span> <span class="n">cluster</span> <span class="o">=</span> <span class="n">c</span><span class="p">,</span> <span class="n">env</span> <span class="o">=</span> <span class="s">'prod'</span><span class="p">)</span> <span class="k">for</span> <span class="n">c</span> <span class="ow">in</span> <span class="p">(</span><span class="s">'cluster1'</span><span class="p">,</span> <span class="s">'cluster2'</span><span class="p">)</span> <span class="p">]</span> <span class="n">jobs</span><span class="o">.</span><span class="n">extend</span><span class="p">(</span> - <span class="n">Job</span><span class="p">(</span><span class="n">cluster</span> <span class="o">=</span> <span class="n">c</span><span class="p">,</span> <span class="n">env</span> <span class="o">=</span> <span class="s">'prod'</span><span class="p">,</span> <span class="n">role</span> <span class="o">=</span> <span class="n">getpass</span><span class="o">.</span><span class="n">getuser</span><span class="p">())</span> <span class="k">for</span> <span class="n">c</span> <span class="ow">in</span> <span class="p">(</span><span class="s">'cluster1'</span><span class="p">,</span> <span class="s">'cluster2'</span><span class="p">))</span> + <span class="n">Job</span><span class="p">(</span><span class="n">cluster</span> <span class="o">=</span> <span class="n">c</span><span class="p">,</span> <span class="n">env</span> <span class="o">=</span> <span class="s">'prod'</span><span class="p">,</span> <span class="n">role</span> <span class="o">=</span> <span class="n">getpass</span><span class="o">.</span><span class="n">getuser</span><span class="p">())</span> <span class="k">for</span> <span class="n">c</span> <span class="ow">in</span> <span class="p">(</span><span class="s">'cluster1'</span><span class="p">,</span> <span class="s">'cluster2'</span><span class="p">))</span> <span class="c"># Hooks disabled for these jobs</span> -</pre> +</code></pre> + <h2 id="command-line">Command Line</h2> <p>All Aurora Command Line commands now accept an <code>.aurora</code> config file as an optional parameter (some, of course, accept it as a required parameter). Whenever a command has a <code>.aurora</code> file parameter, any hooks specified and activated in the <code>.aurora</code> file can be used. For example:</p> -<pre class="highlight text">aurora job restart cluster1/role/env/app myapp.aurora -</pre> +<pre class="highlight plaintext"><code>aurora job restart cluster1/role/env/app myapp.aurora +</code></pre> + <p>The command activates any hooks specified and activated in <code>myapp.aurora</code>. For the <code>restart</code> command, that is the only thing the <code>myapp.aurora</code> parameter does. So, if the command was the following, since there is no <code>.aurora</code> config file to specify any hooks, no hooks on the <code>restart</code> command can run.</p> -<pre class="highlight text">aurora job restart cluster1/role/env/app -</pre> +<pre class="highlight plaintext"><code>aurora job restart cluster1/role/env/app +</code></pre> + <h2 id="hooks-protocol">Hooks Protocol</h2> <p>Any object defined in the <code>.aurora</code> config file can define hook methods. You should define your hook methods within a class, and then use the class name as a value in the <code>hooks</code> list in your config file.</p> @@ -221,21 +226,23 @@ returning <code>True</code>. For example <p>Note that you can define other methods in the class that its hook methods can call; all the logic of a hook does not have to be in its definition.</p> <p>The following example defines a class containing a <code>pre_kill_job</code> hook definition that calls another method defined in the class.</p> -<pre class="highlight python"><span class="c"># Defines a method pre_kill_job</span> +<pre class="highlight python"><code><span class="c"># Defines a method pre_kill_job</span> <span class="k">class</span> <span class="nc">KillConfirmer</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span> <span class="k">def</span> <span class="nf">confirm</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">msg</span><span class="p">):</span> - <span class="k">return</span> <span class="nb">raw_input</span><span class="p">(</span><span class="n">msg</span><span class="p">)</span><span class="o">.</span><span class="n">lower</span><span class="p">()</span> <span class="o">==</span> <span class="s">'yes'</span> + <span class="k">return</span> <span class="nb">raw_input</span><span class="p">(</span><span class="n">msg</span><span class="p">)</span><span class="o">.</span><span class="n">lower</span><span class="p">()</span> <span class="o">==</span> <span class="s">'yes'</span> <span class="k">def</span> <span class="nf">pre_kill_job</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">job_key</span><span class="p">,</span> <span class="n">shards</span><span class="o">=</span><span class="bp">None</span><span class="p">):</span> - <span class="n">shards</span> <span class="o">=</span> <span class="p">(</span><span class="s">'shards </span><span class="si">%</span><span class="s">s'</span> <span class="o">%</span> <span class="n">shards</span><span class="p">)</span> <span class="k">if</span> <span class="n">shards</span> <span class="ow">is</span> <span class="ow">not</span> <span class="bp">None</span> <span class="k">else</span> <span class="s">'all shards'</span> - <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">confirm</span><span class="p">(</span><span class="s">'Are you sure you want to kill </span><span class="si">%</span><span class="s">s (</span><span class="si">%</span><span class="s">s)? (yes/no): '</span> + <span class="n">shards</span> <span class="o">=</span> <span class="p">(</span><span class="s">'shards </span><span class="si">%</span><span class="s">s'</span> <span class="o">%</span> <span class="n">shards</span><span class="p">)</span> <span class="k">if</span> <span class="n">shards</span> <span class="ow">is</span> <span class="ow">not</span> <span class="bp">None</span> <span class="k">else</span> <span class="s">'all shards'</span> + <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">confirm</span><span class="p">(</span><span class="s">'Are you sure you want to kill </span><span class="si">%</span><span class="s">s (</span><span class="si">%</span><span class="s">s)? (yes/no): '</span> <span class="o">%</span> <span class="p">(</span><span class="n">job_key</span><span class="p">,</span> <span class="n">shards</span><span class="p">))</span> -</pre> +</code></pre> + <h3 id="pre_-methods">pre_ Methods</h3> <p><code>pre_</code> methods have the signature:</p> -<pre class="highlight text">pre_<API method name>(self, <associated method's signature>) -</pre> +<pre class="highlight plaintext"><code>pre_<API method name>(self, <associated method's signature>) +</code></pre> + <p><code>pre_</code> methods have the same signature as their associated method, with the addition of <code>self</code> as the first parameter. See the <a href="#Chart">chart</a> above for the mapping of parameters to methods. When writing <code>pre_</code> methods, you can use the <code>*</code> and <code>**</code> syntax to designate that all unspecified parameters are passed in a list to the <code>*</code>ed variable and all named parameters with values are passed as name/value pairs to the <code>**</code>ed variable.</p> <p>If this method returns False, the API command call aborts.</p> @@ -243,8 +250,9 @@ returning <code>True</code>. For example <h3 id="err_-methods">err_ Methods</h3> <p><code>err_</code> methods have the signature:</p> -<pre class="highlight text">err_<API method name>(self, exc, <associated method's signature>) -</pre> +<pre class="highlight plaintext"><code>err_<API method name>(self, exc, <associated method's signature>) +</code></pre> + <p><code>err_</code> methods have the same signature as their associated method, with the addition of a first parameter <code>self</code> and a second parameter <code>exc</code>. <code>exc</code> is either a result with responseCode other than <code>ResponseCode.OK</code> or an <code>Exception</code>. See the <a href="#Chart">chart</a> above for the mapping of parameters to methods. When writing <code>err</code>_ methods, you can use the <code>*</code> and <code>**</code> syntax to designate that all unspecified parameters are passed in a list to the <code>*</code>ed variable and all named parameters with values are passed as name/value pairs to the <code>**</code>ed variable.</p> <p><code>err_</code> method return codes are ignored.</p> @@ -252,8 +260,9 @@ returning <code>True</code>. For example <h3 id="post_-methods">post_ Methods</h3> <p><code>post_</code> methods have the signature:</p> -<pre class="highlight text">post_<API method name>(self, result, <associated method signature>) -</pre> +<pre class="highlight plaintext"><code>post_<API method name>(self, result, <associated method signature>) +</code></pre> + <p><code>post_</code> method parameters are <code>self</code>, then <code>result</code>, followed by the same parameter signature as their associated method. <code>result</code> is the result of the associated method call. See the <a href="#chart">chart</a> above for the mapping of parameters to methods. When writing <code>post_</code> methods, you can use the <code>*</code> and <code>**</code> syntax to designate that all unspecified arguments are passed in a list to the <code>*</code>ed parameter and all unspecified named arguments with values are passed as name/value pairs to the <code>**</code>ed parameter.</p> <p><code>post_</code> method return codes are ignored.</p> @@ -261,8 +270,9 @@ returning <code>True</code>. For example <h2 id="generic-hooks">Generic Hooks</h2> <p>There are seven Aurora API Methods which any of the three hook types can attach to. Thus, there are 21 possible hook/method combinations for a single <code>.aurora</code> config file. Say that you define <code>pre_</code> and <code>post_</code> hooks for the <code>restart</code> method. That leaves 19 undefined hook/method combinations; <code>err_restart</code> and the 3 <code>pre_</code>, <code>post_</code>, and <code>err_</code> hooks for each of the other 6 hookable methods. You can define what happens when any of these otherwise undefined 19 hooks execute via a generic hook, whose signature is:</p> -<pre class="highlight python"><span class="n">generic_hook</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">hook_config</span><span class="p">,</span> <span class="n">event</span><span class="p">,</span> <span class="n">method_name</span><span class="p">,</span> <span class="n">result_or_err</span><span class="p">,</span> <span class="n">args</span><span class="o">*</span><span class="p">,</span> <span class="n">kw</span><span class="o">**</span><span class="p">)</span> -</pre> +<pre class="highlight python"><code><span class="n">generic_hook</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">hook_config</span><span class="p">,</span> <span class="n">event</span><span class="p">,</span> <span class="n">method_name</span><span class="p">,</span> <span class="n">result_or_err</span><span class="p">,</span> <span class="n">args</span><span class="o">*</span><span class="p">,</span> <span class="n">kw</span><span class="o">**</span><span class="p">)</span> +</code></pre> + <p>where:</p> <ul> @@ -280,37 +290,40 @@ returning <code>True</code>. For example </ul> <p>Example:</p> -<pre class="highlight python"><span class="c"># Overrides the standard do-nothing generic_hook by adding a log writing operation.</span> +<pre class="highlight python"><code><span class="c"># Overrides the standard do-nothing generic_hook by adding a log writing operation.</span> <span class="kn">from</span> <span class="nn">twitter.common</span> <span class="kn">import</span> <span class="n">log</span> <span class="k">class</span> <span class="nc">Logger</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span> - <span class="s">'''Adds to the log every time a hookable API method is called'''</span> + <span class="s">'''Adds to the log every time a hookable API method is called'''</span> <span class="k">def</span> <span class="nf">generic_hook</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">hook_config</span><span class="p">,</span> <span class="n">event</span><span class="p">,</span> <span class="n">method_name</span><span class="p">,</span> <span class="n">result_or_err</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kw</span><span class="p">)</span> - <span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s">'</span><span class="si">%</span><span class="s">s: </span><span class="si">%</span><span class="s">s_</span><span class="si">%</span><span class="s">s of </span><span class="si">%</span><span class="s">s'</span> + <span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s">'</span><span class="si">%</span><span class="s">s: </span><span class="si">%</span><span class="s">s_</span><span class="si">%</span><span class="s">s of </span><span class="si">%</span><span class="s">s'</span> <span class="o">%</span> <span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">__class__</span><span class="o">.</span><span class="n">__name__</span><span class="p">,</span> <span class="n">event</span><span class="p">,</span> <span class="n">method_name</span><span class="p">,</span> <span class="n">hook_config</span><span class="o">.</span><span class="n">job_key</span><span class="p">))</span> -</pre> +</code></pre> + <h2 id="hooks-process-checklist">Hooks Process Checklist</h2> <ol> <li>In your <code>.aurora</code> config file, add a <code>hooks</code> variable. Note that you may want to define a <code>.aurora</code> file only for hook definitions and then include this file in multiple other config files that you want to use the same hooks.</li> </ol> -<pre class="highlight python"><span class="n">hooks</span> <span class="o">=</span> <span class="p">[]</span> -</pre> +<pre class="highlight python"><code><span class="n">hooks</span> <span class="o">=</span> <span class="p">[]</span> +</code></pre> + <ol> <li>In the <code>hooks</code> variable, list all objects that define hooks used by <code>Job</code>s defined in this config:</li> </ol> -<pre class="highlight python"><span class="n">hooks</span> <span class="o">=</span> <span class="p">[</span><span class="n">Object_hook_definer1</span><span class="p">,</span> <span class="n">Object_hook_definer2</span><span class="p">]</span> -</pre> +<pre class="highlight python"><code><span class="n">hooks</span> <span class="o">=</span> <span class="p">[</span><span class="n">Object_hook_definer1</span><span class="p">,</span> <span class="n">Object_hook_definer2</span><span class="p">]</span> +</code></pre> + <ol> <li><p>For each job that uses hooks in this config file, add <code>enable_hooks = True</code> to the <code>Job</code> definition. Note that this is necessary even if you only want to use the generic hook.</p></li> <li><p>Write your <code>pre_</code>, <code>post_</code>, and <code>err_</code> hook definitions as part of an object definition in your <code>.aurora</code> config file.</p></li> <li><p>If desired, write your <code>generic_hook</code> definition as part of an object definition in your <code>.aurora</code> config file. Remember, the object must be listed as a member of <code>hooks</code>.</p></li> <li><p>If your Aurora command line command does not otherwise take an <code>.aurora</code> config file argument, add the appropriate <code>.aurora</code> file as an argument in order to define and activate the configuration’s hooks.</p></li> </ol> + </div> </div> </div> - <div class="container-fluid section-footer buffer"> <div class="container"> <div class="row"> @@ -335,5 +348,6 @@ returning <code>True</code>. For example </div> </div> </div> + </body> </html> \ No newline at end of file
Modified: aurora/site/publish/documentation/latest/index.html URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/latest/index.html?rev=1719617&r1=1719616&r2=1719617&view=diff ============================================================================== --- aurora/site/publish/documentation/latest/index.html (original) +++ aurora/site/publish/documentation/latest/index.html Sat Dec 12 01:46:48 2015 @@ -21,12 +21,11 @@ </script> </head> <body> - <div class="container-fluid section-header"> <div class="container"> <div class="nav nav-bar"> <a href="/"><img src="/assets/img/aurora_logo_dkbkg.svg" width="300" alt="Transparent Apache Aurora logo with dark background"/></a> - <ul class="nav navbar-nav navbar-right"> + <ul class="nav navbar-nav navbar-right"> <li><a href="/documentation/latest/">Documentation</a></li> <li><a href="/community/">Community</a></li> <li><a href="/downloads/">Downloads</a></li> @@ -34,7 +33,8 @@ </ul> </div> </div> -</div> +</div> + <div class="container-fluid"> <div class="container content"> <div class="col-md-12 documentation"> @@ -72,23 +72,30 @@ <li><a href="/documentation/latest/storage/">Scheduler Storage</a></li> <li><a href="/documentation/latest/storage-config/">Scheduler Storage and Maintenance</a></li> <li><a href="/documentation/latest/sla/">SLA Measurement</a></li> -<li><a href="/documentation/latest/resource-isolation/">Resource Isolation and Sizing</a></li> +<li><a href="/documentation/latest/resources/">Resource Isolation and Sizing</a></li> <li><a href="/documentation/latest/test-resource-generation/">Generating test resources</a></li> </ul> <h2 id="developers">Developers</h2> <ul> -<li><a href="/documentation/latest/contributing/">Contributing to the project</a></li> +<li><a href="../CONTRIBUTING.md">Contributing to the project</a></li> <li><a href="/documentation/latest/developing-aurora-scheduler/">Developing the Aurora Scheduler</a></li> <li><a href="/documentation/latest/developing-aurora-client/">Developing the Aurora Client</a></li> <li><a href="/documentation/latest/committers/">Committers Guide</a></li> +<li><a href="/documentation/latest/build-system/">Build System</a></li> </ul> + +<h2 id="additional-resources">Additional Resources</h2> + +<ul> +<li><a href="/documentation/latest/presentations/">Presentation videos and slides</a></li> +</ul> + </div> </div> </div> - <div class="container-fluid section-footer buffer"> <div class="container"> <div class="row"> @@ -113,5 +120,6 @@ </div> </div> </div> + </body> </html> \ No newline at end of file Modified: aurora/site/publish/documentation/latest/monitoring/index.html URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/latest/monitoring/index.html?rev=1719617&r1=1719616&r2=1719617&view=diff ============================================================================== --- aurora/site/publish/documentation/latest/monitoring/index.html (original) +++ aurora/site/publish/documentation/latest/monitoring/index.html Sat Dec 12 01:46:48 2015 @@ -21,12 +21,11 @@ </script> </head> <body> - <div class="container-fluid section-header"> <div class="container"> <div class="nav nav-bar"> <a href="/"><img src="/assets/img/aurora_logo_dkbkg.svg" width="300" alt="Transparent Apache Aurora logo with dark background"/></a> - <ul class="nav navbar-nav navbar-right"> + <ul class="nav navbar-nav navbar-right"> <li><a href="/documentation/latest/">Documentation</a></li> <li><a href="/community/">Community</a></li> <li><a href="/downloads/">Downloads</a></li> @@ -34,7 +33,8 @@ </ul> </div> </div> -</div> +</div> + <div class="container-fluid"> <div class="container content"> <div class="col-md-12 documentation"> @@ -49,7 +49,7 @@ since it will give you a global view of <p>The scheduler exposes a <em>lot</em> of instrumentation data via its HTTP interface. You can get a quick peek at the first few of these in our vagrant image:</p> -<pre class="highlight text">$ vagrant ssh -c 'curl -s localhost:8081/vars | head' +<pre class="highlight plaintext"><code>$ vagrant ssh -c 'curl -s localhost:8081/vars | head' async_tasks_completed 1004 attribute_store_fetch_all_events 15 attribute_store_fetch_all_events_per_sec 0.0 @@ -60,24 +60,26 @@ attribute_store_fetch_one_events 3391 attribute_store_fetch_one_events_per_sec 0.0 attribute_store_fetch_one_nanos_per_event 0.0 attribute_store_fetch_one_nanos_total 454690753 -</pre> +</code></pre> + <p>These values are served as <code>Content-Type: text/plain</code>, with each line containing a space-separated metric name and value. Values may be integers, doubles, or strings (note: strings are static, others may be dynamic).</p> <p>If your monitoring infrastructure prefers JSON, the scheduler exports that as well:</p> -<pre class="highlight text">$ vagrant ssh -c 'curl -s localhost:8081/vars.json | python -mjson.tool | head' +<pre class="highlight plaintext"><code>$ vagrant ssh -c 'curl -s localhost:8081/vars.json | python -mjson.tool | head' { - "async_tasks_completed": 1009, - "attribute_store_fetch_all_events": 15, - "attribute_store_fetch_all_events_per_sec": 0.0, - "attribute_store_fetch_all_nanos_per_event": 0.0, - "attribute_store_fetch_all_nanos_total": 3048285, - "attribute_store_fetch_all_nanos_total_per_sec": 0.0, - "attribute_store_fetch_one_events": 3409, - "attribute_store_fetch_one_events_per_sec": 0.0, - "attribute_store_fetch_one_nanos_per_event": 0.0, -</pre> + "async_tasks_completed": 1009, + "attribute_store_fetch_all_events": 15, + "attribute_store_fetch_all_events_per_sec": 0.0, + "attribute_store_fetch_all_nanos_per_event": 0.0, + "attribute_store_fetch_all_nanos_total": 3048285, + "attribute_store_fetch_all_nanos_total_per_sec": 0.0, + "attribute_store_fetch_one_events": 3409, + "attribute_store_fetch_one_events_per_sec": 0.0, + "attribute_store_fetch_one_nanos_per_event": 0.0, +</code></pre> + <p>This will be the same data as above, served with <code>Content-Type: application/json</code>.</p> <h2 id="viewing-live-stat-samples-on-the-scheduler">Viewing live stat samples on the scheduler</h2> @@ -118,177 +120,125 @@ recommend you start with a strict value adjust thresholds as you see fit. Feel free to ask us if you would like to validate that your alerts and thresholds make sense.</p> -<h4 id="code-code"><code>jvm_uptime_secs</code></h4> +<h2 id="important-stats">Important stats</h2> -<p>Type: integer counter</p> +<h3 id="jvm_uptime_secs"><code>jvm_uptime_secs</code></h3> -<h4 id="description">Description</h4> +<p>Type: integer counter</p> <p>The number of seconds the JVM process has been running. Comes from <a href="http://docs.oracle.com/javase/7/docs/api/java/lang/management/RuntimeMXBean.html#getUptime()">RuntimeMXBean#getUptime()</a></p> -<h4 id="alerting">Alerting</h4> - <p>Detecting resets (decreasing values) on this stat will tell you that the scheduler is failing to stay alive.</p> -<h4 id="triage">Triage</h4> - <p>Look at the scheduler logs to identify the reason the scheduler is exiting.</p> -<h4 id="code-code"><code>system_load_avg</code></h4> +<h3 id="system_load_avg"><code>system_load_avg</code></h3> <p>Type: double gauge</p> -<h4 id="description">Description</h4> - <p>The current load average of the system for the last minute. Comes from <a href="http://docs.oracle.com/javase/7/docs/api/java/lang/management/OperatingSystemMXBean.html?is-external=true#getSystemLoadAverage()">OperatingSystemMXBean#getSystemLoadAverage()</a>.</p> -<h4 id="alerting">Alerting</h4> - <p>A high sustained value suggests that the scheduler machine may be over-utilized.</p> -<h4 id="triage">Triage</h4> - <p>Use standard unix tools like <code>top</code> and <code>ps</code> to track down the offending process(es).</p> -<h4 id="code-code"><code>process_cpu_cores_utilized</code></h4> +<h3 id="process_cpu_cores_utilized"><code>process_cpu_cores_utilized</code></h3> <p>Type: double gauge</p> -<h4 id="description">Description</h4> - <p>The current number of CPU cores in use by the JVM process. This should not exceed the number of logical CPU cores on the machine. Derived from <a href="http://docs.oracle.com/javase/7/docs/jre/api/management/extension/com/sun/management/OperatingSystemMXBean.html";>OperatingSystemMXBean#getProcessCpuTime()</a></p> -<h4 id="alerting">Alerting</h4> - <p>A high sustained value indicates that the scheduler is overworked. Due to current internal design limitations, if this value is sustained at <code>1</code>, there is a good chance the scheduler is under water.</p> -<h4 id="triage">Triage</h4> - <p>There are two main inputs that tend to drive this figure: task scheduling attempts and status updates from Mesos. You may see activity in the scheduler logs to give an indication of where time is being spent. Beyond that, it really takes good familiarity with the code to effectively triage this. We suggest engaging with an Aurora developer.</p> -<h4 id="code-code"><code>task_store_LOST</code></h4> +<h3 id="task_store_lost"><code>task_store_LOST</code></h3> <p>Type: integer gauge</p> -<h4 id="description">Description</h4> - <p>The number of tasks stored in the scheduler that are in the <code>LOST</code> state, and have been rescheduled.</p> -<h4 id="alerting">Alerting</h4> - <p>If this value is increasing at a high rate, it is a sign of trouble.</p> -<h4 id="triage">Triage</h4> - <p>There are many sources of <code>LOST</code> tasks in Mesos: the scheduler, master, slave, and executor can all trigger this. The first step is to look in the scheduler logs for <code>LOST</code> to identify where the state changes are originating.</p> -<h4 id="code-code"><code>scheduler_resource_offers</code></h4> +<h3 id="scheduler_resource_offers"><code>scheduler_resource_offers</code></h3> <p>Type: integer counter</p> -<h4 id="description">Description</h4> - <p>The number of resource offers that the scheduler has received.</p> -<h4 id="alerting">Alerting</h4> - <p>For a healthy scheduler, this value must be increasing over time.</p> -<h5 id="triage">Triage</h5> - <p>Assuming the scheduler is up and otherwise healthy, you will want to check if the master thinks it is sending offers. You should also look at the master’s web interface to see if it has a large number of outstanding offers that it is waiting to be returned.</p> -<h4 id="code-code"><code>framework_registered</code></h4> +<h3 id="framework_registered"><code>framework_registered</code></h3> <p>Type: binary integer counter</p> -<h4 id="description">Description</h4> - <p>Will be <code>1</code> for the leading scheduler that is registered with the Mesos master, <code>0</code> for passive schedulers,</p> -<h4 id="alerting">Alerting</h4> - <p>A sustained period without a <code>1</code> (or where <code>sum() != 1</code>) warrants investigation.</p> -<h4 id="triage">Triage</h4> - <p>If there is no leading scheduler, look in the scheduler and master logs for why. If there are multiple schedulers claiming leadership, this suggests a split brain and warrants filing a critical bug.</p> -<h4 id="code-code"><code>rate(scheduler_log_native_append_nanos_total)/rate(scheduler_log_native_append_events)</code></h4> +<h3 id="rate-scheduler_log_native_append_nanos_total-rate-scheduler_log_native_append_events"><code>rate(scheduler_log_native_append_nanos_total)/rate(scheduler_log_native_append_events)</code></h3> <p>Type: rate ratio of integer counters</p> -<h4 id="description">Description</h4> - <p>This composes two counters to compute a windowed figure for the latency of replicated log writes.</p> -<h4 id="alerting">Alerting</h4> - <p>A hike in this value suggests disk bandwidth contention.</p> -<h4 id="triage">Triage</h4> - <p>Look in scheduler logs for any reported oddness with saving to the replicated log. Also use standard tools like <code>vmstat</code> and <code>iotop</code> to identify whether the disk has become slow or over-utilized. We suggest using a dedicated disk for the replicated log to mitigate this.</p> -<h4 id="code-code"><code>timed_out_tasks</code></h4> +<h3 id="timed_out_tasks"><code>timed_out_tasks</code></h3> <p>Type: integer counter</p> -<h4 id="description">Description</h4> - <p>Tracks the number of times the scheduler has given up while waiting (for <code>-transient_task_state_timeout</code>) to hear back about a task that is in a transient state (e.g. <code>ASSIGNED</code>, <code>KILLING</code>), and has moved to <code>LOST</code> before rescheduling.</p> -<h4 id="alerting">Alerting</h4> - <p>This value is currently known to increase occasionally when the scheduler fails over (<a href="https://issues.apache.org/jira/browse/AURORA-740";>AURORA-740</a>). However, any large spike in this value warrants investigation.</p> -<h4 id="triage">Triage</h4> - <p>The scheduler will log when it times out a task. You should trace the task ID of the timed out task into the master, slave, and/or executors to determine where the message was dropped.</p> -<h4 id="code-code"><code>http_500_responses_events</code></h4> +<h3 id="http_500_responses_events"><code>http_500_responses_events</code></h3> <p>Type: integer counter</p> -<h4 id="description">Description</h4> - <p>The total number of HTTP 500 status responses sent by the scheduler. Includes API and asset serving.</p> -<h4 id="alerting">Alerting</h4> - <p>An increase warrants investigation.</p> -<h4 id="triage">Triage</h4> - <p>Look in scheduler logs to identify why the scheduler returned a 500, there should be a stack trace.</p> + </div> </div> </div> - <div class="container-fluid section-footer buffer"> <div class="container"> <div class="row"> @@ -313,5 +263,6 @@ task into the master, slave, and/or exec </div> </div> </div> + </body> </html> \ No newline at end of file Modified: aurora/site/publish/documentation/latest/presentations/index.html URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/latest/presentations/index.html?rev=1719617&r1=1719616&r2=1719617&view=diff ============================================================================== --- aurora/site/publish/documentation/latest/presentations/index.html (original) +++ aurora/site/publish/documentation/latest/presentations/index.html Sat Dec 12 01:46:48 2015 @@ -21,12 +21,11 @@ </script> </head> <body> - <div class="container-fluid section-header"> <div class="container"> <div class="nav nav-bar"> <a href="/"><img src="/assets/img/aurora_logo_dkbkg.svg" width="300" alt="Transparent Apache Aurora logo with dark background"/></a> - <ul class="nav navbar-nav navbar-right"> + <ul class="nav navbar-nav navbar-right"> <li><a href="/documentation/latest/">Documentation</a></li> <li><a href="/community/">Community</a></li> <li><a href="/downloads/">Downloads</a></li> @@ -34,7 +33,8 @@ </ul> </div> </div> -</div> +</div> + <div class="container-fluid"> <div class="container content"> <div class="col-md-12 documentation"> @@ -90,11 +90,11 @@ <p>March 25, 2014 at <a href="https://www.eventbrite.com/e/aurora-and-mesosframeworksmeetup-tickets-10850994617";>Aurora and Mesos Frameworks Meetup</a></p></td> </tr> </table> + </div> </div> </div> - <div class="container-fluid section-footer buffer"> <div class="container"> <div class="row"> @@ -119,5 +119,6 @@ </div> </div> </div> + </body> </html> \ No newline at end of file Modified: aurora/site/publish/documentation/latest/resources/index.html URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/latest/resources/index.html?rev=1719617&r1=1719616&r2=1719617&view=diff ============================================================================== --- aurora/site/publish/documentation/latest/resources/index.html (original) +++ aurora/site/publish/documentation/latest/resources/index.html Sat Dec 12 01:46:48 2015 @@ -21,12 +21,11 @@ </script> </head> <body> - <div class="container-fluid section-header"> <div class="container"> <div class="nav nav-bar"> <a href="/"><img src="/assets/img/aurora_logo_dkbkg.svg" width="300" alt="Transparent Apache Aurora logo with dark background"/></a> - <ul class="nav navbar-nav navbar-right"> + <ul class="nav navbar-nav navbar-right"> <li><a href="/documentation/latest/">Documentation</a></li> <li><a href="/community/">Community</a></li> <li><a href="/downloads/">Downloads</a></li> @@ -34,7 +33,8 @@ </ul> </div> </div> -</div> +</div> + <div class="container-fluid"> <div class="container content"> <div class="col-md-12 documentation"> @@ -206,11 +206,11 @@ that role.</p> <a href="configuration-reference.md#job-objects">production</a> jobs may preempt tasks from any non-production job. A production task may only be preempted by tasks from production jobs in the same role with higher <a href="configuration-reference.md#job-objects">priority</a>.</p> + </div> </div> </div> - <div class="container-fluid section-footer buffer"> <div class="container"> <div class="row"> @@ -235,5 +235,6 @@ higher <a href="configuration-reference. </div> </div> </div> + </body> </html> \ No newline at end of file Modified: aurora/site/publish/documentation/latest/scheduler-storage/index.html URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/latest/scheduler-storage/index.html?rev=1719617&r1=1719616&r2=1719617&view=diff ============================================================================== --- aurora/site/publish/documentation/latest/scheduler-storage/index.html (original) +++ aurora/site/publish/documentation/latest/scheduler-storage/index.html Sat Dec 12 01:46:48 2015 @@ -21,12 +21,11 @@ </script> </head> <body> - <div class="container-fluid section-header"> <div class="container"> <div class="nav nav-bar"> <a href="/"><img src="/assets/img/aurora_logo_dkbkg.svg" width="300" alt="Transparent Apache Aurora logo with dark background"/></a> - <ul class="nav navbar-nav navbar-right"> + <ul class="nav navbar-nav navbar-right"> <li><a href="/documentation/latest/">Documentation</a></li> <li><a href="/community/">Community</a></li> <li><a href="/downloads/">Downloads</a></li> @@ -34,7 +33,8 @@ </ul> </div> </div> -</div> +</div> + <div class="container-fluid"> <div class="container content"> <div class="col-md-12 documentation"> @@ -86,11 +86,11 @@ the <code>-dlog_snapshot_interval</code> <p>To disable deduplication, for example to rollback to Aurora, restart all of the cluster’s schedulers with <code>-deduplicate_snapshots=false</code> and either wait for a snapshot or force one using <code>aurora_admin snapshot</code>.</p> + </div> </div> </div> - <div class="container-fluid section-footer buffer"> <div class="container"> <div class="row"> @@ -115,5 +115,6 @@ using <code>aurora_admin snapshot</code> </div> </div> </div> + </body> </html> \ No newline at end of file Modified: aurora/site/publish/documentation/latest/security/index.html URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/latest/security/index.html?rev=1719617&r1=1719616&r2=1719617&view=diff ============================================================================== --- aurora/site/publish/documentation/latest/security/index.html (original) +++ aurora/site/publish/documentation/latest/security/index.html Sat Dec 12 01:46:48 2015 @@ -21,12 +21,11 @@ </script> </head> <body> - <div class="container-fluid section-header"> <div class="container"> <div class="nav nav-bar"> <a href="/"><img src="/assets/img/aurora_logo_dkbkg.svg" width="300" alt="Transparent Apache Aurora logo with dark background"/></a> - <ul class="nav navbar-nav navbar-right"> + <ul class="nav navbar-nav navbar-right"> <li><a href="/documentation/latest/">Documentation</a></li> <li><a href="/community/">Community</a></li> <li><a href="/downloads/">Downloads</a></li> @@ -34,7 +33,8 @@ </ul> </div> </div> -</div> +</div> + <div class="container-fluid"> <div class="container content"> <div class="col-md-12 documentation"> @@ -103,24 +103,26 @@ considerations</a>.</p> <h3 id="server-configuration">Server Configuration</h3> <p>At a minimum you need to set 4 command-line flags on the scheduler:</p> -<pre class="highlight text">-http_authentication_mechanism=BASIC +<pre class="highlight plaintext"><code>-http_authentication_mechanism=BASIC -shiro_realm_modules=INI_AUTHNZ -shiro_ini_path=path/to/security.ini -</pre> +</code></pre> + <p>And create a security.ini file like so:</p> -<pre class="highlight text">[users] +<pre class="highlight plaintext"><code>[users] sally = apple, admin [roles] admin = * -</pre> +</code></pre> + <p>The details of the security.ini file are explained below. Note that this file contains plaintext, unhashed passwords.</p> <h3 id="client-configuration">Client Configuration</h3> <p>To configure the client for HTTP Basic authentication, add an entry to ~/.netrc with your credentials</p> -<pre class="highlight text">% cat ~/.netrc +<pre class="highlight plaintext"><code>% cat ~/.netrc # ... machine aurora.example.com @@ -128,68 +130,78 @@ login sally password apple # ... -</pre> +</code></pre> + <p>No changes are required to <code>clusters.json</code>.</p> -<h2 id="http-spnego-authentication-kerberos-">HTTP SPNEGO Authentication (Kerberos)</h2> +<h2 id="http-spnego-authentication-kerberos">HTTP SPNEGO Authentication (Kerberos)</h2> <h3 id="server-configuration">Server Configuration</h3> <p>At a minimum you need to set 6 command-line flags on the scheduler:</p> -<pre class="highlight text">-http_authentication_mechanism=NEGOTIATE +<pre class="highlight plaintext"><code>-http_authentication_mechanism=NEGOTIATE -shiro_realm_modules=KERBEROS5_AUTHN,INI_AUTHNZ -kerberos_server_principal=HTTP/[email protected] -kerberos_server_keytab=path/to/aurora.example.com.keytab -shiro_ini_path=path/to/security.ini -</pre> +</code></pre> + <p>And create a security.ini file like so:</p> -<pre class="highlight text">% cat path/to/security.ini +<pre class="highlight plaintext"><code>% cat path/to/security.ini [users] sally = _, admin [roles] admin = * -</pre> +</code></pre> + <p>What’s going on here? First, Aurora must be configured to request Kerberos credentials when presented with an unauthenticated request. This is achieved by setting</p> -<pre class="highlight text">-http_authentication_mechanism=NEGOTIATE -</pre> +<pre class="highlight plaintext"><code>-http_authentication_mechanism=NEGOTIATE +</code></pre> + <p>Next, a Realm module must be configured to <strong>authenticate</strong> the current request using the Kerberos credentials that were requested. Aurora ships with a realm module that can do this</p> -<pre class="highlight text">-shiro_realm_modules=KERBEROS5_AUTHN[,...] -</pre> +<pre class="highlight plaintext"><code>-shiro_realm_modules=KERBEROS5_AUTHN[,...] +</code></pre> + <p>The Kerberos5Realm requires a keytab file and a server principal name. The principal name will usually be in the form <code>HTTP/[email protected]</code>.</p> -<pre class="highlight text">-kerberos_server_principal=HTTP/[email protected] +<pre class="highlight plaintext"><code>-kerberos_server_principal=HTTP/[email protected] -kerberos_server_keytab=path/to/aurora.example.com.keytab -</pre> +</code></pre> + <p>The Kerberos5 realm module is authentication-only. For scheduler security to work you must also enable a realm module that provides an Authorizer implementation. For example, to do this using the IniShiroRealmModule:</p> -<pre class="highlight text">-shiro_realm_modules=KERBEROS5_AUTHN,INI_AUTHNZ -</pre> +<pre class="highlight plaintext"><code>-shiro_realm_modules=KERBEROS5_AUTHN,INI_AUTHNZ +</code></pre> + <p>You can then configure authorization using a security.ini file as described below (the password field is ignored). You must configure the realm module with the path to this file:</p> -<pre class="highlight text">-shiro_ini_path=path/to/security.ini -</pre> +<pre class="highlight plaintext"><code>-shiro_ini_path=path/to/security.ini +</code></pre> + <h3 id="client-configuration">Client Configuration</h3> <p>To use Kerberos on the client-side you must build Kerberos-enabled client binaries. Do this with</p> -<pre class="highlight text">./pants binary src/main/python/apache/aurora/client/cli:kaurora -./pants binary src/main/python/apache/aurora/admin:kaurora_admin -</pre> +<pre class="highlight plaintext"><code>./pants binary src/main/python/apache/aurora/kerberos:kaurora +./pants binary src/main/python/apache/aurora/kerberos:kaurora_admin +</code></pre> + <p>You must also configure each cluster where you’ve enabled Kerberos on the scheduler to use Kerberos authentication. Do this by setting <code>auth_mechanism</code> to <code>KERBEROS</code> in <code>clusters.json</code>.</p> -<pre class="highlight text">% cat ~/.aurora/clusters.json +<pre class="highlight plaintext"><code>% cat ~/.aurora/clusters.json { - "devcluser": { - "auth_mechanism": "KERBEROS", + "devcluser": { + "auth_mechanism": "KERBEROS", ... }, ... } -</pre> +</code></pre> + <h1 id="authorization">Authorization</h1> <p>Given a means to authenticate the entity a client claims they are, we need to define what privileges they have.</p> @@ -202,16 +214,17 @@ likely the preferred approach. However are rapidly changing, or if your access control information already exists in another system.</p> <p>You can enable INI-based configuration with following scheduler command line arguments:</p> -<pre class="highlight text">-http_authentication_mechanism=BASIC +<pre class="highlight plaintext"><code>-http_authentication_mechanism=BASIC -shiro_ini_path=path/to/security.ini -</pre> +</code></pre> + <p><em>note</em> As the argument name reveals, this is using Shiroâs <a href="http://shiro.apache.org/configuration.html#Configuration-INIConfiguration";>IniRealm</a> behind the scenes.</p> <p>The INI file will contain two sections - users and roles. Hereâs an example for what might be in security.ini:</p> -<pre class="highlight text">[users] +<pre class="highlight plaintext"><code>[users] sally = apple, admin jim = 123456, accounting becky = letmein, webapp @@ -222,7 +235,8 @@ steve = password admin = * accounting = thrift.AuroraAdmin:setQuota webapp = thrift.AuroraSchedulerManager:*:webapp -</pre> +</code></pre> + <p>The users section defines user user credentials and the role(s) they are members of. These lines are of the format <code><user> = <password>[, <role>...]</code>. As you probably noticed, the passwords are in plaintext and as a result read access to this file should be restricted.</p> @@ -254,7 +268,7 @@ for more information.</p> <h2 id="packaging-a-realm-module">Packaging a realm module</h2> <p>Package your custom Realm(s) with a Guice module that exposes a <code>Set<Realm></code> multibinding.</p> -<pre class="highlight java"><span class="kn">package</span> <span class="n">com</span><span class="o">.</span><span class="na">example</span><span class="o">;</span> +<pre class="highlight java"><code><span class="kn">package</span> <span class="n">com</span><span class="o">.</span><span class="na">example</span><span class="o">;</span> <span class="kn">import</span> <span class="nn">com.google.inject.AbstractModule</span><span class="o">;</span> <span class="kn">import</span> <span class="nn">com.google.inject.multibindings.Multibinder</span><span class="o">;</span> @@ -272,11 +286,13 @@ for more information.</p> <span class="c1">// Realm implementation.</span> <span class="o">}</span> <span class="o">}</span> -</pre> +</code></pre> + <p>To use your module in the scheduler, include it as a realm module based on its fully-qualified class name:</p> -<pre class="highlight text">-shiro_realm_modules=KERBEROS5_AUTHN,INI_AUTHNZ,com.example.MyRealmModule -</pre> +<pre class="highlight plaintext"><code>-shiro_realm_modules=KERBEROS5_AUTHN,INI_AUTHNZ,com.example.MyRealmModule +</code></pre> + <h1 id="known-issues">Known Issues</h1> <p>While the APIs and SPIs we ship with are stable as of 0.8.0, we are aware of several incremental @@ -289,11 +305,11 @@ improvements. Please follow, vote, or se * <a href="https://issues.apache.org/jira/browse/AURORA-1291";>AURORA-1293</a>: Consider defining a JSON format in place of INI * <a href="https://issues.apache.org/jira/browse/AURORA-1179";>AURORA-1179</a>: Supported hashed passwords in security.ini * <a href="https://issues.apache.org/jira/browse/AURORA-1295";>AURORA-1295</a>: Support security for the ReadOnlyScheduler service</p> + </div> </div> </div> - <div class="container-fluid section-footer buffer"> <div class="container"> <div class="row"> @@ -318,5 +334,6 @@ improvements. Please follow, vote, or se </div> </div> </div> + </body> </html> \ No newline at end of file Modified: aurora/site/publish/documentation/latest/sla/index.html URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/latest/sla/index.html?rev=1719617&r1=1719616&r2=1719617&view=diff ============================================================================== --- aurora/site/publish/documentation/latest/sla/index.html (original) +++ aurora/site/publish/documentation/latest/sla/index.html Sat Dec 12 01:46:48 2015 @@ -21,12 +21,11 @@ </script> </head> <body> - <div class="container-fluid section-header"> <div class="container"> <div class="nav nav-bar"> <a href="/"><img src="/assets/img/aurora_logo_dkbkg.svg" width="300" alt="Transparent Apache Aurora logo with dark background"/></a> - <ul class="nav navbar-nav navbar-right"> + <ul class="nav navbar-nav navbar-right"> <li><a href="/documentation/latest/">Documentation</a></li> <li><a href="/community/">Community</a></li> <li><a href="/downloads/">Downloads</a></li> @@ -34,7 +33,8 @@ </ul> </div> </div> -</div> +</div> + <div class="container-fluid"> <div class="container content"> <div class="col-md-12 documentation"> @@ -60,8 +60,9 @@ Agreements) metrics that defining a contractual relationship between the Aurora/Mesos platform and hosted services.</p> -<p>The Aurora SLA feature currently supports stat collection only for service (non-cron) -production jobs (<code>"production = True"</code> in your <code>.aurora</code> config).</p> +<p>The Aurora SLA feature is by default only enabled for service (non-cron) +production jobs (<code>"production = True"</code> in your <code>.aurora</code> config). It can be enabled for +non-production services via the scheduler command line flag <code>-sla_non_prod_metrics</code>.</p> <p>Counters that track SLA measurements are computed periodically within the scheduler. The individual instance metrics are refreshed every minute (configurable via @@ -145,7 +146,7 @@ percentiles (50th,75th,90th,95th and 99t You can also get customized real-time stats from aurora client. See <code>aurora sla -h</code> for more details.</p> -<h3 id="median-time-to-assigned-mtta-">Median Time To Assigned (MTTA)</h3> +<h3 id="median-time-to-assigned-mtta">Median Time To Assigned (MTTA)</h3> <p><em>Median time a job spends waiting for its tasks to be assigned to a host. This is a combined metric that helps track the dependency of scheduling performance on the requested resources @@ -187,7 +188,7 @@ metric that helps track the dependency o that are still PENDING. This ensures straggler instances (e.g. with unreasonable resource constraints) do not affect metric curves.</p> -<h3 id="median-time-to-running-mttr-">Median Time To Running (MTTR)</h3> +<h3 id="median-time-to-running-mttr">Median Time To Running (MTTR)</h3> <p><em>Median time a job waits for its tasks to reach RUNNING state. This is a comprehensive metric reflecting on the overall time it takes for the Aurora/Mesos to start executing user content.</em></p> @@ -234,11 +235,11 @@ unreasonable resource constraints) do no <li><p>All metrics are calculated at a pre-defined interval (currently set at 1 minute). Scheduler restarts may result in missed collections.</p></li> </ul> + </div> </div> </div> - <div class="container-fluid section-footer buffer"> <div class="container"> <div class="row"> @@ -263,5 +264,6 @@ Scheduler restarts may result in missed </div> </div> </div> + </body> </html> \ No newline at end of file Modified: aurora/site/publish/documentation/latest/storage-config/index.html URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/latest/storage-config/index.html?rev=1719617&r1=1719616&r2=1719617&view=diff ============================================================================== --- aurora/site/publish/documentation/latest/storage-config/index.html (original) +++ aurora/site/publish/documentation/latest/storage-config/index.html Sat Dec 12 01:46:48 2015 @@ -21,12 +21,11 @@ </script> </head> <body> - <div class="container-fluid section-header"> <div class="container"> <div class="nav nav-bar"> <a href="/"><img src="/assets/img/aurora_logo_dkbkg.svg" width="300" alt="Transparent Apache Aurora logo with dark background"/></a> - <ul class="nav navbar-nav navbar-right"> + <ul class="nav navbar-nav navbar-right"> <li><a href="/documentation/latest/">Documentation</a></li> <li><a href="/community/">Community</a></li> <li><a href="/downloads/">Downloads</a></li> @@ -34,7 +33,8 @@ </ul> </div> </div> -</div> +</div> + <div class="container-fluid"> <div class="container content"> <div class="col-md-12 documentation"> @@ -80,18 +80,18 @@ or require attention before deploying in <h3 id="mesos-replicated-log-configuration-flags">Mesos replicated log configuration flags</h3> -<h4 id="-nativelogquorum_size">-native<em>log</em>quorum_size</h4> +<h4 id="nativelogquorum_size">-native<em>log</em>quorum_size</h4> <p>Defines the Mesos replicated log quorum size. See <a href="deploying-aurora-scheduler.md#replicated-log-configuration">the replicated log configuration document</a> on how to choose the right value.</p> -<h4 id="-nativelogfile_path">-native<em>log</em>file_path</h4> +<h4 id="nativelogfile_path">-native<em>log</em>file_path</h4> <p>Location of the Mesos replicated log files. Consider allocating a dedicated disk (preferably SSD) for Mesos replicated log files to ensure optimal storage performance.</p> -<h4 id="-nativelogzkgrouppath">-native<em>log</em>zk<em>group</em>path</h4> +<h4 id="nativelogzkgrouppath">-native<em>log</em>zk<em>group</em>path</h4> <p>ZooKeeper path used for Mesos replicated log quorum discovery.</p> @@ -102,15 +102,15 @@ other available Mesos replicated log con <p>Configuration options for the Aurora scheduler backup manager.</p> -<h4 id="-backup_interval">-backup_interval</h4> +<h4 id="backup_interval">-backup_interval</h4> <p>The interval on which the scheduler writes local storage backups. The default is every hour.</p> -<h4 id="-backup_dir">-backup_dir</h4> +<h4 id="backup_dir">-backup_dir</h4> <p>Directory to write backups to.</p> -<h4 id="-maxsavedbackups">-max<em>saved</em>backups</h4> +<h4 id="maxsavedbackups">-max<em>saved</em>backups</h4> <p>Maximum number of backups to retain before deleting the oldest backup(s).</p> @@ -157,10 +157,10 @@ accomplished by updating the following s <li>Set <code>-mesos_master_address</code> to a non-existent zk address. This will prevent scheduler from registering with Mesos. E.g.: <code>-mesos_master_address=zk://localhost:2181</code></li> <li><code>-max_registration_delay</code> - set to sufficiently long interval to prevent registration timeout -and as a result scheduler suicide. E.g: <code>-max_registration_delay=360min</code></li> -<li>Make sure <code>-gc_executor_path</code> option is not set to prevent accidental task GC. This is -important as scheduler will attempt to reconcile the cluster state and will kill all tasks when -restarted with an empty Mesos replicated log.</li> +and as a result scheduler suicide. E.g: <code>-max_registration_delay=360mins</code></li> +<li>Make sure <code>-reconciliation_initial_delay</code> option is set high enough (e.g.: <code>365days</code>) to +prevent accidental task GC. This is important as scheduler will attempt to reconcile the cluster +state and will kill all tasks when restarted with an empty Mesos replicated log.</li> </ul></li> <li><p>Restart all schedulers</p></li> </ul> @@ -172,7 +172,7 @@ restarted with an empty Mesos replicated <ul> <li>Stop schedulers</li> <li>Delete all files under <code>-native_log_file_path</code> on all schedulers</li> -<li>Initialize Mesos replica’s log file: <code>mesos-log initialize <-native_log_file_path></code></li> +<li>Initialize Mesos replica’s log file: <code>mesos-log initialize --path=<-native_log_file_path></code></li> <li>Restart schedulers</li> </ul> @@ -203,11 +203,11 @@ the provided backup snapshot and initiat <h3 id="cleanup">Cleanup</h3> <p>Undo any modification done during <a href="#preparation">Preparation</a> sequence.</p> + </div> </div> </div> - <div class="container-fluid section-footer buffer"> <div class="container"> <div class="row"> @@ -232,5 +232,6 @@ the provided backup snapshot and initiat </div> </div> </div> + </body> </html> \ No newline at end of file Modified: aurora/site/publish/documentation/latest/storage/index.html URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/latest/storage/index.html?rev=1719617&r1=1719616&r2=1719617&view=diff ============================================================================== --- aurora/site/publish/documentation/latest/storage/index.html (original) +++ aurora/site/publish/documentation/latest/storage/index.html Sat Dec 12 01:46:48 2015 @@ -21,12 +21,11 @@ </script> </head> <body> - <div class="container-fluid section-header"> <div class="container"> <div class="nav nav-bar"> <a href="/"><img src="/assets/img/aurora_logo_dkbkg.svg" width="300" alt="Transparent Apache Aurora logo with dark background"/></a> - <ul class="nav navbar-nav navbar-right"> + <ul class="nav navbar-nav navbar-right"> <li><a href="/documentation/latest/">Documentation</a></li> <li><a href="/community/">Community</a></li> <li><a href="/downloads/">Downloads</a></li> @@ -34,7 +33,8 @@ </ul> </div> </div> -</div> +</div> + <div class="container-fluid"> <div class="container content"> <div class="col-md-12 documentation"> @@ -135,11 +135,11 @@ volatile and replicated writes to succee <p>Any time a scheduler restarts, it restores its volatile state from the most recent position recorded in the replicated log by restoring the snapshot and replaying individual log entries on top to fully recover the state up to the last write.</p> + </div> </div> </div> - <div class="container-fluid section-footer buffer"> <div class="container"> <div class="row"> @@ -164,5 +164,6 @@ recover the state up to the last write.< </div> </div> </div> + </body> </html> \ No newline at end of file Modified: aurora/site/publish/documentation/latest/test-resource-generation/index.html URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/latest/test-resource-generation/index.html?rev=1719617&r1=1719616&r2=1719617&view=diff ============================================================================== --- aurora/site/publish/documentation/latest/test-resource-generation/index.html (original) +++ aurora/site/publish/documentation/latest/test-resource-generation/index.html Sat Dec 12 01:46:48 2015 @@ -21,12 +21,11 @@ </script> </head> <body> - <div class="container-fluid section-header"> <div class="container"> <div class="nav nav-bar"> <a href="/"><img src="/assets/img/aurora_logo_dkbkg.svg" width="300" alt="Transparent Apache Aurora logo with dark background"/></a> - <ul class="nav navbar-nav navbar-right"> + <ul class="nav navbar-nav navbar-right"> <li><a href="/documentation/latest/">Documentation</a></li> <li><a href="/community/">Community</a></li> <li><a href="/downloads/">Downloads</a></li> @@ -34,7 +33,8 @@ </ul> </div> </div> -</div> +</div> + <div class="container-fluid"> <div class="container content"> <div class="col-md-12 documentation"> @@ -46,9 +46,8 @@ <p>The Aurora source repository and distributions contain several <a href="../src/test/resources/org/apache/thermos/root/checkpoints">binary files</a> to qualify the backwards-compatibility of thermos with checkpoint data. Since -thermos persists state to disk, to be read by other components (the GC executor -and the thermos observer), it is important that we have tests that prevent -regressions affecting the ability to parse previously-written data.</p> +thermos persists state to disk, to be read by the thermos observer), it is important that we have +tests that prevent regressions affecting the ability to parse previously-written data.</p> <h2 id="generating-test-files">Generating test files</h2> @@ -66,11 +65,11 @@ accomplished by writing and running a <a href="/documentation/latest/configuration-reference/">job configuration</a> that exercises the feature, and copying the checkpoint file from the sandbox directory, by default this is <code>/var/run/thermos/checkpoints/<aurora task id></code>.</p> + </div> </div> </div> - <div class="container-fluid section-footer buffer"> <div class="container"> <div class="row"> @@ -95,5 +94,6 @@ copying the checkpoint file from the san </div> </div> </div> + </body> </html> \ No newline at end of file Modified: aurora/site/publish/documentation/latest/thrift-deprecation/index.html URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/latest/thrift-deprecation/index.html?rev=1719617&r1=1719616&r2=1719617&view=diff ============================================================================== --- aurora/site/publish/documentation/latest/thrift-deprecation/index.html (original) +++ aurora/site/publish/documentation/latest/thrift-deprecation/index.html Sat Dec 12 01:46:48 2015 @@ -21,12 +21,11 @@ </script> </head> <body> - <div class="container-fluid section-header"> <div class="container"> <div class="nav nav-bar"> <a href="/"><img src="/assets/img/aurora_logo_dkbkg.svg" width="300" alt="Transparent Apache Aurora logo with dark background"/></a> - <ul class="nav navbar-nav navbar-right"> + <ul class="nav navbar-nav navbar-right"> <li><a href="/documentation/latest/">Documentation</a></li> <li><a href="/community/">Community</a></li> <li><a href="/downloads/">Downloads</a></li> @@ -34,7 +33,8 @@ </ul> </div> </div> -</div> +</div> + <div class="container-fluid"> <div class="container content"> <div class="col-md-12 documentation"> @@ -94,11 +94,11 @@ See <a href="../src/main/java/org/apache confidence that you change is backwards compatible. It’s easy to simulate different client/scheduler versions by playing with <code>aurorabuild</code> command. See <a href="/documentation/latest/vagrant/">this document</a> for more.</p> + </div> </div> </div> - <div class="container-fluid section-footer buffer"> <div class="container"> <div class="row"> @@ -123,5 +123,6 @@ for more.</p> </div> </div> </div> + </body> </html> \ No newline at end of file Modified: aurora/site/publish/documentation/latest/tutorial/index.html URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/latest/tutorial/index.html?rev=1719617&r1=1719616&r2=1719617&view=diff ============================================================================== --- aurora/site/publish/documentation/latest/tutorial/index.html (original) +++ aurora/site/publish/documentation/latest/tutorial/index.html Sat Dec 12 01:46:48 2015 @@ -21,12 +21,11 @@ </script> </head> <body> - <div class="container-fluid section-header"> <div class="container"> <div class="nav nav-bar"> <a href="/"><img src="/assets/img/aurora_logo_dkbkg.svg" width="300" alt="Transparent Apache Aurora logo with dark background"/></a> - <ul class="nav navbar-nav navbar-right"> + <ul class="nav navbar-nav navbar-right"> <li><a href="/documentation/latest/">Documentation</a></li> <li><a href="/community/">Community</a></li> <li><a href="/downloads/">Downloads</a></li> @@ -34,7 +33,8 @@ </ul> </div> </div> -</div> +</div> + <div class="container-fluid"> <div class="container content"> <div class="col-md-12 documentation"> @@ -89,21 +89,22 @@ this directory is the same as <code>/vag <!-- NOTE: If you are changing this file, be sure to also update examples/vagrant/test_tutorial.sh. --> -<pre class="highlight python"><span class="kn">import</span> <span class="nn">sys</span> +<pre class="highlight python"><code><span class="kn">import</span> <span class="nn">sys</span> <span class="kn">import</span> <span class="nn">time</span> <span class="k">def</span> <span class="nf">main</span><span class="p">(</span><span class="n">argv</span><span class="p">):</span> <span class="n">SLEEP_DELAY</span> <span class="o">=</span> <span class="mi">10</span> <span class="c"># Python ninjas - ignore this blatant bug.</span> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="n">xrang</span><span class="p">(</span><span class="mi">100</span><span class="p">):</span> - <span class="k">print</span><span class="p">(</span><span class="s">"Hello world! The time is now: </span><span class="si">%</span><span class="s">s. Sleeping for </span><span class="si">%</span><span class="s">d secs"</span> <span class="o">%</span> <span class="p">(</span> + <span class="k">print</span><span class="p">(</span><span class="s">"Hello world! The time is now: </span><span class="si">%</span><span class="s">s. Sleeping for </span><span class="si">%</span><span class="s">d secs"</span> <span class="o">%</span> <span class="p">(</span> <span class="n">time</span><span class="o">.</span><span class="n">asctime</span><span class="p">(),</span> <span class="n">SLEEP_DELAY</span><span class="p">))</span> <span class="n">sys</span><span class="o">.</span><span class="n">stdout</span><span class="o">.</span><span class="n">flush</span><span class="p">()</span> <span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="n">SLEEP_DELAY</span><span class="p">)</span> -<span class="k">if</span> <span class="n">__name__</span> <span class="o">==</span> <span class="s">"__main__"</span><span class="p">:</span> +<span class="k">if</span> <span class="n">__name__</span> <span class="o">==</span> <span class="s">"__main__"</span><span class="p">:</span> <span class="n">main</span><span class="p">(</span><span class="n">sys</span><span class="o">.</span><span class="n">argv</span><span class="p">)</span> -</pre> +</code></pre> + <h2 id="aurora-configuration">Aurora Configuration</h2> <p>Once we have our script/program, we need to create a <em>configuration @@ -112,24 +113,24 @@ code in the file <code>hello_world.auror <!-- NOTE: If you are changing this file, be sure to also update examples/vagrant/test_tutorial.sh. --> -<pre class="highlight python"><span class="n">pkg_path</span> <span class="o">=</span> <span class="s">'/vagrant/hello_world.py'</span> +<pre class="highlight python"><code><span class="n">pkg_path</span> <span class="o">=</span> <span class="s">'/vagrant/hello_world.py'</span> <span class="c"># we use a trick here to make the configuration change with</span> <span class="c"># the contents of the file, for simplicity. in a normal setting, packages would be</span> <span class="c"># versioned, and the version number would be changed in the configuration.</span> <span class="kn">import</span> <span class="nn">hashlib</span> -<span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="n">pkg_path</span><span class="p">,</span> <span class="s">'rb'</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span> - <span class="n">pkg_checksum</span> <span class="o">=</span> <span class="nb">hash</span><span class="n">lib</span><span class="o">.</span><span class="n">md5</span><span class="p">(</span><span class="n">f</span><span class="o">.</span><span class="n">read</span><span class="p">())</span><span class="o">.</span><span class="nb">hex</span><span class="n">digest</span><span class="p">()</span> +<span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="n">pkg_path</span><span class="p">,</span> <span class="s">'rb'</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span> + <span class="n">pkg_checksum</span> <span class="o">=</span> <span class="n">hashlib</span><span class="o">.</span><span class="n">md5</span><span class="p">(</span><span class="n">f</span><span class="o">.</span><span class="n">read</span><span class="p">())</span><span class="o">.</span><span class="n">hexdigest</span><span class="p">()</span> <span class="c"># copy hello_world.py into the local sandbox</span> <span class="n">install</span> <span class="o">=</span> <span class="n">Process</span><span class="p">(</span> - <span class="n">name</span> <span class="o">=</span> <span class="s">'fetch_package'</span><span class="p">,</span> - <span class="n">cmdline</span> <span class="o">=</span> <span class="s">'cp </span><span class="si">%</span><span class="s">s . && echo </span><span class="si">%</span><span class="s">s && chmod +x hello_world.py'</span> <span class="o">%</span> <span class="p">(</span><span class="n">pkg_path</span><span class="p">,</span> <span class="n">pkg_checksum</span><span class="p">))</span> + <span class="n">name</span> <span class="o">=</span> <span class="s">'fetch_package'</span><span class="p">,</span> + <span class="n">cmdline</span> <span class="o">=</span> <span class="s">'cp </span><span class="si">%</span><span class="s">s . && echo </span><span class="si">%</span><span class="s">s && chmod +x hello_world.py'</span> <span class="o">%</span> <span class="p">(</span><span class="n">pkg_path</span><span class="p">,</span> <span class="n">pkg_checksum</span><span class="p">))</span> <span class="c"># run the script</span> <span class="n">hello_world</span> <span class="o">=</span> <span class="n">Process</span><span class="p">(</span> - <span class="n">name</span> <span class="o">=</span> <span class="s">'hello_world'</span><span class="p">,</span> - <span class="n">cmdline</span> <span class="o">=</span> <span class="s">'python hello_world.py'</span><span class="p">)</span> + <span class="n">name</span> <span class="o">=</span> <span class="s">'hello_world'</span><span class="p">,</span> + <span class="n">cmdline</span> <span class="o">=</span> <span class="s">'python hello_world.py'</span><span class="p">)</span> <span class="c"># describe the task</span> <span class="n">hello_world_task</span> <span class="o">=</span> <span class="n">SequentialTask</span><span class="p">(</span> @@ -137,19 +138,20 @@ code in the file <code>hello_world.auror <span class="n">resources</span> <span class="o">=</span> <span class="n">Resources</span><span class="p">(</span><span class="n">cpu</span> <span class="o">=</span> <span class="mi">1</span><span class="p">,</span> <span class="n">ram</span> <span class="o">=</span> <span class="mi">1</span><span class="o">*</span><span class="n">MB</span><span class="p">,</span> <span class="n">disk</span><span class="o">=</span><span class="mi">8</span><span class="o">*</span><span class="n">MB</span><span class="p">))</span> <span class="n">jobs</span> <span class="o">=</span> <span class="p">[</span> - <span class="n">Service</span><span class="p">(</span><span class="n">cluster</span> <span class="o">=</span> <span class="s">'devcluster'</span><span class="p">,</span> - <span class="n">environment</span> <span class="o">=</span> <span class="s">'devel'</span><span class="p">,</span> - <span class="n">role</span> <span class="o">=</span> <span class="s">'www-data'</span><span class="p">,</span> - <span class="n">name</span> <span class="o">=</span> <span class="s">'hello_world'</span><span class="p">,</span> + <span class="n">Service</span><span class="p">(</span><span class="n">cluster</span> <span class="o">=</span> <span class="s">'devcluster'</span><span class="p">,</span> + <span class="n">environment</span> <span class="o">=</span> <span class="s">'devel'</span><span class="p">,</span> + <span class="n">role</span> <span class="o">=</span> <span class="s">'www-data'</span><span class="p">,</span> + <span class="n">name</span> <span class="o">=</span> <span class="s">'hello_world'</span><span class="p">,</span> <span class="n">task</span> <span class="o">=</span> <span class="n">hello_world_task</span><span class="p">)</span> <span class="p">]</span> -</pre> +</code></pre> + <p>For more about Aurora configuration files, see the <a href="/documentation/latest/configuration-tutorial/">Configuration Tutorial</a> and the <a href="/documentation/latest/configuration-reference/">Aurora + Thermos Reference</a> (preferably after finishing this tutorial).</p> -<h2 id="what-39-s-going-on-in-that-configuration-file-">What’s Going On In That Configuration File?</h2> +<h2 id="what-39-s-going-on-in-that-configuration-file">What’s Going On In That Configuration File?</h2> <p>More than you might think.</p> @@ -182,19 +184,22 @@ identical, the job keys identify the sam <p><code>/etc/aurora/clusters.json</code> within the Aurora scheduler has the available cluster names. For Vagrant, from the top-level of your Aurora repository clone, do:</p> -<pre class="highlight text">$ vagrant ssh -</pre> +<pre class="highlight plaintext"><code>$ vagrant ssh +</code></pre> + <p>Followed by:</p> -<pre class="highlight text">vagrant@precise64:~$ cat /etc/aurora/clusters.json -</pre> +<pre class="highlight plaintext"><code>vagrant@precise64:~$ cat /etc/aurora/clusters.json +</code></pre> + <p>You’ll see something like:</p> -<pre class="highlight javascript"><span class="p">[{</span> - <span class="s2">"name"</span><span class="err">:</span> <span class="s2">"devcluster"</span><span class="p">,</span> - <span class="s2">"zk"</span><span class="err">:</span> <span class="s2">"192.168.33.7"</span><span class="p">,</span> - <span class="s2">"scheduler_zk_path"</span><span class="err">:</span> <span class="s2">"/aurora/scheduler"</span><span class="p">,</span> - <span class="s2">"auth_mechanism"</span><span class="err">:</span> <span class="s2">"UNAUTHENTICATED"</span> +<pre class="highlight javascript"><code><span class="p">[{</span> + <span class="s2">"name"</span><span class="p">:</span> <span class="s2">"devcluster"</span><span class="p">,</span> + <span class="s2">"zk"</span><span class="p">:</span> <span class="s2">"192.168.33.7"</span><span class="p">,</span> + <span class="s2">"scheduler_zk_path"</span><span class="p">:</span> <span class="s2">"/aurora/scheduler"</span><span class="p">,</span> + <span class="s2">"auth_mechanism"</span><span class="p">:</span> <span class="s2">"UNAUTHENTICATED"</span> <span class="p">}]</span> -</pre> +</code></pre> + <p>Use a <code>name</code> value for your job key’s cluster value.</p> <p>Role names are user accounts existing on the slave machines. If you don’t know what accounts @@ -204,13 +209,15 @@ are available, contact your sysadmin.</p <p>The Aurora Client command that actually runs our Job is <code>aurora job create</code>. It creates a Job as specified by its job key and configuration file arguments and runs it.</p> -<pre class="highlight text">aurora job create <cluster>/<role>/<environment>/<jobname> <config_file> -</pre> +<pre class="highlight plaintext"><code>aurora job create <cluster>/<role>/<environment>/<jobname> <config_file> +</code></pre> + <p>Or for our example:</p> -<pre class="highlight text">aurora job create devcluster/www-data/devel/hello_world /vagrant/hello_world.aurora -</pre> +<pre class="highlight plaintext"><code>aurora job create devcluster/www-data/devel/hello_world /vagrant/hello_world.aurora +</code></pre> + <p>This returns:</p> -<pre class="highlight text">$ vagrant ssh +<pre class="highlight plaintext"><code>$ vagrant ssh Welcome to Ubuntu 12.04 LTS (GNU/Linux 3.2.0-23-generic x86_64) * Documentation: https://help.ubuntu.com/ @@ -222,7 +229,8 @@ vagrant@precise64:~$ aurora job create d INFO] Response from scheduler: OK (message: 1 new tasks pending for job www-data/devel/hello_world) INFO] Job url: http://precise64:8081/scheduler/www-data/devel/hello_world -</pre> +</code></pre> + <h2 id="watching-the-job-run">Watching the Job Run</h2> <p>Now that our job is running, let’s see what it’s doing. Access the @@ -258,8 +266,9 @@ to <code>stderr</code> on the failed <co <p>It looks like we made a typo in our Python script. We wanted <code>xrange</code>, not <code>xrang</code>. Edit the <code>hello_world.py</code> script to use the correct function and we will try again.</p> -<pre class="highlight text">aurora job update devcluster/www-data/devel/hello_world /vagrant/hello_world.aurora -</pre> +<pre class="highlight plaintext"><code>aurora job update devcluster/www-data/devel/hello_world /vagrant/hello_world.aurora +</code></pre> + <p>This time, the task comes up, we inspect the page, and see that the <code>hello_world</code> process is running.</p> @@ -273,12 +282,13 @@ output:</p> <h2 id="cleanup">Cleanup</h2> <p>Now that we’re done, we kill the job using the Aurora client:</p> -<pre class="highlight text">vagrant@precise64:~$ aurora job killall devcluster/www-data/devel/hello_world +<pre class="highlight plaintext"><code>vagrant@precise64:~$ aurora job killall devcluster/www-data/devel/hello_world INFO] Killing tasks for job: devcluster/www-data/devel/hello_world INFO] Response from scheduler: OK (message: Tasks killed.) INFO] Job url: http://precise64:8081/scheduler/www-data/devel/hello_world vagrant@precise64:~$ -</pre> +</code></pre> + <p>The job page now shows the <code>hello_world</code> tasks as completed.</p> <p><img alt="Killed Task page" src="../images/killedtask.png" /></p> @@ -296,11 +306,11 @@ Thermos work “under the hood&rdquo <li>Explore the Aurora Client - use <code>aurora -h</code>, and read the <a href="/documentation/latest/client-commands/">Aurora Client Commands</a> document.</li> </ul> + </div> </div> </div> - <div class="container-fluid section-footer buffer"> <div class="container"> <div class="row"> @@ -325,5 +335,6 @@ Thermos work “under the hood&rdquo </div> </div> </div> + </body> </html> \ No newline at end of file Modified: aurora/site/publish/documentation/latest/user-guide/index.html URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/latest/user-guide/index.html?rev=1719617&r1=1719616&r2=1719617&view=diff ============================================================================== --- aurora/site/publish/documentation/latest/user-guide/index.html (original) +++ aurora/site/publish/documentation/latest/user-guide/index.html Sat Dec 12 01:46:48 2015 @@ -21,12 +21,11 @@ </script> </head> <body> - <div class="container-fluid section-header"> <div class="container"> <div class="nav nav-bar"> <a href="/"><img src="/assets/img/aurora_logo_dkbkg.svg" width="300" alt="Transparent Apache Aurora logo with dark background"/></a> - <ul class="nav navbar-nav navbar-right"> + <ul class="nav navbar-nav navbar-right"> <li><a href="/documentation/latest/">Documentation</a></li> <li><a href="/community/">Community</a></li> <li><a href="/downloads/">Downloads</a></li> @@ -34,7 +33,8 @@ </ul> </div> </div> -</div> +</div> + <div class="container-fluid"> <div class="container content"> <div class="col-md-12 documentation"> @@ -244,14 +244,16 @@ from the point where the update failed. <p>The Executor implements a protocol for rudimentary control of a task via HTTP. Tasks subscribe for this protocol by declaring a port named <code>health</code>. Take for example this configuration snippet:</p> -<pre class="highlight text">nginx = Process( - name = 'nginx', - cmdline = './run_nginx.sh -port {{thermos.ports[http]}}') -</pre> +<pre class="highlight plaintext"><code>nginx = Process( + name = 'nginx', + cmdline = './run_nginx.sh -port {{thermos.ports[http]}}') +</code></pre> + <p>When this Process is included in a job, the job will be allocated a port, and the command line will be replaced with something like:</p> -<pre class="highlight text">./run_nginx.sh -port 42816 -</pre> +<pre class="highlight plaintext"><code>./run_nginx.sh -port 42816 +</code></pre> + <p>Where 42816 happens to be the allocated. port. Typically, the Executor monitors Processes within a task only by liveness of the forked process. However, when a <code>health</code> port was allocated, it will also send periodic HTTP health checks. A task requesting a <code>health</code> port must handle the following @@ -398,18 +400,20 @@ about the Aurora Client.</p> <p>Part of the output from creating a new Job is a URL for the Job’s scheduler UI page.</p> <p>For example:</p> -<pre class="highlight text"> vagrant@precise64:~$ aurora job create devcluster/www-data/prod/hello \ +<pre class="highlight plaintext"><code> vagrant@precise64:~$ aurora job create devcluster/www-data/prod/hello \ /vagrant/examples/jobs/hello_world.aurora INFO] Creating job hello INFO] Response from scheduler: OK (message: 1 new tasks pending for job www-data/prod/hello) INFO] Job url: http://precise64:8081/scheduler/www-data/prod/hello -</pre> +</code></pre> + <p>The “Job url” goes to the Job’s scheduler UI page. To go to the overall scheduler UI page, stop at the “scheduler” part of the URL, in this case, <code>http://precise64:8081/scheduler</code></p> <p>You can also reach the scheduler UI page via the Client command <code>aurora job open</code>:</p> -<pre class="highlight text"> aurora job open [<cluster>[/<role>[/<env>/<job_name>]]] -</pre> +<pre class="highlight plaintext"><code> aurora job open [<cluster>[/<role>[/<env>/<job_name>]]] +</code></pre> + <p>If only the cluster is specified, it goes directly to that cluster’s scheduler main page. If the role is specified, it goes to the top-level role page. If the full job key is specified, it goes directly to the job page where you can inspect individual tasks.</p> @@ -423,11 +427,11 @@ about the Aurora Client.</p> </ul> <p>See <a href="/documentation/latest/client-commands/">client commands</a>.</p> + </div> </div> </div> - <div class="container-fluid section-footer buffer"> <div class="container"> <div class="row"> @@ -452,5 +456,6 @@ about the Aurora Client.</p> </div> </div> </div> + </body> </html> \ No newline at end of file
