Repository: incubator-airflow-site Updated Branches: refs/heads/asf-site 7d4d76286 -> 1f06fa0e0
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/1f06fa0e/security.html ---------------------------------------------------------------------- diff --git a/security.html b/security.html index 6bf00b5..fa4fa71 100644 --- a/security.html +++ b/security.html @@ -206,6 +206,13 @@ SSH tunnels.</p> <p>It is however possible to switch on authentication by either using one of the supplied backends or creating your own.</p> <p>Be sure to checkout <a class="reference internal" href="api.html"><span class="doc">Experimental Rest API</span></a> for securing the API.</p> +<div class="admonition note"> +<p class="first admonition-title">Note</p> +<p class="last">Airflow uses the config parser of Python. This config parser interpolates +â%â-signs. Make sure escape any <code class="docutils literal notranslate"><span class="pre">%</span></code> signs in your config file (but not +environment variables) as <code class="docutils literal notranslate"><span class="pre">%%</span></code>, otherwise Airflow might leak these +passwords on a config parser exception to a log.</p> +</div> <div class="section" id="web-authentication"> <h2>Web Authentication<a class="headerlink" href="#web-authentication" title="Permalink to this headline">¶</a></h2> <div class="section" id="password"> @@ -244,8 +251,7 @@ Type <span class="s2">"help"</span>, <span class="s2">"copyright& <div class="section" id="ldap"> <h3>LDAP<a class="headerlink" href="#ldap" title="Permalink to this headline">¶</a></h3> <p>To turn on LDAP authentication configure your <code class="docutils literal notranslate"><span class="pre">airflow.cfg</span></code> as follows. Please note that the example uses -an encrypted connection to the ldap server as you probably do not want passwords be readable on the network level. -It is however possible to configure without encryption if you really want to.</p> +an encrypted connection to the ldap server as we do not want passwords be readable on the network level.</p> <p>Additionally, if you are using Active Directory, and are not explicitly specifying an OU that your users are in, you will need to change <code class="docutils literal notranslate"><span class="pre">search_scope</span></code> to âSUBTREEâ.</p> <p>Valid search_scope options can be found in the <a class="reference external" href="http://ldap3.readthedocs.org/searches.html?highlight=search_scope";>ldap3 Documentation</a></p> @@ -439,7 +445,7 @@ backend. In order to setup an application:</p> <div class="section" id="google-authentication"> <h3>Google Authentication<a class="headerlink" href="#google-authentication" title="Permalink to this headline">¶</a></h3> <p>The Google authentication backend can be used to authenticate users -against Google using OAuth2. You must specify the email domains to restrict +against Google using OAuth2. You must specify the domains to restrict login, separated with a comma, to only members of those domains.</p> <div class="highlight-bash notranslate"><div class="highlight"><pre><span></span><span class="o">[</span>webserver<span class="o">]</span> <span class="nv">authenticate</span> <span class="o">=</span> True @@ -488,10 +494,10 @@ standard port 443, youâll need to configure that too. Be aware that super user <p>Enable CeleryExecutor with SSL. Ensure you properly generate client and server certs and keys.</p> <div class="highlight-bash notranslate"><div class="highlight"><pre><span></span><span class="o">[</span>celery<span class="o">]</span> -<span class="nv">CELERY_SSL_ACTIVE</span> <span class="o">=</span> True -<span class="nv">CELERY_SSL_KEY</span> <span class="o">=</span> <path to key> -<span class="nv">CELERY_SSL_CERT</span> <span class="o">=</span> <path to cert> -<span class="nv">CELERY_SSL_CACERT</span> <span class="o">=</span> <path to cacert> +<span class="nv">ssl_active</span> <span class="o">=</span> True +<span class="nv">ssl_key</span> <span class="o">=</span> <path to key> +<span class="nv">ssl_cert</span> <span class="o">=</span> <path to cert> +<span class="nv">ssl_cacert</span> <span class="o">=</span> <path to cacert> </pre></div> </div> </div> @@ -560,20 +566,13 @@ not set.</p> - <script type="text/javascript"> - var DOCUMENTATION_OPTIONS = { - URL_ROOT:'./', - VERSION:'', - LANGUAGE:'None', - COLLAPSE_INDEX:false, - FILE_SUFFIX:'.html', - HAS_SOURCE: true, - SOURCELINK_SUFFIX: '.txt' - }; - </script> - <script type="text/javascript" src="_static/jquery.js"></script> - <script type="text/javascript" src="_static/underscore.js"></script> - <script type="text/javascript" src="_static/doctools.js"></script> + + + <script type="text/javascript" id="documentation_options" data-url_root="./" src="_static/documentation_options.js"></script> + <script type="text/javascript" src="_static/jquery.js"></script> + <script type="text/javascript" src="_static/underscore.js"></script> + <script type="text/javascript" src="_static/doctools.js"></script> + http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/1f06fa0e/start.html ---------------------------------------------------------------------- diff --git a/start.html b/start.html index b5fdcdb..1846ed2 100644 --- a/start.html +++ b/start.html @@ -253,20 +253,13 @@ airflow backfill example_bash_operator -s <span class="m">2015</span>-01-01 -e < - <script type="text/javascript"> - var DOCUMENTATION_OPTIONS = { - URL_ROOT:'./', - VERSION:'', - LANGUAGE:'None', - COLLAPSE_INDEX:false, - FILE_SUFFIX:'.html', - HAS_SOURCE: true, - SOURCELINK_SUFFIX: '.txt' - }; - </script> - <script type="text/javascript" src="_static/jquery.js"></script> - <script type="text/javascript" src="_static/underscore.js"></script> - <script type="text/javascript" src="_static/doctools.js"></script> + + + <script type="text/javascript" id="documentation_options" data-url_root="./" src="_static/documentation_options.js"></script> + <script type="text/javascript" src="_static/jquery.js"></script> + <script type="text/javascript" src="_static/underscore.js"></script> + <script type="text/javascript" src="_static/doctools.js"></script> + http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/1f06fa0e/timezone.html ---------------------------------------------------------------------- diff --git a/timezone.html b/timezone.html index 587f7d6..7fc2526 100644 --- a/timezone.html +++ b/timezone.html @@ -266,6 +266,10 @@ recommended to use <cite>pendulum</cite> for this, but <cite>pytz</cite> (to be <span class="nb">print</span><span class="p">(</span><span class="n">dag</span><span class="o">.</span><span class="n">timezone</span><span class="p">)</span> <span class="c1"># <Timezone [Europe/Amsterdam]></span> </pre></div> </div> +<p>Please note that while it is possible to set a <cite>start_date</cite> and <cite>end_date</cite> for Tasks always the DAG timezone +or global timezone (in that order) will be used to calculate the next execution date. Upon first encounter +the start date or end date will be converted to UTC using the timezone associated with start_date or end_date, +then for calculations this timezone information will be disregarded.</p> <div class="section" id="templates"> <h3>Templates<a class="headerlink" href="#templates" title="Permalink to this headline">¶</a></h3> <p>Airflow returns time zone aware datetimes in templates, but does not convert them to local time so they remain in UTC. @@ -331,20 +335,13 @@ be taken into account.</p> - <script type="text/javascript"> - var DOCUMENTATION_OPTIONS = { - URL_ROOT:'./', - VERSION:'', - LANGUAGE:'None', - COLLAPSE_INDEX:false, - FILE_SUFFIX:'.html', - HAS_SOURCE: true, - SOURCELINK_SUFFIX: '.txt' - }; - </script> - <script type="text/javascript" src="_static/jquery.js"></script> - <script type="text/javascript" src="_static/underscore.js"></script> - <script type="text/javascript" src="_static/doctools.js"></script> + + + <script type="text/javascript" id="documentation_options" data-url_root="./" src="_static/documentation_options.js"></script> + <script type="text/javascript" src="_static/jquery.js"></script> + <script type="text/javascript" src="_static/underscore.js"></script> + <script type="text/javascript" src="_static/doctools.js"></script> + http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/1f06fa0e/tutorial.html ---------------------------------------------------------------------- diff --git a/tutorial.html b/tutorial.html index b3f4395..46ab267 100644 --- a/tutorial.html +++ b/tutorial.html @@ -213,7 +213,7 @@ complicated, a line by line explanation follows below.</p> <span class="c1"># 'end_date': datetime(2016, 1, 1),</span> <span class="p">}</span> -<span class="n">dag</span> <span class="o">=</span> <span class="n">DAG</span><span class="p">(</span><span class="s1">'tutorial'</span><span class="p">,</span> <span class="n">default_args</span><span class="o">=</span><span class="n">default_args</span><span class="p">)</span> +<span class="n">dag</span> <span class="o">=</span> <span class="n">DAG</span><span class="p">(</span><span class="s1">'tutorial'</span><span class="p">,</span> <span class="n">default_args</span><span class="o">=</span><span class="n">default_args</span><span class="p">,</span> <span class="n">schedule_interval</span><span class="o">=</span><span class="n">timedelta</span><span class="p">(</span><span class="n">days</span><span class="o">=</span><span class="mi">1</span><span class="p">))</span> <span class="c1"># t1, t2 and t3 are examples of tasks created by instantiating operators</span> <span class="n">t1</span> <span class="o">=</span> <span class="n">BashOperator</span><span class="p">(</span> @@ -311,7 +311,7 @@ that defines the <code class="docutils literal notranslate"><span class="pre">da We also pass the default argument dictionary that we just defined and define a <code class="docutils literal notranslate"><span class="pre">schedule_interval</span></code> of 1 day for the DAG.</p> <div class="code python highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">dag</span> <span class="o">=</span> <span class="n">DAG</span><span class="p">(</span> - <span class="s1">'tutorial'</span><span class="p">,</span> <span class="n">default_args</span><span class="o">=</span><span class="n">default_args</span><span class="p">,</span> <span class="n">schedule_interval</span><span class="o">=</span><span class="n">timedelta</span><span class="p">(</span><span class="mi">1</span><span class="p">))</span> + <span class="s1">'tutorial'</span><span class="p">,</span> <span class="n">default_args</span><span class="o">=</span><span class="n">default_args</span><span class="p">,</span> <span class="n">schedule_interval</span><span class="o">=</span><span class="n">timedelta</span><span class="p">(</span><span class="n">days</span><span class="o">=</span><span class="mi">1</span><span class="p">))</span> </pre></div> </div> </div> @@ -403,20 +403,33 @@ in templates, make sure to read through the <a class="reference internal" href=" </div> <div class="section" id="setting-up-dependencies"> <h2>Setting up Dependencies<a class="headerlink" href="#setting-up-dependencies" title="Permalink to this headline">¶</a></h2> -<p>We have two simple tasks that do not depend on each other. Hereâs a few ways +<p>We have tasks <cite>t1</cite>, <cite>t2</cite> and <cite>t3</cite> that do not depend on each other. Hereâs a few ways you can define dependencies between them:</p> -<div class="code python highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">t2</span><span class="o">.</span><span class="n">set_upstream</span><span class="p">(</span><span class="n">t1</span><span class="p">)</span> +<div class="code python highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">t1</span><span class="o">.</span><span class="n">set_downstream</span><span class="p">(</span><span class="n">t2</span><span class="p">)</span> <span class="c1"># This means that t2 will depend on t1</span> -<span class="c1"># running successfully to run</span> -<span class="c1"># It is equivalent to</span> -<span class="c1"># t1.set_downstream(t2)</span> +<span class="c1"># running successfully to run.</span> +<span class="c1"># It is equivalent to:</span> +<span class="n">t2</span><span class="o">.</span><span class="n">set_upstream</span><span class="p">(</span><span class="n">t1</span><span class="p">)</span> -<span class="n">t3</span><span class="o">.</span><span class="n">set_upstream</span><span class="p">(</span><span class="n">t1</span><span class="p">)</span> +<span class="c1"># The bit shift operator can also be</span> +<span class="c1"># used to chain operations:</span> +<span class="n">t1</span> <span class="o">>></span> <span class="n">t2</span> + +<span class="c1"># And the upstream dependency with the</span> +<span class="c1"># bit shift operator:</span> +<span class="n">t2</span> <span class="o"><<</span> <span class="n">t1</span> -<span class="c1"># all of this is equivalent to</span> -<span class="c1"># dag.set_dependency('print_date', 'sleep')</span> -<span class="c1"># dag.set_dependency('print_date', 'templated')</span> +<span class="c1"># Chaining multiple dependencies becomes</span> +<span class="c1"># concise with the bit shift operator:</span> +<span class="n">t1</span> <span class="o">>></span> <span class="n">t2</span> <span class="o">>></span> <span class="n">t3</span> + +<span class="c1"># A list of tasks can also be set as</span> +<span class="c1"># dependencies. These operations</span> +<span class="c1"># all have the same effect:</span> +<span class="n">t1</span><span class="o">.</span><span class="n">set_downstream</span><span class="p">([</span><span class="n">t2</span><span class="p">,</span> <span class="n">t3</span><span class="p">])</span> +<span class="n">t1</span> <span class="o">>></span> <span class="p">[</span><span class="n">t2</span><span class="p">,</span> <span class="n">t3</span><span class="p">]</span> +<span class="p">[</span><span class="n">t2</span><span class="p">,</span> <span class="n">t3</span><span class="p">]</span> <span class="o"><<</span> <span class="n">t1</span> </pre></div> </div> <p>Note that when executing your script, Airflow will raise exceptions when @@ -428,8 +441,8 @@ than once.</p> <p>Alright, so we have a pretty basic DAG. At this point your code should look something like this:</p> <div class="code python highlight-default notranslate"><div class="highlight"><pre><span></span><span class="sd">"""</span> -<span class="sd">Code that goes along with the Airflow located at:</span> -<span class="sd">http://airflow.readthedocs.org/en/latest/tutorial.html</span> +<span class="sd">Code that goes along with the Airflow tutorial located at:</span> +<span class="sd">https://github.com/apache/incubator-airflow/blob/master/airflow/example_dags/tutorial.py</span> <span class="sd">"""</span> <span class="kn">from</span> <span class="nn">airflow</span> <span class="k">import</span> <span class="n">DAG</span> <span class="kn">from</span> <span class="nn">airflow.operators.bash_operator</span> <span class="k">import</span> <span class="n">BashOperator</span> @@ -452,7 +465,7 @@ something like this:</p> <span class="p">}</span> <span class="n">dag</span> <span class="o">=</span> <span class="n">DAG</span><span class="p">(</span> - <span class="s1">'tutorial'</span><span class="p">,</span> <span class="n">default_args</span><span class="o">=</span><span class="n">default_args</span><span class="p">,</span> <span class="n">schedule_interval</span><span class="o">=</span><span class="n">timedelta</span><span class="p">(</span><span class="mi">1</span><span class="p">))</span> + <span class="s1">'tutorial'</span><span class="p">,</span> <span class="n">default_args</span><span class="o">=</span><span class="n">default_args</span><span class="p">,</span> <span class="n">schedule_interval</span><span class="o">=</span><span class="n">timedelta</span><span class="p">(</span><span class="n">days</span><span class="o">=</span><span class="mi">1</span><span class="p">))</span> <span class="c1"># t1, t2 and t3 are examples of tasks created by instantiating operators</span> <span class="n">t1</span> <span class="o">=</span> <span class="n">BashOperator</span><span class="p">(</span> @@ -624,20 +637,13 @@ running against it should get it to get triggered and run every day.</p> - <script type="text/javascript"> - var DOCUMENTATION_OPTIONS = { - URL_ROOT:'./', - VERSION:'', - LANGUAGE:'None', - COLLAPSE_INDEX:false, - FILE_SUFFIX:'.html', - HAS_SOURCE: true, - SOURCELINK_SUFFIX: '.txt' - }; - </script> - <script type="text/javascript" src="_static/jquery.js"></script> - <script type="text/javascript" src="_static/underscore.js"></script> - <script type="text/javascript" src="_static/doctools.js"></script> + + + <script type="text/javascript" id="documentation_options" data-url_root="./" src="_static/documentation_options.js"></script> + <script type="text/javascript" src="_static/jquery.js"></script> + <script type="text/javascript" src="_static/underscore.js"></script> + <script type="text/javascript" src="_static/doctools.js"></script> + http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/1f06fa0e/ui.html ---------------------------------------------------------------------- diff --git a/ui.html b/ui.html index 7ebe7b9..e0ebf7e 100644 --- a/ui.html +++ b/ui.html @@ -174,7 +174,7 @@ <div class="section" id="ui-screenshots"> <h1>UI / Screenshots<a class="headerlink" href="#ui-screenshots" title="Permalink to this headline">¶</a></h1> -<p>The Airflow UI make it easy to monitor and troubleshoot your data pipelines. +<p>The Airflow UI makes it easy to monitor and troubleshoot your data pipelines. Hereâs a quick overview of some of the features and visualizations you can find in the Airflow UI.</p> <div class="section" id="dags-view"> @@ -288,20 +288,13 @@ that can take you to more detailed metadata, and perform some actions.</p> - <script type="text/javascript"> - var DOCUMENTATION_OPTIONS = { - URL_ROOT:'./', - VERSION:'', - LANGUAGE:'None', - COLLAPSE_INDEX:false, - FILE_SUFFIX:'.html', - HAS_SOURCE: true, - SOURCELINK_SUFFIX: '.txt' - }; - </script> - <script type="text/javascript" src="_static/jquery.js"></script> - <script type="text/javascript" src="_static/underscore.js"></script> - <script type="text/javascript" src="_static/doctools.js"></script> + + + <script type="text/javascript" id="documentation_options" data-url_root="./" src="_static/documentation_options.js"></script> + <script type="text/javascript" src="_static/jquery.js"></script> + <script type="text/javascript" src="_static/underscore.js"></script> + <script type="text/javascript" src="_static/doctools.js"></script> +
