http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/28a3eb60/concepts.html ---------------------------------------------------------------------- diff --git a/concepts.html b/concepts.html index a329ae8..16d995c 100644 --- a/concepts.html +++ b/concepts.html @@ -13,6 +13,8 @@ + + @@ -81,7 +83,10 @@ - <ul class="current"> + + + + <ul class="current"> <li class="toctree-l1"><a class="reference internal" href="project.html">Project</a></li> <li class="toctree-l1"><a class="reference internal" href="license.html">License</a></li> <li class="toctree-l1"><a class="reference internal" href="start.html">Quick Start</a></li> @@ -208,23 +213,23 @@ workflows.</p> <h2>Core Ideas<a class="headerlink" href="#core-ideas" title="Permalink to this headline">¶</a></h2> <div class="section" id="dags"> <h3>DAGs<a class="headerlink" href="#dags" title="Permalink to this headline">¶</a></h3> -<p>In Airflow, a <code class="docutils literal"><span class="pre">DAG</span></code> – or a Directed Acyclic Graph – is a collection of all +<p>In Airflow, a <code class="docutils literal"><span class="pre">DAG</span></code> â or a Directed Acyclic Graph â is a collection of all the tasks you want to run, organized in a way that reflects their relationships and dependencies.</p> <p>For example, a simple DAG could consist of three tasks: A, B, and C. It could say that A has to run successfully before B can run, but C can run anytime. It could say that task A times out after 5 minutes, and B can be restarted up to 5 times in case it fails. It might also say that the workflow will run every night -at 10pm, but shouldn’t start until a certain date.</p> +at 10pm, but shouldnât start until a certain date.</p> <p>In this way, a DAG describes <em>how</em> you want to carry out your workflow; but -notice that we haven’t said anything about <em>what</em> we actually want to do! A, B, +notice that we havenât said anything about <em>what</em> we actually want to do! A, B, and C could be anything. Maybe A prepares data for B to analyze while C sends an email. Or perhaps A monitors your location so B can open your garage door while -C turns on your house lights. The important thing is that the DAG isn’t +C turns on your house lights. The important thing is that the DAG isnât concerned with what its constituent tasks do; its job is to make sure that whatever they do happens at the right time, or in the right order, or with the right handling of any unexpected issues.</p> -<p>DAGs are defined in standard Python files that are placed in Airflow’s +<p>DAGs are defined in standard Python files that are placed in Airflowâs <code class="docutils literal"><span class="pre">DAG_FOLDER</span></code>. Airflow will execute the code in each file to dynamically build the <code class="docutils literal"><span class="pre">DAG</span></code> objects. You can have as many DAGs as you want, each describing an arbitrary number of tasks. In general, each one should correspond to a single @@ -245,7 +250,7 @@ scope.</p> </div> <p>Sometimes this can be put to good use. For example, a common pattern with <code class="docutils literal"><span class="pre">SubDagOperator</span></code> is to define the subdag inside a function so that Airflow -doesn’t try to load it as a standalone DAG.</p> +doesnât try to load it as a standalone DAG.</p> </div> <div class="section" id="default-arguments"> <h4>Default Arguments<a class="headerlink" href="#default-arguments" title="Permalink to this headline">¶</a></h4> @@ -278,13 +283,13 @@ any of its operators. This makes it easy to apply a common parameter to many ope <p>While DAGs describe <em>how</em> to run a workflow, <code class="docutils literal"><span class="pre">Operators</span></code> determine what actually gets done.</p> <p>An operator describes a single task in a workflow. Operators are usually (but -not always) atomic, meaning they can stand on their own and don’t need to share +not always) atomic, meaning they can stand on their own and donât need to share resources with any other operators. The DAG will make sure that operators run in the correct certain order; other than those dependencies, operators generally run independently. In fact, they may run on two completely different machines.</p> <p>This is a subtle but very important point: in general, if two operators need to share information, like a filename or small amount of data, you should consider -combining them into a single operator. If it absolutely can’t be avoided, +combining them into a single operator. If it absolutely canât be avoided, Airflow does have a feature for operator cross-communication called XCom that is described elsewhere in this document.</p> <p>Airflow provides operators for many common tasks, including:</p> @@ -293,14 +298,14 @@ described elsewhere in this document.</p> <li><code class="docutils literal"><span class="pre">PythonOperator</span></code> - calls an arbitrary Python function</li> <li><code class="docutils literal"><span class="pre">EmailOperator</span></code> - sends an email</li> <li><code class="docutils literal"><span class="pre">HTTPOperator</span></code> - sends an HTTP request</li> -<li><code class="docutils literal"><span class="pre">SqlOperator</span></code> - executes a SQL command</li> -<li><code class="docutils literal"><span class="pre">Sensor</span></code> - waits for a certain time, file, database row, S3 key, etc...</li> +<li><code class="docutils literal"><span class="pre">MySqlOperator</span></code>, <code class="docutils literal"><span class="pre">SqliteOperator</span></code>, <code class="docutils literal"><span class="pre">PostgresOperator</span></code>, <code class="docutils literal"><span class="pre">MsSqlOperator</span></code>, <code class="docutils literal"><span class="pre">OracleOperator</span></code>, <code class="docutils literal"><span class="pre">JdbcOperator</span></code>, etc. - executes a SQL command</li> +<li><code class="docutils literal"><span class="pre">Sensor</span></code> - waits for a certain time, file, database row, S3 key, etcâ¦</li> </ul> <p>In addition to these basic building blocks, there are many more specific operators: <code class="docutils literal"><span class="pre">DockerOperator</span></code>, <code class="docutils literal"><span class="pre">HiveOperator</span></code>, <code class="docutils literal"><span class="pre">S3FileTransferOperator</span></code>, -<code class="docutils literal"><span class="pre">PrestoToMysqlOperator</span></code>, <code class="docutils literal"><span class="pre">SlackOperator</span></code>... you get the idea!</p> +<code class="docutils literal"><span class="pre">PrestoToMysqlOperator</span></code>, <code class="docutils literal"><span class="pre">SlackOperator</span></code>⦠you get the idea!</p> <p>The <code class="docutils literal"><span class="pre">airflow/contrib/</span></code> directory contains yet more operators built by the -community. These operators aren’t always as complete or well-tested as those in +community. These operators arenât always as complete or well-tested as those in the main distribution, but allow users to more easily add new functionality to the platform.</p> <p>Operators are only loaded by Airflow if they are assigned to a DAG.</p> @@ -344,7 +349,7 @@ functionally equivalent:</p> <p>When using the bitshift to compose operators, the relationship is set in the direction that the bitshift operator points. For example, <code class="docutils literal"><span class="pre">op1</span> <span class="pre">>></span> <span class="pre">op2</span></code> means that <code class="docutils literal"><span class="pre">op1</span></code> runs first and <code class="docutils literal"><span class="pre">op2</span></code> runs second. Multiple operators can be -composed – keep in mind the chain is executed left-to-right and the rightmost +composed â keep in mind the chain is executed left-to-right and the rightmost object is always returned. For example:</p> <div class="code python highlight-default"><div class="highlight"><pre><span></span><span class="n">op1</span> <span class="o">>></span> <span class="n">op2</span> <span class="o">>></span> <span class="n">op3</span> <span class="o"><<</span> <span class="n">op4</span> </pre></div> @@ -367,8 +372,7 @@ object is always returned. For example:</p> <p>We can put this all together to build a simple pipeline:</p> <div class="code python highlight-default"><div class="highlight"><pre><span></span><span class="k">with</span> <span class="n">DAG</span><span class="p">(</span><span class="s1">'my_dag'</span><span class="p">,</span> <span class="n">start_date</span><span class="o">=</span><span class="n">datetime</span><span class="p">(</span><span class="mi">2016</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">1</span><span class="p">))</span> <span class="k">as</span> <span class="n">dag</span><span class="p">:</span> <span class="p">(</span> - <span class="n">dag</span> - <span class="o">>></span> <span class="n">DummyOperator</span><span class="p">(</span><span class="n">task_id</span><span class="o">=</span><span class="s1">'dummy_1'</span><span class="p">)</span> + <span class="n">DummyOperator</span><span class="p">(</span><span class="n">task_id</span><span class="o">=</span><span class="s1">'dummy_1'</span><span class="p">)</span> <span class="o">>></span> <span class="n">BashOperator</span><span class="p">(</span> <span class="n">task_id</span><span class="o">=</span><span class="s1">'bash_1'</span><span class="p">,</span> <span class="n">bash_command</span><span class="o">=</span><span class="s1">'echo "HELLO!"'</span><span class="p">)</span> @@ -382,7 +386,7 @@ object is always returned. For example:</p> </div> <div class="section" id="tasks"> <h3>Tasks<a class="headerlink" href="#tasks" title="Permalink to this headline">¶</a></h3> -<p>Once an operator is instantiated, it is referred to as a “task”. The +<p>Once an operator is instantiated, it is referred to as a âtaskâ. The instantiation defines specific values when calling the abstract operator, and the parameterized task becomes a node in a DAG.</p> </div> @@ -390,12 +394,12 @@ the parameterized task becomes a node in a DAG.</p> <h3>Task Instances<a class="headerlink" href="#task-instances" title="Permalink to this headline">¶</a></h3> <p>A task instance represents a specific run of a task and is characterized as the combination of a dag, a task, and a point in time. Task instances also have an -indicative state, which could be “running”, “success”, “failed”, “skipped”, “up -for retry”, etc.</p> +indicative state, which could be ârunningâ, âsuccessâ, âfailedâ, âskippedâ, âup +for retryâ, etc.</p> </div> <div class="section" id="workflows"> <h3>Workflows<a class="headerlink" href="#workflows" title="Permalink to this headline">¶</a></h3> -<p>You’re now familiar with the core building blocks of Airflow. +<p>Youâre now familiar with the core building blocks of Airflow. Some of the concepts may sound very similar, but the vocabulary can be conceptualized like this:</p> <ul class="simple"> @@ -457,8 +461,8 @@ gets prioritized accordingly.</p> reached, runnable tasks get queued and their state will show as such in the UI. As slots free up, queued tasks start running based on the <code class="docutils literal"><span class="pre">priority_weight</span></code> (of the task and its descendants).</p> -<p>Note that by default tasks aren’t assigned to any pool and their -execution parallelism is only limited to the executor’s setting.</p> +<p>Note that by default tasks arenât assigned to any pool and their +execution parallelism is only limited to the executorâs setting.</p> </div> <div class="section" id="connections"> <h3>Connections<a class="headerlink" href="#connections" title="Permalink to this headline">¶</a></h3> @@ -489,7 +493,7 @@ variable to be in a URI format (e.g. <p>When using the CeleryExecutor, the celery queues that tasks are sent to can be specified. <code class="docutils literal"><span class="pre">queue</span></code> is an attribute of BaseOperator, so any task can be assigned to any queue. The default queue for the environment -is defined in the <code class="docutils literal"><span class="pre">airflow.cfg</span></code>‘s <code class="docutils literal"><span class="pre">celery</span> <span class="pre">-></span> <span class="pre">default_queue</span></code>. This defines +is defined in the <code class="docutils literal"><span class="pre">airflow.cfg</span></code>âs <code class="docutils literal"><span class="pre">celery</span> <span class="pre">-></span> <span class="pre">default_queue</span></code>. This defines the queue that tasks get assigned to when not specified, as well as which queue Airflow workers listen to when started.</p> <p>Workers can listen to one or multiple queues of tasks. When a worker is @@ -505,16 +509,16 @@ itself because it needs a very specific environment and security rights).</p> <div class="section" id="xcoms"> <h3>XComs<a class="headerlink" href="#xcoms" title="Permalink to this headline">¶</a></h3> <p>XComs let tasks exchange messages, allowing more nuanced forms of control and -shared state. The name is an abbreviation of “cross-communication”. XComs are +shared state. The name is an abbreviation of âcross-communicationâ. XComs are principally defined by a key, value, and timestamp, but also track attributes like the task/DAG that created the XCom and when it should become visible. Any object that can be pickled can be used as an XCom value, so users should make sure to use objects of appropriate size.</p> -<p>XComs can be “pushed” (sent) or “pulled” (received). When a task pushes an +<p>XComs can be âpushedâ (sent) or âpulledâ (received). When a task pushes an XCom, it makes it generally available to other tasks. Tasks can push XComs at any time by calling the <code class="docutils literal"><span class="pre">xcom_push()</span></code> method. In addition, if a task returns -a value (either from its Operator’s <code class="docutils literal"><span class="pre">execute()</span></code> method, or from a -PythonOperator’s <code class="docutils literal"><span class="pre">python_callable</span></code> function), then an XCom containing that +a value (either from its Operatorâs <code class="docutils literal"><span class="pre">execute()</span></code> method, or from a +PythonOperatorâs <code class="docutils literal"><span class="pre">python_callable</span></code> function), then an XCom containing that value is automatically pushed.</p> <p>Tasks call <code class="docutils literal"><span class="pre">xcom_pull()</span></code> to retrieve XComs, optionally applying filters based on criteria like <code class="docutils literal"><span class="pre">key</span></code>, source <code class="docutils literal"><span class="pre">task_ids</span></code>, and source <code class="docutils literal"><span class="pre">dag_id</span></code>. By @@ -533,7 +537,7 @@ passed, then a correpsonding list of XCom values is returned.</p> <span class="n">value</span> <span class="o">=</span> <span class="n">context</span><span class="p">[</span><span class="s1">'task_instance'</span><span class="p">]</span><span class="o">.</span><span class="n">xcom_pull</span><span class="p">(</span><span class="n">task_ids</span><span class="o">=</span><span class="s1">'pushing_task'</span><span class="p">)</span> </pre></div> </div> -<p>It is also possible to pull XCom directly in a template, here’s an example +<p>It is also possible to pull XCom directly in a template, hereâs an example of what this may look like:</p> <div class="code sql highlight-default"><div class="highlight"><pre><span></span><span class="n">SELECT</span> <span class="o">*</span> <span class="n">FROM</span> <span class="p">{{</span> <span class="n">task_instance</span><span class="o">.</span><span class="n">xcom_pull</span><span class="p">(</span><span class="n">task_ids</span><span class="o">=</span><span class="s1">'foo'</span><span class="p">,</span> <span class="n">key</span><span class="o">=</span><span class="s1">'table_name'</span><span class="p">)</span> <span class="p">}}</span> </pre></div> @@ -575,9 +579,9 @@ directly downstream from the BranchPythonOperator task.</p> will invariably lead to block tasks that depend on their past successes. <code class="docutils literal"><span class="pre">skipped</span></code> states propagates where all directly upstream tasks are <code class="docutils literal"><span class="pre">skipped</span></code>.</p> -<p>If you want to skip some tasks, keep in mind that you can’t have an empty +<p>If you want to skip some tasks, keep in mind that you canât have an empty path, if so make a dummy task.</p> -<p>like this, the dummy task “branch_false” is skipped</p> +<p>like this, the dummy task âbranch_falseâ is skipped</p> <img alt="_images/branch_good.png" src="_images/branch_good.png" /> <p>Not like this, where the join task is skipped</p> <img alt="_images/branch_bad.png" src="_images/branch_bad.png" /> @@ -649,11 +653,11 @@ the tasks contained within the SubDAG:</p> <img alt="_images/subdag_zoom.png" src="_images/subdag_zoom.png" /> <p>Some other tips when using SubDAGs:</p> <ul class="simple"> -<li>by convention, a SubDAG’s <code class="docutils literal"><span class="pre">dag_id</span></code> should be prefixed by its parent and +<li>by convention, a SubDAGâs <code class="docutils literal"><span class="pre">dag_id</span></code> should be prefixed by its parent and a dot. As in <code class="docutils literal"><span class="pre">parent.child</span></code></li> <li>share arguments between the main DAG and the SubDAG by passing arguments to the SubDAG operator (as demonstrated above)</li> -<li>SubDAGs must have a schedule and be enabled. If the SubDAG’s schedule is +<li>SubDAGs must have a schedule and be enabled. If the SubDAGâs schedule is set to <code class="docutils literal"><span class="pre">None</span></code> or <code class="docutils literal"><span class="pre">@once</span></code>, the SubDAG will succeed without having done anything</li> <li>clearing a SubDagOperator also clears the state of the tasks within</li> @@ -685,8 +689,8 @@ directly upstream tasks have succeeded, Airflow allows for more complex dependency settings.</p> <p>All operators have a <code class="docutils literal"><span class="pre">trigger_rule</span></code> argument which defines the rule by which the generated task get triggered. The default value for <code class="docutils literal"><span class="pre">trigger_rule</span></code> is -<code class="docutils literal"><span class="pre">all_success</span></code> and can be defined as “trigger this task when all directly -upstream tasks have succeeded”. All other rules described here are based +<code class="docutils literal"><span class="pre">all_success</span></code> and can be defined as âtrigger this task when all directly +upstream tasks have succeededâ. All other rules described here are based on direct parent tasks and are values that can be passed to any operator while creating tasks:</p> <ul class="simple"> @@ -699,7 +703,7 @@ while creating tasks:</p> </ul> <p>Note that these can be used in conjunction with <code class="docutils literal"><span class="pre">depends_on_past</span></code> (boolean) that, when set to <code class="docutils literal"><span class="pre">True</span></code>, keeps a task from getting triggered if the -previous schedule for the task hasn’t succeeded.</p> +previous schedule for the task hasnât succeeded.</p> </div> <div class="section" id="latest-run-only"> <h3>Latest Run Only<a class="headerlink" href="#latest-run-only" title="Permalink to this headline">¶</a></h3> @@ -772,16 +776,16 @@ and <code class="docutils literal"><span class="pre">task2</span></code> has suc but sometimes unexpectedly.</p> <p>Zombie tasks are characterized by the absence of an heartbeat (emitted by the job periodically) and a <code class="docutils literal"><span class="pre">running</span></code> status -in the database. They can occur when a worker node can’t reach the database, +in the database. They can occur when a worker node canât reach the database, when Airflow processes are killed externally, or when a node gets rebooted -for instance. Zombie killing is performed periodically by the scheduler’s +for instance. Zombie killing is performed periodically by the schedulerâs process.</p> <p>Undead processes are characterized by the existence of a process and a matching -heartbeat, but Airflow isn’t aware of this task as <code class="docutils literal"><span class="pre">running</span></code> in the database. +heartbeat, but Airflow isnât aware of this task as <code class="docutils literal"><span class="pre">running</span></code> in the database. This mismatch typically occurs as the state of the database is altered, -most likely by deleting rows in the “Task Instances” view in the UI. +most likely by deleting rows in the âTask Instancesâ view in the UI. Tasks are instructed to verify their state as part of the heartbeat routine, -and terminate themselves upon figuring out that they are in this “undead” +and terminate themselves upon figuring out that they are in this âundeadâ state.</p> </div> <div class="section" id="cluster-policy"> @@ -792,10 +796,10 @@ attributes. It receives a single argument as a reference to task objects, and is expected to alter its attributes.</p> <p>For example, this function could apply a specific queue property when using a specific operator, or enforce a task timeout policy, making sure -that no tasks run for more than 48 hours. Here’s an example of what this +that no tasks run for more than 48 hours. Hereâs an example of what this may look like inside your <code class="docutils literal"><span class="pre">airflow_settings.py</span></code>:</p> <div class="code python highlight-default"><div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">policy</span><span class="p">(</span><span class="n">task</span><span class="p">):</span> - <span class="k">if</span> <span class="n">task</span><span class="o">.</span><span class="n">__class__</span><span class="o">.</span><span class="n">__name__</span> <span class="o">==</span> <span class="s1">'HivePartitionSensor'</span><span class="p">:</span> + <span class="k">if</span> <span class="n">task</span><span class="o">.</span><span class="vm">__class__</span><span class="o">.</span><span class="vm">__name__</span> <span class="o">==</span> <span class="s1">'HivePartitionSensor'</span><span class="p">:</span> <span class="n">task</span><span class="o">.</span><span class="n">queue</span> <span class="o">=</span> <span class="s2">"sensor_queue"</span> <span class="k">if</span> <span class="n">task</span><span class="o">.</span><span class="n">timeout</span> <span class="o">></span> <span class="n">timedelta</span><span class="p">(</span><span class="n">hours</span><span class="o">=</span><span class="mi">48</span><span class="p">):</span> <span class="n">task</span><span class="o">.</span><span class="n">timeout</span> <span class="o">=</span> <span class="n">timedelta</span><span class="p">(</span><span class="n">hours</span><span class="o">=</span><span class="mi">48</span><span class="p">)</span> @@ -804,8 +808,8 @@ may look like inside your <code class="docutils literal"><span class="pre">airfl </div> <div class="section" id="documentation-notes"> <h3>Documentation & Notes<a class="headerlink" href="#documentation-notes" title="Permalink to this headline">¶</a></h3> -<p>It’s possible to add documentation or notes to your dags & task objects that -become visible in the web interface (“Graph View” for dags, “Task Details” for +<p>Itâs possible to add documentation or notes to your dags & task objects that +become visible in the web interface (âGraph Viewâ for dags, âTask Detailsâ for tasks). There are a set of special task attributes that get rendered as rich content if defined:</p> <table border="1" class="docutils"> @@ -845,7 +849,7 @@ to the related tasks in Airflow.</p> <span class="sd">"""</span> <span class="n">dag</span> <span class="o">=</span> <span class="n">DAG</span><span class="p">(</span><span class="s1">'my_dag'</span><span class="p">,</span> <span class="n">default_args</span><span class="o">=</span><span class="n">default_args</span><span class="p">)</span> -<span class="n">dag</span><span class="o">.</span><span class="n">doc_md</span> <span class="o">=</span> <span class="n">__doc__</span> +<span class="n">dag</span><span class="o">.</span><span class="n">doc_md</span> <span class="o">=</span> <span class="vm">__doc__</span> <span class="n">t</span> <span class="o">=</span> <span class="n">BashOperator</span><span class="p">(</span><span class="s2">"foo"</span><span class="p">,</span> <span class="n">dag</span><span class="o">=</span><span class="n">dag</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">doc_md</span> <span class="o">=</span> <span class="s2">"""</span><span class="se">\</span> @@ -854,11 +858,11 @@ to the related tasks in Airflow.</p> <span class="s2">"""</span> </pre></div> </div> -<p>This content will get rendered as markdown respectively in the “Graph View” and -“Task Details” pages.</p> +<p>This content will get rendered as markdown respectively in the âGraph Viewâ and +âTask Detailsâ pages.</p> </div> <div class="section" id="jinja-templating"> -<h3>Jinja Templating<a class="headerlink" href="#jinja-templating" title="Permalink to this headline">¶</a></h3> +<span id="id1"></span><h3>Jinja Templating<a class="headerlink" href="#jinja-templating" title="Permalink to this headline">¶</a></h3> <p>Airflow leverages the power of <a class="reference external" href="http://jinja.pocoo.org/docs/dev/";>Jinja Templating</a> and this can be a powerful tool to use in combination with macros (see the <a class="reference internal" href="code.html#macros"><span class="std std-ref">Macros</span></a> section).</p> @@ -876,8 +880,9 @@ to a Bash script using the <code class="docutils literal"><span class="pre">Bash <p>Here, <code class="docutils literal"><span class="pre">{{</span> <span class="pre">ds</span> <span class="pre">}}</span></code> is a macro, and because the <code class="docutils literal"><span class="pre">env</span></code> parameter of the <code class="docutils literal"><span class="pre">BashOperator</span></code> is templated with Jinja, the execution date will be available as an environment variable named <code class="docutils literal"><span class="pre">EXECUTION_DATE</span></code> in your Bash script.</p> -<p>You can use Jinja templating with every parameter that is marked as “templated” -in the documentation.</p> +<p>You can use Jinja templating with every parameter that is marked as âtemplatedâ +in the documentation. Template substitution occurs just before the pre_execute +function of your operator is called.</p> </div> </div> <div class="section" id="packaged-dags">
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/28a3eb60/configuration.html ---------------------------------------------------------------------- diff --git a/configuration.html b/configuration.html index ea1d2e9..881716e 100644 --- a/configuration.html +++ b/configuration.html @@ -13,6 +13,8 @@ + + @@ -81,7 +83,10 @@ - <ul class="current"> + + + + <ul class="current"> <li class="toctree-l1"><a class="reference internal" href="project.html">Project</a></li> <li class="toctree-l1"><a class="reference internal" href="license.html">License</a></li> <li class="toctree-l1"><a class="reference internal" href="start.html">Quick Start</a></li> @@ -92,6 +97,7 @@ <li class="toctree-l2"><a class="reference internal" href="#setting-up-a-backend">Setting up a Backend</a></li> <li class="toctree-l2"><a class="reference internal" href="#connections">Connections</a></li> <li class="toctree-l2"><a class="reference internal" href="#scaling-out-with-celery">Scaling Out with Celery</a></li> +<li class="toctree-l2"><a class="reference internal" href="#scaling-out-with-dask">Scaling Out with Dask</a></li> <li class="toctree-l2"><a class="reference internal" href="#logs">Logs</a></li> <li class="toctree-l2"><a class="reference internal" href="#scaling-out-on-mesos-community-contributed">Scaling Out on Mesos (community contributed)</a></li> <li class="toctree-l2"><a class="reference internal" href="#integration-with-systemd">Integration with systemd</a></li> @@ -180,7 +186,7 @@ building a production-grade environment requires a bit more work!</p> <div class="section" id="setting-configuration-options"> <span id="setting-options"></span><h2>Setting Configuration Options<a class="headerlink" href="#setting-configuration-options" title="Permalink to this headline">¶</a></h2> <p>The first time you run Airflow, it will create a file called <code class="docutils literal"><span class="pre">airflow.cfg</span></code> in -your <code class="docutils literal"><span class="pre">$AIRFLOW_HOME</span></code> directory (<code class="docutils literal"><span class="pre">~/airflow</span></code> by default). This file contains Airflow’s configuration and you +your <code class="docutils literal"><span class="pre">$AIRFLOW_HOME</span></code> directory (<code class="docutils literal"><span class="pre">~/airflow</span></code> by default). This file contains Airflowâs configuration and you can edit it to change any of the settings. You can also set options with environment variables by using this format: <code class="docutils literal"><span class="pre">$AIRFLOW__{SECTION}__{KEY}</span></code> (note the double underscores).</p> <p>For example, the @@ -222,10 +228,10 @@ specific schema in the Postgres connection URI, you may want to set a default schema for your role with a command similar to <code class="docutils literal"><span class="pre">ALTER</span> <span class="pre">ROLE</span> <span class="pre">username</span> <span class="pre">SET</span> <span class="pre">search_path</span> <span class="pre">=</span> <span class="pre">airflow,</span> <span class="pre">foobar;</span></code></p> </div> -<p>Once you’ve setup your database to host Airflow, you’ll need to alter the +<p>Once youâve setup your database to host Airflow, youâll need to alter the SqlAlchemy connection string located in your configuration file -<code class="docutils literal"><span class="pre">$AIRFLOW_HOME/airflow.cfg</span></code>. You should then also change the “executor” -setting to use “LocalExecutor”, an executor that can parallelize task +<code class="docutils literal"><span class="pre">$AIRFLOW_HOME/airflow.cfg</span></code>. You should then also change the âexecutorâ +setting to use âLocalExecutorâ, an executor that can parallelize task instances locally.</p> <div class="highlight-bash"><div class="highlight"><pre><span></span><span class="c1"># initialize the database</span> airflow initdb @@ -237,12 +243,35 @@ airflow initdb <p>Airflow needs to know how to connect to your environment. Information such as hostname, port, login and passwords to other systems and services is handled in the <code class="docutils literal"><span class="pre">Admin->Connection</span></code> section of the UI. The pipeline code you -will author will reference the ‘conn_id’ of the Connection objects.</p> +will author will reference the âconn_idâ of the Connection objects.</p> <img alt="_images/connections.png" src="_images/connections.png" /> <p>By default, Airflow will save the passwords for the connection in plain text within the metadata database. The <code class="docutils literal"><span class="pre">crypto</span></code> package is highly recommended during installation. The <code class="docutils literal"><span class="pre">crypto</span></code> package does require that your operating system have libffi-dev installed.</p> +<p>If <code class="docutils literal"><span class="pre">crypto</span></code> package was not installed initially, you can still enable encryption for +connections by following steps below:</p> +<ol class="arabic simple"> +<li>Install crypto package <code class="docutils literal"><span class="pre">pip</span> <span class="pre">install</span> <span class="pre">apache-airflow[crypto]</span></code></li> +<li>Generate fernet_key, using this code snippet below. fernet_key must be a base64-encoded 32-byte key.</li> +</ol> +<div class="code python highlight-default"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">cryptography.fernet</span> <span class="k">import</span> <span class="n">Fernet</span> +<span class="n">fernet_key</span><span class="o">=</span> <span class="n">Fernet</span><span class="o">.</span><span class="n">generate_key</span><span class="p">()</span> +<span class="nb">print</span><span class="p">(</span><span class="n">fernet_key</span><span class="p">)</span> <span class="c1"># your fernet_key, keep it in secured place!</span> +</pre></div> +</div> +<p>3. Replace <code class="docutils literal"><span class="pre">airflow.cfg</span></code> fernet_key value with the one from step 2. +Alternatively, you can store your fernet_key in OS environment variable. You +do not need to change <code class="docutils literal"><span class="pre">airflow.cfg</span></code> in this case as AirFlow will use environment +variable over the value in <code class="docutils literal"><span class="pre">airflow.cfg</span></code>:</p> +<div class="highlight-bash"><div class="highlight"><pre><span></span><span class="c1"># Note the double underscores</span> +EXPORT <span class="nv">AIRFLOW__CORE__FERNET_KEY</span> <span class="o">=</span> your_fernet_key +</pre></div> +</div> +<ol class="arabic simple" start="4"> +<li>Restart AirFlow webserver.</li> +<li>For existing connections (the ones that you had defined before installing <code class="docutils literal"><span class="pre">airflow[crypto]</span></code> and creating a Fernet key), you need to open each connection in the connection admin UI, re-type the password, and save it.</li> +</ol> <p>Connections in Airflow pipelines can be created using environment variables. The environment variable needs to have a prefix of <code class="docutils literal"><span class="pre">AIRFLOW_CONN_</span></code> for Airflow with the value in a URI format to use the connection properly. Please @@ -252,7 +281,7 @@ variables and connections.</p> <div class="section" id="scaling-out-with-celery"> <h2>Scaling Out with Celery<a class="headerlink" href="#scaling-out-with-celery" title="Permalink to this headline">¶</a></h2> <p><code class="docutils literal"><span class="pre">CeleryExecutor</span></code> is one of the ways you can scale out the number of workers. For this -to work, you need to setup a Celery backend (<strong>RabbitMQ</strong>, <strong>Redis</strong>, ...) and +to work, you need to setup a Celery backend (<strong>RabbitMQ</strong>, <strong>Redis</strong>, â¦) and change your <code class="docutils literal"><span class="pre">airflow.cfg</span></code> to point the executor parameter to <code class="docutils literal"><span class="pre">CeleryExecutor</span></code> and provide the related Celery settings.</p> <p>For more information about setting up a Celery broker, refer to the @@ -280,10 +309,37 @@ subcommand</p> </div> <p>Your worker should start picking up tasks as soon as they get fired in its direction.</p> -<p>Note that you can also run “Celery Flower”, a web UI built on top of Celery, +<p>Note that you can also run âCelery Flowerâ, a web UI built on top of Celery, to monitor your workers. You can use the shortcut command <code class="docutils literal"><span class="pre">airflow</span> <span class="pre">flower</span></code> to start a Flower web server.</p> </div> +<div class="section" id="scaling-out-with-dask"> +<h2>Scaling Out with Dask<a class="headerlink" href="#scaling-out-with-dask" title="Permalink to this headline">¶</a></h2> +<p><code class="docutils literal"><span class="pre">DaskExecutor</span></code> allows you to run Airflow tasks in a Dask Distributed cluster.</p> +<p>Dask clusters can be run on a single machine or on remote networks. For complete +details, consult the <a class="reference external" href="https://distributed.readthedocs.io/";>Distributed documentation</a>.</p> +<p>To create a cluster, first start a Scheduler:</p> +<div class="highlight-bash"><div class="highlight"><pre><span></span><span class="c1"># default settings for a local cluster</span> +<span class="nv">DASK_HOST</span><span class="o">=</span><span class="m">127</span>.0.0.1 +<span class="nv">DASK_PORT</span><span class="o">=</span><span class="m">8786</span> + +dask-scheduler --host <span class="nv">$DASK_HOST</span> --port <span class="nv">$DASK_PORT</span> +</pre></div> +</div> +<p>Next start at least one Worker on any machine that can connect to the host:</p> +<div class="highlight-bash"><div class="highlight"><pre><span></span>dask-worker <span class="nv">$DASK_HOST</span>:<span class="nv">$DASK_PORT</span> +</pre></div> +</div> +<p>Edit your <code class="docutils literal"><span class="pre">airflow.cfg</span></code> to set your executor to <code class="docutils literal"><span class="pre">DaskExecutor</span></code> and provide +the Dask Scheduler address in the <code class="docutils literal"><span class="pre">[dask]</span></code> section.</p> +<p>Please note:</p> +<ul class="simple"> +<li>Each Dask worker must be able to import Airflow and any dependencies you +require.</li> +<li>Dask does not support queues. If an Airflow task was created with a queue, a +warning will be raised but the task will be submitted to the cluster.</li> +</ul> +</div> <div class="section" id="logs"> <h2>Logs<a class="headerlink" href="#logs" title="Permalink to this headline">¶</a></h2> <p>Users can specify a logs folder in <code class="docutils literal"><span class="pre">airflow.cfg</span></code>. By default, it is in @@ -302,13 +358,14 @@ To enable this feature, <code class="docutils literal"><span class="pre">airflow <span class="nv">encrypt_s3_logs</span> <span class="o">=</span> False </pre></div> </div> -<p>Remote logging uses an existing Airflow connection to read/write logs. If you don’t +<p>Remote logging uses an existing Airflow connection to read/write logs. If you donât have a connection properly setup, this will fail. In the above example, Airflow will try to use <code class="docutils literal"><span class="pre">S3Hook('MyS3Conn')</span></code>.</p> <p>In the Airflow Web UI, local logs take precedance over remote logs. If local logs can not be found or accessed, the remote logs will be displayed. Note that logs are only sent to remote storage once a task completes (including failure). In other -words, remote logs for running tasks are unavailable.</p> +words, remote logs for running tasks are unavailable. Logs are stored in the log +folder as <code class="docutils literal"><span class="pre">{dag_id}/{task_id}/{execution_date}/{try_number}.log</span></code>.</p> </div> <div class="section" id="scaling-out-on-mesos-community-contributed"> <h2>Scaling Out on Mesos (community contributed)<a class="headerlink" href="#scaling-out-on-mesos-community-contributed" title="Permalink to this headline">¶</a></h2> @@ -317,7 +374,7 @@ For this to work, you need a running mesos cluster and you must perform the foll steps -</p> <ol class="arabic simple"> <li>Install airflow on a machine where web server and scheduler will run, -let’s refer to this as the “Airflow server”.</li> +letâs refer to this as the âAirflow serverâ.</li> <li>On the Airflow server, install mesos python eggs from <a class="reference external" href="http://open.mesosphere.com/downloads/mesos/";>mesos downloads</a>.</li> <li>On the Airflow server, use a database (such as mysql) which can be accessed from mesos slave machines and add configuration in <code class="docutils literal"><span class="pre">airflow.cfg</span></code>.</li> @@ -374,7 +431,7 @@ integrated with upstart</p> </div> <div class="section" id="test-mode"> <h2>Test Mode<a class="headerlink" href="#test-mode" title="Permalink to this headline">¶</a></h2> -<p>Airflow has a fixed set of “test mode” configuration options. You can load these +<p>Airflow has a fixed set of âtest modeâ configuration options. You can load these at any time by calling <code class="docutils literal"><span class="pre">airflow.configuration.load_test_config()</span></code> (note this operation is not reversible!). However, some options (like the DAG_FOLDER) are loaded before you have a chance to call load_test_config(). In order to eagerly load @@ -383,7 +440,7 @@ the test configuration, set test_mode in airflow.cfg:</p> <span class="nv">unit_test_mode</span> <span class="o">=</span> True </pre></div> </div> -<p>Due to Airflow’s automatic environment variable expansion (see <a class="reference internal" href="#setting-options"><span class="std std-ref">Setting Configuration Options</span></a>), +<p>Due to Airflowâs automatic environment variable expansion (see <a class="reference internal" href="#setting-options"><span class="std std-ref">Setting Configuration Options</span></a>), you can also set the env var <code class="docutils literal"><span class="pre">AIRFLOW__CORE__UNIT_TEST_MODE</span></code> to temporarily overwrite airflow.cfg.</p> </div> http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/28a3eb60/faq.html ---------------------------------------------------------------------- diff --git a/faq.html b/faq.html index 7b6c8ab..f223045 100644 --- a/faq.html +++ b/faq.html @@ -13,6 +13,8 @@ + + @@ -81,7 +83,10 @@ - <ul class="current"> + + + + <ul class="current"> <li class="toctree-l1"><a class="reference internal" href="project.html">Project</a></li> <li class="toctree-l1"><a class="reference internal" href="license.html">License</a></li> <li class="toctree-l1"><a class="reference internal" href="start.html">Quick Start</a></li> @@ -98,10 +103,10 @@ <li class="toctree-l1"><a class="reference internal" href="api.html">Experimental Rest API</a></li> <li class="toctree-l1"><a class="reference internal" href="integration.html">Integration</a></li> <li class="toctree-l1 current"><a class="current reference internal" href="#">FAQ</a><ul> -<li class="toctree-l2"><a class="reference internal" href="#why-isn-t-my-task-getting-scheduled">Why isn’t my task getting scheduled?</a></li> -<li class="toctree-l2"><a class="reference internal" href="#how-do-i-trigger-tasks-based-on-another-task-s-failure">How do I trigger tasks based on another task’s failure?</a></li> +<li class="toctree-l2"><a class="reference internal" href="#why-isn-t-my-task-getting-scheduled">Why isnât my task getting scheduled?</a></li> +<li class="toctree-l2"><a class="reference internal" href="#how-do-i-trigger-tasks-based-on-another-task-s-failure">How do I trigger tasks based on another taskâs failure?</a></li> <li class="toctree-l2"><a class="reference internal" href="#why-are-connection-passwords-still-not-encrypted-in-the-metadata-db-after-i-installed-airflow-crypto">Why are connection passwords still not encrypted in the metadata db after I installed airflow[crypto]?</a></li> -<li class="toctree-l2"><a class="reference internal" href="#what-s-the-deal-with-start-date">What’s the deal with <code class="docutils literal"><span class="pre">start_date</span></code>?</a></li> +<li class="toctree-l2"><a class="reference internal" href="#what-s-the-deal-with-start-date">Whatâs the deal with <code class="docutils literal"><span class="pre">start_date</span></code>?</a></li> <li class="toctree-l2"><a class="reference internal" href="#how-can-i-create-dags-dynamically">How can I create DAGs dynamically?</a></li> <li class="toctree-l2"><a class="reference internal" href="#what-are-all-the-airflow-run-commands-in-my-process-list">What are all the <code class="docutils literal"><span class="pre">airflow</span> <span class="pre">run</span></code> commands in my process list?</a></li> </ul> @@ -173,11 +178,11 @@ <div class="section" id="faq"> <h1>FAQ<a class="headerlink" href="#faq" title="Permalink to this headline">¶</a></h1> <div class="section" id="why-isn-t-my-task-getting-scheduled"> -<h2>Why isn’t my task getting scheduled?<a class="headerlink" href="#why-isn-t-my-task-getting-scheduled" title="Permalink to this headline">¶</a></h2> +<h2>Why isnât my task getting scheduled?<a class="headerlink" href="#why-isn-t-my-task-getting-scheduled" title="Permalink to this headline">¶</a></h2> <p>There are very many reasons why your task might not be getting scheduled. Here are some of the common causes:</p> <ul class="simple"> -<li>Does your script “compile”, can the Airflow engine parse it and find your +<li>Does your script âcompileâ, can the Airflow engine parse it and find your DAG object. To test this, you can run <code class="docutils literal"><span class="pre">airflow</span> <span class="pre">list_dags</span></code> and confirm that your DAG shows up in the list. You can also run <code class="docutils literal"><span class="pre">airflow</span> <span class="pre">list_tasks</span> <span class="pre">foo_dag_id</span> <span class="pre">--tree</span></code> and confirm that your task @@ -189,9 +194,9 @@ task soon after the <code class="docutils literal"><span class="pre">start_date< <li>Is your <code class="docutils literal"><span class="pre">schedule_interval</span></code> set properly? The default <code class="docutils literal"><span class="pre">schedule_interval</span></code> is one day (<code class="docutils literal"><span class="pre">datetime.timedelta(1)</span></code>). You must specify a different <code class="docutils literal"><span class="pre">schedule_interval</span></code> directly to the DAG object you instantiate, not as a <code class="docutils literal"><span class="pre">default_param</span></code>, as task instances -do not override their parent DAG’s <code class="docutils literal"><span class="pre">schedule_interval</span></code>.</li> +do not override their parent DAGâs <code class="docutils literal"><span class="pre">schedule_interval</span></code>.</li> <li>Is your <code class="docutils literal"><span class="pre">start_date</span></code> beyond where you can see it in the UI? If you -set your it to some time say 3 months ago, you won’t be able to see +set your <code class="docutils literal"><span class="pre">start_date</span></code> to some time say 3 months ago, you wonât be able to see it in the main view in the UI, but you should be able to see it in the <code class="docutils literal"><span class="pre">Menu</span> <span class="pre">-></span> <span class="pre">Browse</span> <span class="pre">->Task</span> <span class="pre">Instances</span></code>.</li> <li>Are the dependencies for the task met. The task instances directly @@ -203,7 +208,7 @@ what it means. You can view how these properties are set from the <code class="docutils literal"><span class="pre">Task</span> <span class="pre">Instance</span> <span class="pre">Details</span></code> page for your task.</li> <li>Are the DagRuns you need created and active? A DagRun represents a specific -execution of an entire DAG and has a state (running, success, failed, ...). +execution of an entire DAG and has a state (running, success, failed, â¦). The scheduler creates new DagRun as it moves forward, but never goes back in time to create new ones. The scheduler only evaluates <code class="docutils literal"><span class="pre">running</span></code> DagRuns to see what task instances it can trigger. Note that clearing tasks @@ -220,19 +225,17 @@ how many <code class="docutils literal"><span class="pre">running</span></code> sure you fully understand how it proceeds.</p> </div> <div class="section" id="how-do-i-trigger-tasks-based-on-another-task-s-failure"> -<h2>How do I trigger tasks based on another task’s failure?<a class="headerlink" href="#how-do-i-trigger-tasks-based-on-another-task-s-failure" title="Permalink to this headline">¶</a></h2> +<h2>How do I trigger tasks based on another taskâs failure?<a class="headerlink" href="#how-do-i-trigger-tasks-based-on-another-task-s-failure" title="Permalink to this headline">¶</a></h2> <p>Check out the <code class="docutils literal"><span class="pre">Trigger</span> <span class="pre">Rule</span></code> section in the Concepts section of the documentation</p> </div> <div class="section" id="why-are-connection-passwords-still-not-encrypted-in-the-metadata-db-after-i-installed-airflow-crypto"> <h2>Why are connection passwords still not encrypted in the metadata db after I installed airflow[crypto]?<a class="headerlink" href="#why-are-connection-passwords-still-not-encrypted-in-the-metadata-db-after-i-installed-airflow-crypto" title="Permalink to this headline">¶</a></h2> -<ul class="simple"> -<li>Verify that the <code class="docutils literal"><span class="pre">fernet_key</span></code> defined in <code class="docutils literal"><span class="pre">$AIRFLOW_HOME/airflow.cfg</span></code> is a valid Fernet key. It must be a base64-encoded 32-byte key. You need to restart the webserver after you update the key</li> -<li>For existing connections (the ones that you had defined before installing <code class="docutils literal"><span class="pre">airflow[crypto]</span></code> and creating a Fernet key), you need to open each connection in the connection admin UI, re-type the password, and save it</li> -</ul> +<p>Check out the <code class="docutils literal"><span class="pre">Connections</span></code> section in the Configuration section of the +documentation</p> </div> <div class="section" id="what-s-the-deal-with-start-date"> -<h2>What’s the deal with <code class="docutils literal"><span class="pre">start_date</span></code>?<a class="headerlink" href="#what-s-the-deal-with-start-date" title="Permalink to this headline">¶</a></h2> +<h2>Whatâs the deal with <code class="docutils literal"><span class="pre">start_date</span></code>?<a class="headerlink" href="#what-s-the-deal-with-start-date" title="Permalink to this headline">¶</a></h2> <p><code class="docutils literal"><span class="pre">start_date</span></code> is partly legacy from the pre-DagRun era, but it is still relevant in many ways. When creating a new DAG, you probably want to set a global <code class="docutils literal"><span class="pre">start_date</span></code> for your tasks using <code class="docutils literal"><span class="pre">default_args</span></code>. The first @@ -241,7 +244,7 @@ task. From that point on, the scheduler creates new DagRuns based on your <code class="docutils literal"><span class="pre">schedule_interval</span></code> and the corresponding task instances run as your dependencies are met. When introducing new tasks to your DAG, you need to pay special attention to <code class="docutils literal"><span class="pre">start_date</span></code>, and may want to reactivate -inactive DagRuns to get the new task to get onboarded properly.</p> +inactive DagRuns to get the new task onboarded properly.</p> <p>We recommend against using dynamic values as <code class="docutils literal"><span class="pre">start_date</span></code>, especially <code class="docutils literal"><span class="pre">datetime.now()</span></code> as it can be quite confusing. The task is triggered once the period closes, and in theory an <code class="docutils literal"><span class="pre">@hourly</span></code> DAG would never get to @@ -257,16 +260,16 @@ the execution of tasks within the schedule interval. While <code class="docutils literal"><span class="pre">schedule_interval</span></code> does allow specifying a <code class="docutils literal"><span class="pre">datetime.timedelta</span></code> object, we recommend using the macros or cron expressions instead, as it enforces this idea of rounded schedules.</p> -<p>When using <code class="docutils literal"><span class="pre">depends_on_past=True</span></code> it’s important to pay special attention +<p>When using <code class="docutils literal"><span class="pre">depends_on_past=True</span></code> itâs important to pay special attention to <code class="docutils literal"><span class="pre">start_date</span></code> as the past dependency is not enforced only on the specific -schedule of the <code class="docutils literal"><span class="pre">start_date</span></code> specified for the task. It’ also +schedule of the <code class="docutils literal"><span class="pre">start_date</span></code> specified for the task. Itâs also important to watch DagRun activity status in time when introducing new <code class="docutils literal"><span class="pre">depends_on_past=True</span></code>, unless you are planning on running a backfill for the new task(s).</p> <p>Also important to note is that the tasks <code class="docutils literal"><span class="pre">start_date</span></code>, in the context of a -backfill CLI command, get overridden by the backfill’s command <code class="docutils literal"><span class="pre">start_date</span></code>. +backfill CLI command, get overridden by the backfillâs command <code class="docutils literal"><span class="pre">start_date</span></code>. This allows for a backfill on tasks that have <code class="docutils literal"><span class="pre">depends_on_past=True</span></code> to -actually start, if it wasn’t the case, the backfill just wouldn’t start.</p> +actually start, if that wasnât the case, the backfill just wouldnât start.</p> </div> <div class="section" id="how-can-i-create-dags-dynamically"> <h2>How can I create DAGs dynamically?<a class="headerlink" href="#how-can-i-create-dags-dynamically" title="Permalink to this headline">¶</a></h2> @@ -295,7 +298,7 @@ LocalExecutor, that translates into running it in a subprocess pool.</li> command (described below) as a subprocess and is in charge of emitting heartbeats, listening for external kill signals and ensures some cleanup takes place if the subprocess fails</li> -<li>Raw <code class="docutils literal"><span class="pre">airflow</span> <span class="pre">run</span> <span class="pre">--raw</span></code> runs the actual operator’s execute method and +<li>Raw <code class="docutils literal"><span class="pre">airflow</span> <span class="pre">run</span> <span class="pre">--raw</span></code> runs the actual operatorâs execute method and performs the actual work</li> </ul> </div>
