http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/1f06fa0e/howto/write-logs.html ---------------------------------------------------------------------- diff --git a/howto/write-logs.html b/howto/write-logs.html index 7be99f8..79c91b0 100644 --- a/howto/write-logs.html +++ b/howto/write-logs.html @@ -197,7 +197,7 @@ directory.</p> <p>In addition, users can supply a remote location for storing logs and log backups in cloud storage.</p> -<p>In the Airflow Web UI, local logs take precedence over remote logs. If local logs +<p>In the Airflow Web UI, local logs take precedance over remote logs. If local logs can not be found or accessed, the remote logs will be displayed. Note that logs are only sent to remote storage once a task completes (including failure). In other words, remote logs for running tasks are unavailable. Logs are stored in the log @@ -269,59 +269,22 @@ Follow the steps below to enable Azure Blob Storage logging.</p> <div class="section" id="writing-logs-to-google-cloud-storage"> <span id="write-logs-gcp"></span><h2>Writing Logs to Google Cloud Storage<a class="headerlink" href="#writing-logs-to-google-cloud-storage" title="Permalink to this headline">¶</a></h2> <p>Follow the steps below to enable Google Cloud Storage logging.</p> -<ol class="arabic"> -<li><p class="first">Airflowâs logging system requires a custom .py file to be located in the <code class="docutils literal notranslate"><span class="pre">PYTHONPATH</span></code>, so that itâs importable from Airflow. Start by creating a directory to store the config file. <code class="docutils literal notranslate"><span class="pre">$AIRFLOW_HOME/config</span></code> is recommended.</p> -</li> -<li><p class="first">Create empty files called <code class="docutils literal notranslate"><span class="pre">$AIRFLOW_HOME/config/log_config.py</span></code> and <code class="docutils literal notranslate"><span class="pre">$AIRFLOW_HOME/config/__init__.py</span></code>.</p> -</li> -<li><p class="first">Copy the contents of <code class="docutils literal notranslate"><span class="pre">airflow/config_templates/airflow_local_settings.py</span></code> into the <code class="docutils literal notranslate"><span class="pre">log_config.py</span></code> file that was just created in the step above.</p> -</li> -<li><p class="first">Customize the following portions of the template:</p> -<blockquote> -<div><div class="highlight-bash notranslate"><div class="highlight"><pre><span></span><span class="c1"># Add this variable to the top of the file. Note the trailing slash.</span> -<span class="nv">GCS_LOG_FOLDER</span> <span class="o">=</span> <span class="s1">'gs://<bucket where logs should be persisted>/'</span> - -<span class="c1"># Rename DEFAULT_LOGGING_CONFIG to LOGGING CONFIG</span> -<span class="nv">LOGGING_CONFIG</span> <span class="o">=</span> ... - -<span class="c1"># Add a GCSTaskHandler to the 'handlers' block of the LOGGING_CONFIG variable</span> -<span class="s1">'gcs.task'</span>: <span class="o">{</span> - <span class="s1">'class'</span>: <span class="s1">'airflow.utils.log.gcs_task_handler.GCSTaskHandler'</span>, - <span class="s1">'formatter'</span>: <span class="s1">'airflow.task'</span>, - <span class="s1">'base_log_folder'</span>: os.path.expanduser<span class="o">(</span>BASE_LOG_FOLDER<span class="o">)</span>, - <span class="s1">'gcs_log_folder'</span>: GCS_LOG_FOLDER, - <span class="s1">'filename_template'</span>: FILENAME_TEMPLATE, -<span class="o">}</span>, - -<span class="c1"># Update the airflow.task and airflow.task_runner blocks to be 'gcs.task' instead of 'file.task'.</span> -<span class="s1">'loggers'</span>: <span class="o">{</span> - <span class="s1">'airflow.task'</span>: <span class="o">{</span> - <span class="s1">'handlers'</span>: <span class="o">[</span><span class="s1">'gcs.task'</span><span class="o">]</span>, - ... - <span class="o">}</span>, - <span class="s1">'airflow.task_runner'</span>: <span class="o">{</span> - <span class="s1">'handlers'</span>: <span class="o">[</span><span class="s1">'gcs.task'</span><span class="o">]</span>, - ... - <span class="o">}</span>, - <span class="s1">'airflow'</span>: <span class="o">{</span> - <span class="s1">'handlers'</span>: <span class="o">[</span><span class="s1">'console'</span><span class="o">]</span>, - ... - <span class="o">}</span>, -<span class="o">}</span> +<p>To enable this feature, <code class="docutils literal notranslate"><span class="pre">airflow.cfg</span></code> must be configured as in this +example:</p> +<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span><span class="o">[</span>core<span class="o">]</span> +<span class="c1"># Airflow can store logs remotely in AWS S3, Google Cloud Storage or Elastic Search.</span> +<span class="c1"># Users must supply an Airflow connection id that provides access to the storage</span> +<span class="c1"># location. If remote_logging is set to true, see UPDATING.md for additional</span> +<span class="c1"># configuration requirements.</span> +<span class="nv">remote_logging</span> <span class="o">=</span> True +<span class="nv">remote_base_log_folder</span> <span class="o">=</span> gs://my-bucket/path/to/logs +<span class="nv">remote_log_conn_id</span> <span class="o">=</span> MyGCSConn </pre></div> </div> -</div></blockquote> -</li> -<li><p class="first">Make sure a Google Cloud Platform connection hook has been defined in Airflow. The hook should have read and write access to the Google Cloud Storage bucket defined above in <code class="docutils literal notranslate"><span class="pre">GCS_LOG_FOLDER</span></code>.</p> +<ol class="arabic"> +<li><p class="first">Install the <code class="docutils literal notranslate"><span class="pre">gcp_api</span></code> package first, like so: <code class="docutils literal notranslate"><span class="pre">pip</span> <span class="pre">install</span> <span class="pre">apache-airflow[gcp_api]</span></code>.</p> </li> -<li><p class="first">Update <code class="docutils literal notranslate"><span class="pre">$AIRFLOW_HOME/airflow.cfg</span></code> to contain:</p> -<blockquote> -<div><div class="highlight-bash notranslate"><div class="highlight"><pre><span></span><span class="nv">task_log_reader</span> <span class="o">=</span> gcs.task -<span class="nv">logging_config_class</span> <span class="o">=</span> log_config.LOGGING_CONFIG -<span class="nv">remote_log_conn_id</span> <span class="o">=</span> <name of the Google cloud platform hook> -</pre></div> -</div> -</div></blockquote> +<li><p class="first">Make sure a Google Cloud Platform connection hook has been defined in Airflow. The hook should have read and write access to the Google Cloud Storage bucket defined above in <code class="docutils literal notranslate"><span class="pre">remote_base_log_folder</span></code>.</p> </li> <li><p class="first">Restart the Airflow webserver and scheduler, and trigger (or wait for) a new task execution.</p> </li> @@ -340,12 +303,6 @@ Follow the steps below to enable Azure Blob Storage logging.</p> </li> </ol> <p>Note the top line that says itâs reading from the remote log file.</p> -<p>Please be aware that if you were persisting logs to Google Cloud Storage -using the old-style airflow.cfg configuration method, the old logs will no -longer be visible in the Airflow UI, though theyâll still exist in Google -Cloud Storage. This is a backwards incompatbile change. If you are unhappy -with it, you can change the <code class="docutils literal notranslate"><span class="pre">FILENAME_TEMPLATE</span></code> to reflect the old-style -log filename format.</p> </div> </div> @@ -387,20 +344,13 @@ log filename format.</p> - <script type="text/javascript"> - var DOCUMENTATION_OPTIONS = { - URL_ROOT:'../', - VERSION:'', - LANGUAGE:'None', - COLLAPSE_INDEX:false, - FILE_SUFFIX:'.html', - HAS_SOURCE: true, - SOURCELINK_SUFFIX: '.txt' - }; - </script> - <script type="text/javascript" src="../_static/jquery.js"></script> - <script type="text/javascript" src="../_static/underscore.js"></script> - <script type="text/javascript" src="../_static/doctools.js"></script> + + + <script type="text/javascript" id="documentation_options" data-url_root="../" src="../_static/documentation_options.js"></script> + <script type="text/javascript" src="../_static/jquery.js"></script> + <script type="text/javascript" src="../_static/underscore.js"></script> + <script type="text/javascript" src="../_static/doctools.js"></script> +
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/1f06fa0e/index.html ---------------------------------------------------------------------- diff --git a/index.html b/index.html index 099c047..c8a8049 100644 --- a/index.html +++ b/index.html @@ -270,6 +270,17 @@ unit of work and continuity.</p> </li> <li class="toctree-l3"><a class="reference internal" href="howto/operator.html#google-cloud-platform-operators">Google Cloud Platform Operators</a><ul> <li class="toctree-l4"><a class="reference internal" href="howto/operator.html#googlecloudstoragetobigqueryoperator">GoogleCloudStorageToBigQueryOperator</a></li> +<li class="toctree-l4"><a class="reference internal" href="howto/operator.html#gceinstancestartoperator">GceInstanceStartOperator</a></li> +<li class="toctree-l4"><a class="reference internal" href="howto/operator.html#gceinstancestopoperator">GceInstanceStopOperator</a></li> +<li class="toctree-l4"><a class="reference internal" href="howto/operator.html#gcesetmachinetypeoperator">GceSetMachineTypeOperator</a></li> +<li class="toctree-l4"><a class="reference internal" href="howto/operator.html#gcffunctiondeleteoperator">GcfFunctionDeleteOperator</a></li> +<li class="toctree-l4"><a class="reference internal" href="howto/operator.html#gcffunctiondeployoperator">GcfFunctionDeployOperator</a></li> +<li class="toctree-l4"><a class="reference internal" href="howto/operator.html#cloudsqlinstancedatabasecreateoperator">CloudSqlInstanceDatabaseCreateOperator</a></li> +<li class="toctree-l4"><a class="reference internal" href="howto/operator.html#cloudsqlinstancedatabasedeleteoperator">CloudSqlInstanceDatabaseDeleteOperator</a></li> +<li class="toctree-l4"><a class="reference internal" href="howto/operator.html#cloudsqlinstancedatabasepatchoperator">CloudSqlInstanceDatabasePatchOperator</a></li> +<li class="toctree-l4"><a class="reference internal" href="howto/operator.html#cloudsqlinstancedeleteoperator">CloudSqlInstanceDeleteOperator</a></li> +<li class="toctree-l4"><a class="reference internal" href="howto/operator.html#cloudsqlinstancecreateoperator">CloudSqlInstanceCreateOperator</a></li> +<li class="toctree-l4"><a class="reference internal" href="howto/operator.html#cloudsqlinstancepatchoperator">CloudSqlInstancePatchOperator</a></li> </ul> </li> </ul> @@ -280,6 +291,7 @@ unit of work and continuity.</p> <li class="toctree-l3"><a class="reference internal" href="howto/manage-connections.html#creating-a-connection-with-environment-variables">Creating a Connection with Environment Variables</a></li> <li class="toctree-l3"><a class="reference internal" href="howto/manage-connections.html#connection-types">Connection Types</a><ul> <li class="toctree-l4"><a class="reference internal" href="howto/manage-connections.html#google-cloud-platform">Google Cloud Platform</a></li> +<li class="toctree-l4"><a class="reference internal" href="howto/manage-connections.html#mysql">MySQL</a></li> </ul> </li> </ul> @@ -356,6 +368,7 @@ unit of work and continuity.</p> </ul> </li> <li class="toctree-l2"><a class="reference internal" href="concepts.html#packaged-dags">Packaged dags</a></li> +<li class="toctree-l2"><a class="reference internal" href="concepts.html#airflowignore">.airflowignore</a></li> </ul> </li> <li class="toctree-l1"><a class="reference internal" href="profiling.html">Data Profiling</a><ul> @@ -370,20 +383,24 @@ unit of work and continuity.</p> <li class="toctree-l1"><a class="reference internal" href="cli.html">Command Line Interface</a><ul> <li class="toctree-l2"><a class="reference internal" href="cli.html#Positional Arguments">Positional Arguments</a></li> <li class="toctree-l2"><a class="reference internal" href="cli.html#Sub-commands:">Sub-commands:</a><ul> -<li class="toctree-l3"><a class="reference internal" href="cli.html#resetdb">resetdb</a><ul> +<li class="toctree-l3"><a class="reference internal" href="cli.html#version">version</a></li> +<li class="toctree-l3"><a class="reference internal" href="cli.html#initdb">initdb</a></li> +<li class="toctree-l3"><a class="reference internal" href="cli.html#upgradedb">upgradedb</a></li> +<li class="toctree-l3"><a class="reference internal" href="cli.html#delete_dag">delete_dag</a><ul> +<li class="toctree-l4"><a class="reference internal" href="cli.html#Positional Arguments_repeat1">Positional Arguments</a></li> <li class="toctree-l4"><a class="reference internal" href="cli.html#Named Arguments">Named Arguments</a></li> </ul> </li> -<li class="toctree-l3"><a class="reference internal" href="cli.html#render">render</a><ul> -<li class="toctree-l4"><a class="reference internal" href="cli.html#Positional Arguments_repeat1">Positional Arguments</a></li> +<li class="toctree-l3"><a class="reference internal" href="cli.html#task_state">task_state</a><ul> +<li class="toctree-l4"><a class="reference internal" href="cli.html#Positional Arguments_repeat2">Positional Arguments</a></li> <li class="toctree-l4"><a class="reference internal" href="cli.html#Named Arguments_repeat1">Named Arguments</a></li> </ul> </li> -<li class="toctree-l3"><a class="reference internal" href="cli.html#variables">variables</a><ul> +<li class="toctree-l3"><a class="reference internal" href="cli.html#list_dags">list_dags</a><ul> <li class="toctree-l4"><a class="reference internal" href="cli.html#Named Arguments_repeat2">Named Arguments</a></li> </ul> </li> -<li class="toctree-l3"><a class="reference internal" href="cli.html#connections">connections</a><ul> +<li class="toctree-l3"><a class="reference internal" href="cli.html#resetdb">resetdb</a><ul> <li class="toctree-l4"><a class="reference internal" href="cli.html#Named Arguments_repeat3">Named Arguments</a></li> </ul> </li> @@ -391,95 +408,91 @@ unit of work and continuity.</p> <li class="toctree-l4"><a class="reference internal" href="cli.html#Named Arguments_repeat4">Named Arguments</a></li> </ul> </li> -<li class="toctree-l3"><a class="reference internal" href="cli.html#pause">pause</a><ul> -<li class="toctree-l4"><a class="reference internal" href="cli.html#Positional Arguments_repeat2">Positional Arguments</a></li> +<li class="toctree-l3"><a class="reference internal" href="cli.html#webserver">webserver</a><ul> <li class="toctree-l4"><a class="reference internal" href="cli.html#Named Arguments_repeat5">Named Arguments</a></li> </ul> </li> -<li class="toctree-l3"><a class="reference internal" href="cli.html#task_failed_deps">task_failed_deps</a><ul> -<li class="toctree-l4"><a class="reference internal" href="cli.html#Positional Arguments_repeat3">Positional Arguments</a></li> +<li class="toctree-l3"><a class="reference internal" href="cli.html#pool">pool</a><ul> <li class="toctree-l4"><a class="reference internal" href="cli.html#Named Arguments_repeat6">Named Arguments</a></li> </ul> </li> -<li class="toctree-l3"><a class="reference internal" href="cli.html#version">version</a></li> -<li class="toctree-l3"><a class="reference internal" href="cli.html#trigger_dag">trigger_dag</a><ul> -<li class="toctree-l4"><a class="reference internal" href="cli.html#Positional Arguments_repeat4">Positional Arguments</a></li> +<li class="toctree-l3"><a class="reference internal" href="cli.html#scheduler">scheduler</a><ul> <li class="toctree-l4"><a class="reference internal" href="cli.html#Named Arguments_repeat7">Named Arguments</a></li> </ul> </li> -<li class="toctree-l3"><a class="reference internal" href="cli.html#initdb">initdb</a></li> -<li class="toctree-l3"><a class="reference internal" href="cli.html#test">test</a><ul> -<li class="toctree-l4"><a class="reference internal" href="cli.html#Positional Arguments_repeat5">Positional Arguments</a></li> +<li class="toctree-l3"><a class="reference internal" href="cli.html#serve_logs">serve_logs</a></li> +<li class="toctree-l3"><a class="reference internal" href="cli.html#clear">clear</a><ul> +<li class="toctree-l4"><a class="reference internal" href="cli.html#Positional Arguments_repeat3">Positional Arguments</a></li> <li class="toctree-l4"><a class="reference internal" href="cli.html#Named Arguments_repeat8">Named Arguments</a></li> </ul> </li> -<li class="toctree-l3"><a class="reference internal" href="cli.html#unpause">unpause</a><ul> -<li class="toctree-l4"><a class="reference internal" href="cli.html#Positional Arguments_repeat6">Positional Arguments</a></li> +<li class="toctree-l3"><a class="reference internal" href="cli.html#trigger_dag">trigger_dag</a><ul> +<li class="toctree-l4"><a class="reference internal" href="cli.html#Positional Arguments_repeat4">Positional Arguments</a></li> <li class="toctree-l4"><a class="reference internal" href="cli.html#Named Arguments_repeat9">Named Arguments</a></li> </ul> </li> -<li class="toctree-l3"><a class="reference internal" href="cli.html#dag_state">dag_state</a><ul> -<li class="toctree-l4"><a class="reference internal" href="cli.html#Positional Arguments_repeat7">Positional Arguments</a></li> +<li class="toctree-l3"><a class="reference internal" href="cli.html#test">test</a><ul> +<li class="toctree-l4"><a class="reference internal" href="cli.html#Positional Arguments_repeat5">Positional Arguments</a></li> <li class="toctree-l4"><a class="reference internal" href="cli.html#Named Arguments_repeat10">Named Arguments</a></li> </ul> </li> -<li class="toctree-l3"><a class="reference internal" href="cli.html#run">run</a><ul> -<li class="toctree-l4"><a class="reference internal" href="cli.html#Positional Arguments_repeat8">Positional Arguments</a></li> +<li class="toctree-l3"><a class="reference internal" href="cli.html#connections">connections</a><ul> <li class="toctree-l4"><a class="reference internal" href="cli.html#Named Arguments_repeat11">Named Arguments</a></li> </ul> </li> -<li class="toctree-l3"><a class="reference internal" href="cli.html#list_tasks">list_tasks</a><ul> -<li class="toctree-l4"><a class="reference internal" href="cli.html#Positional Arguments_repeat9">Positional Arguments</a></li> +<li class="toctree-l3"><a class="reference internal" href="cli.html#worker">worker</a><ul> <li class="toctree-l4"><a class="reference internal" href="cli.html#Named Arguments_repeat12">Named Arguments</a></li> </ul> </li> -<li class="toctree-l3"><a class="reference internal" href="cli.html#backfill">backfill</a><ul> -<li class="toctree-l4"><a class="reference internal" href="cli.html#Positional Arguments_repeat10">Positional Arguments</a></li> +<li class="toctree-l3"><a class="reference internal" href="cli.html#kerberos">kerberos</a><ul> +<li class="toctree-l4"><a class="reference internal" href="cli.html#Positional Arguments_repeat6">Positional Arguments</a></li> <li class="toctree-l4"><a class="reference internal" href="cli.html#Named Arguments_repeat13">Named Arguments</a></li> </ul> </li> -<li class="toctree-l3"><a class="reference internal" href="cli.html#list_dags">list_dags</a><ul> +<li class="toctree-l3"><a class="reference internal" href="cli.html#pause">pause</a><ul> +<li class="toctree-l4"><a class="reference internal" href="cli.html#Positional Arguments_repeat7">Positional Arguments</a></li> <li class="toctree-l4"><a class="reference internal" href="cli.html#Named Arguments_repeat14">Named Arguments</a></li> </ul> </li> -<li class="toctree-l3"><a class="reference internal" href="cli.html#kerberos">kerberos</a><ul> -<li class="toctree-l4"><a class="reference internal" href="cli.html#Positional Arguments_repeat11">Positional Arguments</a></li> +<li class="toctree-l3"><a class="reference internal" href="cli.html#task_failed_deps">task_failed_deps</a><ul> +<li class="toctree-l4"><a class="reference internal" href="cli.html#Positional Arguments_repeat8">Positional Arguments</a></li> <li class="toctree-l4"><a class="reference internal" href="cli.html#Named Arguments_repeat15">Named Arguments</a></li> </ul> </li> -<li class="toctree-l3"><a class="reference internal" href="cli.html#worker">worker</a><ul> +<li class="toctree-l3"><a class="reference internal" href="cli.html#render">render</a><ul> +<li class="toctree-l4"><a class="reference internal" href="cli.html#Positional Arguments_repeat9">Positional Arguments</a></li> <li class="toctree-l4"><a class="reference internal" href="cli.html#Named Arguments_repeat16">Named Arguments</a></li> </ul> </li> -<li class="toctree-l3"><a class="reference internal" href="cli.html#webserver">webserver</a><ul> +<li class="toctree-l3"><a class="reference internal" href="cli.html#run">run</a><ul> +<li class="toctree-l4"><a class="reference internal" href="cli.html#Positional Arguments_repeat10">Positional Arguments</a></li> <li class="toctree-l4"><a class="reference internal" href="cli.html#Named Arguments_repeat17">Named Arguments</a></li> </ul> </li> -<li class="toctree-l3"><a class="reference internal" href="cli.html#flower">flower</a><ul> +<li class="toctree-l3"><a class="reference internal" href="cli.html#list_tasks">list_tasks</a><ul> +<li class="toctree-l4"><a class="reference internal" href="cli.html#Positional Arguments_repeat11">Positional Arguments</a></li> <li class="toctree-l4"><a class="reference internal" href="cli.html#Named Arguments_repeat18">Named Arguments</a></li> </ul> </li> -<li class="toctree-l3"><a class="reference internal" href="cli.html#scheduler">scheduler</a><ul> +<li class="toctree-l3"><a class="reference internal" href="cli.html#backfill">backfill</a><ul> +<li class="toctree-l4"><a class="reference internal" href="cli.html#Positional Arguments_repeat12">Positional Arguments</a></li> <li class="toctree-l4"><a class="reference internal" href="cli.html#Named Arguments_repeat19">Named Arguments</a></li> </ul> </li> -<li class="toctree-l3"><a class="reference internal" href="cli.html#task_state">task_state</a><ul> -<li class="toctree-l4"><a class="reference internal" href="cli.html#Positional Arguments_repeat12">Positional Arguments</a></li> +<li class="toctree-l3"><a class="reference internal" href="cli.html#dag_state">dag_state</a><ul> +<li class="toctree-l4"><a class="reference internal" href="cli.html#Positional Arguments_repeat13">Positional Arguments</a></li> <li class="toctree-l4"><a class="reference internal" href="cli.html#Named Arguments_repeat20">Named Arguments</a></li> </ul> </li> -<li class="toctree-l3"><a class="reference internal" href="cli.html#pool">pool</a><ul> +<li class="toctree-l3"><a class="reference internal" href="cli.html#variables">variables</a><ul> <li class="toctree-l4"><a class="reference internal" href="cli.html#Named Arguments_repeat21">Named Arguments</a></li> </ul> </li> -<li class="toctree-l3"><a class="reference internal" href="cli.html#serve_logs">serve_logs</a></li> -<li class="toctree-l3"><a class="reference internal" href="cli.html#clear">clear</a><ul> -<li class="toctree-l4"><a class="reference internal" href="cli.html#Positional Arguments_repeat13">Positional Arguments</a></li> +<li class="toctree-l3"><a class="reference internal" href="cli.html#flower">flower</a><ul> <li class="toctree-l4"><a class="reference internal" href="cli.html#Named Arguments_repeat22">Named Arguments</a></li> </ul> </li> -<li class="toctree-l3"><a class="reference internal" href="cli.html#upgradedb">upgradedb</a></li> -<li class="toctree-l3"><a class="reference internal" href="cli.html#delete_dag">delete_dag</a><ul> +<li class="toctree-l3"><a class="reference internal" href="cli.html#unpause">unpause</a><ul> <li class="toctree-l4"><a class="reference internal" href="cli.html#Positional Arguments_repeat14">Positional Arguments</a></li> <li class="toctree-l4"><a class="reference internal" href="cli.html#Named Arguments_repeat23">Named Arguments</a></li> </ul> @@ -500,6 +513,7 @@ unit of work and continuity.</p> <li class="toctree-l2"><a class="reference internal" href="plugins.html#why-build-on-top-of-airflow">Why build on top of Airflow?</a></li> <li class="toctree-l2"><a class="reference internal" href="plugins.html#interface">Interface</a></li> <li class="toctree-l2"><a class="reference internal" href="plugins.html#example">Example</a></li> +<li class="toctree-l2"><a class="reference internal" href="plugins.html#note-on-role-based-views">Note on role based views</a></li> </ul> </li> <li class="toctree-l1"><a class="reference internal" href="security.html">Security</a><ul> @@ -625,6 +639,20 @@ unit of work and continuity.</p> <li class="toctree-l4"><a class="reference internal" href="integration.html#bigqueryhook">BigQueryHook</a></li> </ul> </li> +<li class="toctree-l3"><a class="reference internal" href="integration.html#cloud-sql">Cloud SQL</a><ul> +<li class="toctree-l4"><a class="reference internal" href="integration.html#cloud-sql-operators">Cloud SQL Operators</a></li> +<li class="toctree-l4"><a class="reference internal" href="integration.html#cloud-sql-hook">Cloud SQL Hook</a></li> +</ul> +</li> +<li class="toctree-l3"><a class="reference internal" href="integration.html#compute-engine">Compute Engine</a><ul> +<li class="toctree-l4"><a class="reference internal" href="integration.html#compute-engine-operators">Compute Engine Operators</a></li> +</ul> +</li> +<li class="toctree-l3"><a class="reference internal" href="integration.html#cloud-functions">Cloud Functions</a><ul> +<li class="toctree-l4"><a class="reference internal" href="integration.html#cloud-functions-operators">Cloud Functions Operators</a></li> +<li class="toctree-l4"><a class="reference internal" href="integration.html#cloud-functions-hook">Cloud Functions Hook</a></li> +</ul> +</li> <li class="toctree-l3"><a class="reference internal" href="integration.html#cloud-dataflow">Cloud DataFlow</a><ul> <li class="toctree-l4"><a class="reference internal" href="integration.html#dataflow-operators">DataFlow Operators</a></li> <li class="toctree-l4"><a class="reference internal" href="integration.html#dataflowhook">DataFlowHook</a></li> @@ -680,20 +708,20 @@ unit of work and continuity.</p> <li class="toctree-l3"><a class="reference internal" href="code.html#baseoperator">BaseOperator</a></li> <li class="toctree-l3"><a class="reference internal" href="code.html#basesensoroperator">BaseSensorOperator</a></li> <li class="toctree-l3"><a class="reference internal" href="code.html#core-operators">Core Operators</a><ul> -<li class="toctree-l4"><a class="reference internal" href="code.html#id3">Operators</a></li> +<li class="toctree-l4"><a class="reference internal" href="code.html#id1">Operators</a></li> <li class="toctree-l4"><a class="reference internal" href="code.html#sensors">Sensors</a></li> </ul> </li> <li class="toctree-l3"><a class="reference internal" href="code.html#community-contributed-operators">Community-contributed Operators</a><ul> -<li class="toctree-l4"><a class="reference internal" href="code.html#id4">Operators</a></li> -<li class="toctree-l4"><a class="reference internal" href="code.html#id11">Sensors</a></li> +<li class="toctree-l4"><a class="reference internal" href="code.html#id2">Operators</a></li> +<li class="toctree-l4"><a class="reference internal" href="code.html#id9">Sensors</a></li> </ul> </li> </ul> </li> <li class="toctree-l2"><a class="reference internal" href="code.html#macros">Macros</a><ul> <li class="toctree-l3"><a class="reference internal" href="code.html#default-variables">Default Variables</a></li> -<li class="toctree-l3"><a class="reference internal" href="code.html#id13">Macros</a></li> +<li class="toctree-l3"><a class="reference internal" href="code.html#id11">Macros</a></li> </ul> </li> <li class="toctree-l2"><a class="reference internal" href="code.html#models">Models</a></li> @@ -748,20 +776,13 @@ unit of work and continuity.</p> - <script type="text/javascript"> - var DOCUMENTATION_OPTIONS = { - URL_ROOT:'./', - VERSION:'', - LANGUAGE:'None', - COLLAPSE_INDEX:false, - FILE_SUFFIX:'.html', - HAS_SOURCE: true, - SOURCELINK_SUFFIX: '.txt' - }; - </script> - <script type="text/javascript" src="_static/jquery.js"></script> - <script type="text/javascript" src="_static/underscore.js"></script> - <script type="text/javascript" src="_static/doctools.js"></script> + + + <script type="text/javascript" id="documentation_options" data-url_root="./" src="_static/documentation_options.js"></script> + <script type="text/javascript" src="_static/jquery.js"></script> + <script type="text/javascript" src="_static/underscore.js"></script> + <script type="text/javascript" src="_static/doctools.js"></script> + http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/1f06fa0e/installation.html ---------------------------------------------------------------------- diff --git a/installation.html b/installation.html index a435dee..117406f 100644 --- a/installation.html +++ b/installation.html @@ -176,7 +176,7 @@ </pre></div> </div> <p>You can also install Airflow with support for extra features like <code class="docutils literal notranslate"><span class="pre">s3</span></code> or <code class="docutils literal notranslate"><span class="pre">postgres</span></code>:</p> -<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>pip install apache-airflow<span class="o">[</span>postgres,s3<span class="o">]</span> +<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>pip install <span class="s2">"apache-airflow[s3, postgres]"</span> </pre></div> </div> <div class="admonition note"> @@ -222,37 +222,37 @@ these extra dependencies.</p> </tr> <tr class="row-even"><td>async</td> <td><code class="docutils literal notranslate"><span class="pre">pip</span> <span class="pre">install</span> <span class="pre">apache-airflow[async]</span></code></td> -<td>Async worker classes for Gunicorn</td> +<td>Async worker classes for gunicorn</td> +</tr> +<tr class="row-odd"><td>devel</td> +<td><code class="docutils literal notranslate"><span class="pre">pip</span> <span class="pre">install</span> <span class="pre">apache-airflow[devel]</span></code></td> +<td>Minimum dev tools requirements</td> +</tr> +<tr class="row-even"><td>devel_hadoop</td> +<td><code class="docutils literal notranslate"><span class="pre">pip</span> <span class="pre">install</span> <span class="pre">apache-airflow[devel_hadoop]</span></code></td> +<td>Airflow + dependencies on the Hadoop stack</td> </tr> <tr class="row-odd"><td>celery</td> <td><code class="docutils literal notranslate"><span class="pre">pip</span> <span class="pre">install</span> <span class="pre">apache-airflow[celery]</span></code></td> <td>CeleryExecutor</td> </tr> -<tr class="row-even"><td>cloudant</td> -<td><code class="docutils literal notranslate"><span class="pre">pip</span> <span class="pre">install</span> <span class="pre">apache-airflow[cloudant]</span></code></td> -<td>Cloudant hook</td> -</tr> -<tr class="row-odd"><td>crypto</td> +<tr class="row-even"><td>crypto</td> <td><code class="docutils literal notranslate"><span class="pre">pip</span> <span class="pre">install</span> <span class="pre">apache-airflow[crypto]</span></code></td> <td>Encrypt connection passwords in metadata db</td> </tr> -<tr class="row-even"><td>devel</td> -<td><code class="docutils literal notranslate"><span class="pre">pip</span> <span class="pre">install</span> <span class="pre">apache-airflow[devel]</span></code></td> -<td>Minimum dev tools requirements</td> -</tr> -<tr class="row-odd"><td>devel_hadoop</td> -<td><code class="docutils literal notranslate"><span class="pre">pip</span> <span class="pre">install</span> <span class="pre">apache-airflow[devel_hadoop]</span></code></td> -<td>Airflow + dependencies on the Hadoop stack</td> -</tr> -<tr class="row-even"><td>druid</td> +<tr class="row-odd"><td>druid</td> <td><code class="docutils literal notranslate"><span class="pre">pip</span> <span class="pre">install</span> <span class="pre">apache-airflow[druid]</span></code></td> -<td>Druid related operators & hooks</td> +<td>Druid.io related operators & hooks</td> </tr> -<tr class="row-odd"><td>gcp_api</td> +<tr class="row-even"><td>gcp_api</td> <td><code class="docutils literal notranslate"><span class="pre">pip</span> <span class="pre">install</span> <span class="pre">apache-airflow[gcp_api]</span></code></td> <td>Google Cloud Platform hooks and operators (using <code class="docutils literal notranslate"><span class="pre">google-api-python-client</span></code>)</td> </tr> +<tr class="row-odd"><td>jdbc</td> +<td><code class="docutils literal notranslate"><span class="pre">pip</span> <span class="pre">install</span> <span class="pre">apache-airflow[jdbc]</span></code></td> +<td>JDBC hooks and operators</td> +</tr> <tr class="row-even"><td>hdfs</td> <td><code class="docutils literal notranslate"><span class="pre">pip</span> <span class="pre">install</span> <span class="pre">apache-airflow[hdfs]</span></code></td> <td>HDFS hooks and operators</td> @@ -261,52 +261,44 @@ these extra dependencies.</p> <td><code class="docutils literal notranslate"><span class="pre">pip</span> <span class="pre">install</span> <span class="pre">apache-airflow[hive]</span></code></td> <td>All Hive related operators</td> </tr> -<tr class="row-even"><td>jdbc</td> -<td><code class="docutils literal notranslate"><span class="pre">pip</span> <span class="pre">install</span> <span class="pre">apache-airflow[jdbc]</span></code></td> -<td>JDBC hooks and operators</td> -</tr> -<tr class="row-odd"><td>kerbero s</td> +<tr class="row-even"><td>kerberos</td> <td><code class="docutils literal notranslate"><span class="pre">pip</span> <span class="pre">install</span> <span class="pre">apache-airflow[kerberos]</span></code></td> -<td>Kerberos integration for Kerberized Hadoop</td> +<td>kerberos integration for kerberized hadoop</td> </tr> -<tr class="row-even"><td>ldap</td> +<tr class="row-odd"><td>ldap</td> <td><code class="docutils literal notranslate"><span class="pre">pip</span> <span class="pre">install</span> <span class="pre">apache-airflow[ldap]</span></code></td> -<td>LDAP authentication for users</td> +<td>ldap authentication for users</td> </tr> -<tr class="row-odd"><td>mssql</td> +<tr class="row-even"><td>mssql</td> <td><code class="docutils literal notranslate"><span class="pre">pip</span> <span class="pre">install</span> <span class="pre">apache-airflow[mssql]</span></code></td> -<td>Microsoft SQL Server operators and hook, +<td>Microsoft SQL operators and hook, support as an Airflow backend</td> </tr> -<tr class="row-even"><td>mysql</td> +<tr class="row-odd"><td>mysql</td> <td><code class="docutils literal notranslate"><span class="pre">pip</span> <span class="pre">install</span> <span class="pre">apache-airflow[mysql]</span></code></td> -<td>MySQL operators and hook, support as an Airflow -backend. The version of MySQL server has to be -5.6.4+. The exact version upper bound depends -on version of <code class="docutils literal notranslate"><span class="pre">mysqlclient</span></code> package. For -example, <code class="docutils literal notranslate"><span class="pre">mysqlclient</span></code> 1.3.12 can only be +<td>MySQL operators and hook, support as +an Airflow backend. The version of MySQL server +has to be 5.6.4+. The exact version upper bound +depends on version of <code class="docutils literal notranslate"><span class="pre">mysqlclient</span></code> package. +For example, <code class="docutils literal notranslate"><span class="pre">mysqlclient</span></code> 1.3.12 can only be used with MySQL server 5.6.4 through 5.7.</td> </tr> -<tr class="row-odd"><td>password</td> +<tr class="row-even"><td>password</td> <td><code class="docutils literal notranslate"><span class="pre">pip</span> <span class="pre">install</span> <span class="pre">apache-airflow[password]</span></code></td> -<td>Password authentication for users</td> +<td>Password Authentication for users</td> </tr> -<tr class="row-even"><td>postgres</td> +<tr class="row-odd"><td>postgres</td> <td><code class="docutils literal notranslate"><span class="pre">pip</span> <span class="pre">install</span> <span class="pre">apache-airflow[postgres]</span></code></td> -<td>PostgreSQL operators and hook, support as an -Airflow backend</td> +<td>Postgres operators and hook, support +as an Airflow backend</td> </tr> -<tr class="row-odd"><td>qds</td> +<tr class="row-even"><td>qds</td> <td><code class="docutils literal notranslate"><span class="pre">pip</span> <span class="pre">install</span> <span class="pre">apache-airflow[qds]</span></code></td> -<td>Enable QDS (Qubole Data Service) support</td> +<td>Enable QDS (qubole data services) support</td> </tr> -<tr class="row-even"><td>rabbitmq</td> +<tr class="row-odd"><td>rabbitmq</td> <td><code class="docutils literal notranslate"><span class="pre">pip</span> <span class="pre">install</span> <span class="pre">apache-airflow[rabbitmq]</span></code></td> -<td>RabbitMQ support as a Celery backend</td> -</tr> -<tr class="row-odd"><td>redis</td> -<td><code class="docutils literal notranslate"><span class="pre">pip</span> <span class="pre">install</span> <span class="pre">apache-airflow[redis]</span></code></td> -<td>Redis hooks and sensors</td> +<td>Rabbitmq support as a Celery backend</td> </tr> <tr class="row-even"><td>s3</td> <td><code class="docutils literal notranslate"><span class="pre">pip</span> <span class="pre">install</span> <span class="pre">apache-airflow[s3]</span></code></td> @@ -320,13 +312,18 @@ Airflow backend</td> <td><code class="docutils literal notranslate"><span class="pre">pip</span> <span class="pre">install</span> <span class="pre">apache-airflow[slack]</span></code></td> <td><code class="docutils literal notranslate"><span class="pre">SlackAPIPostOperator</span></code></td> </tr> -<tr class="row-odd"><td>ssh</td> -<td><code class="docutils literal notranslate"><span class="pre">pip</span> <span class="pre">install</span> <span class="pre">apache-airflow[ssh]</span></code></td> -<td>SSH hooks and Operator</td> -</tr> -<tr class="row-even"><td>vertica</td> +<tr class="row-odd"><td>vertica</td> <td><code class="docutils literal notranslate"><span class="pre">pip</span> <span class="pre">install</span> <span class="pre">apache-airflow[vertica]</span></code></td> -<td>Vertica hook support as an Airflow backend</td> +<td>Vertica hook +support as an Airflow backend</td> +</tr> +<tr class="row-even"><td>cloudant</td> +<td><code class="docutils literal notranslate"><span class="pre">pip</span> <span class="pre">install</span> <span class="pre">apache-airflow[cloudant]</span></code></td> +<td>Cloudant hook</td> +</tr> +<tr class="row-odd"><td>redis</td> +<td><code class="docutils literal notranslate"><span class="pre">pip</span> <span class="pre">install</span> <span class="pre">apache-airflow[redis]</span></code></td> +<td>Redis hooks and sensors</td> </tr> </tbody> </table> @@ -383,20 +380,13 @@ run tasks:</p> - <script type="text/javascript"> - var DOCUMENTATION_OPTIONS = { - URL_ROOT:'./', - VERSION:'', - LANGUAGE:'None', - COLLAPSE_INDEX:false, - FILE_SUFFIX:'.html', - HAS_SOURCE: true, - SOURCELINK_SUFFIX: '.txt' - }; - </script> - <script type="text/javascript" src="_static/jquery.js"></script> - <script type="text/javascript" src="_static/underscore.js"></script> - <script type="text/javascript" src="_static/doctools.js"></script> + + + <script type="text/javascript" id="documentation_options" data-url_root="./" src="_static/documentation_options.js"></script> + <script type="text/javascript" src="_static/jquery.js"></script> + <script type="text/javascript" src="_static/underscore.js"></script> + <script type="text/javascript" src="_static/doctools.js"></script> +
