This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch update-docker-stack-documentation in repository https://gitbox.apache.org/repos/asf/airflow-site.git
commit 57557e111fb7310498369852fc9379ccbbcea8b9 Author: Jarek Potiuk <[email protected]> AuthorDate: Thu Jul 15 11:00:34 2021 +0200 Update docker stack documentation to latest one That includes: * 2.1.2 version of airflow * bring back the warning about dynamic package installation --- .../docker-stack/_sources/entrypoint.rst.txt | 57 +++++++++-------- docs-archive/docker-stack/_static/check-solid.svg | 2 +- docs-archive/docker-stack/_static/clipboard.min.js | 8 +-- docs-archive/docker-stack/_static/copy-button.svg | 6 +- docs-archive/docker-stack/_static/copybutton.css | 28 ++++++--- docs-archive/docker-stack/_static/copybutton.js | 5 +- docs-archive/docker-stack/build.html | 4 +- docs-archive/docker-stack/entrypoint.html | 72 ++++++++++++---------- docs-archive/docker-stack/genindex.html | 36 +++++------ docs-archive/docker-stack/searchindex.js | 2 +- 10 files changed, 116 insertions(+), 104 deletions(-) diff --git a/docs-archive/docker-stack/_sources/entrypoint.rst.txt b/docs-archive/docker-stack/_sources/entrypoint.rst.txt index c386a67..8ac7355 100644 --- a/docs-archive/docker-stack/_sources/entrypoint.rst.txt +++ b/docs-archive/docker-stack/_sources/entrypoint.rst.txt @@ -94,32 +94,14 @@ You can read more about it in the "Support arbitrary user ids" chapter in the Waits for Airflow DB connection ------------------------------- -In case Postgres or MySQL DB is used, the entrypoint will wait until the airflow DB connection becomes -available. This happens always when you use the default entrypoint. +The entrypoint is waiting for a connection to the database independent of the database engine. This allows us to increase +the stability of the environment. -The script detects backend type depending on the URL schema and assigns default port numbers if not specified -in the URL. Then it loops until the connection to the host/port specified can be established +Waiting for connection involves executing ``airflow db check`` command, which means that a ``select 1 as is_alive;`` statement +is executed. Then it loops until the the command will be successful. It tries :envvar:`CONNECTION_CHECK_MAX_COUNT` times and sleeps :envvar:`CONNECTION_CHECK_SLEEP_TIME` between checks To disable check, set ``CONNECTION_CHECK_MAX_COUNT=0``. -Supported schemes: - -* ``postgres://`` - default port 5432 -* ``mysql://`` - default port 3306 -* ``sqlite://`` - -In case of SQLite backend, there is no connection to establish and waiting is skipped. - -For older than Airflow 1.10.14, waiting for connection involves checking if a matching port is open. -The host information is derived from the variables :envvar:`AIRFLOW__CORE__SQL_ALCHEMY_CONN` and -:envvar:`AIRFLOW__CORE__SQL_ALCHEMY_CONN_CMD`. If :envvar:`AIRFLOW__CORE__SQL_ALCHEMY_CONN_CMD` variable -is passed to the container, it is evaluated as a command to execute and result of this evaluation is used -as :envvar:`AIRFLOW__CORE__SQL_ALCHEMY_CONN`. The :envvar:`AIRFLOW__CORE__SQL_ALCHEMY_CONN_CMD` variable -takes precedence over the :envvar:`AIRFLOW__CORE__SQL_ALCHEMY_CONN` variable. - -For newer versions, the ``airflow db check`` command is used, which means that a ``select 1 as is_alive;`` query -is executed. This also means that you can keep your password in secret backend. - Waits for celery broker connection ---------------------------------- @@ -155,7 +137,7 @@ if you specify extra arguments. For example: .. code-block:: bash - docker run -it apache/airflow:2.1.0-python3.6 bash -c "ls -la" + docker run -it apache/airflow:2.1.2-python3.6 bash -c "ls -la" total 16 drwxr-xr-x 4 airflow root 4096 Jun 5 18:12 . drwxr-xr-x 1 root root 4096 Jun 5 18:12 .. @@ -167,7 +149,7 @@ you pass extra parameters. For example: .. code-block:: bash - > docker run -it apache/airflow:2.1.0-python3.6 python -c "print('test')" + > docker run -it apache/airflow:2.1.2-python3.6 python -c "print('test')" test If first argument equals to "airflow" - the rest of the arguments is treated as an airflow command @@ -175,14 +157,14 @@ to execute. Example: .. code-block:: bash - docker run -it apache/airflow:2.1.0-python3.6 airflow webserver + docker run -it apache/airflow:2.1.2-python3.6 airflow webserver If there are any other arguments - they are simply passed to the "airflow" command .. code-block:: bash - > docker run -it apache/airflow:2.1.0-python3.6 version - 2.1.0 + > docker run -it apache/airflow:2.1.2-python3.6 version + 2.1.2 Additional quick test options ----------------------------- @@ -262,11 +244,28 @@ and Admin role. They also forward local port ``8080`` to the webserver port and Installing additional requirements .................................. +.. warning:: Installing requirements this way is a very convenient method of running Airflow, very useful for + testing and debugging. However, do not be tricked by its convenience. You should never, ever use it in + production environment. We have deliberately chose to make it a development/test dependency and we print + a warning, whenever it is used. There is an inherent security-related issue with using this method in + production. Installing the requirements this way can happen at literally any time - when your containers + get restarted, when your machines in K8S cluster get restarted. In a K8S Cluster those events can happen + literally any time. This opens you up to a serious vulnerability where your production environment + might be brought down by a single dependency being removed from PyPI - or even dependency of your + dependency. This means that you put your production service availability in hands of 3rd-party developers. + At any time, any moment including weekends and holidays those 3rd party developers might bring your + production Airflow instance down, without you even knowing it. This is a serious vulnerability that + is similar to the infamous + `leftpad <https://qz.com/646467/how-one-programmer-broke-the-internet-by-deleting-a-tiny-piece-of-code/>`_ + problem. You can fully protect against this case by building your own, immutable custom image, where the + dependencies are baked in. You have been warned. + Installing additional requirements can be done by specifying ``_PIP_ADDITIONAL_REQUIREMENTS`` variable. The variable should contain a list of requirements that should be installed additionally when entering the containers. Note that this option slows down starting of Airflow as every time any container starts -it must install new packages. Therefore this option should only be used for testing. When testing is -finished, you should create your custom image with dependencies baked in. +it must install new packages and it opens up huge potential security vulnerability when used in production +(see below). Therefore this option should only be used for testing. When testing is finished, +you should create your custom image with dependencies baked in. Example: diff --git a/docs-archive/docker-stack/_static/check-solid.svg b/docs-archive/docker-stack/_static/check-solid.svg index 9cbca86..92fad4b 100644 --- a/docs-archive/docker-stack/_static/check-solid.svg +++ b/docs-archive/docker-stack/_static/check-solid.svg @@ -1,4 +1,4 @@ -<svg xmlns="http://www.w3.org/2000/svg"; class="icon icon-tabler icon-tabler-check" width="44" height="44" viewBox="0 0 24 24" stroke-width="1.5" stroke="currentColor" fill="none" stroke-linecap="round" stroke-linejoin="round"> +<svg xmlns="http://www.w3.org/2000/svg"; class="icon icon-tabler icon-tabler-check" width="44" height="44" viewBox="0 0 24 24" stroke-width="2" stroke="#22863a" fill="none" stroke-linecap="round" stroke-linejoin="round"> <path stroke="none" d="M0 0h24v24H0z" fill="none"/> <path d="M5 12l5 5l10 -10" /> </svg> diff --git a/docs-archive/docker-stack/_static/clipboard.min.js b/docs-archive/docker-stack/_static/clipboard.min.js index 02c549e..54b3c46 100644 --- a/docs-archive/docker-stack/_static/clipboard.min.js +++ b/docs-archive/docker-stack/_static/clipboard.min.js @@ -1,7 +1,7 @@ /*! - * clipboard.js v2.0.4 - * https://zenorocha.github.io/clipboard.js - * + * clipboard.js v2.0.8 + * https://clipboardjs.com/ + * * Licensed MIT © Zeno Rocha */ -!function(t,e){"object"==typeof exports&&"object"==typeof module?module.exports=e():"function"==typeof define&&define.amd?define([],e):"object"==typeof exports?exports.ClipboardJS=e():t.ClipboardJS=e()}(this,function(){return function(n){var o={};function r(t){if(o[t])return o[t].exports;var e=o[t]={i:t,l:!1,exports:{}};return n[t].call(e.exports,e,e.exports,r),e.l=!0,e.exports}return r.m=n,r.c=o,r.d=function(t,e,n){r.o(t,e)||Object.defineProperty(t,e,{enumerable:!0,get:n})},r.r=function [...] \ No newline at end of file +!function(t,e){"object"==typeof exports&&"object"==typeof module?module.exports=e():"function"==typeof define&&define.amd?define([],e):"object"==typeof exports?exports.ClipboardJS=e():t.ClipboardJS=e()}(this,function(){return n={686:function(t,e,n){"use strict";n.d(e,{default:function(){return o}});var e=n(279),i=n.n(e),e=n(370),u=n.n(e),e=n(817),c=n.n(e);function a(t){try{return document.execCommand(t)}catch(t){return}}var f=function(t){t=c()(t);return a("cut"),t};var l=function(t){var [...] \ No newline at end of file diff --git a/docs-archive/docker-stack/_static/copy-button.svg b/docs-archive/docker-stack/_static/copy-button.svg index 62e0e0d..b888a20 100644 --- a/docs-archive/docker-stack/_static/copy-button.svg +++ b/docs-archive/docker-stack/_static/copy-button.svg @@ -1,5 +1,5 @@ -<svg xmlns="http://www.w3.org/2000/svg"; width="24" height="24" viewBox="0 0 24 24" stroke-width="1.5" stroke="#607D8B" fill="none" stroke-linecap="round" stroke-linejoin="round"> +<svg xmlns="http://www.w3.org/2000/svg"; class="icon icon-tabler icon-tabler-clipboard" width="44" height="44" viewBox="0 0 24 24" stroke-width="1.5" stroke="#2c3e50" fill="none" stroke-linecap="round" stroke-linejoin="round"> <path stroke="none" d="M0 0h24v24H0z" fill="none"/> - <rect x="8" y="8" width="12" height="12" rx="2" /> - <path d="M16 8v-2a2 2 0 0 0 -2 -2h-8a2 2 0 0 0 -2 2v8a2 2 0 0 0 2 2h2" /> + <path d="M9 5h-2a2 2 0 0 0 -2 2v12a2 2 0 0 0 2 2h10a2 2 0 0 0 2 -2v-12a2 2 0 0 0 -2 -2h-2" /> + <rect x="9" y="3" width="6" height="4" rx="2" /> </svg> diff --git a/docs-archive/docker-stack/_static/copybutton.css b/docs-archive/docker-stack/_static/copybutton.css index 3a863dd..5d29149 100644 --- a/docs-archive/docker-stack/_static/copybutton.css +++ b/docs-archive/docker-stack/_static/copybutton.css @@ -1,20 +1,29 @@ /* Copy buttons */ button.copybtn { position: absolute; + display: flex; top: .3em; right: .5em; - width: 1.7rem; - height: 1.7rem; + width: 1.7em; + height: 1.7em; opacity: 0; - transition: opacity 0.3s, border .3s; + transition: opacity 0.3s, border .3s, background-color .3s; user-select: none; padding: 0; border: none; outline: none; + border-radius: 0.4em; + border: #e1e1e1 1px solid; + background-color: rgb(245, 245, 245); +} + +button.copybtn.success { + border-color: #22863a; } button.copybtn img { width: 100%; + padding: .2em; } div.highlight { @@ -22,11 +31,15 @@ div.highlight { } .highlight:hover button.copybtn { - opacity: .7; + opacity: 1; } .highlight button.copybtn:hover { - opacity: 1; + background-color: rgb(235, 235, 235); +} + +.highlight button.copybtn:active { + background-color: rgb(187, 187, 187); } /** @@ -46,11 +59,10 @@ div.highlight { visibility: hidden; position: absolute; content: attr(data-tooltip); - padding: 2px; - top: 0; + padding: .2em; + font-size: .8em; left: -.2em; background: grey; - font-size: 1rem; color: white; white-space: nowrap; z-index: 2; diff --git a/docs-archive/docker-stack/_static/copybutton.js b/docs-archive/docker-stack/_static/copybutton.js index c4a9f92..482bda0 100644 --- a/docs-archive/docker-stack/_static/copybutton.js +++ b/docs-archive/docker-stack/_static/copybutton.js @@ -81,7 +81,9 @@ const clearSelection = () => { // Changes tooltip text for two seconds, then changes it back const temporarilyChangeTooltip = (el, oldText, newText) => { el.setAttribute('data-tooltip', newText) + el.classList.add('success') setTimeout(() => el.setAttribute('data-tooltip', oldText), 2000) + setTimeout(() => el.classList.remove('success'), 2000) } // Changes the copy button icon for two seconds, then changes it back @@ -104,10 +106,9 @@ const addCopyButtonToCodeCells = () => { codeCells.forEach((codeCell, index) => { const id = codeCellId(index) codeCell.setAttribute('id', id) - const pre_bg = getComputedStyle(codeCell).backgroundColor; const clipboardButton = id => - `<button class="copybtn o-tooltip--left" style="background-color: ${pre_bg}" data-tooltip="${messages[locale]['copy']}" data-clipboard-target="#${id}"> + `<button class="copybtn o-tooltip--left" data-tooltip="${messages[locale]['copy']}" data-clipboard-target="#${id}"> <img src="${path_static}copy-button.svg" alt="${messages[locale]['copy_to_clipboard']}"> </button>` codeCell.insertAdjacentHTML('afterend', clipboardButton(id)) diff --git a/docs-archive/docker-stack/build.html b/docs-archive/docker-stack/build.html index bf89c80..1dff47c 100644 --- a/docs-archive/docker-stack/build.html +++ b/docs-archive/docker-stack/build.html @@ -1283,7 +1283,7 @@ to provide this library from you repository if you want to build Airflow image i <div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>rm docker-context-files/*.whl docker-context-files/*.tar.gz docker-context-files/*.txt <span class="o">||</span> <span class="nb">true</span> curl -Lo <span class="s2">"docker-context-files/constraints-3.7.txt"</span> <span class="se">\</span> - https://raw.githubusercontent.com/apache/airflow/constraints-2.0.2/constraints-3.7.txt + https://raw.githubusercontent.com/apache/airflow/constraints-2.1.2/constraints-3.7.txt <span class="c1"># For Airflow pre 2.1 you need to use PIP 20.2.4 to install/download Airflow packages.</span> pip install <span class="nv">pip</span><span class="o">==</span><span class="m">20</span>.2.4 @@ -1323,7 +1323,7 @@ to the below:</p> <div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>docker build . <span class="se">\</span> --build-arg <span class="nv">PYTHON_BASE_IMAGE</span><span class="o">=</span><span class="s2">"python:3.7-slim-buster"</span> <span class="se">\</span> --build-arg <span class="nv">AIRFLOW_INSTALLATION_METHOD</span><span class="o">=</span><span class="s2">"apache-airflow"</span> <span class="se">\</span> - --build-arg <span class="nv">AIRFLOW_VERSION</span><span class="o">=</span><span class="s2">"2.0.2"</span> <span class="se">\</span> + --build-arg <span class="nv">AIRFLOW_VERSION</span><span class="o">=</span><span class="s2">"2.1.2"</span> <span class="se">\</span> --build-arg <span class="nv">INSTALL_MYSQL_CLIENT</span><span class="o">=</span><span class="s2">"false"</span> <span class="se">\</span> --build-arg <span class="nv">AIRFLOW_PRE_CACHED_PIP_PACKAGES</span><span class="o">=</span><span class="s2">"false"</span> <span class="se">\</span> --build-arg <span class="nv">INSTALL_FROM_DOCKER_CONTEXT_FILES</span><span class="o">=</span><span class="s2">"true"</span> <span class="se">\</span> diff --git a/docs-archive/docker-stack/entrypoint.html b/docs-archive/docker-stack/entrypoint.html index abe7398..1350a79 100644 --- a/docs-archive/docker-stack/entrypoint.html +++ b/docs-archive/docker-stack/entrypoint.html @@ -643,27 +643,12 @@ that need group access will also be writable for the group. This can be done for </div> <div class="section" id="waits-for-airflow-db-connection"> <h2>Waits for Airflow DB connection<a class="headerlink" href="#waits-for-airflow-db-connection" title="Permalink to this headline">¶</a></h2> -<p>In case Postgres or MySQL DB is used, the entrypoint will wait until the airflow DB connection becomes -available. This happens always when you use the default entrypoint.</p> -<p>The script detects backend type depending on the URL schema and assigns default port numbers if not specified -in the URL. Then it loops until the connection to the host/port specified can be established +<p>The entrypoint is waiting for a connection to the database independent of the database engine. This allows us to increase +the stability of the environment.</p> +<p>Waiting for connection involves executing <code class="docutils literal notranslate"><span class="pre">airflow</span> <span class="pre">db</span> <span class="pre">check</span></code> command, which means that a <code class="docutils literal notranslate"><span class="pre">select</span> <span class="pre">1</span> <span class="pre">as</span> <span class="pre">is_alive;</span></code> statement +is executed. Then it loops until the the command will be successful. It tries <span class="target" id="index-0"></span><code class="xref std std-envvar docutils literal notranslate"><span class="pre">CONNECTION_CHECK_MAX_COUNT</span></code> times and sleeps <span class="target" id="index-1"></span><code class="xref std std-envvar docutils literal notranslate"><span class="pre">CONNECTION_CHECK_SLEEP_TIME</span></code> between checks To disable check, set <code class="docutils literal notranslate"><span class="pre">CONNECTION_CHECK_MAX_COUNT=0</span></code>.</p> -<p>Supported schemes:</p> -<ul class="simple"> -<li><p><code class="docutils literal notranslate"><span class="pre">postgres://</span></code> - default port 5432</p></li> -<li><p><code class="docutils literal notranslate"><span class="pre">mysql://</span></code> - default port 3306</p></li> -<li><p><code class="docutils literal notranslate"><span class="pre">sqlite://</span></code></p></li> -</ul> -<p>In case of SQLite backend, there is no connection to establish and waiting is skipped.</p> -<p>For older than Airflow 1.10.14, waiting for connection involves checking if a matching port is open. -The host information is derived from the variables <span class="target" id="index-2"></span><code class="xref std std-envvar docutils literal notranslate"><span class="pre">AIRFLOW__CORE__SQL_ALCHEMY_CONN</span></code> and -<span class="target" id="index-3"></span><code class="xref std std-envvar docutils literal notranslate"><span class="pre">AIRFLOW__CORE__SQL_ALCHEMY_CONN_CMD</span></code>. If <span class="target" id="index-4"></span><code class="xref std std-envvar docutils literal notranslate"><span class="pre">AIRFLOW__CORE__SQL_ALCHEMY_CONN_CMD</span></code> variable -is passed to the container, it is evaluated as a command to execute and result of this evaluation is used -as <span class="target" id="index-5"></span><code class="xref std std-envvar docutils literal notranslate"><span class="pre">AIRFLOW__CORE__SQL_ALCHEMY_CONN</span></code>. The <span class="target" id="index-6"></span><code class="xref std std-envvar docutils literal notranslate"><span class="pre">AIRFLOW__CORE__SQL_ALCHEMY_CONN_CMD</span></code> variable -takes precedence over the <span class="target" id="index-7"></span><code class="xref std std-envvar docutils literal notranslate"><span class="pre">AIRFLOW__CORE__SQL_ALCHEMY_CONN</span></code> variable.</p> -<p>For newer versions, the <code class="docutils literal notranslate"><span class="pre">airflow</span> <span class="pre">db</span> <span class="pre">check</span></code> command is used, which means that a <code class="docutils literal notranslate"><span class="pre">select</span> <span class="pre">1</span> <span class="pre">as</span> <span class="pre">is_alive;</span></code> query -is executed. This also means that you can keep your password in secret backend.</p> </div> <div class="section" id="waits-for-celery-broker-connection"> <h2>Waits for celery broker connection<a class="headerlink" href="#waits-for-celery-broker-connection" title="Permalink to this headline">¶</a></h2> @@ -671,7 +656,7 @@ is executed. This also means that you can keep your password in secret backend.< commands are used the entrypoint will wait until the celery broker DB connection is available.</p> <p>The script detects backend type depending on the URL schema and assigns default port numbers if not specified in the URL. Then it loops until connection to the host/port specified can be established -It tries <span class="target" id="index-8"></span><code class="xref std std-envvar docutils literal notranslate"><span class="pre">CONNECTION_CHECK_MAX_COUNT</span></code> times and sleeps <span class="target" id="index-9"></span><code class="xref std std-envvar docutils literal notranslate"><span class="pre">CONNECTION_CHECK_SLEEP_TIME</span></code> between checks. +It tries <span class="target" id="index-2"></span><code class="xref std std-envvar docutils literal notranslate"><span class="pre">CONNECTION_CHECK_MAX_COUNT</span></code> times and sleeps <span class="target" id="index-3"></span><code class="xref std std-envvar docutils literal notranslate"><span class="pre">CONNECTION_CHECK_SLEEP_TIME</span></code> between checks. To disable check, set <code class="docutils literal notranslate"><span class="pre">CONNECTION_CHECK_MAX_COUNT=0</span></code>.</p> <p>Supported schemes:</p> <ul class="simple"> @@ -681,17 +666,17 @@ To disable check, set <code class="docutils literal notranslate"><span class="pr <li><p><code class="docutils literal notranslate"><span class="pre">mysql://</span></code> - default port 3306</p></li> </ul> <p>Waiting for connection involves checking if a matching port is open. -The host information is derived from the variables <span class="target" id="index-10"></span><code class="xref std std-envvar docutils literal notranslate"><span class="pre">AIRFLOW__CELERY__BROKER_URL</span></code> and -<span class="target" id="index-11"></span><code class="xref std std-envvar docutils literal notranslate"><span class="pre">AIRFLOW__CELERY__BROKER_URL_CMD</span></code>. If <span class="target" id="index-12"></span><code class="xref std std-envvar docutils literal notranslate"><span class="pre">AIRFLOW__CELERY__BROKER_URL_CMD</span></code> variable +The host information is derived from the variables <span class="target" id="index-4"></span><code class="xref std std-envvar docutils literal notranslate"><span class="pre">AIRFLOW__CELERY__BROKER_URL</span></code> and +<span class="target" id="index-5"></span><code class="xref std std-envvar docutils literal notranslate"><span class="pre">AIRFLOW__CELERY__BROKER_URL_CMD</span></code>. If <span class="target" id="index-6"></span><code class="xref std std-envvar docutils literal notranslate"><span class="pre">AIRFLOW__CELERY__BROKER_URL_CMD</span></code> variable is passed to the container, it is evaluated as a command to execute and result of this evaluation is used -as <span class="target" id="index-13"></span><code class="xref std std-envvar docutils literal notranslate"><span class="pre">AIRFLOW__CELERY__BROKER_URL</span></code>. The <span class="target" id="index-14"></span><code class="xref std std-envvar docutils literal notranslate"><span class="pre">AIRFLOW__CELERY__BROKER_URL_CMD</span></code> variable -takes precedence over the <span class="target" id="index-15"></span><code class="xref std std-envvar docutils literal notranslate"><span class="pre">AIRFLOW__CELERY__BROKER_URL</span></code> variable.</p> +as <span class="target" id="index-7"></span><code class="xref std std-envvar docutils literal notranslate"><span class="pre">AIRFLOW__CELERY__BROKER_URL</span></code>. The <span class="target" id="index-8"></span><code class="xref std std-envvar docutils literal notranslate"><span class="pre">AIRFLOW__CELERY__BROKER_URL_CMD</span></code> variable +takes precedence over the <span class="target" id="index-9"></span><code class="xref std std-envvar docutils literal notranslate"><span class="pre">AIRFLOW__CELERY__BROKER_URL</span></code> variable.</p> </div> <div class="section" id="executing-commands"> <span id="entrypoint-commands"></span><h2>Executing commands<a class="headerlink" href="#executing-commands" title="Permalink to this headline">¶</a></h2> <p>If first argument equals to “bash” - you are dropped to a bash shell or you can executes bash command if you specify extra arguments. For example:</p> -<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>docker run -it apache/airflow:2.1.0-python3.6 bash -c <span class="s2">"ls -la"</span> +<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>docker run -it apache/airflow:2.1.2-python3.6 bash -c <span class="s2">"ls -la"</span> total <span class="m">16</span> drwxr-xr-x <span class="m">4</span> airflow root <span class="m">4096</span> Jun <span class="m">5</span> <span class="m">18</span>:12 . drwxr-xr-x <span class="m">1</span> root root <span class="m">4096</span> Jun <span class="m">5</span> <span class="m">18</span>:12 .. @@ -701,18 +686,18 @@ drwxr-xr-x <span class="m">2</span> airflow root <span class="m">4096</span> Jun </div> <p>If first argument is equal to <code class="docutils literal notranslate"><span class="pre">python</span></code> - you are dropped in python shell or python commands are executed if you pass extra parameters. For example:</p> -<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>> docker run -it apache/airflow:2.1.0-python3.6 python -c <span class="s2">"print('test')"</span> +<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>> docker run -it apache/airflow:2.1.2-python3.6 python -c <span class="s2">"print('test')"</span> <span class="nb">test</span> </pre></div> </div> <p>If first argument equals to “airflow” - the rest of the arguments is treated as an airflow command to execute. Example:</p> -<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>docker run -it apache/airflow:2.1.0-python3.6 airflow webserver +<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>docker run -it apache/airflow:2.1.2-python3.6 airflow webserver </pre></div> </div> <p>If there are any other arguments - they are simply passed to the “airflow” command</p> -<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>> docker run -it apache/airflow:2.1.0-python3.6 version -<span class="m">2</span>.1.0 +<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>> docker run -it apache/airflow:2.1.2-python3.6 version +<span class="m">2</span>.1.2 </pre></div> </div> </div> @@ -726,7 +711,7 @@ either as maintenance operations on the database or should be embedded in the cu (when you want to add new packages).</p> <div class="section" id="upgrading-airflow-db"> <h3>Upgrading Airflow DB<a class="headerlink" href="#upgrading-airflow-db" title="Permalink to this headline">¶</a></h3> -<p>If you set <span class="target" id="index-16"></span><code class="xref std std-envvar docutils literal notranslate"><span class="pre">_AIRFLOW_DB_UPGRADE</span></code> variable to a non-empty value, the entrypoint will run +<p>If you set <span class="target" id="index-10"></span><code class="xref std std-envvar docutils literal notranslate"><span class="pre">_AIRFLOW_DB_UPGRADE</span></code> variable to a non-empty value, the entrypoint will run the <code class="docutils literal notranslate"><span class="pre">airflow</span> <span class="pre">db</span> <span class="pre">upgrade</span></code> command right after verifying the connection. You can also use this when you are running airflow with internal SQLite database (default) to upgrade the db and create admin users at entrypoint, so that you can start the webserver immediately. Note - using SQLite is @@ -736,10 +721,10 @@ comes to concurrency.</p> <div class="section" id="creating-admin-user"> <h3>Creating admin user<a class="headerlink" href="#creating-admin-user" title="Permalink to this headline">¶</a></h3> <p>The entrypoint can also create webserver user automatically when you enter it. you need to set -<span class="target" id="index-17"></span><code class="xref std std-envvar docutils literal notranslate"><span class="pre">_AIRFLOW_WWW_USER_CREATE</span></code> to a non-empty value in order to do that. This is not intended for +<span class="target" id="index-11"></span><code class="xref std std-envvar docutils literal notranslate"><span class="pre">_AIRFLOW_WWW_USER_CREATE</span></code> to a non-empty value in order to do that. This is not intended for production, it is only useful if you would like to run a quick test with the production image. You need to pass at least password to create such user via <code class="docutils literal notranslate"><span class="pre">_AIRFLOW_WWW_USER_PASSWORD</span></code> or -<span class="target" id="index-18"></span><code class="xref std std-envvar docutils literal notranslate"><span class="pre">_AIRFLOW_WWW_USER_PASSWORD_CMD</span></code> similarly like for other <code class="docutils literal notranslate"><span class="pre">*_CMD</span></code> variables, the content of +<span class="target" id="index-12"></span><code class="xref std std-envvar docutils literal notranslate"><span class="pre">_AIRFLOW_WWW_USER_PASSWORD_CMD</span></code> similarly like for other <code class="docutils literal notranslate"><span class="pre">*_CMD</span></code> variables, the content of the <code class="docutils literal notranslate"><span class="pre">*_CMD</span></code> will be evaluated as shell command and it’s output will be set as password.</p> <p>User creation will fail if none of the <code class="docutils literal notranslate"><span class="pre">PASSWORD</span></code> variables are set - there is no default for password for security reasons.</p> @@ -805,11 +790,30 @@ and Admin role. They also forward local port <code class="docutils literal notra </div> <div class="section" id="installing-additional-requirements"> <h3>Installing additional requirements<a class="headerlink" href="#installing-additional-requirements" title="Permalink to this headline">¶</a></h3> +<div class="admonition warning"> +<p class="admonition-title">Warning</p> +<p>Installing requirements this way is a very convenient method of running Airflow, very useful for +testing and debugging. However, do not be tricked by its convenience. You should never, ever use it in +production environment. We have deliberately chose to make it a development/test dependency and we print +a warning, whenever it is used. There is an inherent security-related issue with using this method in +production. Installing the requirements this way can happen at literally any time - when your containers +get restarted, when your machines in K8S cluster get restarted. In a K8S Cluster those events can happen +literally any time. This opens you up to a serious vulnerability where your production environment +might be brought down by a single dependency being removed from PyPI - or even dependency of your +dependency. This means that you put your production service availability in hands of 3rd-party developers. +At any time, any moment including weekends and holidays those 3rd party developers might bring your +production Airflow instance down, without you even knowing it. This is a serious vulnerability that +is similar to the infamous +<a class="reference external" href="https://qz.com/646467/how-one-programmer-broke-the-internet-by-deleting-a-tiny-piece-of-code/";>leftpad</a> +problem. You can fully protect against this case by building your own, immutable custom image, where the +dependencies are baked in. You have been warned.</p> +</div> <p>Installing additional requirements can be done by specifying <code class="docutils literal notranslate"><span class="pre">_PIP_ADDITIONAL_REQUIREMENTS</span></code> variable. The variable should contain a list of requirements that should be installed additionally when entering the containers. Note that this option slows down starting of Airflow as every time any container starts -it must install new packages. Therefore this option should only be used for testing. When testing is -finished, you should create your custom image with dependencies baked in.</p> +it must install new packages and it opens up huge potential security vulnerability when used in production +(see below). Therefore this option should only be used for testing. When testing is finished, +you should create your custom image with dependencies baked in.</p> <p>Example:</p> <div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>docker run -it -p <span class="m">8080</span>:8080 <span class="se">\</span> --env <span class="s2">"_PIP_ADDITIONAL_REQUIREMENTS=lxml==4.6.3 charset-normalizer==1.4.1"</span> <span class="se">\</span> diff --git a/docs-archive/docker-stack/genindex.html b/docs-archive/docker-stack/genindex.html index ff5cb63..c3fecc8 100644 --- a/docs-archive/docker-stack/genindex.html +++ b/docs-archive/docker-stack/genindex.html @@ -567,13 +567,13 @@ <h2 id="_">_</h2> <table style="width: 100%" class="indextable genindextable"><tr> <td style="width: 33%; vertical-align: top;"><ul> - <li><a href="entrypoint.html#index-16">_AIRFLOW_DB_UPGRADE</a> + <li><a href="entrypoint.html#index-10">_AIRFLOW_DB_UPGRADE</a> </li> </ul></td> <td style="width: 33%; vertical-align: top;"><ul> - <li><a href="entrypoint.html#index-17">_AIRFLOW_WWW_USER_CREATE</a> + <li><a href="entrypoint.html#index-11">_AIRFLOW_WWW_USER_CREATE</a> </li> - <li><a href="entrypoint.html#index-18">_AIRFLOW_WWW_USER_PASSWORD_CMD</a> + <li><a href="entrypoint.html#index-12">_AIRFLOW_WWW_USER_PASSWORD_CMD</a> </li> </ul></td> </tr></table> @@ -581,15 +581,13 @@ <h2 id="A">A</h2> <table style="width: 100%" class="indextable genindextable"><tr> <td style="width: 33%; vertical-align: top;"><ul> - <li><a href="entrypoint.html#index-10">AIRFLOW__CELERY__BROKER_URL</a>, <a href="entrypoint.html#index-13">[1]</a>, <a href="entrypoint.html#index-15">[2]</a> + <li><a href="entrypoint.html#index-4">AIRFLOW__CELERY__BROKER_URL</a>, <a href="entrypoint.html#index-7">[1]</a>, <a href="entrypoint.html#index-9">[2]</a> </li> - <li><a href="entrypoint.html#index-11">AIRFLOW__CELERY__BROKER_URL_CMD</a>, <a href="entrypoint.html#index-12">[1]</a>, <a href="entrypoint.html#index-14">[2]</a> + <li><a href="entrypoint.html#index-5">AIRFLOW__CELERY__BROKER_URL_CMD</a>, <a href="entrypoint.html#index-6">[1]</a>, <a href="entrypoint.html#index-8">[2]</a> </li> </ul></td> <td style="width: 33%; vertical-align: top;"><ul> - <li><a href="entrypoint.html#index-2">AIRFLOW__CORE__SQL_ALCHEMY_CONN</a>, <a href="entrypoint.html#index-5">[1]</a>, <a href="entrypoint.html#index-7">[2]</a>, <a href="index.html#index-1">[3]</a> -</li> - <li><a href="entrypoint.html#index-3">AIRFLOW__CORE__SQL_ALCHEMY_CONN_CMD</a>, <a href="entrypoint.html#index-4">[1]</a>, <a href="entrypoint.html#index-6">[2]</a> + <li><a href="index.html#index-1">AIRFLOW__CORE__SQL_ALCHEMY_CONN</a> </li> <li><a href="index.html#index-0">AIRFLOW_HOME</a> </li> @@ -599,11 +597,11 @@ <h2 id="C">C</h2> <table style="width: 100%" class="indextable genindextable"><tr> <td style="width: 33%; vertical-align: top;"><ul> - <li><a href="entrypoint.html#index-0">CONNECTION_CHECK_MAX_COUNT</a>, <a href="entrypoint.html#index-8">[1]</a> + <li><a href="entrypoint.html#index-0">CONNECTION_CHECK_MAX_COUNT</a>, <a href="entrypoint.html#index-2">[1]</a> </li> </ul></td> <td style="width: 33%; vertical-align: top;"><ul> - <li><a href="entrypoint.html#index-1">CONNECTION_CHECK_SLEEP_TIME</a>, <a href="entrypoint.html#index-9">[1]</a> + <li><a href="entrypoint.html#index-1">CONNECTION_CHECK_SLEEP_TIME</a>, <a href="entrypoint.html#index-3">[1]</a> </li> </ul></td> </tr></table> @@ -615,25 +613,23 @@ environment variable <ul> - <li><a href="entrypoint.html#index-16">_AIRFLOW_DB_UPGRADE</a> -</li> - <li><a href="entrypoint.html#index-17">_AIRFLOW_WWW_USER_CREATE</a> + <li><a href="entrypoint.html#index-10">_AIRFLOW_DB_UPGRADE</a> </li> - <li><a href="entrypoint.html#index-18">_AIRFLOW_WWW_USER_PASSWORD_CMD</a> + <li><a href="entrypoint.html#index-11">_AIRFLOW_WWW_USER_CREATE</a> </li> - <li><a href="entrypoint.html#index-10">AIRFLOW__CELERY__BROKER_URL</a>, <a href="entrypoint.html#index-13">[1]</a>, <a href="entrypoint.html#index-15">[2]</a> + <li><a href="entrypoint.html#index-12">_AIRFLOW_WWW_USER_PASSWORD_CMD</a> </li> - <li><a href="entrypoint.html#index-11">AIRFLOW__CELERY__BROKER_URL_CMD</a>, <a href="entrypoint.html#index-12">[1]</a>, <a href="entrypoint.html#index-14">[2]</a> + <li><a href="entrypoint.html#index-4">AIRFLOW__CELERY__BROKER_URL</a>, <a href="entrypoint.html#index-7">[1]</a>, <a href="entrypoint.html#index-9">[2]</a> </li> - <li><a href="entrypoint.html#index-2">AIRFLOW__CORE__SQL_ALCHEMY_CONN</a>, <a href="entrypoint.html#index-5">[1]</a>, <a href="entrypoint.html#index-7">[2]</a>, <a href="index.html#index-1">[3]</a> + <li><a href="entrypoint.html#index-5">AIRFLOW__CELERY__BROKER_URL_CMD</a>, <a href="entrypoint.html#index-6">[1]</a>, <a href="entrypoint.html#index-8">[2]</a> </li> - <li><a href="entrypoint.html#index-3">AIRFLOW__CORE__SQL_ALCHEMY_CONN_CMD</a>, <a href="entrypoint.html#index-4">[1]</a>, <a href="entrypoint.html#index-6">[2]</a> + <li><a href="index.html#index-1">AIRFLOW__CORE__SQL_ALCHEMY_CONN</a> </li> <li><a href="index.html#index-0">AIRFLOW_HOME</a> </li> - <li><a href="entrypoint.html#index-0">CONNECTION_CHECK_MAX_COUNT</a>, <a href="entrypoint.html#index-8">[1]</a> + <li><a href="entrypoint.html#index-0">CONNECTION_CHECK_MAX_COUNT</a>, <a href="entrypoint.html#index-2">[1]</a> </li> - <li><a href="entrypoint.html#index-1">CONNECTION_CHECK_SLEEP_TIME</a>, <a href="entrypoint.html#index-9">[1]</a> + <li><a href="entrypoint.html#index-1">CONNECTION_CHECK_SLEEP_TIME</a>, <a href="entrypoint.html#index-3">[1]</a> </li> </ul></li> </ul></td> diff --git a/docs-archive/docker-stack/searchindex.js b/docs-archive/docker-stack/searchindex.js index 4ef34de..f80a14e 100644 --- a/docs-archive/docker-stack/searchindex.js +++ b/docs-archive/docker-stack/searchindex.js @@ -1 +1 @@ -Search.setIndex({docnames:["build","build-arg-ref","entrypoint","index","recipes"],envversion:{"sphinx.domains.c":2,"sphinx.domains.changeset":1,"sphinx.domains.citation":1,"sphinx.domains.cpp":3,"sphinx.domains.index":1,"sphinx.domains.javascript":2,"sphinx.domains.math":2,"sphinx.domains.python":2,"sphinx.domains.rst":2,"sphinx.domains.std":1,"sphinx.ext.intersphinx":1,"sphinx.ext.viewcode":1,sphinx:56},filenames:["build.rst","build-arg-ref.rst","entrypoint.rst","index.rst","recipes.rs [...] \ No newline at end of file +Search.setIndex({docnames:["build","build-arg-ref","entrypoint","index","recipes"],envversion:{"sphinx.domains.c":2,"sphinx.domains.changeset":1,"sphinx.domains.citation":1,"sphinx.domains.cpp":3,"sphinx.domains.index":1,"sphinx.domains.javascript":2,"sphinx.domains.math":2,"sphinx.domains.python":2,"sphinx.domains.rst":2,"sphinx.domains.std":1,"sphinx.ext.intersphinx":1,"sphinx.ext.viewcode":1,sphinx:56},filenames:["build.rst","build-arg-ref.rst","entrypoint.rst","index.rst","recipes.rs [...] \ No newline at end of file
