This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch v2-1-test in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 61332baf3b6b023772ab20109d214f601da01a97 Author: Kan Ouivirach <[email protected]> AuthorDate: Tue Aug 3 03:43:12 2021 +0700 Suggest to use secrets backend for variable when it contains sensitive data (#17319) (cherry picked from commit 2c1880a90712aa79dd7c16c78a93b343cd312268) --- docs/apache-airflow/best-practices.rst | 15 ++++++++++++--- docs/apache-airflow/security/index.rst | 2 -- .../security/secrets/secrets-backend/index.rst | 3 +-- 3 files changed, 13 insertions(+), 7 deletions(-) diff --git a/docs/apache-airflow/best-practices.rst b/docs/apache-airflow/best-practices.rst index 9e89865..7d06192 100644 --- a/docs/apache-airflow/best-practices.rst +++ b/docs/apache-airflow/best-practices.rst @@ -93,13 +93,16 @@ Variables --------- You should avoid usage of Variables outside an operator's ``execute()`` method or Jinja templates if possible, -as Variables create a connection to metadata DB of Airflow to fetch the value, which can slow down parsing and place extra load on the DB. +as Variables create a connection to metadata DB of Airflow to fetch the value, which can slow down parsing and +place extra load on the DB. Airflow parses all the DAGs in the background at a specific period. -The default period is set using ``processor_poll_interval`` config, which is by default 1 second. During parsing, Airflow creates a new connection to the metadata DB for each DAG. +The default period is set using the ``processor_poll_interval`` config, which is 1 second by default. +During parsing, Airflow creates a new connection to the metadata DB for each DAG. This can result in a lot of open connections. -The best way of using variables is via a Jinja template, which will delay reading the value until the task execution. The template syntax to do this is: +The best way of using variables is via a Jinja template, which will delay reading the value until the task execution. +The template syntax to do this is: .. code-block:: @@ -111,6 +114,12 @@ or if you need to deserialize a json object from the variable : {{ var.json.<variable_name> }} +For security purpose, you're recommended to use the :ref:`Secrets Backend<secrets_backend_configuration>` +for any variable that contains sensitive data. + +An alternative option is to use environment variables in the top-level Python code or use environment variables to +create and manage Airflow variables. This will avoid new connections to Airflow metadata DB every time +Airflow parses the Python file. For more information, see: :ref:`managing_variables`. Top level Python Code --------------------- diff --git a/docs/apache-airflow/security/index.rst b/docs/apache-airflow/security/index.rst index 2ebc01b..65d02f7 100644 --- a/docs/apache-airflow/security/index.rst +++ b/docs/apache-airflow/security/index.rst @@ -15,8 +15,6 @@ specific language governing permissions and limitations under the License. - - Security ======== diff --git a/docs/apache-airflow/security/secrets/secrets-backend/index.rst b/docs/apache-airflow/security/secrets/secrets-backend/index.rst index 2b212ea..a70556d 100644 --- a/docs/apache-airflow/security/secrets/secrets-backend/index.rst +++ b/docs/apache-airflow/security/secrets/secrets-backend/index.rst @@ -15,8 +15,7 @@ specific language governing permissions and limitations under the License. - -Secrets backend +Secrets Backend --------------- .. versionadded:: 1.10.10
