mik-laj commented on a change in pull request #6863: [AIRFLOW-XXX] add docs showing usage of `Connection.get_uri` URL: https://github.com/apache/airflow/pull/6863#discussion_r360730383
########## File path: docs/howto/connection/index.rst ########## @@ -61,29 +61,162 @@ to the connection you wish to edit in the connection list. Modify the connection properties and click the ``Save`` button to save your changes. -Creating a Connection with Environment Variables +Storing a Connection in Environment Variables ------------------------------------------------ -Connections in Airflow pipelines can be created using environment variables. -The environment variable needs to have a prefix of ``AIRFLOW_CONN_`` for -Airflow with the value in a URI format to use the connection properly. +The environment variable naming convention is ``AIRFLOW_CONN_<conn_id>``, all uppercase. -When referencing the connection in the Airflow pipeline, the ``conn_id`` -should be the name of the variable without the prefix. For example, if the -``conn_id`` is named ``postgres_master`` the environment variable should be -named ``AIRFLOW_CONN_POSTGRES_MASTER`` (note that the environment variable -must be all uppercase). +So if your connection id is ``my_prod_db`` then the variable name should be ``AIRFLOW_CONN_MY_PROD_DB``. -Airflow assumes the value returned from the environment variable to be in a URI -format (e.g. ``postgres://user:password@localhost:5432/master`` or -``s3://accesskey:secretkey@S3``). The underscore character is not allowed -in the scheme part of URI, so it must be changed to a hyphen character -(e.g. ``google-compute-platform`` if ``conn_type`` is ``google_compute_platform``). -Query parameters are parsed to one-dimensional dict and then used to fill extra. +.. note:: + + There is only one underscore surrounding ``CONN``. This is in contrast with the way ``airflow.cfg`` + parameters are stored, where two underscores surround the config section name. + +The value of this environment variable must use airflow's URI format for connections. See the section +:ref:`Generating a Connection URI <generating_connection_uri>` for more details. + +Notes +^^^^^ + +If storing the environment variable in something like ``~/.bashrc``, you would add it ike so: + +.. code-block:: bash + + export AIRFLOW_CONN_MY_PROD_DATABASE='my-conn-type://login:password@host:port/schema?param1=val1¶m2=val2' + +If using with a docker ``.env`` file, you may need to remove the single quotes. + +.. code-block:: + + AIRFLOW_CONN_MY_PROD_DATABASE=my-conn-type://login:password@host:port/schema?param1=val1¶m2=val2 + + +Creating a Connection from the CLI +---------------------------------- + +You may add a connection to the database from the CLI. + +Obtain the URI for your connection (see :ref:`Generating a Connection URI <generating_connection_uri>`). + +Then add connection like so: + +.. code-block:: bash + + airflow connections add --conn_id 'my_prod_db' --conn_uri 'my-conn-type://login:password@host:port/schema?param1=val1¶m2=val2' + +Alternatively you may specify each parameter individually: + +.. code-block:: bash + + airflow connections add \ + --conn_id 'my_prod_db' \ + --login 'login' \ + --password 'password' \ + ... + + +Connection URI format +--------------------- + +In general, Airflow's URI format is like so: + +.. code-block:: + + my-conn-type://my-login:my-password@my-host:5432/my-schema?param1=val1¶m2=val2 + +.. note:: + + The params param1 and param2 are just examples; you may supply arbitrary urlencoded json-serializable data there. + +The above URI would produce a ``Connection`` object equivalent to the following: + +.. code-block:: python + + Connection( + conn_id='', + conn_type='my_conn_type', + login='my-login', + password='my-password', + host='my-host', + port=5432, + schema='my-schema', + extra=json.dumps(dict(param1='val1', param2='val2')) + ) + +You can verify a URI is parsed correctly like so: + +.. code-block:: python Review comment: ```suggestion .. code-block:: pycon ``` ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
