davlum opened a new pull request #15013: URL: https://github.com/apache/airflow/pull/15013
Currently using the Vault secrets backends requires that users store the secrets in connection URI format: https://airflow.apache.org/docs/apache-airflow/stable/howto/connection.html#connection-uri-format Unfortunately the connection URI format is not capable of expressing all vales of the Connection class. In particular the Connection class allows for arbitrary string values for the `extra` parameter, while the URI format requires that this parameter be unnested JSON so that it can serialize into query parameters. ``` >>> Connection(conn_id='id', conn_type='http', extra='foobar').get_uri() [2021-03-25 13:31:07,535] {connection.py:337} ERROR - Expecting value: line 1 column 1 (char 0) Traceback (most recent call last): File "/Users/da.lum/code/python/airflow/airflow/models/connection.py", line 335, in extra_dejson obj = json.loads(self.extra) File "/nix/store/8kzdflq0v06fq0mh9m2fd73gnyqp57xr-python3-3.7.3/lib/python3.7/json/__init__.py", line 348, in loads return _default_decoder.decode(s) File "/nix/store/8kzdflq0v06fq0mh9m2fd73gnyqp57xr-python3-3.7.3/lib/python3.7/json/decoder.py", line 337, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/nix/store/8kzdflq0v06fq0mh9m2fd73gnyqp57xr-python3-3.7.3/lib/python3.7/json/decoder.py", line 355, in raw_decode raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0) [2021-03-25 13:31:07,535] {connection.py:338} ERROR - Failed parsing the json for conn_id id 'http://' ``` As shown, the `extra` data is missing from the return value `http://`. Although there is an error logged, this does not help users who were previously able to store other data. <!-- Thank you for contributing! Please make sure that your code changes are covered with tests. And in case of new features or big changes remember to adjust the documentation. Feel free to ping committers for the review! In case of existing issue, reference it using one of the following: closes: #ISSUE related: #ISSUE How to write a good git commit message: http://chris.beams.io/posts/git-commit/ --> --- **^ Add meaningful description above** Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)** for more information. In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
