William Horton created AIRFLOW-2875:
---------------------------------------
Summary: Env variables should have percent signs escaped before
writing to tmp config
Key: AIRFLOW-2875
URL: https://issues.apache.org/jira/browse/AIRFLOW-2875
Project: Apache Airflow
Issue Type: Bug
Components: configuration
Environment: Ubuntu
Airflow 1.10rc2
Reporter: William Horton
I encountered this when I was using an environment variable for
`AIRFLOW__CELERY__BROKER_URL`. The airflow worker was able to run and
communicate with the SQS queue, but when it received a task and began to run
it, I encountered an error with this trace:
```
[2018-08-08 15:19:24,402] \{base_task_runner.py:108} INFO - Job 13898: Subtask
mirroring Traceback (most recent call last):
[2018-08-08 15:19:24,402] \{base_task_runner.py:108} INFO - Job 13898: Subtask
mirroring File "/opt/airflow/venv/bin/airflow", line 32, in <module>
[2018-08-08 15:19:24,402] \{base_task_runner.py:108} INFO - Job 13898: Subtask
mirroring args.func(args)
[2018-08-08 15:19:24,402] \{base_task_runner.py:108} INFO - Job 13898: Subtask
mirroring File
"/opt/airflow/venv/local/lib/python2.7/site-packages/airflow/utils/cli.py",
line 74, in wrapper
[2018-08-08 15:19:24,402] \{base_task_runner.py:108} INFO - Job 13898: Subtask
mirroring return f(*args, **kwargs)
[2018-08-08 15:19:24,402] \{base_task_runner.py:108} INFO - Job 13898: Subtask
mirroring File
"/opt/airflow/venv/local/lib/python2.7/site-packages/airflow/bin/cli.py", line
460, in run
[2018-08-08 15:19:24,403] \{base_task_runner.py:108} INFO - Job 13898: Subtask
mirroring conf.set(section, option, value)
[2018-08-08 15:19:24,403] \{base_task_runner.py:108} INFO - Job 13898: Subtask
mirroring File
"/opt/airflow/venv/local/lib/python2.7/site-packages/backports/configparser/__init__.py",
line 1239, in set
[2018-08-08 15:19:24,406] \{base_task_runner.py:108} INFO - Job 13898: Subtask
mirroring super(ConfigParser, self).set(section, option, value)
[2018-08-08 15:19:24,406] \{base_task_runner.py:108} INFO - Job 13898: Subtask
mirroring File
"/opt/airflow/venv/local/lib/python2.7/site-packages/backports/configparser/__init__.py",
line 914, in set
[2018-08-08 15:19:24,406] \{base_task_runner.py:108} INFO - Job 13898: Subtask
mirroring value)
[2018-08-08 15:19:24,406] \{base_task_runner.py:108} INFO - Job 13898: Subtask
mirroring File
"/opt/airflow/venv/local/lib/python2.7/site-packages/backports/configparser/__init__.py",
line 392, in before_set
[2018-08-08 15:19:24,406] \{base_task_runner.py:108} INFO - Job 13898: Subtask
mirroring "position %d" % (value, tmp_value.find('%')))
[2018-08-08 15:19:24,406] \{base_task_runner.py:108} INFO - Job 13898: Subtask
mirroring ValueError: invalid interpolation syntax in <redacted-broker-url>
```
The issue was that the broker url had a percent sign, and when the cli called
`conf.set(section, option, value)`, it was throwing because it interpreted the
percent as an interpolation.
To avoid this issue, I would propose that the environment variables be escaped
when being written in `utils.configuration.tmp_configuration_copy`, so that
when `conf.set` is called in `bin/cli`, it doesn't throw on these unescaped
values.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
