Nathan Scully created AIRFLOW-253: ------------------------------------- Summary: Job does not find CONN_ID environmental variable Key: AIRFLOW-253 URL: https://issues.apache.org/jira/browse/AIRFLOW-253 Project: Apache Airflow Issue Type: Bug Components: hooks Affects Versions: 1.7.1.3 Environment: OS: Ubuntu 14.04 airflow@ip-10-0-40-185:~$ pip show airflow --- Name: airflow Version: 1.7.1.3 Reporter: Nathan Scully
I am not sure if this is directly an airflow issue or something causing havoc in my environment but when the scheduler triggers a test job I have setup (@hourly) to ping a Redshift server with a 'select * from x limit 1;' I receive a no conn_id found. However, I can echo the environmental var and if I run the job as a test, or manually query from python console I can resolve it. Any ideas what might be causing it to be missed? Error from AIRFLOW scheduled run: {code} [2016-06-17 06:00:04,402] {models.py:1286} ERROR - The conn_id `OF_REDSHIFT` isn't defined Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/airflow/models.py", line 1245, in run result = task_copy.execute(context=context) File "/usr/local/lib/python2.7/dist-packages/airflow/operators/python_operator.py", line 66, in execute return_value = self.python_callable(*self.op_args, **self.op_kwargs) File "/usr/local/airflow/dags/ping_redshift.py", line 23, in ping_redshift_query df = ph.get_records(sql) File "/usr/local/lib/python2.7/dist-packages/airflow/hooks/dbapi_hook.py", line 72, in get_records conn = self.get_conn() File "/usr/local/lib/python2.7/dist-packages/airflow/hooks/postgres_hook.py", line 18, in get_conn conn = self.get_connection(self.postgres_conn_id) File "/usr/local/lib/python2.7/dist-packages/airflow/hooks/base_hook.py", line 51, in get_connection conn = random.choice(cls.get_connections(conn_id)) File "/usr/local/lib/python2.7/dist-packages/airflow/hooks/base_hook.py", line 39, in get_connections "The conn_id `{0}` isn't defined".format(conn_id)) AirflowException: The conn_id `OF_REDSHIFT` isn't defined [2016-06-17 06:00:04,403] {models.py:1306} INFO - Marking task as FAILED. [2016-06-17 06:00:04,695] {models.py:1327} ERROR - The conn_id `OF_REDSHIFT` isn't defined {code} I can echo the env variable: {code} airflow@ip-xx-x-xx-xxx:~$ echo $AIRFLOW_CONN_OF_REDSHIFT postgres://airflow:xxx@xxx:5439/xxx {code} And can also get it through python console: {code} airflow@ip-xx-x-xx-xxx:~$ python Python 2.7.6 (default, Jun 22 2015, 17:58:13) [GCC 4.8.2] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import os >>> CONN_ENV_PREFIX = 'AIRFLOW_CONN_' >>> conn_id ='OF_REDSHIFT' >>> os.environ.get(CONN_ENV_PREFIX + conn_id.upper()) 'postgres://airflow:xxx@xxx:5439/xxx' {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)