[
https://issues.apache.org/jira/browse/AIRFLOW-253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nathan Scully resolved AIRFLOW-253.
-----------------------------------
Resolution: Not A Problem
Closing as I worked out what the issue was.
I did not include the env AIRFLOW_CONN_OF_REDSHIFT=... and export
AIRFLOW_CONN_OF_REDSHIFT in the /etc/init/airflow-scheduler.conf upstart file
as per http://upstart.ubuntu.com/cookbook/#id98
> Job does not find CONN_ID environmental variable
> ------------------------------------------------
>
> Key: AIRFLOW-253
> URL: https://issues.apache.org/jira/browse/AIRFLOW-253
> Project: Apache Airflow
> Issue Type: Bug
> Components: hooks
> Affects Versions: Airflow 1.7.1.3
> Environment: OS: Ubuntu 14.04
> airflow@ip-10-0-40-185:~$ pip show airflow
> ---
> Name: airflow
> Version: 1.7.1.3
> Reporter: Nathan Scully
>
> I am not sure if this is directly an airflow issue or something causing havoc
> in my environment but when the scheduler triggers a test job I have setup
> (@hourly) to ping a Redshift server with a 'select * from x limit 1;' I
> receive a no conn_id found. However, I can echo the environmental var and if
> I run the job as a test, or manually query from python console I can resolve
> it. Any ideas what might be causing it to be missed?
> Error from AIRFLOW scheduled run:
> {code}
> [2016-06-17 06:00:04,402] {models.py:1286} ERROR - The conn_id
> `OF_REDSHIFT` isn't defined
> Traceback (most recent call last):
> File "/usr/local/lib/python2.7/dist-packages/airflow/models.py", line
> 1245, in run
> result = task_copy.execute(context=context)
> File
> "/usr/local/lib/python2.7/dist-packages/airflow/operators/python_operator.py",
> line 66, in execute
> return_value = self.python_callable(*self.op_args, **self.op_kwargs)
> File "/usr/local/airflow/dags/ping_redshift.py", line 23, in
> ping_redshift_query
> df = ph.get_records(sql)
> File
> "/usr/local/lib/python2.7/dist-packages/airflow/hooks/dbapi_hook.py", line
> 72, in get_records
> conn = self.get_conn()
> File
> "/usr/local/lib/python2.7/dist-packages/airflow/hooks/postgres_hook.py", line
> 18, in get_conn
> conn = self.get_connection(self.postgres_conn_id)
> File
> "/usr/local/lib/python2.7/dist-packages/airflow/hooks/base_hook.py", line 51,
> in get_connection
> conn = random.choice(cls.get_connections(conn_id))
> File
> "/usr/local/lib/python2.7/dist-packages/airflow/hooks/base_hook.py", line 39,
> in get_connections
> "The conn_id `{0}` isn't defined".format(conn_id))
> AirflowException: The conn_id `OF_REDSHIFT` isn't defined
> [2016-06-17 06:00:04,403] {models.py:1306} INFO - Marking task as FAILED.
> [2016-06-17 06:00:04,695] {models.py:1327} ERROR - The conn_id
> `OF_REDSHIFT` isn't defined
> {code}
> I can echo the env variable:
> {code}
> airflow@ip-xx-x-xx-xxx:~$ echo $AIRFLOW_CONN_OF_REDSHIFT
> postgres://airflow:xxx@xxx:5439/xxx
> {code}
> And can also get it through python console:
> {code}
> airflow@ip-xx-x-xx-xxx:~$ python
> Python 2.7.6 (default, Jun 22 2015, 17:58:13)
> [GCC 4.8.2] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
> >>> import os
> >>> CONN_ENV_PREFIX = 'AIRFLOW_CONN_'
> >>> conn_id ='OF_REDSHIFT'
> >>> os.environ.get(CONN_ENV_PREFIX + conn_id.upper())
> 'postgres://airflow:xxx@xxx:5439/xxx'
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
