Hi all,
What is the right way to set environment variables to be used in DAGS.
Also, is it possible to export them such that currently running scheduler and
worker processes can use it?
Context :
I am working on a DAG that involves making an Oracle connection, which requires
the LD_LIBRARY_PATH variable to be set. I added it to /etc/profile.d of the
airflow user. When I start a new shell + python virtualenv and run “airflow
test dag_name task_name”, the task is able to use the path variable above and
make the necessary connection.
However, when it runs through the scheduler, the task fails with the following
error : “Oracle client library cannot be loaded”, presumably because it cannot
read the LD_LIBRARY_PATH.
2) I have also tried the following code in my custom hook, to explicitly set
the path. I see the same error.
if 'LD_LIBRARY_PATH' not in os.environ.keys():
os.environ['LD_LIBRARY_PATH']="/opt/oracle/instantclient_11_2"
logging.info("added ld library path", os.environ['LD_LIBRARY_PATH'])
Any pointers on how I can resolve this?
Thanks,
Rajesh
