Hi Joy, Thank you, “airflow test” does not seem like a good way to test environment in that case :)
Yes, I am running using celery executor and running the tasks in a distributed manner. I currently set the environment variable in the hook just before it tries to get a DB connection. We also use supervisord to manage the worker, scheduler and web server processes and I have changed the worker’s env profile to set the environment variable, but it does not work either. > On Dec 8, 2017, at 2:50 PM, Joy Gao <[email protected]> wrote: > > Hi Rajesh, > > `airflow test` executes the task synchronously in the same python process > (without touching the scheduler), which is probably why it went down the > happy path. > > Are you using celery executor and running the tasks in a distributed > manner? It* sounds like* the task execution env doesn't know about the > variable. > > Also depending on where you added the code to your custom hook, it may be > in a different process from which the task is executed, since os.environ > sets variable only for the scope/duration of the python process (and its > children). > > > On Fri, Dec 8, 2017 at 7:55 AM, Rajesh Chamarthi <[email protected] >> wrote: > >> Hi all, >> >> What is the right way to set environment variables to be used in DAGS. >> Also, is it possible to export them such that currently running scheduler >> and worker processes can use it? >> >> Context : >> >> I am working on a DAG that involves making an Oracle connection, which >> requires the LD_LIBRARY_PATH variable to be set. I added it to >> /etc/profile.d of the airflow user. When I start a new shell + python >> virtualenv and run “airflow test dag_name task_name”, the task is able to >> use the path variable above and make the necessary connection. >> >> However, when it runs through the scheduler, the task fails with the >> following error : “Oracle client library cannot be loaded”, presumably >> because it cannot read the LD_LIBRARY_PATH. >> >> 2) I have also tried the following code in my custom hook, to explicitly >> set the path. I see the same error. >> >> if 'LD_LIBRARY_PATH' not in os.environ.keys(): >> os.environ['LD_LIBRARY_PATH']="/opt/oracle/instantclient_11_2" >> logging.info("added ld library path", os.environ['LD_LIBRARY_PATH']) >> >> Any pointers on how I can resolve this? >> >> Thanks, >> Rajesh >> >>
