Hi Rajesh, Odd that it still doesn't work after setting the worker's env profile. We run LocalExecutor so this has never been an issue. I will leave to others to comment.
(One thing worth trying is to explicitly export the variable `export VAR=VAL` in a subprocess in the operator/hook so that it lives beyond the scope of the python process.) Cheers, Joy On Fri, Dec 8, 2017 at 4:32 PM, Rajesh Chamarthi <[email protected] > wrote: > Hi Joy, > > Thank you, “airflow test” does not seem like a good way to test > environment in that case :) > > Yes, I am running using celery executor and running the tasks in a > distributed manner. I currently set the environment variable in the hook > just before it tries to get a DB connection. > > We also use supervisord to manage the worker, scheduler and web server > processes and I have changed the worker’s env profile to set the > environment variable, but it does not work either. > > > > On Dec 8, 2017, at 2:50 PM, Joy Gao <[email protected]> wrote: > > > > Hi Rajesh, > > > > `airflow test` executes the task synchronously in the same python process > > (without touching the scheduler), which is probably why it went down the > > happy path. > > > > Are you using celery executor and running the tasks in a distributed > > manner? It* sounds like* the task execution env doesn't know about the > > variable. > > > > Also depending on where you added the code to your custom hook, it may be > > in a different process from which the task is executed, since os.environ > > sets variable only for the scope/duration of the python process (and its > > children). > > > > > > On Fri, Dec 8, 2017 at 7:55 AM, Rajesh Chamarthi < > [email protected] > >> wrote: > > > >> Hi all, > >> > >> What is the right way to set environment variables to be used in DAGS. > >> Also, is it possible to export them such that currently running > scheduler > >> and worker processes can use it? > >> > >> Context : > >> > >> I am working on a DAG that involves making an Oracle connection, which > >> requires the LD_LIBRARY_PATH variable to be set. I added it to > >> /etc/profile.d of the airflow user. When I start a new shell + python > >> virtualenv and run “airflow test dag_name task_name”, the task is able > to > >> use the path variable above and make the necessary connection. > >> > >> However, when it runs through the scheduler, the task fails with the > >> following error : “Oracle client library cannot be loaded”, presumably > >> because it cannot read the LD_LIBRARY_PATH. > >> > >> 2) I have also tried the following code in my custom hook, to explicitly > >> set the path. I see the same error. > >> > >> if 'LD_LIBRARY_PATH' not in os.environ.keys(): > >> os.environ['LD_LIBRARY_PATH']="/opt/oracle/instantclient_11_2" > >> logging.info("added ld library path", os.environ['LD_LIBRARY_PATH']) > >> > >> Any pointers on how I can resolve this? > >> > >> Thanks, > >> Rajesh > >> > >> > >
