Will give this a try. I tried the airflow worker’s conf and it did d not work.
I fixed my problem (temporarily) by setting the environment variable in the python __init__ code before it calls the database connection create statement. (Using python os.environ). On Mon, Dec 11, 2017 at 9:50 AM William Pursell <[email protected]> wrote: > It's perhaps not the cleanest solution, but I have had success setting the > environment in the supervisord config file. Eg: add the line > > environment=LD_LIBRARY_PATH="/opt/oracle/instantclient_11_2" > > On Fri, Dec 8, 2017 at 4:32 PM, Rajesh Chamarthi < > [email protected] > > wrote: > > > Hi Joy, > > > > Thank you, “airflow test” does not seem like a good way to test > > environment in that case :) > > > > Yes, I am running using celery executor and running the tasks in a > > distributed manner. I currently set the environment variable in the hook > > just before it tries to get a DB connection. > > > > We also use supervisord to manage the worker, scheduler and web server > > processes and I have changed the worker’s env profile to set the > > environment variable, but it does not work either. > > > > > > > On Dec 8, 2017, at 2:50 PM, Joy Gao <[email protected]> wrote: > > > > > > Hi Rajesh, > > > > > > `airflow test` executes the task synchronously in the same python > process > > > (without touching the scheduler), which is probably why it went down > the > > > happy path. > > > > > > Are you using celery executor and running the tasks in a distributed > > > manner? It* sounds like* the task execution env doesn't know about the > > > variable. > > > > > > Also depending on where you added the code to your custom hook, it may > be > > > in a different process from which the task is executed, since > os.environ > > > sets variable only for the scope/duration of the python process (and > its > > > children). > > > > > > > > > On Fri, Dec 8, 2017 at 7:55 AM, Rajesh Chamarthi < > > [email protected] > > >> wrote: > > > > > >> Hi all, > > >> > > >> What is the right way to set environment variables to be used in DAGS. > > >> Also, is it possible to export them such that currently running > > scheduler > > >> and worker processes can use it? > > >> > > >> Context : > > >> > > >> I am working on a DAG that involves making an Oracle connection, which > > >> requires the LD_LIBRARY_PATH variable to be set. I added it to > > >> /etc/profile.d of the airflow user. When I start a new shell + python > > >> virtualenv and run “airflow test dag_name task_name”, the task is able > > to > > >> use the path variable above and make the necessary connection. > > >> > > >> However, when it runs through the scheduler, the task fails with the > > >> following error : “Oracle client library cannot be loaded”, presumably > > >> because it cannot read the LD_LIBRARY_PATH. > > >> > > >> 2) I have also tried the following code in my custom hook, to > explicitly > > >> set the path. I see the same error. > > >> > > >> if 'LD_LIBRARY_PATH' not in os.environ.keys(): > > >> os.environ['LD_LIBRARY_PATH']="/opt/oracle/instantclient_11_2" > > >> logging.info("added ld library path", > os.environ['LD_LIBRARY_PATH']) > > >> > > >> Any pointers on how I can resolve this? > > >> > > >> Thanks, > > >> Rajesh > > >> > > >> > > > > >
