Hi Rajesh, `airflow test` executes the task synchronously in the same python process (without touching the scheduler), which is probably why it went down the happy path.
Are you using celery executor and running the tasks in a distributed manner? It* sounds like* the task execution env doesn't know about the variable. Also depending on where you added the code to your custom hook, it may be in a different process from which the task is executed, since os.environ sets variable only for the scope/duration of the python process (and its children). On Fri, Dec 8, 2017 at 7:55 AM, Rajesh Chamarthi <[email protected] > wrote: > Hi all, > > What is the right way to set environment variables to be used in DAGS. > Also, is it possible to export them such that currently running scheduler > and worker processes can use it? > > Context : > > I am working on a DAG that involves making an Oracle connection, which > requires the LD_LIBRARY_PATH variable to be set. I added it to > /etc/profile.d of the airflow user. When I start a new shell + python > virtualenv and run “airflow test dag_name task_name”, the task is able to > use the path variable above and make the necessary connection. > > However, when it runs through the scheduler, the task fails with the > following error : “Oracle client library cannot be loaded”, presumably > because it cannot read the LD_LIBRARY_PATH. > > 2) I have also tried the following code in my custom hook, to explicitly > set the path. I see the same error. > > if 'LD_LIBRARY_PATH' not in os.environ.keys(): > os.environ['LD_LIBRARY_PATH']="/opt/oracle/instantclient_11_2" > logging.info("added ld library path", os.environ['LD_LIBRARY_PATH']) > > Any pointers on how I can resolve this? > > Thanks, > Rajesh > >
