If I have multiple airflow dags with some overlapping python package dependencies, how can I keep each of these project deps. decoupled? Eg. if I had project A and B on same server I would run each of them with something like...
source /path/to/virtualenv_a/activate python script_a.py deactivate source /path/to/virtualenv_b/activate python script_b.py deactivate Basically, would like to run dags with the same situation (eg. each dag uses python scripts that have may have overlapping package deps. that I would like to develop separately (ie. not have to update all code using a package when want to update the package just for one project)). Note, I've been using the BashOperator to run python tasks like... do_stuff = BashOperator( task_id='my_task', bash_command='python /path/to/script.py'), execution_timeout=timedelta(minutes=30), dag=dag) Is there a way to get this working? Is there some other best-practice way that airflow intendeds for people to address (or avoid) these kinds of problems? -- This electronic message is intended only for the named recipient, and may contain information that is confidential or privileged. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of the contents of this message is strictly prohibited. If you have received this message in error or are not the named recipient, please notify us immediately by contacting the sender at the electronic mail address noted above, and delete and destroy all copies of this message. Thank you.