All of the Airflow executors use subprocesses (at minimum) for launching tasks. That means that setting environment variables will not propagate back "up" to the parent process.
On Thu, Oct 31, 2019 at 9:48 PM Reed Villanueva <[email protected]> wrote: > What would happen if had airflow dags running concurrently where tasks > (say BashOperators) each modified the same env variable? Would both tasks > see the change from the other? Would airflow run into race conditions or > would the env changes only affect the tasks' scripts in isolation? > > I would normally think that the env changes would be self contained in > this case <https://gist.github.com/reedv/4d1187af80958f05042047ab300d5ce9>, > but asking because did have a problem with ariflow ( > http://mail-archives.apache.org/mod_mbox/airflow-users/201910.mbox/%3CCAKqkst9C7v-_i6RX+pEGEL=we6rqqug=3ne5mqzndngzjm9...@mail.gmail.com%3E) > where > setting os.environ[...] values (at the dag definition level) affected all > dags. > > This electronic message is intended only for the named > recipient, and may contain information that is confidential or > privileged. If you are not the intended recipient, you are > hereby notified that any disclosure, copying, distribution or > use of the contents of this message is strictly prohibited. If > you have received this message in error or are not the named > recipient, please notify us immediately by contacting the > sender at the electronic mail address noted above, and delete > and destroy all copies of this message. Thank you. >
