I see, thanks for the info. On Fri, Nov 1, 2019 at 3:46 AM James Meickle <[email protected]> wrote:
> All of the Airflow executors use subprocesses (at minimum) for launching > tasks. That means that setting environment variables will not propagate > back "up" to the parent process. > > On Thu, Oct 31, 2019 at 9:48 PM Reed Villanueva <[email protected]> > wrote: > >> What would happen if had airflow dags running concurrently where tasks >> (say BashOperators) each modified the same env variable? Would both >> tasks see the change from the other? Would airflow run into race conditions >> or would the env changes only affect the tasks' scripts in isolation? >> >> I would normally think that the env changes would be self contained in >> this case >> <https://gist.github.com/reedv/4d1187af80958f05042047ab300d5ce9>, but >> asking because did have a problem with ariflow ( >> http://mail-archives.apache.org/mod_mbox/airflow-users/201910.mbox/%3CCAKqkst9C7v-_i6RX+pEGEL=we6rqqug=3ne5mqzndngzjm9...@mail.gmail.com%3E) >> where >> setting os.environ[...] values (at the dag definition level) affected >> all dags. >> >> This electronic message is intended only for the named >> recipient, and may contain information that is confidential or >> privileged. If you are not the intended recipient, you are >> hereby notified that any disclosure, copying, distribution or >> use of the contents of this message is strictly prohibited. If >> you have received this message in error or are not the named >> recipient, please notify us immediately by contacting the >> sender at the electronic mail address noted above, and delete >> and destroy all copies of this message. Thank you. >> > -- This electronic message is intended only for the named recipient, and may contain information that is confidential or privileged. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of the contents of this message is strictly prohibited. If you have received this message in error or are not the named recipient, please notify us immediately by contacting the sender at the electronic mail address noted above, and delete and destroy all copies of this message. Thank you.
