ColtenOuO commented on PR #64105:
URL: https://github.com/apache/airflow/pull/64105#issuecomment-5573171404
> I also noticed that using `os.environ` makes the askpass environment
process-wide, which could cause interference between concurrent Git operations.
>
> I considered switching to explicitly passing the environment to each Git
operation, but that would make this PR more invasive. Since the other askpass
helpers have the same pattern, I'm leaning towards addressing this consistently
in a separate PR.
>
> Would you prefer that I address it here, or would a follow-up PR be better?
I noticed that too, but I dug a bit deeper into this today.
Looking at how Airflow's DAG processor workflow operates, each process runs
independently. Because of this, it might not actually affect regular Airflow
users.
The issue would likely only surface if someone wrote a custom Python script
running multiple bundles concurrently in threads, something like this:
```python
from concurrent.futures import ThreadPoolExecutor
from airflow.providers.git.bundles.git import GitDagBundle
bundle_a = GitDagBundle(
name="team_a",
git_conn_id="team_a_git",
tracking_ref="main",
)
bundle_b = GitDagBundle(
name="team_b",
git_conn_id="team_b_git",
tracking_ref="main",
)
with ThreadPoolExecutor(max_workers=2) as executor:
executor.submit(bundle_a.refresh)
executor.submit(bundle_b.refresh)
```
Since typical Airflow users follow the standard DAG processor flow, the
real-world impact is probably minimal. Still, fixing this so it works reliably
whether it runs in the same process or across different processes might be a
nice improvement. (maybe)
These are just my findings and thoughts after looking into it -- take a look
when you have time in case I missed anything!
That said, I think getting input from the maintainers would probably be the
best next step :D
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
