rammmaa opened a new issue, #74851:
URL: https://github.com/apache/airflow/issues/74851
### Under which category would you file this issue?
Airflow Core
### Apache Airflow version
main (91a8ab6950); initially observed on 2.11.2
### What happened and how to reproduce it?
When git-sync atomically replaces the symlink used by the Dag folder during
file discovery, subdirectories can lose the ignore rules inherited from a
parent `.airflowignore`. Files that should be ignored are then included in the
discovery result.
In
`shared/module_loading/src/airflow_shared/module_loading/file_discovery.py`,
`_find_path_from_directory` stores inherited rules in `patterns_by_dir` using
resolved child paths. If the symlink target changes before discovery visits a
child, the lookup resolves into the new revision and returns an empty list of
inherited rules.
The following example reproduces the issue deterministically, without
threads or sleeps:
```python
import os
from pathlib import Path
from tempfile import TemporaryDirectory
from airflow._shared.module_loading import find_path_from_directory
with TemporaryDirectory() as directory:
base = Path(directory)
for revision in ("rev1", "rev2"):
root = base / revision
(root / "tests").mkdir(parents=True)
(root / ".airflowignore").write_text("tests/\n")
(root / "dag.py").touch()
(root / "tests" / "test_dag.py").touch()
dags = base / "dags"
dags.symlink_to(base / "rev1", target_is_directory=True)
paths = find_path_from_directory(dags, ".airflowignore", "regexp")
# Inherited rules have been stored for the child directories
# before discovery yields the root-level file.
first = next(paths)
replacement = base / "replacement"
replacement.symlink_to(base / "rev2", target_is_directory=True)
os.replace(replacement, dags)
print([
Path(path).relative_to(dags).as_posix()
for path in [first, *paths]
])
```
When running directly against the shared package, replace the import with:
```python
from airflow_shared.module_loading import find_path_from_directory
```
For Airflow 2, use:
```python
from airflow.utils.file import find_path_from_directory
```
Expected output:
```text
['dag.py']
```
Actual output after switching the symlink:
```text
['dag.py', 'tests/test_dag.py']
```
Without the symlink switch, the test file is correctly excluded.
With regexp syntax, `tests/` does not match the directory path `tests`
itself, so discovery descends into it and relies on inherited rules to exclude
its files. Glob patterns that exclude descendant files without pruning their
parent directory, such as `tests/**/*.py`, can also encounter this issue.
### What you think should happen instead?
Replacing a symlink during discovery should not silently discard ignore
rules already inherited from a parent directory.
A proposed fix is to key inherited rules by the walked path, which remains
stable across the symlink replacement, and track resolved paths separately for
the existing recursive-link detection.
This would preserve inherited rules without changing pattern matching or
removing recursive-link detection. It would not attempt to provide a consistent
filesystem snapshot across revisions.
### Operating System
Linux
### Deployment
Other
### Apache Airflow Provider(s)
_No response_
### Versions of Apache Airflow Providers
_No response_
### Official Helm Chart version
Not Applicable
### Kubernetes Version
_No response_
### Helm Chart configuration
_No response_
### Docker Image customizations
_No response_
### Anything else?
The affected deployment uses Kubernetes with a git-sync sidecar that
atomically updates a symlink-backed Dag folder.
An unexpectedly discovered test module imported a Dag factory. Parsing that
module reassigned the generated Dags to the test file. On the next scan, the
ignore rules applied normally, the test file disappeared from the discovery
result, and the associated Dags became inactive.
This downstream effect depends on what the discovered module imports. The
file-discovery bug itself can be reproduced independently using the example
above.
I have prepared a focused fix and regression tests for regexp and glob
syntax and intend to submit a PR.
### Are you willing to submit PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
