mhenc commented on code in PR #26509:
URL: https://github.com/apache/airflow/pull/26509#discussion_r978494229
##########
airflow/models/dag.py:
##########
@@ -1189,11 +1195,25 @@ def relative_fileloc(self) -> pathlib.Path:
"""File location of the importable dag 'file' relative to the
configured DAGs folder."""
path = pathlib.Path(self.fileloc)
try:
- return path.relative_to(settings.DAGS_FOLDER)
+ rel_path = path.relative_to(self.dag_processor_dags_folder or
settings.DAGS_FOLDER)
+ if rel_path == pathlib.Path('.'):
+ return path
+ else:
+ return rel_path
except ValueError:
# Not relative to DAGS_FOLDER.
return path
+ @property
+ def dag_processor_dags_folder(self):
Review Comment:
> to make the subdir agree with configured dags folder. e.g. setting subdir
on dag processor command does not force dags_folder to agree. so fundamentally,
they are two different things right? the dag processor can run without them
agreeing, no?
Right, there is no logic/check, we just pass --subdir to the Dag Processor
and it parses the dags from this directory.
> if you can't use relative loc, then the worker can't really rely on ser
dag to know where a dag is located.
Well, if you define different set of workers on different task queues then
it may work, but I agree there is no support from Airflow perspective for such
setup.
But note that dag processor seperation (and multiple dag processors) is just
the first step towards Airflow multit-tenancy. At some point workers isolation
will also be supported.
cc: @potiuk
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
