bugraoz93 commented on PR #63677: URL: https://github.com/apache/airflow/pull/63677#issuecomment-4070086741
At the current stage, I agree with Jens, as we are not aiming to use Dags directly from files in the API. The design of how DagBundles work and how we are fetching in the API aligns. Moving files to the API, which DagProcessor manages the DagBundles over files, can open a can of worms, too, since by design, we should have a separate DagProcessor deployed after v3. This can eliminate isolation while causing certain synchronisation issues between your process and DagBundles. You should be able to achieve the same thing by using API endpoints in your code and acquire the DagBag from the API itself, without even loading it from files directly. Load DagBag in API: https://github.com/apache/airflow/blob/b87ed1d2c456bd23e8f855d90a38294731c05b7e/airflow-core/src/airflow/api_fastapi/common/dagbag.py#L38-L41 Get All Dags (even with good filters) https://github.com/apache/airflow/blob/b87ed1d2c456bd23e8f855d90a38294731c05b7e/airflow-core/src/airflow/api_fastapi/core_api/routes/public/dags.py#L86 Even though there is a specific API for UI only and it doesn't provide backwards compatibility, as I suggest using the Core API even within the UI scope would make more sense from a stability and maintenance perspective. https://github.com/apache/airflow/blob/main/airflow-core/src/airflow/api_fastapi/core_api/routes/ui/dags.py -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
