potiuk commented on code in PR #40916:
URL: https://github.com/apache/airflow/pull/40916#discussion_r1689595912
##########
airflow/dag_processing/processor.py:
##########
@@ -837,13 +877,17 @@ def process_file(
:return: number of dags found, count of import errors, last number of
db queries
"""
self.log.info("Processing file %s for tasks to queue", file_path)
+ try:
+ if InternalApiConfig.get_use_internal_api():
+ dagbag = DagFileProcessor._get_dagbag(file_path)
+ else:
+ with create_session() as session:
Review Comment:
The context manager approach I introduced looks like a very nice approach
for that BTW. I thinl the way I changed it where each DB method counts queries
separately (also to account for the fact it can be remote). And there actually
- session is perfectly fine). The DAGBag parsing should be additionally wrapped
with "engine" event I think to get the queries counted while parsing (but it
should be done conditionally - only when database isolation is disabled
(because there - there will be no engine to register event in the dag file
processor.
So you will likely need a context manager and wrap parsing with it and that
context manager should register event and retrieve the count if db_isolation is
off.
Does it make sense @MaksYermak ? Would you follow-up with it (I am now
working on runinng db isolation tests and we will need that one rather quickly
if we want to make it for Airflow 2.10.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
