potiuk commented on issue #33688:
URL: https://github.com/apache/airflow/issues/33688#issuecomment-1693024376
OK. That gives us some clue. It seems that there are huge delays (20-30
seconds) between some steps that in 2.6.3 have been almost immediate:
2.7.0
```
[2023-08-23, 18:23:04 UTC] {standard_task_runner.py:85} INFO - Job 1180119:
Subtask fetch_header
+ 29 s
[2023-08-23, 18:23:35 UTC] {task_command.py:415} INFO - Running
<TaskInstance: mydagname.fetch_header
mydagname__bot_extract_sales__v10678108plzv__2023-08-21T20:36:29.046083
[running]> on host 90c66b7612c1
+ 29 s
[2023-08-23, 18:24:06 UTC] {taskinstance.py:1660} INFO - Exporting env vars:
AIRFLOW_CTX_DAG_EMAIL='[email protected]' AIRFLOW_CTX_DAG_OWNER='admin'
AIRFLOW_CTX_DAG_ID='mydagname' AIRFLOW_CTX_TASK_ID='fetch_header'
AIRFLOW_CTX_EXECUTION_DATE='2023-08-22T01:36:29+00:00'
AIRFLOW_CTX_TRY_NUMBER='1'
AIRFLOW_CTX_DAG_RUN_ID='mydagname__bot_extract_sales__v10678108plzv__2023-08-21T20:36:29.046083'
[2023-08-23, 18:24:06 UTC] {logging_mixin.py:151} INFO - {censored }
```
Compare it with 2.6.3
```
[2023-08-23, 19:28:45 UTC] {standard_task_runner.py:85} INFO - Job 1182100:
Subtask fetch_header
[2023-08-23, 19:28:45 UTC] {task_command.py:410} INFO - Running
<TaskInstance: mydagname.fetch_header
mydagname__bot_extract_sales__v10678132plzv__2023-08-21T20:36:30.234195
[running]> on host 9b9bc0eb214e
[2023-08-23, 19:28:46 UTC] {taskinstance.py:1545} INFO - Exporting env vars:
AIRFLOW_CTX_DAG_EMAIL='[email protected]' AIRFLOW_CTX_DAG_OWNER='admin'
AIRFLOW_CTX_DAG_ID='mydagname' AIRFLOW_CTX_TASK_ID='fetch_header'
AIRFLOW_CTX_EXECUTION_DATE='2023-08-22T01:36:30+00:00'
AIRFLOW_CTX_TRY_NUMBER='2'
AIRFLOW_CTX_DAG_RUN_ID='mydagname__bot_extract_sales__v10678132plzv__2023-08-21T20:36:30.234195'
[2023-08-23, 19:28:46 UTC] {logging_mixin.py:150} INFO - { censored }
```
So at least we might come up with some hypotheses. I will take a closer look
but maybe others migh have some ideas.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
