potiuk commented on issue #30884: URL: https://github.com/apache/airflow/issues/30884#issuecomment-1525420870
OK. That would explain it then. It is then indeed the same issue as #30593 but the effect of it has been magnified by the fact that your job_heartbeeat sec is THAT long. > Dag processor in 2.5.3 rollout always showed duration for all parse tasks around 56-60 secs, but I didn't know it is that setting that regulates that btw. Yes. That was a bug I introduced when fixing the liveness check for standalone dag file processor in https://github.com/apache/airflow/pull/30278 - we are using that heartbeat historically for all the jobs (it's also used by local_task_job and triggerer btw) so the section where it is is actually wrong and we should move it out of scheduler section. Fix is here: https://github.com/apache/airflow/pull/30899 There are few workarounds for that: * migrate to 2.6.0 when ready (we are going to cancel rc2 and prepare rc3 shortly * apply the #30899 - but it is in a slightly different part of the code (you'd have to apply it to https://github.com/apache/airflow/pull/30278/files#diff-827b7a469438ffd1b172ae295134a84c8a914c9ea4c9ea2a7d3de1f1d5aa6bb6R580 in 2.5.3) * decrease the job_heartbeat_sec -> 60 seconds seems a bit excessive, scheduler should rarely run scheduler lopp for longer than a few seconds, do you know why you have that long of a heartbeat expectations ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
