Re: [PR] Fix HA scheduler try_number double increment [airflow]
yennysu commented on PR #60330: URL: https://github.com/apache/airflow/pull/60330#issuecomment-3835593659 > > @ephraimbuddy Unfortunately, even after applying this fix, we still run onto the same issue. > > Major problem with the issue is reliability of reproduction path. I have reproduced it once but trying again, I can't. Do you have steps for reliable reproduction? While we haven’t been able to reproduce this reliably, we’ve observed that the frequency of occurrences seems to scale with the number of schedulers running. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Fix HA scheduler try_number double increment [airflow]
dabla commented on PR #60330:
URL: https://github.com/apache/airflow/pull/60330#issuecomment-3777395890
Now we also experience the same issue with HttpOperator:
```
[2026-01-21 04:44:19] INFO - Calling HTTP method
[2026-01-21 04:44:19] INFO - The hook_class
'airflow.providers.http.hooks.http.HttpHook' is not fully initialized (UI
widgets will be missing), because the 'flask_appbuilder' package is not
installed, however it is not required for Airflow components to work
[2026-01-21 04:44:25] ERROR - Server indicated the task shouldn't be running
anymore. Terminating process
detail={"detail":{"reason":"not_found","message":"Task Instance not found"}}
[2026-01-21 04:44:30] ERROR - Task killed!
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Fix HA scheduler try_number double increment [airflow]
ephraimbuddy commented on PR #60330: URL: https://github.com/apache/airflow/pull/60330#issuecomment-3755022534 > @ephraimbuddy Unfortunately, even after applying this fix, we still run onto the same issue. Major problem with the issue is reliability of reproduction path. I have reproduced it once but trying again, I can't. Do you have steps for reliable reproduction? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Fix HA scheduler try_number double increment [airflow]
dabla commented on PR #60330: URL: https://github.com/apache/airflow/pull/60330#issuecomment-3750385613 @ephraimbuddy Unfortunately, even after applying this fix, we still run onto the same issue. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Fix HA scheduler try_number double increment [airflow]
dabla commented on PR #60330: URL: https://github.com/apache/airflow/pull/60330#issuecomment-3749736676 @ephraimbuddy Good job, we are actually testing this fix right now, keep you guys posted if it helped. The weird thing is it seems (we're not sure yet) that we only have this issue with DAG's that have tasks that use the WinRMOperator. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Fix HA scheduler try_number double increment [airflow]
ashb commented on code in PR #60330: URL: https://github.com/apache/airflow/pull/60330#discussion_r2676649066 ## airflow-core/src/airflow/models/dagrun.py: ## @@ -2063,25 +2063,30 @@ def schedule_tis( empty_ti_ids.append(ti.id) count = 0 - +# Don't only check if the TI.id is in id_chunk +# but also check if the TI.state is in the schedulable states. +# Plus, a scheduled empty operator should not be scheduled again. +non_null_schedulable_states = tuple(s for s in SCHEDULEABLE_STATES if s is not None) +schedulable_state_clause = or_( +TI.state.is_(None), +TI.state.in_(non_null_schedulable_states), +) Review Comment: We should do some benchmarking/checking of indexes on this query if you haven't already. TI state is almost certainly indexed, but we should check this with some EXPLAIN ANALYZE in a moderatly sized DB if we have one (something to the region of 1-2m TI rows?) Might also be worth checking if a `schedulable_state_clause = TI.state != TaskInstanceState.SCHEDULED` is enough and more-performant? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
