zach-overflow commented on issue #57347: URL: https://github.com/apache/airflow/issues/57347#issuecomment-3456856138
I'll chime in that we are observing a similar behavior, and we have a similar setup (mutliple dedicated "tenant" namespaces, and a singular "core" namespace where the scheduler and shared components run), but with the Celery executor, and the majority of tasks running within the celery workers, rather than as ephemeral k8s task pods. We are running on Airflow version 3.0.6 and are observing the same error, despite increasing the `parallelism` value. It almost seems like the way the base executor tracks "running" tasks is not accounting for task removals properly. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
