GitHub user shaealh added a comment to the discussion: Airflow task failed but spark kube app is running
@rcrchawla Thanks for the details. Reducing schedulers from 2 to 1 may reduce some DB activity, it won’t fix your issue since it seems to be heartbeat/API related can you try reducing worker_concurrency from 16 to 8 or 10 (unfortunately behavior in Airflow 3.0.4 is not always comparable to 2.2.2 at the same concurrency) you can also try increasing task_instance_heartbeat_timeout from 900 to 1800 GitHub link: https://github.com/apache/airflow/discussions/63298#discussioncomment-16092752 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
