xlanor commented on issue #31205: URL: https://github.com/apache/airflow/issues/31205#issuecomment-1564129512
> > @ORuteMa That's very helpful - I will send this over to another team, we've been seeing this issue in 2.4.2 pop up as well and were curious as to what was causing it. > > > > > > Just curious, were you seeing duplicate task execution? We're seeing tasks get picked up twice by different airflow workers. > > > > Yes, I am certain that the task was executed repeatedly. Please take a look at the image below. > > <img width="1061" alt="image" src="https://github.com/apache/airflow/assets/30034544/dd17c940-2feb-4402-b859-6db0ed00ab02";> > > > > It can be observed that attempt 44 started before attempt 43 had finished. This clearly indicates that the Scheduler, while picking up runnable task instances, had at least two schedulers selecting the same task instance and distributing it concurrently, leading to a concurrency issue. Thanks a lot! This helped us with our issue - we're seeing this as well in 2.4.2 and we couldnt replicate it reliably either. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
