The interesting thing is that we didn't see this issue in 220.127.116.11 but we
did when upgrading to 1.8.0.
We aren't seeing any timeout on the task in question to be quite honest.
The state of the task never changes and we have reasonable timeouts on our
tasks that would notify us. The task is in fact "stuck" w/o reporting any
status. There are other cases where tasks do in fact fail and then go into
retry state, which we see normally (this happens quite a bit for us on
deploys). There is clearly some edge case here where the failure -> retry
does not happen and the dagrun never updates.
What we do see timeouts on Sensors that depend on those tasks and we've
added SLAs to some of our important tasks to see issues earlier.
Does anyone know where this code lives? Is that a function of the
On Mon, Aug 7, 2017 at 1:30 PM, Gerard Toonstra <gtoons...@gmail.com> wrote:
> Hi David,
> When tasks are put on the MQ, they are out of the control of the scheduler.
> The scheduler puts the state of that task instance in "queued".
> What happens next:
> 1. A worker picks up the task to run and tries to run it.
> 2. It first executes a couple of checks against the DB prior to executing
> this. These are final instance checks to see
> if it should still run when the worker is about to pick up the task
> (another could have processed, started processing, etc).
> 3. The worker puts the state of the TI in "running".
> 4. The worker does the work as described in the operator
> 5. The worker then updates the database with fail or success.
> If you kill the docker container doing the execution prior to it having
> updated the state to success or fail,
> it will get into a situation where a timeout must occur to get airflow to
> see if the task failed or not. This is because
> the worker is claiming to be processing the message, but this worker/task
> got killed.
> It is actually the task instance updating the database, so if you leave
> that container running, it will possibly finish
> and update the db.
> The task results are also communicated back to the executors and there's a
> check to see if the results agree.
> You can find this code in models.py / Taskinstance / run() and any
> Executor you are using under (airflow/executors).
> The reason why this happens I think is because docker doesn't really care
> what's running at the moment, it's assuming 'services',
> where you may have interruption of services because they are retried all
> the time anyway. In an environment like airflow,
> There's a persistent backend database that doesn't automatically retry
> because it's driven through the scheduler, which only sees
> a "RUNNING" record in the database.
> How to deal with this depends on your situation. If you run only short
> running tasks (up to 5 mins), you could drain the task queue
> by stopping the scheduler first. This means no new messages are sent to the
> queue, so after 10 mins you should have no tasks running on any workers.
> Another way is to update the database inbetween, but I'd personally avoid
> that as much as you can.
> Not sure if anyone wants to chime in here on how to best deal with this in
> On Mon, Aug 7, 2017 at 8:21 PM, David Klosowski <dav...@thinknear.com>
> > Hi Airflow Dev List:
> > Has anyone had cases where tasks get "stuck"? What I mean by "stuck" is
> > that tasks show as running through the Airflow UI but never actually run
> > (and dependent tasks will eventually timeout).
> > This only happens during our deployments and we replace all the hosts in
> > our stack (3 workers and 1 host with the scheduler + webserver + flower)
> > with a dockerized deployment. We've been deploying to the worker hosts
> > after the scheduler + webserver + flower host.
> > It also doesn't occur all the time, which is a bit frustrating to try to
> > debug.
> > We have the following settings:
> > > celery_result_backend = Postgres
> > > sql_alchmey_conn = Postgres
> > > broker_url = Redis
> > > exector = CeleryExecutor
> > Any thoughts from anyone regarding known issues or observed problems? I
> > haven't seen a jira on this after looking through the Airflow jira.
> > Thanks.
> > Regards,
> > David