We are getting the logs like
{local_executor.py:43} INFO - LocalWorker running airflow run <TaskCommand>
{models.py:1595} ERROR - Executor reports task instance %s finished (%s)
although the task says its %s. Was the task killed externally?
{models.py:1616} INFO - Marking task as UP_FOR_RETRY
It seems that in LocalExecutor.py file "subprocess_checkcall" command is
returning successfully without throwing an error but Task state did not get
updated from queued to running in mysql store.
One possibility is that "_check_and_change_state_before_execution" function in
models.py is returning false due to which task state is not getting updated to
"running" state and task get stuck in up_for_retry in handle_failure function.
Other possibility could be unavailability of system resources to create a new
process inside "subprocess_checkcall" function call but I think in that case it
would be throwing an error.
Is there a way to enable the "verbose" logging flag in
_check_and_change_state_before_execution call. It would help us in debugging it
further.
Thanks,
Raman Gupta
On 2018/08/17 15:22:27, Matthias Huschle <[email protected]> wrote:
> Hi Raman,
>
> Does it happen only occasionally, or can it be easily reproduced?
> What happens if you start it with "airflow run" or " airflow test"?
> What is in the logs about it?
> What is your user process limit ("ulimit -u") on that machine?
>
>
> 2018-08-17 15:39 GMT+02:00 [email protected] <[email protected]>:
>
> > Thanks Taylor,
> > We are getting this issue even after restart. We are observing that task
> > instance state is transitioned from
> > scheduled->queued->up_for_retry and dag gets stuck in up_for_retry state.
> > Behind the scenes executor keep on retrying the dag's task exceeding the
> > max retry limit.
> > In normal scenario task state should have following transition
> > scheduled->queued->running->up_for_retry
> > but we are seeing task is not entered in to running state rather it moves
> > directly to up_for_retry from queued state.
> > Any pointer on this would be helpful.
> >
> > Thanks,
> > raman Gupta
> >
> >
> > On 2018/08/16 16:05:31, Taylor Edmiston <[email protected]> wrote:
> > > Does a scheduler restart make a difference?
> > >
> > > *Taylor Edmiston*
> > > Blog <https://blog.tedmiston.com/> | CV
> > > <https://stackoverflow.com/cv/taylor> | LinkedIn
> > > <https://www.linkedin.com/in/tedmiston/> | AngelList
> > > <https://angel.co/taylor> | Stack Overflow
> > > <https://stackoverflow.com/users/149428/taylor-edmiston>
> > >
> > >
> > > On Thu, Aug 16, 2018 at 4:25 AM, [email protected] <
> > [email protected]>
> > > wrote:
> > >
> > > > Hi All,
> > > >
> > > > We are using airflow 1.9 with Local Executor more. Intermittently we
> > are
> > > > observing that tasks are getting stuck in "up_for_retry" mode and are
> > > > getting retried again and again exceeding their configured max retries
> > > > count. like we have configured max retries as 2 but task is retried 15
> > > > times and got stuck in up_for_retry state.
> > > > Any pointer on this would be helpful.
> > > >
> > > > Thanks,
> > > > Raman Gupta
> > > >
> > >
> >
>
>
>
> --
>
> *< Dr. Matthias Huschle />*
>
> Data and Analytics Manager
>
>
>
>
>
> E: [email protected]
>
> Connect with me LinkedIn
> <https://de.linkedin.com/pub/matthias-huschle/b0/912/91>| Xing
> <https://www.xing.com/profile/Matthias_Huschle>
>
>
> Credit & debit cards or PayPal? Now you can have both through the PAYMILL
> API. Find out more » <https://www.paymill.com/payment-methods/paypal>
>
>
> <https://paymill.atlassian.net/wiki/www.paymill.com/payment-methods/paypal>PAYMILL
> GmbH | St.-Martin-Straße 63, 81669 München <https://www.paymill.com/>
> <https://paymill.atlassian.net/wiki/www.paymill.com/payment-methods/paypal>
>
> Follow us @PAYMILL <https://twitter.com/Paymill> | www.PAYMILL.com
> <http://www.paymill.com/>
>
>
> The information in this e-mail is confidential and may be protected by
> professional secrecy. It is intended solely for the addressee. Any access
> to this e-mail is prohibited by persons other than the addressee. If you
> are not the named addressee, you are prohibited to make any attempt to
> publicise, reproduce or distribute this e-mail, this includes the taking or
> refraining of action in regards to the information obtained. Please notify
> the sender by e-mail immediately if you have received this e-mail by
> mistake and delete this e-mail from your system.
> PAYMILL GmbH | Sitz München | Amtsgericht München | HRB 226526 |
> Steuernummer: 143/169/70894 | USt-ID: DE308345749 | Geschäftsführer: Daniel
> Georges.
>
