Alyssa, Are you running UTC everywhere? That is still an unfortunate requirement in some parts of airflow. I only ask because this is a common stumbling point -- one I have debugged for other before.
You need to run your DB and your airflow servers in UTC. -s On Thu, Jun 9, 2016 at 9:03 AM, Alyssa Kwan <[email protected]> wrote: > We're not using LocalExecutor. It's CeleryExecutor. We're running on > virtual hardware and perhaps the signal module is simply slow. (It > shouldn't be this slow.) We *do* get a timeout eventually. Just not at the > 10 minute mark. It doesn't trip until 5 hours. > > > On 8 Jun 2016, at 8:22, Maxime Beauchemin wrote: > > I created this JIRA to document the issue. >> https://issues.apache.org/jira/browse/AIRFLOW-221 >> >> Max >> >> On Wed, Jun 8, 2016 at 8:16 AM, Maxime Beauchemin < >> [email protected]> wrote: >> >> Are you using the LocalExecutor? there's a limitation around using the >>> task timeout in LocalExecutor under some circumstances. If that's the >>> case >>> you should see this somewhere in your logs: >>> >>> >>> https://github.com/apache/incubator-airflow/blob/master/airflow/utils/timeout.py#L45 >>> >>> We may want to move the logic enforcing the timeout from the context >>> manager/signal approach to BaseJob. >>> >>> Max >>> >>> On Tue, Jun 7, 2016 at 5:47 PM, Alyssa Kwan < >>> [email protected]> >>> wrote: >>> >>> Hi, >>>> >>>> We're having trouble with a `PythonOperator`'s `execution_timeout`. It's >>>> being set directly in the constructor as >>>> `datetime.timedelta(minutes=10)`. >>>> However, the task doesn't timeout until almost 5 hours is passed. This >>>> behavior is pretty consistent. It's merely a SQL query issued by the >>>> Python >>>> callable that can get deadlocked. >>>> >>>> Any thoughts? >>>> >>>> Thanks, >>>> Alyssa >>>> >>> >>> >>> >>>
