Cleared tasks don't activate DAG Runs

2017-12-08 Thread Marc Weil
ven though there are uncompleted tasks within all of them. Has anyone ever seen this behavior before? Any tips on how to force Airflow to pick up all these cleared tasks to rerun them? This is on Airflow 1.8.0. Thanks! -- Marc Weil | Lead Engineer | Growth Data (GAME) | New Relic

Re: Meetup Interest?

2017-10-13 Thread Marc Weil
+1 for this meetup idea! We don't use Kube+Airflow, but I'd love to see talks on scaling it out team-wise and some design patterns people have come up with. -- Marc Weil | Lead Engineer | Growth Automation, Marketing, and Engagement | New Relic On Fri, Oct 13, 2017 at 1:03 PM, Christopher

Re: Completed tasks not being marked as completed

2017-08-04 Thread Marc Weil
er v54, I was > wondering if this was related at all. Somehow, airflow seems kinder on > Firefox. > > Does anyone else have a similar experience? > > Best, > Arthur > > On Fri, Aug 4, 2017 at 12:02 PM, Marc Weil <mw...@newrelic.com> wrote: > > > This is on versio

Re: Completed tasks not being marked as completed

2017-08-04 Thread Marc Weil
t; 3. Worker 2 grabs task > > 4. Worker 1 starts task > > 5. Worker 2 starts task > > 6. Worker 2 sees that Worker 1 has started and plans to abort > > 7. Worker 1 finishes and marks task as done > > 8. Worker 2 finishes aborting and marks

Re: Completed tasks not being marked as completed

2017-07-28 Thread Marc Weil
running. Thanks again! ᐧ -- Marc Weil | Lead Engineer | Growth Automation, Marketing, and Engagement | New Relic On Fri, Jul 28, 2017 at 3:15 PM, Maxime Beauchemin < maximebeauche...@gmail.com> wrote: > Are you sure there hasn't been a retry at that point? [One of] the expected > behavior

Re: Completed tasks not being marked as completed

2017-07-28 Thread Marc Weil
but uncompleted in the Airflow DB for many hours, but if I just leave them all sitting there over night they all tend to be marked complete the next morning. Perhaps this points to some sort of Celery timeout or connection retry interval? ᐧ -- Marc Weil | Lead Engineer | Growth Automation

Re: Completed tasks not being marked as completed

2017-07-28 Thread Marc Weil
>From what I can tell, it only affects CeleryExecutor. I've never seen this behavior with LocalExecutor before. Max, do you know anything about this type of failure mode? ᐧ -- Marc Weil | Lead Engineer | Growth Automation, Marketing, and Engagement | New Relic On Fri, Jul 28, 2017 at 5:48

Re: Checking on 1.7.x regression

2017-06-19 Thread Marc Weil
Hi again everyone, Didn't receive a response to this. What is the proper way to try to get attention on a regression bug from 1.7.x that still hasn't been addressed. Referring to this one: https://issues.apache.org/jira/browse/AIRFLOW-1026. Thanks! Marc ᐧ -- Marc Weil | Lead Engineer | Growth

Checking on 1.7.x regression

2017-06-09 Thread Marc Weil
Hello, Does anyone know what the prioritization status is for https://issues.apache.org/jira/browse/AIRFLOW-1026? It's a regression that was introduced into 1.8.0 that is blocking us from upgrading. Thanks! -- Marc Weil | Lead Engineer | Growth Automation, Marketing, and Engagement | New Relic