I have avoided using them because of outstanding issues like the open JIRA issues I linked above, or similar issues that I've read about on blog posts. If it were just GUI or UX issues I'd use them, but many people have reported issues which affect concurrency/stability, consistency, or correctness of results. I believe that it's working for you, but for me, it's not worth the risk to build using them in my environment (even though they could be handy for many of our workflows).
On Fri, Apr 12, 2019 at 11:18 AM Kaxil Naik <kaxiln...@gmail.com> wrote: > I have been using SubDags in production and haven't had much problem with > it. > > Can you list the issues you had? > > Regards, > Kaxil > > > On Fri, Apr 12, 2019, 16:16 James Meickle <jmeic...@quantopian.com > .invalid> > wrote: > > > Given their bad reputation, would it be appropriate to deprecate subDAGs > > now to advertise that they're no longer considered a suitable > > implementation? If a new and better implementation is created, would it > > even be similar enough to subDAGs that we'd want to continue to call the > > feature that? > > > > They feel like a "new Airflow user trap" right now - I have had to tell > my > > team never to use them, because they seem appealing and are in the > official > > docs. > > > > On Fri, Apr 12, 2019 at 10:51 AM Ash Berlin-Taylor <a...@apache.org> > wrote: > > > > > I'd like to find time to fix subdags as they do provide a useful > > > abstraction - but I agree right now they aren't great (I avoid them > > because > > > of this) > > > > > > I have half thoughts of how to it should work, I just need to look at > the > > > code in depth to see if that makes sense. Now 1.10.3 is out I might > have > > a > > > bit more time to do this. > > > > > > -ash > > > > > > > On 12 Apr 2019, at 15:48, James Meickle <jmeic...@quantopian.com > > .INVALID> > > > wrote: > > > > > > > > I think we should deprecate SubDAGs given the complexity they add and > > the > > > > limited usage and use cases. Or, we should invest effort in > redesigning > > > > their API and implementation. I think that having to account for > > > > subdag-introduced complexity makes Airflow's code much harder to > > maintain > > > > and buggier, looking at how many open issues there are that reference > > > > subdags (and how unrelated in topic they are): > > > > > > > > > > https://issues.apache.org/jira/browse/AIRFLOW-3292?jql=project%20%3D%20AIRFLOW%20AND%20status%20%3D%20Open%20AND%20text%20~%20%22subdag%22 > > > > > > > > >