Agree with others who think SubDag should stay, we should fix the SubDag implementation but not remove the abstraction itself.
On Fri, Apr 12, 2019 at 8:42 AM Chen Tong <[email protected]> wrote: > Is it possible to re-implement it in the view-level, not in operator level? > And this operator is just define a different view in GUI, that these tasks > will be collapsed into another view. > > On Fri, Apr 12, 2019 at 11:31 AM James Meickle > <[email protected]> wrote: > > > I have avoided using them because of outstanding issues like the open > JIRA > > issues I linked above, or similar issues that I've read about on blog > > posts. If it were just GUI or UX issues I'd use them, but many people > have > > reported issues which affect concurrency/stability, consistency, or > > correctness of results. I believe that it's working for you, but for me, > > it's not worth the risk to build using them in my environment (even > though > > they could be handy for many of our workflows). > > > > On Fri, Apr 12, 2019 at 11:18 AM Kaxil Naik <[email protected]> wrote: > > > > > I have been using SubDags in production and haven't had much problem > with > > > it. > > > > > > Can you list the issues you had? > > > > > > Regards, > > > Kaxil > > > > > > > > > On Fri, Apr 12, 2019, 16:16 James Meickle <[email protected] > > > .invalid> > > > wrote: > > > > > > > Given their bad reputation, would it be appropriate to deprecate > > subDAGs > > > > now to advertise that they're no longer considered a suitable > > > > implementation? If a new and better implementation is created, would > it > > > > even be similar enough to subDAGs that we'd want to continue to call > > the > > > > feature that? > > > > > > > > They feel like a "new Airflow user trap" right now - I have had to > tell > > > my > > > > team never to use them, because they seem appealing and are in the > > > official > > > > docs. > > > > > > > > On Fri, Apr 12, 2019 at 10:51 AM Ash Berlin-Taylor <[email protected]> > > > wrote: > > > > > > > > > I'd like to find time to fix subdags as they do provide a useful > > > > > abstraction - but I agree right now they aren't great (I avoid them > > > > because > > > > > of this) > > > > > > > > > > I have half thoughts of how to it should work, I just need to look > at > > > the > > > > > code in depth to see if that makes sense. Now 1.10.3 is out I might > > > have > > > > a > > > > > bit more time to do this. > > > > > > > > > > -ash > > > > > > > > > > > On 12 Apr 2019, at 15:48, James Meickle <[email protected] > > > > .INVALID> > > > > > wrote: > > > > > > > > > > > > I think we should deprecate SubDAGs given the complexity they add > > and > > > > the > > > > > > limited usage and use cases. Or, we should invest effort in > > > redesigning > > > > > > their API and implementation. I think that having to account for > > > > > > subdag-introduced complexity makes Airflow's code much harder to > > > > maintain > > > > > > and buggier, looking at how many open issues there are that > > reference > > > > > > subdags (and how unrelated in topic they are): > > > > > > > > > > > > > > > > > > > > > https://issues.apache.org/jira/browse/AIRFLOW-3292?jql=project%20%3D%20AIRFLOW%20AND%20status%20%3D%20Open%20AND%20text%20~%20%22subdag%22 > > > > > > > > > > > > > > > > > > > >
