Re: [DISCUSS] TaskGroup in Tree View

Hi Yu Qian, Sorry I somehow missed this email 😢 It is great to hear that you are also interested in adding TaskGroup to the TreeView, hope you still do 😀 I like how the TaskGroup is collapsible in the prototype, though I wonder if we can replicate the zoom in behavior of SubDag--that might be the

Re: [DISCUSS] TaskGroup in Tree View

Hi, all, I'm interested in contributing to adding TaskGroup to Tree View. Here's a prototype of how it can look like. Suggestions are welcome. I understand AIP-38 plan

Re: [DISCUSS] TaskGroup in Tree View

I agree, but we should see what of those we can implement just on the parsing side - i.e. can we continue to make the scheduler not have to care about Task Groups? If so, then things like the default args example is a small enough change that it doesn't need an AIP (IMO) -ash On 8 March 2021

Re: [DISCUSS] TaskGroup in Tree View

I personally think that TaskGroup should go beyond being “just” a UI concept. I think that there are a lot of use-cases where people might want to perform a single operation across an entire group of tasks. I think that Bin points out a few really good examples (default arguments and group delet

Re: [DISCUSS] TaskGroup in Tree View

Hi, all, it's really exciting to see the great discussions about TaskGroup. There are some interesting ideas here. - Tree View support for TaskGroup: I think this can mostly be achieved at the web layer? Changes probably involve tree.html and www/view.py. Should we change Tree View to organize tas

Re: [DISCUSS] TaskGroup in Tree View

Hi Kaxil, One use case I have is to reuse TaskGroup across different DAGs as a predefined sub-workflow. For example, my team is currently building out a data platform that will allow a certain level of self-serve ability. Users of the platform (mostly analyst and scientist) should focus on busines

Re: [DISCUSS] TaskGroup in Tree View

Hi all, interesting discussion. I would love to hear about some more use-cases where TaskGroup needs to be something more than the UI concept. All of Kevin's use-cases can be achieved while keeping it as a UI concept.Xinbin can you please expand a bit on your use case. Regards, Kaxil On Sat, Mar

Re: [DISCUSS] TaskGroup in Tree View

Hi Kevin, Vikram, and Nathan, I think we don't need to restrict too much on keeping TaskGroup only as a UI concept. We are already using TaskGroup to author DAGs and create dependencies, which already lies a bit outside the UI. To fully replace SubDagOperator, I think it's necessary to expand Task

Re: [DISCUSS] TaskGroup in Tree View

Hi Vikram, Good point. What I had in mind was getting the TaskGroup definition in a sensor, e.g. extract the _task_group field from serialized DAG, and query the DB for the TI states within. You are right that it might not be clean nor does it keep TaskGroup as a UI concept. Cheers, Kevin Y On

Re: [DISCUSS] TaskGroup in Tree View

Kevin, I am not sure I understand your response to Nathan. I agree that it is also a valid use case, but I don't see how it can be cleanly done while keeping TaskGroup only as a UI concept. Would this require extending the TaskGroup concept to the backend? Best regards, Vikram On Fri, Mar 5, 20

Re: [DISCUSS] TaskGroup in Tree View

Hi Nathan, Thanks a lot for your input and it is indeed a valid use case. This can be done either keeping TaskGroup as a UI concept or bringing it into the backend. I'm curious to hear what others think. Cheers, Kevin Y On Thu, Mar 4, 2021 at 12:57 AM Nathan Hadfield wrote: > Hi Kevin, > > >

Re: [DISCUSS] TaskGroup in Tree View

Hi Kevin, A quick piece of input from our recent experiences of working with TaskGroup is that we often have dependencies across DAGs that require waiting upon the completion of all the tasks in a group. At the moment, you basically have two options: 1. Create a sensor task in a DAG for e

Re: [DISCUSS] TaskGroup in Tree View

Yes, those are correct. 👍 On Wed, Mar 3, 2021 at 9:47 PM Vikram Koka wrote: > Hey Kevin, > > One immediate clarifying question: > - For your use case, it seems that you want to continue using TaskGroup > only as a "pure UI concept". > - But, you want it's representation to also be in the Tree Vi

Re: [DISCUSS] TaskGroup in Tree View

Hey Kevin, One immediate clarifying question: - For your use case, it seems that you want to continue using TaskGroup only as a "pure UI concept". - But, you want it's representation to also be in the Tree View. - You are not proposing any "execution or scheduling" enhancements [again for your use