If DAG removed ,master only need to pick up JOBs which reached schedule time, and dispatches them to one worker .However the JOBs will be in the queue before dispatched . The master will be lightweight , and complement work will be easy. The JOBs which have same label will be dispatched to workers and they will execute parallelly or in a sequence relying on their DEPENDENCY
---Original--- From: "GabryWu"<wu_shao_...@qq.com> Date: Wed, May 20, 2020 18:41 PM To: "dev"<dev@dolphinscheduler.apache.org>; Cc: "wenhemin"<wenhe...@apache.org>; Subject: Re: How do you think Task DAG dependency and List dependency？ In that case ,I recommend spliting the 100 SQLs into different JOB which has same label, each JOB has it's own business logic .Finally Add DEPENDENCY to each JOB. JOBs having same label can be shown in front page through DAG ---Original--- From: "Hemin Wen"<wenhe...@apache.org> Date: Wed, May 20, 2020 17:38 PM To: "GabryWu"<wu_shao_...@qq.com>; Subject: Re: How do you think Task DAG dependency and List dependency？ For example, A workflow has 100 sql scripts task, tasks depend on each other through data lineage. This requirement is not suitable for DAG configuration. -------------------- DolphinScheduler(Incubator) Commtter Hemin Wen 温合民 wenhe...@apache.org -------------------- GabryWu <wu_shao_...@qq.com> 于2020年5月20日周三 下午5:31写道： what do you mean batch tasks ------------------ 原始邮件 ------------------ 发件人: "wenhemin"<wenhe...@apache.org>; 发送时间: 2020年5月20日(星期三) 下午5:04 收件人: "dev"<dev@dolphinscheduler.apache.org>; 主题: Re: How do you think Task DAG dependency and List dependency？ I do not recommend remove DAG. I think, DS lacks support for batch tasks. DAG solves the expression of different types of tasks, there is no way to express batch tasks of the same type. I don't know if what I understand is accurate, It is recommended to understand in depth the problem that users want to solve by using "List dependency”. -------------------- DolphinScheduler(Incubator) Commtter Hemin Wen 温合民 wenhe...@apache.org -------------------- lidong dai <dailidon...@gmail.com> 于2020年5月20日周三 下午3:40写道： > I know what you said, there maybe need more people to discuss this topic, > I want to know other's opinion, how do they think this question > > > > Best Regards > --------------- > DolphinScheduler(Incubator) PPMC > Lidong Dai 代立冬 > dailidon...@gmail.com > --------------- > > > GabryWu <wu_shao_...@qq.com> 于2020年5月19日周二 上午7:19写道： > > > not actually. I mean refactor backend code to remove DAG, where only JOB > > and DEPENDENCY remained > > > > ---Original--- > > *From:* "lidong dai"<dailidon...@gmail.com> > > *Date:* Mon, May 18, 2020 22:47 PM > > *To:* "GabryWu"<wu_shao_...@qq.com>; > > *Cc:* "dev"<dev@dolphinscheduler.apache.org>; > > *Subject:* Re: How do you think Task DAG dependency and List dependency？ > > > > I think your said is "List dependency", List dependency is also DAG, it's > > only different in use. when use List dependency, you can add your > upstream > > dependency for the task, this is convenient for hugely tasks in one > > workflow > > > > > > Best Regards > > --------------- > > DolphinScheduler(Incubator) PPMC > > Lidong Dai 代立冬 > > dailidon...@gmail.com > > --------------- > > > > > > GabryWu <wu_shao_...@qq.com> 于2020年5月13日周三 上午11:08写道： > > > >> Until now ,DAG is one physical concept， which means that DAG is heavy > >> class , and introduces other classes and stored in a big json field. > >> On the bigdata platform , DAG is not a good concept for Schedulers, > which > >> can be abandoned. > >> If DAG was abandoned, JOB and DEPENDENCY will simply the architecture > >> and make DolphinScheduler stabilize and extended easily and easier to > >> search jobs > >> However abandoning DAG doesn't mean remove DAG graph in the front end, > >> which is also an important visualization way for JOBs > >> We can add one Job and Dependency manually ,and visualize them in DAG > >> graph automatically > >> > >> > >> > >> ------------------ 原始邮件 ------------------ > >> *发件人:* "lidong dai"<dailidon...@gmail.com>; > >> *发送时间:* 2020年5月12日(星期二) 下午5:26 > >> *收件人:* "dev"<dev@dolphinscheduler.apache.org>; > >> *主题:* Re: How do you think Task DAG dependency and List dependency？ > >> > >> yes, your description is proper, thanks > >> > >> > >> Best Regards > >> --------------- > >> DolphinScheduler(Incubator) PPMC > >> Lidong Dai 代立冬 > >> dailidon...@gmail.com > >> --------------- > >> > >> > >> leon bao <leon...@apache.org> 于2020年5月12日周二 下午5:06写道： > >> > >> > i think you want to show the DAG using list mode. > >> > DS show DAG using graphic mode, that will have some problems: > >> > > >> > 1. once the number of tasks is large, graphic mode will be confusing. > >> > 2. it is difficult to find the specified task in a complex DAG. > >> > > >> > lidong dai <dailidon...@gmail.com> 于2020年5月12日周二 下午4:31写道： > >> > > >> > > I want to say do we need to implement List dependency？ > >> > > > >> > > > >> > > > >> > > Best Regards > >> > > --------------- > >> > > DolphinScheduler(Incubator) PPMC > >> > > Lidong Dai 代立冬 > >> > > dailidon...@gmail.com > >> > > --------------- > >> > > > >> > > > >> > > JUN GAO <gaojun2...@gmail.com> 于2020年5月12日周二 下午12:26写道： > >> > > > >> > > > Hi , @lidong dai > >> > > > Sorry , I don`t know your mean about this . > >> > > > You want to discuss how to implement Task DAG dependency and List > >> > > > dependency ? Or discuss what is Task DAG dependency and List > >> > dependency ? > >> > > > you can show us what`s your need and ideas , So that we can know > the > >> > > topic > >> > > > of discussion. > >> > > > > >> > > > Thank you ! > >> > > > > >> > > > lidong dai <dailidon...@gmail.com> 于2020年5月11日周一 下午10:34写道： > >> > > > > >> > > > > hi ， > >> > > > > > >> > > > > Task dependency between upstream and downstream called List > >> > > > dependency > >> > > > > do you have ideas about this topic? > >> > > > > > >> > > > > > >> > > > > > >> > > > > Best Regards > >> > > > > --------------- > >> > > > > DolphinScheduler(Incubator) PPMC > >> > > > > Lidong Dai 代立冬 > >> > > > > dailidon...@gmail.com > >> > > > > --------------- > >> > > > > > >> > > > > >> > > > > >> > > > -- > >> > > > > >> > > > DolphinScheduler(Incubator) PPMC > >> > > > Jun Gao 高俊 > >> > > > gaojun2...@gmail.com > >> > > > > >> > > > >> > > >> > > >> > -- > >> > DolphinScheduler(Incubator) PPMC > >> > BaoLiang 鲍亮 > >> > leon...@apache.org > >> > > >> > >> >
