I think we won't necessarily run multiple ShuffleMasters. I think it would be better to pass in a leaner interface into the RM to only handle the deletion of the global result partitions.
Letting the TEs handle the deletion of the global result partitions might work as long as we don't have an external shuffle service implementation. Hence, it could be a first step to decrease complexity but in order to complete this feature, I think we need to do it differently. Cheers, Till On Sat, Oct 12, 2019 at 7:39 AM zhijiang <wangzhijiang...@aliyun.com.invalid> wrote: > Sorry for delay catching up with the recent progress. Thanks for the FLIP > update and valuable discussions! > > I also like the term of job/cluster partitions, and agree with most of the > previous comments. > > Only left one concern of ShuffleMaster side: > >However, if the separation of JM/RM into separate processes, as outlined > in FLIP-6, is ever fully realized it necessarily implies that multiple > shuffle master instances may exist for a given shuffle service. > > My previous thought was that one ShuffleService factory is for creating > one shuffleMaster instance. If we have multiple ShuffleMaster instances, we > might also need differentt ShuffleService factories. > And it seems that different ShuffleMaster instances could run in different > components based on demands, e.g. dispatcher, JM, RM. > > Is it also feasible to not touch the ShuffleMaster concept in this FLIP to > make things a bit easy? I mean the ShuffleMaster is still running in JM > component and is responsbile for job partitions. For the case of cluster > partitions, the RM could interact with TE directly. TE would report global > partitions as payloads via heartbeat with RM. And the RM could call > TE#releaseGlobalPartitions directly not via ShuffleMaster. Even the RM > could also pass the global released partitions via payloads in heartbeat > with TE to reduce additional explict RPC call, but this would bring some > delays for releasing partition based on heartbeat interval. > > Best, > Zhijiang > ------------------------------------------------------------------ > From:Chesnay Schepler <ches...@apache.org> > Send Time:2019年10月11日(星期五) 10:21 > To:dev <dev@flink.apache.org>; Till Rohrmann <trohrm...@apache.org> > Subject:Re: [DISCUSS] FLIP-67: Global partitions lifecycle > > ooooh I like job-/cluster partitions. > > On 10/10/2019 16:27, Till Rohrmann wrote: > > I think we should introduce a separate interface for the ResourceManager > so > > that it can list and delete global result partitions from the shuffle > > service implementation. As long as the JM and RM run in the same process, > > this interface could be implemented by the ShuffleMaster implementations. > > However, we should make sure that we don't introduce unnecessary > > concurrency. If that should be the case, then it might be simpler to have > > two separate components. > > > > Some ideas for the naming problem: > > > > local/global: job/cluster, intra/inter > > > > Cheers, > > Till > > > > On Wed, Oct 9, 2019 at 1:35 PM Chesnay Schepler <ches...@apache.org> > wrote: > > > >> Are there any other opinions in regards to the naming scheme? > >> (local/global, promote) > >> > >> On 06/09/2019 15:16, Chesnay Schepler wrote: > >>> Hello, > >>> > >>> FLIP-36 (interactive programming) > >>> < > >> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-36%3A+Support+Interactive+Programming+in+Flink > > > >> > >>> proposes a new programming paradigm where jobs are built incrementally > >>> by the user. > >>> > >>> To support this in an efficient manner I propose to extend partition > >>> life-cycle to support the notion of /global partitions/, which are > >>> partitions that can exist beyond the life-time of a job. > >>> > >>> These partitions could then be re-used by subsequent jobs in a fairly > >>> efficient manner, as they don't have to persisted to an external > >>> storage first and consuming tasks could be scheduled to exploit > >>> data-locality. > >>> > >>> The FLIP outlines the required changes on the JobMaster, TaskExecutor > >>> and ResourceManager to support this from a life-cycle perspective. > >>> > >>> This FLIP does /not/ concern itself with the /usage/ of global > >>> partitions, including client-side APIs, job-submission, scheduling and > >>> reading said partitions; these are all follow-ups that will either be > >>> part of FLIP-36 or spliced out into separate FLIPs. > >>> > >>> > >> > >
