Re: [DISCUSS] FLIP-67: Global partitions lifecycle

#releasePartitions is also triggered not via > >> > ShuffleMaster in some cases, and it can be regareded as a shortcut > >> > way. Of course I am also in favour of via ShuffleMaster to call > >> > the actual release partition always, and the for

Re: [DISCUSS] FLIP-67: Global partitions lifecycle

, and the form seems elegant. >> > >> > I do not expect my inconsequential thought would block this feature >> > ongoing and disturb your previous conclusion. Moreover, Till's recent >> > reply already dispels my previous concern. :) >> > >> > Best,

Re: [DISCUSS] FLIP-67: Global partitions lifecycle

gt; reply already dispels my previous concern. :) > > > > Best, > > Zhijiang > > > > -- > > From:Chesnay Schepler > > Send Time:2019年10月14日(星期一) 07:00 > > To:dev ; Till Rohrmann >

Re: [DISCUSS] FLIP-67: Global partitions lifecycle

already dispels my previous concern. :) Best, Zhijiang -- From:Chesnay Schepler Send Time:2019年10月14日(星期一) 07:00 To:dev ; Till Rohrmann ; zhijiang Subject:Re: [DISCUSS] FLIP-67: Global partitions lifecycle

Re: [DISCUSS] FLIP-67: Global partitions lifecycle

RPC call, but this would bring some >> delays for releasing partition based on heartbeat interval. >> >> Best, >> Zhijiang >> -- >> From:Chesnay Schepler >> Send Time:2019年10月11日(星期五) 10:21 >

Re: [DISCUSS] FLIP-67: Global partitions lifecycle

. Best, Zhijiang -- From:Till Rohrmann Send Time:2019年10月13日(星期日) 17:04 To:zhijiang Cc:dev Subject:Re: [DISCUSS] FLIP-67: Global partitions lifecycle I think we won't necessarily run multiple ShuffleMasters. I think it would

Re: [DISCUSS] FLIP-67: Global partitions lifecycle

sing partition based on heartbeat interval. Best, Zhijiang -- From:Chesnay Schepler Send Time:2019年10月11日(星期五) 10:21 To:dev ; Till Rohrmann Subject:Re: [DISCUSS] FLIP-67: Global partitions lifecycle h I like job-/cluster partitions.

Re: [DISCUSS] FLIP-67: Global partitions lifecycle

delays for releasing partition based on heartbeat interval. > > Best, > Zhijiang > -- > From:Chesnay Schepler > Send Time:2019年10月11日(星期五) 10:21 > To:dev ; Till Rohrmann > Subject:Re: [DISCUSS] FLIP-67: Global partitions lifecycle > > h I like job-/cluster

Re: [DISCUSS] FLIP-67: Global partitions lifecycle

Zhijiang -- From:Chesnay Schepler Send Time:2019年10月11日(星期五) 10:21 To:dev ; Till Rohrmann Subject:Re: [DISCUSS] FLIP-67: Global partitions lifecycle h I like job-/cluster partitions. On 10/10/2019 16:27, Till Rohrmann wrote: > I think we should introduc

Re: [DISCUSS] FLIP-67: Global partitions lifecycle

h I like job-/cluster partitions. On 10/10/2019 16:27, Till Rohrmann wrote: I think we should introduce a separate interface for the ResourceManager so that it can list and delete global result partitions from the shuffle service implementation. As long as the JM and RM run in the same

Re: [DISCUSS] FLIP-67: Global partitions lifecycle

I think we should introduce a separate interface for the ResourceManager so that it can list and delete global result partitions from the shuffle service implementation. As long as the JM and RM run in the same process, this interface could be implemented by the ShuffleMaster implementations.

Re: [DISCUSS] FLIP-67: Global partitions lifecycle

Are there any other opinions in regards to the naming scheme? (local/global, promote) On 06/09/2019 15:16, Chesnay Schepler wrote: Hello, FLIP-36 (interactive programming) proposes a new

Re: [DISCUSS] FLIP-67: Global partitions lifecycle

Then it might be better to move ShuffleMaster outside of JM, and the lifecycle of ShuffleMaster should be consistent with RM. 5. Nit: TM->TE in the section of Proposed Changes: "TMs retain global partitions for successful jobs" Best, Zhijiang ---

Re: [DISCUSS] FLIP-67: Global partitions lifecycle

cycle of ShuffleMaster should be consistent with RM. 5. Nit: TM->TE in the section of Proposed Changes: "TMs retain global partitions for successful jobs" Best, Zhijiang -- From:Till Rohrmann Send Time:2019年9月10日(星期二) 10:10 To:dev Su

Re: [DISCUSS] FLIP-67: Global partitions lifecycle

ng RPC to the >> >> TEs? >> >> Or will the RM do it by itself? >> >> 4. How do we plan to handle the case when there are different shuffle >> services in the same Flink cluster? For example, a shared standalone >> cluster. >> >> 5. Minor: usually REST API uses `?` to pass

Re: [DISCUSS] FLIP-67: Global partitions lifecycle

ifecycle of ShuffleMaster should be consistent with RM. 5. Nit: TM->TE in the section of Proposed Changes: "TMs retain global partitions for successful jobs" Best, Zhijiang ---------------------- From:Till Rohrmann <mailto:t

Re: [DISCUSS] FLIP-67: Global partitions lifecycle

understanding the semantic. Concering whether to use global or > persistent, I prefer the "global" term personally. Because it > > describes the > > scope of partition clearly, and the "persistent" is more like the > > partition > > storing way or imp

Re: [DISCUSS] FLIP-67: Global partitions lifecycle

f JM, and the lifecycle of ShuffleMaster should be consistent with RM. 5. Nit: TM->TE in the section of Proposed Changes: "TMs retain global partitions for successful jobs" Best, Zhijiang ---------- From:Till Rohrmann Send Time

Re: [DISCUSS] FLIP-67: Global partitions lifecycle

gt;> better understanding the semantic. Concering whether to use global or > >>> persistent, I prefer the "global" term personally. Because it > describes the > >>> scope of partition clearly, and the "persistent" is more like the > partition > >&

Re: [DISCUSS] FLIP-67: Global partitions lifecycle

RM. 5. Nit: TM->TE in the section of Proposed Changes: "TMs retain global partitions for successful jobs" Best, Zhijiang ------------------ From:Till Rohrmann Send Time:2019年9月10日(星期二) 10:10 To:dev Subject:Re: [DISCUSS] FLIP-67: Global partiti

Re: [DISCUSS] FLIP-67: Global partitions lifecycle

ons. >> >> 4. Considering ShuffleMaster, it was built inside JM and expected to >> interactive with JM before. Now the RM also needs to interactive with >> ShuffleMaster to release global partitions. Then it might be better to move >> ShuffleMaster outside of JM, and th

Re: [DISCUSS] FLIP-67: Global partitions lifecycle

with RM. > > 5. Nit: TM->TE in the section of Proposed Changes: "TMs retain global > partitions for successful jobs" > > Best, > Zhijiang > > > -------------- > From:Till Rohrmann > Send Time:2019年9月10日(星

Re: [DISCUSS] FLIP-67: Global partitions lifecycle

essful jobs" Best, Zhijiang -- From:Till Rohrmann Send Time:2019年9月10日(星期二) 10:10 To:dev Subject:Re: [DISCUSS] FLIP-67: Global partitions lifecycle Thanks Chesnay for drafting the FLIP and starting this discussion. I have a couple of comments: * I kno

Re: [DISCUSS] FLIP-67: Global partitions lifecycle

Thanks Chesnay for drafting the FLIP and starting this discussion. I have a couple of comments: * I know that I've also coined the terms global/local result partition but maybe it is not the perfect name. Maybe we could rethink the terminology and call them persistent result partitions? * Nit: I

[DISCUSS] FLIP-67: Global partitions lifecycle

Hello, FLIP-36 (interactive programming) proposes a new programming paradigm where jobs are built incrementally by the user. To support this in an efficient manner I propose to extend