[jira] [Commented] (FLINK-5791) Support an optimal matching based slot manager for flip6 yarn mode
[ https://issues.apache.org/jira/browse/FLINK-5791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17194196#comment-17194196 ] Andrey Zagrebin commented on FLINK-5791: SlotManager can have different implementation now (FLINK-14106). [~tiemsn] are you still planning to work on this? > Support an optimal matching based slot manager for flip6 yarn mode > -- > > Key: FLINK-5791 > URL: https://issues.apache.org/jira/browse/FLINK-5791 > Project: Flink > Issue Type: Improvement > Components: Deployment / YARN >Reporter: shuai.xu >Assignee: shuai.xu >Priority: Major > Labels: flip-6, pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > In flip6, for yarn mode, it request containers from yarn according to the > resource profile in slot requests. And when slots come back after container > starting up, we should supporting an optimal matching based slot manager > which can assign the slot to the request whose resource profile closest to > it. This could reduce resource waste and avoid OOM. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-5791) Support an optimal matching based slot manager for flip6 yarn mode
[ https://issues.apache.org/jira/browse/FLINK-5791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16939249#comment-16939249 ] Till Rohrmann commented on FLINK-5791: -- This issue could become easily doable once FLINK-14106 has been completed. > Support an optimal matching based slot manager for flip6 yarn mode > -- > > Key: FLINK-5791 > URL: https://issues.apache.org/jira/browse/FLINK-5791 > Project: Flink > Issue Type: Improvement > Components: Deployment / YARN >Reporter: shuai.xu >Assignee: shuai.xu >Priority: Major > Labels: flip-6, pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > In flip6, for yarn mode, it request containers from yarn according to the > resource profile in slot requests. And when slots come back after container > starting up, we should supporting an optimal matching based slot manager > which can assign the slot to the request whose resource profile closest to > it. This could reduce resource waste and avoid OOM. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-5791) Support an optimal matching based slot manager for flip6 yarn mode
[ https://issues.apache.org/jira/browse/FLINK-5791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16118098#comment-16118098 ] ASF GitHub Bot commented on FLINK-5791: --- Github user shuai-xu commented on the issue: https://github.com/apache/flink/pull/3580 @tillrohrmann Could you please help to review this pr? And there is a question, the pending requests in slot managers are not in order. So requests arrived later may be fulfilled earlier, does this matter? > Support an optimal matching based slot manager for flip6 yarn mode > -- > > Key: FLINK-5791 > URL: https://issues.apache.org/jira/browse/FLINK-5791 > Project: Flink > Issue Type: Improvement > Components: YARN >Reporter: shuai.xu >Assignee: shuai.xu > Labels: flip-6 > > In flip6, for yarn mode, it request containers from yarn according to the > resource profile in slot requests. And when slots come back after container > starting up, we should supporting an optimal matching based slot manager > which can assign the slot to the request whose resource profile closest to > it. This could reduce resource waste and avoid OOM. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (FLINK-5791) Support an optimal matching based slot manager for flip6 yarn mode
[ https://issues.apache.org/jira/browse/FLINK-5791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15934049#comment-15934049 ] ASF GitHub Bot commented on FLINK-5791: --- GitHub user shuai-xu opened a pull request: https://github.com/apache/flink/pull/3580 [FLINK-5791] Support an optimal matching based slot manager for flip6 yarn mode This pr is for jira-#[5791](https://issues.apache.org/jira/browse/FLINK-5791) Main changes: 1. Add a UNIVERSAL resource profile which can match any other one. 2. TaskExecutor will be initialized with the UNIVERSAL resource profile. 3. Add a Optimal matching based slot manager which match slots to the request with closest resource You can merge this pull request into a Git repository by running: $ git pull https://github.com/shuai-xu/flink jira-5791 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/3580.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3580 commit 573065fadae4f021b793895cbbe7b6ac7e5f1c95 Author: shuai.xus Date: 2017-03-16T05:49:14Z [FLINK-5791] Support an optimal matching based slot manager for flip6 yarn mode Summary: 1. Add a UNIVERSAL resource profile which can match any other one. 2. TaskExecutor will be initialized with the UNIVERSAL resource profile. 3. Add a Optimal matching based slot manager which match slots to the request with closest resource Test Plan: NA Reviewers: 淘江, 星罡 Differential Revision: https://aone.alibaba-inc.com/code/D135680 > Support an optimal matching based slot manager for flip6 yarn mode > -- > > Key: FLINK-5791 > URL: https://issues.apache.org/jira/browse/FLINK-5791 > Project: Flink > Issue Type: Improvement > Components: YARN >Reporter: shuai.xu >Assignee: shuai.xu > Labels: flip-6 > > In flip6, for yarn mode, it request containers from yarn according to the > resource profile in slot requests. And when slots come back after container > starting up, we should supporting an optimal matching based slot manager > which can assign the slot to the request whose resource profile closest to > it. This could reduce resource waste and avoid OOM. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (FLINK-5791) Support an optimal matching based slot manager for flip6 yarn mode
[ https://issues.apache.org/jira/browse/FLINK-5791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15925716#comment-15925716 ] ASF GitHub Bot commented on FLINK-5791: --- Github user shuai-xu closed the pull request at: https://github.com/apache/flink/pull/3304 > Support an optimal matching based slot manager for flip6 yarn mode > -- > > Key: FLINK-5791 > URL: https://issues.apache.org/jira/browse/FLINK-5791 > Project: Flink > Issue Type: Improvement > Components: YARN >Reporter: shuai.xu >Assignee: shuai.xu > Labels: flip-6 > > In flip6, for yarn mode, it request containers from yarn according to the > resource profile in slot requests. And when slots come back after container > starting up, we should supporting an optimal matching based slot manager > which can assign the slot to the request whose resource profile closest to > it. This could reduce resource waste and avoid OOM. -- This message was sent by Atlassian JIRA (v6.3.15#6346)