[jira] [Commented] (YARN-2190) Add CPU and memory limit options to the default container executor for Windows containers
[ https://issues.apache.org/jira/browse/YARN-2190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14351521#comment-14351521 ] Hudson commented on YARN-2190: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #125 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/125/]) YARN-2190. Added CPU and memory limit options to the default container executor for Windows containers. Contributed by Chuan Liu (jianhe: rev 21101c01f242439ec8ec40fb3a9ab1991ae0adc7) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestContainerExecutor.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/WindowsSecureContainerExecutor.java * hadoop-yarn-project/CHANGES.txt * BUILDING.txt * hadoop-common-project/hadoop-common/src/main/winutils/winutils.vcxproj * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml * hadoop-common-project/hadoop-common/src/main/winutils/task.c * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/ContainerExecutor.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestWinUtils.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DefaultContainerExecutor.java * hadoop-common-project/hadoop-common/src/main/winutils/win8sdk.props Add CPU and memory limit options to the default container executor for Windows containers - Key: YARN-2190 URL: https://issues.apache.org/jira/browse/YARN-2190 Project: Hadoop YARN Issue Type: New Feature Components: nodemanager Reporter: Chuan Liu Assignee: Chuan Liu Fix For: 2.7.0 Attachments: YARN-2190-prototype.patch, YARN-2190.1.patch, YARN-2190.10.patch, YARN-2190.11.patch, YARN-2190.12.patch, YARN-2190.13.patch, YARN-2190.2.patch, YARN-2190.3.patch, YARN-2190.4.patch, YARN-2190.5.patch, YARN-2190.6.patch, YARN-2190.7.patch, YARN-2190.8.patch, YARN-2190.9.patch Yarn default container executor on Windows does not set the resource limit on the containers currently. The memory limit is enforced by a separate monitoring thread. The container implementation on Windows uses Job Object right now. The latest Windows (8 or later) API allows CPU and memory limits on the job objects. We want to add the new options to the executor that can set the limits on job objects thus provides resource enforcement at OS level. http://msdn.microsoft.com/en-us/library/windows/desktop/ms686216(v=vs.85).aspx -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2190) Add CPU and memory limit options to the default container executor for Windows containers
[ https://issues.apache.org/jira/browse/YARN-2190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14351641#comment-14351641 ] Hudson commented on YARN-2190: -- SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2075 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2075/]) YARN-2190. Added CPU and memory limit options to the default container executor for Windows containers. Contributed by Chuan Liu (jianhe: rev 21101c01f242439ec8ec40fb3a9ab1991ae0adc7) * hadoop-common-project/hadoop-common/src/main/winutils/win8sdk.props * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/ContainerExecutor.java * BUILDING.txt * hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestWinUtils.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DefaultContainerExecutor.java * hadoop-common-project/hadoop-common/src/main/winutils/task.c * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestContainerExecutor.java * hadoop-common-project/hadoop-common/src/main/winutils/winutils.vcxproj * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/WindowsSecureContainerExecutor.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java Add CPU and memory limit options to the default container executor for Windows containers - Key: YARN-2190 URL: https://issues.apache.org/jira/browse/YARN-2190 Project: Hadoop YARN Issue Type: New Feature Components: nodemanager Reporter: Chuan Liu Assignee: Chuan Liu Fix For: 2.7.0 Attachments: YARN-2190-prototype.patch, YARN-2190.1.patch, YARN-2190.10.patch, YARN-2190.11.patch, YARN-2190.12.patch, YARN-2190.13.patch, YARN-2190.2.patch, YARN-2190.3.patch, YARN-2190.4.patch, YARN-2190.5.patch, YARN-2190.6.patch, YARN-2190.7.patch, YARN-2190.8.patch, YARN-2190.9.patch Yarn default container executor on Windows does not set the resource limit on the containers currently. The memory limit is enforced by a separate monitoring thread. The container implementation on Windows uses Job Object right now. The latest Windows (8 or later) API allows CPU and memory limits on the job objects. We want to add the new options to the executor that can set the limits on job objects thus provides resource enforcement at OS level. http://msdn.microsoft.com/en-us/library/windows/desktop/ms686216(v=vs.85).aspx -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2190) Add CPU and memory limit options to the default container executor for Windows containers
[ https://issues.apache.org/jira/browse/YARN-2190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14351529#comment-14351529 ] Hudson commented on YARN-2190: -- FAILURE: Integrated in Hadoop-Yarn-trunk #859 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/859/]) YARN-2190. Added CPU and memory limit options to the default container executor for Windows containers. Contributed by Chuan Liu (jianhe: rev 21101c01f242439ec8ec40fb3a9ab1991ae0adc7) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DefaultContainerExecutor.java * hadoop-common-project/hadoop-common/src/main/winutils/win8sdk.props * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml * hadoop-yarn-project/CHANGES.txt * hadoop-common-project/hadoop-common/src/main/winutils/winutils.vcxproj * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestContainerExecutor.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/WindowsSecureContainerExecutor.java * hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestWinUtils.java * hadoop-common-project/hadoop-common/src/main/winutils/task.c * BUILDING.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/ContainerExecutor.java Add CPU and memory limit options to the default container executor for Windows containers - Key: YARN-2190 URL: https://issues.apache.org/jira/browse/YARN-2190 Project: Hadoop YARN Issue Type: New Feature Components: nodemanager Reporter: Chuan Liu Assignee: Chuan Liu Fix For: 2.7.0 Attachments: YARN-2190-prototype.patch, YARN-2190.1.patch, YARN-2190.10.patch, YARN-2190.11.patch, YARN-2190.12.patch, YARN-2190.13.patch, YARN-2190.2.patch, YARN-2190.3.patch, YARN-2190.4.patch, YARN-2190.5.patch, YARN-2190.6.patch, YARN-2190.7.patch, YARN-2190.8.patch, YARN-2190.9.patch Yarn default container executor on Windows does not set the resource limit on the containers currently. The memory limit is enforced by a separate monitoring thread. The container implementation on Windows uses Job Object right now. The latest Windows (8 or later) API allows CPU and memory limits on the job objects. We want to add the new options to the executor that can set the limits on job objects thus provides resource enforcement at OS level. http://msdn.microsoft.com/en-us/library/windows/desktop/ms686216(v=vs.85).aspx -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2190) Add CPU and memory limit options to the default container executor for Windows containers
[ https://issues.apache.org/jira/browse/YARN-2190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14351610#comment-14351610 ] Hudson commented on YARN-2190: -- SUCCESS: Integrated in Hadoop-Hdfs-trunk-Java8 #116 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/116/]) YARN-2190. Added CPU and memory limit options to the default container executor for Windows containers. Contributed by Chuan Liu (jianhe: rev 21101c01f242439ec8ec40fb3a9ab1991ae0adc7) * hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestWinUtils.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * hadoop-common-project/hadoop-common/src/main/winutils/winutils.vcxproj * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DefaultContainerExecutor.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/ContainerExecutor.java * hadoop-common-project/hadoop-common/src/main/winutils/win8sdk.props * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/WindowsSecureContainerExecutor.java * BUILDING.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestContainerExecutor.java * hadoop-common-project/hadoop-common/src/main/winutils/task.c Add CPU and memory limit options to the default container executor for Windows containers - Key: YARN-2190 URL: https://issues.apache.org/jira/browse/YARN-2190 Project: Hadoop YARN Issue Type: New Feature Components: nodemanager Reporter: Chuan Liu Assignee: Chuan Liu Fix For: 2.7.0 Attachments: YARN-2190-prototype.patch, YARN-2190.1.patch, YARN-2190.10.patch, YARN-2190.11.patch, YARN-2190.12.patch, YARN-2190.13.patch, YARN-2190.2.patch, YARN-2190.3.patch, YARN-2190.4.patch, YARN-2190.5.patch, YARN-2190.6.patch, YARN-2190.7.patch, YARN-2190.8.patch, YARN-2190.9.patch Yarn default container executor on Windows does not set the resource limit on the containers currently. The memory limit is enforced by a separate monitoring thread. The container implementation on Windows uses Job Object right now. The latest Windows (8 or later) API allows CPU and memory limits on the job objects. We want to add the new options to the executor that can set the limits on job objects thus provides resource enforcement at OS level. http://msdn.microsoft.com/en-us/library/windows/desktop/ms686216(v=vs.85).aspx -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2190) Add CPU and memory limit options to the default container executor for Windows containers
[ https://issues.apache.org/jira/browse/YARN-2190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14351599#comment-14351599 ] Hudson commented on YARN-2190: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #2057 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/2057/]) YARN-2190. Added CPU and memory limit options to the default container executor for Windows containers. Contributed by Chuan Liu (jianhe: rev 21101c01f242439ec8ec40fb3a9ab1991ae0adc7) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestContainerExecutor.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/WindowsSecureContainerExecutor.java * hadoop-common-project/hadoop-common/src/main/winutils/winutils.vcxproj * hadoop-common-project/hadoop-common/src/main/winutils/win8sdk.props * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/ContainerExecutor.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DefaultContainerExecutor.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml * BUILDING.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * hadoop-common-project/hadoop-common/src/main/winutils/task.c * hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestWinUtils.java Add CPU and memory limit options to the default container executor for Windows containers - Key: YARN-2190 URL: https://issues.apache.org/jira/browse/YARN-2190 Project: Hadoop YARN Issue Type: New Feature Components: nodemanager Reporter: Chuan Liu Assignee: Chuan Liu Fix For: 2.7.0 Attachments: YARN-2190-prototype.patch, YARN-2190.1.patch, YARN-2190.10.patch, YARN-2190.11.patch, YARN-2190.12.patch, YARN-2190.13.patch, YARN-2190.2.patch, YARN-2190.3.patch, YARN-2190.4.patch, YARN-2190.5.patch, YARN-2190.6.patch, YARN-2190.7.patch, YARN-2190.8.patch, YARN-2190.9.patch Yarn default container executor on Windows does not set the resource limit on the containers currently. The memory limit is enforced by a separate monitoring thread. The container implementation on Windows uses Job Object right now. The latest Windows (8 or later) API allows CPU and memory limits on the job objects. We want to add the new options to the executor that can set the limits on job objects thus provides resource enforcement at OS level. http://msdn.microsoft.com/en-us/library/windows/desktop/ms686216(v=vs.85).aspx -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2190) Add CPU and memory limit options to the default container executor for Windows containers
[ https://issues.apache.org/jira/browse/YARN-2190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14351623#comment-14351623 ] Hudson commented on YARN-2190: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #125 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/125/]) YARN-2190. Added CPU and memory limit options to the default container executor for Windows containers. Contributed by Chuan Liu (jianhe: rev 21101c01f242439ec8ec40fb3a9ab1991ae0adc7) * BUILDING.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/ContainerExecutor.java * hadoop-common-project/hadoop-common/src/main/winutils/win8sdk.props * hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestWinUtils.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DefaultContainerExecutor.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestContainerExecutor.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/WindowsSecureContainerExecutor.java * hadoop-common-project/hadoop-common/src/main/winutils/task.c * hadoop-common-project/hadoop-common/src/main/winutils/winutils.vcxproj * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml Add CPU and memory limit options to the default container executor for Windows containers - Key: YARN-2190 URL: https://issues.apache.org/jira/browse/YARN-2190 Project: Hadoop YARN Issue Type: New Feature Components: nodemanager Reporter: Chuan Liu Assignee: Chuan Liu Fix For: 2.7.0 Attachments: YARN-2190-prototype.patch, YARN-2190.1.patch, YARN-2190.10.patch, YARN-2190.11.patch, YARN-2190.12.patch, YARN-2190.13.patch, YARN-2190.2.patch, YARN-2190.3.patch, YARN-2190.4.patch, YARN-2190.5.patch, YARN-2190.6.patch, YARN-2190.7.patch, YARN-2190.8.patch, YARN-2190.9.patch Yarn default container executor on Windows does not set the resource limit on the containers currently. The memory limit is enforced by a separate monitoring thread. The container implementation on Windows uses Job Object right now. The latest Windows (8 or later) API allows CPU and memory limits on the job objects. We want to add the new options to the executor that can set the limits on job objects thus provides resource enforcement at OS level. http://msdn.microsoft.com/en-us/library/windows/desktop/ms686216(v=vs.85).aspx -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2190) Add CPU and memory limit options to the default container executor for Windows containers
[ https://issues.apache.org/jira/browse/YARN-2190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14350742#comment-14350742 ] Jian He commented on YARN-2190: --- what I meant is if resource != null and knobs are off, it'll pass -1 for the memory and cpu, which is not the same as the original command. Add CPU and memory limit options to the default container executor for Windows containers - Key: YARN-2190 URL: https://issues.apache.org/jira/browse/YARN-2190 Project: Hadoop YARN Issue Type: New Feature Components: nodemanager Reporter: Chuan Liu Assignee: Chuan Liu Attachments: YARN-2190-prototype.patch, YARN-2190.1.patch, YARN-2190.10.patch, YARN-2190.11.patch, YARN-2190.12.patch, YARN-2190.13.patch, YARN-2190.2.patch, YARN-2190.3.patch, YARN-2190.4.patch, YARN-2190.5.patch, YARN-2190.6.patch, YARN-2190.7.patch, YARN-2190.8.patch, YARN-2190.9.patch Yarn default container executor on Windows does not set the resource limit on the containers currently. The memory limit is enforced by a separate monitoring thread. The container implementation on Windows uses Job Object right now. The latest Windows (8 or later) API allows CPU and memory limits on the job objects. We want to add the new options to the executor that can set the limits on job objects thus provides resource enforcement at OS level. http://msdn.microsoft.com/en-us/library/windows/desktop/ms686216(v=vs.85).aspx -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2190) Add CPU and memory limit options to the default container executor for Windows containers
[
https://issues.apache.org/jira/browse/YARN-2190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14350714#comment-14350714
]
Jian He commented on YARN-2190:
---
thanks [~chuanliu], looks good overall, one question is that
- here, if resource is null, it'll pass -1 for the memory and cpu, is this
expected?
{code}
return new String[] { Shell.WINUTILS, task, create, -m,
String.valueOf(memory), -c, String.valueOf(cpuRate), groupId,
cmd /c + command };
{code}
Add CPU and memory limit options to the default container executor for
Windows containers
-
Key: YARN-2190
URL: https://issues.apache.org/jira/browse/YARN-2190
Project: Hadoop YARN
Issue Type: New Feature
Components: nodemanager
Reporter: Chuan Liu
Assignee: Chuan Liu
Attachments: YARN-2190-prototype.patch, YARN-2190.1.patch,
YARN-2190.10.patch, YARN-2190.11.patch, YARN-2190.12.patch,
YARN-2190.13.patch, YARN-2190.2.patch, YARN-2190.3.patch, YARN-2190.4.patch,
YARN-2190.5.patch, YARN-2190.6.patch, YARN-2190.7.patch, YARN-2190.8.patch,
YARN-2190.9.patch
Yarn default container executor on Windows does not set the resource limit on
the containers currently. The memory limit is enforced by a separate
monitoring thread. The container implementation on Windows uses Job Object
right now. The latest Windows (8 or later) API allows CPU and memory limits
on the job objects. We want to add the new options to the executor that can
set the limits on job objects thus provides resource enforcement at OS level.
http://msdn.microsoft.com/en-us/library/windows/desktop/ms686216(v=vs.85).aspx
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2190) Add CPU and memory limit options to the default container executor for Windows containers
[
https://issues.apache.org/jira/browse/YARN-2190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14350772#comment-14350772
]
Chuan Liu commented on YARN-2190:
-
bq. what I meant is if resource != null and knobs are off, it'll pass -1 for
the memory and cpu, which is not the same as the original command.
With the new 'winutils task' change, negative numbers will meaning no limits.
Even without the command line options, the '-1's are still passed to the actual
method {{CreateTaskImpl}}. If you look at {{CreateTaskImpl}}, the method will
not set the corresponding limit when CPU or memory is negative.
Add CPU and memory limit options to the default container executor for
Windows containers
-
Key: YARN-2190
URL: https://issues.apache.org/jira/browse/YARN-2190
Project: Hadoop YARN
Issue Type: New Feature
Components: nodemanager
Reporter: Chuan Liu
Assignee: Chuan Liu
Attachments: YARN-2190-prototype.patch, YARN-2190.1.patch,
YARN-2190.10.patch, YARN-2190.11.patch, YARN-2190.12.patch,
YARN-2190.13.patch, YARN-2190.2.patch, YARN-2190.3.patch, YARN-2190.4.patch,
YARN-2190.5.patch, YARN-2190.6.patch, YARN-2190.7.patch, YARN-2190.8.patch,
YARN-2190.9.patch
Yarn default container executor on Windows does not set the resource limit on
the containers currently. The memory limit is enforced by a separate
monitoring thread. The container implementation on Windows uses Job Object
right now. The latest Windows (8 or later) API allows CPU and memory limits
on the job objects. We want to add the new options to the executor that can
set the limits on job objects thus provides resource enforcement at OS level.
http://msdn.microsoft.com/en-us/library/windows/desktop/ms686216(v=vs.85).aspx
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2190) Add CPU and memory limit options to the default container executor for Windows containers
[ https://issues.apache.org/jira/browse/YARN-2190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14351003#comment-14351003 ] Jian He commented on YARN-2190: --- make sense, thanks for your explanation ! looks good, committing Add CPU and memory limit options to the default container executor for Windows containers - Key: YARN-2190 URL: https://issues.apache.org/jira/browse/YARN-2190 Project: Hadoop YARN Issue Type: New Feature Components: nodemanager Reporter: Chuan Liu Assignee: Chuan Liu Attachments: YARN-2190-prototype.patch, YARN-2190.1.patch, YARN-2190.10.patch, YARN-2190.11.patch, YARN-2190.12.patch, YARN-2190.13.patch, YARN-2190.2.patch, YARN-2190.3.patch, YARN-2190.4.patch, YARN-2190.5.patch, YARN-2190.6.patch, YARN-2190.7.patch, YARN-2190.8.patch, YARN-2190.9.patch Yarn default container executor on Windows does not set the resource limit on the containers currently. The memory limit is enforced by a separate monitoring thread. The container implementation on Windows uses Job Object right now. The latest Windows (8 or later) API allows CPU and memory limits on the job objects. We want to add the new options to the executor that can set the limits on job objects thus provides resource enforcement at OS level. http://msdn.microsoft.com/en-us/library/windows/desktop/ms686216(v=vs.85).aspx -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2190) Add CPU and memory limit options to the default container executor for Windows containers
[ https://issues.apache.org/jira/browse/YARN-2190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14351031#comment-14351031 ] Hudson commented on YARN-2190: -- FAILURE: Integrated in Hadoop-trunk-Commit #7274 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/7274/]) YARN-2190. Added CPU and memory limit options to the default container executor for Windows containers. Contributed by Chuan Liu (jianhe: rev 21101c01f242439ec8ec40fb3a9ab1991ae0adc7) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * BUILDING.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/WindowsSecureContainerExecutor.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DefaultContainerExecutor.java * hadoop-common-project/hadoop-common/src/main/winutils/task.c * hadoop-common-project/hadoop-common/src/main/winutils/winutils.vcxproj * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/ContainerExecutor.java * hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestWinUtils.java * hadoop-common-project/hadoop-common/src/main/winutils/win8sdk.props * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestContainerExecutor.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml Add CPU and memory limit options to the default container executor for Windows containers - Key: YARN-2190 URL: https://issues.apache.org/jira/browse/YARN-2190 Project: Hadoop YARN Issue Type: New Feature Components: nodemanager Reporter: Chuan Liu Assignee: Chuan Liu Fix For: 2.7.0 Attachments: YARN-2190-prototype.patch, YARN-2190.1.patch, YARN-2190.10.patch, YARN-2190.11.patch, YARN-2190.12.patch, YARN-2190.13.patch, YARN-2190.2.patch, YARN-2190.3.patch, YARN-2190.4.patch, YARN-2190.5.patch, YARN-2190.6.patch, YARN-2190.7.patch, YARN-2190.8.patch, YARN-2190.9.patch Yarn default container executor on Windows does not set the resource limit on the containers currently. The memory limit is enforced by a separate monitoring thread. The container implementation on Windows uses Job Object right now. The latest Windows (8 or later) API allows CPU and memory limits on the job objects. We want to add the new options to the executor that can set the limits on job objects thus provides resource enforcement at OS level. http://msdn.microsoft.com/en-us/library/windows/desktop/ms686216(v=vs.85).aspx -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2190) Add CPU and memory limit options to the default container executor for Windows containers
[
https://issues.apache.org/jira/browse/YARN-2190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14349880#comment-14349880
]
Hadoop QA commented on YARN-2190:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12702956/YARN-2190.13.patch
against trunk revision 952640f.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/6878//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6878//console
This message is automatically generated.
Add CPU and memory limit options to the default container executor for
Windows containers
-
Key: YARN-2190
URL: https://issues.apache.org/jira/browse/YARN-2190
Project: Hadoop YARN
Issue Type: New Feature
Components: nodemanager
Reporter: Chuan Liu
Assignee: Chuan Liu
Attachments: YARN-2190-prototype.patch, YARN-2190.1.patch,
YARN-2190.10.patch, YARN-2190.11.patch, YARN-2190.12.patch,
YARN-2190.13.patch, YARN-2190.2.patch, YARN-2190.3.patch, YARN-2190.4.patch,
YARN-2190.5.patch, YARN-2190.6.patch, YARN-2190.7.patch, YARN-2190.8.patch,
YARN-2190.9.patch
Yarn default container executor on Windows does not set the resource limit on
the containers currently. The memory limit is enforced by a separate
monitoring thread. The container implementation on Windows uses Job Object
right now. The latest Windows (8 or later) API allows CPU and memory limits
on the job objects. We want to add the new options to the executor that can
set the limits on job objects thus provides resource enforcement at OS level.
http://msdn.microsoft.com/en-us/library/windows/desktop/ms686216(v=vs.85).aspx
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2190) Add CPU and memory limit options to the default container executor for Windows containers
[
https://issues.apache.org/jira/browse/YARN-2190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14349845#comment-14349845
]
Hadoop QA commented on YARN-2190:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12702948/YARN-2190.12.patch
against trunk revision 952640f.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/6877//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6877//console
This message is automatically generated.
Add CPU and memory limit options to the default container executor for
Windows containers
-
Key: YARN-2190
URL: https://issues.apache.org/jira/browse/YARN-2190
Project: Hadoop YARN
Issue Type: New Feature
Components: nodemanager
Reporter: Chuan Liu
Assignee: Chuan Liu
Attachments: YARN-2190-prototype.patch, YARN-2190.1.patch,
YARN-2190.10.patch, YARN-2190.11.patch, YARN-2190.12.patch,
YARN-2190.13.patch, YARN-2190.2.patch, YARN-2190.3.patch, YARN-2190.4.patch,
YARN-2190.5.patch, YARN-2190.6.patch, YARN-2190.7.patch, YARN-2190.8.patch,
YARN-2190.9.patch
Yarn default container executor on Windows does not set the resource limit on
the containers currently. The memory limit is enforced by a separate
monitoring thread. The container implementation on Windows uses Job Object
right now. The latest Windows (8 or later) API allows CPU and memory limits
on the job objects. We want to add the new options to the executor that can
set the limits on job objects thus provides resource enforcement at OS level.
http://msdn.microsoft.com/en-us/library/windows/desktop/ms686216(v=vs.85).aspx
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
