[jira] [Commented] (YARN-2225) Turn the virtual memory check to be off by default
[ https://issues.apache.org/jira/browse/YARN-2225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15156615#comment-15156615 ] Akira AJISAKA commented on YARN-2225: - bq. IMHO, I prefer to make the default value of the vmem ratio larger. How about closing this issue and doing it on another jira(or, moving HADOOP-11364 to YARN issue) Agree. Closing this issue and moving HADOOP-11364 to YARN. > Turn the virtual memory check to be off by default > -- > > Key: YARN-2225 > URL: https://issues.apache.org/jira/browse/YARN-2225 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Anubhav Dhoot >Assignee: Anubhav Dhoot > Attachments: YARN-2225.patch > > > The virtual memory check may not be the best way to isolate applications. > Virtual memory is not the constrained resource. It would be better if we > limit the swapping of the task using swapiness instead. This patch will turn > this DEFAULT_NM_VMEM_CHECK_ENABLED off by default and let users turn it on if > they need to. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2225) Turn the virtual memory check to be off by default
[ https://issues.apache.org/jira/browse/YARN-2225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15155388#comment-15155388 ] Tsuyoshi Ozawa commented on YARN-2225: -- It's a bit too aggressive to disable vmem-check as discussed on this issue since some user enables vmem-check. IMHO, I prefer to make the default value of the vmem ratio larger. How about closing this issue and doing it on another jira(or, moving HADOOP-11364 to YARN issue) since the addressing problem is different from this issue? > Turn the virtual memory check to be off by default > -- > > Key: YARN-2225 > URL: https://issues.apache.org/jira/browse/YARN-2225 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Anubhav Dhoot >Assignee: Anubhav Dhoot > Attachments: YARN-2225.patch > > > The virtual memory check may not be the best way to isolate applications. > Virtual memory is not the constrained resource. It would be better if we > limit the swapping of the task using swapiness instead. This patch will turn > this DEFAULT_NM_VMEM_CHECK_ENABLED off by default and let users turn it on if > they need to. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2225) Turn the virtual memory check to be off by default
[ https://issues.apache.org/jira/browse/YARN-2225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15148449#comment-15148449 ] Akira AJISAKA commented on YARN-2225: - bq. Why not making vmem-pmem ratio larger to address the problem? Yes, we can avoid the error by making vmem-pmem ratio larger, so the problems are, the default value of the ratio is too small for Java 8 and the virtual memory check is enabled by default. Can we make the ratio larger, or make the vmem check disabled by default? > Turn the virtual memory check to be off by default > -- > > Key: YARN-2225 > URL: https://issues.apache.org/jira/browse/YARN-2225 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Anubhav Dhoot >Assignee: Anubhav Dhoot > Attachments: YARN-2225.patch > > > The virtual memory check may not be the best way to isolate applications. > Virtual memory is not the constrained resource. It would be better if we > limit the swapping of the task using swapiness instead. This patch will turn > this DEFAULT_NM_VMEM_CHECK_ENABLED off by default and let users turn it on if > they need to. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2225) Turn the virtual memory check to be off by default
[ https://issues.apache.org/jira/browse/YARN-2225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15148329#comment-15148329 ] Tsuyoshi Ozawa commented on YARN-2225: -- Why not making vmem-pmem ratio larger to address the problem? > Turn the virtual memory check to be off by default > -- > > Key: YARN-2225 > URL: https://issues.apache.org/jira/browse/YARN-2225 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Anubhav Dhoot >Assignee: Anubhav Dhoot > Attachments: YARN-2225.patch > > > The virtual memory check may not be the best way to isolate applications. > Virtual memory is not the constrained resource. It would be better if we > limit the swapping of the task using swapiness instead. This patch will turn > this DEFAULT_NM_VMEM_CHECK_ENABLED off by default and let users turn it on if > they need to. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2225) Turn the virtual memory check to be off by default
[ https://issues.apache.org/jira/browse/YARN-2225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15148306#comment-15148306 ] Akira AJISAKA commented on YARN-2225: - I'm thinking the default value should be changed in branch-2 as well. We did some incompatible changes in branch-2 if the change is really needed. There are 14 incompatible changes in 2.6.0/2.7.0. https://issues.apache.org/jira/issues/?jql=project%20in%20(YARN%2C%20HADOOP%2C%20HDFS%2C%20MAPREDUCE)%20AND%20status%20in%20(Resolved%2C%20Closed)%20AND%20fixVersion%20in%20(2.6.0%2C%202.7.0)%20AND%20%22Hadoop%20Flags%22%20%3D%20%22Incompatible%20change%22 I fed up with hearing Java8 users' complaints, "The default value does not work.". > Turn the virtual memory check to be off by default > -- > > Key: YARN-2225 > URL: https://issues.apache.org/jira/browse/YARN-2225 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Anubhav Dhoot >Assignee: Anubhav Dhoot > Attachments: YARN-2225.patch > > > The virtual memory check may not be the best way to isolate applications. > Virtual memory is not the constrained resource. It would be better if we > limit the swapping of the task using swapiness instead. This patch will turn > this DEFAULT_NM_VMEM_CHECK_ENABLED off by default and let users turn it on if > they need to. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2225) Turn the virtual memory check to be off by default
[
https://issues.apache.org/jira/browse/YARN-2225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14046480#comment-14046480
]
Hadoop QA commented on YARN-2225:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12652908/YARN-2225.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4126//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4126//console
This message is automatically generated.
Turn the virtual memory check to be off by default
--
Key: YARN-2225
URL: https://issues.apache.org/jira/browse/YARN-2225
Project: Hadoop YARN
Issue Type: Bug
Reporter: Anubhav Dhoot
Assignee: Anubhav Dhoot
Attachments: YARN-2225.patch
The virtual memory check may not be the best way to isolate applications.
Virtual memory is not the constrained resource. It would be better if we
limit the swapping of the task using swapiness instead. This patch will turn
this DEFAULT_NM_VMEM_CHECK_ENABLED off by default and let users turn it on if
they need to.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2225) Turn the virtual memory check to be off by default
[ https://issues.apache.org/jira/browse/YARN-2225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14046489#comment-14046489 ] Vinod Kumar Vavilapalli commented on YARN-2225: --- -1 for changing the default.. This breaks compatibility. bq. The virtual memory check may not be the best way to isolate applications. Virtual memory is not the constrained resource. I still see a lot of apps that needs isolation w.r.t vmem. It's not about which resource is constrained, it is about isolation. We already identify physical memory as constrained and use that as the main scheduling dimension. Turn the virtual memory check to be off by default -- Key: YARN-2225 URL: https://issues.apache.org/jira/browse/YARN-2225 Project: Hadoop YARN Issue Type: Bug Reporter: Anubhav Dhoot Assignee: Anubhav Dhoot Attachments: YARN-2225.patch The virtual memory check may not be the best way to isolate applications. Virtual memory is not the constrained resource. It would be better if we limit the swapping of the task using swapiness instead. This patch will turn this DEFAULT_NM_VMEM_CHECK_ENABLED off by default and let users turn it on if they need to. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2225) Turn the virtual memory check to be off by default
[ https://issues.apache.org/jira/browse/YARN-2225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14046494#comment-14046494 ] Karthik Kambatla commented on YARN-2225: According to your compatibility guide, The default values of Hadoop-defined properties can be changed across minor/major releases, but will remain the same across point releases within a minor release. So, in letter, we can't target 2.4.1 or 2.5.1, but can target 2.5 or 2.6. In spirit, virtual memory check has been a pain and we end up recommending users to turn it off. Turn the virtual memory check to be off by default -- Key: YARN-2225 URL: https://issues.apache.org/jira/browse/YARN-2225 Project: Hadoop YARN Issue Type: Bug Reporter: Anubhav Dhoot Assignee: Anubhav Dhoot Attachments: YARN-2225.patch The virtual memory check may not be the best way to isolate applications. Virtual memory is not the constrained resource. It would be better if we limit the swapping of the task using swapiness instead. This patch will turn this DEFAULT_NM_VMEM_CHECK_ENABLED off by default and let users turn it on if they need to. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2225) Turn the virtual memory check to be off by default
[ https://issues.apache.org/jira/browse/YARN-2225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14046566#comment-14046566 ] Vinod Kumar Vavilapalli commented on YARN-2225: --- It breaks compatibility w.r.t behavior - asking existing users who care about it to turn it on explicitly. bq. In spirit, virtual memory check has been a pain and we end up recommending users to turn it off. I have had a different experience. It indeed is a pain for testing both in Hadoop and in high level frameworks, but it's been invaluable in real life clusters to thwart run away jobs - specifically the non-java ones - from affecting the cluster. Turn the virtual memory check to be off by default -- Key: YARN-2225 URL: https://issues.apache.org/jira/browse/YARN-2225 Project: Hadoop YARN Issue Type: Bug Reporter: Anubhav Dhoot Assignee: Anubhav Dhoot Attachments: YARN-2225.patch The virtual memory check may not be the best way to isolate applications. Virtual memory is not the constrained resource. It would be better if we limit the swapping of the task using swapiness instead. This patch will turn this DEFAULT_NM_VMEM_CHECK_ENABLED off by default and let users turn it on if they need to. -- This message was sent by Atlassian JIRA (v6.2#6252)
