[jira] [Commented] (YARN-2343) Improve error message on token expire exception
[ https://issues.apache.org/jira/browse/YARN-2343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14083523#comment-14083523 ] Hudson commented on YARN-2343: -- FAILURE: Integrated in Hadoop-Yarn-trunk #631 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/631/]) YARN-2343. Improve NMToken expire exception message. Contributed by Li Lu (jianhe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1615270) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/ContainerManagerImpl.java Improve error message on token expire exception --- Key: YARN-2343 URL: https://issues.apache.org/jira/browse/YARN-2343 Project: Hadoop YARN Issue Type: Improvement Reporter: Li Lu Assignee: Li Lu Priority: Trivial Labels: usability Fix For: 2.6.0 Attachments: YARN-2343-072314.patch, YARN-2343-080114.patch Some of token expire exception is triggered by wrong time settings on cluster nodes, but the current exception message does not explicitly address that. It would be helpful to add some message explicitly pointing out that this exception could be caused by machines out of sync in time, or even wrong time zone settings. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2343) Improve error message on token expire exception
[ https://issues.apache.org/jira/browse/YARN-2343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14083568#comment-14083568 ] Hudson commented on YARN-2343: -- SUCCESS: Integrated in Hadoop-Hdfs-trunk #1825 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1825/]) YARN-2343. Improve NMToken expire exception message. Contributed by Li Lu (jianhe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1615270) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/ContainerManagerImpl.java Improve error message on token expire exception --- Key: YARN-2343 URL: https://issues.apache.org/jira/browse/YARN-2343 Project: Hadoop YARN Issue Type: Improvement Reporter: Li Lu Assignee: Li Lu Priority: Trivial Labels: usability Fix For: 2.6.0 Attachments: YARN-2343-072314.patch, YARN-2343-080114.patch Some of token expire exception is triggered by wrong time settings on cluster nodes, but the current exception message does not explicitly address that. It would be helpful to add some message explicitly pointing out that this exception could be caused by machines out of sync in time, or even wrong time zone settings. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2343) Improve error message on token expire exception
[ https://issues.apache.org/jira/browse/YARN-2343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14083588#comment-14083588 ] Hudson commented on YARN-2343: -- SUCCESS: Integrated in Hadoop-Mapreduce-trunk #1850 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1850/]) YARN-2343. Improve NMToken expire exception message. Contributed by Li Lu (jianhe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1615270) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/ContainerManagerImpl.java Improve error message on token expire exception --- Key: YARN-2343 URL: https://issues.apache.org/jira/browse/YARN-2343 Project: Hadoop YARN Issue Type: Improvement Reporter: Li Lu Assignee: Li Lu Priority: Trivial Labels: usability Fix For: 2.6.0 Attachments: YARN-2343-072314.patch, YARN-2343-080114.patch Some of token expire exception is triggered by wrong time settings on cluster nodes, but the current exception message does not explicitly address that. It would be helpful to add some message explicitly pointing out that this exception could be caused by machines out of sync in time, or even wrong time zone settings. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (YARN-2373) WebAppUtils Should Use configuration.getPassword for Accessing SSL Passwords
[ https://issues.apache.org/jira/browse/YARN-2373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Larry McCay updated YARN-2373: -- Attachment: YARN-2373.patch Patch introduces the use of Configuration.getPassword to enable the use of the credential provider API in lieu of clear text passwords - while maintaining backward compatibility. WebAppUtils Should Use configuration.getPassword for Accessing SSL Passwords Key: YARN-2373 URL: https://issues.apache.org/jira/browse/YARN-2373 Project: Hadoop YARN Issue Type: Bug Reporter: Larry McCay Attachments: YARN-2373.patch As part of HADOOP-10904, this jira represents a change to WebAppUtils to uptake the use of the credential provider API through the new method on Configuration called getPassword. This provides an alternative to storing the passwords in clear text within the ssl-server.xml file while maintaining backward compatibility with that behavior. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2373) WebAppUtils Should Use configuration.getPassword for Accessing SSL Passwords
[
https://issues.apache.org/jira/browse/YARN-2373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14083632#comment-14083632
]
Hadoop QA commented on YARN-2373:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12659480/YARN-2373.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4515//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4515//console
This message is automatically generated.
WebAppUtils Should Use configuration.getPassword for Accessing SSL Passwords
Key: YARN-2373
URL: https://issues.apache.org/jira/browse/YARN-2373
Project: Hadoop YARN
Issue Type: Bug
Reporter: Larry McCay
Attachments: YARN-2373.patch
As part of HADOOP-10904, this jira represents a change to WebAppUtils to
uptake the use of the credential provider API through the new method on
Configuration called getPassword.
This provides an alternative to storing the passwords in clear text within
the ssl-server.xml file while maintaining backward compatibility with that
behavior.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2373) WebAppUtils Should Use configuration.getPassword for Accessing SSL Passwords
[ https://issues.apache.org/jira/browse/YARN-2373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14083661#comment-14083661 ] Larry McCay commented on YARN-2373: --- Existing tests may cover backward compatibility. I am considering whether to make the new static method public for general use by WebApps or to make it package protected and add a new test class for it - either way would make testing the new behavior easier. WebAppUtils Should Use configuration.getPassword for Accessing SSL Passwords Key: YARN-2373 URL: https://issues.apache.org/jira/browse/YARN-2373 Project: Hadoop YARN Issue Type: Bug Reporter: Larry McCay Attachments: YARN-2373.patch As part of HADOOP-10904, this jira represents a change to WebAppUtils to uptake the use of the credential provider API through the new method on Configuration called getPassword. This provides an alternative to storing the passwords in clear text within the ssl-server.xml file while maintaining backward compatibility with that behavior. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-796) Allow for (admin) labels on nodes and resource-requests
[ https://issues.apache.org/jira/browse/YARN-796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14083753#comment-14083753 ] Yuliya Feldman commented on YARN-796: - I am out of country now with very poor internet connectivity, so won't be able to answer comprehensively. To: [~ste...@apache.org] Really appreciate your comments I definitely agree with majority of the comments you made. Especially with how much code it takes to add a single method to rmadmin command - may be we missed something, but it is really too much. regarding wrapper on top of LabelManager to behave as a service - in realy life situation service is instantiated once per process - which is exactly what we need, as it is really a singleton, but since UnitTests create service per unit test it created issues with Service States in this case. About waiting for 6 secs between tests - allowing labels ile to reload - can be reduced further. Allow for (admin) labels on nodes and resource-requests --- Key: YARN-796 URL: https://issues.apache.org/jira/browse/YARN-796 Project: Hadoop YARN Issue Type: Sub-task Affects Versions: 2.4.1 Reporter: Arun C Murthy Assignee: Wangda Tan Attachments: LabelBasedScheduling.pdf, Node-labels-Requirements-Design-doc-V1.pdf, YARN-796.patch, YARN-796.patch4 It will be useful for admins to specify labels for nodes. Examples of labels are OS, processor architecture etc. We should expose these labels and allow applications to specify labels on resource-requests. Obviously we need to support admin operations on adding/removing node labels. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (YARN-2373) WebAppUtils Should Use configuration.getPassword for Accessing SSL Passwords
[ https://issues.apache.org/jira/browse/YARN-2373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Larry McCay updated YARN-2373: -- Attachment: YARN-2373.patch Patch to add a test and make the added functionality in webapputils more testable. WebAppUtils Should Use configuration.getPassword for Accessing SSL Passwords Key: YARN-2373 URL: https://issues.apache.org/jira/browse/YARN-2373 Project: Hadoop YARN Issue Type: Bug Reporter: Larry McCay Attachments: YARN-2373.patch, YARN-2373.patch As part of HADOOP-10904, this jira represents a change to WebAppUtils to uptake the use of the credential provider API through the new method on Configuration called getPassword. This provides an alternative to storing the passwords in clear text within the ssl-server.xml file while maintaining backward compatibility with that behavior. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2373) WebAppUtils Should Use configuration.getPassword for Accessing SSL Passwords
[
https://issues.apache.org/jira/browse/YARN-2373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14083779#comment-14083779
]
Hadoop QA commented on YARN-2373:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12659508/YARN-2373.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:red}-1 release audit{color}. The applied patch generated 1
release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4516//testReport/
Release audit warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/4516//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4516//console
This message is automatically generated.
WebAppUtils Should Use configuration.getPassword for Accessing SSL Passwords
Key: YARN-2373
URL: https://issues.apache.org/jira/browse/YARN-2373
Project: Hadoop YARN
Issue Type: Bug
Reporter: Larry McCay
Attachments: YARN-2373.patch, YARN-2373.patch
As part of HADOOP-10904, this jira represents a change to WebAppUtils to
uptake the use of the credential provider API through the new method on
Configuration called getPassword.
This provides an alternative to storing the passwords in clear text within
the ssl-server.xml file while maintaining backward compatibility with that
behavior.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (YARN-2373) WebAppUtils Should Use configuration.getPassword for Accessing SSL Passwords
[ https://issues.apache.org/jira/browse/YARN-2373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Larry McCay updated YARN-2373: -- Attachment: YARN-2373.patch Ooops - forgot to recreate the patch after adding the license. Reattaching with license added. WebAppUtils Should Use configuration.getPassword for Accessing SSL Passwords Key: YARN-2373 URL: https://issues.apache.org/jira/browse/YARN-2373 Project: Hadoop YARN Issue Type: Bug Reporter: Larry McCay Attachments: YARN-2373.patch, YARN-2373.patch, YARN-2373.patch As part of HADOOP-10904, this jira represents a change to WebAppUtils to uptake the use of the credential provider API through the new method on Configuration called getPassword. This provides an alternative to storing the passwords in clear text within the ssl-server.xml file while maintaining backward compatibility with that behavior. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (YARN-1354) Recover applications upon nodemanager restart
[ https://issues.apache.org/jira/browse/YARN-1354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe updated YARN-1354: - Attachment: YARN-1354-v6.patch Thanks for the comments, Junping! bq. One way is to wrapper it as PB object (keep writable fields as bytes) The patch is already wrapping the credentials in a protobuf, specifically ContainerManagerApplicationProto. So if/when credentials are stored differently after YARN-668 then we can obsolete this field and move to other new fields. The core problem is that at this level Credentials and Tokens are effectively opaque -- we can't piece them together ourselves and must delegate to them for load/store. Unfortunately they are Writables which are notoriously problematic when it comes to updating and dealing with different versions. It's possible if they explicitly handle it themselves (e.g.: write out schema versions and switch on it during load), however they don't typically do that. I agree that YARN-668 is the proper place to discuss how best to migrate Credentials and Tokens to a more manageable infrastructure for supporting upgrades. bq. Shall we change the name of finishApplication() to storeFinishedApplication() which sounds more precisely to actual work in store layer? Sounds good. I updated the method name to storeFinishedApplication. I also noticed that the change in YARN-1885 was only wired into the new node registration path. When an NM restarts it is going to go through the reconnected node path which was _not_ expecting applications to be out of sync with the RM. Therefore I updated the reconnected node path to forward the applications running on the node and have the RM inform the NM to finish the application if it is no longer active. Recover applications upon nodemanager restart - Key: YARN-1354 URL: https://issues.apache.org/jira/browse/YARN-1354 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager Affects Versions: 2.3.0 Reporter: Jason Lowe Assignee: Jason Lowe Attachments: YARN-1354-v1.patch, YARN-1354-v2-and-YARN-1987-and-YARN-1362.patch, YARN-1354-v3.patch, YARN-1354-v4.patch, YARN-1354-v5.patch, YARN-1354-v6.patch The set of active applications in the nodemanager context need to be recovered for work-preserving nodemanager restart -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2373) WebAppUtils Should Use configuration.getPassword for Accessing SSL Passwords
[
https://issues.apache.org/jira/browse/YARN-2373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14083807#comment-14083807
]
Hadoop QA commented on YARN-2373:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12659512/YARN-2373.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4517//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4517//console
This message is automatically generated.
WebAppUtils Should Use configuration.getPassword for Accessing SSL Passwords
Key: YARN-2373
URL: https://issues.apache.org/jira/browse/YARN-2373
Project: Hadoop YARN
Issue Type: Bug
Reporter: Larry McCay
Attachments: YARN-2373.patch, YARN-2373.patch, YARN-2373.patch
As part of HADOOP-10904, this jira represents a change to WebAppUtils to
uptake the use of the credential provider API through the new method on
Configuration called getPassword.
This provides an alternative to storing the passwords in clear text within
the ssl-server.xml file while maintaining backward compatibility with that
behavior.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-1354) Recover applications upon nodemanager restart
[
https://issues.apache.org/jira/browse/YARN-1354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14083820#comment-14083820
]
Hadoop QA commented on YARN-1354:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12659513/YARN-1354-v6.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 7 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4518//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4518//console
This message is automatically generated.
Recover applications upon nodemanager restart
-
Key: YARN-1354
URL: https://issues.apache.org/jira/browse/YARN-1354
Project: Hadoop YARN
Issue Type: Sub-task
Components: nodemanager
Affects Versions: 2.3.0
Reporter: Jason Lowe
Assignee: Jason Lowe
Attachments: YARN-1354-v1.patch,
YARN-1354-v2-and-YARN-1987-and-YARN-1362.patch, YARN-1354-v3.patch,
YARN-1354-v4.patch, YARN-1354-v5.patch, YARN-1354-v6.patch
The set of active applications in the nodemanager context need to be
recovered for work-preserving nodemanager restart
--
This message was sent by Atlassian JIRA
(v6.2#6252)
