[jira] [Updated] (YARN-571) User should not be part of ContainerLaunchContext
[ https://issues.apache.org/jira/browse/YARN-571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Omkar Vinit Joshi updated YARN-571: --- Attachment: YARN-571.20130522.2.patch We are removing user from containerLaunchContext as it is redundant because containerTokenIdentifier already contains user name. Now ContainerManager will get username from containerTokenIdentifier. User should not be part of ContainerLaunchContext - Key: YARN-571 URL: https://issues.apache.org/jira/browse/YARN-571 Project: Hadoop YARN Issue Type: Sub-task Reporter: Hitesh Shah Assignee: Omkar Vinit Joshi Attachments: YARN-571-20130415.2.txt, YARN-571-20130418.txt, YARN-571.20130522.1.patch, YARN-571.20130522.2.patch, YARN-571.20130522.patch Today, a user is expected to set the user name in the CLC when either submitting an application or launching a container from the AM. This does not make sense as the user can/has been identified by the RM as part of the RPC layer. Solution would be to move the user information into either the Container object or directly into the ContainerToken which can then be used by the NM to launch the container. This user information would set into the container by the RM. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-578) NodeManager should use SecureIOUtils for serving and aggregating logs
[ https://issues.apache.org/jira/browse/YARN-578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13668505#comment-13668505 ] Omkar Vinit Joshi commented on YARN-578: Thanks [~sandyr] for reviewing.. Fixed all the comments.. NodeManager should use SecureIOUtils for serving and aggregating logs - Key: YARN-578 URL: https://issues.apache.org/jira/browse/YARN-578 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager Reporter: Vinod Kumar Vavilapalli Assignee: Omkar Vinit Joshi Attachments: yarn-578-20130426.patch, YARN-578-20130506.patch, YARN-578-20130520.1.patch, YARN-578-20130520.branch-2.patch, YARN-578-20130520.patch, YARN-578-20130528.patch Log servlets for serving logs and the ShuffleService for serving intermediate outputs both should use SecureIOUtils for avoiding symlink attacks. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-578) NodeManager should use SecureIOUtils for serving and aggregating logs
[ https://issues.apache.org/jira/browse/YARN-578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Omkar Vinit Joshi updated YARN-578: --- Attachment: YARN-578-20130528.patch NodeManager should use SecureIOUtils for serving and aggregating logs - Key: YARN-578 URL: https://issues.apache.org/jira/browse/YARN-578 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager Reporter: Vinod Kumar Vavilapalli Assignee: Omkar Vinit Joshi Attachments: yarn-578-20130426.patch, YARN-578-20130506.patch, YARN-578-20130520.1.patch, YARN-578-20130520.branch-2.patch, YARN-578-20130520.patch, YARN-578-20130528.patch Log servlets for serving logs and the ShuffleService for serving intermediate outputs both should use SecureIOUtils for avoiding symlink attacks. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-714) AMRM protocol changes for sending NMToken list
[ https://issues.apache.org/jira/browse/YARN-714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13668695#comment-13668695 ] Omkar Vinit Joshi commented on YARN-714: updating patch with protocol changes with pbimpl changes. AMRM protocol changes for sending NMToken list -- Key: YARN-714 URL: https://issues.apache.org/jira/browse/YARN-714 Project: Hadoop YARN Issue Type: Sub-task Reporter: Omkar Vinit Joshi Assignee: Omkar Vinit Joshi Attachments: YARN-714-20130521.patch NMToken will be sent to AM on allocate call if 1) AM doesn't already have NMToken for the underlying NM 2) Key rolled over on RM and AM gets new container on the same NM. On allocate call RM will send a consolidated list of all required NMTokens. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-714) AMRM protocol changes for sending NMToken list
[ https://issues.apache.org/jira/browse/YARN-714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Omkar Vinit Joshi updated YARN-714: --- Attachment: YARN-714-20130528.patch AMRM protocol changes for sending NMToken list -- Key: YARN-714 URL: https://issues.apache.org/jira/browse/YARN-714 Project: Hadoop YARN Issue Type: Sub-task Reporter: Omkar Vinit Joshi Assignee: Omkar Vinit Joshi Attachments: YARN-714-20130521.patch, YARN-714-20130528.patch NMToken will be sent to AM on allocate call if 1) AM doesn't already have NMToken for the underlying NM 2) Key rolled over on RM and AM gets new container on the same NM. On allocate call RM will send a consolidated list of all required NMTokens. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-714) AMRM protocol changes for sending NMToken list
[ https://issues.apache.org/jira/browse/YARN-714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13668744#comment-13668744 ] Omkar Vinit Joshi commented on YARN-714: Fixing findbug warnings and javadoc warnings.. AMRM protocol changes for sending NMToken list -- Key: YARN-714 URL: https://issues.apache.org/jira/browse/YARN-714 Project: Hadoop YARN Issue Type: Sub-task Reporter: Omkar Vinit Joshi Assignee: Omkar Vinit Joshi Attachments: YARN-714-20130521.patch, YARN-714-20130528.1.patch, YARN-714-20130528.patch NMToken will be sent to AM on allocate call if 1) AM doesn't already have NMToken for the underlying NM 2) Key rolled over on RM and AM gets new container on the same NM. On allocate call RM will send a consolidated list of all required NMTokens. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-714) AMRM protocol changes for sending NMToken list
[ https://issues.apache.org/jira/browse/YARN-714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Omkar Vinit Joshi updated YARN-714: --- Attachment: YARN-714-20130528.1.patch AMRM protocol changes for sending NMToken list -- Key: YARN-714 URL: https://issues.apache.org/jira/browse/YARN-714 Project: Hadoop YARN Issue Type: Sub-task Reporter: Omkar Vinit Joshi Assignee: Omkar Vinit Joshi Attachments: YARN-714-20130521.patch, YARN-714-20130528.1.patch, YARN-714-20130528.patch NMToken will be sent to AM on allocate call if 1) AM doesn't already have NMToken for the underlying NM 2) Key rolled over on RM and AM gets new container on the same NM. On allocate call RM will send a consolidated list of all required NMTokens. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-435) Make it easier to access cluster topology information in an AM
[ https://issues.apache.org/jira/browse/YARN-435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13669570#comment-13669570 ] Omkar Vinit Joshi commented on YARN-435: After discussion with [~vinodkv] yesterday.. Here are the possible solutions to this * At the time of application master registration along with AMToken also issue ClientDelegationToken or may be on request in AMRMProtocol. ** Advantages: -AM doesn't need to request token to get additional information (no kerberos authentication for secure case). ** Disadvantages: - *** AM has to open 2 connections and manage 2 tokens (AMToken and ClientDelegationToken) *** AM can now do all the activities which earlier only client was allowed to do (getAllApps, forceKillApp, submitApp) * Create a new interface something like, ClusterInfo and add ClusterNodes and ClusterMatrics and similar info to it. Let ClientRMProtocol and AMRMProtocol extend this. ** Advantages: - *** AM has to create only one connection *** AM doesn't get by default or on request ClientDelegationToken (for secure env.). So AM is not allowed to do app activities on ClientRMProtocol (submitApp, killApp, getAppStatus) unless Client itself share this token with AM. *** Connection management will be very simple for AM and will get all required info. ** Disadvantages: - *** AMRMProtocol will get modified. As AM-RM heartbeat is mandatory for all active AMs, adding this will add burden on ApplicationMasterService. * Allow ClientRMProtocol to accept AMToken too along with ClientDelegationToken. Thereby AM can communicate with RM even on ClientRMProtocol. ** Advantages: -No need to share/create ClientDelegationToken ** Disadvantages: -same as issuing ClientDelegationToken. Make it easier to access cluster topology information in an AM -- Key: YARN-435 URL: https://issues.apache.org/jira/browse/YARN-435 Project: Hadoop YARN Issue Type: Sub-task Reporter: Hitesh Shah Assignee: Hitesh Shah ClientRMProtocol exposes a getClusterNodes api that provides a report on all nodes in the cluster including their rack information. However, this requires the AM to open and establish a separate connection to the RM in addition to one for the AMRMProtocol. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-578) NodeManager should use SecureIOUtils for serving and aggregating logs
[ https://issues.apache.org/jira/browse/YARN-578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Omkar Vinit Joshi updated YARN-578: --- Attachment: YARN-578-20130529.patch NodeManager should use SecureIOUtils for serving and aggregating logs - Key: YARN-578 URL: https://issues.apache.org/jira/browse/YARN-578 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager Reporter: Vinod Kumar Vavilapalli Assignee: Omkar Vinit Joshi Attachments: yarn-578-20130426.patch, YARN-578-20130506.patch, YARN-578-20130520.1.patch, YARN-578-20130520.branch-2.patch, YARN-578-20130520.patch, YARN-578-20130528.1.patch, YARN-578-20130528.patch, YARN-578-20130529.patch Log servlets for serving logs and the ShuffleService for serving intermediate outputs both should use SecureIOUtils for avoiding symlink attacks. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (YARN-552) Expose resource metrics as part of YarnClusterMetrics
[ https://issues.apache.org/jira/browse/YARN-552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Omkar Vinit Joshi reassigned YARN-552: -- Assignee: Omkar Vinit Joshi Expose resource metrics as part of YarnClusterMetrics - Key: YARN-552 URL: https://issues.apache.org/jira/browse/YARN-552 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Affects Versions: 2.0.3-alpha Reporter: Harsh J Assignee: Omkar Vinit Joshi Priority: Minor Right now, the YarnClusterMetrics just has the total number of node managers returned in it (when queried from a Client - RM). It would be useful to also expose NodeManager resource capacities and scheduler max/min resource limits over it to allow clients to pre-determine or pre-compute runtime feasibility without having to request an Application first to get some of this information. This does not need to be an incompatible change, and we can continue exposing the same values as part of the GetNewApplicationResponse too. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-578) NodeManager should use SecureIOUtils for serving and aggregating logs
[ https://issues.apache.org/jira/browse/YARN-578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13669891#comment-13669891 ] Omkar Vinit Joshi commented on YARN-578: The patch is same for both branch-2 and trunk.. compiled and tested locally. NodeManager should use SecureIOUtils for serving and aggregating logs - Key: YARN-578 URL: https://issues.apache.org/jira/browse/YARN-578 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager Reporter: Vinod Kumar Vavilapalli Assignee: Omkar Vinit Joshi Attachments: yarn-578-20130426.patch, YARN-578-20130506.patch, YARN-578-20130520.1.patch, YARN-578-20130520.branch-2.patch, YARN-578-20130520.patch, YARN-578-20130528.1.patch, YARN-578-20130528.patch, YARN-578-20130529.patch Log servlets for serving logs and the ShuffleService for serving intermediate outputs both should use SecureIOUtils for avoiding symlink attacks. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-692) Creating AMNMToken master key on RM and sharing it with NM as a part of RM-NM heartbeat.
[ https://issues.apache.org/jira/browse/YARN-692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13670997#comment-13670997 ] Omkar Vinit Joshi commented on YARN-692: As a part of this patch only enabling following features. * During start RM generates NMTokenMasterKey * For Every NodeManager registration master key is shared along with containerTokenMasterKey * If there is a rollover then in the hearbeat master key will be shared. RM at present doesn't send NMToken at container allocation (will be there in YARN-693). AM at present doesn't store NMToken ...(will be there in YARN-693) AM at present doesn't send these tokens to NM for all AM-NM communication (will be there in YARN-694). NM at present doesn't verify the NMToken (will be there in YARN-694). Added test cases to verify RMNMSecretKeys - at rm start and at roll over. Creating AMNMToken master key on RM and sharing it with NM as a part of RM-NM heartbeat. Key: YARN-692 URL: https://issues.apache.org/jira/browse/YARN-692 Project: Hadoop YARN Issue Type: Bug Reporter: Omkar Vinit Joshi Assignee: Omkar Vinit Joshi This is related to YARN-613 . Here we will be implementing AMNMToken generation on RM side and sharing it with NM during RM-NM heartbeat. As a part of this JIRA mater key will only be made available to NM but there will be no validation done until AM-NM communication is fixed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-733) TestNMClient fails occasionally
[
https://issues.apache.org/jira/browse/YARN-733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13671054#comment-13671054
]
Omkar Vinit Joshi commented on YARN-733:
Yes this is occasionally failing for me too ...
TestNMClient fails occasionally
---
Key: YARN-733
URL: https://issues.apache.org/jira/browse/YARN-733
Project: Hadoop YARN
Issue Type: Bug
Reporter: Zhijie Shen
Assignee: Zhijie Shen
The problem happens at:
{code}
// getContainerStatus can be called after stopContainer
try {
ContainerStatus status = nmClient.getContainerStatus(
container.getId(), container.getNodeId(),
container.getContainerToken());
assertEquals(container.getId(), status.getContainerId());
assertEquals(ContainerState.RUNNING, status.getState());
assertTrue( + i, status.getDiagnostics().contains(
Container killed by the ApplicationMaster.));
assertEquals(-1000, status.getExitStatus());
} catch (YarnRemoteException e) {
fail(Exception is not expected);
}
{code}
NMClientImpl#stopContainer returns, but container hasn't been stopped
immediately. ContainerManangerImpl implements stopContainer in async style.
Therefore, the container's status is in transition.
NMClientImpl#getContainerStatus immediately after stopContainer will get
either the RUNNING status or the COMPLETE one.
There will be the similar problem wrt NMClientImpl#startContainer.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-692) Creating NMToken master key on RM and sharing it with NM as a part of RM-NM heartbeat.
[ https://issues.apache.org/jira/browse/YARN-692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Omkar Vinit Joshi updated YARN-692: --- Description: This is related to YARN-613 . Here we will be implementing NMToken generation on RM side and sharing it with NM during RM-NM heartbeat. As a part of this JIRA mater key will only be made available to NM but there will be no validation done until AM-NM communication is fixed. (was: This is related to YARN-613 . Here we will be implementing AMNMToken generation on RM side and sharing it with NM during RM-NM heartbeat. As a part of this JIRA mater key will only be made available to NM but there will be no validation done until AM-NM communication is fixed.) Creating NMToken master key on RM and sharing it with NM as a part of RM-NM heartbeat. -- Key: YARN-692 URL: https://issues.apache.org/jira/browse/YARN-692 Project: Hadoop YARN Issue Type: Bug Reporter: Omkar Vinit Joshi Assignee: Omkar Vinit Joshi Attachments: YARN-692.20130530.1.patch This is related to YARN-613 . Here we will be implementing NMToken generation on RM side and sharing it with NM during RM-NM heartbeat. As a part of this JIRA mater key will only be made available to NM but there will be no validation done until AM-NM communication is fixed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-693) Sending NMToken to AM on allocate call if container is allocated for the first time on underlying NM for given AM
[ https://issues.apache.org/jira/browse/YARN-693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Omkar Vinit Joshi updated YARN-693: --- Description: This is part of YARN-613. Here AM will receive NMToken in allocate call only if * AM is receiving first container on underlying NM. (Exception. if RM has restarted in between then it will receive it again for work preserving environment). This NMToken will be used for AM-NM authentication; however this will be implemented in another ticket. AM will have to remember this token for communicating with NM. was: This is part of YARN-613. Here AM will receive AMNMToken in allocate call only if * AM is receiving first container on underlying NM. (Exception. if RM has restarted in between then it will receive it again for work preserving environment). This AMNMToken will be used for AM-NM authentication; however this will be implemented in another ticket. AM will have to remember this token for communicating with NM. Sending NMToken to AM on allocate call if container is allocated for the first time on underlying NM for given AM - Key: YARN-693 URL: https://issues.apache.org/jira/browse/YARN-693 Project: Hadoop YARN Issue Type: Bug Reporter: Omkar Vinit Joshi Assignee: Omkar Vinit Joshi This is part of YARN-613. Here AM will receive NMToken in allocate call only if * AM is receiving first container on underlying NM. (Exception. if RM has restarted in between then it will receive it again for work preserving environment). This NMToken will be used for AM-NM authentication; however this will be implemented in another ticket. AM will have to remember this token for communicating with NM. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-692) Creating AMNMToken master key on RM and sharing it with NM as a part of RM-NM heartbeat.
[ https://issues.apache.org/jira/browse/YARN-692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Omkar Vinit Joshi updated YARN-692: --- Attachment: YARN-692.20130530.1.patch Creating AMNMToken master key on RM and sharing it with NM as a part of RM-NM heartbeat. Key: YARN-692 URL: https://issues.apache.org/jira/browse/YARN-692 Project: Hadoop YARN Issue Type: Bug Reporter: Omkar Vinit Joshi Assignee: Omkar Vinit Joshi Attachments: YARN-692.20130530.1.patch This is related to YARN-613 . Here we will be implementing AMNMToken generation on RM side and sharing it with NM during RM-NM heartbeat. As a part of this JIRA mater key will only be made available to NM but there will be no validation done until AM-NM communication is fixed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (YARN-738) TestClientRMTokens is failing irregularly while running all yarn tests
Omkar Vinit Joshi created YARN-738: -- Summary: TestClientRMTokens is failing irregularly while running all yarn tests Key: YARN-738 URL: https://issues.apache.org/jira/browse/YARN-738 Project: Hadoop YARN Issue Type: Bug Reporter: Omkar Vinit Joshi -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-692) Creating NMToken master key on RM and sharing it with NM as a part of RM-NM heartbeat.
[ https://issues.apache.org/jira/browse/YARN-692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Omkar Vinit Joshi updated YARN-692: --- Attachment: YARN-692.20130530.2.patch Fixing findbug and release warnings.. Creating NMToken master key on RM and sharing it with NM as a part of RM-NM heartbeat. -- Key: YARN-692 URL: https://issues.apache.org/jira/browse/YARN-692 Project: Hadoop YARN Issue Type: Bug Reporter: Omkar Vinit Joshi Assignee: Omkar Vinit Joshi Attachments: YARN-692.20130530.1.patch, YARN-692.20130530.2.patch This is related to YARN-613 . Here we will be implementing NMToken generation on RM side and sharing it with NM during RM-NM heartbeat. As a part of this JIRA mater key will only be made available to NM but there will be no validation done until AM-NM communication is fixed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-733) TestNMClient fails occasionally
[
https://issues.apache.org/jira/browse/YARN-733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13671712#comment-13671712
]
Omkar Vinit Joshi commented on YARN-733:
[~zjshen] small nit
bq. may still need some time to make the container actually started or stopped
because of its asynchronous
may still need some time to either start or stop the container because of its
asynchronous
I hope we are not doing getContainerStatus after Application is finished in
which case we won't have tokens at NM side for authentication.
TestNMClient fails occasionally
---
Key: YARN-733
URL: https://issues.apache.org/jira/browse/YARN-733
Project: Hadoop YARN
Issue Type: Bug
Reporter: Zhijie Shen
Assignee: Zhijie Shen
Attachments: YARN-733.1.patch
The problem happens at:
{code}
// getContainerStatus can be called after stopContainer
try {
ContainerStatus status = nmClient.getContainerStatus(
container.getId(), container.getNodeId(),
container.getContainerToken());
assertEquals(container.getId(), status.getContainerId());
assertEquals(ContainerState.RUNNING, status.getState());
assertTrue( + i, status.getDiagnostics().contains(
Container killed by the ApplicationMaster.));
assertEquals(-1000, status.getExitStatus());
} catch (YarnRemoteException e) {
fail(Exception is not expected);
}
{code}
NMClientImpl#stopContainer returns, but container hasn't been stopped
immediately. ContainerManangerImpl implements stopContainer in async style.
Therefore, the container's status is in transition.
NMClientImpl#getContainerStatus immediately after stopContainer will get
either the RUNNING status or the COMPLETE one.
There will be the similar problem wrt NMClientImpl#startContainer.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-694) AM uses the AMNMToken to authenticate all communication with NM. NM remembers and updates token across RM restart
[ https://issues.apache.org/jira/browse/YARN-694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13671819#comment-13671819 ] Omkar Vinit Joshi commented on YARN-694: Need to fix NodeId check as a part of this at NM side. (YARN-739) AM uses the AMNMToken to authenticate all communication with NM. NM remembers and updates token across RM restart - Key: YARN-694 URL: https://issues.apache.org/jira/browse/YARN-694 Project: Hadoop YARN Issue Type: Bug Reporter: Omkar Vinit Joshi Assignee: Omkar Vinit Joshi AM uses the AMNMToken to authenticate all the AM-NM communication. NM will validate AMNMToken in below manner * If AMNMToken is using current or previous master key then the AMNMToken is valid. In this case it will update its cache with this key corresponding to appId. * If AMNMToken is using the master key which is present in NM's cache corresponding to AM's appId then it will be validated based on this. * If AMNMToken is invalid then NM will reject AM calls. Modification for ContainerToken * At present RPC validates AM-NM communication based on ContainerToken. It will be replaced with AMNMToken. Also now onwards AM will use AMNMToken per NM (replacing earlier behavior of ContainerToken per container per NM). * startContainer in case of Secured environment is using ContainerToken from UGI YARN-617; however after this it will use it from the payload (Container). * ContainerToken will exist and it will only be used to validate the AM's container start request. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-694) AM uses the NMToken to authenticate all communication with NM. NM remembers and updates token across RM restart
[ https://issues.apache.org/jira/browse/YARN-694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Omkar Vinit Joshi updated YARN-694: --- Summary: AM uses the NMToken to authenticate all communication with NM. NM remembers and updates token across RM restart (was: AM uses the AMNMToken to authenticate all communication with NM. NM remembers and updates token across RM restart) AM uses the NMToken to authenticate all communication with NM. NM remembers and updates token across RM restart --- Key: YARN-694 URL: https://issues.apache.org/jira/browse/YARN-694 Project: Hadoop YARN Issue Type: Bug Reporter: Omkar Vinit Joshi Assignee: Omkar Vinit Joshi AM uses the AMNMToken to authenticate all the AM-NM communication. NM will validate AMNMToken in below manner * If AMNMToken is using current or previous master key then the AMNMToken is valid. In this case it will update its cache with this key corresponding to appId. * If AMNMToken is using the master key which is present in NM's cache corresponding to AM's appId then it will be validated based on this. * If AMNMToken is invalid then NM will reject AM calls. Modification for ContainerToken * At present RPC validates AM-NM communication based on ContainerToken. It will be replaced with AMNMToken. Also now onwards AM will use AMNMToken per NM (replacing earlier behavior of ContainerToken per container per NM). * startContainer in case of Secured environment is using ContainerToken from UGI YARN-617; however after this it will use it from the payload (Container). * ContainerToken will exist and it will only be used to validate the AM's container start request. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-694) AM uses the NMToken to authenticate all communication with NM. NM remembers and updates token across RM restart
[ https://issues.apache.org/jira/browse/YARN-694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Omkar Vinit Joshi updated YARN-694: --- Description: AM uses the NMToken to authenticate all the AM-NM communication. NM will validate NMToken in below manner * If NMToken is using current or previous master key then the NMToken is valid. In this case it will update its cache with this key corresponding to appId. * If NMToken is using the master key which is present in NM's cache corresponding to AM's appId then it will be validated based on this. * If NMToken is invalid then NM will reject AM calls. Modification for ContainerToken * At present RPC validates AM-NM communication based on ContainerToken. It will be replaced with NMToken. Also now onwards AM will use NMToken per NM (replacing earlier behavior of ContainerToken per container per NM). * startContainer in case of Secured environment is using ContainerToken from UGI YARN-617; however after this it will use it from the payload (Container). * ContainerToken will exist and it will only be used to validate the AM's container start request. was: AM uses the AMNMToken to authenticate all the AM-NM communication. NM will validate AMNMToken in below manner * If AMNMToken is using current or previous master key then the AMNMToken is valid. In this case it will update its cache with this key corresponding to appId. * If AMNMToken is using the master key which is present in NM's cache corresponding to AM's appId then it will be validated based on this. * If AMNMToken is invalid then NM will reject AM calls. Modification for ContainerToken * At present RPC validates AM-NM communication based on ContainerToken. It will be replaced with AMNMToken. Also now onwards AM will use AMNMToken per NM (replacing earlier behavior of ContainerToken per container per NM). * startContainer in case of Secured environment is using ContainerToken from UGI YARN-617; however after this it will use it from the payload (Container). * ContainerToken will exist and it will only be used to validate the AM's container start request. AM uses the NMToken to authenticate all communication with NM. NM remembers and updates token across RM restart --- Key: YARN-694 URL: https://issues.apache.org/jira/browse/YARN-694 Project: Hadoop YARN Issue Type: Bug Reporter: Omkar Vinit Joshi Assignee: Omkar Vinit Joshi AM uses the NMToken to authenticate all the AM-NM communication. NM will validate NMToken in below manner * If NMToken is using current or previous master key then the NMToken is valid. In this case it will update its cache with this key corresponding to appId. * If NMToken is using the master key which is present in NM's cache corresponding to AM's appId then it will be validated based on this. * If NMToken is invalid then NM will reject AM calls. Modification for ContainerToken * At present RPC validates AM-NM communication based on ContainerToken. It will be replaced with NMToken. Also now onwards AM will use NMToken per NM (replacing earlier behavior of ContainerToken per container per NM). * startContainer in case of Secured environment is using ContainerToken from UGI YARN-617; however after this it will use it from the payload (Container). * ContainerToken will exist and it will only be used to validate the AM's container start request. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-762) Javadoc params not matching in ContainerManagerImpl.authorizeRequest method
[ https://issues.apache.org/jira/browse/YARN-762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13674575#comment-13674575 ] Omkar Vinit Joshi commented on YARN-762: [~niranjanmaisnam] thanks niranjan... probably we will have to update it even more after YARN-613 Javadoc params not matching in ContainerManagerImpl.authorizeRequest method --- Key: YARN-762 URL: https://issues.apache.org/jira/browse/YARN-762 Project: Hadoop YARN Issue Type: Bug Components: nodemanager Reporter: Niranjan Singh Priority: Minor Attachments: YARN-762.patch In ContainerManagerImpl.authorizeRequest method the number of parameters passed are four where as in Javadoc the params are only three. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-775) stream jobs are not cleaning the Yarn local-dirs after container is released
[ https://issues.apache.org/jira/browse/YARN-775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13677641#comment-13677641 ] Omkar Vinit Joshi commented on YARN-775: No you don't need to explicitly set it to 0, as the default value is 0. stream jobs are not cleaning the Yarn local-dirs after container is released Key: YARN-775 URL: https://issues.apache.org/jira/browse/YARN-775 Project: Hadoop YARN Issue Type: Bug Affects Versions: 2.1.0-beta Reporter: yeshavora Assignee: Omkar Vinit Joshi Fix For: 2.1.0-beta Run a stream job: hadoop jar hadoop-streaming.jar -files file:///tmp/Tmp.py -input Tmp.py -output /tmp/Tmpout -mapper python Tmp.py -reducer NONE Container Dirs are not being cleaned after Stream job is completed/Killed/Failed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (YARN-785) Every startContainer request send a set of information (auxiliary service related information) which is redundant. Can be replace with single API.
Omkar Vinit Joshi created YARN-785: -- Summary: Every startContainer request send a set of information (auxiliary service related information) which is redundant. Can be replace with single API. Key: YARN-785 URL: https://issues.apache.org/jira/browse/YARN-785 Project: Hadoop YARN Issue Type: Improvement Reporter: Omkar Vinit Joshi Assignee: Omkar Vinit Joshi At present we are sending bunch of information mainly related to auxiliary serivces to whoever launches the container. This is an added overhead for NM. Instead we can expose this as an API then using NMToken client can get this information whenever it needs it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-785) Every startContainer request send a set of information (auxiliary service related information) which is redundant. Can be replaced with single NodeManager API.
[ https://issues.apache.org/jira/browse/YARN-785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Omkar Vinit Joshi updated YARN-785: --- Summary: Every startContainer request send a set of information (auxiliary service related information) which is redundant. Can be replaced with single NodeManager API. (was: Every startContainer request send a set of information (auxiliary service related information) which is redundant. Can be replace with single API.) Every startContainer request send a set of information (auxiliary service related information) which is redundant. Can be replaced with single NodeManager API. --- Key: YARN-785 URL: https://issues.apache.org/jira/browse/YARN-785 Project: Hadoop YARN Issue Type: Improvement Reporter: Omkar Vinit Joshi Assignee: Omkar Vinit Joshi At present we are sending bunch of information mainly related to auxiliary serivces to whoever launches the container. This is an added overhead for NM. Instead we can expose this as an API then using NMToken client can get this information whenever it needs it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (YARN-797) Remove KIND field from ContainerTokenIdentifier as it is not useful.
Omkar Vinit Joshi created YARN-797: -- Summary: Remove KIND field from ContainerTokenIdentifier as it is not useful. Key: YARN-797 URL: https://issues.apache.org/jira/browse/YARN-797 Project: Hadoop YARN Issue Type: Bug Reporter: Omkar Vinit Joshi As we already have removed ContainerToken, ClientToken etc. classes there is no point in keeping KIND field. This was used while decodingIdentifier. (Reflection based on KIND). probably either we should remove or update this code as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-692) Creating NMToken master key on RM and sharing it with NM as a part of RM-NM heartbeat.
[ https://issues.apache.org/jira/browse/YARN-692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Omkar Vinit Joshi updated YARN-692: --- Attachment: YARN-692-20130611.patch Creating NMToken master key on RM and sharing it with NM as a part of RM-NM heartbeat. -- Key: YARN-692 URL: https://issues.apache.org/jira/browse/YARN-692 Project: Hadoop YARN Issue Type: Bug Reporter: Omkar Vinit Joshi Assignee: Omkar Vinit Joshi Attachments: YARN-692.20130530.1.patch, YARN-692.20130530.2.patch, YARN-692.20130531.1.patch, YARN-692.20130531.3.patch, YARN-692.20130531.patch, YARN-692-20130611.patch This is related to YARN-613 . Here we will be implementing NMToken generation on RM side and sharing it with NM during RM-NM heartbeat. As a part of this JIRA mater key will only be made available to NM but there will be no validation done until AM-NM communication is fixed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-797) DecodeIdentifier is broken. It was using KIND field for reflection and now we don't have class named as KIND.
[ https://issues.apache.org/jira/browse/YARN-797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Omkar Vinit Joshi updated YARN-797: --- Summary: DecodeIdentifier is broken. It was using KIND field for reflection and now we don't have class named as KIND. (was: Remove KIND field from ContainerTokenIdentifier as it is not useful.) DecodeIdentifier is broken. It was using KIND field for reflection and now we don't have class named as KIND. - Key: YARN-797 URL: https://issues.apache.org/jira/browse/YARN-797 Project: Hadoop YARN Issue Type: Bug Reporter: Omkar Vinit Joshi As we already have removed ContainerToken, ClientToken etc. classes there is no point in keeping KIND field. This was used while decodingIdentifier. (Reflection based on KIND). probably either we should remove or update this code as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-797) DecodeIdentifier is broken. It was using KIND field for reflection and now we don't have class named as KIND.
[ https://issues.apache.org/jira/browse/YARN-797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Omkar Vinit Joshi updated YARN-797: --- Description: We need to fix the reflection code in Token.decodeIdentifier was: We need to fix the reflection code in Token.decodeIdentifier. DecodeIdentifier is broken. It was using KIND field for reflection and now we don't have class named as KIND. - Key: YARN-797 URL: https://issues.apache.org/jira/browse/YARN-797 Project: Hadoop YARN Issue Type: Bug Reporter: Omkar Vinit Joshi We need to fix the reflection code in Token.decodeIdentifier -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-797) DecodeIdentifier is broken. It was using KIND field for reflection and now we don't have class named as KIND.
[ https://issues.apache.org/jira/browse/YARN-797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Omkar Vinit Joshi updated YARN-797: --- Description: We need to fix the reflection code in Token.decodeIdentifier. was: As we already have removed ContainerToken, ClientToken etc. classes there is no point in keeping KIND field. This was used while decodingIdentifier. (Reflection based on KIND). probably either we should remove or update this code as well. DecodeIdentifier is broken. It was using KIND field for reflection and now we don't have class named as KIND. - Key: YARN-797 URL: https://issues.apache.org/jira/browse/YARN-797 Project: Hadoop YARN Issue Type: Bug Reporter: Omkar Vinit Joshi We need to fix the reflection code in Token.decodeIdentifier. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-693) Sending NMToken to AM on allocate call
[ https://issues.apache.org/jira/browse/YARN-693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Omkar Vinit Joshi updated YARN-693: --- Description: This is part of YARN-613. As per the updated design, AM will receive per NM, NMToken in following scenarios * AM is receiving first container on underlying NM. * AM is receiving container on underlying NM after either NM or RM rebooted. ** After RM reboot, as RM doesn't remember (persist) the information about keys issued per AM per NM, it will reissue tokens in case AM gets new container on underlying NM. However on NM side NM will still retain older token until it receives new token to support long running jobs (in work preserving environment). ** After NM reboot, RM will delete the token information corresponding to that AM for all AMs. * AM is receiving container on underlying NM after NMToken master key is rolled over on RM side. In all the cases if AM receives new NMToken then it is suppose to store it for future NM communication until it receives a new one. was: This is part of YARN-613. As per the updated design, AM will receive per NM, NMToken in following scenarios * AM is receiving first container on underlying NM. * AM is receiving container on underlying NM after either NM or RM rebooted. ** After RM reboot, as RM doesn't remember (persist) the information about keys issued per AM per NM, it will reissue tokens in case AM gets new container on underlying NM. However on NM side NM will still retain older token until it receives new token to support long running jobs (in work preserving environment). ** After NM reboot, RM will delete the token information corresponding to all AMs. * AM is receiving container on underlying NM after NMToken master key is rolled over on RM side. In all the cases if AM receives new NMToken then it is suppose to store it for future NM communication until it receives a new one. Sending NMToken to AM on allocate call -- Key: YARN-693 URL: https://issues.apache.org/jira/browse/YARN-693 Project: Hadoop YARN Issue Type: Bug Reporter: Omkar Vinit Joshi Assignee: Omkar Vinit Joshi This is part of YARN-613. As per the updated design, AM will receive per NM, NMToken in following scenarios * AM is receiving first container on underlying NM. * AM is receiving container on underlying NM after either NM or RM rebooted. ** After RM reboot, as RM doesn't remember (persist) the information about keys issued per AM per NM, it will reissue tokens in case AM gets new container on underlying NM. However on NM side NM will still retain older token until it receives new token to support long running jobs (in work preserving environment). ** After NM reboot, RM will delete the token information corresponding to that AM for all AMs. * AM is receiving container on underlying NM after NMToken master key is rolled over on RM side. In all the cases if AM receives new NMToken then it is suppose to store it for future NM communication until it receives a new one. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-693) Sending NMToken to AM on allocate call
[ https://issues.apache.org/jira/browse/YARN-693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Omkar Vinit Joshi updated YARN-693: --- Attachment: YARN-693-20130610.patch Sending NMToken to AM on allocate call -- Key: YARN-693 URL: https://issues.apache.org/jira/browse/YARN-693 Project: Hadoop YARN Issue Type: Bug Reporter: Omkar Vinit Joshi Assignee: Omkar Vinit Joshi Attachments: YARN-693-20130610.patch This is part of YARN-613. As per the updated design, AM will receive per NM, NMToken in following scenarios * AM is receiving first container on underlying NM. * AM is receiving container on underlying NM after either NM or RM rebooted. ** After RM reboot, as RM doesn't remember (persist) the information about keys issued per AM per NM, it will reissue tokens in case AM gets new container on underlying NM. However on NM side NM will still retain older token until it receives new token to support long running jobs (in work preserving environment). ** After NM reboot, RM will delete the token information corresponding to that AM for all AMs. * AM is receiving container on underlying NM after NMToken master key is rolled over on RM side. In all the cases if AM receives new NMToken then it is suppose to store it for future NM communication until it receives a new one. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-693) Sending NMToken to AM on allocate call
[ https://issues.apache.org/jira/browse/YARN-693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Omkar Vinit Joshi updated YARN-693: --- Description: This is part of YARN-613. As per the updated design, AM will receive per NM, NMToken in following scenarios * AM is receiving first container on underlying NM. * AM is receiving container on underlying NM after either NM or RM rebooted. ** After RM reboot, as RM doesn't remember (persist) the information about keys issued per AM per NM, it will reissue tokens in case AM gets new container on underlying NM. However on NM side NM will still retain older token until it receives new token to support long running jobs (in work preserving environment). ** After NM reboot, RM will delete the token information corresponding to that AM for all AMs. * AM is receiving container on underlying NM after NMToken master key is rolled over on RM side. In all the cases if AM receives new NMToken then it is suppose to store it for future NM communication until it receives a new one. AMRMClient should expose these NMToken to client. was: This is part of YARN-613. As per the updated design, AM will receive per NM, NMToken in following scenarios * AM is receiving first container on underlying NM. * AM is receiving container on underlying NM after either NM or RM rebooted. ** After RM reboot, as RM doesn't remember (persist) the information about keys issued per AM per NM, it will reissue tokens in case AM gets new container on underlying NM. However on NM side NM will still retain older token until it receives new token to support long running jobs (in work preserving environment). ** After NM reboot, RM will delete the token information corresponding to that AM for all AMs. * AM is receiving container on underlying NM after NMToken master key is rolled over on RM side. In all the cases if AM receives new NMToken then it is suppose to store it for future NM communication until it receives a new one. Sending NMToken to AM on allocate call -- Key: YARN-693 URL: https://issues.apache.org/jira/browse/YARN-693 Project: Hadoop YARN Issue Type: Bug Reporter: Omkar Vinit Joshi Assignee: Omkar Vinit Joshi Attachments: YARN-693-20130610.patch This is part of YARN-613. As per the updated design, AM will receive per NM, NMToken in following scenarios * AM is receiving first container on underlying NM. * AM is receiving container on underlying NM after either NM or RM rebooted. ** After RM reboot, as RM doesn't remember (persist) the information about keys issued per AM per NM, it will reissue tokens in case AM gets new container on underlying NM. However on NM side NM will still retain older token until it receives new token to support long running jobs (in work preserving environment). ** After NM reboot, RM will delete the token information corresponding to that AM for all AMs. * AM is receiving container on underlying NM after NMToken master key is rolled over on RM side. In all the cases if AM receives new NMToken then it is suppose to store it for future NM communication until it receives a new one. AMRMClient should expose these NMToken to client. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-692) Creating NMToken master key on RM and sharing it with NM as a part of RM-NM heartbeat.
[ https://issues.apache.org/jira/browse/YARN-692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13682458#comment-13682458 ] Omkar Vinit Joshi commented on YARN-692: rebasing after YARN-117 Creating NMToken master key on RM and sharing it with NM as a part of RM-NM heartbeat. -- Key: YARN-692 URL: https://issues.apache.org/jira/browse/YARN-692 Project: Hadoop YARN Issue Type: Bug Reporter: Omkar Vinit Joshi Assignee: Omkar Vinit Joshi Attachments: YARN-692.20130530.1.patch, YARN-692.20130530.2.patch, YARN-692.20130531.1.patch, YARN-692.20130531.3.patch, YARN-692.20130531.patch, YARN-692-20130611.patch This is related to YARN-613 . Here we will be implementing NMToken generation on RM side and sharing it with NM during RM-NM heartbeat. As a part of this JIRA mater key will only be made available to NM but there will be no validation done until AM-NM communication is fixed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-692) Creating NMToken master key on RM and sharing it with NM as a part of RM-NM heartbeat.
[ https://issues.apache.org/jira/browse/YARN-692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Omkar Vinit Joshi updated YARN-692: --- Attachment: YARN-692-20130613.patch Creating NMToken master key on RM and sharing it with NM as a part of RM-NM heartbeat. -- Key: YARN-692 URL: https://issues.apache.org/jira/browse/YARN-692 Project: Hadoop YARN Issue Type: Bug Reporter: Omkar Vinit Joshi Assignee: Omkar Vinit Joshi Attachments: YARN-692.20130530.1.patch, YARN-692.20130530.2.patch, YARN-692.20130531.1.patch, YARN-692.20130531.3.patch, YARN-692.20130531.patch, YARN-692-20130611.patch, YARN-692-20130613.patch This is related to YARN-613 . Here we will be implementing NMToken generation on RM side and sharing it with NM during RM-NM heartbeat. As a part of this JIRA mater key will only be made available to NM but there will be no validation done until AM-NM communication is fixed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-693) Sending NMToken to AM on allocate call
[ https://issues.apache.org/jira/browse/YARN-693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Omkar Vinit Joshi updated YARN-693: --- Attachment: YARN-693-20130613.patch rebasing patch after YARN-117 Sending NMToken to AM on allocate call -- Key: YARN-693 URL: https://issues.apache.org/jira/browse/YARN-693 Project: Hadoop YARN Issue Type: Bug Reporter: Omkar Vinit Joshi Assignee: Omkar Vinit Joshi Attachments: YARN-693-20130610.patch, YARN-693-20130613.patch This is part of YARN-613. As per the updated design, AM will receive per NM, NMToken in following scenarios * AM is receiving first container on underlying NM. * AM is receiving container on underlying NM after either NM or RM rebooted. ** After RM reboot, as RM doesn't remember (persist) the information about keys issued per AM per NM, it will reissue tokens in case AM gets new container on underlying NM. However on NM side NM will still retain older token until it receives new token to support long running jobs (in work preserving environment). ** After NM reboot, RM will delete the token information corresponding to that AM for all AMs. * AM is receiving container on underlying NM after NMToken master key is rolled over on RM side. In all the cases if AM receives new NMToken then it is suppose to store it for future NM communication until it receives a new one. AMRMClient should expose these NMToken to client. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-694) AM uses the NMToken to authenticate all communication with NM. NM remembers and updates token across RM restart
[ https://issues.apache.org/jira/browse/YARN-694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13682793#comment-13682793 ] Omkar Vinit Joshi commented on YARN-694: Addressing below issues in this patch. * NMTokenSecretManagerInNM remembers 2 things ** NMToken will not be updated in the oldKeyMap as a part of RPC - retrievePassword call. It will be updated only if *** it is a start container call using the proxy/connection/NMToken *** and NMToken used for connection/proxy is using current master key Id or previous master key id. (This is to enforce the client to use latest NMToken if received on startContainer call). * All requests will be authenticated using ** For start Container it will be either the current master key or previous master key. ** For stop / get container status it will be authenticated either using current/ previous / saved (oldKeysMap) key. * All requests will be authorized but authorization will be different for different type of requests. ** For start container request. *** First check will be made to make sure that NMToken key id is using current or previous key id. *** Container Token passed in is correct. (retrievedPassword will be verified.) *** ugi.getUserName is correct *** RM Identifier is correct *** ContainerToken is not expired. *** ContainerToken passed in is for correct NodeManager, application attempt(based on NMToken). ** For stop / get container request. *** containerId passed in is correct w.r.t. the NMToken used for authentication. This is to avoid checking status of / stopping containers belonging to different application attempt. *** If container could not be retrieved used will be notified accordingly. ** All application attempts started for application. This is required when we need to clean up app attempt NMToken keys. Otherwise we will have to scan the whole map (oldKeysMap). * Add ContainerManagerProxy to cache connections/proxy per node manager. There will be at max one connection per node manager. Number of proxies can be configured using YarnConfiguration.NM_CLIENT_MAX_NM_PROXY_CONNECTIONS * Api added in NMClient to set NMToken map which client can retrieve from AMRMClient#getNMTokenMap. * NMClientImpl updated to use NMToken and ContainerManagerProxy to communicate with NM Things remaining. * Update TestContainerManagerSecurity to test NMToken changes. I will update this soon. AM uses the NMToken to authenticate all communication with NM. NM remembers and updates token across RM restart --- Key: YARN-694 URL: https://issues.apache.org/jira/browse/YARN-694 Project: Hadoop YARN Issue Type: Bug Reporter: Omkar Vinit Joshi Assignee: Omkar Vinit Joshi AM uses the NMToken to authenticate all the AM-NM communication. NM will validate NMToken in below manner * If NMToken is using current or previous master key then the NMToken is valid. In this case it will update its cache with this key corresponding to appId. * If NMToken is using the master key which is present in NM's cache corresponding to AM's appId then it will be validated based on this. * If NMToken is invalid then NM will reject AM calls. Modification for ContainerToken * At present RPC validates AM-NM communication based on ContainerToken. It will be replaced with NMToken. Also now onwards AM will use NMToken per NM (replacing earlier behavior of ContainerToken per container per NM). * startContainer in case of Secured environment is using ContainerToken from UGI YARN-617; however after this it will use it from the payload (Container). * ContainerToken will exist and it will only be used to validate the AM's container start request. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-694) AM uses the NMToken to authenticate all communication with NM. NM remembers and updates token across RM restart
[ https://issues.apache.org/jira/browse/YARN-694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Omkar Vinit Joshi updated YARN-694: --- Attachment: YARN-694-20130613.patch AM uses the NMToken to authenticate all communication with NM. NM remembers and updates token across RM restart --- Key: YARN-694 URL: https://issues.apache.org/jira/browse/YARN-694 Project: Hadoop YARN Issue Type: Bug Reporter: Omkar Vinit Joshi Assignee: Omkar Vinit Joshi Attachments: YARN-694-20130613.patch AM uses the NMToken to authenticate all the AM-NM communication. NM will validate NMToken in below manner * If NMToken is using current or previous master key then the NMToken is valid. In this case it will update its cache with this key corresponding to appId. * If NMToken is using the master key which is present in NM's cache corresponding to AM's appId then it will be validated based on this. * If NMToken is invalid then NM will reject AM calls. Modification for ContainerToken * At present RPC validates AM-NM communication based on ContainerToken. It will be replaced with NMToken. Also now onwards AM will use NMToken per NM (replacing earlier behavior of ContainerToken per container per NM). * startContainer in case of Secured environment is using ContainerToken from UGI YARN-617; however after this it will use it from the payload (Container). * ContainerToken will exist and it will only be used to validate the AM's container start request. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-692) Creating NMToken master key on RM and sharing it with NM as a part of RM-NM heartbeat.
[ https://issues.apache.org/jira/browse/YARN-692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Omkar Vinit Joshi updated YARN-692: --- Attachment: YARN-692-20130613.1.patch reducing timeout to 180K. Creating NMToken master key on RM and sharing it with NM as a part of RM-NM heartbeat. -- Key: YARN-692 URL: https://issues.apache.org/jira/browse/YARN-692 Project: Hadoop YARN Issue Type: Bug Reporter: Omkar Vinit Joshi Assignee: Omkar Vinit Joshi Attachments: YARN-692.20130530.1.patch, YARN-692.20130530.2.patch, YARN-692.20130531.1.patch, YARN-692.20130531.3.patch, YARN-692.20130531.patch, YARN-692-20130611.patch, YARN-692-20130613.1.patch, YARN-692-20130613.patch This is related to YARN-613 . Here we will be implementing NMToken generation on RM side and sharing it with NM during RM-NM heartbeat. As a part of this JIRA mater key will only be made available to NM but there will be no validation done until AM-NM communication is fixed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-639) Make AM of Distributed Shell Use NMClient
[
https://issues.apache.org/jira/browse/YARN-639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13682914#comment-13682914
]
Omkar Vinit Joshi commented on YARN-639:
[~zjshen] looked at your patch... minor comments
bq. +nodeManager.stop();
* in AM, can you rename nodemanager to something like nmClient? It was
confusing when I saw nodemanager.stop() :).
{code}
+@Override
+public void onContainerStarted(ContainerId containerId,
+MapString, ByteBuffer allServiceResponse) {
+ LOG.info(Succeeded to start Container + containerId);
+ Container container = containers.get(containerId);
+ if (container != null) {
+nodeManager.getContainerStatus(containerId, container.getNodeId(),
+container.getContainerToken());
+ }
+}
{code}
* why do we need to do getContainerStatus after successfully starting it? is it
required?
* also can you change LOG.info in CallBackHandler to LOG.debug / LOG.error as
appropriate?
everything else looks good..
Make AM of Distributed Shell Use NMClient
-
Key: YARN-639
URL: https://issues.apache.org/jira/browse/YARN-639
Project: Hadoop YARN
Issue Type: Bug
Reporter: Zhijie Shen
Assignee: Zhijie Shen
Attachments: YARN-639.1.patch
YARN-422 adds NMClient. AM of Distributed Shell should use it instead of
using ContainerManager directly.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (YARN-610) ClientToken should not be set in the environment
[ https://issues.apache.org/jira/browse/YARN-610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Omkar Vinit Joshi reassigned YARN-610: -- Assignee: Omkar Vinit Joshi (was: Vinod Kumar Vavilapalli) ClientToken should not be set in the environment Key: YARN-610 URL: https://issues.apache.org/jira/browse/YARN-610 Project: Hadoop YARN Issue Type: Sub-task Reporter: Siddharth Seth Assignee: Omkar Vinit Joshi Similar to YARN-579, this can be set via ContainerTokens -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-610) ClientToken should not be set in the environment
[ https://issues.apache.org/jira/browse/YARN-610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13683611#comment-13683611 ] Omkar Vinit Joshi commented on YARN-610: Here we will be making sure that AM has the secret key for authenticating client requests(tokens). Now problem is that we can't share it in environment as this people might be able to read it (windows). Now instead of sharing it here in environment we should send it on AMRMProtocol as a part of AMRegistration. Thereby AM will get secret key after am registration and will be able to authenticate thereafter. any thoughts? ClientToken should not be set in the environment Key: YARN-610 URL: https://issues.apache.org/jira/browse/YARN-610 Project: Hadoop YARN Issue Type: Sub-task Reporter: Siddharth Seth Assignee: Omkar Vinit Joshi Similar to YARN-579, this can be set via ContainerTokens -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-610) ClientToken should not be set in the environment
[ https://issues.apache.org/jira/browse/YARN-610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13683625#comment-13683625 ] Omkar Vinit Joshi commented on YARN-610: also we can't send it via credentials as a token because we are sharing secret key. Renaming name to ClientToAMToken in protocols too. ClientToken should not be set in the environment Key: YARN-610 URL: https://issues.apache.org/jira/browse/YARN-610 Project: Hadoop YARN Issue Type: Sub-task Reporter: Siddharth Seth Assignee: Omkar Vinit Joshi Similar to YARN-579, this can be set via ContainerTokens -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (YARN-822) Rename ApplicationToken to AMRMToken
Omkar Vinit Joshi created YARN-822: -- Summary: Rename ApplicationToken to AMRMToken Key: YARN-822 URL: https://issues.apache.org/jira/browse/YARN-822 Project: Hadoop YARN Issue Type: Bug Reporter: Omkar Vinit Joshi Assignee: Omkar Vinit Joshi API change. At present this token is getting used on scheduler api AMRMProtocol. Right now name wise it is little confusing as it might be useful for the application to talk to complete yarn system (RM/NM) but that is not the case after YARN-694. NM will have specific NMToken so it is better to name it as AMRMToken. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-610) ClientToken should not be set in the environment
[ https://issues.apache.org/jira/browse/YARN-610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Omkar Vinit Joshi updated YARN-610: --- Attachment: YARN-610-20130614.patch ClientToken should not be set in the environment Key: YARN-610 URL: https://issues.apache.org/jira/browse/YARN-610 Project: Hadoop YARN Issue Type: Sub-task Reporter: Siddharth Seth Assignee: Omkar Vinit Joshi Attachments: YARN-610-20130614.patch Similar to YARN-579, this can be set via ContainerTokens -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-693) Sending NMToken to AM on allocate call
[ https://issues.apache.org/jira/browse/YARN-693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13684041#comment-13684041 ] Omkar Vinit Joshi commented on YARN-693: bq. NMToken should be in org.apache.hadoop.yarn.api.records. bq. Similarly NMTokenPBIMpl in org.apache.hadoop.yarn.api.records.impl.pb. fixed. bq. AMRMClient signature can just have a ConcurrentMap instead of ConcurrentHashMap. fixed. bq. I suppose that the test to verify that the previously given out tokens before roll-over will work after roll-over is in YARN-694, right? yes there are more scenarios which will be tested in YARN-694. bq. TestAMRMClient: Also validate NMTokens = nodeCount ? fixed. bq. Static factory for NMToken? bq. And then use the above in NMTokenSecretManagerInRM. fixed. added static method in NMTokenIdentifier.newNMToken. bq. The MR changes can be moved to YARN-694 or its MR companion. I have commented out the code. will uncomment it after YARN-694 :) bq. The correct place for calling register and unregister with NMTokenSecretManagerInRM is RMAppAttemptImpl. I think ApplicationMasterService looks good as it is easier to follow. keeping it in ApplicationMasterService only. Sending NMToken to AM on allocate call -- Key: YARN-693 URL: https://issues.apache.org/jira/browse/YARN-693 Project: Hadoop YARN Issue Type: Bug Reporter: Omkar Vinit Joshi Assignee: Omkar Vinit Joshi Attachments: YARN-693-20130610.patch, YARN-693-20130613.patch This is part of YARN-613. As per the updated design, AM will receive per NM, NMToken in following scenarios * AM is receiving first container on underlying NM. * AM is receiving container on underlying NM after either NM or RM rebooted. ** After RM reboot, as RM doesn't remember (persist) the information about keys issued per AM per NM, it will reissue tokens in case AM gets new container on underlying NM. However on NM side NM will still retain older token until it receives new token to support long running jobs (in work preserving environment). ** After NM reboot, RM will delete the token information corresponding to that AM for all AMs. * AM is receiving container on underlying NM after NMToken master key is rolled over on RM side. In all the cases if AM receives new NMToken then it is suppose to store it for future NM communication until it receives a new one. AMRMClient should expose these NMToken to client. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-693) Sending NMToken to AM on allocate call
[ https://issues.apache.org/jira/browse/YARN-693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Omkar Vinit Joshi updated YARN-693: --- Attachment: YARN-693-20130614.1.patch Sending NMToken to AM on allocate call -- Key: YARN-693 URL: https://issues.apache.org/jira/browse/YARN-693 Project: Hadoop YARN Issue Type: Bug Reporter: Omkar Vinit Joshi Assignee: Omkar Vinit Joshi Attachments: YARN-693-20130610.patch, YARN-693-20130613.patch, YARN-693-20130614.1.patch This is part of YARN-613. As per the updated design, AM will receive per NM, NMToken in following scenarios * AM is receiving first container on underlying NM. * AM is receiving container on underlying NM after either NM or RM rebooted. ** After RM reboot, as RM doesn't remember (persist) the information about keys issued per AM per NM, it will reissue tokens in case AM gets new container on underlying NM. However on NM side NM will still retain older token until it receives new token to support long running jobs (in work preserving environment). ** After NM reboot, RM will delete the token information corresponding to that AM for all AMs. * AM is receiving container on underlying NM after NMToken master key is rolled over on RM side. In all the cases if AM receives new NMToken then it is suppose to store it for future NM communication until it receives a new one. AMRMClient should expose these NMToken to client. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-693) Sending NMToken to AM on allocate call
[ https://issues.apache.org/jira/browse/YARN-693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Omkar Vinit Joshi updated YARN-693: --- Attachment: YARN-693-20130615.patch Sending NMToken to AM on allocate call -- Key: YARN-693 URL: https://issues.apache.org/jira/browse/YARN-693 Project: Hadoop YARN Issue Type: Bug Reporter: Omkar Vinit Joshi Assignee: Omkar Vinit Joshi Attachments: YARN-693-20130610.patch, YARN-693-20130613.patch, YARN-693-20130614.1.patch, YARN-693-20130615.patch This is part of YARN-613. As per the updated design, AM will receive per NM, NMToken in following scenarios * AM is receiving first container on underlying NM. * AM is receiving container on underlying NM after either NM or RM rebooted. ** After RM reboot, as RM doesn't remember (persist) the information about keys issued per AM per NM, it will reissue tokens in case AM gets new container on underlying NM. However on NM side NM will still retain older token until it receives new token to support long running jobs (in work preserving environment). ** After NM reboot, RM will delete the token information corresponding to that AM for all AMs. * AM is receiving container on underlying NM after NMToken master key is rolled over on RM side. In all the cases if AM receives new NMToken then it is suppose to store it for future NM communication until it receives a new one. AMRMClient should expose these NMToken to client. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-693) Sending NMToken to AM on allocate call
[ https://issues.apache.org/jira/browse/YARN-693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13684532#comment-13684532 ] Omkar Vinit Joshi commented on YARN-693: Fixed TestCases and find bug warnings.. bq. I meant creating a newInstance method in NMToken.java. Once you do that, NMTokenSecretManagerInRM.getNMTokens can use that API. We shouldn't be directly using new NMTokenPBImpl(). Fixed it. bq. Also, NMTokenIdentifier.newNMToken() is better placed in NMTokenSecretManagerInRM as RM is where NMTokens are created. fixed Sending NMToken to AM on allocate call -- Key: YARN-693 URL: https://issues.apache.org/jira/browse/YARN-693 Project: Hadoop YARN Issue Type: Bug Reporter: Omkar Vinit Joshi Assignee: Omkar Vinit Joshi Attachments: YARN-693-20130610.patch, YARN-693-20130613.patch, YARN-693-20130614.1.patch, YARN-693-20130615.patch This is part of YARN-613. As per the updated design, AM will receive per NM, NMToken in following scenarios * AM is receiving first container on underlying NM. * AM is receiving container on underlying NM after either NM or RM rebooted. ** After RM reboot, as RM doesn't remember (persist) the information about keys issued per AM per NM, it will reissue tokens in case AM gets new container on underlying NM. However on NM side NM will still retain older token until it receives new token to support long running jobs (in work preserving environment). ** After NM reboot, RM will delete the token information corresponding to that AM for all AMs. * AM is receiving container on underlying NM after NMToken master key is rolled over on RM side. In all the cases if AM receives new NMToken then it is suppose to store it for future NM communication until it receives a new one. AMRMClient should expose these NMToken to client. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-610) ClientToken should not be set in the environment
[ https://issues.apache.org/jira/browse/YARN-610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13684533#comment-13684533 ] Omkar Vinit Joshi commented on YARN-610: patch submitted should be applied on top of YARN-693 which also involves AMRMProtocol changes. ClientToken should not be set in the environment Key: YARN-610 URL: https://issues.apache.org/jira/browse/YARN-610 Project: Hadoop YARN Issue Type: Sub-task Reporter: Siddharth Seth Assignee: Omkar Vinit Joshi Attachments: YARN-610-20130614.patch Similar to YARN-579, this can be set via ContainerTokens -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-35) Move to per-node RM-NM secrets
[ https://issues.apache.org/jira/browse/YARN-35?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13684537#comment-13684537 ] Omkar Vinit Joshi commented on YARN-35: --- * I think we should make it configurable if user wants he can enable it (like for unsecured environment it will unnecessarily increase RM memory). * This would be tricky when we are going to support work preserving mode. As how long we will key per node SecretManager (key) in RM? say for example if NM reconnects then it should be given same key or else nm will reject all the connections with older tokens. Move to per-node RM-NM secrets -- Key: YARN-35 URL: https://issues.apache.org/jira/browse/YARN-35 Project: Hadoop YARN Issue Type: Sub-task Reporter: Vinod Kumar Vavilapalli We should move over to per node secret (RM-NM shared secrets) for security sake. It was what I had in my mind while designing the whole security architecture, but somehow it got lost in all the storm of security patches. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (YARN-835) The name 'ApplicationTokenIdentifier' is misleading
[ https://issues.apache.org/jira/browse/YARN-835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Omkar Vinit Joshi resolved YARN-835. Resolution: Duplicate The name 'ApplicationTokenIdentifier' is misleading --- Key: YARN-835 URL: https://issues.apache.org/jira/browse/YARN-835 Project: Hadoop YARN Issue Type: Sub-task Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli As others have noted in multiple places, it is misleading. Should rename it to AMRMTokenIdentifier or something similar. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-822) Rename ApplicationToken to AMRMToken
[ https://issues.apache.org/jira/browse/YARN-822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Omkar Vinit Joshi updated YARN-822: --- Issue Type: Sub-task (was: Bug) Parent: YARN-386 Rename ApplicationToken to AMRMToken Key: YARN-822 URL: https://issues.apache.org/jira/browse/YARN-822 Project: Hadoop YARN Issue Type: Sub-task Reporter: Omkar Vinit Joshi Assignee: Omkar Vinit Joshi API change. At present this token is getting used on scheduler api AMRMProtocol. Right now name wise it is little confusing as it might be useful for the application to talk to complete yarn system (RM/NM) but that is not the case after YARN-694. NM will have specific NMToken so it is better to name it as AMRMToken. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-610) ClientToken should not be set in the environment
[ https://issues.apache.org/jira/browse/YARN-610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13685005#comment-13685005 ] Omkar Vinit Joshi commented on YARN-610: bq. Seems like the patch has gone stale, can you rebase? rebased.. bq. File a MR ticket too as a followup tracking MR changes. MAPREDUCE-5328 bq. TestClientTokens should also be renamed to TestClientToAMTokens renamed bq. CustomNM is no longer needed? its just implementing an interface..keeping it as it is. bq. There is commented out code that should be removed. fixed. ClientToken should not be set in the environment Key: YARN-610 URL: https://issues.apache.org/jira/browse/YARN-610 Project: Hadoop YARN Issue Type: Sub-task Reporter: Siddharth Seth Assignee: Omkar Vinit Joshi Attachments: YARN-610-20130614.patch Similar to YARN-579, this can be set via ContainerTokens -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-610) ClientToken should not be set in the environment
[ https://issues.apache.org/jira/browse/YARN-610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Omkar Vinit Joshi updated YARN-610: --- Attachment: YARN-610-20130616.patch ClientToken should not be set in the environment Key: YARN-610 URL: https://issues.apache.org/jira/browse/YARN-610 Project: Hadoop YARN Issue Type: Sub-task Reporter: Siddharth Seth Assignee: Omkar Vinit Joshi Attachments: YARN-610-20130614.patch, YARN-610-20130616.patch Similar to YARN-579, this can be set via ContainerTokens -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-610) ClientToken (ClientToAMToken) should not be set in the environment
[ https://issues.apache.org/jira/browse/YARN-610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Omkar Vinit Joshi updated YARN-610: --- Attachment: YARN-610-20130616.3.patch ClientToken (ClientToAMToken) should not be set in the environment -- Key: YARN-610 URL: https://issues.apache.org/jira/browse/YARN-610 Project: Hadoop YARN Issue Type: Sub-task Reporter: Siddharth Seth Assignee: Omkar Vinit Joshi Priority: Blocker Attachments: YARN-610-20130614.patch, YARN-610-20130616.3.patch, YARN-610-20130616.patch Similar to YARN-579, this can be set via ContainerTokens -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-822) Rename ApplicationToken to AMRMToken
[ https://issues.apache.org/jira/browse/YARN-822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Omkar Vinit Joshi updated YARN-822: --- Attachment: YARN-822-20130617.patch Renaming TokenIdentifier, TokenSelector and SecretManager. Did grep on complete source code. Rename ApplicationToken to AMRMToken Key: YARN-822 URL: https://issues.apache.org/jira/browse/YARN-822 Project: Hadoop YARN Issue Type: Sub-task Reporter: Omkar Vinit Joshi Assignee: Omkar Vinit Joshi Attachments: YARN-822-20130617.patch API change. At present this token is getting used on scheduler api AMRMProtocol. Right now name wise it is little confusing as it might be useful for the application to talk to complete yarn system (RM/NM) but that is not the case after YARN-694. NM will have specific NMToken so it is better to name it as AMRMToken. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-610) ClientToken (ClientToAMToken) should not be set in the environment
[ https://issues.apache.org/jira/browse/YARN-610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13685801#comment-13685801 ] Omkar Vinit Joshi commented on YARN-610: updating the patch .. ClientToken (ClientToAMToken) should not be set in the environment -- Key: YARN-610 URL: https://issues.apache.org/jira/browse/YARN-610 Project: Hadoop YARN Issue Type: Sub-task Reporter: Siddharth Seth Assignee: Omkar Vinit Joshi Priority: Blocker Attachments: YARN-610-20130614.patch, YARN-610-20130616.3.patch, YARN-610-20130616.patch Similar to YARN-579, this can be set via ContainerTokens -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (YARN-843) TestPipeApplication should not be using AMRMToken.
Omkar Vinit Joshi created YARN-843: -- Summary: TestPipeApplication should not be using AMRMToken. Key: YARN-843 URL: https://issues.apache.org/jira/browse/YARN-843 Project: Hadoop YARN Issue Type: Bug Reporter: Omkar Vinit Joshi [YARN-822 comment | https://issues.apache.org/jira/browse/YARN-822?focusedCommentId=13685802page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13685802] May be we can just remove the token usage. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-822) Rename ApplicationToken to AMRMToken
[ https://issues.apache.org/jira/browse/YARN-822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13685806#comment-13685806 ] Omkar Vinit Joshi commented on YARN-822: yes I will... YARN-843 Rename ApplicationToken to AMRMToken Key: YARN-822 URL: https://issues.apache.org/jira/browse/YARN-822 Project: Hadoop YARN Issue Type: Sub-task Reporter: Omkar Vinit Joshi Assignee: Omkar Vinit Joshi Attachments: YARN-822-20130617.patch API change. At present this token is getting used on scheduler api AMRMProtocol. Right now name wise it is little confusing as it might be useful for the application to talk to complete yarn system (RM/NM) but that is not the case after YARN-694. NM will have specific NMToken so it is better to name it as AMRMToken. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-610) ClientToken (ClientToAMToken) should not be set in the environment
[ https://issues.apache.org/jira/browse/YARN-610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Omkar Vinit Joshi updated YARN-610: --- Attachment: YARN-610-20130617.patch ClientToken (ClientToAMToken) should not be set in the environment -- Key: YARN-610 URL: https://issues.apache.org/jira/browse/YARN-610 Project: Hadoop YARN Issue Type: Sub-task Reporter: Siddharth Seth Assignee: Omkar Vinit Joshi Priority: Blocker Attachments: YARN-610-20130614.patch, YARN-610-20130616.3.patch, YARN-610-20130616.patch, YARN-610-20130617.patch Similar to YARN-579, this can be set via ContainerTokens -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-694) Start using NMTokens to authenticate all communication with NM
[ https://issues.apache.org/jira/browse/YARN-694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Omkar Vinit Joshi updated YARN-694: --- Attachment: YARN-694-20130617.patch Start using NMTokens to authenticate all communication with NM -- Key: YARN-694 URL: https://issues.apache.org/jira/browse/YARN-694 Project: Hadoop YARN Issue Type: Bug Reporter: Omkar Vinit Joshi Assignee: Omkar Vinit Joshi Attachments: YARN-694-20130613.patch, YARN-694-20130617.patch AM uses the NMToken to authenticate all the AM-NM communication. NM will validate NMToken in below manner * If NMToken is using current or previous master key then the NMToken is valid. In this case it will update its cache with this key corresponding to appId. * If NMToken is using the master key which is present in NM's cache corresponding to AM's appId then it will be validated based on this. * If NMToken is invalid then NM will reject AM calls. Modification for ContainerToken * At present RPC validates AM-NM communication based on ContainerToken. It will be replaced with NMToken. Also now onwards AM will use NMToken per NM (replacing earlier behavior of ContainerToken per container per NM). * startContainer in case of Secured environment is using ContainerToken from UGI YARN-617; however after this it will use it from the payload (Container). * ContainerToken will exist and it will only be used to validate the AM's container start request. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-694) Start using NMTokens to authenticate all communication with NM
[ https://issues.apache.org/jira/browse/YARN-694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13685913#comment-13685913 ] Omkar Vinit Joshi commented on YARN-694: Things pending.. * Fix TestContainerManagerSecurity ... uncomment tests * Distributed shell pass tokens from AMRMClient to NMClient * Add test in TestContainerManagerSecurity to test key roll over. Start using NMTokens to authenticate all communication with NM -- Key: YARN-694 URL: https://issues.apache.org/jira/browse/YARN-694 Project: Hadoop YARN Issue Type: Bug Reporter: Omkar Vinit Joshi Assignee: Omkar Vinit Joshi Attachments: YARN-694-20130613.patch, YARN-694-20130617.patch AM uses the NMToken to authenticate all the AM-NM communication. NM will validate NMToken in below manner * If NMToken is using current or previous master key then the NMToken is valid. In this case it will update its cache with this key corresponding to appId. * If NMToken is using the master key which is present in NM's cache corresponding to AM's appId then it will be validated based on this. * If NMToken is invalid then NM will reject AM calls. Modification for ContainerToken * At present RPC validates AM-NM communication based on ContainerToken. It will be replaced with NMToken. Also now onwards AM will use NMToken per NM (replacing earlier behavior of ContainerToken per container per NM). * startContainer in case of Secured environment is using ContainerToken from UGI YARN-617; however after this it will use it from the payload (Container). * ContainerToken will exist and it will only be used to validate the AM's container start request. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-610) ClientToken (ClientToAMToken) should not be set in the environment
[ https://issues.apache.org/jira/browse/YARN-610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Omkar Vinit Joshi updated YARN-610: --- Attachment: YARN-610-20130617.1.patch ClientToken (ClientToAMToken) should not be set in the environment -- Key: YARN-610 URL: https://issues.apache.org/jira/browse/YARN-610 Project: Hadoop YARN Issue Type: Sub-task Reporter: Siddharth Seth Assignee: Omkar Vinit Joshi Priority: Blocker Attachments: YARN-610-20130614.patch, YARN-610-20130616.3.patch, YARN-610-20130616.patch, YARN-610-20130617.1.patch, YARN-610-20130617.patch Similar to YARN-579, this can be set via ContainerTokens -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-610) ClientToken (ClientToAMToken) should not be set in the environment
[ https://issues.apache.org/jira/browse/YARN-610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Omkar Vinit Joshi updated YARN-610: --- Attachment: YARN-610-20130617.2.patch ClientToken (ClientToAMToken) should not be set in the environment -- Key: YARN-610 URL: https://issues.apache.org/jira/browse/YARN-610 Project: Hadoop YARN Issue Type: Sub-task Reporter: Siddharth Seth Assignee: Omkar Vinit Joshi Priority: Blocker Attachments: YARN-610-20130614.patch, YARN-610-20130616.3.patch, YARN-610-20130616.patch, YARN-610-20130617.1.patch, YARN-610-20130617.2.patch, YARN-610-20130617.patch Similar to YARN-579, this can be set via ContainerTokens -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-694) Start using NMTokens to authenticate all communication with NM
[ https://issues.apache.org/jira/browse/YARN-694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Omkar Vinit Joshi updated YARN-694: --- Attachment: YARN-694-20130617.1.patch Start using NMTokens to authenticate all communication with NM -- Key: YARN-694 URL: https://issues.apache.org/jira/browse/YARN-694 Project: Hadoop YARN Issue Type: Bug Reporter: Omkar Vinit Joshi Assignee: Omkar Vinit Joshi Attachments: YARN-694-20130613.patch, YARN-694-20130617.1.patch, YARN-694-20130617.patch AM uses the NMToken to authenticate all the AM-NM communication. NM will validate NMToken in below manner * If NMToken is using current or previous master key then the NMToken is valid. In this case it will update its cache with this key corresponding to appId. * If NMToken is using the master key which is present in NM's cache corresponding to AM's appId then it will be validated based on this. * If NMToken is invalid then NM will reject AM calls. Modification for ContainerToken * At present RPC validates AM-NM communication based on ContainerToken. It will be replaced with NMToken. Also now onwards AM will use NMToken per NM (replacing earlier behavior of ContainerToken per container per NM). * startContainer in case of Secured environment is using ContainerToken from UGI YARN-617; however after this it will use it from the payload (Container). * ContainerToken will exist and it will only be used to validate the AM's container start request. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-694) Start using NMTokens to authenticate all communication with NM
[ https://issues.apache.org/jira/browse/YARN-694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Omkar Vinit Joshi updated YARN-694: --- Attachment: YARN-694-20130617.2.patch updating patch after YARN-610 Start using NMTokens to authenticate all communication with NM -- Key: YARN-694 URL: https://issues.apache.org/jira/browse/YARN-694 Project: Hadoop YARN Issue Type: Bug Reporter: Omkar Vinit Joshi Assignee: Omkar Vinit Joshi Attachments: YARN-694-20130613.patch, YARN-694-20130617.1.patch, YARN-694-20130617.2.patch, YARN-694-20130617.patch AM uses the NMToken to authenticate all the AM-NM communication. NM will validate NMToken in below manner * If NMToken is using current or previous master key then the NMToken is valid. In this case it will update its cache with this key corresponding to appId. * If NMToken is using the master key which is present in NM's cache corresponding to AM's appId then it will be validated based on this. * If NMToken is invalid then NM will reject AM calls. Modification for ContainerToken * At present RPC validates AM-NM communication based on ContainerToken. It will be replaced with NMToken. Also now onwards AM will use NMToken per NM (replacing earlier behavior of ContainerToken per container per NM). * startContainer in case of Secured environment is using ContainerToken from UGI YARN-617; however after this it will use it from the payload (Container). * ContainerToken will exist and it will only be used to validate the AM's container start request. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-694) Start using NMTokens to authenticate all communication with NM
[ https://issues.apache.org/jira/browse/YARN-694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Omkar Vinit Joshi updated YARN-694: --- Attachment: YARN-694-20130618.patch Start using NMTokens to authenticate all communication with NM -- Key: YARN-694 URL: https://issues.apache.org/jira/browse/YARN-694 Project: Hadoop YARN Issue Type: Bug Reporter: Omkar Vinit Joshi Assignee: Omkar Vinit Joshi Attachments: YARN-694-20130613.patch, YARN-694-20130617.1.patch, YARN-694-20130617.2.patch, YARN-694-20130617.patch, YARN-694-20130618.patch AM uses the NMToken to authenticate all the AM-NM communication. NM will validate NMToken in below manner * If NMToken is using current or previous master key then the NMToken is valid. In this case it will update its cache with this key corresponding to appId. * If NMToken is using the master key which is present in NM's cache corresponding to AM's appId then it will be validated based on this. * If NMToken is invalid then NM will reject AM calls. Modification for ContainerToken * At present RPC validates AM-NM communication based on ContainerToken. It will be replaced with NMToken. Also now onwards AM will use NMToken per NM (replacing earlier behavior of ContainerToken per container per NM). * startContainer in case of Secured environment is using ContainerToken from UGI YARN-617; however after this it will use it from the payload (Container). * ContainerToken will exist and it will only be used to validate the AM's container start request. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-694) Start using NMTokens to authenticate all communication with NM
[ https://issues.apache.org/jira/browse/YARN-694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Omkar Vinit Joshi updated YARN-694: --- Attachment: YARN-694-20130618.1.patch Start using NMTokens to authenticate all communication with NM -- Key: YARN-694 URL: https://issues.apache.org/jira/browse/YARN-694 Project: Hadoop YARN Issue Type: Bug Reporter: Omkar Vinit Joshi Assignee: Omkar Vinit Joshi Attachments: YARN-694-20130613.patch, YARN-694-20130617.1.patch, YARN-694-20130617.2.patch, YARN-694-20130617.patch, YARN-694-20130618.1.patch AM uses the NMToken to authenticate all the AM-NM communication. NM will validate NMToken in below manner * If NMToken is using current or previous master key then the NMToken is valid. In this case it will update its cache with this key corresponding to appId. * If NMToken is using the master key which is present in NM's cache corresponding to AM's appId then it will be validated based on this. * If NMToken is invalid then NM will reject AM calls. Modification for ContainerToken * At present RPC validates AM-NM communication based on ContainerToken. It will be replaced with NMToken. Also now onwards AM will use NMToken per NM (replacing earlier behavior of ContainerToken per container per NM). * startContainer in case of Secured environment is using ContainerToken from UGI YARN-617; however after this it will use it from the payload (Container). * ContainerToken will exist and it will only be used to validate the AM's container start request. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-694) Start using NMTokens to authenticate all communication with NM
[ https://issues.apache.org/jira/browse/YARN-694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Omkar Vinit Joshi updated YARN-694: --- Attachment: (was: YARN-694-20130618.patch) Start using NMTokens to authenticate all communication with NM -- Key: YARN-694 URL: https://issues.apache.org/jira/browse/YARN-694 Project: Hadoop YARN Issue Type: Bug Reporter: Omkar Vinit Joshi Assignee: Omkar Vinit Joshi Attachments: YARN-694-20130613.patch, YARN-694-20130617.1.patch, YARN-694-20130617.2.patch, YARN-694-20130617.patch, YARN-694-20130618.1.patch AM uses the NMToken to authenticate all the AM-NM communication. NM will validate NMToken in below manner * If NMToken is using current or previous master key then the NMToken is valid. In this case it will update its cache with this key corresponding to appId. * If NMToken is using the master key which is present in NM's cache corresponding to AM's appId then it will be validated based on this. * If NMToken is invalid then NM will reject AM calls. Modification for ContainerToken * At present RPC validates AM-NM communication based on ContainerToken. It will be replaced with NMToken. Also now onwards AM will use NMToken per NM (replacing earlier behavior of ContainerToken per container per NM). * startContainer in case of Secured environment is using ContainerToken from UGI YARN-617; however after this it will use it from the payload (Container). * ContainerToken will exist and it will only be used to validate the AM's container start request. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-694) Start using NMTokens to authenticate all communication with NM
[ https://issues.apache.org/jira/browse/YARN-694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Omkar Vinit Joshi updated YARN-694: --- Attachment: YARN-694-20130618.2.patch Start using NMTokens to authenticate all communication with NM -- Key: YARN-694 URL: https://issues.apache.org/jira/browse/YARN-694 Project: Hadoop YARN Issue Type: Bug Reporter: Omkar Vinit Joshi Assignee: Omkar Vinit Joshi Attachments: YARN-694-20130613.patch, YARN-694-20130617.1.patch, YARN-694-20130617.2.patch, YARN-694-20130617.patch, YARN-694-20130618.1.patch, YARN-694-20130618.2.patch AM uses the NMToken to authenticate all the AM-NM communication. NM will validate NMToken in below manner * If NMToken is using current or previous master key then the NMToken is valid. In this case it will update its cache with this key corresponding to appId. * If NMToken is using the master key which is present in NM's cache corresponding to AM's appId then it will be validated based on this. * If NMToken is invalid then NM will reject AM calls. Modification for ContainerToken * At present RPC validates AM-NM communication based on ContainerToken. It will be replaced with NMToken. Also now onwards AM will use NMToken per NM (replacing earlier behavior of ContainerToken per container per NM). * startContainer in case of Secured environment is using ContainerToken from UGI YARN-617; however after this it will use it from the payload (Container). * ContainerToken will exist and it will only be used to validate the AM's container start request. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-694) Start using NMTokens to authenticate all communication with NM
[ https://issues.apache.org/jira/browse/YARN-694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Omkar Vinit Joshi updated YARN-694: --- Attachment: YARN-694-20130618.3.patch Start using NMTokens to authenticate all communication with NM -- Key: YARN-694 URL: https://issues.apache.org/jira/browse/YARN-694 Project: Hadoop YARN Issue Type: Bug Reporter: Omkar Vinit Joshi Assignee: Omkar Vinit Joshi Attachments: YARN-694-20130613.patch, YARN-694-20130617.1.patch, YARN-694-20130617.2.patch, YARN-694-20130617.patch, YARN-694-20130618.1.patch, YARN-694-20130618.2.patch, YARN-694-20130618.3.patch AM uses the NMToken to authenticate all the AM-NM communication. NM will validate NMToken in below manner * If NMToken is using current or previous master key then the NMToken is valid. In this case it will update its cache with this key corresponding to appId. * If NMToken is using the master key which is present in NM's cache corresponding to AM's appId then it will be validated based on this. * If NMToken is invalid then NM will reject AM calls. Modification for ContainerToken * At present RPC validates AM-NM communication based on ContainerToken. It will be replaced with NMToken. Also now onwards AM will use NMToken per NM (replacing earlier behavior of ContainerToken per container per NM). * startContainer in case of Secured environment is using ContainerToken from UGI YARN-617; however after this it will use it from the payload (Container). * ContainerToken will exist and it will only be used to validate the AM's container start request. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-694) Start using NMTokens to authenticate all communication with NM
[ https://issues.apache.org/jira/browse/YARN-694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Omkar Vinit Joshi updated YARN-694: --- Attachment: YARN-694-20130618.4.patch Start using NMTokens to authenticate all communication with NM -- Key: YARN-694 URL: https://issues.apache.org/jira/browse/YARN-694 Project: Hadoop YARN Issue Type: Bug Reporter: Omkar Vinit Joshi Assignee: Omkar Vinit Joshi Attachments: YARN-694-20130613.patch, YARN-694-20130617.1.patch, YARN-694-20130617.2.patch, YARN-694-20130617.patch, YARN-694-20130618.1.patch, YARN-694-20130618.2.patch, YARN-694-20130618.3.patch, YARN-694-20130618.4.patch AM uses the NMToken to authenticate all the AM-NM communication. NM will validate NMToken in below manner * If NMToken is using current or previous master key then the NMToken is valid. In this case it will update its cache with this key corresponding to appId. * If NMToken is using the master key which is present in NM's cache corresponding to AM's appId then it will be validated based on this. * If NMToken is invalid then NM will reject AM calls. Modification for ContainerToken * At present RPC validates AM-NM communication based on ContainerToken. It will be replaced with NMToken. Also now onwards AM will use NMToken per NM (replacing earlier behavior of ContainerToken per container per NM). * startContainer in case of Secured environment is using ContainerToken from UGI YARN-617; however after this it will use it from the payload (Container). * ContainerToken will exist and it will only be used to validate the AM's container start request. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-694) Start using NMTokens to authenticate all communication with NM
[ https://issues.apache.org/jira/browse/YARN-694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Omkar Vinit Joshi updated YARN-694: --- Attachment: YARN-694-20130618.5.patch Start using NMTokens to authenticate all communication with NM -- Key: YARN-694 URL: https://issues.apache.org/jira/browse/YARN-694 Project: Hadoop YARN Issue Type: Bug Reporter: Omkar Vinit Joshi Assignee: Omkar Vinit Joshi Attachments: YARN-694-20130613.patch, YARN-694-20130617.1.patch, YARN-694-20130617.2.patch, YARN-694-20130617.patch, YARN-694-20130618.1.patch, YARN-694-20130618.2.patch, YARN-694-20130618.3.patch, YARN-694-20130618.4.patch, YARN-694-20130618.5.patch AM uses the NMToken to authenticate all the AM-NM communication. NM will validate NMToken in below manner * If NMToken is using current or previous master key then the NMToken is valid. In this case it will update its cache with this key corresponding to appId. * If NMToken is using the master key which is present in NM's cache corresponding to AM's appId then it will be validated based on this. * If NMToken is invalid then NM will reject AM calls. Modification for ContainerToken * At present RPC validates AM-NM communication based on ContainerToken. It will be replaced with NMToken. Also now onwards AM will use NMToken per NM (replacing earlier behavior of ContainerToken per container per NM). * startContainer in case of Secured environment is using ContainerToken from UGI YARN-617; however after this it will use it from the payload (Container). * ContainerToken will exist and it will only be used to validate the AM's container start request. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (YARN-739) NM startContainer should validate the NodeId
[ https://issues.apache.org/jira/browse/YARN-739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Omkar Vinit Joshi resolved YARN-739. Resolution: Fixed Fix Version/s: 2.1.0-beta NM startContainer should validate the NodeId Key: YARN-739 URL: https://issues.apache.org/jira/browse/YARN-739 Project: Hadoop YARN Issue Type: Sub-task Reporter: Siddharth Seth Assignee: Omkar Vinit Joshi Fix For: 2.1.0-beta The NM validates certain fields from the ContainerToken on a startContainer call. It shoudl also validate the NodeId (which needs to be added to the ContianerToken). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-739) NM startContainer should validate the NodeId
[ https://issues.apache.org/jira/browse/YARN-739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13687272#comment-13687272 ] Omkar Vinit Joshi commented on YARN-739: YARN-694 fixes this. closing this NM startContainer should validate the NodeId Key: YARN-739 URL: https://issues.apache.org/jira/browse/YARN-739 Project: Hadoop YARN Issue Type: Sub-task Reporter: Siddharth Seth Assignee: Omkar Vinit Joshi The NM validates certain fields from the ContainerToken on a startContainer call. It shoudl also validate the NodeId (which needs to be added to the ContianerToken). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-739) NM startContainer should validate the NodeId
[ https://issues.apache.org/jira/browse/YARN-739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13687275#comment-13687275 ] Omkar Vinit Joshi commented on YARN-739: This same check is added for NMToken too. NM startContainer should validate the NodeId Key: YARN-739 URL: https://issues.apache.org/jira/browse/YARN-739 Project: Hadoop YARN Issue Type: Sub-task Reporter: Siddharth Seth Assignee: Omkar Vinit Joshi Fix For: 2.1.0-beta The NM validates certain fields from the ContainerToken on a startContainer call. It shoudl also validate the NodeId (which needs to be added to the ContianerToken). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-575) ContainerManager APIs should be user accessible
[ https://issues.apache.org/jira/browse/YARN-575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13687306#comment-13687306 ] Omkar Vinit Joshi commented on YARN-575: I guess this can be closed now. after YARN-694 using NMToken we can communicate with NodeManager. ContainerManager APIs should be user accessible --- Key: YARN-575 URL: https://issues.apache.org/jira/browse/YARN-575 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager Affects Versions: 2.0.4-alpha Reporter: Siddharth Seth Assignee: Vinod Kumar Vavilapalli Priority: Critical Auth for ContainerManager is based on the containerId being accessed - since this is what is used to launch containers (There's likely another jira somewhere to change this to not be containerId based). What this also means is the API is effectively not usable with kerberos credentials. Also, it should be possible to use this API with some generic tokens (RMDelegation?), instead of with Container specific tokens. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-694) Start using NMTokens to authenticate all communication with NM
[ https://issues.apache.org/jira/browse/YARN-694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Omkar Vinit Joshi updated YARN-694: --- Attachment: YARN-694-20130618.patch.yarn-694-branch-2.1-beta Start using NMTokens to authenticate all communication with NM -- Key: YARN-694 URL: https://issues.apache.org/jira/browse/YARN-694 Project: Hadoop YARN Issue Type: Bug Reporter: Omkar Vinit Joshi Assignee: Omkar Vinit Joshi Fix For: 2.1.0-beta Attachments: YARN-694-20130613.patch, YARN-694-20130617.1.patch, YARN-694-20130617.2.patch, YARN-694-20130617.patch, YARN-694-20130618.1.patch, YARN-694-20130618.2.patch, YARN-694-20130618.3.patch, YARN-694-20130618.4.patch, YARN-694-20130618.5.patch, YARN-694-20130618.patch.branch-2, YARN-694-20130618.patch.yarn-694-branch-2.1-beta AM uses the NMToken to authenticate all the AM-NM communication. NM will validate NMToken in below manner * If NMToken is using current or previous master key then the NMToken is valid. In this case it will update its cache with this key corresponding to appId. * If NMToken is using the master key which is present in NM's cache corresponding to AM's appId then it will be validated based on this. * If NMToken is invalid then NM will reject AM calls. Modification for ContainerToken * At present RPC validates AM-NM communication based on ContainerToken. It will be replaced with NMToken. Also now onwards AM will use NMToken per NM (replacing earlier behavior of ContainerToken per container per NM). * startContainer in case of Secured environment is using ContainerToken from UGI YARN-617; however after this it will use it from the payload (Container). * ContainerToken will exist and it will only be used to validate the AM's container start request. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-694) Start using NMTokens to authenticate all communication with NM
[ https://issues.apache.org/jira/browse/YARN-694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Omkar Vinit Joshi updated YARN-694: --- Attachment: YARN-694-20130618.patch.branch-2 Start using NMTokens to authenticate all communication with NM -- Key: YARN-694 URL: https://issues.apache.org/jira/browse/YARN-694 Project: Hadoop YARN Issue Type: Bug Reporter: Omkar Vinit Joshi Assignee: Omkar Vinit Joshi Fix For: 2.1.0-beta Attachments: YARN-694-20130613.patch, YARN-694-20130617.1.patch, YARN-694-20130617.2.patch, YARN-694-20130617.patch, YARN-694-20130618.1.patch, YARN-694-20130618.2.patch, YARN-694-20130618.3.patch, YARN-694-20130618.4.patch, YARN-694-20130618.5.patch, YARN-694-20130618.patch.branch-2, YARN-694-20130618.patch.yarn-694-branch-2.1-beta AM uses the NMToken to authenticate all the AM-NM communication. NM will validate NMToken in below manner * If NMToken is using current or previous master key then the NMToken is valid. In this case it will update its cache with this key corresponding to appId. * If NMToken is using the master key which is present in NM's cache corresponding to AM's appId then it will be validated based on this. * If NMToken is invalid then NM will reject AM calls. Modification for ContainerToken * At present RPC validates AM-NM communication based on ContainerToken. It will be replaced with NMToken. Also now onwards AM will use NMToken per NM (replacing earlier behavior of ContainerToken per container per NM). * startContainer in case of Secured environment is using ContainerToken from UGI YARN-617; however after this it will use it from the payload (Container). * ContainerToken will exist and it will only be used to validate the AM's container start request. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (YARN-851) Share NMTokens using NMTokenCache (api-based) instead of memory based approach which is used currently.
Omkar Vinit Joshi created YARN-851: -- Summary: Share NMTokens using NMTokenCache (api-based) instead of memory based approach which is used currently. Key: YARN-851 URL: https://issues.apache.org/jira/browse/YARN-851 Project: Hadoop YARN Issue Type: Bug Reporter: Omkar Vinit Joshi Assignee: Omkar Vinit Joshi It is a follow up ticket for YARN-694. Changing the way NMTokens are shared. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-851) Share NMTokens using NMTokenCache (api-based) instead of memory based approach which is used currently.
[ https://issues.apache.org/jira/browse/YARN-851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13687409#comment-13687409 ] Omkar Vinit Joshi commented on YARN-851: Thanks [~bikassaha] for identifying this. Share NMTokens using NMTokenCache (api-based) instead of memory based approach which is used currently. --- Key: YARN-851 URL: https://issues.apache.org/jira/browse/YARN-851 Project: Hadoop YARN Issue Type: Bug Reporter: Omkar Vinit Joshi Assignee: Omkar Vinit Joshi It is a follow up ticket for YARN-694. Changing the way NMTokens are shared. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-613) Create NM proxy per NM instead of per container
[ https://issues.apache.org/jira/browse/YARN-613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13687411#comment-13687411 ] Omkar Vinit Joshi commented on YARN-613: As a part of YARN-694 ContainerManagementProtocolProxy was added to support per node manager proxy. As all the things in here are fixed closing it. Create NM proxy per NM instead of per container --- Key: YARN-613 URL: https://issues.apache.org/jira/browse/YARN-613 Project: Hadoop YARN Issue Type: Sub-task Reporter: Bikas Saha Assignee: Omkar Vinit Joshi Attachments: AMNMToken.docx Currently a new NM proxy has to be created per container since the secure authentication is using a containertoken from the container. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (YARN-613) Create NM proxy per NM instead of per container
[ https://issues.apache.org/jira/browse/YARN-613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Omkar Vinit Joshi resolved YARN-613. Resolution: Fixed Fix Version/s: 2.1.0-beta Create NM proxy per NM instead of per container --- Key: YARN-613 URL: https://issues.apache.org/jira/browse/YARN-613 Project: Hadoop YARN Issue Type: Sub-task Reporter: Bikas Saha Assignee: Omkar Vinit Joshi Fix For: 2.1.0-beta Attachments: AMNMToken.docx Currently a new NM proxy has to be created per container since the secure authentication is using a containertoken from the container. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (YARN-797) DecodeIdentifier is broken. It was using KIND field for reflection and now we don't have class named as KIND.
[ https://issues.apache.org/jira/browse/YARN-797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Omkar Vinit Joshi reassigned YARN-797: -- Assignee: Omkar Vinit Joshi DecodeIdentifier is broken. It was using KIND field for reflection and now we don't have class named as KIND. - Key: YARN-797 URL: https://issues.apache.org/jira/browse/YARN-797 Project: Hadoop YARN Issue Type: Bug Reporter: Omkar Vinit Joshi Assignee: Omkar Vinit Joshi We need to fix the reflection code in Token.decodeIdentifier -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (YARN-797) DecodeIdentifier is broken. It was using KIND field for reflection and now we don't have class named as KIND.
[ https://issues.apache.org/jira/browse/YARN-797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Omkar Vinit Joshi resolved YARN-797. Resolution: Invalid DecodeIdentifier is broken. It was using KIND field for reflection and now we don't have class named as KIND. - Key: YARN-797 URL: https://issues.apache.org/jira/browse/YARN-797 Project: Hadoop YARN Issue Type: Bug Reporter: Omkar Vinit Joshi We need to fix the reflection code in Token.decodeIdentifier -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-851) Share NMTokens using NMTokenCache (api-based) instead of memory based approach which is used currently.
[ https://issues.apache.org/jira/browse/YARN-851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Omkar Vinit Joshi updated YARN-851: --- Attachment: YARN-851-20130618.patch Share NMTokens using NMTokenCache (api-based) instead of memory based approach which is used currently. --- Key: YARN-851 URL: https://issues.apache.org/jira/browse/YARN-851 Project: Hadoop YARN Issue Type: Bug Reporter: Omkar Vinit Joshi Assignee: Omkar Vinit Joshi Attachments: YARN-851-20130618.patch It is a follow up ticket for YARN-694. Changing the way NMTokens are shared. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-854) App submission fails on secure deploy
[
https://issues.apache.org/jira/browse/YARN-854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Omkar Vinit Joshi updated YARN-854:
---
Attachment: YARN-854.20130619.patch
App submission fails on secure deploy
-
Key: YARN-854
URL: https://issues.apache.org/jira/browse/YARN-854
Project: Hadoop YARN
Issue Type: Bug
Affects Versions: 2.1.0-beta
Reporter: Ramya Sunil
Assignee: Omkar Vinit Joshi
Priority: Blocker
Fix For: 2.1.0-beta
Attachments: YARN-854.20130619.patch
App submission on secure cluster fails with the following exception:
{noformat}
INFO mapreduce.Job: Job jobID failed with state FAILED due to: Application
applicationID failed 2 times due to AM Container for appattemptID exited with
exitCode: -1000 due to: App initialization failed (255) with output: main :
command provided 0
main : user is qa_user
javax.security.sasl.SaslException: DIGEST-MD5: digest response format
violation. Mismatched response. [Caused by
org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException):
DIGEST-MD5: digest response format violation. Mismatched response.]
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)
at
org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:104)
at
org.apache.hadoop.yarn.server.nodemanager.api.impl.pb.client.LocalizationProtocolPBClientImpl.heartbeat(LocalizationProtocolPBClientImpl.java:65)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.localizeFiles(ContainerLocalizer.java:235)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.runLocalization(ContainerLocalizer.java:169)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.main(ContainerLocalizer.java:348)
Caused by:
org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException):
DIGEST-MD5: digest response format violation. Mismatched response.
at org.apache.hadoop.ipc.Client.call(Client.java:1298)
at org.apache.hadoop.ipc.Client.call(Client.java:1250)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:204)
at $Proxy7.heartbeat(Unknown Source)
at
org.apache.hadoop.yarn.server.nodemanager.api.impl.pb.client.LocalizationProtocolPBClientImpl.heartbeat(LocalizationProtocolPBClientImpl.java:62)
... 3 more
.Failing this attempt.. Failing the application.
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-854) App submission fails on secure deploy
[
https://issues.apache.org/jira/browse/YARN-854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13688588#comment-13688588
]
Omkar Vinit Joshi commented on YARN-854:
Fixing one more thing.. making sure LocalizerTokenSecretManager is used
irrespective of security.
App submission fails on secure deploy
-
Key: YARN-854
URL: https://issues.apache.org/jira/browse/YARN-854
Project: Hadoop YARN
Issue Type: Bug
Affects Versions: 2.1.0-beta
Reporter: Ramya Sunil
Assignee: Omkar Vinit Joshi
Priority: Blocker
Fix For: 2.1.0-beta
Attachments: YARN-854.20130619.patch
App submission on secure cluster fails with the following exception:
{noformat}
INFO mapreduce.Job: Job jobID failed with state FAILED due to: Application
applicationID failed 2 times due to AM Container for appattemptID exited with
exitCode: -1000 due to: App initialization failed (255) with output: main :
command provided 0
main : user is qa_user
javax.security.sasl.SaslException: DIGEST-MD5: digest response format
violation. Mismatched response. [Caused by
org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException):
DIGEST-MD5: digest response format violation. Mismatched response.]
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)
at
org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:104)
at
org.apache.hadoop.yarn.server.nodemanager.api.impl.pb.client.LocalizationProtocolPBClientImpl.heartbeat(LocalizationProtocolPBClientImpl.java:65)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.localizeFiles(ContainerLocalizer.java:235)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.runLocalization(ContainerLocalizer.java:169)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.main(ContainerLocalizer.java:348)
Caused by:
org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException):
DIGEST-MD5: digest response format violation. Mismatched response.
at org.apache.hadoop.ipc.Client.call(Client.java:1298)
at org.apache.hadoop.ipc.Client.call(Client.java:1250)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:204)
at $Proxy7.heartbeat(Unknown Source)
at
org.apache.hadoop.yarn.server.nodemanager.api.impl.pb.client.LocalizationProtocolPBClientImpl.heartbeat(LocalizationProtocolPBClientImpl.java:62)
... 3 more
.Failing this attempt.. Failing the application.
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-854) App submission fails on secure deploy
[
https://issues.apache.org/jira/browse/YARN-854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Omkar Vinit Joshi updated YARN-854:
---
Attachment: YARN-854.20130619.1.patch
App submission fails on secure deploy
-
Key: YARN-854
URL: https://issues.apache.org/jira/browse/YARN-854
Project: Hadoop YARN
Issue Type: Bug
Affects Versions: 2.1.0-beta
Reporter: Ramya Sunil
Assignee: Omkar Vinit Joshi
Priority: Blocker
Fix For: 2.1.0-beta
Attachments: YARN-854.20130619.1.patch, YARN-854.20130619.patch
App submission on secure cluster fails with the following exception:
{noformat}
INFO mapreduce.Job: Job jobID failed with state FAILED due to: Application
applicationID failed 2 times due to AM Container for appattemptID exited with
exitCode: -1000 due to: App initialization failed (255) with output: main :
command provided 0
main : user is qa_user
javax.security.sasl.SaslException: DIGEST-MD5: digest response format
violation. Mismatched response. [Caused by
org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException):
DIGEST-MD5: digest response format violation. Mismatched response.]
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)
at
org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:104)
at
org.apache.hadoop.yarn.server.nodemanager.api.impl.pb.client.LocalizationProtocolPBClientImpl.heartbeat(LocalizationProtocolPBClientImpl.java:65)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.localizeFiles(ContainerLocalizer.java:235)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.runLocalization(ContainerLocalizer.java:169)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.main(ContainerLocalizer.java:348)
Caused by:
org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException):
DIGEST-MD5: digest response format violation. Mismatched response.
at org.apache.hadoop.ipc.Client.call(Client.java:1298)
at org.apache.hadoop.ipc.Client.call(Client.java:1250)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:204)
at $Proxy7.heartbeat(Unknown Source)
at
org.apache.hadoop.yarn.server.nodemanager.api.impl.pb.client.LocalizationProtocolPBClientImpl.heartbeat(LocalizationProtocolPBClientImpl.java:62)
... 3 more
.Failing this attempt.. Failing the application.
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-854) App submission fails on secure deploy
[
https://issues.apache.org/jira/browse/YARN-854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Omkar Vinit Joshi updated YARN-854:
---
Attachment: YARN-854.20130619.2.patch
App submission fails on secure deploy
-
Key: YARN-854
URL: https://issues.apache.org/jira/browse/YARN-854
Project: Hadoop YARN
Issue Type: Bug
Affects Versions: 2.1.0-beta
Reporter: Ramya Sunil
Assignee: Omkar Vinit Joshi
Priority: Blocker
Fix For: 2.1.0-beta
Attachments: YARN-854.20130619.1.patch, YARN-854.20130619.2.patch,
YARN-854.20130619.patch
App submission on secure cluster fails with the following exception:
{noformat}
INFO mapreduce.Job: Job jobID failed with state FAILED due to: Application
applicationID failed 2 times due to AM Container for appattemptID exited with
exitCode: -1000 due to: App initialization failed (255) with output: main :
command provided 0
main : user is qa_user
javax.security.sasl.SaslException: DIGEST-MD5: digest response format
violation. Mismatched response. [Caused by
org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException):
DIGEST-MD5: digest response format violation. Mismatched response.]
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)
at
org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:104)
at
org.apache.hadoop.yarn.server.nodemanager.api.impl.pb.client.LocalizationProtocolPBClientImpl.heartbeat(LocalizationProtocolPBClientImpl.java:65)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.localizeFiles(ContainerLocalizer.java:235)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.runLocalization(ContainerLocalizer.java:169)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.main(ContainerLocalizer.java:348)
Caused by:
org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException):
DIGEST-MD5: digest response format violation. Mismatched response.
at org.apache.hadoop.ipc.Client.call(Client.java:1298)
at org.apache.hadoop.ipc.Client.call(Client.java:1250)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:204)
at $Proxy7.heartbeat(Unknown Source)
at
org.apache.hadoop.yarn.server.nodemanager.api.impl.pb.client.LocalizationProtocolPBClientImpl.heartbeat(LocalizationProtocolPBClientImpl.java:62)
... 3 more
.Failing this attempt.. Failing the application.
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-851) Share NMTokens using NMTokenCache (api-based) instead of memory based approach which is used currently.
[ https://issues.apache.org/jira/browse/YARN-851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Omkar Vinit Joshi updated YARN-851: --- Attachment: YARN-851-20130619.patch Share NMTokens using NMTokenCache (api-based) instead of memory based approach which is used currently. --- Key: YARN-851 URL: https://issues.apache.org/jira/browse/YARN-851 Project: Hadoop YARN Issue Type: Bug Reporter: Omkar Vinit Joshi Assignee: Omkar Vinit Joshi Attachments: YARN-851-20130618.patch, YARN-851-20130619.patch It is a follow up ticket for YARN-694. Changing the way NMTokens are shared. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-851) Share NMTokens using NMTokenCache (api-based) instead of memory based approach which is used currently.
[ https://issues.apache.org/jira/browse/YARN-851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13688812#comment-13688812 ] Omkar Vinit Joshi commented on YARN-851: Fixed above comments. removed getAllNMTokens(). Share NMTokens using NMTokenCache (api-based) instead of memory based approach which is used currently. --- Key: YARN-851 URL: https://issues.apache.org/jira/browse/YARN-851 Project: Hadoop YARN Issue Type: Bug Reporter: Omkar Vinit Joshi Assignee: Omkar Vinit Joshi Attachments: YARN-851-20130618.patch, YARN-851-20130619.patch It is a follow up ticket for YARN-694. Changing the way NMTokens are shared. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-862) ResourceManager and NodeManager versions should match on node registration or error out
[ https://issues.apache.org/jira/browse/YARN-862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13689496#comment-13689496 ] Omkar Vinit Joshi commented on YARN-862: Won't this become a problem in case we are planning to implement rolling upgrades? we will have to bring the whole system down.. right? before updating even minor corrections? ResourceManager and NodeManager versions should match on node registration or error out --- Key: YARN-862 URL: https://issues.apache.org/jira/browse/YARN-862 Project: Hadoop YARN Issue Type: Bug Components: nodemanager, resourcemanager Affects Versions: 0.23.8 Reporter: Robert Parker Assignee: Robert Parker Attachments: YARN-862-b0.23-v1.patch For branch-0.23 the versions of the node manager and the resource manager should match to complete a successful registration. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-851) Share NMTokens using NMTokenCache (api-based) instead of memory based approach which is used currently.
[ https://issues.apache.org/jira/browse/YARN-851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Omkar Vinit Joshi updated YARN-851: --- Attachment: YARN-851-20130619.patch Share NMTokens using NMTokenCache (api-based) instead of memory based approach which is used currently. --- Key: YARN-851 URL: https://issues.apache.org/jira/browse/YARN-851 Project: Hadoop YARN Issue Type: Bug Reporter: Omkar Vinit Joshi Assignee: Omkar Vinit Joshi Attachments: YARN-851-20130618.patch, YARN-851-20130619.patch, YARN-851-20130619.patch It is a follow up ticket for YARN-694. Changing the way NMTokens are shared. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-851) Share NMTokens using NMTokenCache (api-based) instead of memory based approach which is used currently.
[ https://issues.apache.org/jira/browse/YARN-851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Omkar Vinit Joshi updated YARN-851: --- Attachment: YARN-851-20130620.patch Share NMTokens using NMTokenCache (api-based) instead of memory based approach which is used currently. --- Key: YARN-851 URL: https://issues.apache.org/jira/browse/YARN-851 Project: Hadoop YARN Issue Type: Bug Reporter: Omkar Vinit Joshi Assignee: Omkar Vinit Joshi Attachments: YARN-851-20130618.patch, YARN-851-20130619.patch, YARN-851-20130619.patch, YARN-851-20130620.patch It is a follow up ticket for YARN-694. Changing the way NMTokens are shared. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-851) Share NMTokens using NMTokenCache (api-based) instead of memory based approach which is used currently.
[ https://issues.apache.org/jira/browse/YARN-851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Omkar Vinit Joshi updated YARN-851: --- Attachment: (was: YARN-851-20130620.patch) Share NMTokens using NMTokenCache (api-based) instead of memory based approach which is used currently. --- Key: YARN-851 URL: https://issues.apache.org/jira/browse/YARN-851 Project: Hadoop YARN Issue Type: Bug Reporter: Omkar Vinit Joshi Assignee: Omkar Vinit Joshi Attachments: YARN-851-20130618.patch, YARN-851-20130619.patch, YARN-851-20130619.patch, YARN-851-20130620.patch It is a follow up ticket for YARN-694. Changing the way NMTokens are shared. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-851) Share NMTokens using NMTokenCache (api-based) instead of memory based approach which is used currently.
[ https://issues.apache.org/jira/browse/YARN-851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13689792#comment-13689792 ] Omkar Vinit Joshi commented on YARN-851: fixing vinod's comments. marking class and all methods as @Public and @Evolving. Share NMTokens using NMTokenCache (api-based) instead of memory based approach which is used currently. --- Key: YARN-851 URL: https://issues.apache.org/jira/browse/YARN-851 Project: Hadoop YARN Issue Type: Bug Reporter: Omkar Vinit Joshi Assignee: Omkar Vinit Joshi Attachments: YARN-851-20130618.patch, YARN-851-20130619.patch, YARN-851-20130619.patch, YARN-851-20130620.patch It is a follow up ticket for YARN-694. Changing the way NMTokens are shared. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-851) Share NMTokens using NMTokenCache (api-based) instead of memory based approach which is used currently.
[ https://issues.apache.org/jira/browse/YARN-851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Omkar Vinit Joshi updated YARN-851: --- Attachment: YARN-851-20130620.1.patch Share NMTokens using NMTokenCache (api-based) instead of memory based approach which is used currently. --- Key: YARN-851 URL: https://issues.apache.org/jira/browse/YARN-851 Project: Hadoop YARN Issue Type: Bug Reporter: Omkar Vinit Joshi Assignee: Omkar Vinit Joshi Attachments: YARN-851-20130618.patch, YARN-851-20130619.patch, YARN-851-20130619.patch, YARN-851-20130620.1.patch, YARN-851-20130620.patch It is a follow up ticket for YARN-694. Changing the way NMTokens are shared. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
