[jira] [Commented] (MAPREDUCE-6838) [ATSv2 Security] Add timeline delegation token received in allocate response to UGI
[ https://issues.apache.org/jira/browse/MAPREDUCE-6838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16146765#comment-16146765 ] Hudson commented on MAPREDUCE-6838: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12271 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/12271/]) MAPREDUCE-6838. [ATSv2 Security] Add timeline delegation token received (varunsaxena: rev 08f40bcc7f4174857bb1fc7c8eb1108d5caaafb3) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/timelineservice/TestTimelineServiceClientIntegration.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/api/TimelineV2Client.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/api/impl/TimelineV2ClientImpl.java * (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/rm/TestRMContainerAllocator.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/timelineservice/NMTimelinePublisher.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/CollectorInfo.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/timelineservice/security/TestTimelineAuthFilterForV2.java * (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/async/impl/AMRMClientAsyncImpl.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestTimelineClientV2Impl.java MAPREDUCE-6838. Addendum to fix code comment (varunsaxena: rev 16ba4f544f13d614c1ebd6101ee14f7714e0fc54) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/api/impl/TimelineV2ClientImpl.java > [ATSv2 Security] Add timeline delegation token received in allocate response > to UGI > --- > > Key: MAPREDUCE-6838 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6838 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: yarn-5355-merge-blocker > Fix For: YARN-5355, YARN-5355-branch-2 > > Attachments: MAPREDUCE-6838-YARN-5355.01.patch, > MAPREDUCE-6838-YARN-5355.02.patch, MAPREDUCE-6838-YARN-5355.03.patch, > MAPREDUCE-6838-YARN-5355.04.patch, MAPREDUCE-6838-YARN-5355.05.patch, > MAPREDUCE-6838-YARN-5355.06.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6838) [ATSv2 Security] Add timeline delegation token received in allocate response to UGI
[ https://issues.apache.org/jira/browse/MAPREDUCE-6838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16136379#comment-16136379 ] Varun Saxena commented on MAPREDUCE-6838: - Cherry picked MAPREDUCE-6838 to YARN-5355_branch2. > [ATSv2 Security] Add timeline delegation token received in allocate response > to UGI > --- > > Key: MAPREDUCE-6838 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6838 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: yarn-5355-merge-blocker > Fix For: YARN-5355 > > Attachments: MAPREDUCE-6838-YARN-5355.01.patch, > MAPREDUCE-6838-YARN-5355.02.patch, MAPREDUCE-6838-YARN-5355.03.patch, > MAPREDUCE-6838-YARN-5355.04.patch, MAPREDUCE-6838-YARN-5355.05.patch, > MAPREDUCE-6838-YARN-5355.06.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6838) [ATSv2 Security] Add timeline delegation token received in allocate response to UGI
[ https://issues.apache.org/jira/browse/MAPREDUCE-6838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16136345#comment-16136345 ] Varun Saxena commented on MAPREDUCE-6838: - Thanks [~jianhe] for the review and commit. Thanks [~rohithsharma] for reviews. I will resolve branch2 compilation issue and then cherry-pick this as well. > [ATSv2 Security] Add timeline delegation token received in allocate response > to UGI > --- > > Key: MAPREDUCE-6838 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6838 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: yarn-5355-merge-blocker > Fix For: YARN-5355 > > Attachments: MAPREDUCE-6838-YARN-5355.01.patch, > MAPREDUCE-6838-YARN-5355.02.patch, MAPREDUCE-6838-YARN-5355.03.patch, > MAPREDUCE-6838-YARN-5355.04.patch, MAPREDUCE-6838-YARN-5355.05.patch, > MAPREDUCE-6838-YARN-5355.06.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6838) [ATSv2 Security] Add timeline delegation token received in allocate response to UGI
[ https://issues.apache.org/jira/browse/MAPREDUCE-6838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16136323#comment-16136323 ] Jian He commented on MAPREDUCE-6838: Yep, comment race - I just resolved this jira too. > [ATSv2 Security] Add timeline delegation token received in allocate response > to UGI > --- > > Key: MAPREDUCE-6838 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6838 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: yarn-5355-merge-blocker > Fix For: YARN-5355 > > Attachments: MAPREDUCE-6838-YARN-5355.01.patch, > MAPREDUCE-6838-YARN-5355.02.patch, MAPREDUCE-6838-YARN-5355.03.patch, > MAPREDUCE-6838-YARN-5355.04.patch, MAPREDUCE-6838-YARN-5355.05.patch, > MAPREDUCE-6838-YARN-5355.06.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6838) [ATSv2 Security] Add timeline delegation token received in allocate response to UGI
[ https://issues.apache.org/jira/browse/MAPREDUCE-6838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16136322#comment-16136322 ] Vrushali C commented on MAPREDUCE-6838: --- Thanks [~jianhe]! > [ATSv2 Security] Add timeline delegation token received in allocate response > to UGI > --- > > Key: MAPREDUCE-6838 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6838 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: yarn-5355-merge-blocker > Fix For: YARN-5355 > > Attachments: MAPREDUCE-6838-YARN-5355.01.patch, > MAPREDUCE-6838-YARN-5355.02.patch, MAPREDUCE-6838-YARN-5355.03.patch, > MAPREDUCE-6838-YARN-5355.04.patch, MAPREDUCE-6838-YARN-5355.05.patch, > MAPREDUCE-6838-YARN-5355.06.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6838) [ATSv2 Security] Add timeline delegation token received in allocate response to UGI
[ https://issues.apache.org/jira/browse/MAPREDUCE-6838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16136313#comment-16136313 ] Vrushali C commented on MAPREDUCE-6838: --- Thank you [~jianhe]! bq. but looks like YARN-5355_branch2 has compilation error without this patch. Rohith Sharma K S, Varun Saxena, can you check ? If it's ok with you, I will file a new jira to track this branch2 problem and this jira can be resolved with fix version of "YARN-5355"? > [ATSv2 Security] Add timeline delegation token received in allocate response > to UGI > --- > > Key: MAPREDUCE-6838 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6838 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: yarn-5355-merge-blocker > Fix For: YARN-5355 > > Attachments: MAPREDUCE-6838-YARN-5355.01.patch, > MAPREDUCE-6838-YARN-5355.02.patch, MAPREDUCE-6838-YARN-5355.03.patch, > MAPREDUCE-6838-YARN-5355.04.patch, MAPREDUCE-6838-YARN-5355.05.patch, > MAPREDUCE-6838-YARN-5355.06.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6838) [ATSv2 Security] Add timeline delegation token received in allocate response to UGI
[ https://issues.apache.org/jira/browse/MAPREDUCE-6838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16136305#comment-16136305 ] Jian He commented on MAPREDUCE-6838: I tried to commit to YARN-5355_branch2, but looks like YARN-5355_branch2 has compilation error without this patch. [~rohithsharma], [~varun_saxena], can you check ? I've committed the patch to YARN-5355 branch - but I forgot to update the aforementioned codecomment..[~rohithsharma], [~varun_saxena], maybe you can just update it in next whatever patch you have.. > [ATSv2 Security] Add timeline delegation token received in allocate response > to UGI > --- > > Key: MAPREDUCE-6838 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6838 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: yarn-5355-merge-blocker > Fix For: YARN-5355 > > Attachments: MAPREDUCE-6838-YARN-5355.01.patch, > MAPREDUCE-6838-YARN-5355.02.patch, MAPREDUCE-6838-YARN-5355.03.patch, > MAPREDUCE-6838-YARN-5355.04.patch, MAPREDUCE-6838-YARN-5355.05.patch, > MAPREDUCE-6838-YARN-5355.06.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6838) [ATSv2 Security] Add timeline delegation token received in allocate response to UGI
[ https://issues.apache.org/jira/browse/MAPREDUCE-6838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16136288#comment-16136288 ] Jian He commented on MAPREDUCE-6838: bq. The code condition is correct. Will change the comment. No worry, I can fix this at commit, no need to upload a new patch just for this. bq. Could not find any API to remove the token from UGI. Not sure why. Should we add one? Yeah, I think we can open a jira in hadoop-common for this request, and fix the issue later. I'm committing the patch , thanks > [ATSv2 Security] Add timeline delegation token received in allocate response > to UGI > --- > > Key: MAPREDUCE-6838 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6838 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: yarn-5355-merge-blocker > Fix For: YARN-5355 > > Attachments: MAPREDUCE-6838-YARN-5355.01.patch, > MAPREDUCE-6838-YARN-5355.02.patch, MAPREDUCE-6838-YARN-5355.03.patch, > MAPREDUCE-6838-YARN-5355.04.patch, MAPREDUCE-6838-YARN-5355.05.patch, > MAPREDUCE-6838-YARN-5355.06.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6838) [ATSv2 Security] Add timeline delegation token received in allocate response to UGI
[ https://issues.apache.org/jira/browse/MAPREDUCE-6838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16136206#comment-16136206 ] Varun Saxena commented on MAPREDUCE-6838: - Thanks [~jianhe] for the comments. bq. The comment says is OR condition where as the code is AND, which one is true? The code condition is correct. Will change the comment. bq. Also, when will the "delegationToken.getService()" be empty ? These are just checks for sanity. As NodeTimelineCollectorManager belongs to timelineservice module and this to yarn-common. So added these checks because change elsewhere should not break code here. bq. it uses "SecurityUtil.getTokenServiceAddr(timelineToken)" to set the token service. Then next time collectorAddr is not null because timelineServiceAddress is not null, it always call "NetUtils.createSocketAddr(collectorAddr) " to set the token service. Is my understanding correct? why not just consistently use one of them to make it look simpler? So this is because we are polling on timelineservice address in another thread(entity dispatcher) and as soon as it is found, we go on to publish. So there can be a potential race so I first update the token and then the timeline address. I can write a comment in code to make this clear. bq. Does the collector address change if NM restarts? If so, we may have two keys(different address) for two tokens in the UGI. Yes, that's true but the token will be picked up by DelegationTokenAuthenticatedURL based on current collector address. Could not find any API to remove the token from UGI. > [ATSv2 Security] Add timeline delegation token received in allocate response > to UGI > --- > > Key: MAPREDUCE-6838 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6838 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: yarn-5355-merge-blocker > Fix For: YARN-5355 > > Attachments: MAPREDUCE-6838-YARN-5355.01.patch, > MAPREDUCE-6838-YARN-5355.02.patch, MAPREDUCE-6838-YARN-5355.03.patch, > MAPREDUCE-6838-YARN-5355.04.patch, MAPREDUCE-6838-YARN-5355.05.patch, > MAPREDUCE-6838-YARN-5355.06.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6838) [ATSv2 Security] Add timeline delegation token received in allocate response to UGI
[
https://issues.apache.org/jira/browse/MAPREDUCE-6838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16135858#comment-16135858
]
Jian He commented on MAPREDUCE-6838:
- The comment says is OR condition where as the code is AND, which one is true
? Also, when will the "delegationToken.getService()" be empty ? looks like the
NodeTimelineCollectorManager#generateTokenAndSetTimer is always setting the
service field.
{code}
// Token need not be updated if either address or token service does not
// exist.
String service = delegationToken.getService();
if ((service == null || service.isEmpty()) &&
(collectorAddr == null || collectorAddr.isEmpty())) {
LOG.warn("Timeline token does not have service and timeline service " +
"address is not yet set. Not updating the token");
return;
}
{code}
- Here if this method is called for the first time, timelineServiceAddress is
null, and collectorAddr is null
{code}
if (collectorAddr == null || collectorAddr.isEmpty()) {
collectorAddr = timelineServiceAddress;
}
{code}
later here, it uses "SecurityUtil.getTokenServiceAddr(timelineToken)" to set
the token service. Then next time collectorAddr is not null because
timelineServiceAddress is not null, it always call
"NetUtils.createSocketAddr(collectorAddr) " to set the token service. Is my
understanding correct? why not just consistently use one of them to make it
look simpler?
{code}
// Prefer timeline service address over service coming in the token for
// updating the token service.
InetSocketAddress serviceAddr =
(collectorAddr != null && !collectorAddr.isEmpty()) ?
NetUtils.createSocketAddr(collectorAddr) :
SecurityUtil.getTokenServiceAddr(timelineToken);
SecurityUtil.setTokenService(timelineToken, serviceAddr);
authUgi.addToken(timelineToken);
{code}
- Does the collector address change if NM restarts? If so, we may have two
keys(different address) for two tokens in the UGI.
> [ATSv2 Security] Add timeline delegation token received in allocate response
> to UGI
> ---
>
> Key: MAPREDUCE-6838
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6838
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
>Reporter: Varun Saxena
>Assignee: Varun Saxena
> Labels: yarn-5355-merge-blocker
> Fix For: YARN-5355
>
> Attachments: MAPREDUCE-6838-YARN-5355.01.patch,
> MAPREDUCE-6838-YARN-5355.02.patch, MAPREDUCE-6838-YARN-5355.03.patch,
> MAPREDUCE-6838-YARN-5355.04.patch, MAPREDUCE-6838-YARN-5355.05.patch,
> MAPREDUCE-6838-YARN-5355.06.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6838) [ATSv2 Security] Add timeline delegation token received in allocate response to UGI
[
https://issues.apache.org/jira/browse/MAPREDUCE-6838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16135572#comment-16135572
]
Hadoop QA commented on MAPREDUCE-6838:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
|| || || || {color:brown} YARN-5355 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
35s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m
34s{color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
45s{color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
50s{color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
53s{color} | {color:green} YARN-5355 passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
59s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common in
YARN-5355 has 2 extant Findbugs warnings. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
47s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
in YARN-5355 has 5 extant Findbugs warnings. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
33s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client in
YARN-5355 has 2 extant Findbugs warnings. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
39s{color} | {color:red}
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app in
YARN-5355 has 3 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
58s{color} | {color:green} YARN-5355 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 12m
12s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
49s{color} | {color:red} root: The patch generated 6 new + 238 unchanged - 1
fixed = 244 total (was 239) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
53s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
29s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
28s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 13m
9s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 6m
[jira] [Commented] (MAPREDUCE-6838) [ATSv2 Security] Add timeline delegation token received in allocate response to UGI
[ https://issues.apache.org/jira/browse/MAPREDUCE-6838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16135464#comment-16135464 ] Varun Saxena commented on MAPREDUCE-6838: - Test failures are unrelated. They are outstanding issues on trunk > [ATSv2 Security] Add timeline delegation token received in allocate response > to UGI > --- > > Key: MAPREDUCE-6838 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6838 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: yarn-5355-merge-blocker > Fix For: YARN-5355 > > Attachments: MAPREDUCE-6838-YARN-5355.01.patch, > MAPREDUCE-6838-YARN-5355.02.patch, MAPREDUCE-6838-YARN-5355.03.patch, > MAPREDUCE-6838-YARN-5355.04.patch, MAPREDUCE-6838-YARN-5355.05.patch, > MAPREDUCE-6838-YARN-5355.06.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6838) [ATSv2 Security] Add timeline delegation token received in allocate response to UGI
[
https://issues.apache.org/jira/browse/MAPREDUCE-6838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16134491#comment-16134491
]
Hadoop QA commented on MAPREDUCE-6838:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
26s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
|| || || || {color:brown} YARN-5355 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
31s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 25m
56s{color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 22m
27s{color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
54s{color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 6m
3s{color} | {color:green} YARN-5355 passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
24s{color} | {color:red}
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app in
YARN-5355 has 3 extant Findbugs warnings. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
14s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client in
YARN-5355 has 2 extant Findbugs warnings. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
46s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common in
YARN-5355 has 2 extant Findbugs warnings. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
24s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
in YARN-5355 has 5 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m
19s{color} | {color:green} YARN-5355 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
23s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 21m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 21m
36s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 2m
52s{color} | {color:red} root: The patch generated 6 new + 238 unchanged - 1
fixed = 244 total (was 239) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 5m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 9m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m
16s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
2s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
47s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 16m 25s{color}
| {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 7m 59s{color}
|
[jira] [Commented] (MAPREDUCE-6838) [ATSv2 Security] Add timeline delegation token received in allocate response to UGI
[ https://issues.apache.org/jira/browse/MAPREDUCE-6838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16134420#comment-16134420 ] Varun Saxena commented on MAPREDUCE-6838: - Attaching a patch which makes token variable volatile, adds a LOG when collectorinfo is null and does small refactoring. > [ATSv2 Security] Add timeline delegation token received in allocate response > to UGI > --- > > Key: MAPREDUCE-6838 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6838 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: yarn-5355-merge-blocker > Fix For: YARN-5355 > > Attachments: MAPREDUCE-6838-YARN-5355.01.patch, > MAPREDUCE-6838-YARN-5355.02.patch, MAPREDUCE-6838-YARN-5355.03.patch, > MAPREDUCE-6838-YARN-5355.04.patch, MAPREDUCE-6838-YARN-5355.05.patch, > MAPREDUCE-6838-YARN-5355.06.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6838) [ATSv2 Security] Add timeline delegation token received in allocate response to UGI
[ https://issues.apache.org/jira/browse/MAPREDUCE-6838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16134241#comment-16134241 ] Varun Saxena commented on MAPREDUCE-6838: - Thanks [~rohithsharma] for the review. bq. Need to log a WARN message if collector info is null. Ok. bq. currentTimelineToken should be volatile Need not be. Atleast in MR AM. This is used only while AM is updating the token and that happens only from RMContainer Allocator thread so only one thread sees and updates it. While using token is picked from UGI. Will it be likely that token will be updated from 2 separate threads? We anyways do not claim any thread safety for timeline client. Address is volatile and its different because the thread publishing the entity and using the address would be different from the allocator thread which would communicate with RM and update the address. However, making it volatile doesn't cost us anything. As you say. I do not have a strong opinion on this. Thoughts? bq. Creating Token does not required to check service==null. Internally constructor does. And we can ignore token service passed by delegationToken always and set it up collector address. You mean the constructor inside setTimelineDelegationToken method i.e. at L203? Actually the constructor takes service as Text and not String. The check I am making is for service as String. If I do not make the check and call new Text(service), a null service would throw NPE. bq. !delegationToken.getKind().equals(TimelineDelegationTokenIdentifier.KIND_NAME.toString()) check is not required since equals does this comparrission too. Didn't quite get you. This is to avoid updating token for another kind. This is to avoid updating a token altogether i.e. even if we do not have a previous token. The equals check is for not updating the token if it is equal to cached token. If I remove this check, a token of another kind will be added in UGI. bq. In CollectorInfo object, If collector address is null and Token is non-null. Do not add that token into ugi. Check like this required? If token service exists and timeline service address is already updated, should we not update the token, if we look at this piece of code independently. Currently we send both together but the protocol doesn't enforce it. The proto definition of CollectorInfo marks collector address field as optional. These checks are primarily for robustness if we consider the TimelineV2Client code in isolation and not merely go by what we currently know RM sends. If we make assumptions based on current implementation, we are tightly coupling the RM/NM logic with logic here and it is not enforced by protocol either. It is likely to work just fine as developers would take care but I would suggest that ideally if we assume that collector address is carried always, we enforce it in proto definition of CollectorInfo i.e. make collector address as "required" instead of "optional" in it. And we will have to see if address should be "required" in AppCollectorData too. Please note that RM may not have access to collector address initially when AM container is launched. So this change would also mean change in RM to not send collector info at all if address is null. The last comment i.e. suggested refactoring depends on comments above i.e. whether to update the token or not if address is not carried in collector info. Thoughts? > [ATSv2 Security] Add timeline delegation token received in allocate response > to UGI > --- > > Key: MAPREDUCE-6838 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6838 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: yarn-5355-merge-blocker > Fix For: YARN-5355 > > Attachments: MAPREDUCE-6838-YARN-5355.01.patch, > MAPREDUCE-6838-YARN-5355.02.patch, MAPREDUCE-6838-YARN-5355.03.patch, > MAPREDUCE-6838-YARN-5355.04.patch, MAPREDUCE-6838-YARN-5355.05.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6838) [ATSv2 Security] Add timeline delegation token received in allocate response to UGI
[
https://issues.apache.org/jira/browse/MAPREDUCE-6838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16134221#comment-16134221
]
Rohith Sharma K S commented on MAPREDUCE-6838:
--
thanks Varun for updating patch! Overall approach looks reasonable to me.
Some comments on _setTimelineCollectorInfo_ method. Can it be simplified?
# Need to log a WARN message if collector info is null.
# In CollectorInfo object, If collector address is null and Token is non-null.
Do not add that token into ugi.
# it means definitely something wrong in input. .
# currentTimelineToken should be volatile
#
{{!delegationToken.getKind().equals(TimelineDelegationTokenIdentifier.KIND_NAME.toString())}}
check is not required since equals does this comparrission too.
# Creating Token does not required to check service==null. Internally
constructor does. And we can ignore token service passed by delegationToken
always and set it up collector address.
# can this method modified as below?
{code}
public void setTimelineCollectorInfo(CollectorInfo collectorInfo) {
if (collectorInfo == null || collectorInfo.getCollectorAddr()== null) {
LOG.warn(""); // warning message
return;
}
if (collectorInfo.getCollectorToken() != null) {
setTimelineDelegationToken(collectorInfo.getCollectorToken(),
collectorInfo.getCollectorAddr());
}
this.timelineServiceAddress = collectorInfo.getCollectorAddr();
LOG.info("Updated timeline service address to " + timelineServiceAddress);
}
private void setTimelineDelegationToken(Token delegationToken,
String collectorAddr) {
if (currentTimelineToken != null
&& currentTimelineToken.equals(delegationToken)) {
return;
}
currentTimelineToken = delegationToken;
org.apache.hadoop.security.token.Token
timelineToken =
new
org.apache.hadoop.security.token.Token(
delegationToken.getIdentifier().array(),
delegationToken.getPassword().array(),
new Text(delegationToken.getKind()),
null);
InetSocketAddress serviceAddr = NetUtils.createSocketAddr(collectorAddr);
SecurityUtil.setTokenService(timelineToken, serviceAddr);
authUgi.addToken(timelineToken);
LOG.info("Updated timeline delegation token " + timelineToken);
}
{code}
> [ATSv2 Security] Add timeline delegation token received in allocate response
> to UGI
> ---
>
> Key: MAPREDUCE-6838
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6838
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
>Reporter: Varun Saxena
>Assignee: Varun Saxena
> Labels: yarn-5355-merge-blocker
> Fix For: YARN-5355
>
> Attachments: MAPREDUCE-6838-YARN-5355.01.patch,
> MAPREDUCE-6838-YARN-5355.02.patch, MAPREDUCE-6838-YARN-5355.03.patch,
> MAPREDUCE-6838-YARN-5355.04.patch, MAPREDUCE-6838-YARN-5355.05.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6838) [ATSv2 Security] Add timeline delegation token received in allocate response to UGI
[ https://issues.apache.org/jira/browse/MAPREDUCE-6838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16134109#comment-16134109 ] Rohith Sharma K S commented on MAPREDUCE-6838: -- I think No need. We can do here only. How about renaming existing API _setTimelineServiceAddress_ to _setCollectorInfo_? By this way, in future any update on collector info could be re used without any API change. This change can made since it is subjected to alpha modifications still. > [ATSv2 Security] Add timeline delegation token received in allocate response > to UGI > --- > > Key: MAPREDUCE-6838 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6838 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: yarn-5355-merge-blocker > Fix For: YARN-5355 > > Attachments: MAPREDUCE-6838-YARN-5355.01.patch, > MAPREDUCE-6838-YARN-5355.02.patch, MAPREDUCE-6838-YARN-5355.03.patch, > MAPREDUCE-6838-YARN-5355.04.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6838) [ATSv2 Security] Add timeline delegation token received in allocate response to UGI
[ https://issues.apache.org/jira/browse/MAPREDUCE-6838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16134049#comment-16134049 ] Rohith Sharma K S commented on MAPREDUCE-6838: -- bq. Should we follow the same ? The client can construct the tokenService based on the collector address info ? thats fair point. TimelineClient should construct token service. thanks for pointing out this. > [ATSv2 Security] Add timeline delegation token received in allocate response > to UGI > --- > > Key: MAPREDUCE-6838 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6838 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: yarn-5355-merge-blocker > Fix For: YARN-5355 > > Attachments: MAPREDUCE-6838-YARN-5355.01.patch, > MAPREDUCE-6838-YARN-5355.02.patch, MAPREDUCE-6838-YARN-5355.03.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6838) [ATSv2 Security] Add timeline delegation token received in allocate response to UGI
[ https://issues.apache.org/jira/browse/MAPREDUCE-6838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16134029#comment-16134029 ] Varun Saxena commented on MAPREDUCE-6838: - bq. The client can construct the tokenService based on the collector address info ? I guess you were referring to timeline client too. I will update a patch by adding a new API in TimelineV2Client. We will use the token service if it comes in the token, otherwise use the address. > [ATSv2 Security] Add timeline delegation token received in allocate response > to UGI > --- > > Key: MAPREDUCE-6838 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6838 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: yarn-5355-merge-blocker > Fix For: YARN-5355 > > Attachments: MAPREDUCE-6838-YARN-5355.01.patch, > MAPREDUCE-6838-YARN-5355.02.patch, MAPREDUCE-6838-YARN-5355.03.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6838) [ATSv2 Security] Add timeline delegation token received in allocate response to UGI
[ https://issues.apache.org/jira/browse/MAPREDUCE-6838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16134021#comment-16134021 ] Varun Saxena commented on MAPREDUCE-6838: - Maybe what we can do is that provide another API in TimelineV2Client, say, setTimelineToken and that does the job of sanitizing the service and setting the token. That is, move the code from RMContainerAllocator to TimelineV2ClientImpl and set the token in UGI there. Thoughts? > [ATSv2 Security] Add timeline delegation token received in allocate response > to UGI > --- > > Key: MAPREDUCE-6838 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6838 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: yarn-5355-merge-blocker > Fix For: YARN-5355 > > Attachments: MAPREDUCE-6838-YARN-5355.01.patch, > MAPREDUCE-6838-YARN-5355.02.patch, MAPREDUCE-6838-YARN-5355.03.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6838) [ATSv2 Security] Add timeline delegation token received in allocate response to UGI
[ https://issues.apache.org/jira/browse/MAPREDUCE-6838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16134019#comment-16134019 ] Varun Saxena commented on MAPREDUCE-6838: - bq. today, for other delegation tokens RMDelegationToken, the old ATSv1 DelegationToken, the token service is not set at server side, it is set at client side - the client call the SecurityUtils#buildTokenService and then set the token service. I thought about this option too. But the issue I see here is that DelegationTokenAuthenticatedURL(used by TimelineV2ClientImpl) uses use_ip config and then each AM(if it does not use kerberos), will have to then write the code to sanitize the service coming in the token or fill the service based on collector address when token comes. This would not be done transparently. Currently we do this transparently for ATSv1 in YarnClientImpl > [ATSv2 Security] Add timeline delegation token received in allocate response > to UGI > --- > > Key: MAPREDUCE-6838 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6838 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: yarn-5355-merge-blocker > Fix For: YARN-5355 > > Attachments: MAPREDUCE-6838-YARN-5355.01.patch, > MAPREDUCE-6838-YARN-5355.02.patch, MAPREDUCE-6838-YARN-5355.03.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6838) [ATSv2 Security] Add timeline delegation token received in allocate response to UGI
[ https://issues.apache.org/jira/browse/MAPREDUCE-6838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16134014#comment-16134014 ] Jian He commented on MAPREDUCE-6838: Think one other way would be when we create the token service in generateTokenForAppCollector, using the same SecurityUtil#buildTokenService API - doing this approach requires AM and NM be consistent on the use_ip config. > [ATSv2 Security] Add timeline delegation token received in allocate response > to UGI > --- > > Key: MAPREDUCE-6838 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6838 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: yarn-5355-merge-blocker > Fix For: YARN-5355 > > Attachments: MAPREDUCE-6838-YARN-5355.01.patch, > MAPREDUCE-6838-YARN-5355.02.patch, MAPREDUCE-6838-YARN-5355.03.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6838) [ATSv2 Security] Add timeline delegation token received in allocate response to UGI
[ https://issues.apache.org/jira/browse/MAPREDUCE-6838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16134009#comment-16134009 ] Jian He commented on MAPREDUCE-6838: today, for other delegation tokens RMDelegationToken, the old ATSv1 DelegationToken, the token service is not set at server side, it is set at client side - the client call the SecurityUtils#buildTokenService and then set the token service. I don't know what it was done like that - maybe because it avoids the use_ip config inconsistency between client and serve ? Should we follow the same ? The client can construct the tokenService based on the collector address info ? (One caveat is to make sure the old token gets probably replaced properly - in case ip changes ?) The CollectorInfo#getCollectorAddr right now is a string, should it be an address type ? > [ATSv2 Security] Add timeline delegation token received in allocate response > to UGI > --- > > Key: MAPREDUCE-6838 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6838 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: yarn-5355-merge-blocker > Fix For: YARN-5355 > > Attachments: MAPREDUCE-6838-YARN-5355.01.patch, > MAPREDUCE-6838-YARN-5355.02.patch, MAPREDUCE-6838-YARN-5355.03.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6838) [ATSv2 Security] Add timeline delegation token received in allocate response to UGI
[
https://issues.apache.org/jira/browse/MAPREDUCE-6838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16133420#comment-16133420
]
Hadoop QA commented on MAPREDUCE-6838:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 18m
24s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 6 new or modified test
files. {color} |
|| || || || {color:brown} YARN-5355 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
37s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 28m
34s{color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 29m
17s{color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
31s{color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 8m
44s{color} | {color:green} YARN-5355 passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m
41s{color} | {color:red} hadoop-common-project/hadoop-common in YARN-5355 has
17 extant Findbugs warnings. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
34s{color} | {color:red}
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app in
YARN-5355 has 3 extant Findbugs warnings. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m
14s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common in
YARN-5355 has 2 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m
30s{color} | {color:green} YARN-5355 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
29s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 21m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 21m
48s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 2m
51s{color} | {color:red} root: The patch generated 14 new + 164 unchanged - 5
fixed = 178 total (was 169) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 6m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 9m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m
58s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 10m 57s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
59s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
42s{color} | {color:green} hadoop-yarn-server-timelineservice in the patch
passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 5m 52s{color}
| {color:red} hadoop-yarn-server-tests in the patch failed. {color} |
| {color:green}+1{color} |
[jira] [Commented] (MAPREDUCE-6838) [ATSv2 Security] Add timeline delegation token received in allocate response to UGI
[ https://issues.apache.org/jira/browse/MAPREDUCE-6838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16132092#comment-16132092 ] Varun Saxena commented on MAPREDUCE-6838: - bq. Why are changing hadoop-common code to ignore ignoreUseIpConfig? Is this because to add test case? Does test is failing? As discussed offline, this is to ensure that token service resolution is in sync between NM and AM. Also, with default configurations, E2E flow won't work. bq. I think it is better to update token on each heart beat rather then comparing each time. It avoids proto compare. This would involve iterating over all the tokens on each HB when we add token to UGI. Infact depending on how many tokens a AM has in UGI, this may be even more expensive. bq. This appears mixing collectorManager i.e NMCollectorService authentication and MR side. The latest patch file contains code from YARN-7006. I used a dummy patch just to ensure QA runs. Will delete that patch as YARN-7006 has gone in. bq. One thing I observe is TimelineClient is published using login user which is doAS. I think we should publish is as owner of application otherwise we end up in writing data into sub app application table as well. Fair point. We can raise a separate JIRA for this. And create a proxy user based on app user to ensure timeline client publishes with that. bq. I just noticed that TimelineDelegationTokenIdentifier#Renewer has methods renew/cancel. These creates V1 client. Does it going to be a problem? This is not used by us for V2 anywhere and the Renewer class is annotated as Private. I guess no client would explicitly call it and even if they do, it should fail. > [ATSv2 Security] Add timeline delegation token received in allocate response > to UGI > --- > > Key: MAPREDUCE-6838 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6838 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: yarn-5355-merge-blocker > Fix For: YARN-5355 > > Attachments: MAPREDUCE-6838-YARN-5355.01.patch, > MAPREDUCE-6838-YARN-5355.02.patch, MAPREDUCE-6838-YARN-5355.03.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6838) [ATSv2 Security] Add timeline delegation token received in allocate response to UGI
[ https://issues.apache.org/jira/browse/MAPREDUCE-6838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16131811#comment-16131811 ] Rohith Sharma K S commented on MAPREDUCE-6838: -- thanks [~varun_saxena] for the patch. It seems patch do not apply, required to rebase. # Why are changing hadoop-common code to ignore ignoreUseIpConfig? Is this because to add test case? Does test is failing? # I just noticed that TimelineDelegationTokenIdentifier#Renewer has methods renew/cancel. These creates V1 client. Does it going to be a problem? # I think it is better to update token on each heart beat rather then comparing each time. It avoids proto compare. # This appears mixing collectorManager i.e NMCollectorService authentication and MR side. Is both required here or can we raise new JIRA? Was it conflicting with MR publisher ? One thing I observe is TimelineClient is published using login user which is doAS. I think we should publish is as owner of application otherwise we end up in writing data into sub app application table as well. > [ATSv2 Security] Add timeline delegation token received in allocate response > to UGI > --- > > Key: MAPREDUCE-6838 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6838 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: yarn-5355-merge-blocker > Fix For: YARN-5355 > > Attachments: MAPREDUCE-6838-YARN-5355.01.patch, > MAPREDUCE-6838-YARN-5355.02.patch, MAPREDUCE-6838-YARN-5355.03.patch, > MAPREDUCE-6838-YARN-5355.03.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6838) [ATSv2 Security] Add timeline delegation token received in allocate response to UGI
[ https://issues.apache.org/jira/browse/MAPREDUCE-6838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16128398#comment-16128398 ] Varun Saxena commented on MAPREDUCE-6838: - TestSecureMRTimelineEventHandling failure is strange. It passes for me everytime. Here, AM container is exiting with exit code 1. Ideallly LCE should be used but will be difficult to ensure a E2E test with it I guess. [~rohithsharma], can you simulate the failure? > [ATSv2 Security] Add timeline delegation token received in allocate response > to UGI > --- > > Key: MAPREDUCE-6838 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6838 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: yarn-5355-merge-blocker > Fix For: YARN-5355 > > Attachments: MAPREDUCE-6838-YARN-5355.01.patch, > MAPREDUCE-6838-YARN-5355.02.patch, MAPREDUCE-6838-YARN-5355.03.patch, > MAPREDUCE-6838-YARN-5355.03.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6838) [ATSv2 Security] Add timeline delegation token received in allocate response to UGI
[
https://issues.apache.org/jira/browse/MAPREDUCE-6838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16128154#comment-16128154
]
Hadoop QA commented on MAPREDUCE-6838:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 6 new or modified test
files. {color} |
|| || || || {color:brown} YARN-5355 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
36s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
8s{color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m
5s{color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
58s{color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m
12s{color} | {color:green} YARN-5355 passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
21s{color} | {color:red} hadoop-common-project/hadoop-common in YARN-5355 has
17 extant Findbugs warnings. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
2s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common in
YARN-5355 has 2 extant Findbugs warnings. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
52s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
in YARN-5355 has 5 extant Findbugs warnings. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
45s{color} | {color:red}
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app in
YARN-5355 has 3 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
3s{color} | {color:green} YARN-5355 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 12m
30s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 2m
0s{color} | {color:red} root: The patch generated 14 new + 212 unchanged - 5
fixed = 226 total (was 217) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
2s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 32s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
33s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 13m
10s{color} | {color:green}
[jira] [Commented] (MAPREDUCE-6838) [ATSv2 Security] Add timeline delegation token received in allocate response to UGI
[
https://issues.apache.org/jira/browse/MAPREDUCE-6838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16127842#comment-16127842
]
Hadoop QA commented on MAPREDUCE-6838:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 12m
11s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 6 new or modified test
files. {color} |
|| || || || {color:brown} YARN-5355 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
36s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m
49s{color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
33s{color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
41s{color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
33s{color} | {color:green} YARN-5355 passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
13s{color} | {color:red} hadoop-common-project/hadoop-common in YARN-5355 has
17 extant Findbugs warnings. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
55s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common in
YARN-5355 has 2 extant Findbugs warnings. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
40s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
in YARN-5355 has 5 extant Findbugs warnings. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
38s{color} | {color:red}
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app in
YARN-5355 has 3 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
24s{color} | {color:green} YARN-5355 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m
28s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
43s{color} | {color:red} root: The patch generated 14 new + 212 unchanged - 5
fixed = 226 total (was 217) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
42s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 6m 36s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
23s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 13m
0s{color} | {color:green}
[jira] [Commented] (MAPREDUCE-6838) [ATSv2 Security] Add timeline delegation token received in allocate response to UGI
[ https://issues.apache.org/jira/browse/MAPREDUCE-6838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16127518#comment-16127518 ] Varun Saxena commented on MAPREDUCE-6838: - I have added a E2E test case with security enabled, in the patch. This is to primarily check if token goes all the way to AM and used by AM to publish entities. Have used a single kerberos principal for all the components, based on the current user running the test. Moreover, in AM, we use the job config which may not have same values of hadoop.security.token.service.use_ip config as NM which generates the token. In our deployments, we will keep this config same across both client and NM end but not sure if we can make an assumption. So, for ATSv2, in DelegationTokenAuthenticatedURL I have passed a flag to indicate if we can ignore this config while looking for token in UGI. At the NM end, we would not use the config to generate token service as well. Thoughts? The patch depends on YARN-7006 so not submitting it. > [ATSv2 Security] Add timeline delegation token received in allocate response > to UGI > --- > > Key: MAPREDUCE-6838 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6838 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: yarn-5355-merge-blocker > Attachments: MAPREDUCE-6838-YARN-5355.01.patch, > MAPREDUCE-6838-YARN-5355.02.patch, MAPREDUCE-6838-YARN-5355.03.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6838) [ATSv2 Security] Add timeline delegation token received in allocate response to UGI
[
https://issues.apache.org/jira/browse/MAPREDUCE-6838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16109969#comment-16109969
]
Hadoop QA commented on MAPREDUCE-6838:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
11s{color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
22s{color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
25s{color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
18s{color} | {color:green} YARN-5355 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
36s{color} | {color:red}
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app in
YARN-5355 has 3 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
5s{color} | {color:green} hadoop-mapreduce-client-app in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 28m 1s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0ac17dc |
| JIRA Issue | MAPREDUCE-6838 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12879921/MAPREDUCE-6838-YARN-5355.02.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 739a836c8583 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | YARN-5355 / 3088cfc |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| findbugs |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7037/artifact/patchprocess/branch-findbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app-warnings.html
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7037/testReport/ |
| modules | C:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app U:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7037/console |
| Powered by | Apache Yetus 0.4.0 http://yetus.apache.org |
This message was automatically generated.
> [ATSv2 Security] Add timeline delegation
