[jira] [Updated] (MAPREDUCE-6838) [ATSv2 Security] Add timeline delegation token received in allocate response to UGI
[ https://issues.apache.org/jira/browse/MAPREDUCE-6838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vrushali C updated MAPREDUCE-6838: -- Fix Version/s: YARN-5355 > [ATSv2 Security] Add timeline delegation token received in allocate response > to UGI > --- > > Key: MAPREDUCE-6838 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6838 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: yarn-5355-merge-blocker > Fix For: YARN-5355 > > Attachments: MAPREDUCE-6838-YARN-5355.01.patch, > MAPREDUCE-6838-YARN-5355.02.patch, MAPREDUCE-6838-YARN-5355.03.patch, > MAPREDUCE-6838-YARN-5355.03.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6931) Remove TestDFSIO "Total Throughput" calculation
[
https://issues.apache.org/jira/browse/MAPREDUCE-6931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16128224#comment-16128224
]
ASF GitHub Bot commented on MAPREDUCE-6931:
---
Github user dennishuo commented on a diff in the pull request:
https://github.com/apache/hadoop/pull/259#discussion_r133348688
--- Diff:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/fs/TestDFSIO.java
---
@@ -855,7 +855,7 @@ public int run(String[] args) throws IOException {
long tStart = System.currentTimeMillis();
sequentialTest(fs, testType, nrBytes, nrFiles);
long execTime = System.currentTimeMillis() - tStart;
- String resultLine = "Seq Test exec time sec: " + (float)execTime /
1000;
+ String resultLine = "Seq Test exec time sec: " + msToSecs(execTime);
--- End diff --
Good suggestion, done.
> Remove TestDFSIO "Total Throughput" calculation
> ---
>
> Key: MAPREDUCE-6931
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6931
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: benchmarks, test
>Affects Versions: 2.8.0
>Reporter: Dennis Huo
>Priority: Trivial
>
> The new "Total Throughput" line added in
> https://issues.apache.org/jira/browse/HDFS-9153 is currently calculated as
> {{toMB(size) / ((float)execTime)}} and claims to be in units of "MB/s", but
> {{execTime}} is in milliseconds; thus, the reported number is 1/1000x the
> actual value:
> {code:java}
> String resultLines[] = {
> "- TestDFSIO - : " + testType,
> "Date & time: " + new Date(System.currentTimeMillis()),
> "Number of files: " + tasks,
> " Total MBytes processed: " + df.format(toMB(size)),
> " Throughput mb/sec: " + df.format(size * 1000.0 / (time *
> MEGA)),
> "Total Throughput mb/sec: " + df.format(toMB(size) /
> ((float)execTime)),
> " Average IO rate mb/sec: " + df.format(med),
> " IO rate std deviation: " + df.format(stdDev),
> " Test exec time sec: " + df.format((float)execTime / 1000),
> "" };
> {code}
> The different calculated fields can also use toMB and a shared
> milliseconds-to-seconds conversion to make it easier to keep units consistent.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6838) [ATSv2 Security] Add timeline delegation token received in allocate response to UGI
[
https://issues.apache.org/jira/browse/MAPREDUCE-6838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16128154#comment-16128154
]
Hadoop QA commented on MAPREDUCE-6838:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 6 new or modified test
files. {color} |
|| || || || {color:brown} YARN-5355 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
36s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
8s{color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m
5s{color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
58s{color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m
12s{color} | {color:green} YARN-5355 passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
21s{color} | {color:red} hadoop-common-project/hadoop-common in YARN-5355 has
17 extant Findbugs warnings. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
2s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common in
YARN-5355 has 2 extant Findbugs warnings. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
52s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
in YARN-5355 has 5 extant Findbugs warnings. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
45s{color} | {color:red}
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app in
YARN-5355 has 3 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
3s{color} | {color:green} YARN-5355 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 12m
30s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 2m
0s{color} | {color:red} root: The patch generated 14 new + 212 unchanged - 5
fixed = 226 total (was 217) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
2s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 32s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
33s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 13m
10s{color} | {color:green}
[jira] [Commented] (MAPREDUCE-6674) configure parallel tests for mapreduce-client-jobclient
[
https://issues.apache.org/jira/browse/MAPREDUCE-6674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16127901#comment-16127901
]
Hadoop QA commented on MAPREDUCE-6674:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 144 new or modified
test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
11s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 18s{color}
| {color:red}
hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient
generated 2 new + 95 unchanged - 48 fixed = 97 total (was 143) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
36s{color} | {color:red}
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient:
The patch generated 124 new + 2573 unchanged - 306 fixed = 2697 total (was
2879) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
9s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 30m 0s{color}
| {color:red} hadoop-mapreduce-client-jobclient in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 48m 40s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.mapred.TestLocalMRNotification |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | MAPREDUCE-6674 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12882006/MR.003.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit xml findbugs checkstyle |
| uname | Linux 7e625fd2d292 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / dadb0c2 |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| javac |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7073/artifact/patchprocess/diff-compile-javac-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
|
| checkstyle |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7073/artifact/patchprocess/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
|
| unit |
[jira] [Updated] (MAPREDUCE-6674) configure parallel tests for mapreduce-client-jobclient
[ https://issues.apache.org/jira/browse/MAPREDUCE-6674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated MAPREDUCE-6674: Attachment: MR.003.patch > configure parallel tests for mapreduce-client-jobclient > --- > > Key: MAPREDUCE-6674 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6674 > Project: Hadoop Map/Reduce > Issue Type: Test > Components: test >Affects Versions: 3.0.0-alpha1 >Reporter: Allen Wittenauer >Priority: Critical > Attachments: MR.001.patch, MR.002.patch, MR.003.patch > > > mapreduce-client-jobclient takes almost an hour and a half. Configuring > parallel-tests would greatly reduce this run time. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6838) [ATSv2 Security] Add timeline delegation token received in allocate response to UGI
[
https://issues.apache.org/jira/browse/MAPREDUCE-6838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16127842#comment-16127842
]
Hadoop QA commented on MAPREDUCE-6838:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 12m
11s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 6 new or modified test
files. {color} |
|| || || || {color:brown} YARN-5355 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
36s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m
49s{color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
33s{color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
41s{color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
33s{color} | {color:green} YARN-5355 passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
13s{color} | {color:red} hadoop-common-project/hadoop-common in YARN-5355 has
17 extant Findbugs warnings. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
55s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common in
YARN-5355 has 2 extant Findbugs warnings. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
40s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
in YARN-5355 has 5 extant Findbugs warnings. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
38s{color} | {color:red}
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app in
YARN-5355 has 3 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
24s{color} | {color:green} YARN-5355 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m
28s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
43s{color} | {color:red} root: The patch generated 14 new + 212 unchanged - 5
fixed = 226 total (was 217) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
42s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 6m 36s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
23s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 13m
0s{color} | {color:green}
[jira] [Commented] (MAPREDUCE-6931) Remove TestDFSIO "Total Throughput" calculation
[
https://issues.apache.org/jira/browse/MAPREDUCE-6931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16127781#comment-16127781
]
ASF GitHub Bot commented on MAPREDUCE-6931:
---
Github user clehene commented on a diff in the pull request:
https://github.com/apache/hadoop/pull/259#discussion_r133286014
--- Diff:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/fs/TestDFSIO.java
---
@@ -855,7 +855,7 @@ public int run(String[] args) throws IOException {
long tStart = System.currentTimeMillis();
sequentialTest(fs, testType, nrBytes, nrFiles);
long execTime = System.currentTimeMillis() - tStart;
- String resultLine = "Seq Test exec time sec: " + (float)execTime /
1000;
+ String resultLine = "Seq Test exec time sec: " + msToSecs(execTime);
--- End diff --
Perhaps rename variable to includes units too? E.g. `execTimeMs` ?
> Remove TestDFSIO "Total Throughput" calculation
> ---
>
> Key: MAPREDUCE-6931
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6931
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: benchmarks, test
>Affects Versions: 2.8.0
>Reporter: Dennis Huo
>Priority: Trivial
>
> The new "Total Throughput" line added in
> https://issues.apache.org/jira/browse/HDFS-9153 is currently calculated as
> {{toMB(size) / ((float)execTime)}} and claims to be in units of "MB/s", but
> {{execTime}} is in milliseconds; thus, the reported number is 1/1000x the
> actual value:
> {code:java}
> String resultLines[] = {
> "- TestDFSIO - : " + testType,
> "Date & time: " + new Date(System.currentTimeMillis()),
> "Number of files: " + tasks,
> " Total MBytes processed: " + df.format(toMB(size)),
> " Throughput mb/sec: " + df.format(size * 1000.0 / (time *
> MEGA)),
> "Total Throughput mb/sec: " + df.format(toMB(size) /
> ((float)execTime)),
> " Average IO rate mb/sec: " + df.format(med),
> " IO rate std deviation: " + df.format(stdDev),
> " Test exec time sec: " + df.format((float)execTime / 1000),
> "" };
> {code}
> The different calculated fields can also use toMB and a shared
> milliseconds-to-seconds conversion to make it easier to keep units consistent.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6838) [ATSv2 Security] Add timeline delegation token received in allocate response to UGI
[ https://issues.apache.org/jira/browse/MAPREDUCE-6838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated MAPREDUCE-6838: Attachment: MAPREDUCE-6838-YARN-5355.03.patch Thought that I will attach a patch which contains the previous patch and YARN-7006 patch as well so that QA can run. I will delete the patch once YARN-7006 is committed. > [ATSv2 Security] Add timeline delegation token received in allocate response > to UGI > --- > > Key: MAPREDUCE-6838 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6838 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: yarn-5355-merge-blocker > Attachments: MAPREDUCE-6838-YARN-5355.01.patch, > MAPREDUCE-6838-YARN-5355.02.patch, MAPREDUCE-6838-YARN-5355.03.patch, > MAPREDUCE-6838-YARN-5355.03.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6838) [ATSv2 Security] Add timeline delegation token received in allocate response to UGI
[ https://issues.apache.org/jira/browse/MAPREDUCE-6838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated MAPREDUCE-6838: Status: Patch Available (was: In Progress) > [ATSv2 Security] Add timeline delegation token received in allocate response > to UGI > --- > > Key: MAPREDUCE-6838 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6838 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: yarn-5355-merge-blocker > Attachments: MAPREDUCE-6838-YARN-5355.01.patch, > MAPREDUCE-6838-YARN-5355.02.patch, MAPREDUCE-6838-YARN-5355.03.patch, > MAPREDUCE-6838-YARN-5355.03.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Work started] (MAPREDUCE-6838) [ATSv2 Security] Add timeline delegation token received in allocate response to UGI
[ https://issues.apache.org/jira/browse/MAPREDUCE-6838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on MAPREDUCE-6838 started by Varun Saxena. --- > [ATSv2 Security] Add timeline delegation token received in allocate response > to UGI > --- > > Key: MAPREDUCE-6838 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6838 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: yarn-5355-merge-blocker > Attachments: MAPREDUCE-6838-YARN-5355.01.patch, > MAPREDUCE-6838-YARN-5355.02.patch, MAPREDUCE-6838-YARN-5355.03.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6838) [ATSv2 Security] Add timeline delegation token received in allocate response to UGI
[ https://issues.apache.org/jira/browse/MAPREDUCE-6838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16127518#comment-16127518 ] Varun Saxena commented on MAPREDUCE-6838: - I have added a E2E test case with security enabled, in the patch. This is to primarily check if token goes all the way to AM and used by AM to publish entities. Have used a single kerberos principal for all the components, based on the current user running the test. Moreover, in AM, we use the job config which may not have same values of hadoop.security.token.service.use_ip config as NM which generates the token. In our deployments, we will keep this config same across both client and NM end but not sure if we can make an assumption. So, for ATSv2, in DelegationTokenAuthenticatedURL I have passed a flag to indicate if we can ignore this config while looking for token in UGI. At the NM end, we would not use the config to generate token service as well. Thoughts? The patch depends on YARN-7006 so not submitting it. > [ATSv2 Security] Add timeline delegation token received in allocate response > to UGI > --- > > Key: MAPREDUCE-6838 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6838 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: yarn-5355-merge-blocker > Attachments: MAPREDUCE-6838-YARN-5355.01.patch, > MAPREDUCE-6838-YARN-5355.02.patch, MAPREDUCE-6838-YARN-5355.03.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Created] (MAPREDUCE-6939) Follow-up on MAPREDUCE-6870
Peter Bacsko created MAPREDUCE-6939:
---
Summary: Follow-up on MAPREDUCE-6870
Key: MAPREDUCE-6939
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6939
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: client
Reporter: Peter Bacsko
Assignee: Peter Bacsko
Priority: Minor
Some minor changes should be made after MAPREDUCE-6870 was committed upstream:
1. Fix JavaDoc in {{JobImpl.java}}
2. Correct the description of the method, that is, it might not be entirely
clear what the "improvement" is or what it really improves
3. Small typo in the name of the new testcase
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Resolved] (MAPREDUCE-6938) Question
[
https://issues.apache.org/jira/browse/MAPREDUCE-6938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rushabh S Shah resolved MAPREDUCE-6938.
---
Resolution: Invalid
[~remil] This jira board is for bug/improvement/feature tracking system not for
asking some random questions/programs.
Please send an email to {{gene...@hadoop.apache.org}} or
{{u...@hadoop.apache.org}} and hope someone would reply.
Thanks,
Rushabh Shah.
> Question
>
>
> Key: MAPREDUCE-6938
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6938
> Project: Hadoop Map/Reduce
> Issue Type: Task
>Reporter: Remil
>Priority: Minor
>
> I need 2 helps.
> 1) need a Java map reducer sample program where multiple parameters
> are passed from mapper to reducer.
> 2) need a Java map reducer program where there is a write to a file inside
> hdfs filesystem as well as a read from a file inside hdfs other than
> the normal input file and output file mentioned in the mapper and reducer.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6838) [ATSv2 Security] Add timeline delegation token received in allocate response to UGI
[ https://issues.apache.org/jira/browse/MAPREDUCE-6838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated MAPREDUCE-6838: Attachment: MAPREDUCE-6838-YARN-5355.03.patch > [ATSv2 Security] Add timeline delegation token received in allocate response > to UGI > --- > > Key: MAPREDUCE-6838 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6838 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: yarn-5355-merge-blocker > Attachments: MAPREDUCE-6838-YARN-5355.01.patch, > MAPREDUCE-6838-YARN-5355.02.patch, MAPREDUCE-6838-YARN-5355.03.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6838) [ATSv2 Security] Add timeline delegation token received in allocate response to UGI
[ https://issues.apache.org/jira/browse/MAPREDUCE-6838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated MAPREDUCE-6838: Attachment: (was: MAPREDUCE-6838-YARN-5355.03.patch) > [ATSv2 Security] Add timeline delegation token received in allocate response > to UGI > --- > > Key: MAPREDUCE-6838 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6838 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: yarn-5355-merge-blocker > Attachments: MAPREDUCE-6838-YARN-5355.01.patch, > MAPREDUCE-6838-YARN-5355.02.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6838) [ATSv2 Security] Add timeline delegation token received in allocate response to UGI
[ https://issues.apache.org/jira/browse/MAPREDUCE-6838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated MAPREDUCE-6838: Status: Open (was: Patch Available) > [ATSv2 Security] Add timeline delegation token received in allocate response > to UGI > --- > > Key: MAPREDUCE-6838 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6838 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: yarn-5355-merge-blocker > Attachments: MAPREDUCE-6838-YARN-5355.01.patch, > MAPREDUCE-6838-YARN-5355.02.patch, MAPREDUCE-6838-YARN-5355.03.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6838) [ATSv2 Security] Add timeline delegation token received in allocate response to UGI
[ https://issues.apache.org/jira/browse/MAPREDUCE-6838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated MAPREDUCE-6838: Attachment: MAPREDUCE-6838-YARN-5355.03.patch > [ATSv2 Security] Add timeline delegation token received in allocate response > to UGI > --- > > Key: MAPREDUCE-6838 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6838 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: yarn-5355-merge-blocker > Attachments: MAPREDUCE-6838-YARN-5355.01.patch, > MAPREDUCE-6838-YARN-5355.02.patch, MAPREDUCE-6838-YARN-5355.03.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Created] (MAPREDUCE-6938) Question
Remil created MAPREDUCE-6938: Summary: Question Key: MAPREDUCE-6938 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6938 Project: Hadoop Map/Reduce Issue Type: Task Reporter: Remil Priority: Minor I need 2 helps. 1) need a Java map reducer sample program where multiple parameters are passed from mapper to reducer. 2) need a Java map reducer program where there is a write to a file inside hdfs filesystem as well as a read from a file inside hdfs other than the normal input file and output file mentioned in the mapper and reducer. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6931) Remove TestDFSIO "Total Throughput" calculation
[
https://issues.apache.org/jira/browse/MAPREDUCE-6931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16126935#comment-16126935
]
Konstantin Shvachko commented on MAPREDUCE-6931:
Ah yes, missed the refactoring.
+1 on the patch
> Remove TestDFSIO "Total Throughput" calculation
> ---
>
> Key: MAPREDUCE-6931
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6931
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: benchmarks, test
>Affects Versions: 2.8.0
>Reporter: Dennis Huo
>Priority: Trivial
>
> The new "Total Throughput" line added in
> https://issues.apache.org/jira/browse/HDFS-9153 is currently calculated as
> {{toMB(size) / ((float)execTime)}} and claims to be in units of "MB/s", but
> {{execTime}} is in milliseconds; thus, the reported number is 1/1000x the
> actual value:
> {code:java}
> String resultLines[] = {
> "- TestDFSIO - : " + testType,
> "Date & time: " + new Date(System.currentTimeMillis()),
> "Number of files: " + tasks,
> " Total MBytes processed: " + df.format(toMB(size)),
> " Throughput mb/sec: " + df.format(size * 1000.0 / (time *
> MEGA)),
> "Total Throughput mb/sec: " + df.format(toMB(size) /
> ((float)execTime)),
> " Average IO rate mb/sec: " + df.format(med),
> " IO rate std deviation: " + df.format(stdDev),
> " Test exec time sec: " + df.format((float)execTime / 1000),
> "" };
> {code}
> The different calculated fields can also use toMB and a shared
> milliseconds-to-seconds conversion to make it easier to keep units consistent.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6674) configure parallel tests for mapreduce-client-jobclient
[
https://issues.apache.org/jira/browse/MAPREDUCE-6674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16126857#comment-16126857
]
Hadoop QA commented on MAPREDUCE-6674:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
32s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 144 new or modified
test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 25m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
26s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 42s{color}
| {color:red}
hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient
generated 2 new + 96 unchanged - 48 fixed = 98 total (was 144) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
41s{color} | {color:red}
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient:
The patch generated 124 new + 2574 unchanged - 305 fixed = 2698 total (was
2879) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
3s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 49m 12s{color}
| {color:red} hadoop-mapreduce-client-jobclient in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
39s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 88m 58s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.mapreduce.lib.join.TestJoinProperties |
| | hadoop.hdfs.TestNNBench |
| | hadoop.mapred.TestLocalMRNotification |
| Timed out junit tests | org.apache.hadoop.mapreduce.v2.TestUberAM |
| | org.apache.hadoop.mapreduce.v2.TestMRJobs |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | MAPREDUCE-6674 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12881852/MR.002.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit xml findbugs checkstyle |
| uname | Linux 0d66da470aa8 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7
11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 4d7be1d |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| javac |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7071/artifact/patchprocess/diff-compile-javac-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
|
| checkstyle |
