[jira] [Commented] (MAPREDUCE-7077) Pipe mapreduce job fails with Permission denied for jobTokenPassword
[
https://issues.apache.org/jira/browse/MAPREDUCE-7077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16434916#comment-16434916
]
Hadoop QA commented on MAPREDUCE-7077:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
53s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 54s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
43s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 16s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
49s{color} | {color:green} hadoop-mapreduce-client-core in the patch passed.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}134m 18s{color}
| {color:red} hadoop-mapreduce-client-jobclient in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
24s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}194m 55s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8620d2b |
| JIRA Issue | MAPREDUCE-7077 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12918654/MAPREDUCE-7077.01.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 4a708150e7dc 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 0d898b7 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_162 |
| findbugs | v3.1.0-RC1 |
| unit |
[jira] [Commented] (MAPREDUCE-7077) Pipe mapreduce job fails with Permission denied for jobTokenPassword
[
https://issues.apache.org/jira/browse/MAPREDUCE-7077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16434806#comment-16434806
]
Takanobu Asanuma commented on MAPREDUCE-7077:
-
+1 (non-binding) pending Jenkins.
> Pipe mapreduce job fails with Permission denied for jobTokenPassword
>
>
> Key: MAPREDUCE-7077
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7077
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 3.1.0
>Reporter: Yesha Vora
>Assignee: Akira Ajisaka
>Priority: Critical
> Attachments: MAPREDUCE-7077.01.patch
>
>
> Steps:
> Launch wordcount example with pipe
> {code}
> /usr/hdp/current/hadoop-client/bin/hadoop pipes
> "-Dhadoop.pipes.java.recordreader=true"
> "-Dhadoop.pipes.java.recordwriter=true" -input pipeInput -output pipeOutput
> -program bin/wordcount{code}
> The application fails with below stacktrace
> {code:title=AM}
> attempt_1517534613368_0041_r_00_2 is : 0.0
> 2018-02-02 02:40:51,071 ERROR [IPC Server handler 16 on 43391]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Task:
> attempt_1517534613368_0041_r_00_2 - exited :
> java.io.FileNotFoundException:
> /grid/0/hadoop/yarn/local/usercache/hrt_qa/appcache/application_1517534613368_0041/jobTokenPassword
> (Permission denied)
> at java.io.FileOutputStream.open0(Native Method)
> at java.io.FileOutputStream.open(FileOutputStream.java:270)
> at java.io.FileOutputStream.(FileOutputStream.java:213)
> at
> org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.(RawLocalFileSystem.java:236)
> at
> org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.(RawLocalFileSystem.java:219)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.createOutputStreamWithMode(RawLocalFileSystem.java:318)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:307)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:338)
> at
> org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.(ChecksumFileSystem.java:401)
> at
> org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:464)
> at
> org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:443)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1169)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1149)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1038)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1026)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:703)
> at
> org.apache.hadoop.mapred.pipes.Application.writePasswordToLocalFile(Application.java:173)
> at org.apache.hadoop.mapred.pipes.Application.(Application.java:109)
> at
> org.apache.hadoop.mapred.pipes.PipesReducer.startApplication(PipesReducer.java:87)
> at org.apache.hadoop.mapred.pipes.PipesReducer.reduce(PipesReducer.java:65)
> at org.apache.hadoop.mapred.pipes.PipesReducer.reduce(PipesReducer.java:38)
> at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:445)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:393)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1965)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7036) ASF License warning in hadoop-mapreduce-client
[
https://issues.apache.org/jira/browse/MAPREDUCE-7036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16434797#comment-16434797
]
Takanobu Asanuma commented on MAPREDUCE-7036:
-
Sorry for creating the bug, and thanks for catching it.
> ASF License warning in hadoop-mapreduce-client
> --
>
> Key: MAPREDUCE-7036
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7036
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Reporter: Takanobu Asanuma
>Assignee: Takanobu Asanuma
>Priority: Minor
> Fix For: 3.1.0
>
> Attachments: MAPREDUCE-7036.1.patch
>
>
> it occurred in MAPREDUCE-7021 and MAPREDUCE-7034.
> {noformat}
> Lines that start with ? in the ASF License report indicate files that do
> not have an Apache license header: !?
> /testptch/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/jobTokenPassword
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Resolved] (MAPREDUCE-7036) ASF License warning in hadoop-mapreduce-client
[
https://issues.apache.org/jira/browse/MAPREDUCE-7036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka resolved MAPREDUCE-7036.
--
Resolution: Fixed
bq. We should keep this ticket fixed and open a new JIRA.
Okay, fixed.
> ASF License warning in hadoop-mapreduce-client
> --
>
> Key: MAPREDUCE-7036
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7036
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Reporter: Takanobu Asanuma
>Assignee: Takanobu Asanuma
>Priority: Minor
> Fix For: 3.1.0
>
> Attachments: MAPREDUCE-7036.1.patch
>
>
> it occurred in MAPREDUCE-7021 and MAPREDUCE-7034.
> {noformat}
> Lines that start with ? in the ASF License report indicate files that do
> not have an Apache license header: !?
> /testptch/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/jobTokenPassword
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7077) Pipe mapreduce job fails with Permission denied for jobTokenPassword
[
https://issues.apache.org/jira/browse/MAPREDUCE-7077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16434782#comment-16434782
]
Akira Ajisaka commented on MAPREDUCE-7077:
--
01 patch: revert MAPREDUCE-7036
> Pipe mapreduce job fails with Permission denied for jobTokenPassword
>
>
> Key: MAPREDUCE-7077
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7077
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 3.1.0
>Reporter: Yesha Vora
>Assignee: Akira Ajisaka
>Priority: Critical
> Attachments: MAPREDUCE-7077.01.patch
>
>
> Steps:
> Launch wordcount example with pipe
> {code}
> /usr/hdp/current/hadoop-client/bin/hadoop pipes
> "-Dhadoop.pipes.java.recordreader=true"
> "-Dhadoop.pipes.java.recordwriter=true" -input pipeInput -output pipeOutput
> -program bin/wordcount{code}
> The application fails with below stacktrace
> {code:title=AM}
> attempt_1517534613368_0041_r_00_2 is : 0.0
> 2018-02-02 02:40:51,071 ERROR [IPC Server handler 16 on 43391]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Task:
> attempt_1517534613368_0041_r_00_2 - exited :
> java.io.FileNotFoundException:
> /grid/0/hadoop/yarn/local/usercache/hrt_qa/appcache/application_1517534613368_0041/jobTokenPassword
> (Permission denied)
> at java.io.FileOutputStream.open0(Native Method)
> at java.io.FileOutputStream.open(FileOutputStream.java:270)
> at java.io.FileOutputStream.(FileOutputStream.java:213)
> at
> org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.(RawLocalFileSystem.java:236)
> at
> org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.(RawLocalFileSystem.java:219)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.createOutputStreamWithMode(RawLocalFileSystem.java:318)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:307)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:338)
> at
> org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.(ChecksumFileSystem.java:401)
> at
> org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:464)
> at
> org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:443)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1169)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1149)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1038)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1026)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:703)
> at
> org.apache.hadoop.mapred.pipes.Application.writePasswordToLocalFile(Application.java:173)
> at org.apache.hadoop.mapred.pipes.Application.(Application.java:109)
> at
> org.apache.hadoop.mapred.pipes.PipesReducer.startApplication(PipesReducer.java:87)
> at org.apache.hadoop.mapred.pipes.PipesReducer.reduce(PipesReducer.java:65)
> at org.apache.hadoop.mapred.pipes.PipesReducer.reduce(PipesReducer.java:38)
> at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:445)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:393)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1965)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7077) Pipe mapreduce job fails with Permission denied for jobTokenPassword
[
https://issues.apache.org/jira/browse/MAPREDUCE-7077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka updated MAPREDUCE-7077:
-
Attachment: MAPREDUCE-7077.01.patch
> Pipe mapreduce job fails with Permission denied for jobTokenPassword
>
>
> Key: MAPREDUCE-7077
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7077
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 3.1.0
>Reporter: Yesha Vora
>Priority: Major
> Attachments: MAPREDUCE-7077.01.patch
>
>
> Steps:
> Launch wordcount example with pipe
> {code}
> /usr/hdp/current/hadoop-client/bin/hadoop pipes
> "-Dhadoop.pipes.java.recordreader=true"
> "-Dhadoop.pipes.java.recordwriter=true" -input pipeInput -output pipeOutput
> -program bin/wordcount{code}
> The application fails with below stacktrace
> {code:title=AM}
> attempt_1517534613368_0041_r_00_2 is : 0.0
> 2018-02-02 02:40:51,071 ERROR [IPC Server handler 16 on 43391]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Task:
> attempt_1517534613368_0041_r_00_2 - exited :
> java.io.FileNotFoundException:
> /grid/0/hadoop/yarn/local/usercache/hrt_qa/appcache/application_1517534613368_0041/jobTokenPassword
> (Permission denied)
> at java.io.FileOutputStream.open0(Native Method)
> at java.io.FileOutputStream.open(FileOutputStream.java:270)
> at java.io.FileOutputStream.(FileOutputStream.java:213)
> at
> org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.(RawLocalFileSystem.java:236)
> at
> org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.(RawLocalFileSystem.java:219)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.createOutputStreamWithMode(RawLocalFileSystem.java:318)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:307)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:338)
> at
> org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.(ChecksumFileSystem.java:401)
> at
> org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:464)
> at
> org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:443)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1169)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1149)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1038)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1026)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:703)
> at
> org.apache.hadoop.mapred.pipes.Application.writePasswordToLocalFile(Application.java:173)
> at org.apache.hadoop.mapred.pipes.Application.(Application.java:109)
> at
> org.apache.hadoop.mapred.pipes.PipesReducer.startApplication(PipesReducer.java:87)
> at org.apache.hadoop.mapred.pipes.PipesReducer.reduce(PipesReducer.java:65)
> at org.apache.hadoop.mapred.pipes.PipesReducer.reduce(PipesReducer.java:38)
> at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:445)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:393)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1965)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7077) Pipe mapreduce job fails with Permission denied for jobTokenPassword
[
https://issues.apache.org/jira/browse/MAPREDUCE-7077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka updated MAPREDUCE-7077:
-
Assignee: Akira Ajisaka
Target Version/s: 3.2.0, 3.1.1
Status: Patch Available (was: Open)
> Pipe mapreduce job fails with Permission denied for jobTokenPassword
>
>
> Key: MAPREDUCE-7077
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7077
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 3.1.0
>Reporter: Yesha Vora
>Assignee: Akira Ajisaka
>Priority: Major
> Attachments: MAPREDUCE-7077.01.patch
>
>
> Steps:
> Launch wordcount example with pipe
> {code}
> /usr/hdp/current/hadoop-client/bin/hadoop pipes
> "-Dhadoop.pipes.java.recordreader=true"
> "-Dhadoop.pipes.java.recordwriter=true" -input pipeInput -output pipeOutput
> -program bin/wordcount{code}
> The application fails with below stacktrace
> {code:title=AM}
> attempt_1517534613368_0041_r_00_2 is : 0.0
> 2018-02-02 02:40:51,071 ERROR [IPC Server handler 16 on 43391]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Task:
> attempt_1517534613368_0041_r_00_2 - exited :
> java.io.FileNotFoundException:
> /grid/0/hadoop/yarn/local/usercache/hrt_qa/appcache/application_1517534613368_0041/jobTokenPassword
> (Permission denied)
> at java.io.FileOutputStream.open0(Native Method)
> at java.io.FileOutputStream.open(FileOutputStream.java:270)
> at java.io.FileOutputStream.(FileOutputStream.java:213)
> at
> org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.(RawLocalFileSystem.java:236)
> at
> org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.(RawLocalFileSystem.java:219)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.createOutputStreamWithMode(RawLocalFileSystem.java:318)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:307)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:338)
> at
> org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.(ChecksumFileSystem.java:401)
> at
> org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:464)
> at
> org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:443)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1169)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1149)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1038)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1026)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:703)
> at
> org.apache.hadoop.mapred.pipes.Application.writePasswordToLocalFile(Application.java:173)
> at org.apache.hadoop.mapred.pipes.Application.(Application.java:109)
> at
> org.apache.hadoop.mapred.pipes.PipesReducer.startApplication(PipesReducer.java:87)
> at org.apache.hadoop.mapred.pipes.PipesReducer.reduce(PipesReducer.java:65)
> at org.apache.hadoop.mapred.pipes.PipesReducer.reduce(PipesReducer.java:38)
> at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:445)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:393)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1965)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7077) Pipe mapreduce job fails with Permission denied for jobTokenPassword
[
https://issues.apache.org/jira/browse/MAPREDUCE-7077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka updated MAPREDUCE-7077:
-
Priority: Critical (was: Major)
> Pipe mapreduce job fails with Permission denied for jobTokenPassword
>
>
> Key: MAPREDUCE-7077
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7077
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 3.1.0
>Reporter: Yesha Vora
>Assignee: Akira Ajisaka
>Priority: Critical
> Attachments: MAPREDUCE-7077.01.patch
>
>
> Steps:
> Launch wordcount example with pipe
> {code}
> /usr/hdp/current/hadoop-client/bin/hadoop pipes
> "-Dhadoop.pipes.java.recordreader=true"
> "-Dhadoop.pipes.java.recordwriter=true" -input pipeInput -output pipeOutput
> -program bin/wordcount{code}
> The application fails with below stacktrace
> {code:title=AM}
> attempt_1517534613368_0041_r_00_2 is : 0.0
> 2018-02-02 02:40:51,071 ERROR [IPC Server handler 16 on 43391]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Task:
> attempt_1517534613368_0041_r_00_2 - exited :
> java.io.FileNotFoundException:
> /grid/0/hadoop/yarn/local/usercache/hrt_qa/appcache/application_1517534613368_0041/jobTokenPassword
> (Permission denied)
> at java.io.FileOutputStream.open0(Native Method)
> at java.io.FileOutputStream.open(FileOutputStream.java:270)
> at java.io.FileOutputStream.(FileOutputStream.java:213)
> at
> org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.(RawLocalFileSystem.java:236)
> at
> org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.(RawLocalFileSystem.java:219)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.createOutputStreamWithMode(RawLocalFileSystem.java:318)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:307)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:338)
> at
> org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.(ChecksumFileSystem.java:401)
> at
> org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:464)
> at
> org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:443)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1169)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1149)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1038)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1026)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:703)
> at
> org.apache.hadoop.mapred.pipes.Application.writePasswordToLocalFile(Application.java:173)
> at org.apache.hadoop.mapred.pipes.Application.(Application.java:109)
> at
> org.apache.hadoop.mapred.pipes.PipesReducer.startApplication(PipesReducer.java:87)
> at org.apache.hadoop.mapred.pipes.PipesReducer.reduce(PipesReducer.java:65)
> at org.apache.hadoop.mapred.pipes.PipesReducer.reduce(PipesReducer.java:38)
> at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:445)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:393)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1965)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7036) ASF License warning in hadoop-mapreduce-client
[
https://issues.apache.org/jira/browse/MAPREDUCE-7036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16434779#comment-16434779
]
Akira Ajisaka commented on MAPREDUCE-7036:
--
Sorry for breaking MapReduce Pipes. I'm +1 for reverting this in MAPREDUCE-7077
> ASF License warning in hadoop-mapreduce-client
> --
>
> Key: MAPREDUCE-7036
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7036
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Reporter: Takanobu Asanuma
>Assignee: Takanobu Asanuma
>Priority: Minor
> Fix For: 3.1.0
>
> Attachments: MAPREDUCE-7036.1.patch
>
>
> it occurred in MAPREDUCE-7021 and MAPREDUCE-7034.
> {noformat}
> Lines that start with ? in the ASF License report indicate files that do
> not have an Apache license header: !?
> /testptch/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/jobTokenPassword
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7077) Pipe mapreduce job fails with Permission denied for jobTokenPassword
[
https://issues.apache.org/jira/browse/MAPREDUCE-7077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Suma Shivaprasad updated MAPREDUCE-7077:
Affects Version/s: 3.1.0
> Pipe mapreduce job fails with Permission denied for jobTokenPassword
>
>
> Key: MAPREDUCE-7077
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7077
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 3.1.0
>Reporter: Yesha Vora
>Priority: Major
>
> Steps:
> Launch wordcount example with pipe
> {code}
> /usr/hdp/current/hadoop-client/bin/hadoop pipes
> "-Dhadoop.pipes.java.recordreader=true"
> "-Dhadoop.pipes.java.recordwriter=true" -input pipeInput -output pipeOutput
> -program bin/wordcount{code}
> The application fails with below stacktrace
> {code:title=AM}
> attempt_1517534613368_0041_r_00_2 is : 0.0
> 2018-02-02 02:40:51,071 ERROR [IPC Server handler 16 on 43391]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Task:
> attempt_1517534613368_0041_r_00_2 - exited :
> java.io.FileNotFoundException:
> /grid/0/hadoop/yarn/local/usercache/hrt_qa/appcache/application_1517534613368_0041/jobTokenPassword
> (Permission denied)
> at java.io.FileOutputStream.open0(Native Method)
> at java.io.FileOutputStream.open(FileOutputStream.java:270)
> at java.io.FileOutputStream.(FileOutputStream.java:213)
> at
> org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.(RawLocalFileSystem.java:236)
> at
> org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.(RawLocalFileSystem.java:219)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.createOutputStreamWithMode(RawLocalFileSystem.java:318)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:307)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:338)
> at
> org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.(ChecksumFileSystem.java:401)
> at
> org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:464)
> at
> org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:443)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1169)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1149)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1038)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1026)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:703)
> at
> org.apache.hadoop.mapred.pipes.Application.writePasswordToLocalFile(Application.java:173)
> at org.apache.hadoop.mapred.pipes.Application.(Application.java:109)
> at
> org.apache.hadoop.mapred.pipes.PipesReducer.startApplication(PipesReducer.java:87)
> at org.apache.hadoop.mapred.pipes.PipesReducer.reduce(PipesReducer.java:65)
> at org.apache.hadoop.mapred.pipes.PipesReducer.reduce(PipesReducer.java:38)
> at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:445)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:393)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1965)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (MAPREDUCE-7036) ASF License warning in hadoop-mapreduce-client
[
https://issues.apache.org/jira/browse/MAPREDUCE-7036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16434765#comment-16434765
]
Suma Shivaprasad edited comment on MAPREDUCE-7036 at 4/12/18 12:36 AM:
---
Thanks [~vinodkv] [~leftnoteasy] A new Jira has been filed here - MAPREDUCE-7077
was (Author: suma.shivaprasad):
Thanks [~vinodkv] [~leftnoteasy] A new Jira has been filed here -
https://issues.apache.org/jira/browse/MAPREDUCE-7077
> ASF License warning in hadoop-mapreduce-client
> --
>
> Key: MAPREDUCE-7036
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7036
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Reporter: Takanobu Asanuma
>Assignee: Takanobu Asanuma
>Priority: Minor
> Fix For: 3.1.0
>
> Attachments: MAPREDUCE-7036.1.patch
>
>
> it occurred in MAPREDUCE-7021 and MAPREDUCE-7034.
> {noformat}
> Lines that start with ? in the ASF License report indicate files that do
> not have an Apache license header: !?
> /testptch/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/jobTokenPassword
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7036) ASF License warning in hadoop-mapreduce-client
[
https://issues.apache.org/jira/browse/MAPREDUCE-7036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16434765#comment-16434765
]
Suma Shivaprasad commented on MAPREDUCE-7036:
-
Thanks [~vinodkv] [~leftnoteasy] A new Jira has been filed here -
https://issues.apache.org/jira/browse/MAPREDUCE-7077
> ASF License warning in hadoop-mapreduce-client
> --
>
> Key: MAPREDUCE-7036
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7036
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Reporter: Takanobu Asanuma
>Assignee: Takanobu Asanuma
>Priority: Minor
> Fix For: 3.1.0
>
> Attachments: MAPREDUCE-7036.1.patch
>
>
> it occurred in MAPREDUCE-7021 and MAPREDUCE-7034.
> {noformat}
> Lines that start with ? in the ASF License report indicate files that do
> not have an Apache license header: !?
> /testptch/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/jobTokenPassword
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7036) ASF License warning in hadoop-mapreduce-client
[
https://issues.apache.org/jira/browse/MAPREDUCE-7036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16434695#comment-16434695
]
Vinod Kumar Vavilapalli commented on MAPREDUCE-7036:
Just realized reopening this doesn't make sense since 3.1.0 is already
released. We should keep this ticket fixed and open a new JIRA.
> ASF License warning in hadoop-mapreduce-client
> --
>
> Key: MAPREDUCE-7036
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7036
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Reporter: Takanobu Asanuma
>Assignee: Takanobu Asanuma
>Priority: Minor
> Fix For: 3.1.0
>
> Attachments: MAPREDUCE-7036.1.patch
>
>
> it occurred in MAPREDUCE-7021 and MAPREDUCE-7034.
> {noformat}
> Lines that start with ? in the ASF License report indicate files that do
> not have an Apache license header: !?
> /testptch/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/jobTokenPassword
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7042) Killed mapreduce job data does not move to mapreduce.jobhistory.done-dir when ATS v2 is enabled
[
https://issues.apache.org/jira/browse/MAPREDUCE-7042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16434657#comment-16434657
]
Xuan Gong commented on MAPREDUCE-7042:
--
The goal is that we handle the JHS event and ATS event separately. So if there
is anything wrong while we handle the ats event, it should not block the
process of handling JHS event.
The solution we used here is:
* create an additional queue for the timeline event
* In JHSEventHandlingThread, we simply put the timeline event to this
additional timeline event queue.
* Create a separate eventhandlingThread to handle the timeline event
[~rohithsharma] Could you review this, please?
> Killed mapreduce job data does not move to mapreduce.jobhistory.done-dir when
> ATS v2 is enabled
> ---
>
> Key: MAPREDUCE-7042
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7042
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Yesha Vora
>Assignee: Xuan Gong
>Priority: Major
> Attachments: MAPREDUCE-7042.1.patch
>
>
> Steps:
> 1) Start a mapreduce job
> {code}
> hadoop jar
> /usr/hdp/current/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient-3.0.0.3.0.0.0-751-tests.jar
> sleep "-Dmapreduce.job.user.name=hrt_qa" -m 10 -r 1 -mt 1000 -rt
> 1000{code}
> 2) kill job
> 3) Validate job is present at mapreduce.jobhistory.done-dir
> mapreduce.jobhistory.done-dir dir does not have job_1516776025831_0048 entry.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7042) Killed mapreduce job data does not move to mapreduce.jobhistory.done-dir when ATS v2 is enabled
[
https://issues.apache.org/jira/browse/MAPREDUCE-7042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xuan Gong updated MAPREDUCE-7042:
-
Attachment: MAPREDUCE-7042.1.patch
> Killed mapreduce job data does not move to mapreduce.jobhistory.done-dir when
> ATS v2 is enabled
> ---
>
> Key: MAPREDUCE-7042
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7042
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Yesha Vora
>Assignee: Xuan Gong
>Priority: Major
> Attachments: MAPREDUCE-7042.1.patch
>
>
> Steps:
> 1) Start a mapreduce job
> {code}
> hadoop jar
> /usr/hdp/current/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient-3.0.0.3.0.0.0-751-tests.jar
> sleep "-Dmapreduce.job.user.name=hrt_qa" -m 10 -r 1 -mt 1000 -rt
> 1000{code}
> 2) kill job
> 3) Validate job is present at mapreduce.jobhistory.done-dir
> mapreduce.jobhistory.done-dir dir does not have job_1516776025831_0048 entry.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Work started] (MAPREDUCE-7042) Killed mapreduce job data does not move to mapreduce.jobhistory.done-dir when ATS v2 is enabled
[
https://issues.apache.org/jira/browse/MAPREDUCE-7042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on MAPREDUCE-7042 started by Xuan Gong.
> Killed mapreduce job data does not move to mapreduce.jobhistory.done-dir when
> ATS v2 is enabled
> ---
>
> Key: MAPREDUCE-7042
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7042
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Yesha Vora
>Assignee: Xuan Gong
>Priority: Major
> Attachments: MAPREDUCE-7042.1.patch
>
>
> Steps:
> 1) Start a mapreduce job
> {code}
> hadoop jar
> /usr/hdp/current/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient-3.0.0.3.0.0.0-751-tests.jar
> sleep "-Dmapreduce.job.user.name=hrt_qa" -m 10 -r 1 -mt 1000 -rt
> 1000{code}
> 2) kill job
> 3) Validate job is present at mapreduce.jobhistory.done-dir
> mapreduce.jobhistory.done-dir dir does not have job_1516776025831_0048 entry.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7069) Add ability to specify user environment variables individually
[
https://issues.apache.org/jira/browse/MAPREDUCE-7069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16434641#comment-16434641
]
Jim Brennan commented on MAPREDUCE-7069:
[~jlowe], unit test is unrelated. This is ready for review.
> Add ability to specify user environment variables individually
> --
>
> Key: MAPREDUCE-7069
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7069
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
>Reporter: Jim Brennan
>Assignee: Jim Brennan
>Priority: Major
> Attachments: MAPREDUCE-7069.001.patch, MAPREDUCE-7069.002.patch,
> MAPREDUCE-7069.003.patch, MAPREDUCE-7069.004.patch, MAPREDUCE-7069.005.patch,
> MAPREDUCE-7069.006.patch, MAPREDUCE-7069.007.patch
>
>
> As reported in YARN-6830, it is currently not possible to specify an
> environment variable that contains commas via {{mapreduce.map.env}},
> mapreduce.reduce.env, or {{mapreduce.admin.user.env}}.
> To address this, [~aw] proposed in [YARN-6830] that we add the ability to
> specify environment variables individually:
> {quote}e.g, mapreduce.map.env.[foo]=bar gets turned into foo=bar
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7069) Add ability to specify user environment variables individually
[
https://issues.apache.org/jira/browse/MAPREDUCE-7069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16434615#comment-16434615
]
Hadoop QA commented on MAPREDUCE-7069:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
17s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 26m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 1s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
34s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
18s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 25m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 25m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
53s{color} | {color:green} root: The patch generated 0 new + 458 unchanged - 9
fixed = 458 total (was 467) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 18s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
45s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
31s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
58s{color} | {color:green} hadoop-mapreduce-client-core in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
4s{color} | {color:green} hadoop-mapreduce-client-common in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
22s{color} | {color:green} hadoop-mapreduce-client-app in the patch passed.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}130m 58s{color}
| {color:red} hadoop-mapreduce-client-jobclient in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
38s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}277m 8s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8620d2b |
| JIRA Issue | MAPREDUCE-7069 |
| JIRA Patch URL |
[jira] [Commented] (MAPREDUCE-7036) ASF License warning in hadoop-mapreduce-client
[
https://issues.apache.org/jira/browse/MAPREDUCE-7036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16434516#comment-16434516
]
Wangda Tan commented on MAPREDUCE-7036:
---
Thanks [~suma.shivaprasad] and offline help from [~vinodkv] to figure out the
root case.
I suggest to revert this patch, such ASF license warning does not exist in
Hadoop code, which is accidentally introduced by yetus, I think the fix should
be landed in yetus.
Thoughts? [~ajisakaa] / [~tasanuma0829]
> ASF License warning in hadoop-mapreduce-client
> --
>
> Key: MAPREDUCE-7036
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7036
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Reporter: Takanobu Asanuma
>Assignee: Takanobu Asanuma
>Priority: Minor
> Fix For: 3.1.0
>
> Attachments: MAPREDUCE-7036.1.patch
>
>
> it occurred in MAPREDUCE-7021 and MAPREDUCE-7034.
> {noformat}
> Lines that start with ? in the ASF License report indicate files that do
> not have an Apache license header: !?
> /testptch/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/jobTokenPassword
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (MAPREDUCE-7036) ASF License warning in hadoop-mapreduce-client
[
https://issues.apache.org/jira/browse/MAPREDUCE-7036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16434396#comment-16434396
]
Suma Shivaprasad edited comment on MAPREDUCE-7036 at 4/11/18 6:46 PM:
--
[~tasanuma0829] [~ajisakaa] Reopening this issue since this patch breaks
MapReduce Pipes and causes a race condition due to maps and reducers writing to
the same directory in MAPREDUCE-7077. The title of this patch is misleading
since it mentions fix of License but makes code changes instead. Can you edit
Jira summary to reflect the exact changes done ?
was (Author: suma.shivaprasad):
[~tasanuma0829] [~ajisakaa] Reopening this issue since this patch breaks
MapReduce Pipes and causes a race condition due to maps and reducers writing to
the same working directory in MAPREDUCE-7077. The title of this patch is
misleading since it mentions fix of License but makes code changes instead. Can
you edit Jira summary to reflect the exact changes done ?
> ASF License warning in hadoop-mapreduce-client
> --
>
> Key: MAPREDUCE-7036
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7036
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Reporter: Takanobu Asanuma
>Assignee: Takanobu Asanuma
>Priority: Minor
> Fix For: 3.1.0
>
> Attachments: MAPREDUCE-7036.1.patch
>
>
> it occurred in MAPREDUCE-7021 and MAPREDUCE-7034.
> {noformat}
> Lines that start with ? in the ASF License report indicate files that do
> not have an Apache license header: !?
> /testptch/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/jobTokenPassword
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Reopened] (MAPREDUCE-7036) ASF License warning in hadoop-mapreduce-client
[
https://issues.apache.org/jira/browse/MAPREDUCE-7036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Suma Shivaprasad reopened MAPREDUCE-7036:
-
[~tasanuma0829] [~ajisakaa] Reopening this issue since this patch breaks
MapReduce Pipes and causes a race condition due to maps and reducers writing to
the same working directory in MAPREDUCE-7077. The title of this patch is
misleading since it mentions fix of License but makes code changes instead. Can
you edit Jira summary to reflect the exact changes done ?
> ASF License warning in hadoop-mapreduce-client
> --
>
> Key: MAPREDUCE-7036
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7036
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Reporter: Takanobu Asanuma
>Assignee: Takanobu Asanuma
>Priority: Minor
> Fix For: 3.1.0
>
> Attachments: MAPREDUCE-7036.1.patch
>
>
> it occurred in MAPREDUCE-7021 and MAPREDUCE-7034.
> {noformat}
> Lines that start with ? in the ASF License report indicate files that do
> not have an Apache license header: !?
> /testptch/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/jobTokenPassword
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7077) Pipe mapreduce job fails with Permission denied for jobTokenPassword
[
https://issues.apache.org/jira/browse/MAPREDUCE-7077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16434391#comment-16434391
]
Suma Shivaprasad commented on MAPREDUCE-7077:
-
This is occuring due to a race condition caused due to maps and reducers
writing the jobTokenPassword to the same application directory instead of the
tasks's working directory, which was changed in MAPREDUCE-7036
> Pipe mapreduce job fails with Permission denied for jobTokenPassword
>
>
> Key: MAPREDUCE-7077
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7077
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Yesha Vora
>Priority: Major
>
> Steps:
> Launch wordcount example with pipe
> {code}
> /usr/hdp/current/hadoop-client/bin/hadoop pipes
> "-Dhadoop.pipes.java.recordreader=true"
> "-Dhadoop.pipes.java.recordwriter=true" -input pipeInput -output pipeOutput
> -program bin/wordcount{code}
> The application fails with below stacktrace
> {code:title=AM}
> attempt_1517534613368_0041_r_00_2 is : 0.0
> 2018-02-02 02:40:51,071 ERROR [IPC Server handler 16 on 43391]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Task:
> attempt_1517534613368_0041_r_00_2 - exited :
> java.io.FileNotFoundException:
> /grid/0/hadoop/yarn/local/usercache/hrt_qa/appcache/application_1517534613368_0041/jobTokenPassword
> (Permission denied)
> at java.io.FileOutputStream.open0(Native Method)
> at java.io.FileOutputStream.open(FileOutputStream.java:270)
> at java.io.FileOutputStream.(FileOutputStream.java:213)
> at
> org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.(RawLocalFileSystem.java:236)
> at
> org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.(RawLocalFileSystem.java:219)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.createOutputStreamWithMode(RawLocalFileSystem.java:318)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:307)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:338)
> at
> org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.(ChecksumFileSystem.java:401)
> at
> org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:464)
> at
> org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:443)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1169)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1149)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1038)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1026)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:703)
> at
> org.apache.hadoop.mapred.pipes.Application.writePasswordToLocalFile(Application.java:173)
> at org.apache.hadoop.mapred.pipes.Application.(Application.java:109)
> at
> org.apache.hadoop.mapred.pipes.PipesReducer.startApplication(PipesReducer.java:87)
> at org.apache.hadoop.mapred.pipes.PipesReducer.reduce(PipesReducer.java:65)
> at org.apache.hadoop.mapred.pipes.PipesReducer.reduce(PipesReducer.java:38)
> at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:445)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:393)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1965)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Created] (MAPREDUCE-7077) Pipe mapreduce job fails with Permission denied for jobTokenPassword
Yesha Vora created MAPREDUCE-7077:
-
Summary: Pipe mapreduce job fails with Permission denied for
jobTokenPassword
Key: MAPREDUCE-7077
URL: https://issues.apache.org/jira/browse/MAPREDUCE-7077
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Yesha Vora
Steps:
Launch wordcount example with pipe
{code}
/usr/hdp/current/hadoop-client/bin/hadoop pipes
"-Dhadoop.pipes.java.recordreader=true" "-Dhadoop.pipes.java.recordwriter=true"
-input pipeInput -output pipeOutput -program bin/wordcount{code}
The application fails with below stacktrace
{code:title=AM}
attempt_1517534613368_0041_r_00_2 is : 0.0
2018-02-02 02:40:51,071 ERROR [IPC Server handler 16 on 43391]
org.apache.hadoop.mapred.TaskAttemptListenerImpl: Task:
attempt_1517534613368_0041_r_00_2 - exited : java.io.FileNotFoundException:
/grid/0/hadoop/yarn/local/usercache/hrt_qa/appcache/application_1517534613368_0041/jobTokenPassword
(Permission denied)
at java.io.FileOutputStream.open0(Native Method)
at java.io.FileOutputStream.open(FileOutputStream.java:270)
at java.io.FileOutputStream.(FileOutputStream.java:213)
at
org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.(RawLocalFileSystem.java:236)
at
org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.(RawLocalFileSystem.java:219)
at
org.apache.hadoop.fs.RawLocalFileSystem.createOutputStreamWithMode(RawLocalFileSystem.java:318)
at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:307)
at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:338)
at
org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.(ChecksumFileSystem.java:401)
at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:464)
at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:443)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1169)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1149)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1038)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1026)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:703)
at
org.apache.hadoop.mapred.pipes.Application.writePasswordToLocalFile(Application.java:173)
at org.apache.hadoop.mapred.pipes.Application.(Application.java:109)
at
org.apache.hadoop.mapred.pipes.PipesReducer.startApplication(PipesReducer.java:87)
at org.apache.hadoop.mapred.pipes.PipesReducer.reduce(PipesReducer.java:65)
at org.apache.hadoop.mapred.pipes.PipesReducer.reduce(PipesReducer.java:38)
at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:445)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:393)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1965)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7069) Add ability to specify user environment variables individually
[
https://issues.apache.org/jira/browse/MAPREDUCE-7069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16434225#comment-16434225
]
Jim Brennan commented on MAPREDUCE-7069:
[~jlowe], [~shaneku...@gmail.com], thanks again for the reviews, and thanks
[~shaneku...@gmail.com] for testing the patch. I have put up a new patch that
I believe addresses the documentation issues.
> Add ability to specify user environment variables individually
> --
>
> Key: MAPREDUCE-7069
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7069
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
>Reporter: Jim Brennan
>Assignee: Jim Brennan
>Priority: Major
> Attachments: MAPREDUCE-7069.001.patch, MAPREDUCE-7069.002.patch,
> MAPREDUCE-7069.003.patch, MAPREDUCE-7069.004.patch, MAPREDUCE-7069.005.patch,
> MAPREDUCE-7069.006.patch, MAPREDUCE-7069.007.patch
>
>
> As reported in YARN-6830, it is currently not possible to specify an
> environment variable that contains commas via {{mapreduce.map.env}},
> mapreduce.reduce.env, or {{mapreduce.admin.user.env}}.
> To address this, [~aw] proposed in [YARN-6830] that we add the ability to
> specify environment variables individually:
> {quote}e.g, mapreduce.map.env.[foo]=bar gets turned into foo=bar
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7069) Add ability to specify user environment variables individually
[
https://issues.apache.org/jira/browse/MAPREDUCE-7069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jim Brennan updated MAPREDUCE-7069:
---
Attachment: MAPREDUCE-7069.007.patch
> Add ability to specify user environment variables individually
> --
>
> Key: MAPREDUCE-7069
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7069
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
>Reporter: Jim Brennan
>Assignee: Jim Brennan
>Priority: Major
> Attachments: MAPREDUCE-7069.001.patch, MAPREDUCE-7069.002.patch,
> MAPREDUCE-7069.003.patch, MAPREDUCE-7069.004.patch, MAPREDUCE-7069.005.patch,
> MAPREDUCE-7069.006.patch, MAPREDUCE-7069.007.patch
>
>
> As reported in YARN-6830, it is currently not possible to specify an
> environment variable that contains commas via {{mapreduce.map.env}},
> mapreduce.reduce.env, or {{mapreduce.admin.user.env}}.
> To address this, [~aw] proposed in [YARN-6830] that we add the ability to
> specify environment variables individually:
> {quote}e.g, mapreduce.map.env.[foo]=bar gets turned into foo=bar
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6826) Job fails with InvalidStateTransitonException: Invalid event: JOB_TASK_COMPLETED at SUCCEEDED
[
https://issues.apache.org/jira/browse/MAPREDUCE-6826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16434128#comment-16434128
]
Hadoop QA commented on MAPREDUCE-6826:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 26m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 18m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
29m 53s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source: . {color}
|
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m
57s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 25m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 25m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 19m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 19s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source: . {color}
|
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 5m
17s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}127m 43s{color}
| {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
36s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}299m 18s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.web.TestWebHdfsTimeouts |
| | hadoop.hdfs.TestEncryptionZonesWithKMS |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8620d2b |
| JIRA Issue | MAPREDUCE-6826 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12914665/MAPREDUCE-6826-001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 9ec1abd78009 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 7eb783e |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_162 |
| unit |
[jira] [Commented] (MAPREDUCE-7069) Add ability to specify user environment variables individually
[
https://issues.apache.org/jira/browse/MAPREDUCE-7069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16434026#comment-16434026
]
Jim Brennan commented on MAPREDUCE-7069:
I can do this for some of them, but the base properties for
{{mapreduce.map.env}} and {{mapreduce.reduce.env}} are already commented out so
that they don't override {{mapreduce.child.env}}. Do you think it is
appropriate to put the documentation for those in the description for
{{mapreduce.child.env}}?
> Add ability to specify user environment variables individually
> --
>
> Key: MAPREDUCE-7069
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7069
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
>Reporter: Jim Brennan
>Assignee: Jim Brennan
>Priority: Major
> Attachments: MAPREDUCE-7069.001.patch, MAPREDUCE-7069.002.patch,
> MAPREDUCE-7069.003.patch, MAPREDUCE-7069.004.patch, MAPREDUCE-7069.005.patch,
> MAPREDUCE-7069.006.patch
>
>
> As reported in YARN-6830, it is currently not possible to specify an
> environment variable that contains commas via {{mapreduce.map.env}},
> mapreduce.reduce.env, or {{mapreduce.admin.user.env}}.
> To address this, [~aw] proposed in [YARN-6830] that we add the ability to
> specify environment variables individually:
> {quote}e.g, mapreduce.map.env.[foo]=bar gets turned into foo=bar
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7069) Add ability to specify user environment variables individually
[
https://issues.apache.org/jira/browse/MAPREDUCE-7069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16434021#comment-16434021
]
Jason Lowe commented on MAPREDUCE-7069:
---
Given this is XML and not HTML I think we should avoid the use of HTML tags for
now. Those tags will make it harder to read if the consumer is not viewing it
with an HTML viewer. For the mapred-default.xml descriptions I think we just
need to mention that variables can also be specified as separate properties and
give a simple example demonstrating the syntax.
> Add ability to specify user environment variables individually
> --
>
> Key: MAPREDUCE-7069
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7069
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
>Reporter: Jim Brennan
>Assignee: Jim Brennan
>Priority: Major
> Attachments: MAPREDUCE-7069.001.patch, MAPREDUCE-7069.002.patch,
> MAPREDUCE-7069.003.patch, MAPREDUCE-7069.004.patch, MAPREDUCE-7069.005.patch,
> MAPREDUCE-7069.006.patch
>
>
> As reported in YARN-6830, it is currently not possible to specify an
> environment variable that contains commas via {{mapreduce.map.env}},
> mapreduce.reduce.env, or {{mapreduce.admin.user.env}}.
> To address this, [~aw] proposed in [YARN-6830] that we add the ability to
> specify environment variables individually:
> {quote}e.g, mapreduce.map.env.[foo]=bar gets turned into foo=bar
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7069) Add ability to specify user environment variables individually
[
https://issues.apache.org/jira/browse/MAPREDUCE-7069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16433948#comment-16433948
]
Jim Brennan commented on MAPREDUCE-7069:
[~jlowe], [~shaneku...@gmail.com], thanks for the reviews! I am working to
update the documentation and I'll put up another patch soon.
{quote}Looking at the docs generated at
[http://hadoop.apache.org/docs/current/hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml]
I'm wondering if it would be more useful to add the description for the .[var]
form of the properties to the base property instead of having separate,
commented-out properties. Users probably won't ever notice the commented-out
text in that file, especially since it's buried in a .jar when deployed, but
many would notice the docs generated for properties that aren't commented out.
Thoughts?
{quote}
This is what I originally started to do, but switched to the commented out
option because it felt like the descriptions were too cumbersome. But your
point is well taken. I'll take another stab at combining them. I don't see any
precedent in this file for using formatting tags inside the descriptions (,
,...) - do you know if that is allowed?
> Add ability to specify user environment variables individually
> --
>
> Key: MAPREDUCE-7069
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7069
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
>Reporter: Jim Brennan
>Assignee: Jim Brennan
>Priority: Major
> Attachments: MAPREDUCE-7069.001.patch, MAPREDUCE-7069.002.patch,
> MAPREDUCE-7069.003.patch, MAPREDUCE-7069.004.patch, MAPREDUCE-7069.005.patch,
> MAPREDUCE-7069.006.patch
>
>
> As reported in YARN-6830, it is currently not possible to specify an
> environment variable that contains commas via {{mapreduce.map.env}},
> mapreduce.reduce.env, or {{mapreduce.admin.user.env}}.
> To address this, [~aw] proposed in [YARN-6830] that we add the ability to
> specify environment variables individually:
> {quote}e.g, mapreduce.map.env.[foo]=bar gets turned into foo=bar
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7069) Add ability to specify user environment variables individually
[
https://issues.apache.org/jira/browse/MAPREDUCE-7069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16433947#comment-16433947
]
Shane Kumpf commented on MAPREDUCE-7069:
Circling back. Successfully tested with the 006 patch and didn't find any
issues. Once Jason's comments are addressed, I think the patch is ready. I'll
loop back for a final review when the new patch is ready.
> Add ability to specify user environment variables individually
> --
>
> Key: MAPREDUCE-7069
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7069
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
>Reporter: Jim Brennan
>Assignee: Jim Brennan
>Priority: Major
> Attachments: MAPREDUCE-7069.001.patch, MAPREDUCE-7069.002.patch,
> MAPREDUCE-7069.003.patch, MAPREDUCE-7069.004.patch, MAPREDUCE-7069.005.patch,
> MAPREDUCE-7069.006.patch
>
>
> As reported in YARN-6830, it is currently not possible to specify an
> environment variable that contains commas via {{mapreduce.map.env}},
> mapreduce.reduce.env, or {{mapreduce.admin.user.env}}.
> To address this, [~aw] proposed in [YARN-6830] that we add the ability to
> specify environment variables individually:
> {quote}e.g, mapreduce.map.env.[foo]=bar gets turned into foo=bar
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6826) Job fails with InvalidStateTransitonException: Invalid event: JOB_TASK_COMPLETED at SUCCEEDED
[
https://issues.apache.org/jira/browse/MAPREDUCE-6826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16433746#comment-16433746
]
Varun Saxena commented on MAPREDUCE-6826:
-
Submitting patch to invoke QA
> Job fails with InvalidStateTransitonException: Invalid event:
> JOB_TASK_COMPLETED at SUCCEEDED
> -
>
> Key: MAPREDUCE-6826
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6826
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.7.2
>Reporter: Varun Saxena
>Assignee: Bilwa S T
>Priority: Major
> Attachments: MAPREDUCE-6826-001.patch
>
>
> This happens if a container is preempted by scheduler after job starts
> committing.
> And this exception in turn leads to application being marked as FAILED in
> YARN.
> I think we can probably ignore JOB_TASK_COMPLETED event while JobImpl state
> is COMMITTING or SUCCEEDED as job is in the process of finishing.
> Also is there any point in attempting to scheduler another task attempt if
> job is already in COMMITTING or SUCCEEDED state.
> {noformat}
> 2016-12-23 09:10:38,642 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl:
> task_1482404625971_23910_m_04 Task Transitioned from RUNNING to SUCCEEDED
> 2016-12-23 09:10:38,642 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Num completed Tasks: 5
> 2016-12-23 09:10:38,643 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl:
> job_1482404625971_23910Job Transitioned from RUNNING to COMMITTING
> 2016-12-23 09:10:38,644 INFO [ContainerLauncher #5]
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing
> the event EventType: CONTAINER_REMOTE_CLEANUP for container
> container_e55_1482404625971_23910_01_10 taskAttempt
> attempt_1482404625971_23910_m_04_1
> 2016-12-23 09:10:38,644 INFO [ContainerLauncher #5]
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING
> attempt_1482404625971_23910_m_04_1
> 2016-12-23 09:10:38,644 INFO [ContainerLauncher #5]
> org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy:
> Opening proxy : linux-19:26009
> 2016-12-23 09:10:38,644 INFO [CommitterEvent Processor #4]
> org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing
> the event EventType: JOB_COMMIT
> 2016-12-23 09:10:38,724 INFO [IPC Server handler 0 on 27113]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID :
> jvm_1482404625971_23910_m_60473139527690 asked for a task
> 2016-12-23 09:10:38,724 INFO [IPC Server handler 0 on 27113]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID:
> jvm_1482404625971_23910_m_60473139527690 is invalid and will be killed.
> 2016-12-23 09:10:38,797 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Calling handler for
> JobFinishedEvent
> 2016-12-23 09:10:38,797 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl:
> job_1482404625971_23910Job Transitioned from COMMITTING to SUCCEEDED
> 2016-12-23 09:10:38,798 INFO [Thread-93]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Job finished cleanly,
> recording last MRAppMaster retry
> 2016-12-23 09:10:38,798 INFO [Thread-93]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Notify RMCommunicator
> isAMLastRetry: true
> 2016-12-23 09:10:38,798 INFO [Thread-93]
> org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator: RMCommunicator notified
> that shouldUnregistered is: true
> 2016-12-23 09:10:38,799 INFO [Thread-93]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Notify JHEH isAMLastRetry:
> true
> 2016-12-23 09:10:38,799 INFO [Thread-93]
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler:
> JobHistoryEventHandler notified that forceJobCompletion is true
> 2016-12-23 09:10:38,799 INFO [Thread-93]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Calling stop for all the
> services
> 2016-12-23 09:10:38,800 INFO [Thread-93]
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Stopping
> JobHistoryEventHandler. Size of the outstanding queue size is 1
> 2016-12-23 09:10:38,989 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before
> Scheduling: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:1
> AssignedReds:0 CompletedMaps:5 CompletedReds:0 ContAlloc:8 ContRel:0
> HostLocal:0 RackLocal:0
> 2016-12-23 09:10:38,993 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received
> completed container container_e55_1482404625971_23910_01_10
> 2016-12-23 09:10:38,993 INFO
[jira] [Updated] (MAPREDUCE-6826) Job fails with InvalidStateTransitonException: Invalid event: JOB_TASK_COMPLETED at SUCCEEDED
[
https://issues.apache.org/jira/browse/MAPREDUCE-6826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bibin A Chundatt updated MAPREDUCE-6826:
Status: Patch Available (was: Open)
> Job fails with InvalidStateTransitonException: Invalid event:
> JOB_TASK_COMPLETED at SUCCEEDED
> -
>
> Key: MAPREDUCE-6826
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6826
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.7.2
>Reporter: Varun Saxena
>Assignee: Bilwa S T
>Priority: Major
> Attachments: MAPREDUCE-6826-001.patch
>
>
> This happens if a container is preempted by scheduler after job starts
> committing.
> And this exception in turn leads to application being marked as FAILED in
> YARN.
> I think we can probably ignore JOB_TASK_COMPLETED event while JobImpl state
> is COMMITTING or SUCCEEDED as job is in the process of finishing.
> Also is there any point in attempting to scheduler another task attempt if
> job is already in COMMITTING or SUCCEEDED state.
> {noformat}
> 2016-12-23 09:10:38,642 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl:
> task_1482404625971_23910_m_04 Task Transitioned from RUNNING to SUCCEEDED
> 2016-12-23 09:10:38,642 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Num completed Tasks: 5
> 2016-12-23 09:10:38,643 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl:
> job_1482404625971_23910Job Transitioned from RUNNING to COMMITTING
> 2016-12-23 09:10:38,644 INFO [ContainerLauncher #5]
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing
> the event EventType: CONTAINER_REMOTE_CLEANUP for container
> container_e55_1482404625971_23910_01_10 taskAttempt
> attempt_1482404625971_23910_m_04_1
> 2016-12-23 09:10:38,644 INFO [ContainerLauncher #5]
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING
> attempt_1482404625971_23910_m_04_1
> 2016-12-23 09:10:38,644 INFO [ContainerLauncher #5]
> org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy:
> Opening proxy : linux-19:26009
> 2016-12-23 09:10:38,644 INFO [CommitterEvent Processor #4]
> org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing
> the event EventType: JOB_COMMIT
> 2016-12-23 09:10:38,724 INFO [IPC Server handler 0 on 27113]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID :
> jvm_1482404625971_23910_m_60473139527690 asked for a task
> 2016-12-23 09:10:38,724 INFO [IPC Server handler 0 on 27113]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID:
> jvm_1482404625971_23910_m_60473139527690 is invalid and will be killed.
> 2016-12-23 09:10:38,797 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Calling handler for
> JobFinishedEvent
> 2016-12-23 09:10:38,797 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl:
> job_1482404625971_23910Job Transitioned from COMMITTING to SUCCEEDED
> 2016-12-23 09:10:38,798 INFO [Thread-93]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Job finished cleanly,
> recording last MRAppMaster retry
> 2016-12-23 09:10:38,798 INFO [Thread-93]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Notify RMCommunicator
> isAMLastRetry: true
> 2016-12-23 09:10:38,798 INFO [Thread-93]
> org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator: RMCommunicator notified
> that shouldUnregistered is: true
> 2016-12-23 09:10:38,799 INFO [Thread-93]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Notify JHEH isAMLastRetry:
> true
> 2016-12-23 09:10:38,799 INFO [Thread-93]
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler:
> JobHistoryEventHandler notified that forceJobCompletion is true
> 2016-12-23 09:10:38,799 INFO [Thread-93]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Calling stop for all the
> services
> 2016-12-23 09:10:38,800 INFO [Thread-93]
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Stopping
> JobHistoryEventHandler. Size of the outstanding queue size is 1
> 2016-12-23 09:10:38,989 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before
> Scheduling: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:1
> AssignedReds:0 CompletedMaps:5 CompletedReds:0 ContAlloc:8 ContRel:0
> HostLocal:0 RackLocal:0
> 2016-12-23 09:10:38,993 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received
> completed container container_e55_1482404625971_23910_01_10
> 2016-12-23 09:10:38,993 INFO [RMCommunicator Allocator]
>
[jira] [Commented] (MAPREDUCE-7073) Optimize TokenCache#obtainTokensForNamenodesInternal
[
https://issues.apache.org/jira/browse/MAPREDUCE-7073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16433686#comment-16433686
]
Bibin A Chundatt commented on MAPREDUCE-7073:
-
[~jianhe]
Could you please help in review ?
> Optimize TokenCache#obtainTokensForNamenodesInternal
>
>
> Key: MAPREDUCE-7073
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7073
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>Priority: Major
> Attachments: MAPREDUCE-7073.001.patch, MAPREDUCE-7073.002.patch
>
>
> {{FileInputFormat#listStatus}} is too slow file system cache is disabled.
> {{TokenCache#obtainTokensForNamenodesInternal}} for every filesystem instance
> {{Master.getMasterPrincipal(conf)}} is caled which reloads YarnConfiguration .
> For fileInput with 1k file will reload YarnConfiguration 1k times.
> {{Master.getMasterPrincipal(conf)}} can be passed for
> {{obtainTokensForNamenodesInternal}} per filesystem call.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
