[jira] [Commented] (MAPREDUCE-6869) org.apache.hadoop.mapred.ShuffleHandler: Shuffle error in populating headers :
[ https://issues.apache.org/jira/browse/MAPREDUCE-6869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15944493#comment-15944493 ] 翟玉勇 commented on MAPREDUCE-6869: thx > org.apache.hadoop.mapred.ShuffleHandler: Shuffle error in populating headers : > -- > > Key: MAPREDUCE-6869 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6869 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, yarn >Affects Versions: 2.6.0 > Environment: hadoop 2.6.0-cdh5.8.2 >Reporter: 翟玉勇 >Priority: Minor > > nodemanager log > 2017-03-25 21:07:03,071 ERROR org.apache.hadoop.mapred.ShuffleHandler: > Shuffle error in populating headers : > org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find > usercache/master/appcache/application_1489067586592_930490/output/attempt_1489067586592_930490_m_002811_0/file.out.index > in any of the configured local directories > at > org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:488) > at > org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:165) > at > org.apache.hadoop.mapred.ShuffleHandler$Shuffle.getMapOutputInfo(ShuffleHandler.java:1000) > at > org.apache.hadoop.mapred.ShuffleHandler$Shuffle.populateHeaders(ShuffleHandler.java:1022) > at > org.apache.hadoop.mapred.ShuffleHandler$Shuffle.messageReceived(ShuffleHandler.java:908) > at > org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) > at > org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560) > at > org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:787) > at > org.jboss.netty.handler.stream.ChunkedWriteHandler.handleUpstream(ChunkedWriteHandler.java:142) > at > org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560) > at > org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:787) > at > org.jboss.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:148) > at > org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) > at > org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560) > at > org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:787) > at > org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) > at > org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459) > at > org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536) > at > org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) > at > org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) > at > org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560) > at > org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:555) > at > org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) > at > org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) > at > org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) > at > org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:107) > at > org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312) > at > org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88) > at > org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) > at > org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) > at > org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > reduce log > 2017-03-25 05:40:33,919 WARN [fetcher#2] > org.apache.hadoop.mapreduce.task.reduce.Fetcher: Invalid map id > java.lang.IllegalArgumentException: TaskAttemptId string : TTP/1.1 500 > Internal Server Error > Content-Type: text/plain;
[jira] [Commented] (MAPREDUCE-6862) Fragments are not handled correctly by resource limit checking
[
https://issues.apache.org/jira/browse/MAPREDUCE-6862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15944315#comment-15944315
]
Hadoop QA commented on MAPREDUCE-6862:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 22s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
30s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 25s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
18s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 30s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
51s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 22s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
25s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 24s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 24s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 27s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
58s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 21s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 49s
{color} | {color:green} hadoop-mapreduce-client-core in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 19s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12860765/MAPREDUCE-6862-trunk.001.patch
|
| JIRA Issue | MAPREDUCE-6862 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 3c8a2a775080 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 64ea62c |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6926/testReport/ |
| modules | C:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
U:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6926/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Fragments are not handled correctly by resource limit checking
> --
>
> Key: MAPREDUCE-6862
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6862
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.9.0, 3.0.0-alpha1
>Reporter: Chris Trezzo
>Assignee: Chris Trezzo
>Priority: Minor
> Attachments: MAPREDUCE-6862-trunk.001.patch
[jira] [Updated] (MAPREDUCE-6862) Fragments are not handled correctly by resource limit checking
[
https://issues.apache.org/jira/browse/MAPREDUCE-6862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Trezzo updated MAPREDUCE-6862:
Status: Patch Available (was: Open)
> Fragments are not handled correctly by resource limit checking
> --
>
> Key: MAPREDUCE-6862
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6862
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 3.0.0-alpha1, 2.9.0
>Reporter: Chris Trezzo
>Assignee: Chris Trezzo
>Priority: Minor
> Attachments: MAPREDUCE-6862-trunk.001.patch
>
>
> If a user specifies a fragment for a libjar, files, archives path via generic
> options parser and resource limit checking is enabled, the client crashes
> with a FileNotFoundException:
> {noformat}
> java.io.FileNotFoundException: File file:/home/mapred/test.txt#testFrag.txt
> does not exist
> at
> org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:638)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:864)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:628)
> at
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:442)
> at
> org.apache.hadoop.mapreduce.JobResourceUploader.getFileStatus(JobResourceUploader.java:413)
> at
> org.apache.hadoop.mapreduce.JobResourceUploader.explorePath(JobResourceUploader.java:395)
> at
> org.apache.hadoop.mapreduce.JobResourceUploader.checkLocalizationLimits(JobResourceUploader.java:304)
> at
> org.apache.hadoop.mapreduce.JobResourceUploader.uploadResources(JobResourceUploader.java:103)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:102)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:197)
> at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1344)
> at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1341)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1892)
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1341)
> at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1362)
> at
> org.apache.hadoop.examples.QuasiMonteCarlo.estimatePi(QuasiMonteCarlo.java:306)
> at
> org.apache.hadoop.examples.QuasiMonteCarlo.run(QuasiMonteCarlo.java:359)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at
> org.apache.hadoop.examples.QuasiMonteCarlo.main(QuasiMonteCarlo.java:367)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71)
> at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144)
> at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:239)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:153)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6862) Fragments are not handled correctly by resource limit checking
[
https://issues.apache.org/jira/browse/MAPREDUCE-6862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Trezzo updated MAPREDUCE-6862:
Attachment: MAPREDUCE-6862-trunk.001.patch
Attached is a v1 for trunk. This patch does the following:
# Fixes the conversion from string to path when there are fragments present for
the limit checker in JobResourceUploader.
# Adds a unit test.
> Fragments are not handled correctly by resource limit checking
> --
>
> Key: MAPREDUCE-6862
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6862
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.9.0, 3.0.0-alpha1
>Reporter: Chris Trezzo
>Assignee: Chris Trezzo
>Priority: Minor
> Attachments: MAPREDUCE-6862-trunk.001.patch
>
>
> If a user specifies a fragment for a libjar, files, archives path via generic
> options parser and resource limit checking is enabled, the client crashes
> with a FileNotFoundException:
> {noformat}
> java.io.FileNotFoundException: File file:/home/mapred/test.txt#testFrag.txt
> does not exist
> at
> org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:638)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:864)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:628)
> at
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:442)
> at
> org.apache.hadoop.mapreduce.JobResourceUploader.getFileStatus(JobResourceUploader.java:413)
> at
> org.apache.hadoop.mapreduce.JobResourceUploader.explorePath(JobResourceUploader.java:395)
> at
> org.apache.hadoop.mapreduce.JobResourceUploader.checkLocalizationLimits(JobResourceUploader.java:304)
> at
> org.apache.hadoop.mapreduce.JobResourceUploader.uploadResources(JobResourceUploader.java:103)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:102)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:197)
> at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1344)
> at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1341)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1892)
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1341)
> at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1362)
> at
> org.apache.hadoop.examples.QuasiMonteCarlo.estimatePi(QuasiMonteCarlo.java:306)
> at
> org.apache.hadoop.examples.QuasiMonteCarlo.run(QuasiMonteCarlo.java:359)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at
> org.apache.hadoop.examples.QuasiMonteCarlo.main(QuasiMonteCarlo.java:367)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71)
> at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144)
> at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:239)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:153)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6785) ContainerLauncherImpl support for reusing the containers
[
https://issues.apache.org/jira/browse/MAPREDUCE-6785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15944007#comment-15944007
]
Hadoop QA commented on MAPREDUCE-6785:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 34s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
54s {color} | {color:green} MR-6749 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 22s
{color} | {color:green} MR-6749 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
18s {color} | {color:green} MR-6749 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 29s
{color} | {color:green} MR-6749 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} MR-6749 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
41s {color} | {color:green} MR-6749 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 21s
{color} | {color:green} MR-6749 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
22s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 22s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 22s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 16s
{color} | {color:red}
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app:
The patch generated 3 new + 345 unchanged - 9 fixed = 348 total (was 354)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 26s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
46s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 14s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m 49s
{color} | {color:green} hadoop-mapreduce-client-app in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 18s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12860714/MAPREDUCE-6785-MR-6749.006.patch
|
| JIRA Issue | MAPREDUCE-6785 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux aaf2e0c3dcc9 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | MR-6749 / 48a24ec |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6925/artifact/patchprocess/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6925/testReport/ |
| modules | C:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app U:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6925/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> ContainerLauncherImpl support for reusing the containers
>
>
> Key: MAPREDUCE-6785
>
[jira] [Updated] (MAPREDUCE-6785) ContainerLauncherImpl support for reusing the containers
[ https://issues.apache.org/jira/browse/MAPREDUCE-6785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naganarasimha G R updated MAPREDUCE-6785: - Attachment: MAPREDUCE-6785-MR-6749.006.patch attaching the modifications for fixing jenkins reported issues ! > ContainerLauncherImpl support for reusing the containers > > > Key: MAPREDUCE-6785 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6785 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: applicationmaster, mrv2 >Reporter: Devaraj K >Assignee: Devaraj K > Attachments: MAPREDUCE-6785-MR-6749.003.patch, > MAPREDUCE-6785-MR-6749.004.patch, MAPREDUCE-6785-MR-6749.005.patch, > MAPREDUCE-6785-MR-6749.006.patch, MAPREDUCE-6785-v0.patch, > MAPREDUCE-6785-v1.patch, MAPREDUCE-6785-v2.patch > > > Add support to Container Launcher for reuse of the containers. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6785) ContainerLauncherImpl support for reusing the containers
[
https://issues.apache.org/jira/browse/MAPREDUCE-6785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15943510#comment-15943510
]
Hadoop QA commented on MAPREDUCE-6785:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 44s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
27s {color} | {color:green} MR-6749 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 21s
{color} | {color:green} MR-6749 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s {color} | {color:green} MR-6749 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 29s
{color} | {color:green} MR-6749 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s {color} | {color:green} MR-6749 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
36s {color} | {color:green} MR-6749 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 16s
{color} | {color:green} MR-6749 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
20s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 18s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 18s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 17s
{color} | {color:red}
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app:
The patch generated 8 new + 345 unchanged - 9 fixed = 353 total (was 354)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 24s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 2 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 39s
{color} | {color:red}
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 12s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 9m 27s {color}
| {color:red} hadoop-mapreduce-client-app in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 40s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs |
module:hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
|
| | org.apache.hadoop.yarn.api.records.ContainerId is incompatible with
expected argument type org.apache.hadoop.yarn.api.records.Container in
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerReuseRequestor.containerAssigned(Container,
RMContainerRequestor$ContainerRequest, Map) At
RMContainerReuseRequestor.java:argument type
org.apache.hadoop.yarn.api.records.Container in
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerReuseRequestor.containerAssigned(Container,
RMContainerRequestor$ContainerRequest, Map) At
RMContainerReuseRequestor.java:[line 233] |
| Failed junit tests | hadoop.mapreduce.v2.app.rm.TestRMContainerReuseRequestor
|
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12860583/MAPREDUCE-6785-MR-6749.005.patch
|
| JIRA Issue | MAPREDUCE-6785 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 7e300d2cf4cb 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git
[jira] [Updated] (MAPREDUCE-6785) ContainerLauncherImpl support for reusing the containers
[ https://issues.apache.org/jira/browse/MAPREDUCE-6785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naganarasimha G R updated MAPREDUCE-6785: - Status: Patch Available (was: Open) > ContainerLauncherImpl support for reusing the containers > > > Key: MAPREDUCE-6785 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6785 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: applicationmaster, mrv2 >Reporter: Devaraj K >Assignee: Devaraj K > Attachments: MAPREDUCE-6785-MR-6749.003.patch, > MAPREDUCE-6785-MR-6749.004.patch, MAPREDUCE-6785-MR-6749.005.patch, > MAPREDUCE-6785-v0.patch, MAPREDUCE-6785-v1.patch, MAPREDUCE-6785-v2.patch > > > Add support to Container Launcher for reuse of the containers. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6869) org.apache.hadoop.mapred.ShuffleHandler: Shuffle error in populating headers :
[ https://issues.apache.org/jira/browse/MAPREDUCE-6869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15943317#comment-15943317 ] Jason Lowe commented on MAPREDUCE-6869: --- This error is simply stating that a map output file could not be found. That can happen if the disk goes bad, something like tmpwatch is coming along and cleaning things up when it shouldn't, etc. It is not in itself indicative of a problem in Hadoop unless you have supporting evidence for that. Is this reproducible or are there other things that indicate there is something Hadoop itself can do to prevent this? If not then this is more like a user support request which is better served by the [Hadoop mailing lists|http://hadoop.apache.org/mailing_lists.html]. JIRA is for tracking known problems in Hadoop itself and not for general support. > org.apache.hadoop.mapred.ShuffleHandler: Shuffle error in populating headers : > -- > > Key: MAPREDUCE-6869 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6869 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, yarn >Affects Versions: 2.6.0 > Environment: hadoop 2.6.0-cdh5.8.2 >Reporter: 翟玉勇 >Priority: Minor > > nodemanager log > 2017-03-25 21:07:03,071 ERROR org.apache.hadoop.mapred.ShuffleHandler: > Shuffle error in populating headers : > org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find > usercache/master/appcache/application_1489067586592_930490/output/attempt_1489067586592_930490_m_002811_0/file.out.index > in any of the configured local directories > at > org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:488) > at > org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:165) > at > org.apache.hadoop.mapred.ShuffleHandler$Shuffle.getMapOutputInfo(ShuffleHandler.java:1000) > at > org.apache.hadoop.mapred.ShuffleHandler$Shuffle.populateHeaders(ShuffleHandler.java:1022) > at > org.apache.hadoop.mapred.ShuffleHandler$Shuffle.messageReceived(ShuffleHandler.java:908) > at > org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) > at > org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560) > at > org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:787) > at > org.jboss.netty.handler.stream.ChunkedWriteHandler.handleUpstream(ChunkedWriteHandler.java:142) > at > org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560) > at > org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:787) > at > org.jboss.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:148) > at > org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) > at > org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560) > at > org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:787) > at > org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) > at > org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459) > at > org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536) > at > org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) > at > org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) > at > org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560) > at > org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:555) > at > org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) > at > org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) > at > org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) > at > org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:107) > at > org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312) > at > org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88) > at > org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) > at >
[jira] [Commented] (MAPREDUCE-6840) Distcp to support cutoff time
[
https://issues.apache.org/jira/browse/MAPREDUCE-6840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15943218#comment-15943218
]
Steve Loughran commented on MAPREDUCE-6840:
---
milis is a pretty human-unfriendly number; the mechanism Configuration uses to
support ms, s, m, h, d is better. I think it should be possible to use
{{configuration.getTimeDuration()}} to parse the duration arg simply by
creating a no-default Configuration, set the property, then have it parse the
string. Ugly but effective
> Distcp to support cutoff time
> -
>
> Key: MAPREDUCE-6840
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6840
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: distcp
>Affects Versions: 2.6.0
>Reporter: Zheng Shao
>Assignee: Zheng Shao
>Priority: Minor
> Attachments: MAPREDUCE-6840.1.patch
>
>
> To ensure consistency in the datasets on HDFS, some projects like file
> formats on Hive do HDFS operations in a particular order. For example, if a
> file format uses an index file, a new version of the index file will only be
> written to HDFS after all files mentioned by the index are written to HDFS.
> When we do distcp, it's important to preserve that consistency, so that we
> don't break those file formats.
> A typical solution for that is to create a HDFS Snapshot beforehand, and only
> distcp the Snapshot. That could work well if the user has superuser
> privilege to make the directory snapshottable.
> If not, then it will be beneficial to have a cutoff time for distcp, so that
> distcp only copy files modified on/before that cutoff time.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-5653) DistCp does not honour config-overrides for mapreduce.[map,reduce].memory.mb
[ https://issues.apache.org/jira/browse/MAPREDUCE-5653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15943127#comment-15943127 ] Kuhu Shukla commented on MAPREDUCE-5653: Thank you so much [~aw] for the explanation. I too had my doubts whether this change will break a lot of users if we backported the patch but I agree that this change belongs to 3.x line. > DistCp does not honour config-overrides for mapreduce.[map,reduce].memory.mb > > > Key: MAPREDUCE-5653 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5653 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: distcp >Affects Versions: 0.23.9, 2.2.0 >Reporter: Mithun Radhakrishnan >Assignee: Ratandeep Ratti > Fix For: 3.0.0-alpha1 > > Attachments: MAPREDUCE-5653.branch-0.23.patch, > MAPREDUCE-5653.branch-2.patch, MAPREDUCE-5653.trunk.2.patch, > MAPREDUCE-5653.trunk.patch > > > When a DistCp job is run through Oozie (through a Java action that launches > DistCp), one sees that mapred.child.java.opts as set from the caller is > honoured by DistCp. But, DistCp doesn't seem to honour any overrides for > configs mapreduce.[map,reduce].memory.mb. > Problem has been identified. I'll post a patch shortly. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6869) org.apache.hadoop.mapred.ShuffleHandler: Shuffle error in populating headers :
[ https://issues.apache.org/jira/browse/MAPREDUCE-6869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] 翟玉勇 updated MAPREDUCE-6869: --- Description: nodemanager log 2017-03-25 21:07:03,071 ERROR org.apache.hadoop.mapred.ShuffleHandler: Shuffle error in populating headers : org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find usercache/master/appcache/application_1489067586592_930490/output/attempt_1489067586592_930490_m_002811_0/file.out.index in any of the configured local directories at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:488) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:165) at org.apache.hadoop.mapred.ShuffleHandler$Shuffle.getMapOutputInfo(ShuffleHandler.java:1000) at org.apache.hadoop.mapred.ShuffleHandler$Shuffle.populateHeaders(ShuffleHandler.java:1022) at org.apache.hadoop.mapred.ShuffleHandler$Shuffle.messageReceived(ShuffleHandler.java:908) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560) at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:787) at org.jboss.netty.handler.stream.ChunkedWriteHandler.handleUpstream(ChunkedWriteHandler.java:142) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560) at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:787) at org.jboss.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:148) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560) at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:787) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459) at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536) at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:555) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:107) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) reduce log 2017-03-25 05:40:33,919 WARN [fetcher#2] org.apache.hadoop.mapreduce.task.reduce.Fetcher: Invalid map id java.lang.IllegalArgumentException: TaskAttemptId string : TTP/1.1 500 Internal Server Error Content-Type: text/plain; charset=UTF is not properly formed at org.apache.hadoop.mapreduce.TaskAttemptID.forName(TaskAttemptID.java:201) at org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyMapOutput(Fetcher.java:485) at org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java:336) at org.apache.hadoop.mapreduce.task.reduce.Fetcher.run(Fetcher.java:193) was: nodemanager log 2017-03-25 21:07:03,071 ERROR org.apache.hadoop.mapred.ShuffleHandler: Shuffle error in populating headers : org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find
[jira] [Created] (MAPREDUCE-6869) org.apache.hadoop.mapred.ShuffleHandler: Shuffle error in populating headers :
翟玉勇 created MAPREDUCE-6869: -- Summary: org.apache.hadoop.mapred.ShuffleHandler: Shuffle error in populating headers : Key: MAPREDUCE-6869 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6869 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, yarn Affects Versions: 2.6.0 Environment: hadoop 2.6.0-cdh5.8.2 Reporter: 翟玉勇 Priority: Minor nodemanager log 2017-03-25 21:07:03,071 ERROR org.apache.hadoop.mapred.ShuffleHandler: Shuffle error in populating headers : org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find usercache/master/appcache/application_1489067586592_930490/output/attempt_1489067586592_930490_m_002811_0/file.out.index in any of the configured local directories at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:488) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:165) at org.apache.hadoop.mapred.ShuffleHandler$Shuffle.getMapOutputInfo(ShuffleHandler.java:1000) at org.apache.hadoop.mapred.ShuffleHandler$Shuffle.populateHeaders(ShuffleHandler.java:1022) at org.apache.hadoop.mapred.ShuffleHandler$Shuffle.messageReceived(ShuffleHandler.java:908) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560) at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:787) at org.jboss.netty.handler.stream.ChunkedWriteHandler.handleUpstream(ChunkedWriteHandler.java:142) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560) at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:787) at org.jboss.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:148) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560) at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:787) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459) at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536) at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:555) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:107) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-4253) Tests for mapreduce-client-core are lying under mapreduce-client-jobclient
[ https://issues.apache.org/jira/browse/MAPREDUCE-4253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15942774#comment-15942774 ] Akira Ajisaka commented on MAPREDUCE-4253: -- Hi [~ozawa], how is this issue going on? > Tests for mapreduce-client-core are lying under mapreduce-client-jobclient > -- > > Key: MAPREDUCE-4253 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4253 > Project: Hadoop Map/Reduce > Issue Type: Test > Components: client >Affects Versions: 2.0.0-alpha >Reporter: Harsh J >Assignee: Tsuyoshi Ozawa > Attachments: crossing_project_checker.rb, MR-4253.1.patch, > MR-4253.2.patch, result.txt > > > Many of the tests for client libs from mapreduce-client-core are lying under > mapreduce-client-jobclient. > We should investigate if this is the right thing to do and if not, move the > tests back into client-core. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6868) License check for jdiff output files should be ignored
[ https://issues.apache.org/jira/browse/MAPREDUCE-6868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated MAPREDUCE-6868: -- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.0.0-alpha3 2.8.1 2.9.0 Status: Resolved (was: Patch Available) I have commit the patch to trunk, branch-2 and branch-2.8. Thanks [~ajisakaa]! > License check for jdiff output files should be ignored > -- > > Key: MAPREDUCE-6868 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6868 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka > Fix For: 2.9.0, 2.8.1, 3.0.0-alpha3 > > Attachments: MAPREDUCE-6868.01.patch > > > The following commits added jdiff output for Hadoop 2.8.0 but ASF license > header is missing. > * > https://github.com/apache/hadoop/commit/6df029db36412b7219b64313dcbe1874dc1c8b0c > * > https://github.com/apache/hadoop/commit/d174c06b01e1f743d3111b9b760a9824d8106b86 > hadoop-mapreduce-project module does not have a setting to ignore the jdiff > output files, so the license check fails. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6868) License check for jdiff output files should be ignored
[ https://issues.apache.org/jira/browse/MAPREDUCE-6868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15942709#comment-15942709 ] Hudson commented on MAPREDUCE-6868: --- FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #11468 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/11468/]) MAPREDUCE-6868. License check for jdiff output files should be ignored. (junping_du: rev 945b006fe85473c1bb87830dbbfc1df9051c7bb9) * (edit) hadoop-mapreduce-project/pom.xml > License check for jdiff output files should be ignored > -- > > Key: MAPREDUCE-6868 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6868 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka > Attachments: MAPREDUCE-6868.01.patch > > > The following commits added jdiff output for Hadoop 2.8.0 but ASF license > header is missing. > * > https://github.com/apache/hadoop/commit/6df029db36412b7219b64313dcbe1874dc1c8b0c > * > https://github.com/apache/hadoop/commit/d174c06b01e1f743d3111b9b760a9824d8106b86 > hadoop-mapreduce-project module does not have a setting to ignore the jdiff > output files, so the license check fails. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6868) License check for jdiff output files should be ignored
[ https://issues.apache.org/jira/browse/MAPREDUCE-6868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15942692#comment-15942692 ] Junping Du commented on MAPREDUCE-6868: --- Thanks [~ajisakaa] for reporting the issue and fixing here. +1 on the patch. Committing it. > License check for jdiff output files should be ignored > -- > > Key: MAPREDUCE-6868 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6868 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka > Attachments: MAPREDUCE-6868.01.patch > > > The following commits added jdiff output for Hadoop 2.8.0 but ASF license > header is missing. > * > https://github.com/apache/hadoop/commit/6df029db36412b7219b64313dcbe1874dc1c8b0c > * > https://github.com/apache/hadoop/commit/d174c06b01e1f743d3111b9b760a9824d8106b86 > hadoop-mapreduce-project module does not have a setting to ignore the jdiff > output files, so the license check fails. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
