[jira] [Comment Edited] (MAPREDUCE-6190) MR Job is stuck because of one mapper stuck in STARTING
[
https://issues.apache.org/jira/browse/MAPREDUCE-6190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16700028#comment-16700028
]
Zhaohui Xin edited comment on MAPREDUCE-6190 at 11/27/18 7:44 AM:
--
In our production cluster, the new code mentioned above helps job avoid stuck
and also helps cluster maintainers find the problem nodes which maybe disk
error or network error.
All our clusters have more than 50k nodes, it is very useful to actively
discover and repair bad nodes quickly.
was (Author: uranus):
In our production cluster, the new code mentioned above help job avoid stuck
and also helps cluster maintainers find the problem nodes which maybe disk
error or network error.
All our clusters have more than 50k nodes, it is very useful to actively
discover and repair bad nodes quickly.
> MR Job is stuck because of one mapper stuck in STARTING
> ---
>
> Key: MAPREDUCE-6190
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6190
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.6.0, 2.7.0, 2.8.0, 2.9.0, 3.0.0, 3.1.1
>Reporter: Ankit Malhotra
>Assignee: Zhaohui Xin
>Priority: Major
> Attachments: MAPREDUCE-6190.001.patch, MAPREDUCE-6190.002.patch
>
>
> Trying to figure out a weird issue we started seeing on our CDH5.1.0 cluster
> with map reduce jobs on YARN.
> We had a job stuck for hours because one of the mappers never started up
> fully. Basically, the map task had 2 attempts, the first one failed and the
> AM tried to schedule a second one and the second attempt was stuck on STATE:
> STARTING, STATUS: NEW. A node never got assigned and the task along with the
> job was stuck indefinitely.
> The AM logs had this being logged again and again:
> {code}
> 2014-12-09 19:25:12,347 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Ramping down 0
> 2014-12-09 19:25:13,352 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received
> completed container container_1408745633994_450952_02_003807
> 2014-12-09 19:25:13,352 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Reduce preemption
> successful attempt_1408745633994_450952_r_48_1000
> 2014-12-09 19:25:13,352 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Ramping down all
> scheduled reduces:0
> 2014-12-09 19:25:13,352 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Going to preempt 1
> 2014-12-09 19:25:13,353 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Preempting
> attempt_1408745633994_450952_r_50_1000
> 2014-12-09 19:25:13,353 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Recalculating
> schedule, headroom=0
> 2014-12-09 19:25:13,353 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator:
> completedMapPercent 0.99968 totalMemLimit:1722880 finalMapMemLimit:2560
> finalReduceMemLimit:1720320 netScheduledMapMem:2560
> netScheduledReduceMem:1722880
> 2014-12-09 19:25:13,353 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Ramping down 0
> 2014-12-09 19:25:13,353 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling:
> PendingReds:77 ScheduledMaps:1 ScheduledReds:0 AssignedMaps:0
> AssignedReds:673 CompletedMaps:3124 CompletedReds:0 ContAlloc:4789
> ContRel:798 HostLocal:2944 RackLocal:155
> 2014-12-09 19:25:14,353 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before
> Scheduling: PendingReds:78 ScheduledMaps:1 ScheduledReds:0 AssignedMaps:0
> AssignedReds:673 CompletedMaps:3124 CompletedReds:0 ContAlloc:4789
> ContRel:798 HostLocal:2944 RackLocal:155
> 2014-12-09 19:25:14,359 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Recalculating
> schedule, headroom=0
> {code}
> On killing the task manually, the AM started up the task again, scheduled and
> ran it successfully completing the task and the job with it.
> Some quick code grepping led us here:
> http://grepcode.com/file/repo1.maven.org/maven2/org.apache.hadoop/hadoop-mapreduce-client-app/2.3.0/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java#397
> But still dont quite understand why this would happen once in a while and why
> the job would suddenly be ok once the stuck task is manually killed.
> Note: Other jobs succeed on the cluster while this job is stuck.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (MAPREDUCE-6190) MR Job is stuck because of one mapper stuck in STARTING
[
https://issues.apache.org/jira/browse/MAPREDUCE-6190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16700028#comment-16700028
]
Zhaohui Xin commented on MAPREDUCE-6190:
In our production cluster, the new code mentioned above help job avoid stuck
and also helps cluster maintainers find the problem nodes which maybe disk
error or network error.
All our clusters have more than 50k nodes, it is very useful to actively
discover and repair bad nodes quickly.
> MR Job is stuck because of one mapper stuck in STARTING
> ---
>
> Key: MAPREDUCE-6190
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6190
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.6.0, 2.7.0, 2.8.0, 2.9.0, 3.0.0, 3.1.1
>Reporter: Ankit Malhotra
>Assignee: Zhaohui Xin
>Priority: Major
> Attachments: MAPREDUCE-6190.001.patch, MAPREDUCE-6190.002.patch
>
>
> Trying to figure out a weird issue we started seeing on our CDH5.1.0 cluster
> with map reduce jobs on YARN.
> We had a job stuck for hours because one of the mappers never started up
> fully. Basically, the map task had 2 attempts, the first one failed and the
> AM tried to schedule a second one and the second attempt was stuck on STATE:
> STARTING, STATUS: NEW. A node never got assigned and the task along with the
> job was stuck indefinitely.
> The AM logs had this being logged again and again:
> {code}
> 2014-12-09 19:25:12,347 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Ramping down 0
> 2014-12-09 19:25:13,352 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received
> completed container container_1408745633994_450952_02_003807
> 2014-12-09 19:25:13,352 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Reduce preemption
> successful attempt_1408745633994_450952_r_48_1000
> 2014-12-09 19:25:13,352 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Ramping down all
> scheduled reduces:0
> 2014-12-09 19:25:13,352 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Going to preempt 1
> 2014-12-09 19:25:13,353 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Preempting
> attempt_1408745633994_450952_r_50_1000
> 2014-12-09 19:25:13,353 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Recalculating
> schedule, headroom=0
> 2014-12-09 19:25:13,353 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator:
> completedMapPercent 0.99968 totalMemLimit:1722880 finalMapMemLimit:2560
> finalReduceMemLimit:1720320 netScheduledMapMem:2560
> netScheduledReduceMem:1722880
> 2014-12-09 19:25:13,353 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Ramping down 0
> 2014-12-09 19:25:13,353 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling:
> PendingReds:77 ScheduledMaps:1 ScheduledReds:0 AssignedMaps:0
> AssignedReds:673 CompletedMaps:3124 CompletedReds:0 ContAlloc:4789
> ContRel:798 HostLocal:2944 RackLocal:155
> 2014-12-09 19:25:14,353 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before
> Scheduling: PendingReds:78 ScheduledMaps:1 ScheduledReds:0 AssignedMaps:0
> AssignedReds:673 CompletedMaps:3124 CompletedReds:0 ContAlloc:4789
> ContRel:798 HostLocal:2944 RackLocal:155
> 2014-12-09 19:25:14,359 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Recalculating
> schedule, headroom=0
> {code}
> On killing the task manually, the AM started up the task again, scheduled and
> ran it successfully completing the task and the job with it.
> Some quick code grepping led us here:
> http://grepcode.com/file/repo1.maven.org/maven2/org.apache.hadoop/hadoop-mapreduce-client-app/2.3.0/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java#397
> But still dont quite understand why this would happen once in a while and why
> the job would suddenly be ok once the stuck task is manually killed.
> Note: Other jobs succeed on the cluster while this job is stuck.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7162) MapReduce unit test is broken
[
https://issues.apache.org/jira/browse/MAPREDUCE-7162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16700017#comment-16700017
]
Zhaohui Xin commented on MAPREDUCE-7162:
[~ajisakaa], very good suggestion. I attached new patch.:D
> MapReduce unit test is broken
> -
>
> Key: MAPREDUCE-7162
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7162
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Zhaohui Xin
>Assignee: Zhaohui Xin
>Priority: Critical
> Attachments: MAPREDUCE-7162.001.patch, MAPREDUCE-7162.002.patch
>
>
> Mapreduce unit test is broken by
> https://issues.apache.org/jira/browse/MAPREDUCE-7158 .
> *I think we should keep the data consistent to avoid corruption when output,
> so I roll back the previous code and attach the patch.*
> Broken location _is
> org.apache.hadoop.mapreduce.jobhistory.TestEvents#testEvents._
> {code:java}
> org.codehaus.jackson.JsonParseException: Illegal unquoted character
> ((CTRL-CHAR, code 10)): has to be escaped using backslash to be included in
> name
> at [Source: java.io.DataInputStream@25618e91; line: 23, column: 418]
> at org.codehaus.jackson.JsonParser._constructError(JsonParser.java:1433)
> at
> org.codehaus.jackson.impl.JsonParserMinimalBase._reportError(JsonParserMinimalBase.java:521)
> at
> org.codehaus.jackson.impl.JsonParserMinimalBase._throwUnquotedSpace(JsonParserMinimalBase.java:482)
> at
> org.codehaus.jackson.impl.Utf8StreamParser.parseEscapedFieldName(Utf8StreamParser.java:1446)
> at
> org.codehaus.jackson.impl.Utf8StreamParser.parseFieldName(Utf8StreamParser.java:1410)
> at
> org.codehaus.jackson.impl.Utf8StreamParser._parseFieldName(Utf8StreamParser.java:1283)
> at
> org.codehaus.jackson.impl.Utf8StreamParser.nextToken(Utf8StreamParser.java:495)
> at org.apache.avro.io.JsonDecoder.doArrayNext(JsonDecoder.java:367)
> at org.apache.avro.io.JsonDecoder.arrayNext(JsonDecoder.java:361)
> at org.apache.avro.io.ValidatingDecoder.arrayNext(ValidatingDecoder.java:189)
> at
> org.apache.avro.generic.GenericDatumReader.readArray(GenericDatumReader.java:222)
> at
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:153)
> at
> org.apache.avro.generic.GenericDatumReader.readField(GenericDatumReader.java:193)
> at
> org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:183)
> at
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:151)
> at
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:155)
> at
> org.apache.avro.generic.GenericDatumReader.readField(GenericDatumReader.java:193)
> at
> org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:183)
> at
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:151)
> at
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:142)
> at
> org.apache.hadoop.mapreduce.jobhistory.EventReader.getNextEvent(EventReader.java:101)
> at
> org.apache.hadoop.mapreduce.jobhistory.TestEvents.testEvents(TestEvents.java:177)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7162) MapReduce unit test is broken
[
https://issues.apache.org/jira/browse/MAPREDUCE-7162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16700015#comment-16700015
]
Akira Ajisaka commented on MAPREDUCE-7162:
--
The source code before MAPREDUCE-7158 intended to insert '\n' between the event
data by calling {{encoder.flush()}} if the schema is json. MAPREDUCE-7158
dropped the call of {{encoder.flush()}} and then it is not guaranteed that '\n'
is inserted between the event data. That's why the test fails.
> MapReduce unit test is broken
> -
>
> Key: MAPREDUCE-7162
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7162
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Zhaohui Xin
>Assignee: Zhaohui Xin
>Priority: Critical
> Attachments: MAPREDUCE-7162.001.patch, MAPREDUCE-7162.002.patch
>
>
> Mapreduce unit test is broken by
> https://issues.apache.org/jira/browse/MAPREDUCE-7158 .
> *I think we should keep the data consistent to avoid corruption when output,
> so I roll back the previous code and attach the patch.*
> Broken location _is
> org.apache.hadoop.mapreduce.jobhistory.TestEvents#testEvents._
> {code:java}
> org.codehaus.jackson.JsonParseException: Illegal unquoted character
> ((CTRL-CHAR, code 10)): has to be escaped using backslash to be included in
> name
> at [Source: java.io.DataInputStream@25618e91; line: 23, column: 418]
> at org.codehaus.jackson.JsonParser._constructError(JsonParser.java:1433)
> at
> org.codehaus.jackson.impl.JsonParserMinimalBase._reportError(JsonParserMinimalBase.java:521)
> at
> org.codehaus.jackson.impl.JsonParserMinimalBase._throwUnquotedSpace(JsonParserMinimalBase.java:482)
> at
> org.codehaus.jackson.impl.Utf8StreamParser.parseEscapedFieldName(Utf8StreamParser.java:1446)
> at
> org.codehaus.jackson.impl.Utf8StreamParser.parseFieldName(Utf8StreamParser.java:1410)
> at
> org.codehaus.jackson.impl.Utf8StreamParser._parseFieldName(Utf8StreamParser.java:1283)
> at
> org.codehaus.jackson.impl.Utf8StreamParser.nextToken(Utf8StreamParser.java:495)
> at org.apache.avro.io.JsonDecoder.doArrayNext(JsonDecoder.java:367)
> at org.apache.avro.io.JsonDecoder.arrayNext(JsonDecoder.java:361)
> at org.apache.avro.io.ValidatingDecoder.arrayNext(ValidatingDecoder.java:189)
> at
> org.apache.avro.generic.GenericDatumReader.readArray(GenericDatumReader.java:222)
> at
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:153)
> at
> org.apache.avro.generic.GenericDatumReader.readField(GenericDatumReader.java:193)
> at
> org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:183)
> at
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:151)
> at
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:155)
> at
> org.apache.avro.generic.GenericDatumReader.readField(GenericDatumReader.java:193)
> at
> org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:183)
> at
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:151)
> at
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:142)
> at
> org.apache.hadoop.mapreduce.jobhistory.EventReader.getNextEvent(EventReader.java:101)
> at
> org.apache.hadoop.mapreduce.jobhistory.TestEvents.testEvents(TestEvents.java:177)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7162) MapReduce unit test is broken
[
https://issues.apache.org/jira/browse/MAPREDUCE-7162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhaohui Xin updated MAPREDUCE-7162:
---
Attachment: MAPREDUCE-7162.002.patch
> MapReduce unit test is broken
> -
>
> Key: MAPREDUCE-7162
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7162
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Zhaohui Xin
>Assignee: Zhaohui Xin
>Priority: Critical
> Attachments: MAPREDUCE-7162.001.patch, MAPREDUCE-7162.002.patch
>
>
> Mapreduce unit test is broken by
> https://issues.apache.org/jira/browse/MAPREDUCE-7158 .
> *I think we should keep the data consistent to avoid corruption when output,
> so I roll back the previous code and attach the patch.*
> Broken location _is
> org.apache.hadoop.mapreduce.jobhistory.TestEvents#testEvents._
> {code:java}
> org.codehaus.jackson.JsonParseException: Illegal unquoted character
> ((CTRL-CHAR, code 10)): has to be escaped using backslash to be included in
> name
> at [Source: java.io.DataInputStream@25618e91; line: 23, column: 418]
> at org.codehaus.jackson.JsonParser._constructError(JsonParser.java:1433)
> at
> org.codehaus.jackson.impl.JsonParserMinimalBase._reportError(JsonParserMinimalBase.java:521)
> at
> org.codehaus.jackson.impl.JsonParserMinimalBase._throwUnquotedSpace(JsonParserMinimalBase.java:482)
> at
> org.codehaus.jackson.impl.Utf8StreamParser.parseEscapedFieldName(Utf8StreamParser.java:1446)
> at
> org.codehaus.jackson.impl.Utf8StreamParser.parseFieldName(Utf8StreamParser.java:1410)
> at
> org.codehaus.jackson.impl.Utf8StreamParser._parseFieldName(Utf8StreamParser.java:1283)
> at
> org.codehaus.jackson.impl.Utf8StreamParser.nextToken(Utf8StreamParser.java:495)
> at org.apache.avro.io.JsonDecoder.doArrayNext(JsonDecoder.java:367)
> at org.apache.avro.io.JsonDecoder.arrayNext(JsonDecoder.java:361)
> at org.apache.avro.io.ValidatingDecoder.arrayNext(ValidatingDecoder.java:189)
> at
> org.apache.avro.generic.GenericDatumReader.readArray(GenericDatumReader.java:222)
> at
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:153)
> at
> org.apache.avro.generic.GenericDatumReader.readField(GenericDatumReader.java:193)
> at
> org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:183)
> at
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:151)
> at
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:155)
> at
> org.apache.avro.generic.GenericDatumReader.readField(GenericDatumReader.java:193)
> at
> org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:183)
> at
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:151)
> at
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:142)
> at
> org.apache.hadoop.mapreduce.jobhistory.EventReader.getNextEvent(EventReader.java:101)
> at
> org.apache.hadoop.mapreduce.jobhistory.TestEvents.testEvents(TestEvents.java:177)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7162) MapReduce unit test is broken
[
https://issues.apache.org/jira/browse/MAPREDUCE-7162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka updated MAPREDUCE-7162:
-
Target Version/s: 3.1.2, 3.3.0, 3.2.1
> MapReduce unit test is broken
> -
>
> Key: MAPREDUCE-7162
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7162
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Zhaohui Xin
>Assignee: Zhaohui Xin
>Priority: Critical
> Attachments: MAPREDUCE-7162.001.patch
>
>
> Mapreduce unit test is broken by
> https://issues.apache.org/jira/browse/MAPREDUCE-7158 .
> *I think we should keep the data consistent to avoid corruption when output,
> so I roll back the previous code and attach the patch.*
> Broken location _is
> org.apache.hadoop.mapreduce.jobhistory.TestEvents#testEvents._
> {code:java}
> org.codehaus.jackson.JsonParseException: Illegal unquoted character
> ((CTRL-CHAR, code 10)): has to be escaped using backslash to be included in
> name
> at [Source: java.io.DataInputStream@25618e91; line: 23, column: 418]
> at org.codehaus.jackson.JsonParser._constructError(JsonParser.java:1433)
> at
> org.codehaus.jackson.impl.JsonParserMinimalBase._reportError(JsonParserMinimalBase.java:521)
> at
> org.codehaus.jackson.impl.JsonParserMinimalBase._throwUnquotedSpace(JsonParserMinimalBase.java:482)
> at
> org.codehaus.jackson.impl.Utf8StreamParser.parseEscapedFieldName(Utf8StreamParser.java:1446)
> at
> org.codehaus.jackson.impl.Utf8StreamParser.parseFieldName(Utf8StreamParser.java:1410)
> at
> org.codehaus.jackson.impl.Utf8StreamParser._parseFieldName(Utf8StreamParser.java:1283)
> at
> org.codehaus.jackson.impl.Utf8StreamParser.nextToken(Utf8StreamParser.java:495)
> at org.apache.avro.io.JsonDecoder.doArrayNext(JsonDecoder.java:367)
> at org.apache.avro.io.JsonDecoder.arrayNext(JsonDecoder.java:361)
> at org.apache.avro.io.ValidatingDecoder.arrayNext(ValidatingDecoder.java:189)
> at
> org.apache.avro.generic.GenericDatumReader.readArray(GenericDatumReader.java:222)
> at
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:153)
> at
> org.apache.avro.generic.GenericDatumReader.readField(GenericDatumReader.java:193)
> at
> org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:183)
> at
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:151)
> at
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:155)
> at
> org.apache.avro.generic.GenericDatumReader.readField(GenericDatumReader.java:193)
> at
> org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:183)
> at
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:151)
> at
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:142)
> at
> org.apache.hadoop.mapreduce.jobhistory.EventReader.getNextEvent(EventReader.java:101)
> at
> org.apache.hadoop.mapreduce.jobhistory.TestEvents.testEvents(TestEvents.java:177)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7162) MapReduce unit test is broken
[
https://issues.apache.org/jira/browse/MAPREDUCE-7162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1676#comment-1676
]
Akira Ajisaka commented on MAPREDUCE-7162:
--
Hi [~uranus], thank you for your patch.
I'm thinking {{encoder.flush()}} is required only if the schema is json. Would
you add {{encoder.flush()}} in the if clause?
> MapReduce unit test is broken
> -
>
> Key: MAPREDUCE-7162
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7162
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Zhaohui Xin
>Assignee: Zhaohui Xin
>Priority: Critical
> Attachments: MAPREDUCE-7162.001.patch
>
>
> Mapreduce unit test is broken by
> https://issues.apache.org/jira/browse/MAPREDUCE-7158 .
> *I think we should keep the data consistent to avoid corruption when output,
> so I roll back the previous code and attach the patch.*
> Broken location _is
> org.apache.hadoop.mapreduce.jobhistory.TestEvents#testEvents._
> {code:java}
> org.codehaus.jackson.JsonParseException: Illegal unquoted character
> ((CTRL-CHAR, code 10)): has to be escaped using backslash to be included in
> name
> at [Source: java.io.DataInputStream@25618e91; line: 23, column: 418]
> at org.codehaus.jackson.JsonParser._constructError(JsonParser.java:1433)
> at
> org.codehaus.jackson.impl.JsonParserMinimalBase._reportError(JsonParserMinimalBase.java:521)
> at
> org.codehaus.jackson.impl.JsonParserMinimalBase._throwUnquotedSpace(JsonParserMinimalBase.java:482)
> at
> org.codehaus.jackson.impl.Utf8StreamParser.parseEscapedFieldName(Utf8StreamParser.java:1446)
> at
> org.codehaus.jackson.impl.Utf8StreamParser.parseFieldName(Utf8StreamParser.java:1410)
> at
> org.codehaus.jackson.impl.Utf8StreamParser._parseFieldName(Utf8StreamParser.java:1283)
> at
> org.codehaus.jackson.impl.Utf8StreamParser.nextToken(Utf8StreamParser.java:495)
> at org.apache.avro.io.JsonDecoder.doArrayNext(JsonDecoder.java:367)
> at org.apache.avro.io.JsonDecoder.arrayNext(JsonDecoder.java:361)
> at org.apache.avro.io.ValidatingDecoder.arrayNext(ValidatingDecoder.java:189)
> at
> org.apache.avro.generic.GenericDatumReader.readArray(GenericDatumReader.java:222)
> at
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:153)
> at
> org.apache.avro.generic.GenericDatumReader.readField(GenericDatumReader.java:193)
> at
> org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:183)
> at
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:151)
> at
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:155)
> at
> org.apache.avro.generic.GenericDatumReader.readField(GenericDatumReader.java:193)
> at
> org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:183)
> at
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:151)
> at
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:142)
> at
> org.apache.hadoop.mapreduce.jobhistory.EventReader.getNextEvent(EventReader.java:101)
> at
> org.apache.hadoop.mapreduce.jobhistory.TestEvents.testEvents(TestEvents.java:177)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (MAPREDUCE-6190) MR Job is stuck because of one mapper stuck in STARTING
[
https://issues.apache.org/jira/browse/MAPREDUCE-6190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16698105#comment-16698105
]
Zhaohui Xin edited comment on MAPREDUCE-6190 at 11/27/18 2:44 AM:
--
Hi [~ajisakaa], [~jianhe], can you help me review this patch? :)
was (Author: uranus):
Hi [~ajisakaa], can you help me review this patch? :)
> MR Job is stuck because of one mapper stuck in STARTING
> ---
>
> Key: MAPREDUCE-6190
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6190
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.6.0, 2.7.0, 2.8.0, 2.9.0, 3.0.0, 3.1.1
>Reporter: Ankit Malhotra
>Assignee: Zhaohui Xin
>Priority: Major
> Attachments: MAPREDUCE-6190.001.patch, MAPREDUCE-6190.002.patch
>
>
> Trying to figure out a weird issue we started seeing on our CDH5.1.0 cluster
> with map reduce jobs on YARN.
> We had a job stuck for hours because one of the mappers never started up
> fully. Basically, the map task had 2 attempts, the first one failed and the
> AM tried to schedule a second one and the second attempt was stuck on STATE:
> STARTING, STATUS: NEW. A node never got assigned and the task along with the
> job was stuck indefinitely.
> The AM logs had this being logged again and again:
> {code}
> 2014-12-09 19:25:12,347 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Ramping down 0
> 2014-12-09 19:25:13,352 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received
> completed container container_1408745633994_450952_02_003807
> 2014-12-09 19:25:13,352 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Reduce preemption
> successful attempt_1408745633994_450952_r_48_1000
> 2014-12-09 19:25:13,352 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Ramping down all
> scheduled reduces:0
> 2014-12-09 19:25:13,352 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Going to preempt 1
> 2014-12-09 19:25:13,353 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Preempting
> attempt_1408745633994_450952_r_50_1000
> 2014-12-09 19:25:13,353 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Recalculating
> schedule, headroom=0
> 2014-12-09 19:25:13,353 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator:
> completedMapPercent 0.99968 totalMemLimit:1722880 finalMapMemLimit:2560
> finalReduceMemLimit:1720320 netScheduledMapMem:2560
> netScheduledReduceMem:1722880
> 2014-12-09 19:25:13,353 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Ramping down 0
> 2014-12-09 19:25:13,353 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling:
> PendingReds:77 ScheduledMaps:1 ScheduledReds:0 AssignedMaps:0
> AssignedReds:673 CompletedMaps:3124 CompletedReds:0 ContAlloc:4789
> ContRel:798 HostLocal:2944 RackLocal:155
> 2014-12-09 19:25:14,353 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before
> Scheduling: PendingReds:78 ScheduledMaps:1 ScheduledReds:0 AssignedMaps:0
> AssignedReds:673 CompletedMaps:3124 CompletedReds:0 ContAlloc:4789
> ContRel:798 HostLocal:2944 RackLocal:155
> 2014-12-09 19:25:14,359 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Recalculating
> schedule, headroom=0
> {code}
> On killing the task manually, the AM started up the task again, scheduled and
> ran it successfully completing the task and the job with it.
> Some quick code grepping led us here:
> http://grepcode.com/file/repo1.maven.org/maven2/org.apache.hadoop/hadoop-mapreduce-client-app/2.3.0/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java#397
> But still dont quite understand why this would happen once in a while and why
> the job would suddenly be ok once the stuck task is manually killed.
> Note: Other jobs succeed on the cluster while this job is stuck.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7164) FileOutputCommitter does not report progress while merging paths.
[
https://issues.apache.org/jira/browse/MAPREDUCE-7164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699689#comment-16699689
]
Hadoop QA commented on MAPREDUCE-7164:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
12s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 21s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
15s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 38s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m
21s{color} | {color:green} hadoop-mapreduce-client-core in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 52m 58s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | MAPREDUCE-7164 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12949559/MAPREDUCE-7164.001.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux fd387671cc9e 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2
17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 0d84061 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7542/testReport/ |
| Max. process+thread count | 1573 (vs. ulimit of 1) |
| modules | C:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
U:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7542/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> FileOutputCommitter does not report
[jira] [Updated] (MAPREDUCE-7164) FileOutputCommitter does not report progress while merging paths.
[ https://issues.apache.org/jira/browse/MAPREDUCE-7164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kuhu Shukla updated MAPREDUCE-7164: --- Attachment: MAPREDUCE-7164.001.patch > FileOutputCommitter does not report progress while merging paths. > - > > Key: MAPREDUCE-7164 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7164 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Affects Versions: 3.0.3, 2.8.5, 2.9.2 >Reporter: Kuhu Shukla >Assignee: Kuhu Shukla >Priority: Major > Attachments: MAPREDUCE-7164.001.patch > > > In cases where the rename and merge path logic takes more time than usual, > the committer does not report progress and can cause job failure. This > behavior was not present in Hadoop 1.x. This JIRA will fix it so that the old > behavior for 1.x is restored. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7164) FileOutputCommitter does not report progress while merging paths.
[ https://issues.apache.org/jira/browse/MAPREDUCE-7164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kuhu Shukla updated MAPREDUCE-7164: --- Status: Patch Available (was: Open) > FileOutputCommitter does not report progress while merging paths. > - > > Key: MAPREDUCE-7164 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7164 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Affects Versions: 2.9.2, 2.8.5, 3.0.3 >Reporter: Kuhu Shukla >Assignee: Kuhu Shukla >Priority: Major > Attachments: MAPREDUCE-7164.001.patch > > > In cases where the rename and merge path logic takes more time than usual, > the committer does not report progress and can cause job failure. This > behavior was not present in Hadoop 1.x. This JIRA will fix it so that the old > behavior for 1.x is restored. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7164) FileOutputCommitter does not report progress while merging paths.
[ https://issues.apache.org/jira/browse/MAPREDUCE-7164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699631#comment-16699631 ] Kuhu Shukla commented on MAPREDUCE-7164: A patch that does not fully cover all cases of fileoutput committer calling into mergePaths but does add the taskAttemptContext as a member. > FileOutputCommitter does not report progress while merging paths. > - > > Key: MAPREDUCE-7164 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7164 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Affects Versions: 3.0.3, 2.8.5, 2.9.2 >Reporter: Kuhu Shukla >Assignee: Kuhu Shukla >Priority: Major > Attachments: MAPREDUCE-7164.001.patch > > > In cases where the rename and merge path logic takes more time than usual, > the committer does not report progress and can cause job failure. This > behavior was not present in Hadoop 1.x. This JIRA will fix it so that the old > behavior for 1.x is restored. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7164) FileOutputCommitter does not report progress while merging paths.
[ https://issues.apache.org/jira/browse/MAPREDUCE-7164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699606#comment-16699606 ] Kuhu Shukla commented on MAPREDUCE-7164: Would it be useful to add progress() functionality to JobContext , similar to the TaskAttemptContext? If not, the change to mergePaths() simply calling progress gets tricky as there are paths like commitJobInternal() that only have the job context to pass on to this method. I do not think it would be ok to report progress only when the task context is being passed. I can do an instanceof check but that seems like a bad choice. > FileOutputCommitter does not report progress while merging paths. > - > > Key: MAPREDUCE-7164 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7164 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Affects Versions: 3.0.3, 2.8.5, 2.9.2 >Reporter: Kuhu Shukla >Assignee: Kuhu Shukla >Priority: Major > > In cases where the rename and merge path logic takes more time than usual, > the committer does not report progress and can cause job failure. This > behavior was not present in Hadoop 1.x. This JIRA will fix it so that the old > behavior for 1.x is restored. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7164) FileOutputCommitter does not report progress while merging paths.
[ https://issues.apache.org/jira/browse/MAPREDUCE-7164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kuhu Shukla updated MAPREDUCE-7164: --- Description: In cases where the rename and merge path logic takes more time than usual, the committer does not report progress and can cause job failure. This behavior was not present in Hadoop 1.x. This JIRA will fix it so that the old behavior for 1.x is restored. > FileOutputCommitter does not report progress while merging paths. > - > > Key: MAPREDUCE-7164 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7164 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Affects Versions: 3.0.3, 2.8.5, 2.9.2 >Reporter: Kuhu Shukla >Assignee: Kuhu Shukla >Priority: Major > > In cases where the rename and merge path logic takes more time than usual, > the committer does not report progress and can cause job failure. This > behavior was not present in Hadoop 1.x. This JIRA will fix it so that the old > behavior for 1.x is restored. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Created] (MAPREDUCE-7164) FileOutputCommitter does not report progress while merging paths.
Kuhu Shukla created MAPREDUCE-7164: -- Summary: FileOutputCommitter does not report progress while merging paths. Key: MAPREDUCE-7164 URL: https://issues.apache.org/jira/browse/MAPREDUCE-7164 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 2.9.2, 2.8.5, 3.0.3 Environment: In cases where the rename and merge path logic takes more time than usual, the committer does not report progress and can cause job failure. This behavior was not present in Hadoop 1.x. This JIRA will fix it so that the old behavior for 1.x is restored. Reporter: Kuhu Shukla Assignee: Kuhu Shukla -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7164) FileOutputCommitter does not report progress while merging paths.
[ https://issues.apache.org/jira/browse/MAPREDUCE-7164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kuhu Shukla updated MAPREDUCE-7164: --- Environment: (was: In cases where the rename and merge path logic takes more time than usual, the committer does not report progress and can cause job failure. This behavior was not present in Hadoop 1.x. This JIRA will fix it so that the old behavior for 1.x is restored.) > FileOutputCommitter does not report progress while merging paths. > - > > Key: MAPREDUCE-7164 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7164 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Affects Versions: 3.0.3, 2.8.5, 2.9.2 >Reporter: Kuhu Shukla >Assignee: Kuhu Shukla >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7160) Job end notification not sent and client service not stopped after YarnRuntimeException in shutDownJob in MRAppMaster.java despite finally block
[
https://issues.apache.org/jira/browse/MAPREDUCE-7160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Thomason updated MAPREDUCE-7160:
-
Description:
If a YarnRuntimeException occurs during shutDownJob (in this case, at the line
'MRAppMaster.this.stop();') in MRAppMaster.java, the try/catch block will catch
it but the 'finally' block will not get executed. Additionally, the client
service should be stopped if an exception occurs and is not currently.
For example, if this error occurs:
{code:java}
2018-11-20 12:38:46,173 INFO [Thread-171]
org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: In stop, writing
event TASK_FINISHED
2018-11-20 12:38:46,173 ERROR [Thread-171]
org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Error writing
History Event:
org.apache.hadoop.mapreduce.jobhistory.TaskFinishedEvent@3b1a7bb5
java.io.IOException: All datanodes
[DatanodeInfoWithStorage[10.11.1.227:9866,DS-3539e7b8-5d87-45e5-880a-1897f11577d2,DISK]]
are bad. Aborting...
at
org.apache.hadoop.hdfs.DataStreamer.handleBadDatanode(DataStreamer.java:1561)
at
org.apache.hadoop.hdfs.DataStreamer.setupPipelineInternal(DataStreamer.java:1495)
at
org.apache.hadoop.hdfs.DataStreamer.setupPipelineForAppendOrRecovery(DataStreamer.java:1481)
at
org.apache.hadoop.hdfs.DataStreamer.processDatanodeOrExternalError(DataStreamer.java:1256)
at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:667) 2018-11-20
12:38:46,174 INFO [Thread-171] org.apache.hadoop.service.AbstractService:
Service JobHistoryEventHandler failed in state STOPPED
{code}
Then the line
{code:java}
catch (Throwable t) {
LOG.warn("Graceful stop failed. Exiting.. ", t);
exitMRAppMaster(1, t);
}
{code}
causes the MRAppMaster to exit before the following finally block can execute.
This means that the job end notification is not sent despite
[https://jira.apache.org/jira/browse/MAPREDUCE-6895?attachmentOrder=desc].
Additionally, this finally block should also call 'clientService.stop();', as
otherwise the following errors occur on the client side:
{code:java}
2018-11-07 19:50:59,602 INFO mapreduce.Job: map 100% reduce 98%
2018-11-07 19:51:03,617 INFO mapreduce.Job: map 100% reduce 99%
2018-11-07 19:51:07,648 INFO mapreduce.Job: map 100% reduce 100%
2018-11-07 19:51:07,656 INFO mapreduce.Job: Job job_1541647829228_0001
completed successfully
2018-11-07 19:51:08,162 INFO mapred.ClientServiceDelegate: Application state is
completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server
2018-11-07 19:51:09,169 INFO ipc.Client: Retrying connect to server:
0.0.0.0/0.0.0.0:10020. Already tried 0 time(s); retry policy is
RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2018-11-07 19:51:10,170 INFO ipc.Client: Retrying connect to server:
0.0.0.0/0.0.0.0:10020. Already tried 1 time(s); retry policy is
RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2018-11-07 19:51:11,171 INFO ipc.Client: Retrying connect to server:
0.0.0.0/0.0.0.0:10020. Already tried 2 time(s); retry policy is
RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
..
2018-11-07 19:51:38,407 INFO ipc.Client: Retrying connect to server:
0.0.0.0/0.0.0.0:10020. Already tried 9 time(s); retry policy is
RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
java.io.IOException: java.net.ConnectException: Your endpoint configuration is
wrong; For more details see: http://wiki.apache.org/hadoop/UnsetHostnameOrPort
at
org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:344)
at
org.apache.hadoop.mapred.ClientServiceDelegate.getJobCounters(ClientServiceDelegate.java:382)
at org.apache.hadoop.mapred.YARNRunner.getJobCounters(YARNRunner.java:859) at
org.apache.hadoop.mapreduce.Job$8.run(Job.java:820) at
org.apache.hadoop.mapreduce.Job$8.run(Job.java:817) at
java.security.AccessController.doPrivileged(Native Method) at
javax.security.auth.Subject.doAs(Subject.java:422) at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1686)
at org.apache.hadoop.mapreduce.Job.getCounters(Job.java:817) at
org.apache.hadoop.mapreduce.Job.monitorAndPrintJob(Job.java:1663) at
org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1591) at
org.apache.hadoop.examples.terasort.TeraSort.run(TeraSort.java:334) at
org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at
org.apache.hadoop.examples.terasort.TeraSort.main(TeraSort.java:343) at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498) at
[jira] [Updated] (MAPREDUCE-7160) Job end notification not sent and client service not stopped after YarnRuntimeException in shutDownJob in MRAppMaster.java despite finally block
[
https://issues.apache.org/jira/browse/MAPREDUCE-7160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Thomason updated MAPREDUCE-7160:
-
Description:
If a YarnRuntimeException occurs during shutDownJob (in this case, at the line
'MRAppMaster.this.stop();') in MRAppMaster.java, the try/catch block will catch
it but the 'finally' block will not get executed. Additionally, the client
service should be stopped if an exception occurs and is not currently.
For example, if this error occurs:
{code:java}
2018-11-20 12:38:46,173 INFO [Thread-171]
org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: In stop, writing
event TASK_FINISHED
2018-11-20 12:38:46,173 ERROR [Thread-171]
org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Error writing
History Event:
org.apache.hadoop.mapreduce.jobhistory.TaskFinishedEvent@3b1a7bb5
java.io.IOException: All datanodes
[DatanodeInfoWithStorage[10.11.1.227:9866,DS-3539e7b8-5d87-45e5-880a-1897f11577d2,DISK]]
are bad. Aborting...
at
org.apache.hadoop.hdfs.DataStreamer.handleBadDatanode(DataStreamer.java:1561)
at
org.apache.hadoop.hdfs.DataStreamer.setupPipelineInternal(DataStreamer.java:1495)
at
org.apache.hadoop.hdfs.DataStreamer.setupPipelineForAppendOrRecovery(DataStreamer.java:1481)
at
org.apache.hadoop.hdfs.DataStreamer.processDatanodeOrExternalError(DataStreamer.java:1256)
at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:667) 2018-11-20
12:38:46,174 INFO [Thread-171] org.apache.hadoop.service.AbstractService:
Service JobHistoryEventHandler failed in state STOPPED
{code}
Then the line
{code:java}
catch (Throwable t) {
LOG.warn("Graceful stop failed. Exiting.. ", t);
exitMRAppMaster(1, t);
}
{code}
causes the MRAppMaster to exit before the following finally block can execute.
This means that the job end notification is not sent despite
[https://jira.apache.org/jira/browse/MAPREDUCE-6895?attachmentOrder=desc].
Additionally, this finally block should also call 'clientService.stop();', as
otherwise the following errors occur on the client side:
{code:java}
2018-11-07 19:50:59,602 INFO mapreduce.Job: map 100% reduce 98%
2018-11-07 19:51:03,617 INFO mapreduce.Job: map 100% reduce 99%
2018-11-07 19:51:07,648 INFO mapreduce.Job: map 100% reduce 100%
2018-11-07 19:51:07,656 INFO mapreduce.Job: Job job_1541647829228_0001
completed successfully
2018-11-07 19:51:08,162 INFO mapred.ClientServiceDelegate: Application state is
completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server
2018-11-07 19:51:09,169 INFO ipc.Client: Retrying connect to server:
0.0.0.0/0.0.0.0:10020. Already tried 0 time(s); retry policy is
RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2018-11-07 19:51:10,170 INFO ipc.Client: Retrying connect to server:
0.0.0.0/0.0.0.0:10020. Already tried 1 time(s); retry policy is
RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2018-11-07 19:51:11,171 INFO ipc.Client: Retrying connect to server:
0.0.0.0/0.0.0.0:10020. Already tried 2 time(s); retry policy is
RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
..
2018-11-07 19:51:38,407 INFO ipc.Client: Retrying connect to server:
0.0.0.0/0.0.0.0:10020. Already tried 9 time(s); retry policy is
RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
java.io.IOException: java.net.ConnectException: Your endpoint configuration is
wrong; For more details see: http://wiki.apache.org/hadoop/UnsetHostnameOrPort
at
org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:344)
at
org.apache.hadoop.mapred.ClientServiceDelegate.getJobCounters(ClientServiceDelegate.java:382)
at org.apache.hadoop.mapred.YARNRunner.getJobCounters(YARNRunner.java:859) at
org.apache.hadoop.mapreduce.Job$8.run(Job.java:820) at
org.apache.hadoop.mapreduce.Job$8.run(Job.java:817) at
java.security.AccessController.doPrivileged(Native Method) at
javax.security.auth.Subject.doAs(Subject.java:422) at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1686)
at org.apache.hadoop.mapreduce.Job.getCounters(Job.java:817) at
org.apache.hadoop.mapreduce.Job.monitorAndPrintJob(Job.java:1663) at
org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1591) at
org.apache.hadoop.examples.terasort.TeraSort.run(TeraSort.java:334) at
org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at
org.apache.hadoop.examples.terasort.TeraSort.main(TeraSort.java:343) at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498) at
[jira] [Commented] (MAPREDUCE-7163) MapReduce add NullInputFormat
[
https://issues.apache.org/jira/browse/MAPREDUCE-7163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699217#comment-16699217
]
Hadoop QA commented on MAPREDUCE-7163:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
40s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 33s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
22s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 23s{color} | {color:orange}
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core:
The patch generated 7 new + 0 unchanged - 0 fixed = 7 total (was 0) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 34s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m
24s{color} | {color:green} hadoop-mapreduce-client-core in the patch passed.
{color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
26s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 57m 55s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | MAPREDUCE-7163 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12949510/MAPREDUCE-7163.001.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 5b886f41903b 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2
17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / b098281 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7541/artifact/out/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7541/testReport/ |
| asflicense |
[jira] [Updated] (MAPREDUCE-7163) MapReduce add NullInputFormat
[
https://issues.apache.org/jira/browse/MAPREDUCE-7163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhaohui Xin updated MAPREDUCE-7163:
---
Description:
When job's inputformat set to NullInputFormat, we can use mapreduce as
distributed shell.
{code:java}
hadoop jar hadoop-streaming-xxx.jar \
-D mapreduce.job.name=distributed_shell \
-D mapreduce.job.maps=100 \
-D mapred.reduce.tasks=0 \
-inputformat org.apache.hadoop.mapred.NullInputFormat \
-input /user/test/in \
-output /user/test/out \
-mapper shell.sh
{code}
was:
When job's inputformat set to NullInputFormat, we can use mapreduce as
distributed shell.
> MapReduce add NullInputFormat
> -
>
> Key: MAPREDUCE-7163
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7163
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
>Reporter: Zhaohui Xin
>Assignee: Zhaohui Xin
>Priority: Minor
> Attachments: MAPREDUCE-7163.001.patch
>
>
> When job's inputformat set to NullInputFormat, we can use mapreduce as
> distributed shell.
>
> {code:java}
> hadoop jar hadoop-streaming-xxx.jar \
> -D mapreduce.job.name=distributed_shell \
> -D mapreduce.job.maps=100 \
> -D mapred.reduce.tasks=0 \
> -inputformat org.apache.hadoop.mapred.NullInputFormat \
> -input /user/test/in \
> -output /user/test/out \
> -mapper shell.sh
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7163) MapReduce add NullInputFormat
[
https://issues.apache.org/jira/browse/MAPREDUCE-7163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhaohui Xin updated MAPREDUCE-7163:
---
Description:
When job's inputformat set to NullInputFormat, we can use mapreduce as
distributed shell. The following is an example,
{code:java}
hadoop jar hadoop-streaming-xxx.jar \
-D mapreduce.job.name=distributed_shell \
-D mapreduce.job.maps=100 \
-D mapred.reduce.tasks=0 \
-inputformat org.apache.hadoop.mapred.NullInputFormat \
-input /user/test/in \
-output /user/test/out \
-mapper shell.sh
{code}
was:
When job's inputformat set to NullInputFormat, we can use mapreduce as
distributed shell.
{code:java}
hadoop jar hadoop-streaming-xxx.jar \
-D mapreduce.job.name=distributed_shell \
-D mapreduce.job.maps=100 \
-D mapred.reduce.tasks=0 \
-inputformat org.apache.hadoop.mapred.NullInputFormat \
-input /user/test/in \
-output /user/test/out \
-mapper shell.sh
{code}
> MapReduce add NullInputFormat
> -
>
> Key: MAPREDUCE-7163
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7163
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
>Reporter: Zhaohui Xin
>Assignee: Zhaohui Xin
>Priority: Minor
> Attachments: MAPREDUCE-7163.001.patch
>
>
> When job's inputformat set to NullInputFormat, we can use mapreduce as
> distributed shell. The following is an example,
> {code:java}
> hadoop jar hadoop-streaming-xxx.jar \
> -D mapreduce.job.name=distributed_shell \
> -D mapreduce.job.maps=100 \
> -D mapred.reduce.tasks=0 \
> -inputformat org.apache.hadoop.mapred.NullInputFormat \
> -input /user/test/in \
> -output /user/test/out \
> -mapper shell.sh
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7163) MapReduce add NullInputFormat
[ https://issues.apache.org/jira/browse/MAPREDUCE-7163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhaohui Xin updated MAPREDUCE-7163: --- Attachment: MAPREDUCE-7163.001.patch Status: Patch Available (was: Open) > MapReduce add NullInputFormat > - > > Key: MAPREDUCE-7163 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7163 > Project: Hadoop Map/Reduce > Issue Type: New Feature >Reporter: Zhaohui Xin >Assignee: Zhaohui Xin >Priority: Minor > Attachments: MAPREDUCE-7163.001.patch > > > When job's inputformat set to NullInputFormat, we can use mapreduce as > distributed shell. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Created] (MAPREDUCE-7163) MapReduce add NullInputFormat
Zhaohui Xin created MAPREDUCE-7163: -- Summary: MapReduce add NullInputFormat Key: MAPREDUCE-7163 URL: https://issues.apache.org/jira/browse/MAPREDUCE-7163 Project: Hadoop Map/Reduce Issue Type: New Feature Reporter: Zhaohui Xin Assignee: Zhaohui Xin When job's inputformat set to NullInputFormat, we can use mapreduce as distributed shell. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7162) MapReduce unit test is broken
[
https://issues.apache.org/jira/browse/MAPREDUCE-7162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhaohui Xin updated MAPREDUCE-7162:
---
Description:
Mapreduce unit test is broken by
https://issues.apache.org/jira/browse/MAPREDUCE-7158 .
*I think we should keep the data consistent to avoid corruption when output, so
I roll back the previous code and attach the patch.*
Broken location _is
org.apache.hadoop.mapreduce.jobhistory.TestEvents#testEvents._
{code:java}
org.codehaus.jackson.JsonParseException: Illegal unquoted character
((CTRL-CHAR, code 10)): has to be escaped using backslash to be included in name
at [Source: java.io.DataInputStream@25618e91; line: 23, column: 418]
at org.codehaus.jackson.JsonParser._constructError(JsonParser.java:1433)
at
org.codehaus.jackson.impl.JsonParserMinimalBase._reportError(JsonParserMinimalBase.java:521)
at
org.codehaus.jackson.impl.JsonParserMinimalBase._throwUnquotedSpace(JsonParserMinimalBase.java:482)
at
org.codehaus.jackson.impl.Utf8StreamParser.parseEscapedFieldName(Utf8StreamParser.java:1446)
at
org.codehaus.jackson.impl.Utf8StreamParser.parseFieldName(Utf8StreamParser.java:1410)
at
org.codehaus.jackson.impl.Utf8StreamParser._parseFieldName(Utf8StreamParser.java:1283)
at
org.codehaus.jackson.impl.Utf8StreamParser.nextToken(Utf8StreamParser.java:495)
at org.apache.avro.io.JsonDecoder.doArrayNext(JsonDecoder.java:367)
at org.apache.avro.io.JsonDecoder.arrayNext(JsonDecoder.java:361)
at org.apache.avro.io.ValidatingDecoder.arrayNext(ValidatingDecoder.java:189)
at
org.apache.avro.generic.GenericDatumReader.readArray(GenericDatumReader.java:222)
at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:153)
at
org.apache.avro.generic.GenericDatumReader.readField(GenericDatumReader.java:193)
at
org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:183)
at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:151)
at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:155)
at
org.apache.avro.generic.GenericDatumReader.readField(GenericDatumReader.java:193)
at
org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:183)
at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:151)
at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:142)
at
org.apache.hadoop.mapreduce.jobhistory.EventReader.getNextEvent(EventReader.java:101)
at
org.apache.hadoop.mapreduce.jobhistory.TestEvents.testEvents(TestEvents.java:177)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.lang.Thread.run(Thread.java:748)
{code}
was:
Mapreduce unit test is broken by
https://issues.apache.org/jira/browse/MAPREDUCE-7158 .
*I think we should keep the data consistent to avoid corruption when output, so
I roll back the previous code. Please see the patch.*
Broken location _is
org.apache.hadoop.mapreduce.jobhistory.TestEvents#testEvents._
{code:java}
org.codehaus.jackson.JsonParseException: Illegal unquoted character
((CTRL-CHAR, code 10)): has to be escaped using backslash to be included in name
at [Source: java.io.DataInputStream@25618e91; line: 23, column: 418]
at org.codehaus.jackson.JsonParser._constructError(JsonParser.java:1433)
at
org.codehaus.jackson.impl.JsonParserMinimalBase._reportError(JsonParserMinimalBase.java:521)
at
org.codehaus.jackson.impl.JsonParserMinimalBase._throwUnquotedSpace(JsonParserMinimalBase.java:482)
at
org.codehaus.jackson.impl.Utf8StreamParser.parseEscapedFieldName(Utf8StreamParser.java:1446)
at
org.codehaus.jackson.impl.Utf8StreamParser.parseFieldName(Utf8StreamParser.java:1410)
at
org.codehaus.jackson.impl.Utf8StreamParser._parseFieldName(Utf8StreamParser.java:1283)
at
org.codehaus.jackson.impl.Utf8StreamParser.nextToken(Utf8StreamParser.java:495)
at org.apache.avro.io.JsonDecoder.doArrayNext(JsonDecoder.java:367)
at org.apache.avro.io.JsonDecoder.arrayNext(JsonDecoder.java:361)
at org.apache.avro.io.ValidatingDecoder.arrayNext(ValidatingDecoder.java:189)
at
[jira] [Commented] (MAPREDUCE-7159) FrameworkUploader: ensure proper permissions of generated framework tar.gz if restrictive umask is used
[ https://issues.apache.org/jira/browse/MAPREDUCE-7159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16698672#comment-16698672 ] Peter Bacsko commented on MAPREDUCE-7159: - [~wilfreds] could you take a quick look at this? Thanks. > FrameworkUploader: ensure proper permissions of generated framework tar.gz if > restrictive umask is used > --- > > Key: MAPREDUCE-7159 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7159 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 3.1.1 >Reporter: Peter Bacsko >Assignee: Peter Bacsko >Priority: Major > Attachments: MAPREDUCE-7159-001.patch, MAPREDUCE-7159-002.patch, > MAPREDUCE-7159-003.patch > > > Using certain umask values (like 027) makes files unreadable to "others". > This causes problems if the FrameworkUploader > (https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-uploader/src/main/java/org/apache/hadoop/mapred/uploader/FrameworkUploader.java) > is used - it's necessary that the compressed MR framework is readable by all > users, otherwise they won't be able to run MR jobs. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
