[jira] [Updated] (MAPREDUCE-6577) MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set
[
https://issues.apache.org/jira/browse/MAPREDUCE-6577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Junping Du updated MAPREDUCE-6577:
--
Fix Version/s: (was: 2.9.0)
> MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set
>
>
> Key: MAPREDUCE-6577
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6577
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mr-am
>Affects Versions: 2.6.0
>Reporter: Sangjin Lee
>Assignee: Sangjin Lee
>Priority: Critical
> Fix For: 2.8.0, 2.7.3, 2.6.4, 3.0.0-alpha1
>
> Attachments: MAPREDUCE-6577.01.patch, MAPREDUCE-6577.02.patch,
> MAPREDUCE-6577.03.patch, MAPREDUCE-6577.04.patch
>
>
> If yarn.app.mapreduce.am.admin.user.env (or yarn.app.mapreduce.am.env) is not
> configured to set LD_LIBRARY_PATH, MR AM will fail to load the native library:
> {noformat}
> 2015-12-15 21:29:22,473 WARN [main] org.apache.hadoop.util.NativeCodeLoader:
> Unable to load native-hadoop library for your platform... using builtin-java
> classes where applicable
> {noformat}
> As a result, any code that needs the hadoop native library in the MR AM will
> fail. For example, an uber-AM with lz4 compression for the mapper task will
> fail:
> {noformat}
> 2015-12-15 21:30:17,575 WARN [uber-SubtaskRunner]
> org.apache.hadoop.mapred.LocalContainerLauncher: Exception running local
> (uberized) 'child' : java.lang.RuntimeException: native lz4 library not
> available
> at
> org.apache.hadoop.io.compress.Lz4Codec.getCompressorType(Lz4Codec.java:125)
> at
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:148)
> at
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:163)
> at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:114)
> at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:97)
> at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1602)
> at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1482)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:457)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
> at
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:391)
> at
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:309)
> at
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200(LocalContainerLauncher.java:195)
> at
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(LocalContainerLauncher.java:238)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-5883) "Total megabyte-seconds" in job counters is slightly misleading
[ https://issues.apache.org/jira/browse/MAPREDUCE-5883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated MAPREDUCE-5883: -- Fix Version/s: 2.8.0 > "Total megabyte-seconds" in job counters is slightly misleading > --- > > Key: MAPREDUCE-5883 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5883 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.4.0, 3.0.0-alpha1 >Reporter: Nathan Roberts >Assignee: Nathan Roberts >Priority: Minor > Labels: BB2015-05-TBR > Fix For: 2.8.0, 2.7.2, 2.6.3, 3.0.0-alpha1 > > Attachments: MAPREDUCE-5883.patch > > > The following counters are in milliseconds so "megabyte-seconds" might be > better stated as "megabyte-milliseconds" > MB_MILLIS_MAPS.name= Total megabyte-seconds taken by all map > tasks > MB_MILLIS_REDUCES.name=Total megabyte-seconds taken by all reduce > tasks > VCORES_MILLIS_MAPS.name= Total vcore-seconds taken by all map tasks > VCORES_MILLIS_REDUCES.name=Total vcore-seconds taken by all reduce > tasks -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6387) Serialize the recently added Task#encryptedSpillKey field at the end
[
https://issues.apache.org/jira/browse/MAPREDUCE-6387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Junping Du updated MAPREDUCE-6387:
--
Fix Version/s: 2.8.0
> Serialize the recently added Task#encryptedSpillKey field at the end
>
>
> Key: MAPREDUCE-6387
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6387
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.7.0
>Reporter: Arun Suresh
>Assignee: Arun Suresh
>Priority: Minor
> Fix For: 2.8.0, 2.7.1, 3.0.0-alpha1
>
> Attachments: MAPREDUCE-6387.1.patch
>
>
> There was a recent addition of an {{encryptedSpillKey}} to the Task object.
> And when serialized, this field was written out somewhere in the middle. This
> caused deployments that do not use DistributedCache to push job jars before
> running the job to fail rolling upgrade.
> Although deploying via Distributed Cache is the recommended method, there
> might still be deployments that use the node local classpath to pick up the
> MR framework classes (eg. for efficiency purposes, since this does not
> require the jar being copied to hdfs and then to all the nodes)
> Ensuring that it is the last field written and read when the Task object is
> serialized would alleviate this issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6451) DistCp has incorrect chunkFilePath for multiple jobs when strategy is dynamic
[ https://issues.apache.org/jira/browse/MAPREDUCE-6451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated MAPREDUCE-6451: -- Fix Version/s: 2.8.0 > DistCp has incorrect chunkFilePath for multiple jobs when strategy is dynamic > - > > Key: MAPREDUCE-6451 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6451 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: distcp >Affects Versions: 2.6.0 >Reporter: Kuhu Shukla >Assignee: Kuhu Shukla > Fix For: 2.8.0, 2.7.2, 3.0.0-alpha1 > > Attachments: MAPREDUCE-6451-v1.patch, MAPREDUCE-6451-v2.patch, > MAPREDUCE-6451-v3.patch, MAPREDUCE-6451-v4.patch, MAPREDUCE-6451-v5.patch > > > DistCp when used with dynamic strategy does not update the chunkFilePath and > other static variables any time other than for the first job. This is seen > when DistCp::run() is used. > A single copy succeeds but multiple jobs finish successfully without any real > copying. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6518) Set SO_KEEPALIVE on shuffle connections
[ https://issues.apache.org/jira/browse/MAPREDUCE-6518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated MAPREDUCE-6518: -- Fix Version/s: 2.8.0 > Set SO_KEEPALIVE on shuffle connections > --- > > Key: MAPREDUCE-6518 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6518 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, nodemanager >Affects Versions: 2.7.1 >Reporter: Nathan Roberts >Assignee: Chang Li > Fix For: 2.8.0, 2.7.2, 3.0.0-alpha1 > > Attachments: MAPREDUCE-6518.4.patch, YARN-4052.2.patch, > YARN-4052.3.patch, YARN-4052.patch > > > Shuffle handler does not set SO_KEEPALIVE so we've seen cases where > FDs/sockets get stuck in ESTABLISHED state indefinitely because the server > did not see the client leave (network cut or otherwise). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6497) Fix wrong value of JOB_FINISHED event in JobHistoryEventHandler
[ https://issues.apache.org/jira/browse/MAPREDUCE-6497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated MAPREDUCE-6497: -- Fix Version/s: 2.8.0 > Fix wrong value of JOB_FINISHED event in JobHistoryEventHandler > --- > > Key: MAPREDUCE-6497 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6497 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.7.1 >Reporter: Shinichi Yamashita >Assignee: Shinichi Yamashita > Fix For: 2.8.0, 2.7.2, 2.6.2, 3.0.0-alpha1 > > Attachments: MAPREDUCE-6497.001.patch > > > It seems that "MAP_COUNTER_GROUPS" values use total_counter value. > We should fix to use map_counter value. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6474) ShuffleHandler can possibly exhaust nodemanager file descriptors
[ https://issues.apache.org/jira/browse/MAPREDUCE-6474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated MAPREDUCE-6474: -- Fix Version/s: 2.8.0 > ShuffleHandler can possibly exhaust nodemanager file descriptors > > > Key: MAPREDUCE-6474 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6474 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, nodemanager >Affects Versions: 2.5.0 >Reporter: Nathan Roberts >Assignee: Kuhu Shukla > Fix For: 2.8.0, 2.7.2, 3.0.0-alpha1 > > Attachments: YARN-2410-v1.patch, YARN-2410-v10.patch, > YARN-2410-v11.patch, YARN-2410-v2.patch, YARN-2410-v3.patch, > YARN-2410-v4.patch, YARN-2410-v5.patch, YARN-2410-v6.patch, > YARN-2410-v7.patch, YARN-2410-v8.patch, YARN-2410-v9.patch > > > The async nature of the shufflehandler can cause it to open a huge number of > file descriptors, when it runs out it crashes. > Scenario: > Job with 6K reduces, slow start set to 0.95, about 40 map outputs per node. > Let's say all 6K reduces hit a node at about same time asking for their > outputs. Each reducer will ask for all 40 map outputs over a single socket in > a > single request (not necessarily all 40 at once, but with coalescing it is > likely to be a large number). > sendMapOutput() will open the file for random reading and then perform an > async transfer of the particular portion of this file(). This will > theoretically > happen 6000*40=24 times which will run the NM out of file descriptors and > cause it to crash. > The algorithm should be refactored a little to not open the fds until they're > actually needed. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-5817) Mappers get rescheduled on node transition even after all reducers are completed
[ https://issues.apache.org/jira/browse/MAPREDUCE-5817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated MAPREDUCE-5817: -- Fix Version/s: 2.8.0 > Mappers get rescheduled on node transition even after all reducers are > completed > > > Key: MAPREDUCE-5817 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5817 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: applicationmaster >Affects Versions: 2.3.0 >Reporter: Sangjin Lee >Assignee: Sangjin Lee > Fix For: 2.8.0, 2.7.3, 2.6.5, 3.0.0-alpha1 > > Attachments: MAPREDUCE-5817.001.patch, MAPREDUCE-5817.002.patch, > mapreduce-5817.patch > > > We're seeing a behavior where a job runs long after all reducers were already > finished. We found that the job was rescheduling and running a number of > mappers beyond the point of reducer completion. In one situation, the job ran > for some 9 more hours after all reducers completed! > This happens because whenever a node transition (to an unusable state) comes > into the app master, it just reschedules all mappers that already ran on the > node in all cases. > Therefore, if any node transition has a potential to extend the job period. > Once this window opens, another node transition can prolong it, and this can > happen indefinitely in theory. > If there is some instability in the pool (unhealthy, etc.) for a duration, > then any big job is severely vulnerable to this problem. > If all reducers have been completed, JobImpl.actOnUnusableNode() should not > reschedule mapper tasks. If all reducers are completed, the mapper outputs > are no longer needed, and there is no need to reschedule mapper tasks as they > would not be consumed anyway. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-5982) Task attempts that fail from the ASSIGNED state can disappear
[ https://issues.apache.org/jira/browse/MAPREDUCE-5982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated MAPREDUCE-5982: -- Fix Version/s: 2.8.0 > Task attempts that fail from the ASSIGNED state can disappear > - > > Key: MAPREDUCE-5982 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5982 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mr-am >Affects Versions: 0.23.10, 2.2.1, 2.7.1 >Reporter: Jason Lowe >Assignee: Chang Li > Fix For: 2.8.0, 2.7.2, 2.6.4, 3.0.0-alpha1 > > Attachments: MAPREDUCE-5982.2.patch, MAPREDUCE-5982.3.patch, > MAPREDUCE-5982.4.patch, MAPREDUCE-5982.5.patch, MAPREDUCE-5982.6.patch, > MAPREDUCE-5982.branch.2.7.patch, MAPREDUCE-5982.patch > > > If a task attempt fails in the ASSIGNED state, e.g.: container launch fails, > then it can disappear from the job history. The task overview page will show > subsequent attempts but the attempt in question is simply missing. For > example attempt ID 1 appears but the attempt ID 0 is missing. Similarly in > the job overview page the task attempt doesn't appear in any of the > failed/killed/succeeded counts or pages. It's as if the task attempt never > existed, but the AM logs show otherwise. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6715) Fix Several Unsafe Practices
[
https://issues.apache.org/jira/browse/MAPREDUCE-6715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15803294#comment-15803294
]
Hudson commented on MAPREDUCE-6715:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11079 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/11079/])
MAPREDUCE-6715. Fix Several Unsafe Practices (Contributed by Yufei Gu
(templedf: rev 0b8a7c18ddbe73b356b3c9baf4460659ccaee095)
* (edit)
hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/dancing/Pentomino.java
* (edit)
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/MapTask.java
* (edit)
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/ShuffleSchedulerImpl.java
* (edit)
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/TextOutputFormat.java
* (edit)
hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/terasort/TeraScheduler.java
* (edit)
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/CleanupQueue.java
> Fix Several Unsafe Practices
>
>
> Key: MAPREDUCE-6715
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6715
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.9.0
>Reporter: Yufei Gu
>Assignee: Yufei Gu
> Fix For: 2.9.0, 3.0.0-alpha2
>
> Attachments: MAPREDUCE-6715.001.patch, MAPREDUCE-6715.002.patch,
> MAPREDUCE-6715.003.patch, MAPREDUCE-6715.004.patch, MAPREDUCE-6715.005.patch,
> MAPREDUCE-6715.006.patch
>
>
> {code}
> Null DereferenceCleanupQueue.java:139
> Weak SecurityManager Check: Overridable Method
> LocalDistributedCacheManager.java:229
> Null DereferenceTextOutputFormat.java:137
> Null DereferenceShuffleSchedulerImpl.java:422
> Null DereferenceMapTask.java:415
> Null DereferencePentomino.java:160
> Unreleased Resource: StreamsTeraScheduler.java:77
> Unreleased Resource: StreamsCLI.java:570
> Null DereferenceCLI.java:370
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6715) Fix Several Unsafe Practices
[
https://issues.apache.org/jira/browse/MAPREDUCE-6715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Templeton updated MAPREDUCE-6715:
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.0.0-alpha2
2.9.0
Status: Resolved (was: Patch Available)
Thanks for the patch, [~yufeigu]. Committed to trunk and branch-2
> Fix Several Unsafe Practices
>
>
> Key: MAPREDUCE-6715
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6715
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.9.0
>Reporter: Yufei Gu
>Assignee: Yufei Gu
> Fix For: 2.9.0, 3.0.0-alpha2
>
> Attachments: MAPREDUCE-6715.001.patch, MAPREDUCE-6715.002.patch,
> MAPREDUCE-6715.003.patch, MAPREDUCE-6715.004.patch, MAPREDUCE-6715.005.patch,
> MAPREDUCE-6715.006.patch
>
>
> {code}
> Null DereferenceCleanupQueue.java:139
> Weak SecurityManager Check: Overridable Method
> LocalDistributedCacheManager.java:229
> Null DereferenceTextOutputFormat.java:137
> Null DereferenceShuffleSchedulerImpl.java:422
> Null DereferenceMapTask.java:415
> Null DereferencePentomino.java:160
> Unreleased Resource: StreamsTeraScheduler.java:77
> Unreleased Resource: StreamsCLI.java:570
> Null DereferenceCLI.java:370
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6715) Fix Several Unsafe Practices
[
https://issues.apache.org/jira/browse/MAPREDUCE-6715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Templeton updated MAPREDUCE-6715:
Summary: Fix Several Unsafe Practices (was: Fix Bad Practices)
> Fix Several Unsafe Practices
>
>
> Key: MAPREDUCE-6715
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6715
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.9.0
>Reporter: Yufei Gu
>Assignee: Yufei Gu
> Attachments: MAPREDUCE-6715.001.patch, MAPREDUCE-6715.002.patch,
> MAPREDUCE-6715.003.patch, MAPREDUCE-6715.004.patch, MAPREDUCE-6715.005.patch,
> MAPREDUCE-6715.006.patch
>
>
> {code}
> Null DereferenceCleanupQueue.java:139
> Weak SecurityManager Check: Overridable Method
> LocalDistributedCacheManager.java:229
> Null DereferenceTextOutputFormat.java:137
> Null DereferenceShuffleSchedulerImpl.java:422
> Null DereferenceMapTask.java:415
> Null DereferencePentomino.java:160
> Unreleased Resource: StreamsTeraScheduler.java:77
> Unreleased Resource: StreamsCLI.java:570
> Null DereferenceCLI.java:370
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6191) TestJavaSerialization fails with getting incorrect MR job result
[ https://issues.apache.org/jira/browse/MAPREDUCE-6191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated MAPREDUCE-6191: -- Fix Version/s: 2.8.0 > TestJavaSerialization fails with getting incorrect MR job result > > > Key: MAPREDUCE-6191 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6191 > Project: Hadoop Map/Reduce > Issue Type: Test > Components: test >Affects Versions: 2.6.0 >Reporter: sam liu >Assignee: sam liu >Priority: Minor > Fix For: 2.8.0, 2.7.3, 2.6.5, 3.0.0-alpha1 > > Attachments: MAPREDUCE-6191.patch > > > TestJavaSerialization#testMapReduceJob() fails with getting incorrect MR job > result: > "junit.framework.ComparisonFailure: expected:<[a ]1> but was:<[0 1]1>" -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6472) MapReduce AM should have java.io.tmpdir=./tmp to be consistent with tasks
[ https://issues.apache.org/jira/browse/MAPREDUCE-6472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated MAPREDUCE-6472: -- Fix Version/s: 2.8.0 > MapReduce AM should have java.io.tmpdir=./tmp to be consistent with tasks > - > > Key: MAPREDUCE-6472 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6472 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mr-am >Affects Versions: 2.6.0 >Reporter: Jason Lowe >Assignee: Naganarasimha G R > Fix For: 2.8.0, 2.7.2, 3.0.0-alpha1 > > Attachments: MAPREDUCE-6472.20150914-1.patch, > MAPREDUCE-6472.20150914-2.patch > > > MapReduceChildJVM.getVMCommand ensures that all tasks have > -Djava.io.tmpdir=./tmp set as part of the task command-line, but this is only > used for tasks. The AM itself does not have a corresponding java.io.tmpdir > setting. It should also use the same tmpdir setting to avoid cases where the > AM JVM wants to place files in /tmp by default. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6621) Memory Leak in JobClient#submitJobInternal()
[
https://issues.apache.org/jira/browse/MAPREDUCE-6621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Junping Du updated MAPREDUCE-6621:
--
Fix Version/s: 2.8.0
> Memory Leak in JobClient#submitJobInternal()
>
>
> Key: MAPREDUCE-6621
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6621
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Xuan Gong
>Assignee: Xuan Gong
> Fix For: 2.8.0, 2.7.3, 2.6.4, 3.0.0-alpha1
>
> Attachments: MAPREDUCE-6621.1.patch, MAPREDUCE-6621.2.patch
>
>
> In JobClient:
> {code}
> public RunningJob submitJobInternal(final JobConf conf)
> throws FileNotFoundException, IOException {
> try {
> conf.setBooleanIfUnset("mapred.mapper.new-api", false);
> conf.setBooleanIfUnset("mapred.reducer.new-api", false);
> Job job = clientUgi.doAs(new PrivilegedExceptionAction () {
> @Override
> public Job run() throws IOException, ClassNotFoundException,
> InterruptedException {
> Job job = Job.getInstance(conf);
> job.submit();
> return job;
> }
> });
> // update our Cluster instance with the one created by Job for
> submission
> // (we can't pass our Cluster instance to Job, since Job wraps the
> config
> // instance, and the two configs would then diverge)
> cluster = job.getCluster();
> return new NetworkedJob(job);
> } catch (InterruptedException ie) {
> throw new IOException("interrupted", ie);
> }
> }
> {code}
> We will replace the cluster object with the cluster object from Job, but the
> previous old cluster object would never be closed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6618) YarnClientProtocolProvider leaking the YarnClient thread.
[
https://issues.apache.org/jira/browse/MAPREDUCE-6618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Junping Du updated MAPREDUCE-6618:
--
Fix Version/s: 2.8.0
> YarnClientProtocolProvider leaking the YarnClient thread.
> --
>
> Key: MAPREDUCE-6618
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6618
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Xuan Gong
>Assignee: Xuan Gong
> Fix For: 2.8.0, 2.7.3, 2.6.4, 3.0.0-alpha1
>
> Attachments: MAPREDUCE-6618.1.patch, MAPREDUCE-6618.2.patch,
> MAPREDUCE-6618.3.patch, MAPREDUCE-6618.4.patch, MAPREDUCE-6618.5.patch,
> MAPREDUCE-6618.6.patch
>
>
> YarnClientProtocolProvider creates YarnRunner which includes
> ResourceMgrDelegate. In ResourceMgrDelegate, we would initiate and start
> yarnclient. The yarnClient thread would be leaked due to
> {code}
> @Override
> public void close(ClientProtocol clientProtocol) throws IOException {
> // nothing to do
> }
> {code} in YarnClientProtocolProvider
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6637) Testcase Failure : TestFileInputFormat.testSplitLocationInfo
[
https://issues.apache.org/jira/browse/MAPREDUCE-6637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Junping Du updated MAPREDUCE-6637:
--
Fix Version/s: 2.8.0
> Testcase Failure : TestFileInputFormat.testSplitLocationInfo
>
>
> Key: MAPREDUCE-6637
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6637
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
> Fix For: 2.8.0, 2.7.3, 2.6.5, 3.0.0-alpha1
>
> Attachments: MAPREDUCE-6637.patch
>
>
> Following testcase is failing after HADOOP-12810
> {noformat}
> FAILED: org.apache.hadoop.mapred.TestFileInputFormat.testSplitLocationInfo[0]
> Error Message:
> expected:<2> but was:<1>
> Stack Trace:
> java.lang.AssertionError: expected:<2> but was:<1>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at
> org.apache.hadoop.mapred.TestFileInputFormat.testSplitLocationInfo(TestFileInputFormat.java:115)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6633) AM should retry map attempts if the reduce task encounters commpression related errors.
[
https://issues.apache.org/jira/browse/MAPREDUCE-6633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Junping Du updated MAPREDUCE-6633:
--
Fix Version/s: 2.8.0
> AM should retry map attempts if the reduce task encounters commpression
> related errors.
> ---
>
> Key: MAPREDUCE-6633
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6633
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.7.2
>Reporter: Rushabh S Shah
>Assignee: Rushabh S Shah
> Fix For: 2.8.0, 2.7.3, 3.0.0-alpha1
>
> Attachments: MAPREDUCE-6633.patch
>
>
> When reduce task encounters compression related errors, AM doesn't retry the
> corresponding map task.
> In one of the case we encountered, here is the stack trace.
> {noformat}
> 2016-01-27 13:44:28,915 WARN [main] org.apache.hadoop.mapred.YarnChild:
> Exception running child :
> org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in
> shuffle in fetcher#29
> at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:134)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:376)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1694)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> Caused by: java.lang.ArrayIndexOutOfBoundsException
> at
> com.hadoop.compression.lzo.LzoDecompressor.setInput(LzoDecompressor.java:196)
> at
> org.apache.hadoop.io.compress.BlockDecompressorStream.decompress(BlockDecompressorStream.java:104)
> at
> org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:85)
> at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:192)
> at
> org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput.shuffle(InMemoryMapOutput.java:97)
> at
> org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyMapOutput(Fetcher.java:537)
> at
> org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java:336)
> at org.apache.hadoop.mapreduce.task.reduce.Fetcher.run(Fetcher.java:193)
> {noformat}
> In this case, the node on which the map task ran had a bad drive.
> If the AM had retried running that map task somewhere else, the job
> definitely would have succeeded.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6549) multibyte delimiters with LineRecordReader cause duplicate records
[ https://issues.apache.org/jira/browse/MAPREDUCE-6549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated MAPREDUCE-6549: -- Fix Version/s: 2.8.0 > multibyte delimiters with LineRecordReader cause duplicate records > -- > > Key: MAPREDUCE-6549 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6549 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv1, mrv2 >Affects Versions: 2.7.2 >Reporter: Dustin Cote >Assignee: Wilfred Spiegelenburg > Fix For: 2.8.0, 2.7.2, 2.6.3, 3.0.0-alpha1 > > Attachments: MAPREDUCE-6549-1.patch, MAPREDUCE-6549-2.patch, > MAPREDUCE-6549.3.patch > > > LineRecorderReader currently produces duplicate records under certain > scenarios such as: > 1) input string: "abc+++def++ghi++" > delimiter string: "+++" > test passes with all sizes of the split > 2) input string: "abc++def+++ghi++" > delimiter string: "+++" > test fails with a split size of 4 > 2) input string: "abc+++def++ghi++" > delimiter string: "++" > test fails with a split size of 5 > 3) input string "abc+++defg++hij++" > delimiter string: "++" > test fails with a split size of 4 > 4) input string "abc++def+++ghi++" > delimiter string: "++" > test fails with a split size of 9 -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6540) TestMRTimelineEventHandling fails
[
https://issues.apache.org/jira/browse/MAPREDUCE-6540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Junping Du updated MAPREDUCE-6540:
--
Fix Version/s: 2.8.0
> TestMRTimelineEventHandling fails
> -
>
> Key: MAPREDUCE-6540
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6540
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Reporter: Sangjin Lee
>Assignee: Sangjin Lee
> Fix For: 2.8.0, 2.7.2, 2.6.3, 3.0.0-alpha1
>
> Attachments: MAPREDUCE-6540.001.patch
>
>
> TestMRTimelineEventHandling fails after YARN-2859 is merged because it
> changed the port the AHS binds to in a mini cluster.
> {noformat}
> Running org.apache.hadoop.mapred.TestMRTimelineEventHandling
> Tests run: 3, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 184.38 sec
> <<< FAILURE! - in org.apache.hadoop.mapred.TestMRTimelineEventHandling
> testMRTimelineEventHandling(org.apache.hadoop.mapred.TestMRTimelineEventHandling)
> Time elapsed: 70.528 sec <<< ERROR!
> java.io.IOException: Job didn't finish in 30 seconds
> at
> org.apache.hadoop.mapred.UtilsForTests.runJobSucceed(UtilsForTests.java:622)
> at
> org.apache.hadoop.mapred.TestMRTimelineEventHandling.testMRTimelineEventHandling(TestMRTimelineEventHandling.java:99)
> testMapreduceJobTimelineServiceEnabled(org.apache.hadoop.mapred.TestMRTimelineEventHandling)
> Time elapsed: 84.312 sec <<< ERROR!
> java.io.IOException: Job didn't finish in 30 seconds
> at
> org.apache.hadoop.mapred.UtilsForTests.runJobSucceed(UtilsForTests.java:622)
> at
> org.apache.hadoop.mapred.TestMRTimelineEventHandling.testMapreduceJobTimelineServiceEnabled(TestMRTimelineEventHandling.java:162)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6426) TestShuffleHandler#testGetMapOutputInfo is failing
[
https://issues.apache.org/jira/browse/MAPREDUCE-6426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Junping Du updated MAPREDUCE-6426:
--
Fix Version/s: 2.8.0
> TestShuffleHandler#testGetMapOutputInfo is failing
> --
>
> Key: MAPREDUCE-6426
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6426
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Affects Versions: 2.7.2
>Reporter: Devaraj K
>Assignee: zhihai xu
> Fix For: 2.8.0, 2.7.2, 3.0.0-alpha1
>
> Attachments: MAPREDUCE-6426.000.patch, MAPREDUCE-6426.001.patch
>
>
> {code:xml}
> expected:<1> but was:<0>
> Stacktrace
> java.lang.AssertionError: expected:<1> but was:<0>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at
> org.apache.hadoop.mapred.TestShuffleHandler.testGetMapOutputInfo(TestShuffleHandler.java:927)
> {code}
> https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2195/testReport/junit/org.apache.hadoop.mapred/TestShuffleHandler/testGetMapOutputInfo/
> https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/247/testReport/org.apache.hadoop.mapred/TestShuffleHandler/testGetMapOutputInfo/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6425) ShuffleHandler passes wrong "base" parameter to getMapOutputInfo if mapId is not in the cache.
[
https://issues.apache.org/jira/browse/MAPREDUCE-6425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Junping Du updated MAPREDUCE-6425:
--
Fix Version/s: 2.8.0
> ShuffleHandler passes wrong "base" parameter to getMapOutputInfo if mapId is
> not in the cache.
> --
>
> Key: MAPREDUCE-6425
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6425
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, nodemanager
>Reporter: zhihai xu
>Assignee: zhihai xu
> Fix For: 2.8.0, 2.7.2, 3.0.0-alpha1
>
> Attachments: MAPREDUCE-6425.000.patch, MAPREDUCE-6425.001.patch
>
>
> ShuffleHandler passes wrong {{base}} parameter to {{getMapOutputInfo}} if
> mapId is not in the cache.
> {{getMapOutputInfo}} expected the {{base}} parameter is
> {{getBaseLocation(jobId, user) + mapId}}
> When it is called inside populateHeaders, the {{base}} parameter is set
> correctly
> {code}
> String base = outputBaseStr + mapId;
> MapOutputInfo outputInfo = getMapOutputInfo(base, mapId, reduce,
> user);
> {code}
> When it is called outside populateHeaders, the {{base}} parameter is set
> wrongly to outputBasePathStr after number of mapId cached exceeds
> {{mapOutputMetaInfoCacheSize}}.
> {code}
> String outputBasePathStr = getBaseLocation(jobId, user);
> MapOutputInfo info = mapOutputInfoMap.get(mapId);
> if (info == null) {
> info = getMapOutputInfo(outputBasePathStr, mapId, reduceId, user);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6251) JobClient needs additional retries at a higher level to address not-immediately-consistent dfs corner cases
[ https://issues.apache.org/jira/browse/MAPREDUCE-6251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated MAPREDUCE-6251: -- Fix Version/s: 2.8.0 > JobClient needs additional retries at a higher level to address > not-immediately-consistent dfs corner cases > --- > > Key: MAPREDUCE-6251 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6251 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver, mrv2 >Affects Versions: 2.6.0 >Reporter: Craig Welch >Assignee: Craig Welch > Labels: BB2015-05-TBR > Fix For: 2.8.0, 2.7.1, 3.0.0-alpha1 > > Attachments: MAPREDUCE-6251.0.patch, MAPREDUCE-6251.1.patch, > MAPREDUCE-6251.2.patch, MAPREDUCE-6251.3.patch, MAPREDUCE-6251.4.patch, > MAPREDUCE-6251.6.patch, MAPREDUCE-6251.7.patch, MAPREDUCE-6251.8.patch, > MAPREDUCE-6251.8.patch > > > The JobClient is used to get job status information for running and completed > jobs. Final state and history for a job is communicated from the application > master to the job history server via a distributed file system - where the > history is uploaded by the application master to the dfs and then > scanned/loaded by the jobhistory server. While HDFS has strong consistency > guarantees not all Hadoop DFS's do. When used in conjunction with a > distributed file system which does not have this guarantee there will be > cases where the history server may not see an uploaded file, resulting in the > dreaded "no such job" and a null value for the RunningJob in the client. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Created] (MAPREDUCE-6830) Add a customized uncaughtexceptionhandler for fair scheduler
Yufei Gu created MAPREDUCE-6830: --- Summary: Add a customized uncaughtexceptionhandler for fair scheduler Key: MAPREDUCE-6830 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6830 Project: Hadoop Map/Reduce Issue Type: Improvement Components: yarn Reporter: Yufei Gu Assignee: Yufei Gu There are several threads in fair scheduler. The thread will quit when there is a runtime exception inside it. We should bring down the RM when that happens. Otherwise, there may be some weird behavior in RM. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-5649) Reduce cannot use more than 2G memory for the final merge
[ https://issues.apache.org/jira/browse/MAPREDUCE-5649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated MAPREDUCE-5649: -- Fix Version/s: 2.8.0 > Reduce cannot use more than 2G memory for the final merge > -- > > Key: MAPREDUCE-5649 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5649 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Reporter: stanley shi >Assignee: Gera Shegalov > Labels: 2.6.1-candidate, 2.7.2-candidate > Fix For: 2.6.1, 2.8.0, 2.7.2, 3.0.0-alpha1 > > Attachments: MAPREDUCE-5649.001.patch, MAPREDUCE-5649.002.patch, > MAPREDUCE-5649.003.patch > > > In the org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl.java file, in > the finalMerge method: > int maxInMemReduce = (int)Math.min( > Runtime.getRuntime().maxMemory() * maxRedPer, Integer.MAX_VALUE); > > This means no matter how much memory user has, reducer will not retain more > than 2G data in memory before the reduce phase starts. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6715) Fix Bad Practices
[
https://issues.apache.org/jira/browse/MAPREDUCE-6715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15803071#comment-15803071
]
Daniel Templeton commented on MAPREDUCE-6715:
-
Yay! LGTM +1 I'll commit shortly.
> Fix Bad Practices
> -
>
> Key: MAPREDUCE-6715
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6715
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.9.0
>Reporter: Yufei Gu
>Assignee: Yufei Gu
> Attachments: MAPREDUCE-6715.001.patch, MAPREDUCE-6715.002.patch,
> MAPREDUCE-6715.003.patch, MAPREDUCE-6715.004.patch, MAPREDUCE-6715.005.patch,
> MAPREDUCE-6715.006.patch
>
>
> {code}
> Null DereferenceCleanupQueue.java:139
> Weak SecurityManager Check: Overridable Method
> LocalDistributedCacheManager.java:229
> Null DereferenceTextOutputFormat.java:137
> Null DereferenceShuffleSchedulerImpl.java:422
> Null DereferenceMapTask.java:415
> Null DereferencePentomino.java:160
> Unreleased Resource: StreamsTeraScheduler.java:77
> Unreleased Resource: StreamsCLI.java:570
> Null DereferenceCLI.java:370
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-4785) TestMRApp occasionally fails
[
https://issues.apache.org/jira/browse/MAPREDUCE-4785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Junping Du updated MAPREDUCE-4785:
--
Fix Version/s: 2.8.0
> TestMRApp occasionally fails
>
>
> Key: MAPREDUCE-4785
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4785
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, test
>Affects Versions: 2.0.3-alpha
>Reporter: Jason Lowe
>Assignee: Haibo Chen
> Fix For: 2.8.0, 2.7.3, 3.0.0-alpha1
>
> Attachments: mapreduce4785.001.patch, mapreduce4785.prelim.patch
>
>
> TestMRApp is failing occasionally with this error:
> {noformat}
> testUpdatedNodes(org.apache.hadoop.mapreduce.v2.app.TestMRApp): Expecting 2
> more completion events for killed expected:<4> but was:<2>
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6259) IllegalArgumentException due to missing job submit time
[
https://issues.apache.org/jira/browse/MAPREDUCE-6259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Junping Du updated MAPREDUCE-6259:
--
Fix Version/s: 2.8.0
> IllegalArgumentException due to missing job submit time
> ---
>
> Key: MAPREDUCE-6259
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6259
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver
>Reporter: zhihai xu
>Assignee: zhihai xu
> Fix For: 2.8.0, 2.7.1, 3.0.0-alpha1
>
> Attachments: MAPREDUCE-6259.000.patch
>
>
> -1 job submit time cause IllegalArgumentException when parse the Job history
> file name and JOB_INIT_FAILED cause -1 job submit time in JobIndexInfo.
> We found the following job history file name which cause
> IllegalArgumentException when parse the job status in the job history file
> name.
> {code}
> job_1418398645407_115853--1-worun-kafka%2Dto%2Dhdfs%5Btwo%5D%5B15+topic%28s%29%5D-1423572836007-0-0-FAILED-root.journaling-1423572836007.jhist
> {code}
> The stack trace for the IllegalArgumentException is
> {code}
> 2015-02-10 04:54:01,863 WARN org.apache.hadoop.mapreduce.v2.hs.PartialJob:
> Exception while parsing job state. Defaulting to KILLED
> java.lang.IllegalArgumentException: No enum constant
> org.apache.hadoop.mapreduce.v2.api.records.JobState.0
> at java.lang.Enum.valueOf(Enum.java:236)
> at
> org.apache.hadoop.mapreduce.v2.api.records.JobState.valueOf(JobState.java:21)
> at
> org.apache.hadoop.mapreduce.v2.hs.PartialJob.getState(PartialJob.java:82)
> at
> org.apache.hadoop.mapreduce.v2.hs.PartialJob.(PartialJob.java:59)
> at
> org.apache.hadoop.mapreduce.v2.hs.CachedHistoryStorage.getAllPartialJobs(CachedHistoryStorage.java:159)
> at
> org.apache.hadoop.mapreduce.v2.hs.CachedHistoryStorage.getPartialJobs(CachedHistoryStorage.java:173)
> at
> org.apache.hadoop.mapreduce.v2.hs.JobHistory.getPartialJobs(JobHistory.java:284)
> at
> org.apache.hadoop.mapreduce.v2.hs.webapp.HsWebServices.getJobs(HsWebServices.java:212)
> at sun.reflect.GeneratedMethodAccessor63.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
> at
> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185)
> at
> com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
> at
> com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
> at
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> at
> com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
> at
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> at
> com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
> at
> com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:886)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:834)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:795)
> at
> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163)
> at
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> at
> com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118)
> at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113)
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
> at
>
[jira] [Updated] (MAPREDUCE-6454) MapReduce doesn't set the HADOOP_CLASSPATH for jar lib in distributed cache.
[ https://issues.apache.org/jira/browse/MAPREDUCE-6454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated MAPREDUCE-6454: -- Fix Version/s: 2.8.0 > MapReduce doesn't set the HADOOP_CLASSPATH for jar lib in distributed cache. > > > Key: MAPREDUCE-6454 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6454 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Junping Du >Assignee: Junping Du >Priority: Critical > Fix For: 2.8.0, 2.7.2, 2.6.2 > > Attachments: MAPREDUCE-6454-v2.1.patch, MAPREDUCE-6454-v2.patch, > MAPREDUCE-6454-v3.1.patch, MAPREDUCE-6454-v3.patch, MAPREDUCE-6454.patch > > > We already set lib jars on distributed-cache to CLASSPATH. However, in some > corner cases (like: MR local mode, Hive Map side local join, etc.), we need > these jars on HADOOP_CLASSPATH so hadoop scripts can take it in launching > runjar process. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6554) MRAppMaster servicestart failing with NPE in MRAppMaster#parsePreviousJobHistory
[
https://issues.apache.org/jira/browse/MAPREDUCE-6554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Junping Du updated MAPREDUCE-6554:
--
Fix Version/s: 2.8.0
> MRAppMaster servicestart failing with NPE in
> MRAppMaster#parsePreviousJobHistory
> -
>
> Key: MAPREDUCE-6554
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6554
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>Priority: Critical
> Fix For: 2.8.0, 2.7.3, 2.6.4, 3.0.0-alpha1
>
> Attachments: 0001-MAPREDUCE-6554.patch, 0002-MAPREDUCE-6554.patch,
> 0003-MAPREDUCE-6554.patch, MAPREDUCE-6554-branch-2.7.001.patch,
> MAPREDUCE-6554-branch-2.7.patch
>
>
> Create scenario so that MR app master gets preempted.
> On next MRAppMaster launch tried to recover previous job history file
> {{MRAppMaster#parsePreviousJobHistory}}
> {noformat}
> 2015-11-21 13:52:27,722 INFO [main]
> org.apache.hadoop.service.AbstractService: Service
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster failed in state STARTED;
> cause: java.lang.NullPointerException
> java.lang.NullPointerException
> at java.io.StringReader.(StringReader.java:50)
> at org.apache.avro.Schema$Parser.parse(Schema.java:917)
> at org.apache.avro.Schema.parse(Schema.java:966)
> at
> org.apache.hadoop.mapreduce.jobhistory.EventReader.(EventReader.java:75)
> at
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryParser.parse(JobHistoryParser.java:139)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.parsePreviousJobHistory(MRAppMaster.java:1256)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.processRecovery(MRAppMaster.java:1225)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1087)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$4.run(MRAppMaster.java:1570)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1673)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1566)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1499)
> 2015-11-21 13:52:27,725 INFO [main]
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Stopping
> JobHistoryEventHandler. Size of the outstanding queue size is 0
> {noformat}
> EventReader(EventReader stream)
> {noformat}
> this.version = in.readLine();
> ...
> Schema myschema = new
> SpecificData(Event.class.getClassLoader()).getSchema(Event.class);
> this.schema = Schema.parse(in.readLine());
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6481) LineRecordReader may give incomplete record and wrong position/key information for uncompressed input sometimes.
[
https://issues.apache.org/jira/browse/MAPREDUCE-6481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Junping Du updated MAPREDUCE-6481:
--
Fix Version/s: 2.8.0
> LineRecordReader may give incomplete record and wrong position/key
> information for uncompressed input sometimes.
>
>
> Key: MAPREDUCE-6481
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6481
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 2.7.0
>Reporter: zhihai xu
>Assignee: zhihai xu
>Priority: Critical
> Fix For: 2.8.0, 2.7.2, 2.6.3, 3.0.0-alpha1
>
> Attachments: MAPREDUCE-6481.000.patch
>
>
> LineRecordReader may give incomplete record and wrong position/key
> information for uncompressed input sometimes.
> There are two issues:
> # LineRecordReader may give incomplete record: some characters cut off at the
> end of record.
> # LineRecordReader may give wrong position/key information.
> The first issue only happens for Custom Delimiter, which is caused by the
> following code at {{LineReader#readCustomLine}}:
> {code}
> if (appendLength > 0) {
> if (ambiguousByteCount > 0) {
> str.append(recordDelimiterBytes, 0, ambiguousByteCount);
> //appending the ambiguous characters (refer case 2.2)
> bytesConsumed += ambiguousByteCount;
> ambiguousByteCount=0;
> }
> str.append(buffer, startPosn, appendLength);
> txtLength += appendLength;
> }
> {code}
> If {{appendLength}} is 0 and {{ambiguousByteCount}} is not 0, this bug will
> be triggered. For example, input is "123456789aab", Custom Delimiter is "ab",
> bufferSize is 10 and splitLength is 12, the correct record should be
> "123456789a" with length 10, but we get incomplete record "123456789" with
> length 9 from current code.
> The second issue can happen for both Custom Delimiter and Default Delimiter,
> which is caused by the code in {{UncompressedSplitLineReader#readLine}}.
> {{UncompressedSplitLineReader#readLine}} may report wrong size information at
> some corner cases. The reason is {{unusedBytes}} in the following code:
> {code}
> bytesRead += unusedBytes;
> unusedBytes = bufferSize - getBufferPosn();
> bytesRead -= unusedBytes;
> {code}
> If the last bytes read (bufferLength) is less than bufferSize, the previous
> {{unusedBytes}} will be wrong, which should be {{bufferLength}} -
> {{bufferPosn}} instead of bufferSize - {{bufferPosn}}. It will return larger
> value.
> For example, input is "1234567890ab12ab345", Custom Delimiter is "ab",
> bufferSize is 10 and two splits:first splitLength is 15 and second
> splitLength 4:
> the current code will give the following result:
> First record: Key:0 Value:"1234567890"
> Second record: Key:12 Value:"12"
> Third Record: Key:21 Value:"345"
> You can see the Key for the third record is wrong, it should be 16 instead of
> 21. It is due to wrong {{unusedBytes}}. {{fillBuffer}} read 10 bytes for the
> first time, for the second times, it only read 5 bytes, which is 5 bytes less
> than the bufferSize. That is why the key we get is 5 bytes larger than the
> correct one.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6492) AsyncDispatcher exit with NPE on TaskAttemptImpl#sendJHStartEventForAssignedFailTask
[
https://issues.apache.org/jira/browse/MAPREDUCE-6492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Junping Du updated MAPREDUCE-6492:
--
Fix Version/s: 2.8.0
> AsyncDispatcher exit with NPE on
> TaskAttemptImpl#sendJHStartEventForAssignedFailTask
>
>
> Key: MAPREDUCE-6492
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6492
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>Priority: Critical
> Fix For: 2.8.0, 2.7.2, 2.6.4, 3.0.0-alpha1
>
> Attachments: 0002-MAPREDUCE-6492.patch, 0003-MAPREDUCE-6492.patch,
> MAPREDUCE-6492.patch
>
>
> For {{TaskAttemptImpl#DeallocateContainerTransition}}
> {{sendJHStartEventForAssignedFailTask}} is send for
> TaskAttemptStateInternal.UNASSIGNED also .
> Causing NPE on {{taskAttempt.container.getNodeHttpAddress()}}
> {noformat}
> 2015-09-28 18:01:48,656 FATAL [AsyncDispatcher event handler]
> org.apache.hadoop.yarn.event.AsyncDispatcher: Error in dispatcher thread
> java.lang.NullPointerException
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.sendJHStartEventForAssignedFailTask(TaskAttemptImpl.java:1494)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.access$2900(TaskAttemptImpl.java:147)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl$DeallocateContainerTransition.transition(TaskAttemptImpl.java:1700)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl$DeallocateContainerTransition.transition(TaskAttemptImpl.java:1686)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$SingleInternalArc.doTransition(StateMachineFactory.java:362)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$3(StateMachineFactory.java:290)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.handle(TaskAttemptImpl.java:1190)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.handle(TaskAttemptImpl.java:146)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher.handle(MRAppMaster.java:1415)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher.handle(MRAppMaster.java:1407)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:183)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:109)
> at java.lang.Thread.run(Thread.java:745)
> 2015-09-28 18:01:48,660 INFO [AsyncDispatcher ShutDown handler]
> org.apache.hadoop.yarn.event.AsyncDispatcher: Exiting, bbye..
> 2015-09-28 18:01:48,660 INFO [ContainerLauncher #6]
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing
> the event EventType: CONTAINER_REMOTE_CLEANUP for container
> container_e04_1443430524957_0006_01_59 taskAttempt
> attempt_1443430524957_0006_m_00_9
> {noformat}
> Log aggregation fail for mapreduce application.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6528) Memory leak for HistoryFileManager.getJobSummary()
[
https://issues.apache.org/jira/browse/MAPREDUCE-6528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Junping Du updated MAPREDUCE-6528:
--
Fix Version/s: 2.8.0
> Memory leak for HistoryFileManager.getJobSummary()
> --
>
> Key: MAPREDUCE-6528
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6528
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver
>Reporter: Junping Du
>Assignee: Junping Du
>Priority: Critical
> Fix For: 2.8.0, 2.7.2, 2.6.3, 3.0.0-alpha1
>
> Attachments: MAPREDUCE-6528.patch
>
>
> We meet memory leak issues for JHS in a large cluster which is caused by code
> below doesn't release FSDataInputStream in exception case. MAPREDUCE-6273
> should fix most cases that exceptions get thrown. However, we still need to
> fix the memory leak for occasional case.
> {code}
> private String getJobSummary(FileContext fc, Path path) throws IOException {
> Path qPath = fc.makeQualified(path);
> FSDataInputStream in = fc.open(qPath);
> String jobSummaryString = in.readUTF();
> in.close();
> return jobSummaryString;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6410) Aggregated Logs Deletion doesnt work after refreshing Log Retention Settings in secure cluster
[
https://issues.apache.org/jira/browse/MAPREDUCE-6410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Junping Du updated MAPREDUCE-6410:
--
Fix Version/s: 2.8.0
> Aggregated Logs Deletion doesnt work after refreshing Log Retention Settings
> in secure cluster
> --
>
> Key: MAPREDUCE-6410
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6410
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.7.0
> Environment: mrV2, secure mode
>Reporter: Zhang Wei
>Assignee: Varun Saxena
>Priority: Critical
> Labels: historyserver
> Fix For: 2.8.0, 2.7.1, 3.0.0-alpha1
>
> Attachments: MAPREDUCE-6410.04.patch, MAPREDUCE-6410.05.patch,
> YARN-3779.01.patch, YARN-3779.02.patch, YARN-3779.03.patch,
> log_aggr_deletion_on_refresh_error.log, log_aggr_deletion_on_refresh_fix.log
>
>
> {{GSSException}} is thrown everytime log aggregation deletion is attempted
> after executing bin/mapred hsadmin -refreshLogRetentionSettings in a secure
> cluster.
> The problem can be reproduced by following steps:
> 1. startup historyserver in secure cluster.
> 2. Log deletion happens as per expectation.
> 3. execute {{mapred hsadmin -refreshLogRetentionSettings}} command to refresh
> the configuration value.
> 4. All the subsequent attempts of log deletion fail with {{GSSException}}
> Following exception can be found in historyserver's log if log deletion is
> enabled.
> {noformat}
> 2015-06-04 14:14:40,070 | ERROR | Timer-3 | Error reading root log dir this
> deletion attempt is being aborted | AggregatedLogDeletionService.java:127
> java.io.IOException: Failed on local exception: java.io.IOException:
> javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Failed to find
> any Kerberos tgt)]; Host Details : local host is: "vm-31/9.91.12.31";
> destination host is: "vm-33":25000;
> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:764)
> at org.apache.hadoop.ipc.Client.call(Client.java:1414)
> at org.apache.hadoop.ipc.Client.call(Client.java:1363)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
> at com.sun.proxy.$Proxy9.getListing(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getListing(ClientNamenodeProtocolTranslatorPB.java:519)
> at sun.reflect.GeneratedMethodAccessor16.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
> at com.sun.proxy.$Proxy10.getListing(Unknown Source)
> at org.apache.hadoop.hdfs.DFSClient.listPaths(DFSClient.java:1767)
> at org.apache.hadoop.hdfs.DFSClient.listPaths(DFSClient.java:1750)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:691)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.access$600(DistributedFileSystem.java:102)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$15.doCall(DistributedFileSystem.java:753)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$15.doCall(DistributedFileSystem.java:749)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:749)
> at
> org.apache.hadoop.yarn.logaggregation.AggregatedLogDeletionService$LogDeletionTask.run(AggregatedLogDeletionService.java:68)
> at java.util.TimerThread.mainLoop(Timer.java:555)
> at java.util.TimerThread.run(Timer.java:505)
> Caused by: java.io.IOException: javax.security.sasl.SaslException: GSS
> initiate failed [Caused by GSSException: No valid credentials provided
> (Mechanism level: Failed to find any Kerberos tgt)]
> at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:677)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1641)
> at
> org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:640)
> at
> org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:724)
> at
>
[jira] [Updated] (MAPREDUCE-6361) NPE issue in shuffle caused by concurrent issue between copySucceeded() in one thread and copyFailed() in another thread on the same host
[ https://issues.apache.org/jira/browse/MAPREDUCE-6361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated MAPREDUCE-6361: -- Fix Version/s: 2.8.0 > NPE issue in shuffle caused by concurrent issue between copySucceeded() in > one thread and copyFailed() in another thread on the same host > - > > Key: MAPREDUCE-6361 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6361 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.7.0 >Reporter: Junping Du >Assignee: Junping Du >Priority: Critical > Labels: 2.6.1-candidate > Fix For: 2.6.1, 2.8.0, 2.7.1, 3.0.0-alpha1 > > Attachments: MAPREDUCE-6361-v1.patch > > > The failure in log: > 2015-05-08 21:00:00,513 WARN [main] org.apache.hadoop.mapred.YarnChild: > Exception running child : > org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in > shuffle in fetcher#25 > at > org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:134) > at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:376) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) > Caused by: java.lang.NullPointerException > at > org.apache.hadoop.mapreduce.task.reduce.ShuffleSchedulerImpl.copyFailed(ShuffleSchedulerImpl.java:267) > at > org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java:308) > at > org.apache.hadoop.mapreduce.task.reduce.Fetcher.run(Fetcher.java:193) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-5948) org.apache.hadoop.mapred.LineRecordReader does not handle multibyte record delimiters well
[ https://issues.apache.org/jira/browse/MAPREDUCE-5948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated MAPREDUCE-5948: -- Fix Version/s: 2.8.0 > org.apache.hadoop.mapred.LineRecordReader does not handle multibyte record > delimiters well > -- > > Key: MAPREDUCE-5948 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5948 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.20.2, 0.23.9, 2.2.0 > Environment: CDH3U2 Redhat linux 5.7 >Reporter: Kris Geusebroek >Assignee: Akira Ajisaka >Priority: Critical > Fix For: 2.8.0, 2.7.2, 2.6.3, 3.0.0-alpha1 > > Attachments: HADOOP-9867.patch, HADOOP-9867.patch, HADOOP-9867.patch, > HADOOP-9867.patch, MAPREDUCE-5948.002.patch, MAPREDUCE-5948.003.patch > > > Having defined a recorddelimiter of multiple bytes in a new InputFileFormat > sometimes has the effect of skipping records from the input. > This happens when the input splits are split off just after a > recordseparator. Starting point for the next split would be non zero and > skipFirstLine would be true. A seek into the file is done to start - 1 and > the text until the first recorddelimiter is ignored (due to the presumption > that this record is already handled by the previous maptask). Since the re > ord delimiter is multibyte the seek only got the last byte of the delimiter > into scope and its not recognized as a full delimiter. So the text is skipped > until the next delimiter (ignoring a full record!!) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6339) Job history file is not flushed correctly because isTimerActive flag is not set true when flushTimerTask is scheduled.
[ https://issues.apache.org/jira/browse/MAPREDUCE-6339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated MAPREDUCE-6339: -- Fix Version/s: 2.8.0 > Job history file is not flushed correctly because isTimerActive flag is not > set true when flushTimerTask is scheduled. > -- > > Key: MAPREDUCE-6339 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6339 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 2.7.0 >Reporter: zhihai xu >Assignee: zhihai xu >Priority: Critical > Fix For: 2.8.0, 2.7.1, 3.0.0-alpha1 > > Attachments: MAPREDUCE-6339.000.patch, MAPREDUCE-6339.test.patch, > MAPREDUCE-6339.test1.patch > > > Job history file is not flushed correctly because isTimerActive flag is not > set true when flushTimerTask is scheduled. > It looks like we should set isTimerActive to true when flushTimerTask is > scheduled. Otherwise if a new qualified event comes before the current flush > timer is expired, flushTimerTask will be canceled and rescheduled. > Also I didn't find any code which set isTimerActive flag to true, So > isTimerActive is useless in current code. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6238) MR2 can't run local jobs with -libjars command options which is a regression from MR1
[
https://issues.apache.org/jira/browse/MAPREDUCE-6238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Junping Du updated MAPREDUCE-6238:
--
Fix Version/s: 2.8.0
> MR2 can't run local jobs with -libjars command options which is a regression
> from MR1
> -
>
> Key: MAPREDUCE-6238
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6238
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Reporter: zhihai xu
>Assignee: zhihai xu
>Priority: Critical
> Labels: 2.6.1-candidate
> Fix For: 2.6.1, 2.8.0, 2.7.1, 3.0.0-alpha1
>
> Attachments: MAPREDUCE-6238.000.patch
>
>
> MR2 can't run local jobs with -libjars command options which is a regression
> from MR1.
> When run MR2 job with -jt local and -libjars, the job fails with
> java.io.FileNotFoundException: File does not exist:
> hdfs://XXX.jar.
> But the same command is working in MR1.
> I find the problem is
> 1.
> because when MR2 run local job using LocalJobRunner
> from JobSubmitter, the JobSubmitter#jtFs is local filesystem,
> So copyRemoteFiles will return from [the middle of the
> function|https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmitter.java#L138]
> because source and destination file system are same.
> {code}
> if (compareFs(remoteFs, jtFs)) {
> return originalPath;
> }
> {code}
> The following code at
> [JobSubmitter.java|https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmitter.java#L219]
> try to add the destination file to DistributedCache which introduce a bug for
> local job.
> {code}
> Path newPath = copyRemoteFiles(libjarsDir, tmp, conf, replication);
> DistributedCache.addFileToClassPath(
> new Path(newPath.toUri().getPath()), conf);
> {code}
> Because new Path(newPath.toUri().getPath()) will lose the filesystem
> information from newPath, the file added to DistributedCache will use the
> default Uri filesystem hdfs based on the following code. This causes the
> FileNotFoundException when we access the file later at
>
> [determineTimestampsAndCacheVisibilities|https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmitter.java#L270]
> {code}
> public static void addFileToClassPath(Path file, Configuration conf)
> throws IOException {
> addFileToClassPath(file, conf, file.getFileSystem(conf));
> }
> public static void addFileToClassPath
>(Path file, Configuration conf, FileSystem fs)
> throws IOException {
> String classpath = conf.get(MRJobConfig.CLASSPATH_FILES);
> conf.set(MRJobConfig.CLASSPATH_FILES, classpath == null ? file.toString()
> : classpath + "," + file.toString());
> URI uri = fs.makeQualified(file).toUri();
> addCacheFile(uri, conf);
> }
> {code}
> Compare to the following [MR1
> code|https://github.com/apache/hadoop/blob/branch-1/src/mapred/org/apache/hadoop/mapred/JobClient.java#L811]:
> {code}
> Path newPath = copyRemoteFiles(fs, libjarsDir, tmp, job, replication);
> DistributedCache.addFileToClassPath(
> new Path(newPath.toUri().getPath()), job, fs);
> {code}
> You will see why MR1 doesn't have this issue.
> because it passes the local filesystem into
> DistributedCache#addFileToClassPath instead of using the default Uri
> filesystem hdfs.
> 2.
> Another incompatible change in MR2 is in
> [LocalDistributedCacheManager#setup|https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapred/LocalDistributedCacheManager.java#L113]
> {code}
> // Find which resources are to be put on the local classpath
> Map classpaths = new HashMap();
> Path[] archiveClassPaths = DistributedCache.getArchiveClassPaths(conf);
> if (archiveClassPaths != null) {
> for (Path p : archiveClassPaths) {
> FileSystem remoteFS = p.getFileSystem(conf);
> p = remoteFS.resolvePath(p.makeQualified(remoteFS.getUri(),
> remoteFS.getWorkingDirectory()));
> classpaths.put(p.toUri().getPath().toString(), p);
> }
> }
> Path[] fileClassPaths = DistributedCache.getFileClassPaths(conf);
> if (fileClassPaths != null) {
> for (Path p : fileClassPaths) {
> FileSystem remoteFS = p.getFileSystem(conf);
> p =
[jira] [Updated] (MAPREDUCE-6514) Job hangs as ask is not updated after ramping down of all reducers
[
https://issues.apache.org/jira/browse/MAPREDUCE-6514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Junping Du updated MAPREDUCE-6514:
--
Fix Version/s: 2.8.0
> Job hangs as ask is not updated after ramping down of all reducers
> --
>
> Key: MAPREDUCE-6514
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6514
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: applicationmaster
>Reporter: Varun Saxena
>Assignee: Varun Saxena
>Priority: Blocker
> Fix For: 2.8.0, 2.7.3, 2.6.5, 3.0.0-alpha1
>
> Attachments: MAPREDUCE-6514.01.patch, MAPREDUCE-6514.02.patch
>
>
> In RMContainerAllocator#preemptReducesIfNeeded, we simply clear the scheduled
> reduces map and put these reducers to pending. This is not updated in ask. So
> RM keeps on assigning and AM is not able to assign as no reducer is
> scheduled(check logs below the code).
> If this is updated immediately, RM will be able to schedule mappers
> immediately which anyways is the intention when we ramp down reducers.
> Scheduler need not allocate for ramped down reducers
> This if not handled can lead to map starvation as pointed out in
> MAPREDUCE-6513
> {code}
> LOG.info("Ramping down all scheduled reduces:"
> + scheduledRequests.reduces.size());
> for (ContainerRequest req : scheduledRequests.reduces.values()) {
> pendingReduces.add(req);
> }
> scheduledRequests.reduces.clear();
> {code}
> {noformat}
> 2015-10-13 04:55:04,912 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Container not
> assigned : container_1437451211867_1485_01_000215
> 2015-10-13 04:55:04,912 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Cannot assign
> container Container: [ContainerId: container_1437451211867_1485_01_000216,
> NodeId: hdszzdcxdat6g06u04p:26009, NodeHttpAddress:
> hdszzdcxdat6g06u04p:26010, Resource: , Priority: 10,
> Token: Token { kind: ContainerToken, service: 10.2.33.236:26009 }, ] for a
> reduce as either container memory less than required 4096 or no pending
> reduce tasks - reduces.isEmpty=true
> 2015-10-13 04:55:04,912 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Container not
> assigned : container_1437451211867_1485_01_000216
> 2015-10-13 04:55:04,912 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Cannot assign
> container Container: [ContainerId: container_1437451211867_1485_01_000217,
> NodeId: hdszzdcxdat6g06u06p:26009, NodeHttpAddress:
> hdszzdcxdat6g06u06p:26010, Resource: , Priority: 10,
> Token: Token { kind: ContainerToken, service: 10.2.33.239:26009 }, ] for a
> reduce as either container memory less than required 4096 or no pending
> reduce tasks - reduces.isEmpty=true
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6324) Uber jobs fail to update AMRM token when it rolls over
[ https://issues.apache.org/jira/browse/MAPREDUCE-6324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated MAPREDUCE-6324: -- Fix Version/s: 2.8.0 > Uber jobs fail to update AMRM token when it rolls over > -- > > Key: MAPREDUCE-6324 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6324 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mr-am >Affects Versions: 2.6.0 >Reporter: Jason Lowe >Assignee: Jason Lowe >Priority: Blocker > Labels: 2.6.1-candidate > Fix For: 2.6.1, 2.8.0, 2.7.1, 3.0.0-alpha1 > > Attachments: MAPREDUCE-6324.001.patch, MAPREDUCE-6324.002.patch > > > When the RM rolls a new AMRM master key the AMs are supposed to receive a new > AMRM token on subsequent heartbeats between the time when the new key is > rolled and when it is activated. This is not occurring for uber jobs. If > the connection to the RM needs to be re-established after the new key is > activated (e.g.: RM restart or network hiccup) then the uber job AM will be > unable to reconnect to the RM. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6829) Add peak memory usage counter for each task
[
https://issues.apache.org/jira/browse/MAPREDUCE-6829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15802919#comment-15802919
]
Hadoop QA commented on MAPREDUCE-6829:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 16s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
30s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 25s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
19s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 28s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
47s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 21s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
24s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 21s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 21s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 25s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
58s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 24s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 54s
{color} | {color:green} hadoop-mapreduce-client-core in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 9s {color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12845889/MAPREDUCE-6829.000.patch
|
| JIRA Issue | MAPREDUCE-6829 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux ca4cbe8e8c8a 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 02766b6 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6861/testReport/ |
| modules | C:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
U:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6861/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Add peak memory usage counter for each task
> ---
>
> Key: MAPREDUCE-6829
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6829
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
>Reporter: Yufei Gu
>Assignee: Miklos Szegedi
> Attachments: MAPREDUCE-6829.000.patch
>
>
> Each task has counters PHYSICAL_MEMORY_BYTES and VIRTUAL_MEMORY_BYTES, which
>
[jira] [Commented] (MAPREDUCE-6715) Fix Bad Practices
[
https://issues.apache.org/jira/browse/MAPREDUCE-6715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15802891#comment-15802891
]
Hadoop QA commented on MAPREDUCE-6715:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 20s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 17s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
32s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 48s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
32s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 48s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
30s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
22s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 39s
{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
56s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 54s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 54s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
30s {color} | {color:green} hadoop-mapreduce-project: The patch generated 0 new
+ 316 unchanged - 20 fixed = 316 total (was 336) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 43s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
25s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
33s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 31s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 48s
{color} | {color:green} hadoop-mapreduce-client-core in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 31s
{color} | {color:green} hadoop-mapreduce-examples in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 34m 12s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12845880/MAPREDUCE-6715.006.patch
|
| JIRA Issue | MAPREDUCE-6715 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 26a1e7d91916 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 02766b6 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6860/testReport/ |
| modules | C:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
hadoop-mapreduce-project/hadoop-mapreduce-examples U: hadoop-mapreduce-project |
| Console output |
[jira] [Updated] (MAPREDUCE-6829) Add peak memory usage counter for each task
[ https://issues.apache.org/jira/browse/MAPREDUCE-6829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Szegedi updated MAPREDUCE-6829: -- Status: Patch Available (was: Open) > Add peak memory usage counter for each task > --- > > Key: MAPREDUCE-6829 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6829 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Reporter: Yufei Gu >Assignee: Miklos Szegedi > Attachments: MAPREDUCE-6829.000.patch > > > Each task has counters PHYSICAL_MEMORY_BYTES and VIRTUAL_MEMORY_BYTES, which > are snapshots of memory usage of that task. They are not sufficient for users > to understand peak memory usage by that task, e.g. in order to diagnose task > failures, tune job parameters or change application design. This new feature > will add two more counters for each task: PHYSICAL_MEMORY_BYTES_MAX and > VIRTUAL_MEMORY_BYTES_MAX. > This JIRA has the same feature from MAPREDUCE-4710. I file this new YARN > JIRA since MAPREDUCE-4710 is pretty old one from MR 1.x era, it more or less > assumes a branch-1 architecture, should be close at this point. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6829) Add peak memory usage counter for each task
[ https://issues.apache.org/jira/browse/MAPREDUCE-6829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Szegedi updated MAPREDUCE-6829: -- Attachment: MAPREDUCE-6829.000.patch Attaching a patch that adds two memory peak counters. > Add peak memory usage counter for each task > --- > > Key: MAPREDUCE-6829 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6829 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Reporter: Yufei Gu >Assignee: Miklos Szegedi > Attachments: MAPREDUCE-6829.000.patch > > > Each task has counters PHYSICAL_MEMORY_BYTES and VIRTUAL_MEMORY_BYTES, which > are snapshots of memory usage of that task. They are not sufficient for users > to understand peak memory usage by that task, e.g. in order to diagnose task > failures, tune job parameters or change application design. This new feature > will add two more counters for each task: PHYSICAL_MEMORY_BYTES_MAX and > VIRTUAL_MEMORY_BYTES_MAX. > This JIRA has the same feature from MAPREDUCE-4710. I file this new YARN > JIRA since MAPREDUCE-4710 is pretty old one from MR 1.x era, it more or less > assumes a branch-1 architecture, should be close at this point. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6715) Fix Bad Practices
[
https://issues.apache.org/jira/browse/MAPREDUCE-6715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yufei Gu updated MAPREDUCE-6715:
Attachment: MAPREDUCE-6715.006.patch
Thanks [~templedf], makes sense to me, uploaded a new patch.
> Fix Bad Practices
> -
>
> Key: MAPREDUCE-6715
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6715
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.9.0
>Reporter: Yufei Gu
>Assignee: Yufei Gu
> Attachments: MAPREDUCE-6715.001.patch, MAPREDUCE-6715.002.patch,
> MAPREDUCE-6715.003.patch, MAPREDUCE-6715.004.patch, MAPREDUCE-6715.005.patch,
> MAPREDUCE-6715.006.patch
>
>
> {code}
> Null DereferenceCleanupQueue.java:139
> Weak SecurityManager Check: Overridable Method
> LocalDistributedCacheManager.java:229
> Null DereferenceTextOutputFormat.java:137
> Null DereferenceShuffleSchedulerImpl.java:422
> Null DereferenceMapTask.java:415
> Null DereferencePentomino.java:160
> Unreleased Resource: StreamsTeraScheduler.java:77
> Unreleased Resource: StreamsCLI.java:570
> Null DereferenceCLI.java:370
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6715) Fix Bad Practices
[
https://issues.apache.org/jira/browse/MAPREDUCE-6715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15802785#comment-15802785
]
Hadoop QA commented on MAPREDUCE-6715:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 26s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 39s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
18s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 42s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
31s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 46s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
29s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
15s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 35s
{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 8s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
39s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 40s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 40s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
29s {color} | {color:green} hadoop-mapreduce-project: The patch generated 0 new
+ 315 unchanged - 20 fixed = 315 total (was 335) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 41s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
25s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
26s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 32s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 46s
{color} | {color:green} hadoop-mapreduce-client-core in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 29s
{color} | {color:green} hadoop-mapreduce-examples in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 32m 57s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12845864/MAPREDUCE-6715.005.patch
|
| JIRA Issue | MAPREDUCE-6715 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 7f5232d4f5d2 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29
20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 02766b6 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6859/testReport/ |
| modules | C:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
hadoop-mapreduce-project/hadoop-mapreduce-examples U: hadoop-mapreduce-project |
| Console output |
[jira] [Commented] (MAPREDUCE-6715) Fix Bad Practices
[
https://issues.apache.org/jira/browse/MAPREDUCE-6715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15802758#comment-15802758
]
Daniel Templeton commented on MAPREDUCE-6715:
-
That part looks fine to me. I'm looking more closely at the
{{ShuffleSchedulerImpl}} changes now. First, throwing {{InterruptedException}}
in the wrong thing to do. Second, the analyzer is wrong--{{host}} can't be
null. Maybe if you rearrange the code like this: {code} Iterator
iter = pendingHosts.iterator();
// Safe to take one because we know pendingHosts isn't empty
host = iter.next();
int numToPick = random.nextInt(pendingHosts.size());
for (int i=0; i < numToPick; ++i) {
host = iter.next();
}{code} the analyzer would be happier?
> Fix Bad Practices
> -
>
> Key: MAPREDUCE-6715
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6715
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.9.0
>Reporter: Yufei Gu
>Assignee: Yufei Gu
> Attachments: MAPREDUCE-6715.001.patch, MAPREDUCE-6715.002.patch,
> MAPREDUCE-6715.003.patch, MAPREDUCE-6715.004.patch, MAPREDUCE-6715.005.patch
>
>
> {code}
> Null DereferenceCleanupQueue.java:139
> Weak SecurityManager Check: Overridable Method
> LocalDistributedCacheManager.java:229
> Null DereferenceTextOutputFormat.java:137
> Null DereferenceShuffleSchedulerImpl.java:422
> Null DereferenceMapTask.java:415
> Null DereferencePentomino.java:160
> Unreleased Resource: StreamsTeraScheduler.java:77
> Unreleased Resource: StreamsCLI.java:570
> Null DereferenceCLI.java:370
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6334) Fetcher#copyMapOutput is leaking usedMemory upon IOException during InMemoryMapOutput shuffle handler
[ https://issues.apache.org/jira/browse/MAPREDUCE-6334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated MAPREDUCE-6334: -- Fix Version/s: 2.8.0 > Fetcher#copyMapOutput is leaking usedMemory upon IOException during > InMemoryMapOutput shuffle handler > - > > Key: MAPREDUCE-6334 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6334 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.7.0 >Reporter: Eric Payne >Assignee: Eric Payne >Priority: Blocker > Fix For: 2.8.0, 2.7.1, 2.6.2, 3.0.0-alpha1 > > Attachments: MAPREDUCE-6334.001.patch, MAPREDUCE-6334.002.patch > > > We are seeing this happen when > - an NM's disk goes bad during the creation of map output(s) > - the reducer's fetcher can read the shuffle header and reserve the memory > - but gets an IOException when trying to shuffle for InMemoryMapOutput > - shuffle fetch retry is enabled -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6715) Fix Bad Practices
[
https://issues.apache.org/jira/browse/MAPREDUCE-6715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15802717#comment-15802717
]
Yufei Gu commented on MAPREDUCE-6715:
-
Thanks for the review. After deeper looking, I realize this complain about NPE
is because the code analyzer cannot tell whether two separate code section are
in the same condition. For example, section1 and section2 in the following code
are under the same condition, but code analyzer cannot tell. I adjusted the
code a little, hopefully it will work for the code analyzer, but it is NOT an
issue anyway. We may need a smarter code analyzer.
{code}
CompressionCodec codec = null;
String extension = "";
if (isCompressed) {
// section1
Class codecClass =
getOutputCompressorClass(job, GzipCodec.class);
codec = (CompressionCodec) ReflectionUtils.newInstance(codecClass, conf);
extension = codec.getDefaultExtension();
}
Path file = getDefaultWorkFile(job, extension);
FileSystem fs = file.getFileSystem(conf);
if (!isCompressed) {
FSDataOutputStream fileOut = fs.create(file, false);
return new LineRecordWriter(fileOut, keyValueSeparator);
} else {
//section2
FSDataOutputStream fileOut = fs.create(file, false);
return new LineRecordWriter(new DataOutputStream
(codec.createOutputStream(fileOut)),
keyValueSeparator);
}
{code}
> Fix Bad Practices
> -
>
> Key: MAPREDUCE-6715
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6715
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.9.0
>Reporter: Yufei Gu
>Assignee: Yufei Gu
> Attachments: MAPREDUCE-6715.001.patch, MAPREDUCE-6715.002.patch,
> MAPREDUCE-6715.003.patch, MAPREDUCE-6715.004.patch, MAPREDUCE-6715.005.patch
>
>
> {code}
> Null DereferenceCleanupQueue.java:139
> Weak SecurityManager Check: Overridable Method
> LocalDistributedCacheManager.java:229
> Null DereferenceTextOutputFormat.java:137
> Null DereferenceShuffleSchedulerImpl.java:422
> Null DereferenceMapTask.java:415
> Null DereferencePentomino.java:160
> Unreleased Resource: StreamsTeraScheduler.java:77
> Unreleased Resource: StreamsCLI.java:570
> Null DereferenceCLI.java:370
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6715) Fix Bad Practices
[
https://issues.apache.org/jira/browse/MAPREDUCE-6715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yufei Gu updated MAPREDUCE-6715:
Attachment: MAPREDUCE-6715.005.patch
> Fix Bad Practices
> -
>
> Key: MAPREDUCE-6715
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6715
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.9.0
>Reporter: Yufei Gu
>Assignee: Yufei Gu
> Attachments: MAPREDUCE-6715.001.patch, MAPREDUCE-6715.002.patch,
> MAPREDUCE-6715.003.patch, MAPREDUCE-6715.004.patch, MAPREDUCE-6715.005.patch
>
>
> {code}
> Null DereferenceCleanupQueue.java:139
> Weak SecurityManager Check: Overridable Method
> LocalDistributedCacheManager.java:229
> Null DereferenceTextOutputFormat.java:137
> Null DereferenceShuffleSchedulerImpl.java:422
> Null DereferenceMapTask.java:415
> Null DereferencePentomino.java:160
> Unreleased Resource: StreamsTeraScheduler.java:77
> Unreleased Resource: StreamsCLI.java:570
> Null DereferenceCLI.java:370
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6829) Add peak memory usage counter for each task
[ https://issues.apache.org/jira/browse/MAPREDUCE-6829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yufei Gu updated MAPREDUCE-6829: Assignee: Miklos Szegedi > Add peak memory usage counter for each task > --- > > Key: MAPREDUCE-6829 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6829 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Reporter: Yufei Gu >Assignee: Miklos Szegedi > > Each task has counters PHYSICAL_MEMORY_BYTES and VIRTUAL_MEMORY_BYTES, which > are snapshots of memory usage of that task. They are not sufficient for users > to understand peak memory usage by that task, e.g. in order to diagnose task > failures, tune job parameters or change application design. This new feature > will add two more counters for each task: PHYSICAL_MEMORY_BYTES_MAX and > VIRTUAL_MEMORY_BYTES_MAX. > This JIRA has the same feature from MAPREDUCE-4710. I file this new YARN > JIRA since MAPREDUCE-4710 is pretty old one from MR 1.x era, it more or less > assumes a branch-1 architecture, should be close at this point. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Moved] (MAPREDUCE-6829) Add peak memory usage counter for each task
[ https://issues.apache.org/jira/browse/MAPREDUCE-6829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yufei Gu moved YARN-5044 to MAPREDUCE-6829: --- Component/s: (was: yarn) mrv2 Key: MAPREDUCE-6829 (was: YARN-5044) Project: Hadoop Map/Reduce (was: Hadoop YARN) > Add peak memory usage counter for each task > --- > > Key: MAPREDUCE-6829 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6829 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Reporter: Yufei Gu > > Each task has counters PHYSICAL_MEMORY_BYTES and VIRTUAL_MEMORY_BYTES, which > are snapshots of memory usage of that task. They are not sufficient for users > to understand peak memory usage by that task, e.g. in order to diagnose task > failures, tune job parameters or change application design. This new feature > will add two more counters for each task: PHYSICAL_MEMORY_BYTES_MAX and > VIRTUAL_MEMORY_BYTES_MAX. > This JIRA has the same feature from MAPREDUCE-4710. I file this new YARN > JIRA since MAPREDUCE-4710 is pretty old one from MR 1.x era, it more or less > assumes a branch-1 architecture, should be close at this point. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6441) LocalDistributedCacheManager for concurrent sqoop processes fails to create unique directories
[
https://issues.apache.org/jira/browse/MAPREDUCE-6441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15801919#comment-15801919
]
Hadoop QA commented on MAPREDUCE-6441:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 14s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
55s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 23s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
19s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 23s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
43s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 23s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
20s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 17s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 17s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s {color} | {color:green}
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common:
The patch generated 0 new + 78 unchanged - 7 fixed = 78 total (was 85) {color}
|
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 19s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
42s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 18s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 44s
{color} | {color:green} hadoop-mapreduce-client-common in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 21m 39s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12845729/MAPREDUCE-6441.005.patch
|
| JIRA Issue | MAPREDUCE-6441 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 20d979dc0412 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / a605ff3 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6858/testReport/ |
| modules | C:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common
U:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common
|
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6858/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> LocalDistributedCacheManager for concurrent sqoop processes fails to create
> unique directories
> --
>
> Key: MAPREDUCE-6441
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6441
> Project: Hadoop Map/Reduce
> Issue
[jira] [Commented] (MAPREDUCE-6715) Fix Bad Practices
[
https://issues.apache.org/jira/browse/MAPREDUCE-6715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15801651#comment-15801651
]
Daniel Templeton commented on MAPREDUCE-6715:
-
LGTM. About that pointless null check... If it's there to make a code
analyzer happy, I'm OK with that. It might be good to add a comment to explain
that, though, so that someone else doesn't remove it later.
> Fix Bad Practices
> -
>
> Key: MAPREDUCE-6715
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6715
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.9.0
>Reporter: Yufei Gu
>Assignee: Yufei Gu
> Attachments: MAPREDUCE-6715.001.patch, MAPREDUCE-6715.002.patch,
> MAPREDUCE-6715.003.patch, MAPREDUCE-6715.004.patch
>
>
> {code}
> Null DereferenceCleanupQueue.java:139
> Weak SecurityManager Check: Overridable Method
> LocalDistributedCacheManager.java:229
> Null DereferenceTextOutputFormat.java:137
> Null DereferenceShuffleSchedulerImpl.java:422
> Null DereferenceMapTask.java:415
> Null DereferencePentomino.java:160
> Unreleased Resource: StreamsTeraScheduler.java:77
> Unreleased Resource: StreamsCLI.java:570
> Null DereferenceCLI.java:370
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6662) Clear ASF Warnings on test data files
[ https://issues.apache.org/jira/browse/MAPREDUCE-6662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15800940#comment-15800940 ] Brahma Reddy Battula commented on MAPREDUCE-6662: - Thanks [~ajisakaa] > Clear ASF Warnings on test data files > - > > Key: MAPREDUCE-6662 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6662 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Vinayakumar B >Assignee: Vinayakumar B >Priority: Minor > Fix For: 2.8.0, 3.0.0-alpha1 > > Attachments: MAPREDUCE-6662.01.patch, MAPREDUCE-6662.02.patch, > MAPREDUCE-6662.03.patch, MAPREDUCE-6662.04.patch, MAPREDUCE-6662.05.patch > > > In all MAPREDUCE QA runs, '14 ASF Warnings' will mark the entire QA result as > '-1 Overall'. > These warnings are from the files generated in tests. > It will be good to see green +1s in QA report than RED -1(s). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6662) Clear ASF Warnings on test data files
[ https://issues.apache.org/jira/browse/MAPREDUCE-6662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15800834#comment-15800834 ] Akira Ajisaka commented on MAPREDUCE-6662: -- Cherry-picked this to branch-2.8. > Clear ASF Warnings on test data files > - > > Key: MAPREDUCE-6662 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6662 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Vinayakumar B >Assignee: Vinayakumar B >Priority: Minor > Fix For: 2.8.0, 3.0.0-alpha1 > > Attachments: MAPREDUCE-6662.01.patch, MAPREDUCE-6662.02.patch, > MAPREDUCE-6662.03.patch, MAPREDUCE-6662.04.patch, MAPREDUCE-6662.05.patch > > > In all MAPREDUCE QA runs, '14 ASF Warnings' will mark the entire QA result as > '-1 Overall'. > These warnings are from the files generated in tests. > It will be good to see green +1s in QA report than RED -1(s). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6662) Clear ASF Warnings on test data files
[ https://issues.apache.org/jira/browse/MAPREDUCE-6662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka updated MAPREDUCE-6662: - Fix Version/s: (was: 2.9.0) 2.8.0 > Clear ASF Warnings on test data files > - > > Key: MAPREDUCE-6662 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6662 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Vinayakumar B >Assignee: Vinayakumar B >Priority: Minor > Fix For: 2.8.0, 3.0.0-alpha1 > > Attachments: MAPREDUCE-6662.01.patch, MAPREDUCE-6662.02.patch, > MAPREDUCE-6662.03.patch, MAPREDUCE-6662.04.patch, MAPREDUCE-6662.05.patch > > > In all MAPREDUCE QA runs, '14 ASF Warnings' will mark the entire QA result as > '-1 Overall'. > These warnings are from the files generated in tests. > It will be good to see green +1s in QA report than RED -1(s). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6662) Clear ASF Warnings on test data files
[ https://issues.apache.org/jira/browse/MAPREDUCE-6662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15800832#comment-15800832 ] Akira Ajisaka commented on MAPREDUCE-6662: -- TestMiniMRChildTask succeed with the patch on CentOS 6. Checking this in. > Clear ASF Warnings on test data files > - > > Key: MAPREDUCE-6662 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6662 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Vinayakumar B >Assignee: Vinayakumar B >Priority: Minor > Fix For: 2.9.0, 3.0.0-alpha1 > > Attachments: MAPREDUCE-6662.01.patch, MAPREDUCE-6662.02.patch, > MAPREDUCE-6662.03.patch, MAPREDUCE-6662.04.patch, MAPREDUCE-6662.05.patch > > > In all MAPREDUCE QA runs, '14 ASF Warnings' will mark the entire QA result as > '-1 Overall'. > These warnings are from the files generated in tests. > It will be good to see green +1s in QA report than RED -1(s). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6828) TestMiniMRChildTask is failing in branch-2 and branch-2.8 on mac
[
https://issues.apache.org/jira/browse/MAPREDUCE-6828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka updated MAPREDUCE-6828:
-
Priority: Minor (was: Major)
Summary: TestMiniMRChildTask is failing in branch-2 and branch-2.8 on mac
(was: TestMiniMRChildTask is failing in branch-2 and branch-2.8)
Looks like this issue happens only on mac.
> TestMiniMRChildTask is failing in branch-2 and branch-2.8 on mac
>
>
> Key: MAPREDUCE-6828
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6828
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Reporter: Akira Ajisaka
>Priority: Minor
>
> {noformat}
> Running org.apache.hadoop.mapred.TestMiniMRChildTask
> Tests run: 3, Failures: 3, Errors: 0, Skipped: 0, Time elapsed: 107.644 sec
> <<< FAILURE! - in org.apache.hadoop.mapred.TestMiniMRChildTask
> testTaskEnv(org.apache.hadoop.mapred.TestMiniMRChildTask) Time elapsed:
> 24.369 sec <<< FAILURE!
> java.lang.AssertionError: The environment checker job failed.
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at
> org.apache.hadoop.mapred.TestMiniMRChildTask.runTestTaskEnv(TestMiniMRChildTask.java:550)
> at
> org.apache.hadoop.mapred.TestMiniMRChildTask.testTaskEnv(TestMiniMRChildTask.java:472)
> testTaskOldEnv(org.apache.hadoop.mapred.TestMiniMRChildTask) Time elapsed:
> 18.773 sec <<< FAILURE!
> java.lang.AssertionError: The environment checker job failed.
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at
> org.apache.hadoop.mapred.TestMiniMRChildTask.runTestTaskEnv(TestMiniMRChildTask.java:550)
> at
> org.apache.hadoop.mapred.TestMiniMRChildTask.testTaskOldEnv(TestMiniMRChildTask.java:496)
> testMapRedExecutionEnv(org.apache.hadoop.mapred.TestMiniMRChildTask) Time
> elapsed: 23.931 sec <<< FAILURE!
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.hadoop.mapred.TestMiniMRChildTask.launchTest(TestMiniMRChildTask.java:194)
> at
> org.apache.hadoop.mapred.TestMiniMRChildTask.testMapRedExecutionEnv(TestMiniMRChildTask.java:448)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6662) Clear ASF Warnings on test data files
[ https://issues.apache.org/jira/browse/MAPREDUCE-6662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15800805#comment-15800805 ] Akira Ajisaka commented on MAPREDUCE-6662: -- Looks like this issue only happens on mac. > Clear ASF Warnings on test data files > - > > Key: MAPREDUCE-6662 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6662 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Vinayakumar B >Assignee: Vinayakumar B >Priority: Minor > Fix For: 2.9.0, 3.0.0-alpha1 > > Attachments: MAPREDUCE-6662.01.patch, MAPREDUCE-6662.02.patch, > MAPREDUCE-6662.03.patch, MAPREDUCE-6662.04.patch, MAPREDUCE-6662.05.patch > > > In all MAPREDUCE QA runs, '14 ASF Warnings' will mark the entire QA result as > '-1 Overall'. > These warnings are from the files generated in tests. > It will be good to see green +1s in QA report than RED -1(s). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Created] (MAPREDUCE-6828) TestMiniMRChildTask is failing in branch-2 and branch-2.8
Akira Ajisaka created MAPREDUCE-6828:
Summary: TestMiniMRChildTask is failing in branch-2 and branch-2.8
Key: MAPREDUCE-6828
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6828
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: test
Reporter: Akira Ajisaka
{noformat}
Running org.apache.hadoop.mapred.TestMiniMRChildTask
Tests run: 3, Failures: 3, Errors: 0, Skipped: 0, Time elapsed: 107.644 sec <<<
FAILURE! - in org.apache.hadoop.mapred.TestMiniMRChildTask
testTaskEnv(org.apache.hadoop.mapred.TestMiniMRChildTask) Time elapsed: 24.369
sec <<< FAILURE!
java.lang.AssertionError: The environment checker job failed.
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.assertTrue(Assert.java:41)
at
org.apache.hadoop.mapred.TestMiniMRChildTask.runTestTaskEnv(TestMiniMRChildTask.java:550)
at
org.apache.hadoop.mapred.TestMiniMRChildTask.testTaskEnv(TestMiniMRChildTask.java:472)
testTaskOldEnv(org.apache.hadoop.mapred.TestMiniMRChildTask) Time elapsed:
18.773 sec <<< FAILURE!
java.lang.AssertionError: The environment checker job failed.
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.assertTrue(Assert.java:41)
at
org.apache.hadoop.mapred.TestMiniMRChildTask.runTestTaskEnv(TestMiniMRChildTask.java:550)
at
org.apache.hadoop.mapred.TestMiniMRChildTask.testTaskOldEnv(TestMiniMRChildTask.java:496)
testMapRedExecutionEnv(org.apache.hadoop.mapred.TestMiniMRChildTask) Time
elapsed: 23.931 sec <<< FAILURE!
java.lang.AssertionError: null
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertTrue(Assert.java:52)
at
org.apache.hadoop.mapred.TestMiniMRChildTask.launchTest(TestMiniMRChildTask.java:194)
at
org.apache.hadoop.mapred.TestMiniMRChildTask.testMapRedExecutionEnv(TestMiniMRChildTask.java:448)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6662) Clear ASF Warnings on test data files
[ https://issues.apache.org/jira/browse/MAPREDUCE-6662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15800751#comment-15800751 ] Akira Ajisaka commented on MAPREDUCE-6662: -- TestMiniMRChildTask is failing in branch-2.8 (even without the patch). All other tests succeeded. > Clear ASF Warnings on test data files > - > > Key: MAPREDUCE-6662 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6662 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Vinayakumar B >Assignee: Vinayakumar B >Priority: Minor > Fix For: 2.9.0, 3.0.0-alpha1 > > Attachments: MAPREDUCE-6662.01.patch, MAPREDUCE-6662.02.patch, > MAPREDUCE-6662.03.patch, MAPREDUCE-6662.04.patch, MAPREDUCE-6662.05.patch > > > In all MAPREDUCE QA runs, '14 ASF Warnings' will mark the entire QA result as > '-1 Overall'. > These warnings are from the files generated in tests. > It will be good to see green +1s in QA report than RED -1(s). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6662) Clear ASF Warnings on test data files
[ https://issues.apache.org/jira/browse/MAPREDUCE-6662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15800681#comment-15800681 ] Akira Ajisaka commented on MAPREDUCE-6662: -- +1 for cherry-picking to branch-2.8. I'll run the tests updated by the patch locally. > Clear ASF Warnings on test data files > - > > Key: MAPREDUCE-6662 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6662 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Vinayakumar B >Assignee: Vinayakumar B >Priority: Minor > Fix For: 2.9.0, 3.0.0-alpha1 > > Attachments: MAPREDUCE-6662.01.patch, MAPREDUCE-6662.02.patch, > MAPREDUCE-6662.03.patch, MAPREDUCE-6662.04.patch, MAPREDUCE-6662.05.patch > > > In all MAPREDUCE QA runs, '14 ASF Warnings' will mark the entire QA result as > '-1 Overall'. > These warnings are from the files generated in tests. > It will be good to see green +1s in QA report than RED -1(s). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
