Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

[Apache Jenkins Server]

For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/554/

[Oct 10, 2017 5:16:36 PM] (wang) HADOOP-14939. Update project release notes 
with HDFS-10467 for 3.0.0.
[Oct 10, 2017 5:18:42 PM] (rkanter) YARN-7309. 
TestClientRMService#testUpdateApplicationPriorityRequest and
[Oct 10, 2017 5:38:05 PM] (jitendra) HDFS-11575. Supporting HDFS NFS gateway 
with Federated HDFS. Contributed
[Oct 10, 2017 6:00:07 PM] (jitendra) HADOOP-14910. Upgrade netty-all jar to 
latest 4.0.x.Final. Contributed
[Oct 10, 2017 7:47:25 PM] (jlowe) HADOOP-14912. FairCallQueue may defer 
servicing calls. Contributed by
[Oct 10, 2017 9:23:29 PM] (manojpec) HDFS-12573. Divide the total blocks 
metrics into replicated and erasure
[Oct 11, 2017 12:35:49 AM] (wang) HDFS-12547. Extend TestQuotaWithStripedBlocks 
with a random EC policy.
[Oct 11, 2017 1:42:32 AM] (aajisaka) HADOOP-13514. Upgrade maven surefire 
plugin to 2.19.1.




-1 overall


The following subsystems voted -1:
unit


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

Failed junit tests :

   hadoop.security.TestRaceWhenRelogin 
   hadoop.net.TestDNS 
   hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency 
   hadoop.hdfs.server.namenode.TestDecommissioningStatus 
   hadoop.yarn.server.nodemanager.scheduler.TestDistributedScheduler 
   hadoop.yarn.server.resourcemanager.ahs.TestRMApplicationHistoryWriter 
   hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSAppAttempt 
   hadoop.yarn.server.TestContainerManagerSecurity 
   hadoop.yarn.server.router.webapp.TestRouterWebServicesREST 

Timed out junit tests :

   
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.conf.TestZKConfigurationStore
 
   
org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.TestRMContainerImpl 
   org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart 
   
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler
 
   
org.apache.hadoop.yarn.server.resourcemanager.scheduler.TestAbstractYarnScheduler
 
   org.apache.hadoop.yarn.server.resourcemanager.TestRMHA 
   
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestNodeLabelContainerAllocation
 
   
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler 
   
org.apache.hadoop.yarn.server.resourcemanager.recovery.TestFSRMStateStore 
   
org.apache.hadoop.yarn.server.resourcemanager.recovery.TestZKRMStateStore 
   
org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesAppsModification
 
   
org.apache.hadoop.yarn.server.resourcemanager.TestApplicationMasterService 
   org.apache.hadoop.yarn.server.resourcemanager.TestResourceTrackerService 
   org.apache.hadoop.yarn.server.resourcemanager.TestLeaderElectorService 
   
org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesReservation
 
   
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue 
   
org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesSchedulerActivities
 
   
org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesApps 
   org.apache.hadoop.yarn.server.resourcemanager.TestRMProxyUsersConf 
   org.apache.hadoop.yarn.server.resourcemanager.TestContainerResourceUsage 
   
org.apache.hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA 
   
org.apache.hadoop.yarn.server.resourcemanager.rmapp.TestApplicationLifetimeMonitor
 
   
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerAsyncScheduling
 
   
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestIncreaseAllocationExpirer
 
   
org.apache.hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart 
   
org.apache.hadoop.yarn.server.resourcemanager.recovery.TestZKRMStateStorePerf 
   
org.apache.hadoop.yarn.server.resourcemanager.TestWorkPreservingRMRestart 
   org.apache.hadoop.yarn.server.resourcemanager.security.TestAMRMTokens 
   
org.apache.hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer
 
   org.apache.hadoop.yarn.client.api.impl.TestAMRMClient 
  

   cc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/554/artifact/out/diff-compile-cc-root.txt
  [4.0K]

   javac:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/554/artifact/out/diff-compile-javac-root.txt
  [288K]

   checkstyle:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/554/artifact/out/diff-checkstyle-root.txt
  [17M]

   pylint:

   

[jira] [Created] (MAPREDUCE-6981) Map Progress is misleading for Distcp job

[Prabhu Joseph (JIRA)]

Prabhu Joseph created MAPREDUCE-6981:


 Summary: Map Progress is misleading for Distcp job
 Key: MAPREDUCE-6981
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6981
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: distcp
Affects Versions: 2.7.3
Reporter: Prabhu Joseph
Priority: Minor


The Progress displayed by client when running Distcp job is misleading. The Map 
Progress reaches 100% earlier than the map tasks finishes. The issue reproduced 
by just running Distcp with multiple huge files. 

JobImpl returns progress 1.0 when either task finishes or task progress is 1.0. 
The MapTask of Distcp gets the progress from SequenceFileRecordReader which 
looks like updates the progress after reading the list of files and which does 
not account the time taken to copy the files into Destination.

{code}
17/10/11 13:33:29 INFO mapreduce.Job:  map 100% reduce 0%
17/10/11 13:34:47 INFO mapreduce.Job: Job job_1506610341926_0016 completed 
successfully
{code}

The MapTask Progress is displayed at 17/10/11 13:33:29 whereas the last map 
task finishes at 2017-10-11 13:34:45

{code}
2017-10-11 13:34:45,159 INFO [AsyncDispatcher event handler] 
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: 
task_1506610341926_0016_m_02 Task Transitioned from RUNNING to SUCCEEDED
{code}

Attaching the client and application logs.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org