Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

2017-11-25 Thread Apache Jenkins Server 
For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/604/

[Nov 25, 2017 7:32:46 AM] (yufei) YARN-7290. Method canContainerBePreempted can 
return true when it




-1 overall


The following subsystems voted -1:
asflicense findbugs unit


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

FindBugs :

   module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api 
   org.apache.hadoop.yarn.api.records.Resource.getResources() may expose 
internal representation by returning Resource.resources At Resource.java:by 
returning Resource.resources At Resource.java:[line 213] 

Failed junit tests :

   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure 
   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150 
   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure030 
   hadoop.hdfs.web.TestWebHdfsTimeouts 
   hadoop.fs.TestUnbuffer 
   hadoop.hdfs.server.balancer.TestBalancerRPCDelay 
   hadoop.hdfs.TestErasureCodingPolicies 
   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure140 
   
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestNodeLabelContainerAllocation
 
  

   cc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/604/artifact/out/diff-compile-cc-root.txt
  [4.0K]

   javac:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/604/artifact/out/diff-compile-javac-root.txt
  [276K]

   checkstyle:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/604/artifact/out/diff-checkstyle-root.txt
  [17M]

   pylint:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/604/artifact/out/diff-patch-pylint.txt
  [20K]

   shellcheck:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/604/artifact/out/diff-patch-shellcheck.txt
  [20K]

   shelldocs:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/604/artifact/out/diff-patch-shelldocs.txt
  [12K]

   whitespace:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/604/artifact/out/whitespace-eol.txt
  [8.8M]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/604/artifact/out/whitespace-tabs.txt
  [288K]

   findbugs:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/604/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api-warnings.html
  [8.0K]

   javadoc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/604/artifact/out/diff-javadoc-javadoc-root.txt
  [760K]

   unit:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/604/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
  [352K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/604/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
  [80K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/604/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
  [84K]

   asflicense:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/604/artifact/out/patch-asflicense-problems.txt
  [4.0K]

Powered by Apache Yetus 0.7.0-SNAPSHOT   http://yetus.apache.org

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.0.0 RC0

2017-11-25 Thread Allen Wittenauer 

> On Nov 21, 2017, at 2:16 PM, Vinod Kumar Vavilapalli  
> wrote:
> 
>>> - $HADOOP_YARN_HOME/sbin/yarn-daemon.sh start historyserver doesn't even 
>>> work. Not just deprecated in favor of timelineserver as was advertised.
>> 
>>  This works for me in trunk and the bash code doesn’t appear to have 
>> changed in a very long time.  Probably something local to your install.  (I 
>> do notice that the deprecation message says “starting” which is awkward when 
>> the stop command is given though.)  Also: is the deprecation message even 
>> true at this point?
> 
> 
> Sorry, I mischaracterized the problem.
> 
> The real issue is that I cannot use this command line when the MapReduce 
> JobHistoryServer is already started on the same machine.

The specific string is:

hadoop-${HADOOP_IDENT_STRING}-${HADOOP_SUBCMD}.pid

More specifically, the pid handling code will conflict if the following 
are true:

* same machine (obviously)
* same subcommand name
* same HADOOP_IDENT_USER: which by default is the user name of 
whatever starts it… but was designed to be overridden way back in hadoop 0.X.

… which means for most production setups, this is probably not real a 
problem.


> So, it looks like in shell-scripts, there can ever be only one daemon of a 
> given name, irrespective of which daemon scripts are invoked.

Correct.  Naming multiple, different daemons the same thing is 
extremely anti-user.   In fact, I thought this was originally about the “other” 
history server.

> 
> We need to figure out two things here
>  (a) The behavior of this command. Clearly, it will conflict with the 
> MapReduce JHS - only one of them can be started on the same node.

… by the same user, by default.  Started by a different user or 
different HADOOP_IDENT_USER, it will come up just fine.

>  (b) We need to figure out if this V1 TimelineService should even be support 
> given ATSv2.

If ATSv1 isn’t replaced by ATSv2, then why is it marked deprecated?

> On Nov 22, 2017, at 9:45 AM, Brahma Reddy Battula  wrote:
> 
> 1) Change the name
> 2) Create PID based on the CLASS Name, here applicationhistoryserver and 
> jobhistoryserver
> 3) Use same as branch-2.9..i.e suffixing with mapred or yarn
> 
> 
> @allen, any thoughts on this..?

Using the classname works in this instance, but just as we saw with the 
router daemons, people tend to use the same class names when building different 
components. It also means that if different daemons can be started in different 
ways from the same class dependent upon options, this conflict will still 
exist.  Also, with dynamic commands, it is very possible to run the same daemon 
from multiple start points.

As part of this discussion, I think it’s important to recognize:

a) This is likely to be primarily impacting developers.
b) We’re talking about two daemons where one has been deprecated.
c) Calling two different daemons “history server” is just awful from an end 
user perspective.
d) There is already a work around in place if one absolutely needs to run both 
on the same node as the same user, just as people do with datanode and 
nodemanager today.



-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Apache Hadoop qbt Report: branch2+JDK7 on Linux/x86

2017-11-25 Thread Apache Jenkins Server 
For more details, see 
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/50/

[Nov 23, 2017 5:02:23 PM] (xiao) HADOOP-15067. GC time percentage reported in 
JvmMetrics should be a
[Nov 24, 2017 4:06:16 AM] (wangda) YARN-7509. AsyncScheduleThread and 
ResourceCommitterService are still




-1 overall


The following subsystems voted -1:
asflicense unit xml


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

Unreaped Processes :

   hadoop-common:1 
   hadoop-hdfs:22 
   bkjournal:5 
   hadoop-yarn-server-timelineservice:1 
   hadoop-yarn-server-resourcemanager:1 
   hadoop-yarn-client:3 
   hadoop-yarn-applications-distributedshell:1 
   hadoop-mapreduce-client-jobclient:14 
   hadoop-distcp:3 

Failed junit tests :

   
hadoop.yarn.server.resourcemanager.reservation.TestCapacityOverTimePolicy 
   
hadoop.yarn.applications.distributedshell.TestDistributedShellWithNodeLabels 
   hadoop.mapreduce.v2.TestMRAMWithNonNormalizedCapabilities 
   hadoop.mapreduce.TestMapReduceLazyOutput 
   hadoop.mapreduce.v2.TestMRJobsWithHistoryService 
   hadoop.tools.TestIntegration 
   hadoop.tools.TestDistCpViewFs 
   hadoop.resourceestimator.solver.impl.TestLpSolver 
   hadoop.resourceestimator.service.TestResourceEstimatorService 

Timed out junit tests :

   org.apache.hadoop.log.TestLogLevel 
   org.apache.hadoop.hdfs.TestLeaseRecovery2 
   org.apache.hadoop.hdfs.TestDatanodeRegistration 
   org.apache.hadoop.hdfs.TestBlocksScheduledCounter 
   org.apache.hadoop.hdfs.TestDFSClientFailover 
   org.apache.hadoop.hdfs.TestDFSClientRetries 
   org.apache.hadoop.hdfs.web.TestWebHdfsTokens 
   org.apache.hadoop.hdfs.TestDFSInotifyEventInputStream 
   org.apache.hadoop.hdfs.TestFileAppendRestart 
   org.apache.hadoop.hdfs.web.TestWebHdfsWithRestCsrfPreventionFilter 
   org.apache.hadoop.hdfs.TestDatanodeReport 
   org.apache.hadoop.hdfs.web.TestWebHDFS 
   org.apache.hadoop.hdfs.web.TestWebHDFSXAttr 
   org.apache.hadoop.hdfs.web.TestWebHdfsWithMultipleNameNodes 
   org.apache.hadoop.metrics2.sink.TestRollingFileSystemSinkWithHdfs 
   org.apache.hadoop.hdfs.TestMiniDFSCluster 
   org.apache.hadoop.hdfs.TestDistributedFileSystem 
   org.apache.hadoop.hdfs.web.TestWebHDFSForHA 
   org.apache.hadoop.hdfs.TestTrashWithEncryptionZones 
   org.apache.hadoop.hdfs.TestSetTimes 
   org.apache.hadoop.hdfs.TestReplaceDatanodeFailureReplication 
   org.apache.hadoop.hdfs.TestDFSShell 
   org.apache.hadoop.hdfs.web.TestWebHDFSAcl 
   org.apache.hadoop.contrib.bkjournal.TestBootstrapStandbyWithBKJM 
   org.apache.hadoop.contrib.bkjournal.TestBookKeeperJournalManager 
   org.apache.hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints 
   org.apache.hadoop.contrib.bkjournal.TestBookKeeperAsHASharedDir 
   org.apache.hadoop.contrib.bkjournal.TestBookKeeperSpeculativeRead 
   
org.apache.hadoop.yarn.server.timelineservice.reader.TestTimelineReaderWebServices
 
   
org.apache.hadoop.yarn.server.resourcemanager.reservation.TestFairSchedulerPlanFollower
 
   org.apache.hadoop.yarn.client.TestRMFailover 
   org.apache.hadoop.yarn.client.TestApplicationClientProtocolOnHA 
   org.apache.hadoop.yarn.client.api.impl.TestAMRMClient 
   
org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell 
   org.apache.hadoop.mapreduce.TestMRJobClient 
   org.apache.hadoop.mapred.TestMiniMRClasspath 
   org.apache.hadoop.mapred.TestClusterMapReduceTestCase 
   org.apache.hadoop.mapreduce.v2.TestUberAM 
   org.apache.hadoop.mapreduce.v2.TestSpeculativeExecution 
   org.apache.hadoop.mapred.TestMRTimelineEventHandling 
   org.apache.hadoop.mapreduce.v2.TestMRAppWithCombiner 
   org.apache.hadoop.mapreduce.v2.TestMRJobs 
   org.apache.hadoop.mapreduce.v2.TestMRJobsWithProfiler 
   org.apache.hadoop.mapreduce.lib.output.TestJobOutputCommitter 
   org.apache.hadoop.mapreduce.v2.TestMROldApiJobs 
   org.apache.hadoop.mapreduce.TestChild 
   org.apache.hadoop.mapreduce.v2.TestRMNMInfo 
   org.apache.hadoop.mapreduce.v2.TestMiniMRProxyUser 
   org.apache.hadoop.tools.TestDistCpSync 
   org.apache.hadoop.tools.TestDistCpSyncReverseFromTarget 
   org.apache.hadoop.tools.TestDistCpSyncReverseFromSource 
  

   cc:

   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/50/artifact/out/diff-compile-cc-root.txt
  [4.0K]

   javac:

   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/50/artifact/out/diff-compile-javac-root.txt
  [324K]

   checkstyle: