Re: [VOTE] Merge Absolute resource configuration support in Capacity Scheduler (YARN-5881) to trunk

2017-12-07 Thread Jian He 
My late +1 

> On Dec 7, 2017, at 7:38 PM, Sunil G  wrote:
> 
> Thank You all.
> 
> We merged the branch to trunk and updated jiras accordingly. Thanks
> everyone who helped in this feature.
> 
> - Sunil and Wangda
> 
> 
> On Thu, Dec 7, 2017 at 10:13 PM Sunil G  wrote:
> 
>> And lastly +1 (binding) from myself.
>> Vote passes with 6 (+1) bindings by considering Weiwei's vote as binding
>> itself.
>> 
>> Thank you very much for all who voted. I’ll merge to trunk by the end of
>> today.
>> 
>> 
>> - Sunil
>> 
>> 
>> 
>> On Thu, Dec 7, 2017 at 8:08 AM Subramaniam V K  wrote:
>> 
>>> +1.
>>> 
>>> Skimmed through the design doc and uber patch and seems to be reasonable.
>>> 
>>> This is a welcome addition especially w.r.t. cloud deployments so thanks
>>> to everyone who worked on this.
>>> 
>>> On Mon, Dec 4, 2017 at 8:18 PM, Rohith Sharma K S <
>>> rohithsharm...@apache.org> wrote:
>>> 
 +1
 
 On Nov 30, 2017 7:26 AM, "Sunil G"  wrote:
 
> Hi All,
> 
> 
> Based on the discussion at [1], I'd like to start a vote to merge
 feature
> branch
> 
> YARN-5881 to trunk. Vote will run for 7 days, ending Wednesday Dec 6 at
> 6:00PM PDT.
> 
> 
> This branch adds support to configure queue capacity as absolute
 resource
> in
> 
> capacity scheduler. This will help admins who want fine control of
> resources of queues.
> 
> 
> Feature development is done at YARN-5881 [2], jenkins build is here
> (YARN-7510 [3]).
> 
> All required tasks for this feature are committed. This feature changes
> RM’s Capacity Scheduler only,
> 
> and we did extensive tests for the feature in the last couple of months
> including performance tests.
> 
> 
> Key points:
> 
> - The feature is turned off by default, and have to configure absolute
> resource to enable same.
> 
> - Detailed documentation about how to use this feature is done as part
 of
> [4].
> 
> - No major performance degradation is observed with this branch work.
 SLS
> and UT performance
> 
> tests are done.
> 
> 
> There were 11 subtasks completed for this feature.
> 
> 
> Huge thanks to everyone who helped with reviews, commits, guidance, and
> 
> technical discussion/design, including Wangda Tan, Vinod Vavilapalli,
> Rohith Sharma K S, Eric Payne .
> 
> 
> [1] :
> http://mail-archives.apache.org/mod_mbox/hadoop-yarn-dev/201711.mbox/%
> 3CCACYiTuhKhF1JCtR7ZFuZSEKQ4sBvN_n_tV5GHsbJ3YeyJP%2BP4Q%
> 40mail.gmail.com%3E
> 
> [2] : https://issues.apache.org/jira/browse/YARN-5881
> 
> [3] : https://issues.apache.org/jira/browse/YARN-7510
> 
> [4] : https://issues.apache.org/jira/browse/YARN-7533
> 
> 
> Regards
> 
> Sunil and Wangda
> 
 
>>> 
>>>

[VOTE] Release Apache Hadoop 2.7.5 (RC1)

2017-12-07 Thread Konstantin Shvachko 
Hi everybody,

I updated CHANGES.txt and fixed documentation links.
Also committed  MAPREDUCE-6165, which fixes a consistently failing test.

This is RC1 for the next dot release of Apache Hadoop 2.7 line. The
previous one 2.7.4 was release August 4, 2017.
Release 2.7.5 includes critical bug fixes and optimizations. See more
details in Release Note:
http://home.apache.org/~shv/hadoop-2.7.5-RC1/releasenotes.html

The RC0 is available at: http://home.apache.org/~shv/hadoop-2.7.5-RC1/

Please give it a try and vote on this thread. The vote will run for 5 days
ending 12/13/2017.

My up to date public key is available from:
https://dist.apache.org/repos/dist/release/hadoop/common/KEYS

Thanks,
--Konstantin

[jira] [Created] (MAPREDUCE-7022) Fast fail rogue jobs based on task scratch dir size

2017-12-07 Thread Johan Gustavsson (JIRA) 
Johan Gustavsson created MAPREDUCE-7022:
---

 Summary: Fast fail rogue jobs based on task scratch dir size
 Key: MAPREDUCE-7022
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-7022
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: task
Reporter: Johan Gustavsson


With the introduction of MAPREDUCE-6489 there are some options to kill rogue 
tasks based on writes to local disk writes. In our environment are we mainly 
run Hive based jobs we noticed that this counter and the size of the local 
scratch dirs were very different. We had tasks where BYTES_WRITTEN counter were 
at 300Gb and where it was at 10Tb both producing around 200Gb on local disk, so 
it didn't help us much. So to extend this feature tasks should monitor local 
scratchdir size and fail if they pass the limit. In these cases the tasks 
should not be retried either but instead the job should fast fail.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: VOTE] Release Apache Hadoop 2.7.5 (RC0)

2017-12-07 Thread Konstantin Shvachko 
Hey Brahma,

Yes, I see some commits did not update CHANGES.txt. I see 2 options
1. Commit CHANGES.txt to 2.7.6, so that they are not lost, and make sure
they are available while publishing documentation for the release.
2. Update CHANGES.txt on 2.7.5, build and put RC1 for another vote.

LMK what people think?

Thanks,
--Konstantin

On Wed, Dec 6, 2017 at 2:46 PM, Erik Krogen  wrote:

> +1 (non-binding)
>
> - Verified signatures, MD5, RMD160, SHA* for bin and src tarballs
> - Built from source on macOS 10.12.6 and RHEL 6.6
> - Ran local HDFS cluster, ran basic commands, verified read and write
> capability.
> - Ran 3000 node cluster via Dynamometer and do not see significant
> performance variation from 2.7.4 expectations
>
> @Brahma, I was able to find HDFS-12831, HADOOP-14881, and HADOOP-14827 in
> CHANGES.txt, but agree with you on the others listed. I was, however, able
> to find all of them in the linked releasenotes.html.
>
> Thanks Konstantin!
>
> Erik
>
> On 12/4/17, 10:50 PM, "Brahma Reddy Battula"  com> wrote:
>
> +1  (non-binding), thanks Konstantin for driving this.
>
>
> --Built from the source
> --Installed 3 Node HA Cluster
> --Ran basic shell commands
> --Verified append/snapshot/truncate
> --Ran sample jobs like pi,wordcount
>
>
> Looks follow commits are missed in changes.txt.
>
> MAPREDUCE-6975
> HADOOP-14919
> HDFS-12596
> YARN-7084
> HADOOP-14881
> HADOOP-14827
> HDFS-12832
>
>
> --Brahma Reddy Battula
>
> -Original Message-
> From: Konstantin Shvachko [mailto:shv.had...@gmail.com]
> Sent: 02 December 2017 10:13
> To: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org;
> mapreduce-dev@hadoop.apache.org; yarn-...@hadoop.apache.org
> Subject: VOTE] Release Apache Hadoop 2.7.5 (RC0)
>
> Hi everybody,
>
> This is the next dot release of Apache Hadoop 2.7 line. The previous
> one
> 2.7.4 was release August 4, 2017.
> Release 2.7.5 includes critical bug fixes and optimizations. See more
> details in Release Note:
> http://home.apache.org/~shv/hadoop-2.7.5-RC0/releasenotes.html
>
> The RC0 is available at: http://home.apache.org/~shv/hadoop-2.7.5-RC0/
>
> Please give it a try and vote on this thread. The vote will run for 5
> days ending 12/08/2017.
>
> My up to date public key is available from:
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
> Thanks,
> --Konstantin
>
>
>

Re: VOTE] Release Apache Hadoop 2.7.5 (RC0)

2017-12-07 Thread Konstantin Shvachko 
Hey Nagarasimha,

Good find. thanks.
Indeed we need those as they are linked from the side bar.
I'll make sure to add CHANGES.txt when publishing new documentation.

Thanks,
--Konstantin


On Wed, Dec 6, 2017 at 4:02 PM, Naganarasimha Garla <
naganarasimha...@apache.org> wrote:

> Thanks for the release Konstantin.
>
> Verified the following:
> - Downloaded the tar on Ubuntu and verified the signatures
> - Deployed pseudo cluster
> - Sanity checks
> - Basic hdfs operations
> - Spark PyWordcount & few MR jobs
> - Accessed most of the web UI's
>
> when accessing the docs(from the tar) was able to notice :
> -  Release Notes, Common, HDFS, MapReduce Changes showing file not
> found
> -  I observed that changes for all components were not available
> for 2.7.4 as well (http://hadoop.apache.org/docs/r2.7.4/hadoop-project-
> dist/hadoop-common/CHANGES.txt)
>
> So not sure whether its missed or not required, else everything else is
> fine.
>
> Regards,
> + Naga
>
>
> On Tue, Dec 5, 2017 at 2:50 PM, Brahma Reddy Battula <
> brahmareddy.batt...@huawei.com> wrote:
>
>> +1  (non-binding), thanks Konstantin for driving this.
>>
>>
>> --Built from the source
>> --Installed 3 Node HA Cluster
>> --Ran basic shell commands
>> --Verified append/snapshot/truncate
>> --Ran sample jobs like pi,wordcount
>>
>>
>> Looks follow commits are missed in changes.txt.
>>
>> MAPREDUCE-6975
>> HADOOP-14919
>> HDFS-12596
>> YARN-7084
>> HADOOP-14881
>> HADOOP-14827
>> HDFS-12832
>>
>>
>> --Brahma Reddy Battula
>>
>> -Original Message-
>> From: Konstantin Shvachko [mailto:shv.had...@gmail.com]
>> Sent: 02 December 2017 10:13
>> To: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org;
>> mapreduce-dev@hadoop.apache.org; yarn-...@hadoop.apache.org
>> Subject: VOTE] Release Apache Hadoop 2.7.5 (RC0)
>>
>> Hi everybody,
>>
>> This is the next dot release of Apache Hadoop 2.7 line. The previous one
>> 2.7.4 was release August 4, 2017.
>> Release 2.7.5 includes critical bug fixes and optimizations. See more
>> details in Release Note:
>> http://home.apache.org/~shv/hadoop-2.7.5-RC0/releasenotes.html
>>
>> The RC0 is available at: http://home.apache.org/~shv/hadoop-2.7.5-RC0/
>>
>> Please give it a try and vote on this thread. The vote will run for 5
>> days ending 12/08/2017.
>>
>> My up to date public key is available from:
>> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>>
>> Thanks,
>> --Konstantin
>>
>
>

Apache Hadoop qbt Report: branch2+JDK7 on Linux/x86

2017-12-07 Thread Apache Jenkins Server 
For more details, see 
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/61/

[Dec 4, 2017 6:40:22 PM] (xiao) HDFS-12396. Webhdfs file system should get 
delegation token from kms
[Dec 4, 2017 9:18:04 PM] (rkanter) YARN-5594. Handle old RMDelegationToken 
format when recovering RM
[Dec 5, 2017 5:31:44 AM] (aajisaka) HADOOP-14985. Remove subversion related 
code from VersionInfoMojo.java.
[Dec 5, 2017 2:31:33 PM] (brahma) HDFS-11751. DFSZKFailoverController daemon 
exits with wrong status code.
[Dec 6, 2017 2:42:51 AM] (aajisaka) HDFS-12889. Router UI is missing robots.txt 
file. Contributed by Bharat




-1 overall


The following subsystems voted -1:
unit xml


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

Unreaped Processes :

   hadoop-common:1 
   hadoop-hdfs:23 
   bkjournal:5 
   hadoop-yarn-common:1 
   hadoop-yarn-server-timelineservice:1 
   hadoop-yarn-server-resourcemanager:1 
   hadoop-yarn-client:8 
   hadoop-yarn-applications-distributedshell:1 
   hadoop-mapreduce-client-app:1 
   hadoop-mapreduce-client-jobclient:2 
   hadoop-distcp:3 

Failed junit tests :

   hadoop.hdfs.TestFsShellPermission 
   hadoop.hdfs.TestParallelShortCircuitRead 
   
hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesDelegationTokenAuthentication
 
   
hadoop.yarn.server.resourcemanager.reservation.TestCapacityOverTimePolicy 
   hadoop.yarn.server.TestDiskFailures 
   
hadoop.yarn.applications.distributedshell.TestDistributedShellWithNodeLabels 
   hadoop.mapreduce.v2.TestUberAM 
   hadoop.tools.TestDistCpViewFs 
   hadoop.tools.TestIntegration 
   hadoop.resourceestimator.service.TestResourceEstimatorService 
   hadoop.resourceestimator.solver.impl.TestLpSolver 

Timed out junit tests :

   org.apache.hadoop.log.TestLogLevel 
   org.apache.hadoop.hdfs.TestEncryptionZones 
   org.apache.hadoop.hdfs.TestLargeBlock 
   org.apache.hadoop.hdfs.TestDFSStartupVersions 
   org.apache.hadoop.hdfs.TestHdfsAdmin 
   org.apache.hadoop.hdfs.TestDFSClientFailover 
   org.apache.hadoop.hdfs.TestQuota 
   org.apache.hadoop.hdfs.TestFileLengthOnClusterRestart 
   org.apache.hadoop.hdfs.TestDecommission 
   org.apache.hadoop.hdfs.client.impl.TestBlockReaderLocal 
   org.apache.hadoop.hdfs.TestDFSRemove 
   org.apache.hadoop.hdfs.TestHDFSServerPorts 
   org.apache.hadoop.hdfs.TestRollingUpgradeRollback 
   org.apache.hadoop.hdfs.client.impl.TestBlockReaderFactory 
   org.apache.hadoop.hdfs.client.impl.TestBlockReaderLocalLegacy 
   org.apache.hadoop.hdfs.TestPipelines 
   org.apache.hadoop.hdfs.TestFileConcurrentReader 
   org.apache.hadoop.hdfs.TestDFSClientSocketSize 
   org.apache.hadoop.hdfs.TestFSOutputSummer 
   org.apache.hadoop.fs.permission.TestStickyBit 
   org.apache.hadoop.hdfs.TestGetFileChecksum 
   org.apache.hadoop.tracing.TestTraceAdmin 
   org.apache.hadoop.hdfs.TestAbandonBlock 
   org.apache.hadoop.cli.TestHDFSCLI 
   org.apache.hadoop.contrib.bkjournal.TestBookKeeperJournalManager 
   org.apache.hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints 
   org.apache.hadoop.contrib.bkjournal.TestBookKeeperEditLogStreams 
   org.apache.hadoop.contrib.bkjournal.TestCurrentInprogress 
   org.apache.hadoop.contrib.bkjournal.TestBookKeeperConfiguration 
   org.apache.hadoop.yarn.webapp.TestWebApp 
   
org.apache.hadoop.yarn.server.timelineservice.reader.TestTimelineReaderWebServices
 
   
org.apache.hadoop.yarn.server.resourcemanager.recovery.TestFSRMStateStore 
   org.apache.hadoop.yarn.client.api.impl.TestAMRMProxy 
   org.apache.hadoop.yarn.client.TestRMFailover 
   
org.apache.hadoop.yarn.client.api.impl.TestOpportunisticContainerAllocationE2E 
   org.apache.hadoop.yarn.client.TestApplicationClientProtocolOnHA 
   org.apache.hadoop.yarn.client.api.impl.TestYarnClientWithReservation 
   org.apache.hadoop.yarn.client.api.impl.TestYarnClient 
   org.apache.hadoop.yarn.client.api.impl.TestAMRMClient 
   org.apache.hadoop.yarn.client.api.impl.TestNMClient 
   
org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell 
   org.apache.hadoop.mapreduce.v2.app.TestJobEndNotifier 
   org.apache.hadoop.mapred.TestClusterMapReduceTestCase 
   org.apache.hadoop.mapred.TestMRTimelineEventHandling 
   org.apache.hadoop.tools.TestDistCpSync 
   org.apache.hadoop.tools.TestDistCpSyncReverseFromTarget 
   org.apache.hadoop.tools.TestDistCpSyncReverseFromSource 
  

   cc:

   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/61/artifact/out/diff-compile-cc-root.txt