Apache Hadoop qbt Report: trunk+JDK8 on Windows/x64

2018-04-04 Thread Apache Jenkins Server 
For more details, see https://builds.apache.org/job/hadoop-trunk-win/427/

[Apr 3, 2018 2:59:20 PM] (haibochen) YARN-8051. 
TestRMEmbeddedElector#testCallbackSynchronization is flaky.
[Apr 3, 2018 3:31:34 PM] (stevel) HADOOP-14758. S3GuardTool.prune to handle 
UnsupportedOperationException.
[Apr 3, 2018 5:01:00 PM] (szegedim) YARN-8035. Uncaught exception in 
ContainersMonitorImpl during relaunch
[Apr 4, 2018 3:52:35 AM] (sunilg) YARN-7764. Findbugs warning: 
Resource#getResources may expose internal
[Apr 4, 2018 4:06:24 AM] (wangda) YARN-8106. Update 
LogAggregationIndexedFileController to use readFull




-1 overall


The following subsystems voted -1:
compile mvninstall unit


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc javac


The following subsystems are considered long running:
(runtime bigger than 1h 00m 00s)
unit


Specific tests:

Failed junit tests :

   hadoop.crypto.TestCryptoStreamsWithOpensslAesCtrCryptoCodec 
   hadoop.fs.contract.rawlocal.TestRawlocalContractAppend 
   hadoop.fs.TestFileUtil 
   hadoop.fs.TestFsShellCopy 
   hadoop.fs.TestFsShellList 
   hadoop.fs.TestLocalFileSystem 
   hadoop.fs.TestRawLocalFileSystemContract 
   hadoop.fs.TestSymlinkLocalFSFileContext 
   hadoop.fs.TestTrash 
   hadoop.http.TestHttpServer 
   hadoop.http.TestHttpServerLogs 
   hadoop.io.nativeio.TestNativeIO 
   hadoop.ipc.TestSocketFactory 
   hadoop.metrics2.impl.TestStatsDMetrics 
   hadoop.metrics2.sink.TestRollingFileSystemSinkWithLocal 
   hadoop.security.TestSecurityUtil 
   hadoop.security.TestShellBasedUnixGroupsMapping 
   hadoop.security.token.TestDtUtilShell 
   hadoop.util.TestNativeCodeLoader 
   hadoop.util.TestWinUtils 
   hadoop.fs.TestResolveHdfsSymlink 
   hadoop.hdfs.crypto.TestHdfsCryptoStreams 
   hadoop.hdfs.qjournal.client.TestQuorumJournalManager 
   hadoop.hdfs.qjournal.server.TestJournalNode 
   hadoop.hdfs.qjournal.server.TestJournalNodeSync 
   hadoop.hdfs.server.blockmanagement.TestNameNodePrunesMissingStorages 
   hadoop.hdfs.server.blockmanagement.TestOverReplicatedBlocks 
   hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl 
   hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistFiles 
   hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistLockedMemory 
   hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistPolicy 
   
hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaPlacement 
   
hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery 
   hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyWriter 
   hadoop.hdfs.server.datanode.fsdataset.impl.TestProvidedImpl 
   hadoop.hdfs.server.datanode.fsdataset.impl.TestSpaceReservation 
   hadoop.hdfs.server.datanode.fsdataset.impl.TestWriteToReplica 
   hadoop.hdfs.server.datanode.TestBlockPoolSliceStorage 
   hadoop.hdfs.server.datanode.TestBlockRecovery 
   hadoop.hdfs.server.datanode.TestBlockScanner 
   hadoop.hdfs.server.datanode.TestDataNodeErasureCodingMetrics 
   hadoop.hdfs.server.datanode.TestDataNodeFaultInjector 
   hadoop.hdfs.server.datanode.TestDataNodeMetrics 
   hadoop.hdfs.server.datanode.TestDataNodeRollingUpgrade 
   hadoop.hdfs.server.datanode.TestDataNodeUUID 
   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure 
   hadoop.hdfs.server.datanode.TestDirectoryScanner 
   hadoop.hdfs.server.datanode.TestHSync 
   hadoop.hdfs.server.datanode.TestNNHandlesBlockReportPerStorage 
   hadoop.hdfs.server.datanode.web.TestDatanodeHttpXFrame 
   hadoop.hdfs.server.diskbalancer.command.TestDiskBalancerCommand 
   hadoop.hdfs.server.diskbalancer.TestDiskBalancerRPC 
   hadoop.hdfs.server.mover.TestMover 
   hadoop.hdfs.server.mover.TestStorageMover 
   hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA 
   hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA 
   hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics 
   
hadoop.hdfs.server.namenode.snapshot.TestINodeFileUnderConstructionWithSnapshot 
   hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot 
   hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots 
   hadoop.hdfs.server.namenode.snapshot.TestSnapRootDescendantDiff 
   hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport 
   hadoop.hdfs.server.namenode.TestAddBlock 
   hadoop.hdfs.server.namenode.TestAuditLoggerWithCommands 
   hadoop.hdfs.server.namenode.TestCheckpoint 
   hadoop.hdfs.server.namenode.TestDiskspaceQuotaUpdate 
   hadoop.hdfs.server.namenode.TestEditLogRace 
   hadoop.hdfs.server.namenode.TestFileTruncate 
   hadoop.hdfs.server.namenode.TestFsck 
   hadoop.hdfs.server.namenode.TestFSImage 
   hadoop.hdfs.server.namenode.TestFSImageWithSnapshot

Apache Hadoop qbt Report: branch2+JDK7 on Linux/x86

2018-04-04 Thread Apache Jenkins Server 
For more details, see 
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/185/

[Apr 4, 2018 2:16:20 PM] (stevel) HADOOP-14651. Update okhttp version to 2.7.5. 
Contributed by Ray Chiang




-1 overall


The following subsystems voted -1:
docker


Powered by Apache Yetus 0.8.0-SNAPSHOT   http://yetus.apache.org

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Created] (YARN-8118) Better utilize gracefully decommissioning node managers

2018-04-04 Thread Karthik Palaniappan (JIRA) 
Karthik Palaniappan created YARN-8118:
-

 Summary: Better utilize gracefully decommissioning node managers
 Key: YARN-8118
 URL: https://issues.apache.org/jira/browse/YARN-8118
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: yarn
Affects Versions: 2.8.2
 Environment: * Google Compute Engine (Dataproc)
 * Java 8
 * Hadoop 2.8.2 using client-mode graceful decommissioning
Reporter: Karthik Palaniappan


Proposal design doc with background + details: (will link – please comment on 
that doc)

tl;dr Right now, DECOMMISSIONING nodes must wait for in-progress applications 
to complete before shutting down, but they cannot run new containers from those 
in-progress applications. This is wasteful, particularly in environments where 
you are billed by resource usage (e.g. EC2).

Proposal: YARN should schedule containers from in-progress applications on 
DECOMMISSIONING nodes, but should still avoid scheduling containers from new 
applications. That will make in-progress applications complete faster and let 
nodes decommission faster. Overall, this should be cheaper.

I have a working patch without unit tests (will attach) that's surprisingly 
just a few real lines of code. If folks are happy with the proposal, I'll write 
unit tests and also write a patch targeted at trunk.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.0.1 (RC1)

2018-04-04 Thread Lei Xu 
Hi, All

Thanks Arpit to find that 3.0.1 artifacts missed shaded jars.  I will
create a new 3.0.2 release with the same codebase as 3.0.1, but deploy
it as shaded jars. A new vote thread will start later this week.

After that, 3.0.3 release will be cut from branch-3.0 and includes the
new bug fixes. Please change the target version of new fixes to 3.0.3.
I will also go over JIRAs to fix targeted versions for committed changes.

Best,

On Mon, Apr 2, 2018 at 6:08 PM, Lei Xu  wrote:
> Hi, Arpit
>
> I followed this instruction
> https://wiki.apache.org/hadoop/HowToRelease
>
> It instructs to use "mvn deploy -Psign -DskipTests -DskipShade".
>
> It seems wrong, given that 3.0.0 is a shaded jar.
> I will do another deploy to fix it.
>
>
> On Mon, Apr 2, 2018 at 5:31 PM, Arpit Agarwal  
> wrote:
>> Hi Lei,
>>
>> It looks like the release artefacts have dummy shaded jars. E.g.
>>
>> Repository Path:  
>> /org/apache/hadoop/hadoop-client-runtime/3.0.1/hadoop-client-runtime-3.0.1.jar
>> Uploaded by:  lei
>> Size: 44.47 KB
>> Uploaded Date:Fri Mar 16 2018 15:50:42 GMT-0700 (PDT)
>> Last Modified:Fri Mar 16 2018 15:50:42 GMT-0700 (PDT)
>>
>> https://repository.apache.org/index.html#view-repositories;releases~browsestorage~/org/apache/hadoop/hadoop-client-runtime/3.0.1/hadoop-client-runtime-3.0.1.jar
>>
>> Am I looking at this wrong or is this supposed to be the shaded jar which is 
>> ~20MB?
>>
>> Thanks,
>> Arpit
>>
>>
>>
>> On 3/23/18, 10:18 AM, "Lei Xu"  wrote:
>>
>> Hi, All
>>
>> Thanks everyone for voting! The vote passes successfully with 6
>> binding +1s, 7 non-binding +1s and no -1s.
>>
>> I will work on the staging and releases.
>>
>> Best,
>>
>>
>> On Fri, Mar 23, 2018 at 5:10 AM, Kuhu Shukla  
>> wrote:
>> > +1 (non-binding)
>> >
>> > Built from source.
>> > Installed on a pseudo distributed cluster.
>> > Ran word count job and basic hdfs commands.
>> >
>> > Thank you for the effort on this release.
>> >
>> > Regards,
>> > Kuhu
>> >
>> > On Thu, Mar 22, 2018 at 5:25 PM, Elek, Marton  wrote:
>> >
>> >>
>> >> +1 (non binding)
>> >>
>> >> I did a full build from source code, created a docker container and 
>> did
>> >> various basic level tests with robotframework based automation and
>> >> docker-compose based pseudo clusters[1].
>> >>
>> >> Including:
>> >>
>> >> * Hdfs federation smoke test
>> >> * Basic ViewFS configuration
>> >> * Yarn example jobs
>> >> * Spark example jobs (with and without yarn)
>> >> * Simple hive table creation
>> >>
>> >> Marton
>> >>
>> >>
>> >> [1]: https://github.com/flokkr/runtime-compose
>> >>
>> >> On 03/18/2018 05:11 AM, Lei Xu wrote:
>> >>
>> >>> Hi, all
>> >>>
>> >>> I've created release candidate RC-1 for Apache Hadoop 3.0.1
>> >>>
>> >>> Apache Hadoop 3.0.1 will be the first bug fix release for Apache
>> >>> Hadoop 3.0 release. It includes 49 bug fixes and security fixes, 
>> which
>> >>> include 12
>> >>> blockers and 17 are critical.
>> >>>
>> >>> Please note:
>> >>> * HDFS-12990. Change default NameNode RPC port back to 8020. It makes
>> >>> incompatible changes to Hadoop 3.0.0.  After 3.0.1 releases, Apache
>> >>> Hadoop 3.0.0 will be deprecated due to this change.
>> >>>
>> >>> The release page is:
>> >>> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+3.0+Release
>> >>>
>> >>> New RC is available at: http://home.apache.org/~lei/hadoop-3.0.1-RC1/
>> >>>
>> >>> The git tag is release-3.0.1-RC1, and the latest commit is
>> >>> 496dc57cc2e4f4da117f7a8e3840aaeac0c1d2d0
>> >>>
>> >>> The maven artifacts are available at:
>> >>> 
>> https://repository.apache.org/content/repositories/orgapachehadoop-1081/
>> >>>
>> >>> Please try the release and vote; the vote will run for the usual 5
>> >>> days, ending on 3/22/2017 6pm PST time.
>> >>>
>> >>> Thanks!
>> >>>
>> >>> -
>> >>> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
>> >>> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
>> >>>
>> >>>
>> >> -
>> >> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
>> >> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>> >>
>> >>
>>
>>
>>
>> --
>> Lei (Eddy) Xu
>> Software Engineer, Cloudera
>>
>> -
>> To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
>> For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
>>
>>
>>

[jira] [Resolved] (YARN-7213) [Umbrella] Test and validate HBase-2.0.x with Atsv2

2018-04-04 Thread Haibo Chen (JIRA) 

 [ 
https://issues.apache.org/jira/browse/YARN-7213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Haibo Chen resolved YARN-7213.
--
   Resolution: Done
Fix Version/s: 3.2.0

> [Umbrella] Test and validate HBase-2.0.x with Atsv2
> ---
>
> Key: YARN-7213
> URL: https://issues.apache.org/jira/browse/YARN-7213
> Project: Hadoop YARN
>  Issue Type: Task
>Reporter: Rohith Sharma K S
>Priority: Major
> Fix For: 3.2.0
>
> Attachments: YARN-7213.prelim.patch, YARN-7213.prelim.patch, 
> YARN-7213.wip.patch
>
>
> Hbase-2.0.x officially support hadoop-alpha compilations. And also they are 
> getting ready for Hadoop-beta release so that HBase can release their 
> versions compatible with Hadoop-beta. So, this JIRA is to keep track of 
> HBase-2.0 integration issues. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)

2018-04-04 Thread Suma Shivaprasad 
+1 (non binding)


*Verified - User Group Queue mapping - Node labels with New UI- Dynamic
queuesThanksSuma*


On Wed, Apr 4, 2018 at 11:48 AM, Steve Loughran 
wrote:

> that's "dangerously interesting". I think you are right, and I also think
> it'll just be the version files which get generated
>
> anyway, +1 binding
>
> * ran my new Hadoop-3 profile on spark (SPARK-23807), with the committer
> binding, then my downstream tests. All is well, provided you also have a
> spark hive JAR patched to accept hadoop 3 as a legitimate hadoop version.
> That's an ongoing issue in the Spark project. With that JAR on my CP my
> downstream tests were all happy (yesterday)
>
> * today the staging files seem to be missing, at least maven is unable to
> find them even when I turn the spark snapshots-and-staging profile on.
> That'll be the maven dist process at play, nothing else
>
> On 4 Apr 2018, at 04:13, Wangda Tan  eele...@gmail.com>> wrote:
>
> Hi Vinod / Arpit,
>
> I checked following versions:
> - 2.6.5 / 2.7.5 / 2.8.3 / 2.9.0 / 3.0.1:
>
> Jars in maven repo [1] are *always* different from jars in the binary
> tarball [2]: (I only checked hadoop-yarn-api-version.jar)
>
> (Following numbers are sizes of the jar)
> 2.6.5:
> - Jar in Maven: 1896185
> - Jar in tarball: 1891485
>
> 2.7.5:
> - Jar in Maven: 2039371 (md5: 15e76f7c734b49315ef2bce952509ddf)
> - Jar in tarball: 2039371 (md5: 0ef9f42f587401f5b49b39f27459f3ef)
> (Even size is same, md5 is different)
>
> 2.8.3:
> - Jar in Maven: 2451433
> - Jar in tarball: 2438975
>
> 2.9.0:
> - Jar in Maven: 2791477
> - Jar in tarball: 289
>
> 3.0.1:
> - Jar in Maven: 2852604
> - Jar in tarball: 2851373
>
> I guess the differences come from our release process.
>
> Thanks,
> Wangda
>
> [1] Maven jars are downloaded from
> https://repository.apache.org/service/local/repositories/
> releases/content/org/apache/hadoop/hadoop-yarn-api/
> /hadoop-yarn-api-.jar
> [2] Binary tarballs downloaded from http://apache.claz.org/hadoop/common/
>
>
> On Tue, Apr 3, 2018 at 4:25 PM, Vinod Kumar Vavilapalli <
> vino...@apache.org>
> wrote:
>
> We vote on the source code. The binaries are convenience artifacts.
>
> This is what I would do - (a) Just replace both the maven jars as well as
> the binaries to be consistent and correct. And then (b) Give a couple more
> days for folks who tested on the binaries to reverify - I count one such
> clear vote as of now.
>
> Thanks
> +Vinod
>
>
> On Apr 3, 2018, at 3:30 PM, Wangda Tan  eele...@gmail.com>> wrote:
>
> HI Arpit,
>
> I think it won't match if we do rebuild. It should be fine as far as
> they're signed, correct? I don't see any policy doesn't allow this.
>
> Thanks,
> Wangda
>
>
> On Tue, Apr 3, 2018 at 9:33 AM, Arpit Agarwal  mailto:aagar...@hortonworks.com>>
> wrote:
>
> Thanks Wangda, I see the shaded jars now.
>
> Are the repo jars required to be the same as the binary release? They
> don’t match right now, probably they got rebuilt.
>
> +1 (binding), modulo that remaining question.
>
> * Verified signatures
> * Verified checksums for source and binary artefacts
> * Sanity checked jars on r.a.o.
> * Built from source
> * Deployed to 3 node secure cluster with NameNode HA
> * Verified HDFS web UIs
> * Tried out HDFS shell commands
> * Ran sample MapReduce jobs
>
> Thanks!
>
>
> --
> From: Wangda Tan >
> Date: Monday, April 2, 2018 at 9:25 PM
> To: Arpit Agarwal  com>>
> Cc: Gera Shegalov >, Sunil G <
> sun...@apache.org>, "
> yarn-dev@hadoop.apache.org" <
> yarn-dev@hadoop.apache.org>, Hdfs-dev <
> hdfs-...@hadoop.apache.org>, Hadoop
> Common  >>,
> "mapreduce-...@hadoop.apache.org"
> >,
> Vinod Kumar Vavilapalli >
> Subject: Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
>
> As pointed by Arpit, the previously deployed shared jars are incorrect.
> Just redeployed jars and staged. @Arpit, could you please check the updated
> Maven repo? https://repository.apache.org/content/repositories/
> orgapachehadoop-1092
>
> Since the jars inside binary tarballs are correct (
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1/). I think we don't
> need roll another RC, just update Maven repo should be sufficient.
>
> Best,
> Wangda
>
>
> On Mon, Apr 2, 2018 at 2:39 PM, Wangda Tan 
> wrote:
> Hi Arpit,
>
> Thanks for

Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)

2018-04-04 Thread Steve Loughran 
that's "dangerously interesting". I think you are right, and I also think it'll 
just be the version files which get generated

anyway, +1 binding

* ran my new Hadoop-3 profile on spark (SPARK-23807), with the committer 
binding, then my downstream tests. All is well, provided you also have a spark 
hive JAR patched to accept hadoop 3 as a legitimate hadoop version. That's an 
ongoing issue in the Spark project. With that JAR on my CP my downstream tests 
were all happy (yesterday)

* today the staging files seem to be missing, at least maven is unable to find 
them even when I turn the spark snapshots-and-staging profile on. That'll be 
the maven dist process at play, nothing else

On 4 Apr 2018, at 04:13, Wangda Tan 
> wrote:

Hi Vinod / Arpit,

I checked following versions:
- 2.6.5 / 2.7.5 / 2.8.3 / 2.9.0 / 3.0.1:

Jars in maven repo [1] are *always* different from jars in the binary
tarball [2]: (I only checked hadoop-yarn-api-version.jar)

(Following numbers are sizes of the jar)
2.6.5:
- Jar in Maven: 1896185
- Jar in tarball: 1891485

2.7.5:
- Jar in Maven: 2039371 (md5: 15e76f7c734b49315ef2bce952509ddf)
- Jar in tarball: 2039371 (md5: 0ef9f42f587401f5b49b39f27459f3ef)
(Even size is same, md5 is different)

2.8.3:
- Jar in Maven: 2451433
- Jar in tarball: 2438975

2.9.0:
- Jar in Maven: 2791477
- Jar in tarball: 289

3.0.1:
- Jar in Maven: 2852604
- Jar in tarball: 2851373

I guess the differences come from our release process.

Thanks,
Wangda

[1] Maven jars are downloaded from
https://repository.apache.org/service/local/repositories/releases/content/org/apache/hadoop/hadoop-yarn-api/
/hadoop-yarn-api-.jar
[2] Binary tarballs downloaded from http://apache.claz.org/hadoop/common/


On Tue, Apr 3, 2018 at 4:25 PM, Vinod Kumar Vavilapalli 
>
wrote:

We vote on the source code. The binaries are convenience artifacts.

This is what I would do - (a) Just replace both the maven jars as well as
the binaries to be consistent and correct. And then (b) Give a couple more
days for folks who tested on the binaries to reverify - I count one such
clear vote as of now.

Thanks
+Vinod


On Apr 3, 2018, at 3:30 PM, Wangda Tan 
> wrote:

HI Arpit,

I think it won't match if we do rebuild. It should be fine as far as
they're signed, correct? I don't see any policy doesn't allow this.

Thanks,
Wangda


On Tue, Apr 3, 2018 at 9:33 AM, Arpit Agarwal 
>
wrote:

Thanks Wangda, I see the shaded jars now.

Are the repo jars required to be the same as the binary release? They
don’t match right now, probably they got rebuilt.

+1 (binding), modulo that remaining question.

* Verified signatures
* Verified checksums for source and binary artefacts
* Sanity checked jars on r.a.o.
* Built from source
* Deployed to 3 node secure cluster with NameNode HA
* Verified HDFS web UIs
* Tried out HDFS shell commands
* Ran sample MapReduce jobs

Thanks!


--
From: Wangda Tan >
Date: Monday, April 2, 2018 at 9:25 PM
To: Arpit Agarwal >
Cc: Gera Shegalov >, Sunil G 
>, "
yarn-dev@hadoop.apache.org" 
>, Hdfs-dev <
hdfs-...@hadoop.apache.org>, Hadoop Common 
>,
"mapreduce-...@hadoop.apache.org" 
>,
Vinod Kumar Vavilapalli >
Subject: Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)

As pointed by Arpit, the previously deployed shared jars are incorrect.
Just redeployed jars and staged. @Arpit, could you please check the updated
Maven repo? https://repository.apache.org/content/repositories/
orgapachehadoop-1092

Since the jars inside binary tarballs are correct (
http://people.apache.org/~wangda/hadoop-3.1.0-RC1/). I think we don't
need roll another RC, just update Maven repo should be sufficient.

Best,
Wangda


On Mon, Apr 2, 2018 at 2:39 PM, Wangda Tan 
wrote:
Hi Arpit,

Thanks for pointing out this.

I just removed all .md5 files from artifacts. I found md5 checksums still
exist in .mds files and I didn't remove them from .mds file because it is
generated by create-release script and Apache guidance is "should not"
instead of "must not". Please let me know if you think they need to be
removed as well.

- Wangda



On Mon, Apr 2, 2018 at 1:37 PM, Arpit Agarwal mailto:aagar...@hortonworks.com>> wrote:
Thanks for

Re: Apache Hadoop 2.9.1 Release Plan

2018-04-04 Thread Wei-Chiu Chuang 
Sorry Sammi I was late to this thread.
Please considering incorporating HDFS-11915. Sync rbw dir on the first
hsync() to avoid file lost on power failure.
I thought it was already in 2.9.1 but turns out it didn't land. The cherry
pick to branch-2.9 is conflict free.

On Mon, Apr 2, 2018 at 4:34 AM, Chen, Sammi  wrote:

> Hi All,
>
> Today I have created branch-2.9.1 from branch-2.9 and started creating the
> RC0  based on branch-2.9.1.   But due to the corporate network conditions
> and my not full privileges on Hadoop,   it will take a while for RC0 to
> come out.
>
> If you have anything want to commit to branch-2.9.1,  please let me know.
>
> Also I will update fix version of all  2.9.1 JIRAs and moved all
> unresolved JIRA with target version = 2.9.1 to 2.9.2.
>
>
>
> Bests,
> Sammi Chen
>
> -Original Message-
> From: Chen, Sammi [mailto:sammi.c...@intel.com]
> Sent: Friday, March 30, 2018 3:55 PM
> To: hdfs-dev ; mapreduce-...@hadoop.apache.org;
> common-...@hadoop.apache.org; yarn-dev@hadoop.apache.org
> Subject: Apache Hadoop 2.9.1 Release Plan
>
> Hi All,
>
> We have 47 changes on 2.9 branch since last release on Nov. 2017.   There
> are 7 blockers, 5 critical issues and rest are normal bug fixes and feature
> improvements.
>
>
>
>
>
> Here are current tasks targeting for 2.9.1.  No critical and blockers so
> far.
>
> https://issues.apache.org/jira/issues/?jql=%22Target+
> Version%2Fs%22+%3D+2.9.1+AND+%28project+%3D+hadoop+OR+
> project+%3D+%22Hadoop+HDFS%22+OR+project+%3D+%22Hadoop+YARN%
> 22+OR+project+%3D+%22Hadoop+Map%2FReduce%22+OR+project+%
> 3D+%22Hadoop+Common%22%29+AND+status+%21%3D+resolved+ORDER+
> BY+priority+DESC
>
>
> I plan to cut the 2.9.1 branch today, and try to deliver the RC0  ASAP.
>  Please let me know if you have any objections or suggestions.
>
>
>
>
>
>
> Bests,
>
> Sammi
>
>
>
>
> -
> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
>
>


-- 
A very happy Clouderan

[jira] [Created] (YARN-8117) Fix TestRMWebServicesNodes test failure

2018-04-04 Thread Bibin A Chundatt (JIRA) 
Bibin A Chundatt created YARN-8117:
--

 Summary: Fix TestRMWebServicesNodes test failure
 Key: YARN-8117
 URL: https://issues.apache.org/jira/browse/YARN-8117
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Bibin A Chundatt
Assignee: Bibin A Chundatt


Need to fix conflict in TestRMWebServices  due to YARN-7792 and YARN-8092 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Created] (YARN-8116) Nodemanager fails with NumberFormatException: For input string: ""

2018-04-04 Thread Yesha Vora (JIRA) 
Yesha Vora created YARN-8116:


 Summary: Nodemanager fails with NumberFormatException: For input 
string: ""
 Key: YARN-8116
 URL: https://issues.apache.org/jira/browse/YARN-8116
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Yesha Vora


Steps followed.
1) Update nodemanager debug delay config
{code}

  yarn.nodemanager.delete.debug-delay-sec
  350
{code}
2) Launch distributed shell application multiple times
{code}
/usr/hdp/current/hadoop-yarn-client/bin/yarn  jar 
hadoop-yarn-applications-distributedshell-*.jar  -shell_command "sleep 120" 
-num_containers 1 -shell_env YARN_CONTAINER_RUNTIME_TYPE=docker -shell_env 
YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=centos/httpd-24-centos7:latest -shell_env 
YARN_CONTAINER_RUNTIME_DOCKER_DELAYED_REMOVAL=true -jar 
hadoop-yarn-applications-distributedshell-*.jar{code}
3) restart NM

Nodemanager fails to start with below error.
{code}

{code:title=NM log}
2018-03-23 21:32:14,437 INFO  monitor.ContainersMonitorImpl 
(ContainersMonitorImpl.java:serviceInit(181)) - ContainersMonitor enabled: true
2018-03-23 21:32:14,439 INFO  logaggregation.LogAggregationService 
(LogAggregationService.java:serviceInit(130)) - rollingMonitorInterval is set 
as 3600. The logs will be aggregated every 3600 seconds
2018-03-23 21:32:14,455 INFO  service.AbstractService 
(AbstractService.java:noteFailure(267)) - Service 
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl 
failed in state INITED
java.lang.NumberFormatException: For input string: ""
at 
java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
at java.lang.Long.parseLong(Long.java:601)
at java.lang.Long.parseLong(Long.java:631)
at 
org.apache.hadoop.yarn.server.nodemanager.recovery.NMLeveldbStateStoreService.loadContainerState(NMLeveldbStateStoreService.java:350)
at 
org.apache.hadoop.yarn.server.nodemanager.recovery.NMLeveldbStateStoreService.loadContainersState(NMLeveldbStateStoreService.java:253)
at 
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.recover(ContainerManagerImpl.java:365)
at 
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceInit(ContainerManagerImpl.java:316)
at 
org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
at 
org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108)
at 
org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:464)
at 
org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
at 
org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:899)
at 
org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:960)
2018-03-23 21:32:14,458 INFO  logaggregation.LogAggregationService 
(LogAggregationService.java:serviceStop(148)) - 
org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService
 waiting for pending aggregation during exit
2018-03-23 21:32:14,460 INFO  service.AbstractService 
(AbstractService.java:noteFailure(267)) - Service NodeManager failed in state 
INITED
java.lang.NumberFormatException: For input string: ""
at 
java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
at java.lang.Long.parseLong(Long.java:601)
at java.lang.Long.parseLong(Long.java:631)
at 
org.apache.hadoop.yarn.server.nodemanager.recovery.NMLeveldbStateStoreService.loadContainerState(NMLeveldbStateStoreService.java:350)
at 
org.apache.hadoop.yarn.server.nodemanager.recovery.NMLeveldbStateStoreService.loadContainersState(NMLeveldbStateStoreService.java:253)
at 
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.recover(ContainerManagerImpl.java:365)
at 
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceInit(ContainerManagerImpl.java:316)
at 
org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
at 
org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108)
at 
org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:464)
at 
org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
at 
org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:899)
at 
org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:960)
2018-03-23 21:32:14,463 INFO  impl.MetricsSystemImpl 
(MetricsSystemImpl.java:stop(210)) - Stopping NodeManager metrics system...
2018-03-23 21:32:14,464 INFO  impl.MetricsSinkAdapter 
(MetricsSinkAdapter.java:publishMetricsFromQueue(141)) - timeline thread

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

2018-04-04 Thread Apache Jenkins Server 
For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/741/

[Apr 3, 2018 5:48:26 AM] (xiao) HADOOP-15317. Improve NetworkTopology 
chooseRandom's loop.
[Apr 3, 2018 6:10:08 AM] (xiao) HADOOP-15355. TestCommonConfigurationFields is 
broken by HADOOP-15312.
[Apr 3, 2018 7:08:40 AM] (yqlin) HDFS-13364. RBF: Support NamenodeProtocol in 
the Router. Contributed by
[Apr 3, 2018 2:59:20 PM] (haibochen) YARN-8051. 
TestRMEmbeddedElector#testCallbackSynchronization is flaky.
[Apr 3, 2018 3:31:34 PM] (stevel) HADOOP-14758. S3GuardTool.prune to handle 
UnsupportedOperationException.
[Apr 3, 2018 5:01:00 PM] (szegedim) YARN-8035. Uncaught exception in 
ContainersMonitorImpl during relaunch




-1 overall


The following subsystems voted -1:
findbugs unit xml


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

FindBugs :

   module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api 
   org.apache.hadoop.yarn.api.records.Resource.getResources() may expose 
internal representation by returning Resource.resources At Resource.java:by 
returning Resource.resources At Resource.java:[line 234] 

Failed junit tests :

   hadoop.hdfs.server.datanode.TestDataNodeUUID 
   hadoop.hdfs.TestDFSClientRetries 
   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure 
   hadoop.hdfs.server.namenode.TestReencryptionWithKMS 
   hadoop.hdfs.TestDFSStripedOutputStreamWithFailureWithRandomECPolicy 
   hadoop.hdfs.web.TestWebHdfsTimeouts 
  

   cc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/741/artifact/out/diff-compile-cc-root.txt
  [4.0K]

   javac:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/741/artifact/out/diff-compile-javac-root.txt
  [288K]

   checkstyle:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/741/artifact/out/diff-checkstyle-root.txt
  [17M]

   pylint:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/741/artifact/out/diff-patch-pylint.txt
  [24K]

   shellcheck:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/741/artifact/out/diff-patch-shellcheck.txt
  [20K]

   shelldocs:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/741/artifact/out/diff-patch-shelldocs.txt
  [12K]

   whitespace:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/741/artifact/out/whitespace-eol.txt
  [9.2M]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/741/artifact/out/whitespace-tabs.txt
  [1.1M]

   xml:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/741/artifact/out/xml.txt
  [4.0K]

   findbugs:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/741/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api-warnings.html
  [8.0K]

   javadoc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/741/artifact/out/diff-javadoc-javadoc-root.txt
  [760K]

   unit:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/741/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
  [424K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/741/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
  [76K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/741/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
  [84K]

Powered by Apache Yetus 0.8.0-SNAPSHOT   http://yetus.apache.org

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Created] (YARN-8115) [UI2] URL data like nodeHTTPAddress must be encoded in UI before using for NM/ATS

2018-04-04 Thread Sunil G (JIRA) 
Sunil G created YARN-8115:
-

 Summary: [UI2] URL data like nodeHTTPAddress must be encoded in UI 
before using for NM/ATS
 Key: YARN-8115
 URL: https://issues.apache.org/jira/browse/YARN-8115
 Project: Hadoop YARN
  Issue Type: Bug
  Components: yarn-ui-v2
Reporter: Sunil G
Assignee: Sreenath Somarajapuram


nodeHTTPAddress is picked from RM ,but it should be encoded before processing 
it from UI



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Resolved] (YARN-2466) Umbrella issue for Yarn launched Docker Containers

2018-04-04 Thread Shane Kumpf (JIRA) 

 [ 
https://issues.apache.org/jira/browse/YARN-2466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shane Kumpf resolved YARN-2466.
---
Resolution: Won't Fix

> Umbrella issue for Yarn launched Docker Containers
> --
>
> Key: YARN-2466
> URL: https://issues.apache.org/jira/browse/YARN-2466
> Project: Hadoop YARN
>  Issue Type: New Feature
>Affects Versions: 2.4.1
>Reporter: Abin Shahab
>Priority: Major
>
> Docker (https://www.docker.io/) is, increasingly, a very popular container 
> technology.
> In context of YARN, the support for Docker will provide a very elegant 
> solution to allow applications to package their software into a Docker 
> container (entire Linux file system incl. custom versions of perl, python 
> etc.) and use it as a blueprint to launch all their YARN containers with 
> requisite software environment. This provides both consistency (all YARN 
> containers will have the same software environment) and isolation (no 
> interference with whatever is installed on the physical machine).
> In addition to software isolation mentioned above, Docker containers will 
> provide resource, network, and user-namespace isolation. 
> Docker provides resource isolation through cgroups, similar to 
> LinuxContainerExecutor. This prevents one job from taking other jobs 
> resource(memory and CPU) on the same hadoop cluster. 
> User-namespace isolation will ensure that the root on the container is mapped 
> an unprivileged user on the host. This is currently being added to Docker.
> Network isolation will ensure that one user’s network traffic is completely 
> isolated from another user’s network traffic. 
> Last but not the least, the interaction of Docker and Kerberos will have to 
> be worked out. These Docker containers must work in a secure hadoop 
> environment.
> Additional details are here: 
> https://wiki.apache.org/hadoop/dineshs/IsolatingYarnAppsInDockerContainers



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Resolved] (YARN-3290) DockerContainerExecutor should optionally limit memory and cpu

2018-04-04 Thread Shane Kumpf (JIRA) 

 [ 
https://issues.apache.org/jira/browse/YARN-3290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shane Kumpf resolved YARN-3290.
---
Resolution: Won't Fix

> DockerContainerExecutor should optionally limit memory and cpu
> --
>
> Key: YARN-3290
> URL: https://issues.apache.org/jira/browse/YARN-3290
> Project: Hadoop YARN
>  Issue Type: Improvement
>Reporter: Abin Shahab
>Priority: Major
>
> Currently, DockerContainerExecutor does not set cgroup limits on memory and 
> cpu. It should follow LCE's example to set cgroup limits.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Resolved] (YARN-3095) Enable DockerContainerExecutor to update Docker image

2018-04-04 Thread Shane Kumpf (JIRA) 

 [ 
https://issues.apache.org/jira/browse/YARN-3095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shane Kumpf resolved YARN-3095.
---
Resolution: Won't Fix

> Enable DockerContainerExecutor to update Docker image
> -
>
> Key: YARN-3095
> URL: https://issues.apache.org/jira/browse/YARN-3095
> Project: Hadoop YARN
>  Issue Type: New Feature
>Affects Versions: 2.6.0
>Reporter: Chen He
>Assignee: Chen He
>Priority: Major
>  Labels: security
>
> This JIRA allows DCE to check and update docker image before running a 
> container. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Resolved] (YARN-3289) Docker images should be downloaded during localization

2018-04-04 Thread Shane Kumpf (JIRA) 

 [ 
https://issues.apache.org/jira/browse/YARN-3289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shane Kumpf resolved YARN-3289.
---
Resolution: Duplicate

> Docker images should be downloaded during localization
> --
>
> Key: YARN-3289
> URL: https://issues.apache.org/jira/browse/YARN-3289
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Ravi Prakash
>Priority: Major
>
> We currently call docker run on images while launching containers. If the 
> image size if sufficiently big, the task will timeout. We should download the 
> image we want to run during localization (if possible) to prevent this



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Created] (YARN-8114) oozie using router submit job error

2018-04-04 Thread Yiran Wu (JIRA) 
Yiran Wu created YARN-8114:
--

 Summary: oozie using router submit job error
 Key: YARN-8114
 URL: https://issues.apache.org/jira/browse/YARN-8114
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Yiran Wu



{code:java}
 call org.apache.hadoop.yarn.api.ApplicationClientProtocolPB.getApplications 
from 172.21.14.106:14410 Call#13 Retry#0
org.apache.commons.lang.NotImplementedException: Code is not implemented
at 
org.apache.hadoop.yarn.server.router.clientrm.FederationClientInterceptor.getApplications(JDFederationClientInterceptor.java:643)
at 
org.apache.hadoop.yarn.server.router.clientrm.RouterClientRMService.getApplications(RouterClientRMService.java:354)
at 
org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.getApplications(ApplicationClientProtocolPBServiceImpl.java:234)
at 
org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:425)
at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2090)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2086)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1803)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2086)

{code}




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org