Re: [VOTE] Release Apache Hadoop 2.9.1 (RC0)

[Rakesh Radhakrishnan]

Thanks Sammi for getting this out!

+1 (binding)

Verified the following and looks fine to me.

 * Built from source.
 * Deployed 3 node cluster with NameNode HA.
 * Verified HDFS web UIs.
 * Tried out HDFS shell commands.
 * Ran Mover, Balancer tools.
 * Ran sample MapReduce jobs.


Rakesh

On Thu, Apr 19, 2018 at 2:27 PM, Chen, Sammi  wrote:

> Hi all,
>
> This is the first dot release of Apache Hadoop 2.9 line since 2.9.0 was
> released on November 17, 2017.
>
> It includes 208 changes. Among them, 9 blockers, 15 critical issues and
> rest are normal bug fixes and feature improvements.
>
> Thanks to the many who contributed to the 2.9.1 development.
>
> The artifacts are available here:  https://dist.apache.org/repos/
> dist/dev/hadoop/2.9.1-RC0/
>
> The RC tag in git is release-2.9.1-RC0. Last git commit SHA is
> e30710aea4e6e55e69372929106cf119af06fd0e.
>
> The maven artifacts are available at:
> https://repository.apache.org/content/repositories/orgapachehadoop-1115/
>
> My public key is available from:
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
> Please try the release and vote; the vote will run for the usual 5 days,
> ending on 4/25/2018 PST time.
>
> Also I would like thank Lei(Eddy) Xu and Chris Douglas for your help
> during the RC preparation.
>
> Bests,
> Sammi Chen
>

Apache Hadoop qbt Report: trunk+JDK8 on Windows/x64

[Apache Jenkins Server]

For more details, see https://builds.apache.org/job/hadoop-trunk-win/450/

[Apr 25, 2018 10:50:52 PM] (omalley) HDFS-8456. Introduce 
STORAGE_CONTAINER_SERVICE as a new NodeType.
[Apr 25, 2018 10:51:00 PM] (omalley) HDFS-8614. OzoneHandler : Add Quota 
Support. (Contributed by Anu
[Apr 25, 2018 10:51:00 PM] (omalley) HDFS-8448. Create REST Interface for 
Volumes. Contributed by Anu
[Apr 25, 2018 10:51:00 PM] (omalley) HDFS-8637. OzoneHandler : Add Error Table. 
(Contributed by Anu Engineer)
[Apr 25, 2018 10:51:00 PM] (omalley) HDFS-8634. OzoneHandler: Add userAuth 
Interface and Simple userAuth
[Apr 25, 2018 10:51:00 PM] (omalley) HDFS-8644. OzoneHandler : Add volume 
handler. (Contributed by Anu
[Apr 25, 2018 10:51:00 PM] (omalley) HDFS-8654. OzoneHandler : Add ACL support. 
(Contributed by Anu Engineer)
[Apr 25, 2018 10:51:00 PM] (omalley) HDFS-8680. OzoneHandler : Add Local 
StorageHandler support for volumes.
[Apr 25, 2018 10:51:00 PM] (omalley) HDFS-8717. OzoneHandler : Add common 
bucket objects. (Contributed by Anu
[Apr 25, 2018 10:51:00 PM] (omalley) HDFS-8753. Ozone: Unify 
StorageContainerConfiguration with
[Apr 25, 2018 10:51:00 PM] (omalley) HDFS-8695. OzoneHandler : Add Bucket REST 
Interface. (aengineer)
[Apr 25, 2018 10:51:36 PM] (omalley) HDFS-8527. OzoneHandler: Integration of 
REST interface and container
[Apr 25, 2018 10:51:39 PM] (omalley) HDFS-8757 : OzoneHandler : Add 
localStorageHandler support for Buckets.
[Apr 25, 2018 10:51:39 PM] (omalley) HDFS-9834. OzoneHandler : Enable 
MiniDFSCluster based testing for Ozone.
[Apr 25, 2018 10:51:39 PM] (omalley) HDFS-9845. OzoneHandler : Support List and 
Info Volumes. Contributed by
[Apr 25, 2018 10:51:51 PM] (omalley) HDFS-9853. Ozone: Add container 
definitions. Contributed by Anu
[Apr 25, 2018 10:51:53 PM] (omalley) HDFS-9848. Ozone: Add Ozone Client lib for 
volume handling. Contributed
[Apr 25, 2018 10:51:53 PM] (omalley) HDFS-9907. Exclude Ozone 
protobuf-generated classes from Findbugs
[Apr 25, 2018 10:51:53 PM] (omalley) HDFS-9873. Ozone: Add container transport 
server. Contributed by Anu
[Apr 25, 2018 10:51:58 PM] (omalley) HDFS-9891. Ozone: Add container transport 
client. Contributed by Anu
[Apr 25, 2018 10:52:00 PM] (omalley) HDFS-9920. Stop tracking CHANGES.txt in 
the HDFS-7240 feature branch.
[Apr 25, 2018 10:52:00 PM] (omalley) HDFS-9916. OzoneHandler : Add Key handler. 
Contributed by Anu Engineer.
[Apr 25, 2018 10:52:00 PM] (omalley) HDFS-9916. OzoneHandler : Add Key handler. 
Contributed by Anu Engineer.
[Apr 25, 2018 10:52:00 PM] (omalley) HDFS-9925. Ozone: Add Ozone Client lib for 
bucket handling. Contributed
[Apr 25, 2018 10:52:00 PM] (omalley) HDFS-9926. ozone : Add volume commands to 
CLI. Contributed by Anu
[Apr 25, 2018 10:52:00 PM] (omalley) HDFS-9944. Ozone : Add container 
dispatcher. Contributed by Anu
[Apr 25, 2018 10:52:00 PM] (omalley) HDFS-9961. Ozone: Add buckets commands to 
CLI. Contributed by Anu
[Apr 25, 2018 10:52:00 PM] (omalley) HDFS-10180. Ozone: Refactor container 
Namespace. Contributed by Anu
[Apr 25, 2018 10:52:00 PM] (omalley) HDFS-9960. OzoneHandler : Add localstorage 
support for keys. Contributed
[Apr 25, 2018 10:52:07 PM] (omalley) HDFS-10179. Ozone: Adding logging support. 
Contributed by Anu Engineer.
[Apr 25, 2018 10:52:09 PM] (omalley) HDFS-10196. Ozone : Enable better error 
reporting for failed commands in
[Apr 25, 2018 10:52:25 PM] (omalley) HDFS-10202. ozone : Add key commands to 
CLI. Contributed by Anu
[Apr 25, 2018 10:52:33 PM] (omalley) HDFS-10195. Ozone: Add container 
persistence. Contributed by Anu
[Apr 25, 2018 10:52:46 PM] (omalley) HDFS-8210. Ozone: Implement storage 
container manager. Contributed by
[Apr 25, 2018 10:52:48 PM] (omalley) HDFS-10238. Ozone : Add chunk persistance. 
Contributed by Anu Engineer.
[Apr 25, 2018 10:52:48 PM] (omalley) HDFS-10250. Ozone: Add key Persistence. 
Contributed by Anu Engineer.
[Apr 25, 2018 10:52:56 PM] (omalley) HDFS-10268. Ozone: end-to-end integration 
for create/get volumes,
[Apr 25, 2018 10:52:58 PM] (omalley) HDFS-10251. Ozone: Shutdown cleanly. 
Contributed by Anu Engineer
[Apr 25, 2018 10:52:58 PM] (omalley) HDFS-10278. Ozone: Add paging support to 
list Volumes. Contributed by
[Apr 25, 2018 10:53:05 PM] (omalley) HDFS-10232. Ozone: Make config key naming 
consistent. Contributed by Anu
[Apr 25, 2018 10:53:07 PM] (omalley) HDFS-10349. Ozone: StorageContainerManager 
fails to compile after merge
[Apr 25, 2018 10:53:07 PM] (omalley) HDFS-10351. Ozone: Optimize key writes to 
chunks by providing a bulk
[Apr 25, 2018 10:53:12 PM] (omalley) HDFS-10361. Support starting 
StorageContainerManager as a daemon.
[Apr 25, 2018 10:53:19 PM] (omalley) HDFS-10363. Ozone: Introduce new config 
keys for SCM service endpoints.
[Apr 25, 2018 10:53:21 PM] (omalley) HDFS-10420. Fix Ozone unit tests to use 
MiniOzoneCluster. Contributed by
[Apr 25, 2018 10:53:21 PM] (omalley) HDFS-10897. Ozone: SCM: Add NodeManager. 
Contributed by Anu Engineer.

[jira] [Created] (YARN-8231) Dshell application fails when one of the docker container gets killed

[Yesha Vora (JIRA)]

Yesha Vora created YARN-8231:


 Summary: Dshell application fails when one of the docker container 
gets killed
 Key: YARN-8231
 URL: https://issues.apache.org/jira/browse/YARN-8231
 Project: Hadoop YARN
  Issue Type: Bug
  Components: yarn-native-services
Reporter: Yesha Vora


1) Launch dshell application
{code}
yarn  jar hadoop-yarn-applications-distributedshell-*.jar  -shell_command 
"sleep 300" -num_containers 2 -shell_env YARN_CONTAINER_RUNTIME_TYPE=docker 
-shell_env YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=centos/httpd-24-centos7:latest 
-keep_containers_across_application_attempts -jar 
/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-applications-distributedshell-*.jar{code}
2) Kill container_1524681858728_0012_01_02

Expected behavior:
Application should start new instance and finish successfully

Actual behavior:
Application Failed as soon as container was killed

{code:title=AM log}
18/04/27 23:05:12 INFO distributedshell.ApplicationMaster: Got response from RM 
for container ask, completedCnt=1
18/04/27 23:05:12 INFO distributedshell.ApplicationMaster: 
appattempt_1524681858728_0012_01 got container status for 
containerID=container_1524681858728_0012_01_02, state=COMPLETE, 
exitStatus=137, diagnostics=[2018-04-27 23:05:09.310]Container killed on 
request. Exit code is 137
[2018-04-27 23:05:09.331]Container exited with a non-zero exit code 137. 
[2018-04-27 23:05:09.332]Killed by external signal

18/04/27 23:08:46 INFO distributedshell.ApplicationMaster: Got response from RM 
for container ask, completedCnt=1
18/04/27 23:08:46 INFO distributedshell.ApplicationMaster: 
appattempt_1524681858728_0012_01 got container status for 
containerID=container_1524681858728_0012_01_03, state=COMPLETE, 
exitStatus=0, diagnostics=
18/04/27 23:08:46 INFO distributedshell.ApplicationMaster: Container completed 
successfully., containerId=container_1524681858728_0012_01_03
18/04/27 23:08:46 INFO distributedshell.ApplicationMaster: Application 
completed. Stopping running containers
18/04/27 23:08:46 INFO distributedshell.ApplicationMaster: Application 
completed. Signalling finish to RM
18/04/27 23:08:46 INFO distributedshell.ApplicationMaster: Diagnostics., 
total=2, completed=2, allocated=2, failed=1
18/04/27 23:08:46 INFO impl.AMRMClientImpl: Waiting for application to be 
successfully unregistered.{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Created] (YARN-8229) exp

[Sumana Sathish (JIRA)]

Sumana Sathish created YARN-8229:


 Summary: exp
 Key: YARN-8229
 URL: https://issues.apache.org/jira/browse/YARN-8229
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Sumana Sathish






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Resolved] (YARN-8227) TestPlacementConstraintTransformations is failing in trunk

[Shane Kumpf (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-8227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shane Kumpf resolved YARN-8227.
---
Resolution: Duplicate

This is being tracked under YARN-8225, going to close this one as a duplicate. 
Thanks.

> TestPlacementConstraintTransformations is failing in trunk
> --
>
> Key: YARN-8227
> URL: https://issues.apache.org/jira/browse/YARN-8227
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Botong Huang
>Priority: Major
>
> [ERROR] Tests run: 4, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.1 s 
> <<< FAILURE! - in 
> org.apache.hadoop.yarn.api.resource.TestPlacementConstraintTransformations
> [ERROR] 
> testCardinalityConstraint(org.apache.hadoop.yarn.api.resource.TestPlacementConstraintTransformations)
>   Time elapsed: 0.007 s  <<< FAILURE!
> java.lang.AssertionError: expected: java.util.HashSet<[hb]> but was: 
> java.util.HashSet<[hb]>
>   at org.junit.Assert.fail(Assert.java:88)
>   at org.junit.Assert.failNotEquals(Assert.java:743)
>   at org.junit.Assert.assertEquals(Assert.java:118)
>   at org.junit.Assert.assertEquals(Assert.java:144)
>   at 
> org.apache.hadoop.yarn.api.resource.TestPlacementConstraintTransformations.testCardinalityConstraint(TestPlacementConstraintTransformations.java:116)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Created] (YARN-8227) TestPlacementConstraintTransformations is failing in trunk

[Botong Huang (JIRA)]

Botong Huang created YARN-8227:
--

 Summary: TestPlacementConstraintTransformations is failing in trunk
 Key: YARN-8227
 URL: https://issues.apache.org/jira/browse/YARN-8227
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Botong Huang


[ERROR] Tests run: 4, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.1 s 
<<< FAILURE! - in 
org.apache.hadoop.yarn.api.resource.TestPlacementConstraintTransformations
[ERROR] 
testCardinalityConstraint(org.apache.hadoop.yarn.api.resource.TestPlacementConstraintTransformations)
  Time elapsed: 0.007 s  <<< FAILURE!
java.lang.AssertionError: expected: java.util.HashSet<[hb]> but was: 
java.util.HashSet<[hb]>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:144)
at 
org.apache.hadoop.yarn.api.resource.TestPlacementConstraintTransformations.testCardinalityConstraint(TestPlacementConstraintTransformations.java:116)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 2.9.1 (RC0)

[Eric Badger]

+1 (non-binding)

- Verified all hashes and checksums
- Built from source on macOS 10.13.4, Java 1.8.0u65
- Deployed a pseudo cluster
- Ran some example jobs

Eric

On Thu, Apr 26, 2018 at 11:16 PM, Takanobu Asanuma 
wrote:

> Thanks for working on this, Sammi!
>
> +1 (non-binding)
>- Verified checksums
>- Succeeded "mvn clean package -Pdist,native -Dtar -DskipTests"
>- Started hadoop cluster with 1 master and 5 slaves
>- Run TeraGen/TeraSort
>- Verified some hdfs operations
>- Verified Web UI (NameNode, ResourceManager(classic and V2),
> JobHistory, Timeline)
>
> Thanks,
> -Takanobu
>
> > -Original Message-
> > From: Jinhu Wu [mailto:jinhu.wu@gmail.com]
> > Sent: Friday, April 27, 2018 12:39 PM
> > To: Gabor Bota 
> > Cc: Chen, Sammi ; junping...@apache.org; Hadoop
> > Common ; Rushabh Shah ;
> > hdfs-dev ; mapreduce-...@hadoop.apache.org;
> > yarn-dev@hadoop.apache.org
> > Subject: Re: [VOTE] Release Apache Hadoop 2.9.1 (RC0)
> >
> > Thanks Sammi for driving the release work!
> >
> > +1 (non-binding)
> >
> > based on following verification work:
> > - built succeed from source on Mac OSX 10.13.4, java version "1.8.0_151"
> > - run hadoop-aliyun tests successfully on cn-shanghai endpoint
> > - deployed a one node cluster and verified PI job
> > - verfied word-count job by using hadoop-aliyun as storage.
> >
> > Thanks,
> > jinhu
> >
> > On Fri, Apr 27, 2018 at 12:45 AM, Gabor Bota 
> > wrote:
> >
> > >   Thanks for the work Sammi!
> > >
> > >   +1 (non-binding)
> > >
> > >-   checked out git tag release-2.9.1-RC0
> > >-   S3A unit (mvn test) and integration (mvn verify) test run were
> > >successful on us-west-2
> > >-   built from source on Mac OS X 10.13.4, openjdk 1.8.0_144 (zulu)
> > >-   deployed on a 3 node cluster
> > >-   verified pi job, teragen, terasort and teravalidate
> > >
> > >
> > >   Regards,
> > >   Gabor Bota
> > >
> > > On Wed, Apr 25, 2018 at 7:12 AM, Chen, Sammi 
> wrote:
> > >
> > > >
> > > > Paste the links here,
> > > >
> > > > The artifacts are available here:  https://dist.apache.org/repos/
> > > > dist/dev/hadoop/2.9.1-RC0/
> > > >
> > > > The RC tag in git is release-2.9.1-RC0. Last git commit SHA is
> > > > e30710aea4e6e55e69372929106cf119af06fd0e.
> > > >
> > > > The maven artifacts are available at:
> > > >
> > https://repository.apache.org/content/repositories/orgapachehadoop-1
> > > > 115/
> > > >
> > > > My public key is available from:
> > > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> > > >
> > > >
> > > > Bests,
> > > > Sammi
> > > > -Original Message-
> > > > From: Chen, Sammi [mailto:sammi.c...@intel.com]
> > > > Sent: Wednesday, April 25, 2018 12:02 PM
> > > > To: junping...@apache.org
> > > > Cc: Hadoop Common ; Rushabh Shah <
> > > > rusha...@oath.com>; hdfs-dev ;
> > > > mapreduce-...@hadoop.apache.org; yarn-dev@hadoop.apache.org
> > > > Subject: RE: [VOTE] Release Apache Hadoop 2.9.1 (RC0)
> > > >
> > > >
> > > > Thanks Jason Lowe for the quick investigation to find out that the
> > > > test failures belong to the test only.
> > > >
> > > > Based on the current facts, I would like to continue calling the
> > > > VOTE for
> > > > 2.9.1 RC0,  and extend the vote deadline to end of this week 4/27.
> > > >
> > > >
> > > > I will add following note to the final release notes,
> > > >
> > > > HADOOP-15385
> > Test
> > > > case failures in Hadoop-distcp project doesn’t impact the distcp
> > > > function in 2.9.1
> > > >
> > > >
> > > > Bests,
> > > > Sammi
> > > > From: 俊平堵 [mailto:junping...@apache.org]
> > > > Sent: Tuesday, April 24, 2018 11:50 PM
> > > > To: Chen, Sammi 
> > > > Cc: Hadoop Common ; Rushabh Shah <
> > > > rusha...@oath.com>; hdfs-dev ;
> > > > mapreduce-...@hadoop.apache.org; yarn-dev@hadoop.apache.org
> > > > Subject: Re: [VOTE] Release Apache Hadoop 2.9.1 (RC0)
> > > >
> > > > Thanks for reporting the issue, Rushabh! Actually, we found that
> > > > these test failures belong to test issues but not production issue,
> > > > so not
> > > really
> > > > a solid blocker for release. Anyway, I will let RM of 2.9.1 to
> > > > decide if
> > > to
> > > > cancel RC or not for this test issue.
> > > >
> > > > Thanks,
> > > >
> > > > Junping
> > > >
> > > >
> > > > Chen, Sammi >
> > > 于2018年4月24日
> > > > 周二下午7:50写道：
> > > > Hi Rushabh,
> > > >
> > > > Thanks for reporting the issue.  I will upload a new RC candidate
> > > > soon after the test failing issue is resolved.
> > > >
> > > >
> > > > Bests,
> > > > Sammi Chen
> > > > From: Rushabh 

Re: [NOTIFICATION] Hadoop trunk rebased

[Allen Wittenauer]

Did the patch that fixes the mountain of maven warnings get missed?

> On Apr 26, 2018, at 11:52 PM, Akira Ajisaka  wrote:
> 
> + common-dev and mapreduce-dev
> 
> On 2018/04/27 6:23, Owen O'Malley wrote:
>> As we discussed in hdfs-dev@hadoop, I did a force push to Hadoop's trunk to
>> replace the Ozone merge with a rebase.
>> That means that you'll need to rebase your branches.
>> .. Owen
> 
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
> 


-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Created] (YARN-8226) Improve anti-affinity section description in YARN Service API doc

[Charan Hebri (JIRA)]

Charan Hebri created YARN-8226:
--

 Summary: Improve anti-affinity section description in YARN Service 
API doc
 Key: YARN-8226
 URL: https://issues.apache.org/jira/browse/YARN-8226
 Project: Hadoop YARN
  Issue Type: Improvement
  Components: docs, documentation
Reporter: Charan Hebri


The anti-affinity section in the YARN services doc says,
{noformat}
Note, that the 3 containers will come up on 3 different nodes. If there are 
less than 3 NMs running in the cluster, then all 3 container requests will not 
be fulfilled and the service will be in non-STABLE state.{noformat}
Based on the description the expected behavior for the case of number of NMs 
less than the number of containers requested isn't very obvious. Opening issue 
to improve the comment.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

[Apache Jenkins Server]

For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/764/

No changes




-1 overall


The following subsystems voted -1:
asflicense findbugs unit xml


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

FindBugs :

   module:hadoop-hdds/common 
   Exceptional return value of java.io.File.mkdirs() ignored in 
org.apache.hadoop.utils.LevelDBStore.openDB(File, Options) At 
LevelDBStore.java:ignored in org.apache.hadoop.utils.LevelDBStore.openDB(File, 
Options) At LevelDBStore.java:[line 79] 

Failed junit tests :

   hadoop.hdfs.server.namenode.TestReencryptionWithKMS 
   hadoop.hdfs.TestDFSStripedOutputStreamWithFailureWithRandomECPolicy 
   hadoop.hdfs.web.TestWebHdfsTimeouts 
   
hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery 
   hadoop.yarn.api.resource.TestPlacementConstraintTransformations 
   hadoop.yarn.applications.distributedshell.TestDistributedShell 
  

   cc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/764/artifact/out/diff-compile-cc-root.txt
  [4.0K]

   javac:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/764/artifact/out/diff-compile-javac-root.txt
  [288K]

   checkstyle:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/764/artifact/out/diff-checkstyle-root.txt
  [17M]

   pylint:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/764/artifact/out/diff-patch-pylint.txt
  [24K]

   shellcheck:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/764/artifact/out/diff-patch-shellcheck.txt
  [20K]

   shelldocs:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/764/artifact/out/diff-patch-shelldocs.txt
  [12K]

   whitespace:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/764/artifact/out/whitespace-eol.txt
  [9.4M]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/764/artifact/out/whitespace-tabs.txt
  [1.1M]

   xml:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/764/artifact/out/xml.txt
  [4.0K]

   findbugs:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/764/artifact/out/branch-findbugs-hadoop-hdds_client.txt
  [4.0K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/764/artifact/out/branch-findbugs-hadoop-hdds_common-warnings.html
  [8.0K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/764/artifact/out/branch-findbugs-hadoop-hdds_container-service.txt
  [36K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/764/artifact/out/branch-findbugs-hadoop-hdds_framework.txt
  [4.0K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/764/artifact/out/branch-findbugs-hadoop-hdds_server-scm.txt
  [24K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/764/artifact/out/branch-findbugs-hadoop-hdds_tools.txt
  [20K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/764/artifact/out/branch-findbugs-hadoop-ozone_client.txt
  [8.0K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/764/artifact/out/branch-findbugs-hadoop-ozone_common.txt
  [4.0K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/764/artifact/out/branch-findbugs-hadoop-ozone_objectstore-service.txt
  [8.0K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/764/artifact/out/branch-findbugs-hadoop-ozone_ozone-manager.txt
  [8.0K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/764/artifact/out/branch-findbugs-hadoop-ozone_tools.txt
  [8.0K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/764/artifact/out/branch-findbugs-hadoop-tools_hadoop-ozone.txt
  [12K]

   javadoc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/764/artifact/out/diff-javadoc-javadoc-root.txt
  [760K]

   unit:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/764/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
  [500K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/764/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt
  [40K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/764/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications_hadoop-yarn-applications-distributedshell.txt
  [12K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/764/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
  [84K]
   

[jira] [Created] (YARN-8225) YARN precommit build failing in TestPlacementConstraintTransformations

[Billie Rinaldi (JIRA)]

Billie Rinaldi created YARN-8225:


 Summary: YARN precommit build failing in 
TestPlacementConstraintTransformations
 Key: YARN-8225
 URL: https://issues.apache.org/jira/browse/YARN-8225
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Billie Rinaldi


The HashSet comparison is not working for some reason:
{noformat}
java.lang.AssertionError: expected: java.util.HashSet<[hb]> but was: 
java.util.HashSet<[hb]>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:144)
at 
org.apache.hadoop.yarn.api.resource.TestPlacementConstraintTransformations.testCardinalityConstraint(TestPlacementConstraintTransformations.java:116)
{noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Created] (YARN-8224) LogAggregation status TIME_OUT for absent container misleading

[Prabhu Joseph (JIRA)]

Prabhu Joseph created YARN-8224:
---

 Summary: LogAggregation status TIME_OUT for absent container 
misleading
 Key: YARN-8224
 URL: https://issues.apache.org/jira/browse/YARN-8224
 Project: Hadoop YARN
  Issue Type: Bug
  Components: log-aggregation
Affects Versions: 2.7.3
Reporter: Prabhu Joseph


When a container is not launched on NM and it is absent, RM still tries to get 
the Log Aggregation Status and reports the status as TIME_OUT. (attached 
screenshot)

{code}
2018-04-26 12:47:38,403 WARN  containermanager.ContainerManagerImpl 
(ContainerManagerImpl.java:handle(1070)) - Event EventType: KILL_CONTAINER sent 
to absent container container_e361_1524687599273_2110_01_000770

2018-04-26 12:49:31,743 WARN  containermanager.ContainerManagerImpl 
(ContainerManagerImpl.java:handle(1086)) - Event EventType: FINISH_APPLICATION 
sent to absent application application_1524687599273_2110

{code}






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Resolved] (YARN-6639) Docker container can not find launch_container.sh in workdir

[Shane Kumpf (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-6639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shane Kumpf resolved YARN-6639.
---
Resolution: Cannot Reproduce

> Docker container can not find launch_container.sh in workdir
> 
>
> Key: YARN-6639
> URL: https://issues.apache.org/jira/browse/YARN-6639
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: nodemanager
>Affects Versions: 3.0.0-alpha2, 3.0.0-alpha4
>Reporter: Wei Chen
>Priority: Major
>  Labels: easyfix
>
> I am testing with DockerLinuxContainerRuntime. I found applications failed to 
> start during initializing the docker container. That's caused by the docker 
> can not find launch_contaienr.sh in its workdir.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Created] (YARN-8223) ClassNotFoundException when auxiliary service is loaded from HDFS

[Charan Hebri (JIRA)]

Charan Hebri created YARN-8223:
--

 Summary: ClassNotFoundException when auxiliary service is loaded 
from HDFS
 Key: YARN-8223
 URL: https://issues.apache.org/jira/browse/YARN-8223
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Charan Hebri


Loading an auxiliary jar from a local location on a node manager works as 
expected,
{noformat}
2018-04-26 15:09:26,179 INFO  util.ApplicationClassLoader 
(ApplicationClassLoader.java:(98)) - classpath: 
[file:/grid/0/hadoop/yarn/local/aux-service-local.jar]
2018-04-26 15:09:26,179 INFO  util.ApplicationClassLoader 
(ApplicationClassLoader.java:(99)) - system classes: [java., 
javax.accessibility., javax.activation., javax.activity., javax.annotation., 
javax.annotation.processing., javax.crypto., javax.imageio., javax.jws., 
javax.lang.model., -javax.management.j2ee., javax.management., javax.naming., 
javax.net., javax.print., javax.rmi., javax.script., 
-javax.security.auth.message., javax.security.auth., javax.security.cert., 
javax.security.sasl., javax.sound., javax.sql., javax.swing., javax.tools., 
javax.transaction., -javax.xml.registry., -javax.xml.rpc., javax.xml., 
org.w3c.dom., org.xml.sax., org.apache.commons.logging., org.apache.log4j., 
-org.apache.hadoop.hbase., org.apache.hadoop., core-default.xml, 
hdfs-default.xml, mapred-default.xml, yarn-default.xml]
2018-04-26 15:09:26,181 INFO  containermanager.AuxServices 
(AuxServices.java:serviceInit(252)) - The aux service:test_aux_local are using 
the custom classloader
2018-04-26 15:09:26,182 WARN  containermanager.AuxServices 
(AuxServices.java:serviceInit(268)) - The Auxiliary Service named 
'test_aux_local' in the configuration is for class 
org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxiliaryServiceWithCustomClassLoader
 which has a name of 'org.apache.auxtest.AuxServiceFromLocal with custom class 
loader'. Because these are not the same tools trying to send ServiceData and 
read Service Meta Data may have issues unless the refer to the name in the 
config.
2018-04-26 15:09:26,182 INFO  containermanager.AuxServices 
(AuxServices.java:addService(103)) - Adding auxiliary service 
org.apache.auxtest.AuxServiceFromLocal with custom class loader, 
"test_aux_local"{noformat}
But loading the same jar from a location on HDFS fails with a 
ClassNotFoundException.
{noformat}
018-04-26 15:14:39,683 INFO  util.ApplicationClassLoader 
(ApplicationClassLoader.java:(98)) - classpath: []
2018-04-26 15:14:39,683 INFO  util.ApplicationClassLoader 
(ApplicationClassLoader.java:(99)) - system classes: [java., 
javax.accessibility., javax.activation., javax.activity., javax.annotation., 
javax.annotation.processing., javax.crypto., javax.imageio., javax.jws., 
javax.lang.model., -javax.management.j2ee., javax.management., javax.naming., 
javax.net., javax.print., javax.rmi., javax.script., 
-javax.security.auth.message., javax.security.auth., javax.security.cert., 
javax.security.sasl., javax.sound., javax.sql., javax.swing., javax.tools., 
javax.transaction., -javax.xml.registry., -javax.xml.rpc., javax.xml., 
org.w3c.dom., org.xml.sax., org.apache.commons.logging., org.apache.log4j., 
-org.apache.hadoop.hbase., org.apache.hadoop., core-default.xml, 
hdfs-default.xml, mapred-default.xml, yarn-default.xml]
2018-04-26 15:14:39,687 INFO  service.AbstractService 
(AbstractService.java:noteFailure(267)) - Service 
org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices failed 
in state INITED
java.lang.ClassNotFoundException: org.apache.auxtest.AuxServiceFromLocal
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:338)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at 
org.apache.hadoop.util.ApplicationClassLoader.loadClass(ApplicationClassLoader.java:189)
at 
org.apache.hadoop.util.ApplicationClassLoader.loadClass(ApplicationClassLoader.java:157)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at 
org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxiliaryServiceWithCustomClassLoader.getInstance(AuxiliaryServiceWithCustomClassLoader.java:169)
at 
org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.serviceInit(AuxServices.java:249)
at 
org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
at 
org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108)
at 
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceInit(ContainerManagerImpl.java:316)
at 
org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
at 

Re: [NOTIFICATION] Hadoop trunk rebased

[Steve Loughran]

owen, did you really, really have to do this? I have an large number of patches 
against trunk and they are all now broken. Couldn't a rebase of the ozone tree 
& a merge have worked?

> On 26 Apr 2018, at 22:23, Owen O'Malley  wrote:
> 
> As we discussed in hdfs-dev@hadoop, I did a force push to Hadoop's trunk to
> replace the Ozone merge with a rebase.
> 
> That means that you'll need to rebase your branches.
> 
> .. Owen


-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Created] (YARN-8222) NPE in RMContainerImpl$FinishedTransition#updateAttemptMetrics

[Tao Yang (JIRA)]

Tao Yang created YARN-8222:
--

 Summary: NPE in 
RMContainerImpl$FinishedTransition#updateAttemptMetrics
 Key: YARN-8222
 URL: https://issues.apache.org/jira/browse/YARN-8222
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Tao Yang
Assignee: Tao Yang


This NPE looks like may happen when node heartbeat delay and try to update 
attempt metrics for a non-exist application.
Reference code of RMContainerImpl$FinishedTransition#updateAttemptMetrics:
{code:java}
private static void updateAttemptMetrics(RMContainerImpl container) {
  Resource resource = container.getContainer().getResource();
  RMAppAttempt rmAttempt = container.rmContext.getRMApps()
  .get(container.getApplicationAttemptId().getApplicationId())
  .getCurrentAppAttempt();

  if (rmAttempt != null) {
 //
  }
}
{code}
We can add a null check for application before getting attempt form it.

Error log:
{noformat}
java.lang.NullPointerException
        at 
org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl$FinishedTransition.updateAttemptMetrics(RMContainerImpl.java:742)
        at 
org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl$FinishedTransition.transition(RMContainerImpl.java:715)
        at 
org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl$FinishedTransition.transition(RMContainerImpl.java:699)
        at 
org.apache.hadoop.yarn.state.StateMachineFactory$SingleInternalArc.doTransition(StateMachineFactory.java:362)
        at 
org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
        at 
org.apache.hadoop.yarn.state.StateMachineFactory.access$500(StateMachineFactory.java:46)
        at 
org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:487)
        at 
org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl.handle(RMContainerImpl.java:482)
        at 
org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl.handle(RMContainerImpl.java:64)
        at 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerApp.containerCompleted(FiCaSchedulerApp.java:195)
        at 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.completedContainer(LeafQueue.java:1793)
        at 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.completedContainerInternal(CapacityScheduler.java:2624)
        at 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.AbstractYarnScheduler.completedContainer(AbstractYarnScheduler.java:663)
        at 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.doneApplicationAttempt(CapacityScheduler.java:1514)
        at 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:2396)
        at 
org.apache.hadoop.yarn.sls.scheduler.SLSCapacityScheduler.handle(SLSCapacityScheduler.java:205)
        at 
org.apache.hadoop.yarn.sls.scheduler.SLSCapacityScheduler.handle(SLSCapacityScheduler.java:60)
        at 
org.apache.hadoop.yarn.event.EventDispatcher$EventProcessor.run(EventDispatcher.java:66)
        at java.lang.Thread.run(Thread.java:834)
{noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [NOTIFICATION] Hadoop trunk rebased

[Akira Ajisaka]

+ common-dev and mapreduce-dev

On 2018/04/27 6:23, Owen O'Malley wrote:

As we discussed in hdfs-dev@hadoop, I did a force push to Hadoop's trunk to
replace the Ozone merge with a rebase.

That means that you'll need to rebase your branches.

.. Owen



-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org