date:20171215

[jira] [Resolved] (YARN-6950) Invalid event: LAUNCH_FAILED at FAILED

2017-12-15 Thread lujie (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-6950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

lujie resolved YARN-6950.
-
Resolution: Duplicate

> Invalid event: LAUNCH_FAILED at FAILED
> --
>
> Key: YARN-6950
> URL: https://issues.apache.org/jira/browse/YARN-6950
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: yarn
>Affects Versions: 2.6.0
>Reporter: lujie
> Fix For: 2.7.0
>
>
> A RMAppAttemptImpl fail due to some reason,meanwhile AM fails to launch a 
> container and send event  LAUNCH_FAILED,and the StateMachine can not handle 
> it:
> {code:java}
> 2017-07-05 03:33:09,013 ERROR 
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: 
> Can't handle this event at current state
> org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event: 
> LAUNCH_FAILED at FAILED
> at 
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305)
> at 
> org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
> at 
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.handle(RMAppAttemptImpl.java:757)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.handle(RMAppAttemptImpl.java:106)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationAttemptEventDispatcher.handle(ResourceManager.java:834)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationAttemptEventDispatcher.handle(ResourceManager.java:815)
> at 
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:173)
> at 
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:106)
> at java.lang.Thread.run(Thread.java:745)
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

2017-12-15 Thread Apache Jenkins Server

For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/623/

[Dec 14, 2017 6:58:16 PM] (xyao) HADOOP-10054. ViewFsFileStatus.toString() is 
broken. Contributed by
[Dec 14, 2017 8:58:47 PM] (kihwal) HDFS-12907. Allow read-only access to 
reserved raw for non-superusers.
[Dec 14, 2017 10:03:55 PM] (subru) YARN-7630. Fix AMRMToken rollover handling 
in AMRMProxy. Contributed by
[Dec 15, 2017 1:10:37 AM] (vinodkv) YARN-7565. Addendum to fix an incompatible 
change. Contributed by Wangda
[Dec 15, 2017 3:22:51 AM] (xiao) HDFS-12910. Secure Datanode Starter should log 
the port when it fails to
[Dec 15, 2017 3:58:52 AM] (wwei) YARN-7642. Add test case to verify context 
update after container
[Dec 15, 2017 6:09:24 AM] (yqlin) HDFS-12895. RBF: Add ACL support for mount 
table. Contributed by Yiqun
[Dec 15, 2017 7:30:57 AM] (sunilg) YARN-7638. Unit tests related to preemption 
for auto created leaf queues
[Dec 15, 2017 4:20:28 PM] (rohithsharmaks) YARN-7190. Ensure only NM classpath 
in 2.x gets TSv2 related hbase jars,




-1 overall


The following subsystems voted -1:
asflicense findbugs unit


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

FindBugs :

   module:hadoop-hdfs-project/hadoop-hdfs 
   Possible null pointer dereference of replication in 
org.apache.hadoop.hdfs.server.namenode.INodeFile$HeaderFormat.getBlockLayoutRedundancy(BlockType,
 Short, Byte) Dereferenced at INodeFile.java:replication in 
org.apache.hadoop.hdfs.server.namenode.INodeFile$HeaderFormat.getBlockLayoutRedundancy(BlockType,
 Short, Byte) Dereferenced at INodeFile.java:[line 210] 

FindBugs :

   module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api 
   org.apache.hadoop.yarn.api.records.Resource.getResources() may expose 
internal representation by returning Resource.resources At Resource.java:by 
returning Resource.resources At Resource.java:[line 234] 

Failed junit tests :

   hadoop.fs.viewfs.TestViewFileSystemLocalFileSystem 
   hadoop.fs.viewfs.TestViewFileSystemWithAuthorityLocalFileSystem 
   hadoop.hdfs.server.balancer.TestBalancerWithSaslDataTransfer 
   hadoop.hdfs.server.datanode.TestBlockReplacement 
   hadoop.hdfs.qjournal.server.TestJournalNodeSync 
   hadoop.hdfs.server.namenode.TestNameNodeRetryCacheMetrics 
   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure020 
   hadoop.hdfs.TestReadStripedFileWithMissingBlocks 
   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150 
   hadoop.hdfs.server.namenode.TestEditLogAutoroll 
   hadoop.hdfs.server.namenode.snapshot.TestSnapshotDeletion 
   hadoop.hdfs.server.namenode.TestCreateEditsLog 
   hadoop.hdfs.web.TestWebHdfsTimeouts 
   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure190 
   hadoop.hdfs.server.namenode.TestEditLog 
   hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup 
   hadoop.hdfs.server.balancer.TestBalancerRPCDelay 
   hadoop.hdfs.TestErasureCodingPolicies 
   hadoop.hdfs.server.federation.router.TestNamenodeHeartbeat 
   hadoop.hdfs.TestReconstructStripedFile 
   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure140 
   
hadoop.yarn.server.nodemanager.containermanager.launcher.TestContainerLaunch 
   
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestNodeLabelContainerAllocation
 
   hadoop.yarn.client.api.impl.TestAMRMClientOnRMRestart 
   hadoop.mapreduce.v2.app.rm.TestRMContainerAllocator 
   hadoop.mapreduce.v2.TestUberAM 
  

   cc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/623/artifact/out/diff-compile-cc-root.txt
  [4.0K]

   javac:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/623/artifact/out/diff-compile-javac-root.txt
  [280K]

   checkstyle:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/623/artifact/out/diff-checkstyle-root.txt
  [17M]

   pylint:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/623/artifact/out/diff-patch-pylint.txt
  [24K]

   shellcheck:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/623/artifact/out/diff-patch-shellcheck.txt
  [20K]

   shelldocs:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/623/artifact/out/diff-patch-shelldocs.txt
  [12K]

   whitespace:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/623/artifact/out/whitespace-eol.txt
  [8.8M]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/623/artifact/out/whitespace-tabs.txt
  [288K]

   findbugs:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/623/artifact/out/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
  [8.0K]

Hadoop 3.1.0 release discussion

2017-12-15 Thread Wangda Tan

Hi all,

Congratulations on the 3.0.0-GA release!

As we discussed in the previous email thread [1], I'd like to restart 3.1.0
release plans.

a) Quick summary:
a.1 Release status
We started 3.1 release discussion on Sep 6, 2017 [1]. As of today, there’re
232 patches loaded on 3.1.0 alone [2], besides 6 open blockers and 22 open
critical issues.

a.2 Release date update
Considering delays of 3.0-GA release by month-and-a-half, I propose to move
the dates as follows
 - feature freeze date from Dec 15, 2017, to Jan 30, 2018 - last date for
any branches to get merged too;
 - code freeze (blockers & critical only) date to Feb 08, 2018;
 - release voting start by Feb 18, 2018, leaving time for at least two RCx
 - release date from Jan 15, 2018, to Feb 28, 2018;

Unlike before, I added an additional milestone for release-vote-start so
that we can account for voting time-period also.

This overall is still 5 1/2 months of release-timeline unlike the faster
cadence we hoped for, but this, in my opinion, is the best-updated timeline
given the delays of the final release of 3.0-GA.

b) Individual feature status:
I spoke to several feature owners and checked the status of un-finished
features, following are status of features planned to 3.1.0:

b.1 Merged & Completed features:
* (Sunil) YARN-5881: Support absolute value in CapacityScheduler.
* (Wangda) YARN-6223: GPU support on YARN. Features in trunk and works
end-to-end.
* (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN native services.
* (Steve Loughran): HADOOP-13786: S3Guard committer for zero-rename commits.
* (Suma): YARN-7117: Capacity Scheduler: Support Auto Creation of Leaf
Queues While Doing Queue Mapping.

b.2 Features close to finish:
* (Chris Douglas) HDFS-9806: HDFS Tiered Storage. Being voting now.
* (Zhankun) YARN-5983: FPGA support. Majority implementations completed and
merged to trunk. Except for UI/documentation.
* (Uma) HDFS-10285: HDFS SPS. Majority implementations are done, some
discussions going on about implementation.

b.3 Tentative features:
* (Arun Suresh). YARN-5972: Support pausing/freezing opportunistic
containers. Only one pending patch. Plan to finish before Jan 7th.
* (Haibo Chen). YARN-1011: Resource overcommitment. Looks challenging to be
done before Jan 2018.
* (Arun Suresh / Kostas / Wangda). YARN-6592: New SchedulingRequest and
anti-affinity support. Tentative will figure out by Jan 1st.
* (Anu): HDFS-7240: Ozone. Given the discussion on HDFS-7240. Looks
challenging to be done before Jan 2018.
* (Varun V) YARN-5673: container-executor write. Given security refactoring
of c-e (YARN-6623) is already landed, IMHO other stuff may be moved to 3.2.

b.4 Additional release drivers
* More exhaustive upgrade testing from 2.x to 3.x.

c) Regarding branch cut:

We will keep pointing trunk to 3.1 and cut branch-3.1 until: A. some
feature planned to 3.2 has to be landed on trunk or B. After feature freeze
date, whichever comes first.

I've also talked offline with Vinod to get help on release-management given
this is my first release. He agreed to help do this release jointly.

Thoughts?

Thanks,
Wangda Tan

[1]
https://lists.apache.org/thread.html/c11506c3250c9481852130616b3cb09a0e222f5c2465c015f9906dab@%3Cyarn-dev.hadoop.apache.org%3E
[2] "project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
AND fixVersion not in (3.0.0,2.9.0) ORDER BY priority DESC”

[jira] [Created] (YARN-7664) Several javadoc errors

2017-12-15 Thread Sean Mackrory (JIRA)

Sean Mackrory created YARN-7664:
---

 Summary: Several javadoc errors
 Key: YARN-7664
 URL: https://issues.apache.org/jira/browse/YARN-7664
 Project: Hadoop YARN
  Issue Type: Bug
Affects Versions: 3.1.0
Reporter: Sean Mackrory
Priority: Blocker


Not sure if I'm somehow on a different version as our Yetus infra or what, but 
I'm unable to build Hadoop due to some recent changes that are throwing JavaDoc 
errors. Most significantly:

{code}
[ERROR] 
/home/sean/src/apache/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/AllocateResponse.java:379:
 error: self-closing element not allowed
[ERROR] * 
[ERROR] ^
[ERROR] 
/home/sean/src/apache/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/AllocateResponse.java:397:
 error: self-closing element not allowed
[ERROR] * 
[ERROR] ^
[ERROR] 
/home/sean/src/apache/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/util/resource/ResourceUtils.java:425:
 error: reference not found
[ERROR] * @throws IllegalArgumentExcpetion if units contain non alpha characters
[ERROR] ^
{code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: Apache YARN Committers & Contributors Meetup #5

2017-12-15 Thread Vinod Kumar Vavilapalli

Thanks for that update, Sunil! Good to hear that we had such a successful 
meet-up in Bangalore! Hoping we get back to a regular cadence!

Is it worth doing a simple blog post on this at 
https://blogs.apache.org/hadoop/  , may be 
with pics etc?

Plus Daniel explicitly too. Daniel, can you post the minutes from our YARN 
meetup in the Western hemisphere too? Shall we do a blog post on that too or 
combine it with Sunil's?

Thanks
+Vinod

> On Dec 14, 2017, at 11:08 AM, Sunil G  wrote:
> 
> Hi All,
> 
> Bangalore Hadoop Meetup was recently conducted @Microsoft campus on Dec 8th
> almost in line with YARN Contributors Meetup @CA. Many thanks to Newton
> from MS for organizing this quickly.
> 
> Quick Update about this:
> 1. We had *150*+ participants for this event which was one of the best
> turnout for Hadoop specific meetup in recent times.
> 2. There were 5 talks in total and majority of the talks were on YARN
> (scale/GPU etc). Also shared meetup link below for detailed schedule.
> 3. We were happy to see a excellent turn out from all Hadoop/YARN
> committers and contributors @Bangalore for this event.
> 4. Discussed majorly about upcoming Hadoop 3.0 and next major releases in
> same line. Lot of interests on Resource types / GPU scheduling / EC etc and
> also doubts about rolling upgrade to Hadoop 3.
> 
> Meetup Link: (Slides and Videos will be shared in same link shortly)
> http://meetup.com/Bangalore-Hadoop-Meetups/events/245123037/
> 
> Thanks once again to all folks who joined. We will plan a Bug Bash in same
> timeline when Meetup #5 is going to happen at CA.
> 
> Warm Regards,
> Sunil G
> 
> 
> 
> On Sat, Dec 2, 2017 at 3:55 AM Daniel Templeton  wrote:
> 
>> And thanks to Vinod we now have an official Meetup online:
>> 
>> https://www.meetup.com/Hadoop-Contributors/events/245569075/
>> 
>> Daniel
>> 
>> On 11/22/17 12:20 PM, Daniel Templeton wrote:
>>> We're long past due for another contributors meetup.  Cloudera is
>>> excited to host this event at our new Palo Alto Galactic Headquarters
>>> (
>> https://www.google.com/maps/place/Cloudera+Galactic+HQ/@37.4254615,-122.1413431,17z
>> )
>>> on December 6th from 2pm-5pm PST.  We will have a Google Hangout set
>>> up so that remote participants can dial in.  We will also provide
>>> drinks and snacks.
>>> 
>>> Our agenda will be roughly:
>>> 
>>> * Celebrate Hadoop 3.0.0, identify any loose ends, and talk about 3.0.1
>>> * Planning around Hadoop 3.1.0
>>> * Planning around the branch-2 releases, Hadoop 2.10.0, etc.
>>> * Key signing
>>> * Plan for the next bug bash
>>> 
>>> Hope to see you there!
>>> Daniel
>>> 
>>> PS: I wasn't able to figure out how to create an event in our meetup
>>> group for this.  Anyone want to give me a pointer or set it up?
>> 
>> 
>> -
>> To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
>> For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
>> 
>>

[RESULT] [VOTE] Release Apache Hadoop 2.7.5 (RC1)

2017-12-15 Thread Konstantin Shvachko

Updating again to reflect all votes.

Correction:
With 8 binding and 4 non-binding +1s and no -1s the vote for Apache Release
2.7.5 passes.
Thank you everybody for contributing to the release, testing it, and voting.

Binding +1s
Kihwal Lee
Jason Lowe
John Zhuge
Rohith Sharma K S
Eric Payne
Zhe Zhang
Konstantin Shvachko
Naganarasimha Garla

Non-binding +1s
Erik Krogen
Brahma Reddy Battula
Eric Badger
Jonathan Hung



>> On Thu, Dec 7, 2017 at 7:22 PM, Konstantin Shvachko > > wrote:
>>
>>> Hi everybody,
>>>
>>> I updated CHANGES.txt and fixed documentation links.
>>> Also committed  MAPREDUCE-6165, which fixes a consistently failing test.
>>>
>>> This is RC1 for the next dot release of Apache Hadoop 2.7 line. The
>>> previous one 2.7.4 was release August 4, 2017.
>>> Release 2.7.5 includes critical bug fixes and optimizations. See more
>>> details in Release Note:
>>> http://home.apache.org/~shv/hadoop-2.7.5-RC1/releasenotes.html
>>>
>>> The RC0 is available at: http://home.apache.org/~shv/hadoop-2.7.5-RC1/
>>>
>>> Please give it a try and vote on this thread. The vote will run for 5
>>> days ending 12/13/2017.
>>>
>>> My up to date public key is available from:
>>> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>>>
>>> Thanks,
>>> --Konstantin
>>>
>>
>>
>

Re: [ANNOUNCE] Apache Hadoop 3.0.0 GA is released

2017-12-15 Thread Andrew Wang

Hi Arpit,

If you look at the release announcements, it's made clear that the
changelog for 3.0.0 is diffed based on beta1. This is important since users
need to know what's different from the previous 3.0.0-* releases if they're
upgrading.

I agree there's additional value to making combined release notes, but it'd
be something additive rather than replacing what's there.

Best,
Andrew

On Fri, Dec 15, 2017 at 8:27 AM, Arpit Agarwal 
wrote:

>
> Hi Andrew,
>
> Thank you for all the hard work on this release. I was out the last few
> days and didn’t get a chance to evaluate RC1 earlier.
>
> The changelog looks incorrect. E.g. This gives an impression that there
> are just 5 incompatible changes in 3.0.0.
> http://hadoop.apache.org/docs/r3.0.0/hadoop-project-dist/
> hadoop-common/release/3.0.0/CHANGES.3.0.0.html
>
> I assume you only counted 3.0.0 changes in this log excluding
> alphas/betas. However, users shouldn’t have to manually compile
> incompatibilities by summing up a/b release notes. Can we fix the changelog
> after the fact?
>
>
>
>
> On 12/14/17, 10:45 AM, "Andrew Wang"  wrote:
>
> Hi all,
>
> I'm pleased to announce that Apache Hadoop 3.0.0 is generally available
> (GA).
>
> 3.0.0 GA consists of 302 bug fixes, improvements, and other
> enhancements
> since 3.0.0-beta1. This release marks a point of quality and stability
> for
> the 3.0.0 release line, and users of earlier 3.0.0-alpha and -beta
> releases
> are encouraged to upgrade.
>
> Looking back, 3.0.0 GA is the culmination of over a year of work on the
> 3.0.0 line, starting with 3.0.0-alpha1 which was released in September
> 2016. Altogether, 3.0.0 incorporates 6,242 changes since 2.7.0.
>
> Users are encouraged to read the overview of major changes
>  in 3.0.0. The GA
> release
> notes
>  dist/hadoop-common/release/3.0.0/RELEASENOTES.3.0.0.html>
>  and changelog
>  dist/hadoop-common/release/3.0.0/CHANGES.3.0.0.html>
> detail
> the changes since 3.0.0-beta1.
>
> The ASF press release provides additional color and highlights some of
> the
> major features:
>
> https://globenewswire.com/news-release/2017/12/14/
> 1261879/0/en/The-Apache-Software-Foundation-Announces-
> Apache-Hadoop-v3-0-0-General-Availability.html
>
> Let me end by thanking the many, many contributors who helped with this
> release line. We've only had three major releases in Hadoop's 10 year
> history, and this is our biggest major release ever. It's an incredible
> accomplishment for our community, and I'm proud to have worked with
> all of
> you.
>
> Best,
> Andrew
>
>
>
>
>
>
>
>

Re: [RESULT] [VOTE] Release Apache Hadoop 2.7.5 (RC1)

2017-12-15 Thread Naganarasimha Garla

Sorry for the delayed reply,  and thanks for the work Konstantin
+1 Binding

Verified the following:
- Downloaded the tar on Ubuntu and verified the signatures
- Deployed pseudo cluster
- Sanity checks
- Basic hdfs operations
- Spark PyWordcount & few MR jobs (sleep , word count ..)
- Accessed most of the web UI's

Regards,
+ Naga

On Sat, Dec 16, 2017 at 12:29 AM, Konstantin Shvachko 
wrote:

> Correction:
> With 7 binding and 4 non-binding +1s and no -1s the vote for Apache Release
> 2.7.5 passes.
> Thank you everybody for contributing to the release, testing it, and
> voting.
>
> Binding +1s
> Kihwal Lee
> Jason Lowe
> John Zhuge
> Rohith Sharma K S
> Eric Payne
> Zhe Zhang
> Konstantin Shvachko
>
> Non-binding +1s
> Erik Krogen
> Brahma Reddy Battula
> Eric Badger
> Jonathan Hung
>
>
> On Fri, Dec 15, 2017 at 8:25 AM, Konstantin Shvachko  >
> wrote:
>
> > Hi everybody,
> >
> > With 6 binding and 4 non-binding +1s and no -1s the vote for Apache
> > Release 2.7.5 passes.
> > Thank you everybody for contributing to the release, testing it, and
> > voting.
> >
> > Binding +1s
> > Kihwal Lee
> > Jason Lowe
> > John Zhuge
> > Rohith Sharma K S
> > Eric Payne
> > Zhe Zhang
> >
> > Non-binding +1s
> > Erik Krogen
> > Brahma Reddy Battula
> > Eric Badger
> > Jonathan Hung
> >
> >
> > On Thu, Dec 7, 2017 at 7:22 PM, Konstantin Shvachko <
> shv.had...@gmail.com>
> > wrote:
> >
> >> Hi everybody,
> >>
> >> I updated CHANGES.txt and fixed documentation links.
> >> Also committed  MAPREDUCE-6165, which fixes a consistently failing test.
> >>
> >> This is RC1 for the next dot release of Apache Hadoop 2.7 line. The
> >> previous one 2.7.4 was release August 4, 2017.
> >> Release 2.7.5 includes critical bug fixes and optimizations. See more
> >> details in Release Note:
> >> http://home.apache.org/~shv/hadoop-2.7.5-RC1/releasenotes.html
> >>
> >> The RC0 is available at: http://home.apache.org/~shv/hadoop-2.7.5-RC1/
> >>
> >> Please give it a try and vote on this thread. The vote will run for 5
> >> days ending 12/13/2017.
> >>
> >> My up to date public key is available from:
> >> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >>
> >> Thanks,
> >> --Konstantin
> >>
> >
> >
>

[RESULT] [VOTE] Release Apache Hadoop 2.7.5 (RC1)

2017-12-15 Thread Konstantin Shvachko

Hi everybody,

With 6 binding and 4 non-binding +1s and no -1s the vote for Apache Release
2.7.5 passes.
Thank you everybody for contributing to the release, testing it, and voting.

Binding +1s
Kihwal Lee
Jason Lowe
John Zhuge
Rohith Sharma K S
Eric Payne
Zhe Zhang

Non-binding +1s
Erik Krogen
Brahma Reddy Battula
Eric Badger
Jonathan Hung


On Thu, Dec 7, 2017 at 7:22 PM, Konstantin Shvachko 
wrote:

> Hi everybody,
>
> I updated CHANGES.txt and fixed documentation links.
> Also committed  MAPREDUCE-6165, which fixes a consistently failing test.
>
> This is RC1 for the next dot release of Apache Hadoop 2.7 line. The
> previous one 2.7.4 was release August 4, 2017.
> Release 2.7.5 includes critical bug fixes and optimizations. See more
> details in Release Note:
> http://home.apache.org/~shv/hadoop-2.7.5-RC1/releasenotes.html
>
> The RC0 is available at: http://home.apache.org/~shv/hadoop-2.7.5-RC1/
>
> Please give it a try and vote on this thread. The vote will run for 5
> days ending 12/13/2017.
>
> My up to date public key is available from:
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
> Thanks,
> --Konstantin
>

Re: [VOTE] Release Apache Hadoop 2.7.5 (RC1)

2017-12-15 Thread Konstantin Shvachko

Here is my formal +1.

Thanks,
--Konstantin

On Thu, Dec 7, 2017 at 7:22 PM, Konstantin Shvachko 
wrote:

> Hi everybody,
>
> I updated CHANGES.txt and fixed documentation links.
> Also committed  MAPREDUCE-6165, which fixes a consistently failing test.
>
> This is RC1 for the next dot release of Apache Hadoop 2.7 line. The
> previous one 2.7.4 was release August 4, 2017.
> Release 2.7.5 includes critical bug fixes and optimizations. See more
> details in Release Note:
> http://home.apache.org/~shv/hadoop-2.7.5-RC1/releasenotes.html
>
> The RC0 is available at: http://home.apache.org/~shv/hadoop-2.7.5-RC1/
>
> Please give it a try and vote on this thread. The vote will run for 5
> days ending 12/13/2017.
>
> My up to date public key is available from:
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
> Thanks,
> --Konstantin
>

[jira] [Resolved] (YARN-7659) NodeManager metrics return wrong value after update resource

2017-12-15 Thread Jason Lowe (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-7659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Lowe resolved YARN-7659.
--
Resolution: Duplicate

> NodeManager metrics return wrong value after update resource
> 
>
> Key: YARN-7659
> URL: https://issues.apache.org/jira/browse/YARN-7659
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Yang Wang
>
> {code:title=NodeManagerMetrics.java}
>   public void addResource(Resource res) {
> availableMB = availableMB + res.getMemorySize();
> availableGB.incr((int)Math.floor(availableMB/1024d));
> availableVCores.incr(res.getVirtualCores());
>   }
> {code}
> When the node resource was updated through RM-NM heartbeat, the NM metric 
> will get wrong value. 
> The root cause of this issue is that new resource has been added to 
> availableMB, so not needed to increase for availableGB again.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Created] (YARN-7662) [Atsv2] Remove timeline.service.address dependency for starting Timeline Reader in secure mode

2017-12-15 Thread Rohith Sharma K S (JIRA)

Rohith Sharma K S created YARN-7662:
---

 Summary: [Atsv2] Remove timeline.service.address dependency for 
starting Timeline Reader in secure mode 
 Key: YARN-7662
 URL: https://issues.apache.org/jira/browse/YARN-7662
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Rohith Sharma K S


While starting Timeline Reader in secure mode, login happens using 
timeline.service.address even though timeline.service.bindhost is configured 
with 0.0.0.0. This causes exact principal name that matches address name to be 
present in keytabs. 

It is always better to login using getLocalHost that gives machine hostname 
which is configured in /etc/hosts unlike NodeManager does in serviceStart. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Apache Hadoop qbt Report: branch2+JDK7 on Linux/x86

2017-12-15 Thread Apache Jenkins Server

For more details, see 
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/69/

[Dec 14, 2017 8:03:34 AM] (yqlin) HDFS-12883. RBF: Document Router and State 
Store metrics. Contributed by




-1 overall


The following subsystems voted -1:
asflicense unit xml


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

Unreaped Processes :

   hadoop-common:1 
   hadoop-hdfs:25 
   bkjournal:5 
   hadoop-yarn-common:1 
   hadoop-yarn-server-applicationhistoryservice:1 
   hadoop-yarn-server-timelineservice:1 
   hadoop-yarn-server-resourcemanager:2 
   hadoop-yarn-client:5 
   hadoop-yarn-applications-distributedshell:1 
   hadoop-mapreduce-client-jobclient:13 
   hadoop-distcp:2 
   hadoop-archives:1 

Failed junit tests :

   hadoop.hdfs.TestSnapshotCommands 
   hadoop.hdfs.TestDFSClientFailover 
   
hadoop.hdfs.server.datanode.metrics.TestDataNodeOutlierDetectionViaMetrics 
   hadoop.hdfs.server.datanode.TestDataNodeFaultInjector 
   hadoop.hdfs.server.datanode.TestIncrementalBlockReports 
   hadoop.hdfs.server.datanode.metrics.TestSlowNodeDetector 
   hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations 
   hadoop.hdfs.web.TestWebHdfsWithAuthenticationFilter 
   hadoop.hdfs.server.datanode.TestIncrementalBrVariations 
   hadoop.hdfs.TestSeekBug 
   hadoop.hdfs.TestDFSFinalize 
   hadoop.hdfs.server.datanode.TestTransferRbw 
   hadoop.hdfs.web.TestWebHdfsFileSystemContract 
   hadoop.hdfs.server.datanode.TestBlockRecovery 
   hadoop.hdfs.server.datanode.TestDataNodeFSDataSetSink 
   hadoop.hdfs.web.TestWebHdfsWithMultipleNameNodes 
   hadoop.hdfs.server.datanode.TestNNHandlesCombinedBlockReport 
   hadoop.hdfs.web.TestFSMainOperationsWebHdfs 
   hadoop.yarn.server.timeline.webapp.TestTimelineWebServicesWithSSL 
   
hadoop.yarn.server.timeline.security.TestTimelineAuthenticationFilterForV1 
   
hadoop.yarn.server.resourcemanager.scheduler.fair.TestContinuousScheduling 
   hadoop.yarn.server.TestDiskFailures 
   
hadoop.yarn.applications.distributedshell.TestDistributedShellWithNodeLabels 
   hadoop.mapreduce.TestMRJobClient 
   hadoop.mapreduce.security.ssl.TestEncryptedShuffle 
   hadoop.mapreduce.security.TestBinaryTokenFile 
   hadoop.fs.TestFileSystem 
   hadoop.mapreduce.TestChild 
   hadoop.mapreduce.security.TestMRCredentials 
   hadoop.conf.TestNoDefaultsJobConf 
   hadoop.fs.TestDFSIO 
   hadoop.mapred.TestJobSysDirWithDFS 
   hadoop.tools.TestIntegration 
   hadoop.tools.TestDistCpViewFs 
   hadoop.resourceestimator.solver.impl.TestLpSolver 
   hadoop.resourceestimator.service.TestResourceEstimatorService 

Timed out junit tests :

   org.apache.hadoop.log.TestLogLevel 
   org.apache.hadoop.hdfs.TestLeaseRecovery2 
   org.apache.hadoop.hdfs.server.datanode.TestRefreshNamenodes 
   org.apache.hadoop.hdfs.server.datanode.TestLargeBlockReport 
   org.apache.hadoop.hdfs.TestRead 
   org.apache.hadoop.hdfs.web.TestWebHdfsTokens 
   org.apache.hadoop.hdfs.TestDFSInotifyEventInputStream 
   org.apache.hadoop.hdfs.TestFileAppendRestart 
   org.apache.hadoop.hdfs.TestReadWhileWriting 
   org.apache.hadoop.hdfs.security.TestDelegationToken 
   org.apache.hadoop.hdfs.web.TestWebHdfsWithRestCsrfPreventionFilter 
   org.apache.hadoop.hdfs.TestDFSMkdirs 
   org.apache.hadoop.hdfs.TestDFSOutputStream 
   org.apache.hadoop.hdfs.web.TestWebHDFS 
   org.apache.hadoop.metrics2.sink.TestRollingFileSystemSinkWithSecureHdfs 
   org.apache.hadoop.metrics2.sink.TestRollingFileSystemSinkWithHdfs 
   org.apache.hadoop.hdfs.server.datanode.TestBlockScanner 
   org.apache.hadoop.hdfs.TestDistributedFileSystem 
   org.apache.hadoop.hdfs.TestReplaceDatanodeFailureReplication 
   org.apache.hadoop.hdfs.TestDFSShell 
   org.apache.hadoop.contrib.bkjournal.TestBootstrapStandbyWithBKJM 
   org.apache.hadoop.contrib.bkjournal.TestBookKeeperJournalManager 
   org.apache.hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints 
   org.apache.hadoop.contrib.bkjournal.TestBookKeeperAsHASharedDir 
   org.apache.hadoop.contrib.bkjournal.TestBookKeeperSpeculativeRead 
   org.apache.hadoop.yarn.webapp.TestWebApp 
   
org.apache.hadoop.yarn.server.applicationhistoryservice.TestApplicationHistoryServer
 
   
org.apache.hadoop.yarn.server.timelineservice.reader.TestTimelineReaderWebServices
 
   
org.apache.hadoop.yarn.server.resourcemanager.recovery.TestFSRMStateStore 
   org.apache.hadoop.yarn.server.resourcemanager.TestRMAdminService 
   org.apache.hadoop.yarn.client.TestRMFailover

[jira] [Created] (YARN-7661) NodeManager metrics return wrong value after update node resource

2017-12-15 Thread Yang Wang (JIRA)

Yang Wang created YARN-7661:
---

 Summary: NodeManager metrics return wrong value after update node 
resource
 Key: YARN-7661
 URL: https://issues.apache.org/jira/browse/YARN-7661
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Yang Wang


{code:title=NodeManagerMetrics.java}
  public void addResource(Resource res) {
availableMB = availableMB + res.getMemorySize();
availableGB.incr((int)Math.floor(availableMB/1024d));
availableVCores.incr(res.getVirtualCores());
  }
{code}
When the node resource was updated through RM-NM heartbeat, the NM metric will 
get wrong value. 
The root cause of this issue is that new resource has been added to 
availableMB, so not needed to increase for availableGB again.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Created] (YARN-7660) NodeManager metrics return wrong value after update node resource

2017-12-15 Thread Yang Wang (JIRA)

Yang Wang created YARN-7660:
---

 Summary: NodeManager metrics return wrong value after update node 
resource
 Key: YARN-7660
 URL: https://issues.apache.org/jira/browse/YARN-7660
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Yang Wang


{code:title=NodeManagerMetrics.java}
  public void addResource(Resource res) {
availableMB = availableMB + res.getMemorySize();
availableGB.incr((int)Math.floor(availableMB/1024d));
availableVCores.incr(res.getVirtualCores());
  }
{code}
When the node resource was updated through RM-NM heartbeat, the NM metric will 
get wrong value. 
The root cause of this issue is that new resource has been added to 
availableMB, so not needed to increase for availableGB again.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Created] (YARN-7659) NodeManager metrics return wrong value after update resource

2017-12-15 Thread Yang Wang (JIRA)

Yang Wang created YARN-7659:
---

 Summary: NodeManager metrics return wrong value after update 
resource
 Key: YARN-7659
 URL: https://issues.apache.org/jira/browse/YARN-7659
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Yang Wang


{code:title=NodeManagerMetrics.java}
  public void addResource(Resource res) {
availableMB = availableMB + res.getMemorySize();
availableGB.incr((int)Math.floor(availableMB/1024d));
availableVCores.incr(res.getVirtualCores());
  }
{code}
When the node resource was updated through RM-NM heartbeat, the NM metric will 
get wrong value. 
The root cause of this issue is that new resource has been added to 
availableMB, so not needed to increase for availableGB again.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Resolved] (YARN-6950) Invalid event: LAUNCH_FAILED at FAILED

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

Hadoop 3.1.0 release discussion

[jira] [Created] (YARN-7664) Several javadoc errors

Re: Apache YARN Committers & Contributors Meetup #5

[RESULT] [VOTE] Release Apache Hadoop 2.7.5 (RC1)

Re: [ANNOUNCE] Apache Hadoop 3.0.0 GA is released

Re: [RESULT] [VOTE] Release Apache Hadoop 2.7.5 (RC1)

[RESULT] [VOTE] Release Apache Hadoop 2.7.5 (RC1)

Re: [VOTE] Release Apache Hadoop 2.7.5 (RC1)

[jira] [Resolved] (YARN-7659) NodeManager metrics return wrong value after update resource

[jira] [Created] (YARN-7662) [Atsv2] Remove timeline.service.address dependency for starting Timeline Reader in secure mode

Apache Hadoop qbt Report: branch2+JDK7 on Linux/x86

[jira] [Created] (YARN-7661) NodeManager metrics return wrong value after update node resource

[jira] [Created] (YARN-7660) NodeManager metrics return wrong value after update node resource

[jira] [Created] (YARN-7659) NodeManager metrics return wrong value after update resource

16 matches

Site Navigation

Mail list logo

Footer information