Re: [VOTE] Release Apache Hadoop 2.8.3 (RC0)

[Jian He]

+1
I built from source code
deployed a cluster
and successfully ran jobs while restarting RM as well

Jian

 
> On Dec 12, 2017, at 4:28 PM, John Zhuge  wrote:
> 
> Thanks Junping for the great effort!
> 
> 
>   - Verified checksums and signatures of all tarballs
>   - Built source with native, Azul Java 1.7.0_161 on Mac OS X 10.13.2
>   - Verified cloud connectors:
>  - All S3A integration tests
>   - Deployed both binary and built source to a pseudo cluster, passed the
>   following sanity tests in insecure and SSL mode:
>  - HDFS basic and ACL
>  - DistCp basic
>  - MapReduce wordcount
>  - KMS and HttpFS basic
>  - Balancer start/stop
> 
> 
> Non-blockers
> 
>   - HADOOP-13030 Handle special characters in passwords in KMS startup
>   script. Fixed in 2.8+.
>   - NameNode servlets test failures: 403 User dr.who is unauthorized to
>   access this page. Researching. Could be just test configuration issue.
> 
> John
> 
> On Tue, Dec 12, 2017 at 1:10 PM, Eric Badger 
> wrote:
> 
>> Thanks, Junping
>> 
>> +1 (non-binding) looks good from my end
>> 
>> - Verified all hashes and checksums
>> - Built from source on macOS 10.12.6, Java 1.8.0u65
>> - Deployed a pseudo cluster
>> - Ran some example jobs
>> 
>> Eric
>> 
>> On Tue, Dec 12, 2017 at 12:55 PM, Konstantin Shvachko <
>> shv.had...@gmail.com>
>> wrote:
>> 
>>> Downloaded again, now the checksums look good. Sorry my fault
>>> 
>>> Thanks,
>>> --Konstantin
>>> 
>>> On Mon, Dec 11, 2017 at 5:03 PM, Junping Du  wrote:
>>> 
 Hi Konstantin,
 
 Thanks for verification and comments. I was verifying your example
 below but found it is actually matched:
 
 
 *jduMBP:hadoop-2.8.3 jdu$ md5 ~/Downloads/hadoop-2.8.3-src.tar.gz*
 *MD5 (/Users/jdu/Downloads/hadoop-2.8.3-src.tar.gz) =
 e53d04477b85e8b58ac0a26468f04736*
 
 What's your md5 checksum for given source tar ball?
 
 
 Thanks,
 
 
 Junping
 
 
 --
 *From:* Konstantin Shvachko 
 *Sent:* Saturday, December 9, 2017 11:06 AM
 *To:* Junping Du
 *Cc:* common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org;
 mapreduce-dev@hadoop.apache.org; yarn-...@hadoop.apache.org
 *Subject:* Re: [VOTE] Release Apache Hadoop 2.8.3 (RC0)
 
 Hey Junping,
 
 Could you pls upload mds relative to the tar.gz etc. files rather than
 their full path
 /build/source/target/artifacts/hadoop-2.8.3-src.tar.gz:
   MD5 = E5 3D 04 47 7B 85 E8 B5  8A C0 A2 64 68 F0 47 36
 
 Otherwise mds don't match for me.
 
 Thanks,
 --Konstantin
 
 On Tue, Dec 5, 2017 at 1:58 AM, Junping Du 
>> wrote:
 
> Hi all,
> I've created the first release candidate (RC0) for Apache Hadoop
> 2.8.3. This is our next maint release to follow up 2.8.2. It includes
>> 79
> important fixes and improvements.
> 
>  The RC artifacts are available at:
>> http://home.apache.org/~junpin
> g_du/hadoop-2.8.3-RC0
> 
>  The RC tag in git is: release-2.8.3-RC0
> 
>  The maven artifacts are available via repository.apache.org at:
> https://repository.apache.org/content/repositories/
>> orgapachehadoop-1072
> 
>  Please try the release and vote; the vote will run for the
>> usual 5
> working days, ending on 12/12/2017 PST time.
> 
> Thanks,
> 
> Junping
> 
 
 
>>> 
>> 
> 
> 
> 
> -- 
> John


-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [VOTE] Merge Absolute resource configuration support in Capacity Scheduler (YARN-5881) to trunk

[Jian He]

My late +1 

> On Dec 7, 2017, at 7:38 PM, Sunil G  wrote:
> 
> Thank You all.
> 
> We merged the branch to trunk and updated jiras accordingly. Thanks
> everyone who helped in this feature.
> 
> - Sunil and Wangda
> 
> 
> On Thu, Dec 7, 2017 at 10:13 PM Sunil G  wrote:
> 
>> And lastly +1 (binding) from myself.
>> Vote passes with 6 (+1) bindings by considering Weiwei's vote as binding
>> itself.
>> 
>> Thank you very much for all who voted. I’ll merge to trunk by the end of
>> today.
>> 
>> 
>> - Sunil
>> 
>> 
>> 
>> On Thu, Dec 7, 2017 at 8:08 AM Subramaniam V K  wrote:
>> 
>>> +1.
>>> 
>>> Skimmed through the design doc and uber patch and seems to be reasonable.
>>> 
>>> This is a welcome addition especially w.r.t. cloud deployments so thanks
>>> to everyone who worked on this.
>>> 
>>> On Mon, Dec 4, 2017 at 8:18 PM, Rohith Sharma K S <
>>> rohithsharm...@apache.org> wrote:
>>> 
 +1
 
 On Nov 30, 2017 7:26 AM, "Sunil G"  wrote:
 
> Hi All,
> 
> 
> Based on the discussion at [1], I'd like to start a vote to merge
 feature
> branch
> 
> YARN-5881 to trunk. Vote will run for 7 days, ending Wednesday Dec 6 at
> 6:00PM PDT.
> 
> 
> This branch adds support to configure queue capacity as absolute
 resource
> in
> 
> capacity scheduler. This will help admins who want fine control of
> resources of queues.
> 
> 
> Feature development is done at YARN-5881 [2], jenkins build is here
> (YARN-7510 [3]).
> 
> All required tasks for this feature are committed. This feature changes
> RM’s Capacity Scheduler only,
> 
> and we did extensive tests for the feature in the last couple of months
> including performance tests.
> 
> 
> Key points:
> 
> - The feature is turned off by default, and have to configure absolute
> resource to enable same.
> 
> - Detailed documentation about how to use this feature is done as part
 of
> [4].
> 
> - No major performance degradation is observed with this branch work.
 SLS
> and UT performance
> 
> tests are done.
> 
> 
> There were 11 subtasks completed for this feature.
> 
> 
> Huge thanks to everyone who helped with reviews, commits, guidance, and
> 
> technical discussion/design, including Wangda Tan, Vinod Vavilapalli,
> Rohith Sharma K S, Eric Payne .
> 
> 
> [1] :
> http://mail-archives.apache.org/mod_mbox/hadoop-yarn-dev/201711.mbox/%
> 3CCACYiTuhKhF1JCtR7ZFuZSEKQ4sBvN_n_tV5GHsbJ3YeyJP%2BP4Q%
> 40mail.gmail.com%3E
> 
> [2] : https://issues.apache.org/jira/browse/YARN-5881
> 
> [3] : https://issues.apache.org/jira/browse/YARN-7510
> 
> [4] : https://issues.apache.org/jira/browse/YARN-7533
> 
> 
> Regards
> 
> Sunil and Wangda
> 
 
>>> 
>>> 

Re: [VOTE] Release Apache Hadoop 2.9.0 (RC3)

[Jian He]

+1 (binding)

Built from source code.
Successfully ran jobs and in the meantime tried to restart RM.

Jian

> On Nov 13, 2017, at 4:10 PM, Arun Suresh  wrote:
> 
> Hi Folks,
> 
> Apache Hadoop 2.9.0 is the first release of Hadoop 2.9 line and will be the
> starting release for Apache Hadoop 2.9.x line - it includes 30 New Features
> with 500+ subtasks, 407 Improvements, 790 Bug fixes new fixed issues since
> 2.8.2.
> 
> More information about the 2.9.0 release plan can be found here:
> *https://cwiki.apache.org/confluence/display/HADOOP/Roadmap#Roadmap-Version2.9
> *
> 
> New RC is available at: *https://home.apache.org/~asuresh/hadoop-2.9.0-RC3/
> *
> 
> The RC tag in git is: release-2.9.0-RC3, and the latest commit id is:
> 756ebc8394e473ac25feac05fa493f6d612e6c50.
> 
> The maven artifacts are available via repository.apache.org at:
> *https://repository.apache.org/content/repositories/orgapachehadoop-1068/
> *
> 
> We are carrying over the votes from the previous RC given that the delta is
> the license fix.
> 
> Given the above - we are also going to stick with the original deadline for
> the vote : ending on Friday 17th November 2017 2pm PT time.
> 
> Thanks,
> -Arun/Subru


-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [VOTE] Merge yarn-native-services branch into trunk

[Jian He]

Okay, I just merged the branch to trunk (108 commits in total !)
Again, thanks for all who contributed to this feature!

Jian

On Nov 6, 2017, at 1:26 PM, Jian He 
<j...@hortonworks.com<mailto:j...@hortonworks.com>> wrote:

Here’s +1 from myself.
The vote passes with 7 (+1) bindings and 2 (+1) non-bindings.

Thanks for all who voted. I’ll merge to trunk by the end of today.

Jian

On Nov 6, 2017, at 8:38 AM, Billie Rinaldi 
<billie.rina...@gmail.com<mailto:billie.rina...@gmail.com>> wrote:

+1 (binding)

On Mon, Oct 30, 2017 at 1:06 PM, Jian He 
<j...@hortonworks.com<mailto:j...@hortonworks.com>> wrote:
Hi All,

I would like to restart the vote for merging yarn-native-services to trunk.
Since last vote, we have been working on several issues in documentation, DNS, 
CLI modifications etc. We believe now the feature is in a much better shape.

Some back ground:
At a high level, the following are the key feautres implemented.
- YARN-5079[1]. A native YARN framework (ApplicationMaster) to orchestrate 
existing services to YARN either docker or non-docker based.
- YARN-4793[2]. A Rest API service embeded in RM (optional)  for user to deploy 
a service via a simple JSON spec
- YARN-4757[3]. Extending today's service registry with a simple DNS service to 
enable users to discover services deployed on YARN via standard DNS lookup
- YARN-6419[4]. UI support for native-services on the new YARN UI
All these new services are optional and are sitting outside of the existing 
system, and have no impact on existing system if disabled.

Special thanks to a team of folks who worked hard towards this: Billie Rinaldi, 
Gour Saha, Vinod Kumar Vavilapalli, Jonathan Maron, Rohith Sharma K S, Sunil G, 
Akhil PB, Eric Yang. This effort could not be possible without their ideas and 
hard work.
Also thanks Allen for some review and verifications.

Thanks,
Jian

[1] https://issues.apache.org/jira/browse/YARN-5079
[2] https://issues.apache.org/jira/browse/YARN-4793
[3] https://issues.apache.org/jira/browse/YARN-4757
[4] https://issues.apache.org/jira/browse/YARN-6419

Re: [VOTE] Merge yarn-native-services branch into trunk

[Jian He]

Here’s +1 from myself.
The vote passes with 7 (+1) bindings and 2 (+1) non-bindings.

Thanks for all who voted. I’ll merge to trunk by the end of today.

Jian

On Nov 6, 2017, at 8:38 AM, Billie Rinaldi 
<billie.rina...@gmail.com<mailto:billie.rina...@gmail.com>> wrote:

+1 (binding)

On Mon, Oct 30, 2017 at 1:06 PM, Jian He 
<j...@hortonworks.com<mailto:j...@hortonworks.com>> wrote:
Hi All,

I would like to restart the vote for merging yarn-native-services to trunk.
Since last vote, we have been working on several issues in documentation, DNS, 
CLI modifications etc. We believe now the feature is in a much better shape.

Some back ground:
At a high level, the following are the key feautres implemented.
- YARN-5079[1]. A native YARN framework (ApplicationMaster) to orchestrate 
existing services to YARN either docker or non-docker based.
- YARN-4793[2]. A Rest API service embeded in RM (optional)  for user to deploy 
a service via a simple JSON spec
- YARN-4757[3]. Extending today's service registry with a simple DNS service to 
enable users to discover services deployed on YARN via standard DNS lookup
- YARN-6419[4]. UI support for native-services on the new YARN UI
All these new services are optional and are sitting outside of the existing 
system, and have no impact on existing system if disabled.

Special thanks to a team of folks who worked hard towards this: Billie Rinaldi, 
Gour Saha, Vinod Kumar Vavilapalli, Jonathan Maron, Rohith Sharma K S, Sunil G, 
Akhil PB, Eric Yang. This effort could not be possible without their ideas and 
hard work.
Also thanks Allen for some review and verifications.

Thanks,
Jian

[1] https://issues.apache.org/jira/browse/YARN-5079
[2] https://issues.apache.org/jira/browse/YARN-4793
[3] https://issues.apache.org/jira/browse/YARN-4757
[4] https://issues.apache.org/jira/browse/YARN-6419

Re: [VOTE] Merge yarn-native-services branch into trunk

[Jian He]

Few more things:

This is the document for trying a non-docker service on YARN.
https://github.com/apache/hadoop/blob/yarn-native-services/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/yarn-service/QuickStart.md

And the document for a docker based service
https://github.com/apache/hadoop/blob/yarn-native-services/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/yarn-service/Examples.md

And the vote lasts 7 days as usual.

Thanks,
Jian

On Oct 30, 2017, at 1:06 PM, Jian He 
<j...@hortonworks.com<mailto:j...@hortonworks.com>> wrote:

Hi All,

I would like to restart the vote for merging yarn-native-services to trunk.
Since last vote, we have been working on several issues in documentation, DNS, 
CLI modifications etc. We believe now the feature is in a much better shape.

Some back ground:
At a high level, the following are the key feautres implemented.
- YARN-5079[1]. A native YARN framework (ApplicationMaster) to orchestrate 
existing services to YARN either docker or non-docker based.
- YARN-4793[2]. A Rest API service embeded in RM (optional)  for user to deploy 
a service via a simple JSON spec
- YARN-4757[3]. Extending today's service registry with a simple DNS service to 
enable users to discover services deployed on YARN via standard DNS lookup
- YARN-6419[4]. UI support for native-services on the new YARN UI
All these new services are optional and are sitting outside of the existing 
system, and have no impact on existing system if disabled.

Special thanks to a team of folks who worked hard towards this: Billie Rinaldi, 
Gour Saha, Vinod Kumar Vavilapalli, Jonathan Maron, Rohith Sharma K S, Sunil G, 
Akhil PB, Eric Yang. This effort could not be possible without their ideas and 
hard work.
Also thanks Allen for some review and verifications.

Thanks,
Jian

[1] https://issues.apache.org/jira/browse/YARN-5079
[2] https://issues.apache.org/jira/browse/YARN-4793
[3] https://issues.apache.org/jira/browse/YARN-4757
[4] https://issues.apache.org/jira/browse/YARN-6419

[VOTE] Merge yarn-native-services branch into trunk

[Jian He]

Hi All,

I would like to restart the vote for merging yarn-native-services to trunk.
Since last vote, we have been working on several issues in documentation, DNS, 
CLI modifications etc. We believe now the feature is in a much better shape.

Some back ground:
At a high level, the following are the key feautres implemented.
- YARN-5079[1]. A native YARN framework (ApplicationMaster) to orchestrate 
existing services to YARN either docker or non-docker based.
- YARN-4793[2]. A Rest API service embeded in RM (optional)  for user to deploy 
a service via a simple JSON spec
- YARN-4757[3]. Extending today's service registry with a simple DNS service to 
enable users to discover services deployed on YARN via standard DNS lookup
- YARN-6419[4]. UI support for native-services on the new YARN UI
All these new services are optional and are sitting outside of the existing 
system, and have no impact on existing system if disabled.

Special thanks to a team of folks who worked hard towards this: Billie Rinaldi, 
Gour Saha, Vinod Kumar Vavilapalli, Jonathan Maron, Rohith Sharma K S, Sunil G, 
Akhil PB, Eric Yang. This effort could not be possible without their ideas and 
hard work.
Also thanks Allen for some review and verifications.

Thanks,
Jian

[1] https://issues.apache.org/jira/browse/YARN-5079
[2] https://issues.apache.org/jira/browse/YARN-4793
[3] https://issues.apache.org/jira/browse/YARN-4757
[4] https://issues.apache.org/jira/browse/YARN-6419

Re: YARN native services Re: 2017-10-06 Hadoop 3 release status update

[Jian He]

Allen,

I was under the impression (and, maybe this was my misunderstanding. if so, 
sorry) that “the goal” for this first pass was to integrate the existing Apache 
Slider functionality into YARN.  As it stands, I don’t think those goals have 
been met.  It doesn’t seem to be much different than just writing a shell 
profile to call slider directly:
The goal of this feature is to support container-based services on YARN. The 
team started with merging slider but built many new stuff like the REST 
service, the DNS which don’t exist in slider and also rewrote a bunch of stuff 
in the core.
This thread was supposed for release update. Let’s move the feature discussion 
to the jira YARN-7127.

Thanks,
Jian


On Oct 9, 2017, at 5:51 PM, Allen Wittenauer 
> wrote:


On Oct 6, 2017, at 5:51 PM, Eric Yang 
> wrote:
yarn application -deploy –f spec.json
yarn application -stop 
yarn application -restart 
yarn application -remove 

and

yarn application –list will display both application list from RM as well as 
docker services?

IMO, that makes much more sense. [*] I’m trying think of a reason why I’d care 
if something was using this API or not.  It’s not like users can’t run whatever 
they want as part of their job now.  The break out is really only necessary so 
I have an idea if something is running that is using the REST API daemon. But 
more on that later….

I think the development team was concerned that command structure overload 
between batch applications and long running services.  In my view, there is no 
difference, they are all applications.  The only distinction is the launching 
and shutdown of services may be different from batch jobs.  I think user can 
get used to these command structures without creating additional command 
grouping.

I pretty much agree.  In fact, I’d love to see ‘yarn application’ even replace 
‘yarn jar’. One Interface To Rule Them All.

I was under the impression (and, maybe this was my misunderstanding. if so, 
sorry) that “the goal” for this first pass was to integrate the existing Apache 
Slider functionality into YARN.  As it stands, I don’t think those goals have 
been met.  It doesn’t seem to be much different than just writing a shell 
profile to call slider directly:

---
function yarn_subcommand_service
{
  exec slider “$@“
}


(or whatever). Plus doing it this way, one gets the added benefit of the 
SIGNIFICANTLY better documentation. (Seriously: well done that team)

From an outside perspective, the extra daemon for running the REST API seems 
like when it should have clicked that the project is going off the rails and 
missing the whole “integration” aspect. Integrating the REST API into the RM 
from day one and the command separation would have also stuck out. If the RM 
runs the REST API, it now becomes a problem of “how does a user launch more 
than just a jar easily?” A problem that Hadoop has had since nearly day one.  
Redefining the “application” subcommand sounds like a reasonable way to move 
forward on that problem while also dropping the generic sounding "service" 
subcommand.

But all that said, it feels like direct integration was avoided from the 
beginning and I’m unclear as to why. Take this line from the quick start 
documentation:

"Start all the hadoop components HDFS, YARN as usual.”

a) This sentence is pretty much a declaration that this feature set isn’t part 
of “YARN”.
b) Minimally, this should link to ClusterSetup.

Anyway, yes, please work on removing all of these extra adoption barriers and 
increased workload on admin teams with Yet Another Daemon to monitor and 
collect metrics.

Thanks!

[*] - I’m reminded of a conversation I had with a PMC member year or three ago 
about HDFS. They proudly almost defiantly stated that the HDFS command 
structure is such because it resembles the protocols and that was great. Guess 
what: users’ don’t care about how something is implemented, much less the 
protocols that are used to drive it. They care about consistency, EOU, and all 
those feel good things that make applications a joy to use. They have more 
important stuff to do. Copying the protocols onto the command line only help 
the person who wrote it and no one else. It’s hard not to walk away from 
playing with YARN in this branch as exhibiting those same anti-user behaviors.

Re: 2017-10-06 Hadoop 3 release status update

[Jian He]

Hi Allen,

Thanks for spending the time reviewing it.
A new patch was uploaded yesterday on YARN-7198 to address the documentation of 
missing config, you might want to check.
The api-server is basically a REST server which accepts user requests to deploy 
services, it now has an option to be run as part of RM, which eliminates one 
separate daemon.

We are open to naming suggestions. So far we used ‘service’ keyword to indicate 
this feature. E.g. 
"yarn service” sub-command is used to manage services deployed on YARN such as:

yarn service create -f service-spec.json
yarn service stop 

Jian

> On Oct 6, 2017, at 3:12 PM, Allen Wittenauer  
> wrote:
> 
> 
>> On Oct 6, 2017, at 1:31 PM, Andrew Wang  wrote:
>> 
>>  - Still waiting on Allen to review YARN native services feature.
> 
>   Fake news.  
> 
>   I’m still -1 on it, at least prior to a patch that posted late 
> yesterday. I’ll probably have a chance to play with it early next week.
> 
> 
> Key problems:
> 
>   * still haven’t been able to bring up dns daemon due to lacking 
> documentation
> 
>   * it really needs better naming and command structures.  When put into 
> the larger YARN context, it’s very problematic:
> 
> $ yarn —daemon start resourcemanager
> 
>   vs.
> 
> $ yarn —daemon start apiserver 
> 
>   if you awoke from a deep sleep from inside a cave, which one 
> would you expect to “start YARN”? Made worse that the feature is called 
> “YARN services” all over the place.
> 
> $ yarn service foo
> 
>   … what does this even mean?
> 
>   It would be great if other outsiders really looked hard at this branch 
> to give the team feedback.   Once it gets released, it’s gonna be too late to 
> change it….
> 
> As a sidenote:
> 
>   It’d be great if the folks working on YARN spent some time 
> consolidating daemons.  With this branch, it now feels like we’re approaching 
> the double digit area of daemons to turn on all the features.  It’s well past 
> ridiculous, especially considering we still haven’t replaced the MRJHS’s 
> feature set to the point we can turn it off.
> 
> 
> -
> To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
> 

Re: [VOTE] Merge yarn-native-services branch into trunk

[Jian He]

Hi Allen,

Thanks for sharing the feedback. I opened YARN-7191 for addressing the feedback.
We can move the discussions there. 

Thanks,
Jian

> On Sep 13, 2017, at 10:10 AM, Allen Wittenauer <a...@effectivemachines.com> 
> wrote:
> 
> 
>> On Sep 8, 2017, at 9:25 AM, Jian He <j...@hortonworks.com> wrote:
>> 
>> Hi Allen,
>> The documentations are committed. Please check QuickStart.md and others in 
>> the same folder.
>> YarnCommands.md doc is updated to include new commands.
>> DNS default port is also documented. 
>> Would you like to give a look and see if it address your concerns ?
> 
>   Somewhat. Greatly improved, but there’s still way too much “we’re 
> working on this” and “here’s a link to a JIRA” and just general brokenness 
> going on.
> 
>   Here’s some examples from concepts.  Concepts!  The document I’d expect 
> to give me very basic “when we talk about X, we mean Y” definitions:
> 
> "A host of scheduling features are being developed to support long running 
> services.”
> 
>   Yeah, ok?  How is this a concept?
> 
>  or
> 
>   "[YARN-3998](https://issues.apache.org/jira/browse/YARN-3998) 
> implements a retry-policy to let NM re-launch a service container when it 
> fails.”
> 
> 
>   The patch itself went through nine revisions and a long discussion. 
> Would an end user care about the details in that JIRA?  
> 
>   If the answer to the last question is YES, then the documentation has 
> failed.  The whole point of documentation is so they don’t have to go digging 
> into the details of the implementation, the decision process that got us 
> there, etc.  If they care enough about the details, they’ll run through the 
> changelog and click on the JIRA link there.  If the summary line of the 
> changelog isn’t obvious, well… then we need better summaries.
> 
>   etc, etc.
> 
> ...
> 
>   The sleep example is nice.  Now, let’s see a non-toy example:  multiple 
> instances of Apache httpd or MariaDB or something real and not from the 
> Hadoop echo chamber (e.g., non-JVM-based).  If this is for “native” services, 
> this shouldn’t be a problem, right?  Give a real example and users will buy 
> what you’re selling.  I also think writing the docs and providing an example 
> of doing something big and outside the team’s comfort zone will clarify where 
> end users are going to need more help than what’s being provided.  Getting a 
> MariaDB instance or three up will help tremendously here.
> 
>   Which reminds me: something the documentation doesn’t cover is storage. 
> What happens to it, where does it come from, etc, etc.  That’s an important 
> detail that I didn’t see covered.  (I may have missed it.)  
> 
> …
> 
>   Why are there directions to enable other, partially unrelated services 
> in here?  Shouldn’t there be pointers to their specific documentation?  Is 
> the expectation that if the requirements for those other services change that 
> contributors will need to update multiple documents?
> 
> "Start the DNS server”
> 
>   Just… yikes.
> 
>   a) yarn classname … This is not how we do user-facing things. 
> The fact it’s not really possible for a *daemon* to be put in the 
> YarnCommands.md doc should be a giant red flag that something isn’t going 
> correctly here.
>   b) no jsvc support for something that it’s strongly hinted at 
> wanting to run privileged = an instant -1 for failing basic security 
> practices.  There’s zero reason for it to be running continually as root.
>   c) If this would have been hooked into the shell scripts 
> appropriately, logs, user switching, etc would have been had for free.
>   d) Where’s stop?  Right. Since it’s outside the scripts, there 
> is no pid support so one has to do all of that manually….
> 
> 
> Given:
> 
>"3. Supports reverse lookups (name based on IP). Note, this works only 
> for Docker containers.”
> 
> then:
> 
>   "It should not be used as a fully-functional corporate DNS.”
> 
> Scratch corporate.  It’s not a fully functional DNS server if it can’t do 
> reverse lookups.  (Which, ironically, means it’s not suitable for use with 
> Apache Hadoop, given it requires both fwd and rev DNS ...)
> 
> 

Re: [DISCUSS] Merge yarn-native-services branch into trunk

[Jian He]

Hi Arun

Sorry for late reply.
* Is there a branch-2 merge planned for this ?
Branch-2 is not planned for this merge.

* I understand YARN-7126 has some introductory documentation, But I think we 
need to flesh it up a bit more before release, I would also like to see steps 
to deploy a sample service.
We have added more documentations, QuickStart.md, Overview.md and others in the 
same folder.
YarnCommands.md is also updated to document the new shell commands.

I encourage everyone to try  and share suggestions.

As said in another email thread, we decided to drop this for beta and re-target 
it for GA.

Thanks,
Jian

On Sep 5, 2017, at 1:37 PM, Arun Suresh 
<asur...@apache.org<mailto:asur...@apache.org>> wrote:

Thanks for all the work on this folks.
I know the VOTE thread has started for this.

But I did have a couple of questions:
* Is there a branch-2 merge planned for this ?
* I understand YARN-7126 has some introductory documentation, But I think we 
need to flesh it up a bit more before release, I would also like to see steps 
to deploy a sample service.

Cheers
-Arun

On Thu, Aug 31, 2017 at 12:40 AM, Jian He 
<j...@hortonworks.com<mailto:j...@hortonworks.com>> wrote:
Update:
I’ve chatted with Andrew offline, we’ll proceed with merging 
yarn-native-services into trunk for beta.
We’ll advertise this feature as “alpha"
Currently, we have completed all the jiras for this merge - I’ve also moved out 
the subtasks that are not blocking this merge.

I’ve created YARN-7127 to run the entire patch against trunk, once that goes 
green, I plan to start a formal vote.

Thanks,
Jian

On Aug 18, 2017, at 2:48 PM, Andrew Wang 
<andrew.w...@cloudera.com<mailto:andrew.w...@cloudera.com><mailto:andrew.w...@cloudera.com<mailto:andrew.w...@cloudera.com>>>
 wrote:

Hi Jian, thanks for the reply,

On Thu, Aug 17, 2017 at 1:03 PM, Jian He 
<j...@hortonworks.com<mailto:j...@hortonworks.com><mailto:j...@hortonworks.com<mailto:j...@hortonworks.com>>>
 wrote:
Thanks Andrew for the comments. Answers below:

- There are no new APIs added in YARN/Hadoop core. In fact, all the new code 
are running outside of existing system and they are optional and require users 
to explicitly opt in. The new system’s own rest API is not stable and will be 
evolving.

Great! That adds a lot more confidence that this is safe to merge.

Are these new APIs listed in user documentation, and described as unstable?

- We have been running/testing a version of the entire system internally for 
quite a while.

Do you mind elaborating on the level of testing? Number of nodes, types of 
applications, production or test workload, etc. It'd help us build confidence.

- I’d like to see this in hadoop3-beta1. Of course, we’ll take responsibility 
of moving fast and not block the potential timeline.

Few more questions:

How should we advertise this feature in the release? Since the APIs are 
unstable, I'd propose calling it "alpha" in the release notes, like we do the 
TSv2.

Could you move out subtasks from YARN-5079 that are not blocking the merge? 
This would make it easier to understand what's remaining.

Thanks,
Andrew

Re: [VOTE] Merge yarn-native-services branch into trunk

[Jian He]

Hi Andrew,

At this point, there are no more release blockers including documentations from 
our side - all work done.
But I agree it is too close to the release, after talking with other team 
members, we are fine to drop  this from beta,

And we want to target this for GA.
I’m withdrawing this vote and will start afresh vote later for GA. 
Thanks all who voted this effort !

Thanks,
Jian


> On Sep 7, 2017, at 3:59 PM, Andrew Wang <andrew.w...@cloudera.com> wrote:
> 
> Hi folks,
> 
> This vote closes today. I see a -1 from Allen on inclusion in beta1. I see
> there's active fixing going on, but given that we're one week out from RC0,
> I think we should drop this from beta1.
> 
> Allen, Jian, others, is this reasonable? What release should we retarget
> this for? I don't have a sense for how much work there is left to do, but
> as a reminder, we're planning GA for Nov 1st, and 3.1.0 for January.
> 
> Best,
> Andrew
> 
> On Wed, Sep 6, 2017 at 10:19 AM, Jian He <j...@hortonworks.com> wrote:
> 
>>>  Please correct me if I’m wrong, but the current summary of the
>> branch, post these changes, looks like:
>> Sorry for confusion, I was actively writing the formal documentation for
>> how to use/how it works etc. and will post soon in a few hours.
>> 
>> 
>>> On Sep 6, 2017, at 10:15 AM, Allen Wittenauer <a...@effectivemachines.com>
>> wrote:
>>> 
>>> 
>>>> On Sep 5, 2017, at 6:23 PM, Jian He <j...@hortonworks.com> wrote:
>>>> 
>>>>>If it doesn’t have all the bells and whistles, then it shouldn’t
>> be on port 53 by default.
>>>> Sure, I’ll change the default port to not use 53 and document it.
>>>>>*how* is it getting launched on a privileged port? It sounds like
>> the expectation is to run “command” as root.   *ALL* of the previous
>> daemons in Hadoop that needed a privileged port used jsvc.  Why isn’t this
>> one? These questions matter from a security standpoint.
>>>> Yes, it is running as “root” to be able to use the privileged port. The
>> DNS server is not yet integrated with the hadoop script.
>>>> 
>>>>> Check the output.  It’s pretty obviously borked:
>>>> Thanks for pointing out. Missed this when rebasing onto trunk.
>>> 
>>> 
>>>  Please correct me if I’m wrong, but the current summary of the
>> branch, post these changes, looks like:
>>> 
>>>  * A bunch of mostly new Java code that may or may not have
>> javadocs (post-revert YARN-6877, still working out HADOOP-14835)
>>>  * ~1/3 of the docs are roadmap/TBD
>>>  * ~1/3 of the docs are for an optional DNS daemon that has
>> no end user hook to start it
>>>  * ~1/3 of the docs are for a REST API that comes from some
>> undefined daemon (apiserver?)
>>>  * Two new, but undocumented, subcommands to yarn
>>>  * There are no docs for admins or users on how to actually
>> start or use this completely new/separate/optional feature
>>> 
>>>  How are outside people (e.g., non-branch committers) supposed to
>> test this new feature under these conditions?
>>> 
>> 
>> 
>> -
>> To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
>> For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
>> 
>> 

Re: [VOTE] Merge yarn-native-services branch into trunk

[Jian He]

Hi Allen,
The documentations are committed. Please check QuickStart.md and others in the 
same folder.
YarnCommands.md doc is updated to include new commands.
DNS default port is also documented. 
Would you like to give a look and see if it address your concerns ?

Jian

> On Sep 6, 2017, at 10:19 AM, Jian He <j...@hortonworks.com> wrote:
> 
>>  Please correct me if I’m wrong, but the current summary of the branch, 
>> post these changes, looks like:
> Sorry for confusion, I was actively writing the formal documentation for how 
> to use/how it works etc. and will post soon in a few hours.
> 
> 
>> On Sep 6, 2017, at 10:15 AM, Allen Wittenauer <a...@effectivemachines.com> 
>> wrote:
>> 
>> 
>>> On Sep 5, 2017, at 6:23 PM, Jian He <j...@hortonworks.com> wrote:
>>> 
>>>>If it doesn’t have all the bells and whistles, then it shouldn’t be on 
>>>> port 53 by default.
>>> Sure, I’ll change the default port to not use 53 and document it.
>>>>*how* is it getting launched on a privileged port? It sounds like the 
>>>> expectation is to run “command” as root.   *ALL* of the previous daemons 
>>>> in Hadoop that needed a privileged port used jsvc.  Why isn’t this one? 
>>>> These questions matter from a security standpoint.  
>>> Yes, it is running as “root” to be able to use the privileged port. The DNS 
>>> server is not yet integrated with the hadoop script. 
>>> 
>>>> Check the output.  It’s pretty obviously borked:
>>> Thanks for pointing out. Missed this when rebasing onto trunk.
>> 
>> 
>>  Please correct me if I’m wrong, but the current summary of the branch, 
>> post these changes, looks like:
>> 
>>  * A bunch of mostly new Java code that may or may not have 
>> javadocs (post-revert YARN-6877, still working out HADOOP-14835)
>>  * ~1/3 of the docs are roadmap/TBD
>>  * ~1/3 of the docs are for an optional DNS daemon that has no 
>> end user hook to start it
>>  * ~1/3 of the docs are for a REST API that comes from some 
>> undefined daemon (apiserver?)
>>  * Two new, but undocumented, subcommands to yarn
>>  * There are no docs for admins or users on how to actually 
>> start or use this completely new/separate/optional feature
>> 
>>  How are outside people (e.g., non-branch committers) supposed to test 
>> this new feature under these conditions?
>> 
> 

Re: [VOTE] Merge yarn-native-services branch into trunk

[Jian He]

>   Please correct me if I’m wrong, but the current summary of the branch, 
> post these changes, looks like:
Sorry for confusion, I was actively writing the formal documentation for how to 
use/how it works etc. and will post soon in a few hours.


> On Sep 6, 2017, at 10:15 AM, Allen Wittenauer <a...@effectivemachines.com> 
> wrote:
> 
> 
>> On Sep 5, 2017, at 6:23 PM, Jian He <j...@hortonworks.com> wrote:
>> 
>>> If it doesn’t have all the bells and whistles, then it shouldn’t be on 
>>> port 53 by default.
>> Sure, I’ll change the default port to not use 53 and document it.
>>> *how* is it getting launched on a privileged port? It sounds like the 
>>> expectation is to run “command” as root.   *ALL* of the previous daemons in 
>>> Hadoop that needed a privileged port used jsvc.  Why isn’t this one? These 
>>> questions matter from a security standpoint.  
>> Yes, it is running as “root” to be able to use the privileged port. The DNS 
>> server is not yet integrated with the hadoop script. 
>> 
>>> Check the output.  It’s pretty obviously borked:
>> Thanks for pointing out. Missed this when rebasing onto trunk.
> 
> 
>   Please correct me if I’m wrong, but the current summary of the branch, 
> post these changes, looks like:
> 
>   * A bunch of mostly new Java code that may or may not have 
> javadocs (post-revert YARN-6877, still working out HADOOP-14835)
>   * ~1/3 of the docs are roadmap/TBD
>   * ~1/3 of the docs are for an optional DNS daemon that has no 
> end user hook to start it
>   * ~1/3 of the docs are for a REST API that comes from some 
> undefined daemon (apiserver?)
>   * Two new, but undocumented, subcommands to yarn
>   * There are no docs for admins or users on how to actually 
> start or use this completely new/separate/optional feature
> 
>   How are outside people (e.g., non-branch committers) supposed to test 
> this new feature under these conditions?
> 


-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [VOTE] Merge yarn-native-services branch into trunk

[Jian He]

>   If it doesn’t have all the bells and whistles, then it shouldn’t be on 
> port 53 by default.
Sure, I’ll change the default port to not use 53 and document it.
>   *how* is it getting launched on a privileged port? It sounds like the 
> expectation is to run “command” as root.   *ALL* of the previous daemons in 
> Hadoop that needed a privileged port used jsvc.  Why isn’t this one? These 
> questions matter from a security standpoint.  
Yes, it is running as “root” to be able to use the privileged port. The DNS 
server is not yet integrated with the hadoop script. 

> Check the output.  It’s pretty obviously borked:
Thanks for pointing out. Missed this when rebasing onto trunk.

> On Sep 5, 2017, at 3:11 PM, Allen Wittenauer <a...@effectivemachines.com> 
> wrote:
> 
> 
>> On Sep 5, 2017, at 2:53 PM, Jian He <j...@hortonworks.com> wrote:
>> 
>>> Based on the documentation, this doesn’t appear to be a fully function DNS 
>>> server as an admin would expect (e.g., BIND, Knot, whatever).  Where’s 
>>> forwarding? How do I setup notify? Are secondaries even supported? etc, etc.
>> 
>> It seems like this is a rehash of some of the discussion you and others had 
>> on the JIRA. The DNS here is a thin layer backed by service registry. My 
>> understanding from the JIRA is that there are no claims that this is already 
>> a DNS with all the bells and whistles - its goal is mainly to expose dynamic 
>> services running on YARN as end-points. Clearly, this is an optional daemon, 
>> if the provided feature set is deemed insufficient, an alternative solution 
>> can be plugged in by specific admins because the DNS piece is completely 
>> decoupled from the rest of native-services. 
> 
>   If it doesn’t have all the bells and whistles, then it shouldn’t be on 
> port 53 by default. It should also be documented that one *can’t* do these 
> things.  If the standard config is likely to be a “real” server on port 53 
> either acting as a secondary to the YARN one or at least able to forward 
> queries to it, then these need to get documented.  As it stands, operations 
> folks are going to be taken completely by surprise by some relatively random 
> process sitting on a very well established port.
> 
>>> In fact:  was this even tested on port 53? How does this get launched such 
>>> that it even has access to open port 53?  I don’t see any calls to use the 
>>> secure daemon code in the shell scripts. Is there any jsvc voodoo or is it 
>>> just “run X as root”?
>> 
>> Yes, we have tested this DNS server on port 53 on a cluster by running the 
>> DNS server as root user. The port is clearly configurable, so the admin has 
>> two options. Run as root + port 53. Run as non-root + non-privileged port. 
>> We tested and left it as port 53 to keep it on a standard DNS port. It is 
>> already documented as such though I can see that part can be improved a 
>> little.
> 
>   *how* is it getting launched on a privileged port? It sounds like the 
> expectation is to run “command” as root.   *ALL* of the previous daemons in 
> Hadoop that needed a privileged port used jsvc.  Why isn’t this one? These 
> questions matter from a security standpoint.  
> 
>>> 4) Post-merge, yarn usage information is broken.  This is especially 
>>> bad since it doesn’t appear that YarnCommands was ever updated to include 
>>> the new sub-commands.
>> 
>> The “yarn” usage command is working for me. what do you mean ? 
> 
> Check the output.  It’s pretty obviously borked:
> 
> ===snip
> 
>Daemon Commands:
> 
> nodemanager  run a nodemanager on each worker
> proxyserver  run the web app proxy server
> resourcemanager  run the ResourceManager
> router   run the Router daemon
> timelineserver   run the timeline server
> 
>Run a service Commands:
> 
> service  run a service
> 
>Run yarn-native-service rest server Commands:
> 
> apiserverrun yarn-native-service rest server
> 
> 
> ===snip===
> 
>> Yeah, looks like some previous features also forgot to update 
>> YarnCommands.md for the new sub commands 
> 
>   Likely.  But I was actually interested in playing with this one to 
> compare it to the competition.  [Lucky you. ;) ]  But with pretty much zero 
> documentation….
> 
> 

Re: [VOTE] Merge yarn-native-services branch into trunk

[Jian He]

> 1) Did I miss it or is there no actual end-user documentation on how to use 
> this? 

Yes, we are in the process of finishing up the the doc and posting it. We 
considered this a release blocker for 3.0.0-beta1 and so working on it in 
parallel while the branch merge happens.

>   2) Lots of markdown problems in the NativeServicesDiscovery.md 
> document.  This includes things like ‘yarnsite.xml’ (missing a dash.)  Also, 
> I’m also confused why it’s called that when the title is YARN DNS, but 
> whatever.


Thanks for pointing out. We will fix this.

>  Based on the documentation, this doesn’t appear to be a fully function DNS 
> server as an admin would expect (e.g., BIND, Knot, whatever).  Where’s 
> forwarding? How do I setup notify? Are secondaries even supported? etc, etc.

It seems like this is a rehash of some of the discussion you and others had on 
the JIRA. The DNS here is a thin layer backed by service registry. My 
understanding from the JIRA is that there are no claims that this is already a 
DNS with all the bells and whistles - its goal is mainly to expose dynamic 
services running on YARN as end-points. Clearly, this is an optional daemon, if 
the provided feature set is deemed insufficient, an alternative solution can be 
plugged in by specific admins because the DNS piece is completely decoupled 
from the rest of native-services. 

> In fact:  was this even tested on port 53? How does this get launched such 
> that it even has access to open port 53?  I don’t see any calls to use the 
> secure daemon code in the shell scripts. Is there any jsvc voodoo or is it 
> just “run X as root”?

Yes, we have tested this DNS server on port 53 on a cluster by running the DNS 
server as root user. The port is clearly configurable, so the admin has two 
options. Run as root + port 53. Run as non-root + non-privileged port. We 
tested and left it as port 53 to keep it on a standard DNS port. It is already 
documented as such though I can see that part can be improved a little.

>   4) Post-merge, yarn usage information is broken.  This is especially 
> bad since it doesn’t appear that YarnCommands was ever updated to include the 
> new sub-commands.

The “yarn” usage command is working for me. what do you mean ? 
Yeah, looks like some previous features also forgot to update YarnCommands.md 
for the new sub commands 

Re: [DISCUSS] Merge yarn-native-services branch into trunk

[Jian He]

Update:
I’ve chatted with Andrew offline, we’ll proceed with merging 
yarn-native-services into trunk for beta.
We’ll advertise this feature as “alpha"
Currently, we have completed all the jiras for this merge - I’ve also moved out 
the subtasks that are not blocking this merge.

I’ve created YARN-7127 to run the entire patch against trunk, once that goes 
green, I plan to start a formal vote.

Thanks,
Jian

On Aug 18, 2017, at 2:48 PM, Andrew Wang 
<andrew.w...@cloudera.com<mailto:andrew.w...@cloudera.com>> wrote:

Hi Jian, thanks for the reply,

On Thu, Aug 17, 2017 at 1:03 PM, Jian He 
<j...@hortonworks.com<mailto:j...@hortonworks.com>> wrote:
Thanks Andrew for the comments. Answers below:

- There are no new APIs added in YARN/Hadoop core. In fact, all the new code 
are running outside of existing system and they are optional and require users 
to explicitly opt in. The new system’s own rest API is not stable and will be 
evolving.

Great! That adds a lot more confidence that this is safe to merge.

Are these new APIs listed in user documentation, and described as unstable?

- We have been running/testing a version of the entire system internally for 
quite a while.

Do you mind elaborating on the level of testing? Number of nodes, types of 
applications, production or test workload, etc. It'd help us build confidence.

- I’d like to see this in hadoop3-beta1. Of course, we’ll take responsibility 
of moving fast and not block the potential timeline.

Few more questions:

How should we advertise this feature in the release? Since the APIs are 
unstable, I'd propose calling it "alpha" in the release notes, like we do the 
TSv2.

Could you move out subtasks from YARN-5079 that are not blocking the merge? 
This would make it easier to understand what's remaining.

Thanks,
Andrew

Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk

[Jian He]

+1 (binding)

I’ve participated in the review of ATSv2 security related patch.

JIan

> On Aug 28, 2017, at 1:39 AM, Rohith Sharma K S <rohithsharm...@apache.org> 
> wrote:
> 
> +1 (binding)
> 
> Thank you very much for the great team work!
> 
> Built from source and deployed in secured cluster. The below are the test
> result.
> 
> Deployment :
>Standard hadoop security deployment authentication and authorization as
> well.
>Branch-2 Hadoop and Hbase security cluster.
>Branch-3 Hadoop security cluster. HBase client is pointing to Branch-2
> hbase cluster.
>All security configurations are set in-place.
>Each service is running with its own user. Say, HDFS is running with
> hdfs, YARN user is running with yarn, Hbase is running with hbase
>Smoke test user : test-user
> 
> Test Cases :
> 
> Authentication :
> Verify for all daemons start up successful : OK
> Run a MR job using test-user : OK
> Verify for REST API’s with in the scope of application : OK
> Verify for REST API’s newly added I.e outside scope of application : OK.
> RM Restart/ NM restart / RM_work-preserving restart has executed and
> verified for data : OK. (Entity validation is done, but not entity data
> validation!
> Token redistribution to AM, NM is verified.
> 
> Authorization :
> 1 . Basic whitelisting of users to read has been validated. Works as
> expected!
> 
> Disabling TSv2 configuration is also being tested.
> 
> 
> Thanks & Regards
> Rohith Sharma K S
> 
> On 22 August 2017 at 12:02, Vrushali Channapattan <vrushalic2...@gmail.com>
> wrote:
> 
>> Hi folks,
>> 
>> Per earlier discussion [1], I'd like to start a formal vote to merge
>> feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The vote will
>> run for 7 days, and will end August 29 11:00 PM PDT.
>> 
>> We have previously completed one merge onto trunk [3] and Timeline Service
>> v2 has been part of Hadoop release 3.0.0-alpha1.
>> 
>> Since then, we have been working on extending the capabilities of Timeline
>> Service v2 in a feature branch [2] for a while, and we are reasonably
>> confident that the state of the feature meets the criteria to be merged
>> onto trunk and we'd love folks to get their hands on it in a test capacity
>> and provide valuable feedback so that we can make it production-ready.
>> 
>> In a nutshell, Timeline Service v.2 delivers significant scalability and
>> usability improvements based on a new architecture. What we would like to
>> merge to trunk is termed "alpha 2" (milestone 2). The feature has a
>> complete end-to-end read/write flow with security and read level
>> authorization via whitelists. You should be able to start setting it up and
>> testing it.
>> 
>> At a high level, the following are the key features that have been
>> implemented since alpha1:
>> - Security via Kerberos Authentication and delegation tokens
>> - Read side simple authorization via whitelist
>> - Client configurable entity sort ordering
>> - Richer REST APIs for apps, app attempts, containers, fetching metrics by
>> timerange, pagination, sub-app entities
>> - Support for storing sub-application entities (entities that exist outside
>> the scope of an application)
>> - Configurable TTLs (time-to-live) for tables, configurable table prefixes,
>> configurable hbase cluster
>> - Flow level aggregations done as dynamic (table level) coprocessors
>> - Uses latest stable HBase release 1.2.6
>> 
>> There are a total of 82 subtasks that were completed as part of this
>> effort.
>> 
>> We paid close attention to ensure that once disabled Timeline Service v.2
>> does not impact existing functionality when disabled (by default).
>> 
>> Special thanks to a team of folks who worked hard and contributed towards
>> this effort with patches, reviews and guidance: Rohith Sharma K S, Varun
>> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
>> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
>> 
>> Regards,
>> Vrushali
>> 
>> [1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg27383.html
>> [2] https://issues.apache.org/jira/browse/YARN-5355
>> [3] https://issues.apache.org/jira/browse/YARN-2928
>> [4] https://github.com/apache/hadoop/commits/YARN-5355
>> 


-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [DISCUSS] Merge yarn-native-services branch into trunk

[Jian He]

Thanks Andrew for the comments. Answers below:

- There are no new APIs added in YARN/Hadoop core. In fact, all the new code 
are running outside of existing system and they are optional and require users 
to explicitly opt in. The new system’s own rest API is not stable and will be 
evolving.
- We have been running/testing a version of the entire system internally for 
quite a while.
- I’d like to see this in hadoop3-beta1. Of course, we’ll take responsibility 
of moving fast and not block the potential timeline.

Jian

On Aug 16, 2017, at 10:29 AM, Andrew Wang 
<andrew.w...@cloudera.com<mailto:andrew.w...@cloudera.com>> wrote:

Hi Jian,

Hadoop 3.0.0-beta1 is planned for mid-September. If the plan is to merge in 
hopefully the next two weeks, that's very, very close to the goal release date. 
We've already got a pile of blockers and criticals to resolve before then.

Could you comment on testing and API stability for this branch? YARN Federation 
was run at high scale and did not add new APIs, which provided a lot of 
confidence in the merge.

I'll also raise the option of cutting branch-3 or branch-3.0 for the 3.0.0 
efforts, and targeting this for 3.1.0.

Best,
Andrew

On Tue, Aug 15, 2017 at 1:56 PM, Jian He 
<j...@hortonworks.com<mailto:j...@hortonworks.com>> wrote:
Hi All,
I would like to bring up the discussion of merging yarn-native-services branch 
into trunk in a few weeks. There are a few issues left under YARN-5079 that are 
being actively worked upon. As soon as they are resolved, we plan on start a 
vote hopefully in next 2 weeks. The goal is to get this in for hadoop3 beta.

The major work in this branch include below umbrella jiras:
 - YARN-5079. A native YARN framework (ApplicationMaster) to migrate and 
orchestrate existing services to YARN either docker or non-docker based.
 - YARN-4793. A Rest API server for user to deploy a service via a simple JSON 
spec
 - YARN-4757. Extending today's service registry with a simple DNS service to 
enable users to discover services deployed on YARN
 - YARN-6419. UI support for native-services on the new YARN UI
All these new services are optional and have to be explicitly enabled.

Special thanks to a team of folks who worked hard towards this: Billie Rinaldi, 
Gour Saha, Vinod Kumar Vavilapalli, Jonathan Maron, Rohith Sharma K S, Sunil G, 
Akhil PB. This effort could not be possible without their ideas and hard work.

Please share your thoughts. Thanks.

Jian


-
To unsubscribe, e-mail: 
yarn-dev-unsubscr...@hadoop.apache.org<mailto:yarn-dev-unsubscr...@hadoop.apache.org>
For additional commands, e-mail: 
yarn-dev-h...@hadoop.apache.org<mailto:yarn-dev-h...@hadoop.apache.org>

[DISCUSS] Merge yarn-native-services branch into trunk

[Jian He]

Hi All,
I would like to bring up the discussion of merging yarn-native-services branch 
into trunk in a few weeks. There are a few issues left under YARN-5079 that are 
being actively worked upon. As soon as they are resolved, we plan on start a 
vote hopefully in next 2 weeks. The goal is to get this in for hadoop3 beta.

The major work in this branch include below umbrella jiras:
 - YARN-5079. A native YARN framework (ApplicationMaster) to migrate and 
orchestrate existing services to YARN either docker or non-docker based.
 - YARN-4793. A Rest API server for user to deploy a service via a simple JSON 
spec
 - YARN-4757. Extending today's service registry with a simple DNS service to 
enable users to discover services deployed on YARN
 - YARN-6419. UI support for native-services on the new YARN UI
All these new services are optional and have to be explicitly enabled. 

Special thanks to a team of folks who worked hard towards this: Billie Rinaldi, 
Gour Saha, Vinod Kumar Vavilapalli, Jonathan Maron, Rohith Sharma K S, Sunil G, 
Akhil PB. This effort could not be possible without their ideas and hard work.

Please share your thoughts. Thanks.

Jian


-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 2.8.0 (RC3)

[Jian He]

+1 (binding)

- built from source
- deployed a pseudo cluster
- ran basic example tests.
- Navigate the UI a bit, looks good.

Jian

> On Mar 22, 2017, at 9:03 PM, larry mccay  wrote:
> 
> +1 (non-binding)
> 
> - verified signatures
> - built from source and ran tests
> - deployed pseudo cluster
> - ran basic tests for hdfs, wordcount, credential provider API and related
> commands
> - tested webhdfs with knox
> 
> 
> On Wed, Mar 22, 2017 at 7:21 AM, Ravi Prakash  wrote:
> 
>> Thanks for all the effort Junping!
>> 
>> +1 (binding)
>> + Verified signature and MD5, SHA1, SHA256 checksum of tarball
>> + Verified SHA ID in git corresponds to RC3 tag
>> + Verified wordcount for one small text file produces same output as
>> hadoop-2.7.3.
>> + HDFS Namenode UI looks good.
>> 
>> I agree none of the issues reported so far are blockers. Looking forward to
>> another great release.
>> 
>> Thanks
>> Ravi
>> 
>> On Tue, Mar 21, 2017 at 8:10 PM, Junping Du  wrote:
>> 
>>> Thanks all for response with verification work and vote!
>>> 
>>> 
>>> Sounds like we are hitting several issues here, although none seems to be
>>> blockers so far. Given the large commit set - 2000+ commits first landed
>> in
>>> branch-2 release, we may should follow 2.7.0 practice that to claim this
>>> release is not for production cluster, just like Vinod's suggestion in
>>> previous email. We should quickly come up with 2.8.1 release in next 1
>> or 2
>>> month for production deployment.
>>> 
>>> 
>>> We will close the vote in next 24 hours. For people who haven't vote,
>>> please keep on verification work and report any issues if founded - I
>> will
>>> check if another round of RC is needed based on your findings. Thanks!
>>> 
>>> 
>>> Thanks,
>>> 
>>> 
>>> Junping
>>> 
>>> 
>>> 
>>> From: Kuhu Shukla 
>>> Sent: Tuesday, March 21, 2017 3:17 PM
>>> Cc: Junping Du; common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org
>> ;
>>> yarn-...@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
>>> Subject: Re: [VOTE] Release Apache Hadoop 2.8.0 (RC3)
>>> 
>>> 
>>> +1 (non-binding)
>>> 
>>> - Verified signatures.
>>> - Downloaded and built from source tar.gz.
>>> - Deployed a pseudo-distributed cluster on Mac Sierra.
>>> - Ran example Sleep job successfully.
>>> - Deployed latest Apache Tez 0.9 and ran sample Tez orderedwordcount
>>> successfully.
>>> 
>>> Thank you Junping and everyone else who worked on getting this release
>> out.
>>> 
>>> Warm Regards,
>>> Kuhu
>>> On Tuesday, March 21, 2017, 3:42:46 PM CDT, Eric Badger
>>>  wrote:
>>> +1 (non-binding)
>>> 
>>> - Verified checksums and signatures of all files
>>> - Built from source on MacOS Sierra via JDK 1.8.0 u65
>>> - Deployed single-node cluster
>>> - Successfully ran a few sample jobs
>>> 
>>> Thanks,
>>> 
>>> Eric
>>> 
>>> On Tuesday, March 21, 2017 2:56 PM, John Zhuge 
>>> wrote:
>>> 
>>> 
>>> 
>>> +1. Thanks for the great effort, Junping!
>>> 
>>> 
>>>  - Verified checksums and signatures of the tarballs
>>>  - Built source code with Java 1.8.0_66-b17 on Mac OS X 10.12.3
>>>  - Built source and native code with Java 1.8.0_111 on Centos 7.2.1511
>>>  - Cloud connectors:
>>>  - s3a: integration tests, basic fs commands
>>>  - adl: live unit tests, basic fs commands. See notes below.
>>>  - Deployed a pseudo cluster, passed the following sanity tests in
>>>  both insecure and SSL mode:
>>>  - HDFS: basic dfs, distcp, ACL commands
>>>  - KMS and HttpFS: basic tests
>>>  - MapReduce wordcount
>>>  - balancer start/stop
>>> 
>>> 
>>> Needs the following JIRAs to pass all ADL tests:
>>> 
>>>  - HADOOP-14205. No FileSystem for scheme: adl. Contributed by John
>> Zhuge.
>>>  - HDFS-11132. Allow AccessControlException in contract tests when
>>>  getFileStatus on subdirectory of existing files. Contributed by
>>> Vishwajeet
>>>  Dusane
>>>  - HADOOP-13928. TestAdlFileContextMainOperatio
>> nsLive.testGetFileContext1
>>>  runtime error. (John Zhuge via lei)
>>> 
>>> 
>>> On Mon, Mar 20, 2017 at 10:31 AM, John Zhuge 
>> wrote:
>>> 
 Yes, it only affects ADL. There is a workaround of adding these 2
 properties to core-site.xml:
 
 
   fs.adl.impl
   org.apache.hadoop.fs.adl.AdlFileSystem
 
 
 
   fs.AbstractFileSystem.adl.impl
   org.apache.hadoop.fs.adl.Adl
 
 
 I have the initial patch ready but hitting these live unit test
>> failures:
 
 Failed tests:
 
 TestAdlFileSystemContractLive.runTest:60->FileSystemContractBaseTest.
 testListStatus:257
 expected:<1> but was:<10>
 
 Tests in error:
 
 TestAdlFileContextMainOperationsLive>FileContextMainOperationsBaseT
>> est.
 testMkdirsFailsForSubdirectoryOfExistingFile:254
 » AccessControl
 
 

Re: [VOTE] Release Apache Hadoop 2.7.3 RC2

[Jian He]

+1

Built from source code.
Deployed single node cluster.
Successfully ran some example jobs.

Jian

> On Aug 18, 2016, at 10:05 AM, Vinod Kumar Vavilapalli  
> wrote:
> 
> Hi all,
> 
> I've created a new release candidate RC2 for Apache Hadoop 2.7.3.
> 
> As discussed before, this is the next maintenance release to follow up 2.7.2.
> 
> The RC is available for validation at: 
> http://home.apache.org/~vinodkv/hadoop-2.7.3-RC2/ 
> 
> 
> The RC tag in git is: release-2.7.3-RC2
> 
> The maven artifacts are available via repository.apache.org 
>  at 
> https://repository.apache.org/content/repositories/orgapachehadoop-1046 
> 
> 
> The release-notes are inside the tar-balls at location 
> hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I hosted 
> this at http://home.apache.org/~vinodkv/hadoop-2.7.3-RC2/releasenotes.html 
>  for your 
> quick perusal.
> 
> As you may have noted,
> - few issues with RC0 forced a RC1 [1]
> - few more issues with RC1 forced a RC2 [2]
> - a very long fix-cycle for the License & Notice issues (HADOOP-12893) caused 
> 2.7.3 (along with every other Hadoop release) to slip by quite a bit. This 
> release's related discussion thread is linked below: [3].
> 
> Please try the release and vote; the vote will run for the usual 5 days.
> 
> Thanks,
> Vinod
> 
> [1] [VOTE] Release Apache Hadoop 2.7.3 RC0: 
> https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106 
> 
> [2] [VOTE] Release Apache Hadoop 2.7.3 RC1: 
> https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg26336.html 
> 
> [3] 2.7.3 release plan: 
> https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html 
> 


-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 2.7.3 RC0

[Jian He]

+1 for the source tarball.

- Compiled and built from the source code
- Deployed a cluster
- Successfully ran some sample jobs.

Thanks,
Jian

> On Jul 27, 2016, at 10:11 AM, Akira Ajisaka  
> wrote:
> 
> +1 for the source tarball.
> 
> - Downloaded source tarball and binary tarball
> - Verified signatures and checksums
> - Compiled and built a single node cluster
> - Compiled Hive 2.1.0/1.2.1 and Tez 0.8.4/0.7.1 using Hadoop 2.7.3 pom 
> successfully
> - Ran some Hive on Tez queries successfully
> 
> Thanks,
> Akira
> 
> On 7/27/16 04:12, Vinod Kumar Vavilapalli wrote:
>> But, everyone please do continue your sanity checking on RC0 in case there 
>> are more issues to be fixed.
>> 
>> Thanks
>> +Vinod
>> 
>>> On Jul 26, 2016, at 12:11 PM, Vinod Kumar Vavilapalli  
>>> wrote:
>>> 
>>> Thanks Daniel and Wei.
>>> 
>>> I think these are worth fixing, I’m withdrawing this RC. Will look at 
>>> fixing these issues and roll a new candidate with the fixes as soon as 
>>> possible.
>>> 
>>> Thanks
>>> +Vinod
>>> 
 On Jul 26, 2016, at 11:05 AM, Wei-Chiu Chuang > wrote:
 
 I noticed two issues:
 
 (1) I ran hadoop checknative, but it seems the binary tarball was not 
 compiled with native library for Linux. On the contrary, the Hadoop built 
 from source tarball with maven -Pnative can find the native libraries on 
 the same host.
 
 (2) I noticed that the release dates in CHANGES.txt in tag 
 release-2.7.3-RC0 are set to Release 2.7.3 - 2016-07-27.
 However, the release dates in CHANGES.txt in the source and binary tar 
 balls are set to Release 2.7.3 - 2016-08-01. This is probably a non-issue 
 though.
 
 * Downloaded source and binary.
 * Verified signature.
 * Verified checksum.
 * Built from source using 64-bit Java 7 (1.7.0.75) and 8 (1.8.0.05). Both 
 went fine.
 * Ran hadoop checknative
 
 On Tue, Jul 26, 2016 at 9:12 AM, Rushabh Shah 
 > 
 wrote:
 Thanks Vinod for all the release work !
 +1 (non-binding).
 * Downloaded from source and built it.* Deployed a pseudo distributed 
 cluster.
 * Ran some sample jobs: sleep, pi* Ran some dfs commands.* Everything 
 works fine.
 
 
On Friday, July 22, 2016 9:16 PM, Vinod Kumar Vavilapalli 
 > wrote:
 
 
 Hi all,
 
 I've created a release candidate RC0 for Apache Hadoop 2.7.3.
 
 As discussed before, this is the next maintenance release to follow up 
 2.7.2.
 
 The RC is available for validation at: 
 http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/ 
  
 >
 
 The RC tag in git is: release-2.7.3-RC0
 
 The maven artifacts are available via repository.apache.org 
  > at 
 https://repository.apache.org/content/repositories/orgapachehadoop-1040/ 
  
 >
 
 The release-notes are inside the tar-balls at location 
 hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I 
 hosted this at 
 http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/releasenotes.html 
  
 > 
 for your quick perusal.
 
 As you may have noted, a very long fix-cycle for the License & Notice 
 issues (HADOOP-12893) caused 2.7.3 (along with every other Hadoop release) 
 to slip by quite a bit. This release's related discussion thread is linked 
 below: [1].
 
 Please try the release and vote; the vote will run for the usual 5 days.
 
 Thanks,
 Vinod
 
 [1]: 2.7.3 release plan: 
 https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html 
  
 >
 
 
 
>>> 
>> 
>> 
> 
> 
> -
> To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
> 
> 

Re: [Release thread] 2.8.0 release activities

[Jian He]

For MapReduce/YARN, I closed a few staled ones. Only 4 jiras needs attention 
for 2.8

MAPREDUCE-6288
YARN-1815
YARN-4685
YARN-4844

The rest are either improvements or long-standing issues and does not qualify 
release blocker, IMO.
I think we’ll try to get these 4 jiras in asap. The rest will be on best 
effort, resolve as much as possible and move them out if not resolved in time.

Jian

On May 11, 2016, at 5:37 PM, Wangda Tan 
> wrote:

Sounds good to me :).

Jian and I have looked at all existing 2.8.0 blockers and criticals today.
To me more than half of MR/YARN blockers/criticals of 2.8 should be moved
out. Left comments on these JIRAs asked original owners, plan to update
target version of these JIRAs early next week.

Will keep this thread updated.

Thanks,
Wangda


On Wed, May 11, 2016 at 5:06 PM, Sangjin Lee 
> wrote:

How about this? I'll review the HADOOP/HDFS bugs in that list to come up
with true blockers for 2.8.0 or JIRAs that are close to being ready. I'll
report the list here. Then folks can chime in if you agree

Perhaps Wangda, you can go over the YARN/MR bugs. Sound like a plan?

Thanks,
Sangjin

On Wed, May 11, 2016 at 4:26 PM, Wangda Tan 
> wrote:

+1, we should close such staled JIRAs to avoid doing unnecessary checks
for
every releases.

I'm working on reviewing YARN/MR critical/blocker patches currently, it
gonna very helpful if someone else can help with reviewing Common/HDFS
JIRAs.

Thanks,
Wangda


On Wed, May 11, 2016 at 4:20 PM, Sangjin Lee 
> wrote:

Where do we stand in terms of closing out blocker/critical issues for
2.8.0? I still see 50 open JIRAs in Vinod's list:
https://issues.apache.org/jira/issues/?filter=12334985

But I see a lot of JIRAs with no patches or very stale patches. It
would be
a good exercise to come up with the list of JIRAs that we need to block
2.8.0 for and focus our attention on closing them out. Thoughts?

Thanks,
Sangjin

On Sat, Apr 23, 2016 at 5:05 AM, Steve Loughran 


wrote:


On 23 Apr 2016, at 01:24, Vinod Kumar Vavilapalli <
vino...@apache.org>
wrote:

We are not converging - there’s still 58 more. I need help from the
community in addressing / review 2.8.0 blockers. If folks can start
with
reviewing Patch available tickets, that’ll be great.




I'm still doing the s3a stuff, other people testing and reviewing this
stuff welcome.

in particular, I could do with others playing with this patch of mine,
which adds counters and things into S3a, based on the azure
instrumentation

https://issues.apache.org/jira/browse/HADOOP-13028

[jira] [Resolved] (MAPREDUCE-6099) Adding getSplits(JobContext job, List stats) to mapreduce CombineFileInputFormat

[Jian He (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jian He resolved MAPREDUCE-6099.

Resolution: Won't Fix

Close as Jason mentioned 

> Adding  getSplits(JobContext job, List stats) to mapreduce 
> CombineFileInputFormat
> -
>
> Key: MAPREDUCE-6099
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6099
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: client
>Affects Versions: 2.4.1
>Reporter: Pankit Thapar
>Priority: Critical
> Attachments: MAPREDUCE-6099.patch
>
>
> Currently we have getSplits(JobContext job) in CombineFileInputFormat. 
> This api does not give freedom to the client to create a list if file status 
> it self and then create splits on the resultant List stats.
> The client might be able to perform some filtering on its end on the File 
> sets in the input paths. For the reasons, above it would be a good idea to 
> have getSplits(JobContext, List).
> Please let me know what you think about this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 2.7.2 RC2

[Jian He]

+1,
- built from source code
- deployed a single cluster
- run sample jobs which pass successfully.

Jian

> On Jan 22, 2016, at 6:52 AM, Sunil Govind  wrote:
> 
> +1 (Non Binding)
> 
> * Built tar ball from source and deployed
> * Verified few MR Jobs for various nodelabel and preemption cases.
> * Verified RM Web UI and REST queries. looks fine.
> 
> Thanks and Regards
> Sunil G
> 
> On Fri, Jan 15, 2016 at 10:27 AM Vinod Kumar Vavilapalli 
> wrote:
> 
>> Hi all,
>> 
>> I've created an updated release candidate RC2 for Apache Hadoop 2.7.2.
>> 
>> As discussed before, this is the next maintenance release to follow up
>> 2.7.1.
>> 
>> The RC is available for validation at:
>> http://people.apache.org/~vinodkv/hadoop-2.7.2-RC2/
>> 
>> The RC tag in git is: release-2.7.2-RC2
>> 
>> The maven artifacts are available via repository.apache.org <
>> http://repository.apache.org/> at
>> https://repository.apache.org/content/repositories/orgapachehadoop-1027 <
>> https://repository.apache.org/content/repositories/orgapachehadoop-1027>
>> 
>> The release-notes are inside the tar-balls at location
>> hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I
>> hosted this at
>> http://people.apache.org/~vinodkv/hadoop-2.7.2-RC2/releasenotes.html <
>> http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html> for
>> your quick perusal.
>> 
>> As you may have noted,
>> - I terminated the RC1 related voting thread after finding out that we
>> didn’t have a bunch of patches that are already in the released 2.6.3
>> version. After a brief discussion, we decided to keep the parallel 2.6.x
>> and 2.7.x releases incremental, see [4] for this discussion.
>> - The RC0 related voting thread got halted due to some critical issues.
>> It took a while again for getting all those blockers out of the way. See
>> the previous voting thread [3] for details.
>> - Before RC0, an unusually long 2.6.3 release caused 2.7.2 to slip by
>> quite a bit. This release's related discussion threads are linked below:
>> [1] and [2].
>> 
>> Please try the release and vote; the vote will run for the usual 5 days.
>> 
>> Thanks,
>> Vinod
>> 
>> [1]: 2.7.2 release plan: http://markmail.org/message/oozq3gvd4nhzsaes <
>> http://markmail.org/message/oozq3gvd4nhzsaes>
>> [2]: Planning Apache Hadoop 2.7.2
>> http://markmail.org/message/iktqss2qdeykgpqk <
>> http://markmail.org/message/iktqss2qdeykgpqk>
>> [3]: [VOTE] Release Apache Hadoop 2.7.2 RC0:
>> http://markmail.org/message/5txhvr2qdiqglrwc <
>> http://markmail.org/message/5txhvr2qdiqglrwc>
>> [4] Retracted [VOTE] Release Apache Hadoop 2.7.2 RC1:
>> http://markmail.org/thread/n7ljbsnquihn3wlw

Re: [VOTE] Release Apache Hadoop 2.6.3 RC0

[Jian He]

+1,

Build and deploy the cluster from source code.
Ran a few example jobs and passed successfully. 

Jian

> On Dec 16, 2015, at 2:19 PM, Vinod Kumar Vavilapalli  
> wrote:
> 
> +1 (binding) for the RC except for my question below about the tag.
> 
> I checked 2.6.3-RC0, based on my check-list:
> 
> - Signatures and message digests all are good in general.
> - The top level full LICENSE, NOTICE and README for the source artifacts are 
> good - CHANGES.txt for common, hdfs and mapped are correctly located.
> - Able to build the tars out of the source tar ball using JDK 7. (Don’t have 
> a JDK 6 at hand)
> 
> Testing: All testing on single node, unsecured, default mode.
> 
> - Started HDFS daemons successfully , created directories.
> - Successfully started YARN daemons - ResourceManager, NodeManager and 
> Timeline Service
> - Successfully started MapReduce history server.
> - Ran DistributedShell as a native YARN app.
> - Ran wordcount, pi, random writer, sort, grep and they all pass just fine.
> - Navigated through the RM, NM and Timeline UIs to make sure the views are 
> working well.
> - Navigated through the MapReduce UI to make sure the views are working well.
> 
> Thanks,
> +Vinod
> 
> 
>> On Dec 16, 2015, at 11:32 AM, Vinod Kumar Vavilapalli  
>> wrote:
>> 
>> So, the original voting mail mentions we are voting on release-2.6.3-RC0 tag.
>> 
>> Are we still doing that? What are the RC0.1 and RC1 tags doing then?
>> 
>> +Vinod
>> 
>>> On Dec 16, 2015, at 2:13 AM, Junping Du  wrote:
>>> 
>>> Thanks Akira for notice this. I don't think we can remove these tags as 
>>> they should be immutable as branches. I created these duplicated tags as 
>>> after I cut off RC0 out, some commit lands on 2.6.3 unexpected but I didn't 
>>> realize I could still push to original tag by forcefully. The best thing I 
>>> can do then is to make them point to the same commit as now it is.
>>> 
>>> Thanks,
>>> 
>>> Junping
>>> 
>>> From: Akira AJISAKA 
>>> Sent: Wednesday, December 16, 2015 6:41 AM
>>> To: common-...@hadoop.apache.org; yarn-...@hadoop.apache.org; 
>>> mapreduce-dev@hadoop.apache.org; hdfs-...@hadoop.apache.org
>>> Subject: Re: [VOTE] Release Apache Hadoop 2.6.3 RC0
>>> 
>>> Thanks Junping for starting release process.
>>> I noticed there are duplicated tags:
>>> 
>>> * release-2.6.3-RC0
>>> * release-2.6.3-RC0.1
>>> * release-2.6.3-RC1
>>> 
>>> Could you remove RC0.1 and RC1?
>>> 
>>> Regards,
>>> Akira
>>> 
>>> On 12/16/15 10:17, yliu wrote:
 Thanks Junping, +1.
 Download the tarball and deploy a small HDFS/YARN cluster, and verify few
 basic functionalities.
 
 Regards,
 Yi Liu
 
 On Wed, Dec 16, 2015 at 6:42 AM, Chang Li  wrote:
 
> Thanks Junping, + 1(non binding). Downloaded the tarball, compiled and
> built locally. Ran some MR jobs successfully.
> 
> Best,
> Chang
> 
> On Tue, Dec 15, 2015 at 3:17 PM, Wangda Tan  wrote:
> 
>> Thanks Junping,
>> 
>> +1 (binding). Deploy a cluster locally, run distributed shell and MR job,
>> both successfully finished.
>> 
>> Regards,
>> Wangda
>> 
>> 
>> On Tue, Dec 15, 2015 at 12:43 PM, Naganarasimha Garla <
>> naganarasimha...@gmail.com> wrote:
>> 
>>> Hi Junping,
>>> 
>>> +0 (non binding)
>>> 
>>> Though everything else is working  fine (downloaded the tar ball and
>>> installed single node cluster setup and  verified  few MR jobs),
>>> submission of Unmanaged AM is getting the RM down. YARN-4452 has
>>> already been raised and i am working on it. Will provide the patch for
>>> the trunk and the 2.6.3 version asap
>>> 
>>> Regards,
>>> 
>>> + Naga
>>> 
>>> 
>>> 
>>> Thanks for the work Junping! Downloaded the src tarball. Built locally
>>> and successfully ran
>>> in single node mode with a few map reduce jobs. LGTM.
>>> 
>>> Li Lu
>>> 
>>> On Dec 14, 2015, at 04:23, Junping Du
>>> >
>>> wrote:
>>> 
>>> ?Thanks Sarjeet and Tsuyoshi for reporting this. I just fix permission
>>> issue and download
>>> should work now. Please try to download it again. Thanks!
>>> 
>>> 
>>> Thanks,
>>> 
>>> 
>>> Junping
>>> 
>>> 
>>> 
>>> From: sarjeet singh >
>>> Sent: Sunday, December 13, 2015 6:44 PM
>>> To: common-...@hadoop.apache.org
>>> Cc: mapreduce-dev@hadoop.apache.org> mapreduce-dev@hadoop.apache.org
 ;
>>> hdfs-...@hadoop.apache.org;
>>> 

Re: Planning Apache Hadoop 2.7.2

[Jian He]

sorry, missed the cut.. 
I’d like to merge YARN-4127 into branch-2.7.2 also. 
That’s important because it prevents RM to startup, if the problem is hit.


> On Nov 3, 2015, at 3:24 PM, Vinod Vavilapalli  wrote:
> 
> I’m closing down 2.7.2 now, almost there.
> 
> Created a new branch-2.7.2 for the release so that any 2.7.3 patches can go 
> into branch-2.7 directly.
> 
> Please reach out to me if you have any questions.
> 
> Thanks
> +Vinod
> 
> On Nov 2, 2015, at 2:15 PM, Vinod Kumar Vavilapalli 
> > wrote:
> 
> We are down to 4 now 
> (https://issues.apache.org/jira/issues/?filter=12332867), appreciate help 
> moving forward with them.
> 
> Thanks
> +Vinod
> 
> On Oct 26, 2015, at 11:34 AM, Vinod Kumar Vavilapalli 
> > wrote:
> 
> Got swamped again.
> 
> We already have about 112 patches in 2.7.2 already.
> 
> There are 15 open tickets in progress. I’ll push progress on them for an RC 
> towards end of this week.
> 
> Thanks
> +Vinod
> 
> On Sep 25, 2015, at 3:27 PM, Vinod Kumar Vavilapalli 
> > wrote:
> 
> Hi all,
> 
> We released 2.7.1 nearly 2.5 months ago. I got caught up with a very long 
> release process for 2.6.1 so couldn't make progress on a 2.7.2. Now is the 
> time!
> 
> Things to do
> 
> (#1) Branch
>-- Branch 2.7 has been open to 2.7.2 commits for a while.
>-- In order to converge on a release, I will branch out 2.7.2 soon.
> 
>  (#2) Patches
>-- 2.7.2 already has a boat load [1] of fixes.
>-- The list of open blocker / critical tickets [2] is not small. I'll 
> start triaging and see what can make it in a week or so of time.
> 
>  (#3) Release
>-- Even if we can get half of the blocker / critical tickets in, the full 
> list [3] will be big enough for us to start voting on an RC in no less than a 
> week.
>-- Leaving aside some buffer time, I plan to start RC process by end of 
> first week of October.
> 
> Thoughts?
> 
> Appreciate help in moving open tickets [2] forward.
> 
> A general note:  Please consider putting any critical / blocker tickets on 
> 2.8 into 2.6.2 and 2.7.2 releases.
> 
> Thanks
> +Vinod
> 
> [1] 2.7.2 Fixed Tickets: 
> https://issues.apache.org/jira/issues/?filter=12333473
> [2] 2.7.2 Open Blockers / Critical Tickets: 
> https://issues.apache.org/jira/issues/?filter=12332867
> [3] 2.7.2 Release Tickets: 
> https://issues.apache.org/jira/issues/?filter=12333461
> 
> 
> 

Re: [VOTE] Release Apache Hadoop 2.6.1 RC1

[Jian He]

+1 (binding)

Build from source code.
Deployed a local cluster.
Validated sample jobs passed.

Jian

> On Sep 18, 2015, at 7:34 AM, Wangda Tan  wrote:
> 
> Deployed a local cluster, verified configured cluster with node labels, run
> jobs with/without node labels.
> 
> +1 (non-binding)
> 
> Thanks!
> 
> On Thu, Sep 17, 2015 at 2:40 PM, Xuan Gong  wrote:
> 
>> Update my vote from +1 (non-binding) to +1 binding
>> 
>> Thanks
>> 
>> Xuan Gong
>> 
>>> On Sep 17, 2015, at 2:05 PM, Xuan Gong  wrote:
>>> 
>>> +1 (non-binding)
>>> Download and compile the source code, run several MR jobs.
>>> 
>>> Xuan Gong
>>> 
 On Sep 16, 2015, at 7:10 PM, Vinod Kumar Vavilapalli <
>> vino...@apache.org> wrote:
 
 Hi all,
 
 After a nearly month long [1] toil, with loads of help from Sangjin Lee
>> and
 Akira Ajisaka, and 153 (RC0)+7(RC1) commits later, I've created a
>> release
 candidate RC1 for hadoop-2.6.1.
 
 RC1 is RC0 [0] (for which I opened and closed a vote last week) + UI
>> fixes
 for the issue Sangjin raised (YARN-3171 and the dependencies YARN-3779,
 YARN-3248), additional fix to avoid incompatibility (YARN-3740), other
>> UI
 bugs (YARN-1884, YARN-3544) and the MiniYARNCluster issue (right patch
>> for
 YARN-2890) that Jeff Zhang raised.
 
 The RC is available at:
>> http://people.apache.org/~vinodkv/hadoop-2.6.1-RC1/
 
 The RC tag in git is: release-2.6.1-RC1
 
 The maven artifacts are available via repository.apache.org at
 https://repository.apache.org/content/repositories/orgapachehadoop-1021
 
 Some notes from our release process
 -  - Sangjin and I moved out a bunch of items pending from 2.6.1 [2] -
 non-committed but desired patches. 2.6.1 is already big as is and is
>> late
 by any standard, we can definitely include them in the next release.
 - The 2.6.1 wiki page [3] captures some (but not all) of the context of
 the patches that we pushed in.
 - Given the number of fixes pushed [4] in, we had to make a bunch of
 changes to our original plan - we added a few improvements that helped
>> us
 backport patches easier (or in many cases made backports possible), and
>> we
 dropped a few that didn't make sense (HDFS-7831, HDFS-7926, HDFS-7676,
 HDFS-7611, HDFS-7843, HDFS-8850).
 - I ran all the unit tests which (surprisingly?) passed. (Except for
>> one,
 which pointed out a missing fix HDFS-7552).
 
 As discussed before [5]
 - This release is the first point release after 2.6.0
 - I’d like to use this as a starting release for 2.6.2 in a few weeks
>> and
 then follow up with more of these.
 
 Please try the release and vote; the vote will run for the usual 5 days.
 
 Thanks,
 Vinod
 
 [0] Hadoop 2.6.1 RC0 vote: http://markmail.org/thread/ubut2rn3lodc55iy
 [1] Hadoop 2.6.1 Release process thread:
 http://markmail.org/thread/wkbgkxkhntx5tlux
 [2] 2.6.1 Pending tickets:
 https://issues.apache.org/jira/issues/?filter=12331711
 [3] 2.6.1 Wiki page: https://wiki.apache.org/hadoop/Release-2.6.1
 -Working-Notes
 [4] List of 2.6.1 patches pushed:
 https://issues.apache.org/jira/issues/?jql=fixVersion%20%3D%202.6.1
 %20and%20labels%20%3D%20%222.6.1-candidate%22
 [5] Planning Hadoop 2.6.1 release:
 http://markmail.org/thread/sbykjn5xgnksh6wg
 
 PS:
 - Note that branch-2.6 which will be the base for 2.6.2 doesn't have
>> these
 fixes yet. Once 2.6.1 goes through, I plan to rebase branch-2.6 based
>> off
 2.6.1.
 - The additional patches in RC1 that got into 2.6.1 all the way from 2.8
 are NOT in 2.7.2 yet, this will be done as a followup.
>>> 
>> 
>> 

Re: Planning Hadoop 2.6.1 release

[Jian He]

Can we include YARN-3999 too,  if the list is not yet frozen.

 On Aug 11, 2015, at 1:49 PM, Hitesh Shah hit...@apache.org wrote:
 
 Jason Lowe just filed YARN-4047 ( patch available too ) which probably is 
 also a good candidate for 2.6.1 or 2.6.2 if the list is frozen. I have 
 labelled the jira as a 2.6.1 candidate. Feel free to switch to 2.6.2 if 
 needed. 
 
 thanks
 — Hitesh
 
 On Aug 10, 2015, at 4:15 PM, Vinod Kumar Vavilapalli 
 vino...@hortonworks.com wrote:
 
 As of now, I am freezing the list. We have 139 tickets to go through, find 
 the right order, get them committed etc.
 
 I’ll start a separate thread for this one-time long release-process of 2.6.1.
 
 Thanks everyone for pitching in.
 +Vinod
 
 On Aug 10, 2015, at 4:13 PM, Vinod Kumar Vavilapalli 
 vino...@hortonworks.com wrote:
 
 Added them all, tx Rohit.
 
 +Vinod
 
 On Aug 5, 2015, at 9:51 PM, Rohith Sharma K S rohithsharm...@huawei.com 
 wrote:
 
 Can we add following fixes to 2.6.1?
 
 YARN-3733 
 YARN-2865
 YARN-3990
 YARN-2894
 
 
 Thanks  Regards
 Rohith Sharma K S
 
 
 -Original Message-
 From: Allan Wilson [mailto:awils...@pandora.com] 
 Sent: 05 August 2015 23:25
 To: common-...@hadoop.apache.org; mapreduce-dev@hadoop.apache.org; 
 yarn-...@hadoop.apache.org
 Cc: hdfs-...@hadoop.apache.org
 Subject: Re: Planning Hadoop 2.6.1 release
 
 Another +1 to add those fixes.
 
 YARN-3487 bug can grind a large cluster to a halt repeatedly
 
 
 
 -Allan
 
 
 Allan Wilson | Sr. Software Engineer | Pandora m 919.841.2449 |  
 awils...@pandora.com
 
 
 
 
 
 On 8/5/15, 1:52 PM, Rich Haase rha...@pandora.com wrote:
 
 +1 to add those fixes.
 
 
 Rich Haase | Sr. Software Engineer | Pandora m 303.887.1146 | 
 rha...@pandora.com
 
 
 
 
 On 8/5/15, 11:42 AM, Wangda Tan wheele...@gmail.com wrote:
 
 Can we add following two fixes to 2.6.1?
 
 https://issues.apache.org/jira/browse/YARN-2922 and 
 https://issues.apache.org/jira/browse/YARN-3487.
 
 They're not fatal issue, but they can cause lots of issue in a large 
 cluster.
 
 Thanks,
 Wangda
 
 
 On Mon, Aug 3, 2015 at 1:21 PM, Sangjin Lee sj...@apache.org wrote:
 
 See my later update in the thread. HDFS-7704 is in the list.
 
 Thanks,
 Sangjin
 
 On Mon, Aug 3, 2015 at 1:19 PM, Vinod Kumar Vavilapalli  
 vino...@hortonworks.com wrote:
 
 Makes sense, it was caused by HDFS-7704 which got into 2.7.0 only 
 and
 is
 not part of the candidate list. Removed HDFS-7916 from the list.
 
 Thanks
 +Vinod
 
 On Jul 24, 2015, at 6:32 PM, Sangjin Lee sj...@apache.org wrote:
 
 Out of the JIRAs we proposed, please remove HDFS-7916. I don't
 think it
 applies to 2.6.
 
 Thanks,
 Sangjin
 
 
 
 
 
 
 
 
 
 

Re: [VOTE] Release Apache Hadoop 2.6.0

[Jian He]

+1,

Built from source.
Deployed a single node cluster.
Ran sample MapReduce jobs while restarting RM successfully.

Jian

On Tue, Nov 18, 2014 at 8:47 AM, Eric Payne erichadoo...@yahoo.com.invalid
wrote:

 +1 . Thanks Arun, for producing this release. I downloaded and built
 the source. I started local cluster and ran wordcount, sleep, and streaming
 jobs.

  - I ran a distributed shell job which tested preserving containers across
 AM restart by setting the -keep_containers_across_application_attempts flag
 and killing the first AM once the containers start. I checked the results
 by looking in the timeline server and comparing the start times of the
 non-AM containers against the start times of the later AM container. -
 I enabled the preemption feature and verified containers were preempted and
 queues were adjusted to guaranteed levels.
 - I ran unit tests for hadoop-yarn-server-resourcemanage. All passed with
 the exception of TestContainerResourceUsage. - I ran unit tests for
 hadoop-hdfs. All passed with the exception of
 TestBPOfferService#testBasicFunctionality (HDFS-3930) Thank you,-Eric Payne




   From: Arun C Murthy a...@hortonworks.com
  To: common-...@hadoop.apache.org common-...@hadoop.apache.org; 
 hdfs-...@hadoop.apache.org hdfs-...@hadoop.apache.org; 
 yarn-...@hadoop.apache.org yarn-...@hadoop.apache.org; 
 mapreduce-dev@hadoop.apache.org mapreduce-dev@hadoop.apache.org
  Sent: Thursday, November 13, 2014 5:08 PM
  Subject: [VOTE] Release Apache Hadoop 2.6.0

 Folks,

 I've created another release candidate (rc1) for hadoop-2.6.0 based on the
 feedback.

 The RC is available at:
 http://people.apache.org/~acmurthy/hadoop-2.6.0-rc1
 The RC tag in git is: release-2.6.0-rc1

 The maven artifacts are available via repository.apache.org at
 https://repository.apache.org/content/repositories/orgapachehadoop-1013.

 Please try the release and vote; the vote will run for the usual 5 days.

 thanks,
 Arun


 --
 CONFIDENTIALITY NOTICE
 NOTICE: This message is intended for the use of the individual or entity to
 which it is addressed and may contain information that is confidential,
 privileged and exempt from disclosure under applicable law. If the reader
 of this message is not the intended recipient, you are hereby notified that
 any printing, copying, dissemination, distribution, disclosure or
 forwarding of this communication is strictly prohibited. If you have
 received this communication in error, please contact the sender immediately
 and delete it from your system. Thank You.





-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

[jira] [Created] (MAPREDUCE-6087) MRJobConfig#MR_CLIENT_TO_AM_IPC_MAX_RETRIES_ON_TIMEOUTS config name is wrong

[Jian He (JIRA)]

Jian He created MAPREDUCE-6087:
--

 Summary: MRJobConfig#MR_CLIENT_TO_AM_IPC_MAX_RETRIES_ON_TIMEOUTS 
config name is wrong
 Key: MAPREDUCE-6087
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6087
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Reporter: Jian He


The config name for MRJobConfig#MR_CLIENT_TO_AM_IPC_MAX_RETRIES_ON_TIMEOUTS now 
has double prefix as yarn.app.mapreduce. + 
yarn.app.mapreduce.client-am.ipc.max-retries-on-timeouts



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: hadoop-2.5 - June end?

[Jian He]

Just found the roadmap includes a wrong jira link for RM Webservices
YARN-1731, it should be YARN-1695
https://issues.apache.org/jira/browse/YARN-1695.

Jian


On Mon, Jun 23, 2014 at 2:51 PM, Karthik Kambatla ka...@cloudera.com
wrote:

 While YARN-1492 would greatly benefit performance, I think it might be a
 little too ambitious at this point to include it in 2.5 without delaying
 it.

 As I mentioned before and from my offline conversation with Arun, we might
 want to release 2.6 end of July and that might be a better vehicle for
 YARN-1492.


 On Mon, Jun 23, 2014 at 2:30 PM, Sangjin Lee sjl...@gmail.com wrote:

  It would be great if we can consider the shared cache (YARN-1492) as part
  of 2.5. It obviously will depend on the reviews from folks, but the
 feature
  itself should be reasonably solid.
 
  Thanks,
  Sangjin
 
 
  On Mon, Jun 23, 2014 at 2:24 PM, Arun C Murthy a...@hortonworks.com
  wrote:
 
   Thanks Karthik!
  
   I've updated https://wiki.apache.org/hadoop/Roadmap with features
 which
   are very close to completion. Let's see if that makes sense and if we
 get
   any further feedback.
  
   Arun
  
   On Jun 23, 2014, at 2:09 PM, Karthik Kambatla ka...@cloudera.com
  wrote:
  
I can pick up the RM duties for 2.5. If I run into any HDFS doubts, I
   might
need some help from someone more familiar with HDFS.
   
   
On Mon, Jun 23, 2014 at 12:07 PM, Arun C Murthy a...@hortonworks.com
 
   wrote:
   
Folks,
   
I'd appreciate some help here. Due to family reasons (all good
 ones),
I'll be away for a couple of weeks. Can someone else pick up the RM
   duties
for hadoop-2.5? Maybe Andrew since he's expressed interest in the
  past?
   I
will pick up the thread again for hadoop-2.6, but I don't want to
  block
hadoop-2.5 due to my non-availability.
   
thanks,
Arun
   
On Jun 9, 2014, at 9:39 AM, Arun C Murthy a...@hortonworks.com
  wrote:
   
Folks,
   
As you can see from the Roadmap wiki, it looks like several items
 are
still a bit away from being ready.
   
I think rather than wait for them, it will be useful to create an
intermediate release (2.5) this month - I think ATS security is
 pretty
close, so we can ship that. I'm thinking of creating hadoop-2.5 by
 end
   of
the month, with a branch a couple of weeks prior.
   
Thoughts?
   
thanks,
Arun
   
   
   
   
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or
   entity to
which it is addressed and may contain information that is
  confidential,
privileged and exempt from disclosure under applicable law. If the
   reader
of this message is not the intended recipient, you are hereby
 notified
   that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender
   immediately
and delete it from your system. Thank You.
   
  
   --
   Arun C. Murthy
   Hortonworks Inc.
   http://hortonworks.com/hdp/
  
  
  
   --
   CONFIDENTIALITY NOTICE
   NOTICE: This message is intended for the use of the individual or
 entity
  to
   which it is addressed and may contain information that is confidential,
   privileged and exempt from disclosure under applicable law. If the
 reader
   of this message is not the intended recipient, you are hereby notified
  that
   any printing, copying, dissemination, distribution, disclosure or
   forwarding of this communication is strictly prohibited. If you have
   received this communication in error, please contact the sender
  immediately
   and delete it from your system. Thank You.
  
 


-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: [VOTE] Release Apache Hadoop 2.4.1

[Jian He]

+1,

built from source code.
installed single node cluster.
ran a few sample jobs successfully.

Jian


On Wed, Jun 18, 2014 at 12:13 PM, Vinod Kumar Vavilapalli 
vino...@apache.org wrote:


 There is one item [MAPREDUCE-5830 HostUtil.getTaskLogUrl is not backwards
 binary compatible with 2.3] marked for 2.4. Should we include it?

 There is no patch there yet, it doesn't really help much other than
 letting older clients compile - even if we put the API back in, the URL
 returned is invalid.

 +Vinod

 On Jun 16, 2014, at 9:27 AM, Arun C Murthy a...@hortonworks.com wrote:

  Folks,
 
  I've created a release candidate (rc0) for hadoop-2.4.1 (bug-fix
 release) that I would like to push out.
 
  The RC is available at:
 http://people.apache.org/~acmurthy/hadoop-2.4.1-rc0
  The RC tag in svn is here:
 https://svn.apache.org/repos/asf/hadoop/common/tags/release-2.4.1-rc0
 
  The maven artifacts are available via repository.apache.org.
 
  Please try the release and vote; the vote will run for the usual 7 days.
 
  thanks,
  Arun
 
 
 
  --
  Arun C. Murthy
  Hortonworks Inc.
  http://hortonworks.com/hdp/
 
 
 
  --
  CONFIDENTIALITY NOTICE
  NOTICE: This message is intended for the use of the individual or entity
 to
  which it is addressed and may contain information that is confidential,
  privileged and exempt from disclosure under applicable law. If the reader
  of this message is not the intended recipient, you are hereby notified
 that
  any printing, copying, dissemination, distribution, disclosure or
  forwarding of this communication is strictly prohibited. If you have
  received this communication in error, please contact the sender
 immediately
  and delete it from your system. Thank You.


 --
 CONFIDENTIALITY NOTICE
 NOTICE: This message is intended for the use of the individual or entity to
 which it is addressed and may contain information that is confidential,
 privileged and exempt from disclosure under applicable law. If the reader
 of this message is not the intended recipient, you are hereby notified that
 any printing, copying, dissemination, distribution, disclosure or
 forwarding of this communication is strictly prohibited. If you have
 received this communication in error, please contact the sender immediately
 and delete it from your system. Thank You.


-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

[jira] [Created] (MAPREDUCE-5838) TestRMDelegationTokens#testRMDTMasterKeyStateOnRollingMasterKey is failing intermittently

[Jian He (JIRA)]

Jian He created MAPREDUCE-5838:
--

 Summary: 
TestRMDelegationTokens#testRMDTMasterKeyStateOnRollingMasterKey is failing 
intermittently
 Key: MAPREDUCE-5838
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5838
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Reporter: Jian He
Assignee: Jian He






--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (MAPREDUCE-5832) TestJobClient fails sometimes on Windows

[Jian He (JIRA)]

Jian He created MAPREDUCE-5832:
--

 Summary: TestJobClient fails sometimes on Windows
 Key: MAPREDUCE-5832
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5832
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Reporter: Jian He
Assignee: Jian He






--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: [VOTE] Release Apache Hadoop 2.4.0

[Jian He]

+1 non-binding

Built from source code, tested on a single node cluster. Successfully ran a
few MR sample jobs.
Tested RM restart while job is running.

Thanks,
Jian

On Tue, Apr 1, 2014 at 5:42 PM, Travis Thompson tthomp...@linkedin.comwrote:

 +1 non-binding

 Built from git. Started with 120 node 2.3.0 cluster with security and
 non HA, ran upgrade (non rolling) to 2.4.0.  Confirmed fsimage is OK and
 HDFS successfully upgraded.  Also successfully ran some pig jobs and
 mapreduce examples.  Haven't found any issues yet but will continue
 testing.  Did not test Timeline Server since I'm using security.

 Thanks,
 Travis

 On 03/31/2014 02:24 AM, Arun C Murthy wrote:
  Folks,
 
  I've created a release candidate (rc0) for hadoop-2.4.0 that I would
 like to get released.
 
  The RC is available at:
 http://people.apache.org/~acmurthy/hadoop-2.4.0-rc0
  The RC tag in svn is here:
 https://svn.apache.org/repos/asf/hadoop/common/tags/release-2.4.0-rc0
 
  The maven artifacts are available via repository.apache.org.
 
  Please try the release and vote; the vote will run for the usual 7 days.
 
  thanks,
  Arun
 
  --
  Arun C. Murthy
  Hortonworks Inc.
  http://hortonworks.com/
 
 
 


-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

[jira] [Created] (MAPREDUCE-5818) hsadmin cmd is missing in mapred.cmd

[Jian He (JIRA)]

Jian He created MAPREDUCE-5818:
--

 Summary: hsadmin cmd is missing in mapred.cmd
 Key: MAPREDUCE-5818
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5818
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Reporter: Jian He
Assignee: Jian He






--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Resolved] (MAPREDUCE-5397) AM crashes because Webapp failed to start on multi node cluster

[Jian He (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-5397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jian He resolved MAPREDUCE-5397.


Resolution: Cannot Reproduce

 AM crashes because Webapp failed to start on multi node cluster
 ---

 Key: MAPREDUCE-5397
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5397
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Reporter: Jian He
Assignee: Jian He
 Attachments: log.txt


 I set up a 12 nodes cluster and tried submitting jobs but get this exception.
 But job is able to succeed after AM crashes and retry a few times(2 or 3)
 {code}
 2013-07-12 18:56:28,438 INFO [main] org.mortbay.log: Extract 
 jar:file:/grid/0/dev/jhe/hadoop-2.1.0-beta/share/hadoop/yarn/hadoop-yarn-common-2.1.0-beta.jar!/webapps/mapreduce
  to /tmp/Jetty_0_0_0_0_43554_mapreduceljbmlg/webapp
 2013-07-12 18:56:28,528 WARN [main] org.mortbay.log: Failed startup of 
 context 
 org.mortbay.jetty.webapp.WebAppContext@2726b2{/,jar:file:/grid/0/dev/jhe/hadoop-2.1.0-beta/share/hadoop/yarn/hadoop-yarn-common-2.1.0-beta.jar!/webapps/mapreduce}
 java.io.FileNotFoundException: 
 /tmp/Jetty_0_0_0_0_43554_mapreduceljbmlg/webapp/webapps/mapreduce/.keep 
 (No such file or directory)
   at java.io.FileOutputStream.open(Native Method)
   at java.io.FileOutputStream.init(FileOutputStream.java:194)
   at java.io.FileOutputStream.init(FileOutputStream.java:145)
   at org.mortbay.resource.JarResource.extract(JarResource.java:215)
   at 
 org.mortbay.jetty.webapp.WebAppContext.resolveWebApp(WebAppContext.java:974)
   at 
 org.mortbay.jetty.webapp.WebAppContext.getWebInf(WebAppContext.java:832)
   at 
 org.mortbay.jetty.webapp.WebInfConfiguration.configureClassLoader(WebInfConfiguration.java:62)
   at 
 org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:489)
   at 
 org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
   at 
 org.mortbay.jetty.handler.HandlerCollection.doStart(HandlerCollection.java:152)
   at 
 org.mortbay.jetty.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:156)
   at 
 org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
   at 
 org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130)
   at org.mortbay.jetty.Server.doStart(Server.java:224)
   at 
 org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
   at org.apache.hadoop.http.HttpServer.start(HttpServer.java:684)
   at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:211)
   at 
 org.apache.hadoop.mapreduce.v2.app.client.MRClientService.serviceStart(MRClientService.java:134)
   at 
 org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
   at 
 org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:101)
   at 
 org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1019)
   at 
 org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
   at 
 org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:1394)
   at java.security.AccessController.doPrivileged(Native Method)
   at javax.security.auth.Subject.doAs(Subject.java:396)
   at 
 org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1477)
   at 
 org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1390)
 {code}



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

Re: [VOTE] Release Apache Hadoop 2.2.0

[Jian He]

+1 non-binding

Built from source code, and ran a few sample jobs on single node cluster,
tested RM and AM recovery.

Thanks,
Jian


On Thu, Oct 10, 2013 at 10:59 AM, Chris Nauroth cnaur...@hortonworks.comwrote:

 +1 non-binding

 I verified the checksum and signature.  I deployed the tarball to a small
 cluster of Ubuntu VMs: 1 * NameNode, 1 * ResourceManager, 2 * DataNode, 2 *
 NodeManager, 1 * SecondaryNameNode.  I ran a few HDFS commands and sample
 MapReduce jobs.  I verified that the 2NN can take a checkpoint
 successfully.  Everything worked as expected.

 The outcome of the recent discussions on HDFS symlinks was that we need to
 disable the feature in this release.  Just to be certain that this patch
 took, I wrote a small client to call FileSystem.createSymlink and tried to
 run it in my 2.2.0 cluster.  It threw UnsupportedOperationException, which
 is the expected behavior.

 Chris Nauroth
 Hortonworks
 http://hortonworks.com/



 On Thu, Oct 10, 2013 at 10:18 AM, Bikas Saha bi...@hortonworks.com
 wrote:

  +1 (non binding)
 
  -Original Message-
  From: Arpit Gupta [mailto:ar...@hortonworks.com]
  Sent: Thursday, October 10, 2013 10:06 AM
  To: common-...@hadoop.apache.org
  Cc: hdfs-...@hadoop.apache.org; yarn-...@hadoop.apache.org;
  mapreduce-dev@hadoop.apache.org
  Subject: Re: [VOTE] Release Apache Hadoop 2.2.0
 
  +1 (non binding)
 
  Ran secure and non secure multi node clusters and tested HA and RM
  recovery tests.
 
  --
  Arpit Gupta
  Hortonworks Inc.
  http://hortonworks.com/
 
  On Oct 7, 2013, at 12:00 AM, Arun C Murthy a...@hortonworks.com wrote:
 
   Folks,
  
   I've created a release candidate (rc0) for hadoop-2.2.0 that I would
  like to get released - this release fixes a small number of bugs and some
  protocol/api issues which should ensure they are now stable and will not
  change in hadoop-2.x.
  
   The RC is available at:
   http://people.apache.org/~acmurthy/hadoop-2.2.0-rc0
   The RC tag in svn is here:
   http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.2.0-rc0
  
   The maven artifacts are available via repository.apache.org.
  
   Please try the release and vote; the vote will run for the usual 7
 days.
  
   thanks,
   Arun
  
   P.S.: Thanks to Colin, Andrew, Daryn, Chris and others for helping nail
  down the symlinks-related issues. I'll release note the fact that we have
  disabled it in 2.2. Also, thanks to Vinod for some heavy-lifting on the
  YARN side in the last couple of weeks.
  
  
  
  
  
   --
   Arun C. Murthy
   Hortonworks Inc.
   http://hortonworks.com/
  
  
  
   --
   CONFIDENTIALITY NOTICE
   NOTICE: This message is intended for the use of the individual or
   entity to which it is addressed and may contain information that is
   confidential, privileged and exempt from disclosure under applicable
   law. If the reader of this message is not the intended recipient, you
   are hereby notified that any printing, copying, dissemination,
   distribution, disclosure or forwarding of this communication is
   strictly prohibited. If you have received this communication in error,
   please contact the sender immediately and delete it from your system.
  Thank You.
 
 
  --
  CONFIDENTIALITY NOTICE
  NOTICE: This message is intended for the use of the individual or entity
  to
  which it is addressed and may contain information that is confidential,
  privileged and exempt from disclosure under applicable law. If the reader
  of this message is not the intended recipient, you are hereby notified
  that
  any printing, copying, dissemination, distribution, disclosure or
  forwarding of this communication is strictly prohibited. If you have
  received this communication in error, please contact the sender
  immediately
  and delete it from your system. Thank You.
 
  --
  CONFIDENTIALITY NOTICE
  NOTICE: This message is intended for the use of the individual or entity
 to
  which it is addressed and may contain information that is confidential,
  privileged and exempt from disclosure under applicable law. If the reader
  of this message is not the intended recipient, you are hereby notified
 that
  any printing, copying, dissemination, distribution, disclosure or
  forwarding of this communication is strictly prohibited. If you have
  received this communication in error, please contact the sender
 immediately
  and delete it from your system. Thank You.
 

 --
 CONFIDENTIALITY NOTICE
 NOTICE: This message is intended for the use of the individual or entity to
 which it is addressed and may contain information that is confidential,
 privileged and exempt from disclosure under applicable law. If the reader
 of this message is not the intended recipient, you are hereby notified that
 any printing, copying, dissemination, distribution, disclosure or
 forwarding of this communication is strictly prohibited. If you have
 received this communication in error, please contact the sender immediately
 and delete it from your system. Thank 

[jira] [Created] (MAPREDUCE-5568) JHS return invalid string for reducer completion percentage if AM restarts with 0 reducer.

[Jian He (JIRA)]

Jian He created MAPREDUCE-5568:
--

 Summary: JHS return invalid string for reducer completion 
percentage if AM restarts with 0 reducer.
 Key: MAPREDUCE-5568
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5568
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Reporter: Jian He


JobCLient shows like:
{code}
13/10/05 16:26:09 INFO mapreduce.Job:  map 100% reduce NaN%
13/10/05 16:26:09 INFO mapreduce.Job: Job job_1381015536254_0001 completed 
successfully
13/10/05 16:26:09 INFO mapreduce.Job: Counters: 26
File System Counters
FILE: Number of bytes read=0
FILE: Number of bytes written=76741
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=48
HDFS: Number of bytes written=0
HDFS: Number of read operations=1
HDFS: Number of large read operations=0
HDFS: Number of write operations=0
{code}
With mapped job -status command, it shows:
{code}
Uber job : false
Number of maps: 1
Number of reduces: 0
map() completion: 1.0
reduce() completion: NaN
Job state: SUCCEEDED
retired: false
reason for failure:
{code}



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Created] (MAPREDUCE-5535) TestClusterMRNotification.testMR is failing

[Jian He (JIRA)]

Jian He created MAPREDUCE-5535:
--

 Summary: TestClusterMRNotification.testMR is failing
 Key: MAPREDUCE-5535
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5535
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Reporter: Jian He


{code}
testMR(org.apache.hadoop.mapred.TestClusterMRNotification)  Time elapsed: 
35.222 sec   FAILURE!
junit.framework.AssertionFailedError: expected:2 but was:0
at junit.framework.Assert.fail(Assert.java:50)
at junit.framework.Assert.failNotEquals(Assert.java:287)
at junit.framework.Assert.assertEquals(Assert.java:67)
at junit.framework.Assert.assertEquals(Assert.java:199)
at junit.framework.Assert.assertEquals(Assert.java:205)
at 
org.apache.hadoop.mapred.NotificationTestCase.testMR(NotificationTestCase.java:163)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at junit.framework.TestCase.runTest(TestCase.java:168)
at junit.framework.TestCase.runBare(TestCase.java:134)
at junit.framework.TestResult$1.protect(TestResult.java:110)
at junit.framework.TestResult.runProtected(TestResult.java:128)
at junit.framework.TestResult.run(TestResult.java:113)
at junit.framework.TestCase.run(TestCase.java:124)
at junit.framework.TestSuite.runTest(TestSuite.java:243)
at junit.framework.TestSuite.run(TestSuite.java:238)
at 
org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:83)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:264)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124)
at 
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200)
at 
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153)
at 
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
{code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (MAPREDUCE-5516) TestMRJobClient fails on trunk

[Jian He (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-5516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jian He resolved MAPREDUCE-5516.


Resolution: Duplicate

 TestMRJobClient fails on trunk
 --

 Key: MAPREDUCE-5516
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5516
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Reporter: Jian He



--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (MAPREDUCE-5516) TestMRJobClient fails, TestUberAM hangs on trunk

[Jian He (JIRA)]

Jian He created MAPREDUCE-5516:
--

 Summary: TestMRJobClient fails,  TestUberAM hangs on trunk
 Key: MAPREDUCE-5516
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5516
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Reporter: Jian He




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (MAPREDUCE-5471) Succeed job tries to restart after RMrestart

[Jian He (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-5471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jian He resolved MAPREDUCE-5471.


Resolution: Duplicate

Closed as a duplicate of YARN-540

 Succeed job tries to restart after RMrestart
 

 Key: MAPREDUCE-5471
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5471
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Reporter: Yesha Vora
Assignee: Jian He
Priority: Blocker
 Attachments: MR5471-1AM.log, MR5471-2AM.log


 Run a job , restart RM when job just finished. It should not restart the job 
 once it Succeed.
 After RM restart, The AM of restarted job fails with below error.
 AM log after Rmrestart:
 013-08-19 17:29:21,144 INFO [main] 
 org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Stopping 
 JobHistoryEventHandler. Size of the outstanding queue size is 0
 2013-08-19 17:29:21,145 INFO [main] 
 org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Stopped 
 JobHistoryEventHandler. super.stop()
 2013-08-19 17:29:21,146 INFO [main] 
 org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Deleting staging directory 
 hdfs://host1:port1/user/ABC/.staging/job_1376933101704_0001
 2013-08-19 17:29:21,156 FATAL [main] 
 org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster
 org.apache.hadoop.yarn.exceptions.YarnRuntimeException: 
 java.io.FileNotFoundException: File does not exist: 
 hdfs://host1:port1/ABC/.staging/job_1376933101704_0001/job.splitmetainfo
 at 
 org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$InitTransition.createSplits(JobImpl.java:1469)
 at 
 org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$InitTransition.transition(JobImpl.java:1324)
 at 
 org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$InitTransition.transition(JobImpl.java:1291)
 at 
 org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:385)
 at 
 org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
 at 
 org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
 at 
 org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
 at 
 org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:922)
 at 
 org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:131)
 at 
 org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher.handle(MRAppMaster.java:1184)
 at 
 org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:995)
 at 
 org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
 at 
 org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:1394)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:396)
 at 
 org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1477)
 at 
 org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1390)
 at 
 org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1323)
 Caused by: java.io.FileNotFoundException: File does not exist: 
 hdfs://host1:port1/ABC/.staging/job_1376933101704_0001/job.splitmetainfo
 at 
 org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1121)
 at 
 org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1113)
 at 
 org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:78)
 at 
 org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1113)
 at 
 org.apache.hadoop.mapreduce.split.SplitMetaInfoReader.readSplitMetaInfo(SplitMetaInfoReader.java:51)
 at 
 org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$InitTransition.createSplits(JobImpl.java:1464)
 ... 17 more
 2013-08-19 17:29:21,158 INFO [Thread-2] 
 org.apache.hadoop.mapreduce.v2.app.MRAppMaster: MRAppMaster received a 
 signal. Signaling RMCommunicator and JobHistoryEventHandler.
 2013-08-19 17:29:21,159 WARN [Thread-2] 
 org.apache.hadoop.util.ShutdownHookManager: ShutdownHook 
 'MRAppMasterShutdownHook' failed, java.lang.NullPointerException
 java.lang.NullPointerException
 at 
 org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter.setSignalled(MRAppMaster.java:805)
 at 
 org.apache.hadoop.mapreduce.v2.app.MRAppMaster$MRAppMasterShutdownHook.run(MRAppMaster.java:1344)
 at 
 org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54)

--
This message

[jira] [Created] (MAPREDUCE-5505) Clients should be notified job finished after job successfully unregistered

[Jian He (JIRA)]

Jian He created MAPREDUCE-5505:
--

 Summary: Clients should be notified job finished after job 
successfully unregistered 
 Key: MAPREDUCE-5505
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5505
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Reporter: Jian He


This is to make sure user is notified job finished after job is really done. 
This does increase client latency but can reduce some races during unregister 
like YARN-540

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (MAPREDUCE-5497) '5s sleep' in MRAppMaster.shutDownJob is only needed before stopping ClientService

[Jian He (JIRA)]

Jian He created MAPREDUCE-5497:
--

 Summary: '5s sleep'  in MRAppMaster.shutDownJob is only needed 
before stopping ClientService
 Key: MAPREDUCE-5497
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5497
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Reporter: Jian He
Assignee: Jian He


Since the '5s sleep' is for the purpose to let clients know the final states, 
put it after other services are stopped and only before stopping ClientService 
is enough. This can reduce some race conditions like MAPREDUCE-5471

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (MAPREDUCE-5436) Better exception handling for Invalid AMRMToken exception for MapReduce

[Jian He (JIRA)]

Jian He created MAPREDUCE-5436:
--

 Summary: Better exception handling for Invalid AMRMToken exception 
for MapReduce
 Key: MAPREDUCE-5436
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5436
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Reporter: Jian He
Assignee: Jian He


Nowadays, even though MR gets an InvalidToken Exception from RM, RMCommunicator 
just ignores it and continues looping and retry. we should explicitly handle 
such exception.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (MAPREDUCE-5412) Change MR to use multiple containers API of ContainerManager after YARN-926

[Jian He (JIRA)]

Jian He created MAPREDUCE-5412:
--

 Summary: Change MR to use multiple containers API of 
ContainerManager after YARN-926
 Key: MAPREDUCE-5412
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5412
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Reporter: Jian He
Assignee: Jian He




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (MAPREDUCE-5397) AM crashes because Webapp failed to start on multi node cluster

[Jian He (JIRA)]

Jian He created MAPREDUCE-5397:
--

 Summary: AM crashes because Webapp failed to start on multi node 
cluster
 Key: MAPREDUCE-5397
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5397
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Reporter: Jian He
Assignee: Jian He


I set up a 12 nodes cluster and tried submitting jobs but get this exception.
But job is able to succeed after AM crashes and retry a few times(2 or 3)

{code}
2013-07-12 18:56:28,438 INFO [main] org.mortbay.log: Extract 
jar:file:/grid/0/dev/jhe/hadoop-2.1.0-beta/share/hadoop/yarn/hadoop-yarn-common-2.1.0-beta.jar!/webapps/mapreduce
 to /tmp/Jetty_0_0_0_0_43554_mapreduceljbmlg/webapp
2013-07-12 18:56:28,528 WARN [main] org.mortbay.log: Failed startup of context 
org.mortbay.jetty.webapp.WebAppContext@2726b2{/,jar:file:/grid/0/dev/jhe/hadoop-2.1.0-beta/share/hadoop/yarn/hadoop-yarn-common-2.1.0-beta.jar!/webapps/mapreduce}
java.io.FileNotFoundException: 
/tmp/Jetty_0_0_0_0_43554_mapreduceljbmlg/webapp/webapps/mapreduce/.keep (No 
such file or directory)
at java.io.FileOutputStream.open(Native Method)
at java.io.FileOutputStream.init(FileOutputStream.java:194)
at java.io.FileOutputStream.init(FileOutputStream.java:145)
at org.mortbay.resource.JarResource.extract(JarResource.java:215)
at 
org.mortbay.jetty.webapp.WebAppContext.resolveWebApp(WebAppContext.java:974)
at 
org.mortbay.jetty.webapp.WebAppContext.getWebInf(WebAppContext.java:832)
at 
org.mortbay.jetty.webapp.WebInfConfiguration.configureClassLoader(WebInfConfiguration.java:62)
at 
org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:489)
at 
org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
at 
org.mortbay.jetty.handler.HandlerCollection.doStart(HandlerCollection.java:152)
at 
org.mortbay.jetty.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:156)
at 
org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
at 
org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130)
at org.mortbay.jetty.Server.doStart(Server.java:224)
at 
org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
at org.apache.hadoop.http.HttpServer.start(HttpServer.java:684)
at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:211)
at 
org.apache.hadoop.mapreduce.v2.app.client.MRClientService.serviceStart(MRClientService.java:134)
at 
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at 
org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:101)
at 
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1019)
at 
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at 
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:1394)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1477)
at 
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1390)
{code}


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (MAPREDUCE-5303) Changes on MR after moving ProtoBase to package impl.pb on YARN-724

[Jian He (JIRA)]

Jian He created MAPREDUCE-5303:
--

 Summary: Changes on MR after moving ProtoBase to package impl.pb 
on YARN-724
 Key: MAPREDUCE-5303
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5303
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Reporter: Jian He
Assignee: Jian He




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (MAPREDUCE-5297) Update MR App after YARN-748

[Jian He (JIRA)]

Jian He created MAPREDUCE-5297:
--

 Summary: Update MR App  after YARN-748
 Key: MAPREDUCE-5297
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5297
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Reporter: Jian He




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (MAPREDUCE-5285) Update MR App to use immutable ApplicationAttemptID, ContainerID, NodeID after YARN-735

[Jian He (JIRA)]

Jian He created MAPREDUCE-5285:
--

 Summary: Update MR App to use immutable ApplicationAttemptID, 
ContainerID, NodeID after YARN-735
 Key: MAPREDUCE-5285
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5285
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Reporter: Jian He




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (MAPREDUCE-5270) Migrate from using BuilderUtil factory methods to individual record factory method on MapReduce side

[Jian He (JIRA)]

Jian He created MAPREDUCE-5270:
--

 Summary: Migrate from using BuilderUtil factory methods to 
individual record factory method on MapReduce side
 Key: MAPREDUCE-5270
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5270
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Reporter: Jian He
Assignee: Jian He


 Migrate the factory method on map reduce side.


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira