[jira] [Created] (MAPREDUCE-6942) Add descriptions for jobhistory.done-dir configs

[Yeliang Cang (JIRA)]

Yeliang Cang created MAPREDUCE-6942:
---

 Summary: Add descriptions for jobhistory.done-dir configs
 Key: MAPREDUCE-6942
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6942
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
Reporter: Yeliang Cang
Priority: Minor






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

[jira] [Created] (MAPREDUCE-6941) The default setting doesn't work for MapReduce job

[Junping Du (JIRA)]

Junping Du created MAPREDUCE-6941:
-

 Summary: The default setting doesn't work for MapReduce job
 Key: MAPREDUCE-6941
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6941
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Affects Versions: 3.0.0-beta1
Reporter: Junping Du
Priority: Blocker


On the deployment of hadoop 3 cluster (based on current trunk branch) with 
default settings, the MR job will get failed as following exceptions:
{noformat}
2017-08-16 13:00:03,846 INFO mapreduce.Job: Job job_1502913552390_0001 running 
in uber mode : false
2017-08-16 13:00:03,847 INFO mapreduce.Job:  map 0% reduce 0%
2017-08-16 13:00:03,864 INFO mapreduce.Job: Job job_1502913552390_0001 failed 
with state FAILED due to: Application application_1502913552390_0001 failed 2 
times due to AM Container for appattempt_1502913552390_0001_02 exited with  
exitCode: 1
Failing this attempt.Diagnostics: [2017-08-16 13:00:02.963]Exception from 
container-launch.
Container id: container_1502913552390_0001_02_01
Exit code: 1
Stack trace: ExitCodeException exitCode=1:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:994)
at org.apache.hadoop.util.Shell.run(Shell.java:887)
at 
org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1212)
at 
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:295)
at 
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.launchContainer(ContainerLaunch.java:455)
at 
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:275)
at 
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:90)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

{noformat}

This is because mapreduce related jar are not added into yarn setup by default. 
To make MR job run successful, we need to add following configurations to 
yarn-site.xml now:
{noformat}

  yarn.application.classpath
  
...
/share/hadoop/mapreduce/*,
/share/hadoop/mapreduce/lib/*
...
  
{noformat}
But this config is not necessary for previous version of Hadoop. We should fix 
this issue before beta release otherwise it will be a regression for 
configuration changes.

This could be more like a YARN issue (if so, we should move), depends on how we 
fix it finally.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [DISCUSS] Merging YARN-5355 (Timeline Service v.2) to trunk

[Andrew Wang]

Great, thanks Vrushali! Sounds good to me.

I have a few procedural release notes comments I'll put on YARN-5355, to
make sure we advertise this to our users appropriately.

On Wed, Aug 16, 2017 at 11:32 AM, Vrushali Channapattan <
vrushal...@gmail.com> wrote:

> Hi Andrew,
>
> Thanks for your response!
>
> There have been no changes to existing APIs since alpha1.
>
> We at Twitter have tested the feature to demonstrate it works at what we
> consider moderate scale but this did not include the security related
> testing. The security testing is in progress at present by Timeline Service
> V2 team in the community and we think we will have more details on this
> very soon.
>
> About the jiras under YARN-5355: Only 3 of those sub-tasks are what we
> think of as "merge-blockers". The issues being targeted for merge are in
> [link1] below. There are about 59 jiras of which 56 are completed.
>
> We plan to make a new umbrella jira after the merge to trunk. We will then
> create a new branch with the new jira name and move these open jiras under
> YARN-5355 as subtasks of that new umbrella jira.
>
> thanks
> Vrushali
> [link1] https://issues.apache.org/jira/projects/YARN/versions/12337991
>
>
> On Wed, Aug 16, 2017 at 10:47 AM, Andrew Wang 
> wrote:
>
>> Hi Vrushali,
>>
>> Glad to hear this major dev milestone is nearing completion!
>>
>> Repeating my request on other merge [DISCUSS] threads, could you comment
>> on testing and API stability of this merge? Our timeline for beta1 is about
>> a month out, so there's not much time to fix things beforehand.
>>
>> Looking at YARN-5355 there are also many unresolved subtasks. Should most
>> of these be moved out to a new umbrella? I'm wondering what needs to be
>> completed before sending the merge vote.
>>
>> Given that TSv2 is committed for 3.0.0 GA, I'm more willing to flex the
>> beta1 release date for this feature than others. Hopefully that won't be
>> necessary though :)
>>
>> Best,
>> Andrew
>>
>> On Wed, Aug 16, 2017 at 10:26 AM, Vrushali Channapattan <
>> vrushalic2...@gmail.com> wrote:
>>
>>> Looks like some of the hyperlinks appear messed up, my apologies,
>>> resending
>>> the same email with hopefully better looking content:
>>>
>>> Hi All,
>>>
>>> I'd like to open a discussion for merging Timeline Service v2 (YARN-5355)
>>> to trunk in a few weeks.
>>>
>>> We have previously completed one merge onto trunk [1] and Timeline
>>> Service
>>> v2 has been part of Hadoop release 3.0.0-alpha1.
>>>
>>> Since then, we have been working on extending the capabilities of
>>> Timeline
>>> Service v2 in a feature branch [2].  There are a few related issues
>>> pending
>>> that are being actively worked upon and tested. As soon as they are
>>> resolved, we plan on starting a merge vote within the next two weeks. The
>>> goal is to get this into hadoop3 beta.
>>>
>>> We have paid close attention to ensure that  once disabled Timeline
>>> Service
>>> v2 does not impact existing functionality when disabled (by default).
>>>
>>> At a high level, following are the key features that have been
>>> implemented
>>> since 3.0.0-alpha1:
>>> - Security (via Kerberos Authentication & delegation tokens) [YARN-3053]
>>> - Timeline server usability improvements [timeline-server
>>> >> %20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-5355%20AND%20c
>>> omponent%20%3D%20timelineserver%20AND%20labels%20!%3D%20atsv
>>> 2-hbase%20ORDER%20BY%20updated%20ASC%2C%20priority%20DESC%
>>> 2C%20created%20ASC>
>>> ]
>>> - HBase specific improvements [atsv2-hbase
>>> >> %20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-5355%20AND%20l
>>> abels%20%3D%20atsv2-hbase%20ORDER%20BY%20updated%20DESC%2C%
>>> 20affectedVersion%20DESC%2C%20priority%20DESC%2C%20created%20ASC>
>>> ]
>>> - REST API additions and improvements [timeline-reader
>>> >> %20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-5355%20AND%20c
>>> omponent%20in%20(timelineclient%2C%20timelinereader)%
>>> 20ORDER%20BY%20updated%20ASC%2C%20priority%20DESC%2C%20created%20ASC>
>>> ]
>>> - Reader side simple authorization via whitelist [YARN-6820]
>>>
>>> We would love to get your thoughts on this before we open a real voting
>>> thread.
>>>
>>> Special thanks to a team of folks who worked hard and contributed towards
>>> this effort via patches, reviews and guidance: Rohith Sharma K S, Varun
>>> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
>>> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
>>>
>>> Thanks
>>> Vrushali
>>> [1] Merge to trunk: http://www.mail-archive.com/yarn-dev@hadoop.apache.
>>> org/msg23897.html
>>> 
>>> [2] feature branch YARN-5355 commits: https://github.com/ap
>>> ache/hadoop/commits/YARN-5355
>>>
>>> 

Re: [DISCUSS] Merging YARN-5355 (Timeline Service v.2) to trunk

[Vrushali Channapattan]

Hi Andrew,

Thanks for your response!

There have been no changes to existing APIs since alpha1.

We at Twitter have tested the feature to demonstrate it works at what we
consider moderate scale but this did not include the security related
testing. The security testing is in progress at present by Timeline Service
V2 team in the community and we think we will have more details on this
very soon.

About the jiras under YARN-5355: Only 3 of those sub-tasks are what we
think of as "merge-blockers". The issues being targeted for merge are in
[link1] below. There are about 59 jiras of which 56 are completed.

We plan to make a new umbrella jira after the merge to trunk. We will then
create a new branch with the new jira name and move these open jiras under
YARN-5355 as subtasks of that new umbrella jira.

thanks
Vrushali
[link1] https://issues.apache.org/jira/projects/YARN/versions/12337991


On Wed, Aug 16, 2017 at 10:47 AM, Andrew Wang 
wrote:

> Hi Vrushali,
>
> Glad to hear this major dev milestone is nearing completion!
>
> Repeating my request on other merge [DISCUSS] threads, could you comment
> on testing and API stability of this merge? Our timeline for beta1 is about
> a month out, so there's not much time to fix things beforehand.
>
> Looking at YARN-5355 there are also many unresolved subtasks. Should most
> of these be moved out to a new umbrella? I'm wondering what needs to be
> completed before sending the merge vote.
>
> Given that TSv2 is committed for 3.0.0 GA, I'm more willing to flex the
> beta1 release date for this feature than others. Hopefully that won't be
> necessary though :)
>
> Best,
> Andrew
>
> On Wed, Aug 16, 2017 at 10:26 AM, Vrushali Channapattan <
> vrushalic2...@gmail.com> wrote:
>
>> Looks like some of the hyperlinks appear messed up, my apologies,
>> resending
>> the same email with hopefully better looking content:
>>
>> Hi All,
>>
>> I'd like to open a discussion for merging Timeline Service v2 (YARN-5355)
>> to trunk in a few weeks.
>>
>> We have previously completed one merge onto trunk [1] and Timeline Service
>> v2 has been part of Hadoop release 3.0.0-alpha1.
>>
>> Since then, we have been working on extending the capabilities of Timeline
>> Service v2 in a feature branch [2].  There are a few related issues
>> pending
>> that are being actively worked upon and tested. As soon as they are
>> resolved, we plan on starting a merge vote within the next two weeks. The
>> goal is to get this into hadoop3 beta.
>>
>> We have paid close attention to ensure that  once disabled Timeline
>> Service
>> v2 does not impact existing functionality when disabled (by default).
>>
>> At a high level, following are the key features that have been implemented
>> since 3.0.0-alpha1:
>> - Security (via Kerberos Authentication & delegation tokens) [YARN-3053]
>> - Timeline server usability improvements [timeline-server
>> > %20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-5355%20AND%
>> 20component%20%3D%20timelineserver%20AND%20labels%20!%3D%
>> 20atsv2-hbase%20ORDER%20BY%20updated%20ASC%2C%20priority%
>> 20DESC%2C%20created%20ASC>
>> ]
>> - HBase specific improvements [atsv2-hbase
>> > %20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-5355%20AND%
>> 20labels%20%3D%20atsv2-hbase%20ORDER%20BY%20updated%20DESC%
>> 2C%20affectedVersion%20DESC%2C%20priority%20DESC%2C%20created%20ASC>
>> ]
>> - REST API additions and improvements [timeline-reader
>> > %20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-5355%20AND%
>> 20component%20in%20(timelineclient%2C%20timelinere
>> ader)%20ORDER%20BY%20updated%20ASC%2C%20priority%20DESC%2C%
>> 20created%20ASC>
>> ]
>> - Reader side simple authorization via whitelist [YARN-6820]
>>
>> We would love to get your thoughts on this before we open a real voting
>> thread.
>>
>> Special thanks to a team of folks who worked hard and contributed towards
>> this effort via patches, reviews and guidance: Rohith Sharma K S, Varun
>> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
>> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
>>
>> Thanks
>> Vrushali
>> [1] Merge to trunk: http://www.mail-archive.com/yarn-dev@hadoop.apache.
>> org/msg23897.html
>> 
>> [2] feature branch YARN-5355 commits: https://github.com/ap
>> ache/hadoop/commits/YARN-5355
>>
>> 
>>
>>
>>
>>
>>
>>
>> On Wed, Aug 16, 2017 at 10:02 AM, Vrushali Channapattan <
>> vrushal...@gmail.com> wrote:
>>
>> > Hi All
>> >
>> > I’d like to open a discussion for merging Timeline Service v.2
>> (YARN-5355)
>> > to trunk in a few weeks.
>> >
>> > We have previously completed one merge onto trunk [1] and Timeline
>> Service
>> > v2 has been part of Hadoop release 3.0.0-alpha1.
>> >
>> 

Re: [DISCUSS] Merging YARN-5355 (Timeline Service v.2) to trunk

[Andrew Wang]

Hi Vrushali,

Glad to hear this major dev milestone is nearing completion!

Repeating my request on other merge [DISCUSS] threads, could you comment on
testing and API stability of this merge? Our timeline for beta1 is about a
month out, so there's not much time to fix things beforehand.

Looking at YARN-5355 there are also many unresolved subtasks. Should most
of these be moved out to a new umbrella? I'm wondering what needs to be
completed before sending the merge vote.

Given that TSv2 is committed for 3.0.0 GA, I'm more willing to flex the
beta1 release date for this feature than others. Hopefully that won't be
necessary though :)

Best,
Andrew

On Wed, Aug 16, 2017 at 10:26 AM, Vrushali Channapattan <
vrushalic2...@gmail.com> wrote:

> Looks like some of the hyperlinks appear messed up, my apologies, resending
> the same email with hopefully better looking content:
>
> Hi All,
>
> I'd like to open a discussion for merging Timeline Service v2 (YARN-5355)
> to trunk in a few weeks.
>
> We have previously completed one merge onto trunk [1] and Timeline Service
> v2 has been part of Hadoop release 3.0.0-alpha1.
>
> Since then, we have been working on extending the capabilities of Timeline
> Service v2 in a feature branch [2].  There are a few related issues pending
> that are being actively worked upon and tested. As soon as they are
> resolved, we plan on starting a merge vote within the next two weeks. The
> goal is to get this into hadoop3 beta.
>
> We have paid close attention to ensure that  once disabled Timeline Service
> v2 does not impact existing functionality when disabled (by default).
>
> At a high level, following are the key features that have been implemented
> since 3.0.0-alpha1:
> - Security (via Kerberos Authentication & delegation tokens) [YARN-3053]
> - Timeline server usability improvements [timeline-server
>  project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-
> 5355%20AND%20component%20%3D%20timelineserver%20AND%
> 20labels%20!%3D%20atsv2-hbase%20ORDER%20BY%20updated%20ASC%
> 2C%20priority%20DESC%2C%20created%20ASC>
> ]
> - HBase specific improvements [atsv2-hbase
>  project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-
> 5355%20AND%20labels%20%3D%20atsv2-hbase%20ORDER%20BY%20updated%20DESC%2C%
> 20affectedVersion%20DESC%2C%20priority%20DESC%2C%20created%20ASC>
> ]
> - REST API additions and improvements [timeline-reader
>  project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-
> 5355%20AND%20component%20in%20(timelineclient%2C%
> 20timelinereader)%20ORDER%20BY%20updated%20ASC%2C%20priority%20DESC%2C%
> 20created%20ASC>
> ]
> - Reader side simple authorization via whitelist [YARN-6820]
>
> We would love to get your thoughts on this before we open a real voting
> thread.
>
> Special thanks to a team of folks who worked hard and contributed towards
> this effort via patches, reviews and guidance: Rohith Sharma K S, Varun
> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
>
> Thanks
> Vrushali
> [1] Merge to trunk: http://www.mail-archive.com/yarn-dev@hadoop.apache.
> org/msg23897.html
> [2] feature branch YARN-5355 commits: https://github.com/ap
> ache/hadoop/commits/YARN-5355
> 
>
>
>
>
>
>
> On Wed, Aug 16, 2017 at 10:02 AM, Vrushali Channapattan <
> vrushal...@gmail.com> wrote:
>
> > Hi All
> >
> > I’d like to open a discussion for merging Timeline Service v.2
> (YARN-5355)
> > to trunk in a few weeks.
> >
> > We have previously completed one merge onto trunk [1] and Timeline
> Service
> > v2 has been part of Hadoop release 3.0.0-alpha1.
> >
> > Since then, we have been working on extending the capabilities of
> Timeline
> > Service v2 in a feature branch [2]. There are a few related issues
> pending
> > that are being actively worked upon & tested. As soon as they are
> resolved,
> > we plan on starting a merge vote within the next 2 weeks. The goal is to
> > get this in for Hadoop3 beta.
> >
> > We have paid close attention to ensure that once disabled Timeline
> Service
> > v.2 does not impact existing functionality when disabled (by default).
> >
> > At a high level, following are the key features that have been
> implemented
> > since 3.0.0-alpha1:
> > - Security (via Kerberos Authentication & delegation tokens) at the
> writer
> > [YARN-3053]
> > - Timeline server usability improvements [timeline-server
> >  > project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-
> > 5355%20AND%20component%20%3D%20timelineserver%20AND%
> > 20labels%20%21%3D%20atsv2-hbase%20ORDER%20BY%20updated%
> > 20ASC%2C%20priority%20DESC%2C%20created%20ASC>
> > ]
> > - HBase specific improvements [atsv2-hbase
> > 

Re: [DISCUSS] Merge yarn-native-services branch into trunk

[Andrew Wang]

Hi Jian,

Hadoop 3.0.0-beta1 is planned for mid-September. If the plan is to merge in
hopefully the next two weeks, that's very, very close to the goal release
date. We've already got a pile of blockers and criticals to resolve before
then.

Could you comment on testing and API stability for this branch? YARN
Federation was run at high scale and did not add new APIs, which provided a
lot of confidence in the merge.

I'll also raise the option of cutting branch-3 or branch-3.0 for the 3.0.0
efforts, and targeting this for 3.1.0.

Best,
Andrew

On Tue, Aug 15, 2017 at 1:56 PM, Jian He  wrote:

> Hi All,
> I would like to bring up the discussion of merging yarn-native-services
> branch into trunk in a few weeks. There are a few issues left under
> YARN-5079 that are being actively worked upon. As soon as they are
> resolved, we plan on start a vote hopefully in next 2 weeks. The goal is to
> get this in for hadoop3 beta.
>
> The major work in this branch include below umbrella jiras:
>  - YARN-5079. A native YARN framework (ApplicationMaster) to migrate and
> orchestrate existing services to YARN either docker or non-docker based.
>  - YARN-4793. A Rest API server for user to deploy a service via a simple
> JSON spec
>  - YARN-4757. Extending today's service registry with a simple DNS service
> to enable users to discover services deployed on YARN
>  - YARN-6419. UI support for native-services on the new YARN UI
> All these new services are optional and have to be explicitly enabled.
>
> Special thanks to a team of folks who worked hard towards this: Billie
> Rinaldi, Gour Saha, Vinod Kumar Vavilapalli, Jonathan Maron, Rohith Sharma
> K S, Sunil G, Akhil PB. This effort could not be possible without their
> ideas and hard work.
>
> Please share your thoughts. Thanks.
>
> Jian
>
>
> -
> To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
>
>

Re: [DISCUSS] Merging YARN-5355 (Timeline Service v.2) to trunk

[Vrushali Channapattan]

Looks like some of the hyperlinks appear messed up, my apologies, resending
the same email with hopefully better looking content:

Hi All,

I'd like to open a discussion for merging Timeline Service v2 (YARN-5355)
to trunk in a few weeks.

We have previously completed one merge onto trunk [1] and Timeline Service
v2 has been part of Hadoop release 3.0.0-alpha1.

Since then, we have been working on extending the capabilities of Timeline
Service v2 in a feature branch [2].  There are a few related issues pending
that are being actively worked upon and tested. As soon as they are
resolved, we plan on starting a merge vote within the next two weeks. The
goal is to get this into hadoop3 beta.

We have paid close attention to ensure that  once disabled Timeline Service
v2 does not impact existing functionality when disabled (by default).

At a high level, following are the key features that have been implemented
since 3.0.0-alpha1:
- Security (via Kerberos Authentication & delegation tokens) [YARN-3053]
- Timeline server usability improvements [timeline-server

]
- HBase specific improvements [atsv2-hbase

]
- REST API additions and improvements [timeline-reader

]
- Reader side simple authorization via whitelist [YARN-6820]

We would love to get your thoughts on this before we open a real voting
thread.

Special thanks to a team of folks who worked hard and contributed towards
this effort via patches, reviews and guidance: Rohith Sharma K S, Varun
Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.

Thanks
Vrushali
[1] Merge to trunk: http://www.mail-archive.com/yarn-dev@hadoop.apache.
org/msg23897.html
[2] feature branch YARN-5355 commits: https://github.com/ap
ache/hadoop/commits/YARN-5355







On Wed, Aug 16, 2017 at 10:02 AM, Vrushali Channapattan <
vrushal...@gmail.com> wrote:

> Hi All
>
> I’d like to open a discussion for merging Timeline Service v.2 (YARN-5355)
> to trunk in a few weeks.
>
> We have previously completed one merge onto trunk [1] and Timeline Service
> v2 has been part of Hadoop release 3.0.0-alpha1.
>
> Since then, we have been working on extending the capabilities of Timeline
> Service v2 in a feature branch [2]. There are a few related issues pending
> that are being actively worked upon & tested. As soon as they are resolved,
> we plan on starting a merge vote within the next 2 weeks. The goal is to
> get this in for Hadoop3 beta.
>
> We have paid close attention to ensure that once disabled Timeline Service
> v.2 does not impact existing functionality when disabled (by default).
>
> At a high level, following are the key features that have been implemented
> since 3.0.0-alpha1:
> - Security (via Kerberos Authentication & delegation tokens) at the writer
> [YARN-3053]
> - Timeline server usability improvements [timeline-server
>  project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-
> 5355%20AND%20component%20%3D%20timelineserver%20AND%
> 20labels%20%21%3D%20atsv2-hbase%20ORDER%20BY%20updated%
> 20ASC%2C%20priority%20DESC%2C%20created%20ASC>
> ]
> - HBase specific improvements [atsv2-hbase
>  project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-
> 5355%20AND%20labels%20%3D%20atsv2-hbase%20ORDER%20BY%20updated%20DESC%2C%
> 20affectedVersion%20DESC%2C%20priority%20DESC%2C%20created%20ASC>
> ]
> - REST API additions & improvements [timeline-reader
>  project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-
> 5355%20AND%20component%20in%20%28timelineclient%2C%
> 20timelinereader%29%20ORDER%20BY%20updated%20ASC%2C%20priority%20DESC%2C%
> 20created%20ASC>
> ]
> - Reader side simple authorization via whitelist [YARN-6820]
>
> We would love to get your thoughts on these and more before we open a real
> voting thread.
>
> Special thanks to a team of folks who worked hard and contributed towards
> this effort with patches, reviews and guidance: Rohith Sharma K S, Varun
> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
> Rottinghuis, Jason Lowe, Jian He

[DISCUSS] Merging YARN-5355 (Timeline Service v.2) to trunk

[Vrushali Channapattan]

Hi All

I’d like to open a discussion for merging Timeline Service v.2 (YARN-5355)
to trunk in a few weeks.

We have previously completed one merge onto trunk [1] and Timeline Service
v2 has been part of Hadoop release 3.0.0-alpha1.

Since then, we have been working on extending the capabilities of Timeline
Service v2 in a feature branch [2]. There are a few related issues pending
that are being actively worked upon & tested. As soon as they are resolved,
we plan on starting a merge vote within the next 2 weeks. The goal is to
get this in for Hadoop3 beta.

We have paid close attention to ensure that once disabled Timeline Service
v.2 does not impact existing functionality when disabled (by default).

At a high level, following are the key features that have been implemented
since 3.0.0-alpha1:
- Security (via Kerberos Authentication & delegation tokens) at the writer
[YARN-3053]
- Timeline server usability improvements [timeline-server

]
- HBase specific improvements [atsv2-hbase

]
- REST API additions & improvements [timeline-reader

]
- Reader side simple authorization via whitelist [YARN-6820]

We would love to get your thoughts on these and more before we open a real
voting thread.

Special thanks to a team of folks who worked hard and contributed towards
this effort with patches, reviews and guidance: Rohith Sharma K S, Varun
Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.

Thanks!
Vrushali

[1] Merge to trunk: http://www.mail-archive.com/yarn-dev@hadoop.
apache.org/msg23897.html
[2] feature branch YARN-5355 commits: https://github.com/
apache/hadoop/commits/YARN-5355

Re: Map reduce sample program

[Daniel Templeton]

Can you clarify what you mean for #1?  For #2, try this:

https://tutorials.techmytalk.com/2014/08/16/hadoop-hdfs-java-api/

Daniel

On 8/16/17 8:17 AM, Remil Mohanan wrote:







Hi there,

Please help me to get a sample program for each scenario.

1) need a Java map reducer sample program where multiple parameters
are passed from mapper to reducer.
2) need a Java map reducer program where there is a write to a file inside
hdfs filesystem as well as a read from a file inside hdfs other than
the normal input file and output file mentioned in the mapper and reducer.

Have a nice day

Thanks

Remil

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org




-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

[Apache Jenkins Server]

For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/494/

[Aug 15, 2017 8:48:49 AM] (yqlin) HDFS-11696. Fix warnings from Spotbugs in 
hadoop-hdfs. Contributed by
[Aug 15, 2017 2:38:43 PM] (weichiu) HDFS-12054. 
FSNamesystem#addErasureCodingPolicies should call
[Aug 15, 2017 2:41:43 PM] (weichiu) HDFS-12066. When Namenode is in 
safemode,may not allowed to remove an
[Aug 15, 2017 4:28:44 PM] (sunilg) YARN-5146. Support for Fair Scheduler in new 
YARN UI. Contributed by
[Aug 15, 2017 8:52:48 PM] (nroberts) YARN-7014. Fix off-by-one error causing 
heap corruption (Jason Lowe via
[Aug 15, 2017 10:44:59 PM] (raviprak) HDFS-12301. NN File Browser UI: Navigate 
to a path when enter is pressed
[Aug 15, 2017 11:53:59 PM] (subu) HADOOP-14773. Extend ZKCuratorManager API for 
more reusability. (Íñigo
[Aug 16, 2017 5:06:22 AM] (aajisaka) YARN-6965. Duplicate instantiation in 
FairSchedulerQueueInfo.




-1 overall


The following subsystems voted -1:
findbugs unit


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

FindBugs :

   
module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
 
   Hard coded reference to an absolute pathname in 
org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DockerLinuxContainerRuntime.launchContainer(ContainerRuntimeContext)
 At DockerLinuxContainerRuntime.java:absolute pathname in 
org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DockerLinuxContainerRuntime.launchContainer(ContainerRuntimeContext)
 At DockerLinuxContainerRuntime.java:[line 490] 

Failed junit tests :

   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080 
   hadoop.yarn.server.nodemanager.containermanager.TestContainerManager 
   
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation 
   hadoop.yarn.sls.appmaster.TestAMSimulator 
   hadoop.yarn.sls.nodemanager.TestNMSimulator 
  

   cc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/494/artifact/out/diff-compile-cc-root.txt
  [4.0K]

   javac:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/494/artifact/out/diff-compile-javac-root.txt
  [300K]

   checkstyle:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/494/artifact/out/diff-checkstyle-root.txt
  [17M]

   pylint:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/494/artifact/out/diff-patch-pylint.txt
  [20K]

   shellcheck:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/494/artifact/out/diff-patch-shellcheck.txt
  [20K]

   shelldocs:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/494/artifact/out/diff-patch-shelldocs.txt
  [12K]

   whitespace:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/494/artifact/out/whitespace-eol.txt
  [11M]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/494/artifact/out/whitespace-tabs.txt
  [1.2M]

   findbugs:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/494/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html
  [8.0K]

   javadoc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/494/artifact/out/diff-javadoc-javadoc-root.txt
  [1.9M]

   unit:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/494/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
  [236K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/494/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
  [40K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/494/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
  [64K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/494/artifact/out/patch-unit-hadoop-tools_hadoop-sls.txt
  [16K]

Powered by Apache Yetus 0.6.0-SNAPSHOT   http://yetus.apache.org

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Map reduce sample program

[Remil Mohanan]

> 
> 
> 
>> 
>>> 
>>> Hi there,
>>> 
>>> Please help me to get a sample program for each scenario.
>>> 
>>> 1) need a Java map reducer sample program where multiple parameters
>>> are passed from mapper to reducer.
>>> 2) need a Java map reducer program where there is a write to a file inside
>>> hdfs filesystem as well as a read from a file inside hdfs other than
>>> the normal input file and output file mentioned in the mapper and reducer.
>>> 
>>> Have a nice day
>>> 
>>> Thanks
>>> 
>>> Remil

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

[jira] [Created] (MAPREDUCE-6940) Copy-paste error in the TaskAttemptUnsuccessfulCompletionEvent constructor

[Oleg Danilov (JIRA)]

Oleg Danilov created MAPREDUCE-6940:
---

 Summary: Copy-paste error in the 
TaskAttemptUnsuccessfulCompletionEvent constructor
 Key: MAPREDUCE-6940
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6940
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Reporter: Oleg Danilov
Priority: Minor


This constructor seems to be copy-pasted from another one, but it doesn't pass 
allSplits parameter as expected.

Lines 126-133:

{code:java}
  public TaskAttemptUnsuccessfulCompletionEvent
  (TaskAttemptID id, TaskType taskType,
   String status, long finishTime,
   String hostname, int port, String rackName,
   String error, int[][] allSplits) {
this(id, taskType, status, finishTime, hostname, port,
rackName, error, EMPTY_COUNTERS, null);
  }
{code}




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org