Re: Getting close to a vote on a merge of S3Guard., HADOOP-13345

[Andrew Wang]

Thanks for the detailed explanation Aaron. Given that this has gone through
Cloudera's QA cycle and is run in production, that adds a lot of confidence
in the feature. Looking forward to having this in 3.0.0-beta1!

Best,
Andrew

On Wed, Aug 16, 2017 at 2:17 PM, Aaron Fabbri  wrote:

>
>
> On Wed, Aug 16, 2017 at 1:39 PM, Andrew Wang 
> wrote:
>
>> Hi Steve,
>>
>> What's the target release vehicle, and the timeline for merging this? The
>> target date for beta1 is mid-September, so any large code movements make
>> me
>> nervous.
>>
>
> I think this is ready to get in before beta1.  Most of upstream s3a dev
> has been happening on this branch so it has a lot of improvements and
> testing.
>
>
>> Could you comment on testing and API stability of this branch? I'm
>> trusting
>> the judgement of the contributors involved, since there isn't much time to
>> fix things before beta1.
>>
>>
> We've done a ton of testing on this branch:
>
> - List consistency tests with failure injection. (HADOOP-13793) This
> integration test forces a delay in visibility of certain files by wrapping
> the AWS S3 client. It asserts listing is consistent. The test fails without
> S3Guard, and succeeds with it.
>
> - All existing S3 integration tests with and without S3Guard. The
> filesystem contract tests have been invaluable here. (HADOOP-13589 makes
> these very easy to run).
>
> - MetadataStore contract tests that ensure that the API semantics of the
> DynamoDB and in-memory reference implementations are correct.
>
> - MetadataStore scale tests that can be used to force DynamoDB service
> throttling and ensure we are robust to that.
>
> - Unit tests for different parts of the S3Guard logic.
>
> As you probably know, at Cloudera we are using this codebase in
> production, and have run all of our downstream tests including Hive, Spark,
> Impala on the new S3A client code, with and without S3Guard enabled.
>
> In terms of API compatibility, the new features sit behind the FileSystem
> / FileContext APIs, which have not changed.  Applications don't require any
> changes.  Internal APIs for S3Guard, such as MetadataStore (currently
> private / evolving), should be properly annotated already.  The S3Guard
> work has been active for quite a while now, so the APIs are fairly stable
> in practice.
>
> Probably my biggest goal in writing the S3AFileSystem integration code
> (HADOOP-13651) was to preserve existing logic and correctness when S3Guard
> is not enabled.  One design choice which has worked well was to define a
> "null" implementation of the MetadataStore (the API that filesystem clients
> use to log metadata changes):
>
> https://github.com/apache/hadoop/blob/HADOOP-13345/
> hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/
> NullMetadataStore.java
>
> This is used in S3A by default. This made it easier to reason about
> correctness and minimized the size of the diff to the FS client as well.
>
> Other questions welcomed!
>
> Cheers,
> Aaron
>
>
>
> Best,
>> Andrew
>>
>> On Wed, Aug 16, 2017 at 5:25 AM, Steve Loughran 
>> wrote:
>>
>> >
>> > FYI, We're getting ready for a patch to merge the current S3Guard
>> branch,
>> > HADOOP-13345, via a patch https://issues.apache.org/
>> > jira/browse/HADOOP-13998
>> >
>> > After that's done, we do plan to have a second iteration, work on a
>> > 0-rename committer (HADOOP-13786) with all the other tuning and
>> > improvements; We'd add a new uber-JIRA & move stuff over, maybe branch,
>> > and/or do things patch-by-patch .
>> >
>> > Anyway, now is a great time for people to download and play
>> >
>> > https://github.com/apache/hadoop/blob/HADOOP-13345/
>> > hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/s3guard.md
>> >
>> > testing this
>> >
>> > https://github.com/apache/hadoop/blob/HADOOP-13345/
>> > hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testing.md
>> >
>> > The Inconsistent AWS Client is also something everyone is free to use
>> for
>> > injecting inconsistencies (and soon faults) into their own apps by way
>> of
>> > 2-3 config options. Want to know how your code handles S3A being
>> observably
>> > inconsistent? We'll let you do that.
>> >
>> > -Steve
>> >
>> >
>> >
>>
>
>

Re: HADOOP-14163 proposal for new hadoop.apache.org

[Akira Ajisaka]

Thanks a lot!
First I will work on the 1. and 2. in HADOOP-14163.

On 2017/08/09 5:03, Allen Wittenauer wrote:



On Aug 8, 2017, at 12:36 AM, Akira Ajisaka  wrote:

Now I'm okay with not creating another repo.
I'm thinking the following procedures may work:

1. Create ./asf-site directory
2. Add the content of https://github.com/elek/hadoop-site-proposal to the 
directory
3. Generate web pages and push them to asf-site branch
4. Create a CI job to run 3. automatically when ./asf-site directory is changed



Yup.  To be more specific on the Jenkins part:

MultiSCM build. Build should be set to poll SCM, probably @daily or 
equally reasonable.

first SCM: clone hadoop/trunk to one dir
second SCM: clone hadoop/asf-site to another dir

(Letting Jenkins manage those dirs takes quite a bit of the work out of 
it)

Run a (modified?) form of create-release so that you get an exact 
replica of what a released site looks like.

Take site tarball and unpack it into asf-site/.../trunk (or current? or 
whatever?)

build main site then commit back to asf-site

commit an empty commit to asf-site to work around  INFRA-10751.  
Recommend comment be the git hash of the current hadoop/trunk

FWIW, what we do in Yetus is we actually have the src for the main 
yetus site as part of our source tree.  It gets built as part of the release.



-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-14782) Added sw_64 CPU architecture

[Xing (JIRA)]

Xing created HADOOP-14782:
-

 Summary: Added sw_64 CPU architecture
 Key: HADOOP-14782
 URL: https://issues.apache.org/jira/browse/HADOOP-14782
 Project: Hadoop Common
  Issue Type: Improvement
 Environment: OS is Raise release 2.0.5 (Beta) using RPM package manager
CPU is SW1610
Reporter: Xing






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-14781) Clarify that HADOOP_CONF_DIR shouldn't actually be set in hadoop-env.sh

[Allen Wittenauer (JIRA)]

Allen Wittenauer created HADOOP-14781:
-

 Summary: Clarify that HADOOP_CONF_DIR shouldn't actually be set in 
hadoop-env.sh
 Key: HADOOP-14781
 URL: https://issues.apache.org/jira/browse/HADOOP-14781
 Project: Hadoop Common
  Issue Type: Improvement
  Components: scripts
Affects Versions: 3.0.0-beta1
Reporter: Allen Wittenauer






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: Getting close to a vote on a merge of S3Guard., HADOOP-13345

[Aaron Fabbri]

On Wed, Aug 16, 2017 at 1:39 PM, Andrew Wang 
wrote:

> Hi Steve,
>
> What's the target release vehicle, and the timeline for merging this? The
> target date for beta1 is mid-September, so any large code movements make me
> nervous.
>

I think this is ready to get in before beta1.  Most of upstream s3a dev has
been happening on this branch so it has a lot of improvements and testing.


> Could you comment on testing and API stability of this branch? I'm trusting
> the judgement of the contributors involved, since there isn't much time to
> fix things before beta1.
>
>
We've done a ton of testing on this branch:

- List consistency tests with failure injection. (HADOOP-13793) This
integration test forces a delay in visibility of certain files by wrapping
the AWS S3 client. It asserts listing is consistent. The test fails without
S3Guard, and succeeds with it.

- All existing S3 integration tests with and without S3Guard. The
filesystem contract tests have been invaluable here. (HADOOP-13589 makes
these very easy to run).

- MetadataStore contract tests that ensure that the API semantics of the
DynamoDB and in-memory reference implementations are correct.

- MetadataStore scale tests that can be used to force DynamoDB service
throttling and ensure we are robust to that.

- Unit tests for different parts of the S3Guard logic.

As you probably know, at Cloudera we are using this codebase in production,
and have run all of our downstream tests including Hive, Spark, Impala on
the new S3A client code, with and without S3Guard enabled.

In terms of API compatibility, the new features sit behind the FileSystem /
FileContext APIs, which have not changed.  Applications don't require any
changes.  Internal APIs for S3Guard, such as MetadataStore (currently
private / evolving), should be properly annotated already.  The S3Guard
work has been active for quite a while now, so the APIs are fairly stable
in practice.

Probably my biggest goal in writing the S3AFileSystem integration code
(HADOOP-13651) was to preserve existing logic and correctness when S3Guard
is not enabled.  One design choice which has worked well was to define a
"null" implementation of the MetadataStore (the API that filesystem clients
use to log metadata changes):

https://github.com/apache/hadoop/blob/HADOOP-13345/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/NullMetadataStore.java

This is used in S3A by default. This made it easier to reason about
correctness and minimized the size of the diff to the FS client as well.

Other questions welcomed!

Cheers,
Aaron



Best,
> Andrew
>
> On Wed, Aug 16, 2017 at 5:25 AM, Steve Loughran 
> wrote:
>
> >
> > FYI, We're getting ready for a patch to merge the current S3Guard branch,
> > HADOOP-13345, via a patch https://issues.apache.org/
> > jira/browse/HADOOP-13998
> >
> > After that's done, we do plan to have a second iteration, work on a
> > 0-rename committer (HADOOP-13786) with all the other tuning and
> > improvements; We'd add a new uber-JIRA & move stuff over, maybe branch,
> > and/or do things patch-by-patch .
> >
> > Anyway, now is a great time for people to download and play
> >
> > https://github.com/apache/hadoop/blob/HADOOP-13345/
> > hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/s3guard.md
> >
> > testing this
> >
> > https://github.com/apache/hadoop/blob/HADOOP-13345/
> > hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testing.md
> >
> > The Inconsistent AWS Client is also something everyone is free to use for
> > injecting inconsistencies (and soon faults) into their own apps by way of
> > 2-3 config options. Want to know how your code handles S3A being
> observably
> > inconsistent? We'll let you do that.
> >
> > -Steve
> >
> >
> >
>

Re: [DISCUSS] Merging YARN-5355 (Timeline Service v.2) to trunk

[Andrew Wang]

Great, thanks Vrushali! Sounds good to me.

I have a few procedural release notes comments I'll put on YARN-5355, to
make sure we advertise this to our users appropriately.

On Wed, Aug 16, 2017 at 11:32 AM, Vrushali Channapattan <
vrushal...@gmail.com> wrote:

> Hi Andrew,
>
> Thanks for your response!
>
> There have been no changes to existing APIs since alpha1.
>
> We at Twitter have tested the feature to demonstrate it works at what we
> consider moderate scale but this did not include the security related
> testing. The security testing is in progress at present by Timeline Service
> V2 team in the community and we think we will have more details on this
> very soon.
>
> About the jiras under YARN-5355: Only 3 of those sub-tasks are what we
> think of as "merge-blockers". The issues being targeted for merge are in
> [link1] below. There are about 59 jiras of which 56 are completed.
>
> We plan to make a new umbrella jira after the merge to trunk. We will then
> create a new branch with the new jira name and move these open jiras under
> YARN-5355 as subtasks of that new umbrella jira.
>
> thanks
> Vrushali
> [link1] https://issues.apache.org/jira/projects/YARN/versions/12337991
>
>
> On Wed, Aug 16, 2017 at 10:47 AM, Andrew Wang 
> wrote:
>
>> Hi Vrushali,
>>
>> Glad to hear this major dev milestone is nearing completion!
>>
>> Repeating my request on other merge [DISCUSS] threads, could you comment
>> on testing and API stability of this merge? Our timeline for beta1 is about
>> a month out, so there's not much time to fix things beforehand.
>>
>> Looking at YARN-5355 there are also many unresolved subtasks. Should most
>> of these be moved out to a new umbrella? I'm wondering what needs to be
>> completed before sending the merge vote.
>>
>> Given that TSv2 is committed for 3.0.0 GA, I'm more willing to flex the
>> beta1 release date for this feature than others. Hopefully that won't be
>> necessary though :)
>>
>> Best,
>> Andrew
>>
>> On Wed, Aug 16, 2017 at 10:26 AM, Vrushali Channapattan <
>> vrushalic2...@gmail.com> wrote:
>>
>>> Looks like some of the hyperlinks appear messed up, my apologies,
>>> resending
>>> the same email with hopefully better looking content:
>>>
>>> Hi All,
>>>
>>> I'd like to open a discussion for merging Timeline Service v2 (YARN-5355)
>>> to trunk in a few weeks.
>>>
>>> We have previously completed one merge onto trunk [1] and Timeline
>>> Service
>>> v2 has been part of Hadoop release 3.0.0-alpha1.
>>>
>>> Since then, we have been working on extending the capabilities of
>>> Timeline
>>> Service v2 in a feature branch [2].  There are a few related issues
>>> pending
>>> that are being actively worked upon and tested. As soon as they are
>>> resolved, we plan on starting a merge vote within the next two weeks. The
>>> goal is to get this into hadoop3 beta.
>>>
>>> We have paid close attention to ensure that  once disabled Timeline
>>> Service
>>> v2 does not impact existing functionality when disabled (by default).
>>>
>>> At a high level, following are the key features that have been
>>> implemented
>>> since 3.0.0-alpha1:
>>> - Security (via Kerberos Authentication & delegation tokens) [YARN-3053]
>>> - Timeline server usability improvements [timeline-server
>>> >> %20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-5355%20AND%20c
>>> omponent%20%3D%20timelineserver%20AND%20labels%20!%3D%20atsv
>>> 2-hbase%20ORDER%20BY%20updated%20ASC%2C%20priority%20DESC%
>>> 2C%20created%20ASC>
>>> ]
>>> - HBase specific improvements [atsv2-hbase
>>> >> %20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-5355%20AND%20l
>>> abels%20%3D%20atsv2-hbase%20ORDER%20BY%20updated%20DESC%2C%
>>> 20affectedVersion%20DESC%2C%20priority%20DESC%2C%20created%20ASC>
>>> ]
>>> - REST API additions and improvements [timeline-reader
>>> >> %20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-5355%20AND%20c
>>> omponent%20in%20(timelineclient%2C%20timelinereader)%
>>> 20ORDER%20BY%20updated%20ASC%2C%20priority%20DESC%2C%20created%20ASC>
>>> ]
>>> - Reader side simple authorization via whitelist [YARN-6820]
>>>
>>> We would love to get your thoughts on this before we open a real voting
>>> thread.
>>>
>>> Special thanks to a team of folks who worked hard and contributed towards
>>> this effort via patches, reviews and guidance: Rohith Sharma K S, Varun
>>> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
>>> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
>>>
>>> Thanks
>>> Vrushali
>>> [1] Merge to trunk: http://www.mail-archive.com/yarn-dev@hadoop.apache.
>>> org/msg23897.html
>>> 
>>> [2] feature branch YARN-5355 commits: https://github.com/ap
>>> ache/hadoop/commits/YARN-5355
>>>
>>> 

Re: [DISCUSS] Merging YARN-5355 (Timeline Service v.2) to trunk

[Vrushali Channapattan]

Hi Andrew,

Thanks for your response!

There have been no changes to existing APIs since alpha1.

We at Twitter have tested the feature to demonstrate it works at what we
consider moderate scale but this did not include the security related
testing. The security testing is in progress at present by Timeline Service
V2 team in the community and we think we will have more details on this
very soon.

About the jiras under YARN-5355: Only 3 of those sub-tasks are what we
think of as "merge-blockers". The issues being targeted for merge are in
[link1] below. There are about 59 jiras of which 56 are completed.

We plan to make a new umbrella jira after the merge to trunk. We will then
create a new branch with the new jira name and move these open jiras under
YARN-5355 as subtasks of that new umbrella jira.

thanks
Vrushali
[link1] https://issues.apache.org/jira/projects/YARN/versions/12337991


On Wed, Aug 16, 2017 at 10:47 AM, Andrew Wang 
wrote:

> Hi Vrushali,
>
> Glad to hear this major dev milestone is nearing completion!
>
> Repeating my request on other merge [DISCUSS] threads, could you comment
> on testing and API stability of this merge? Our timeline for beta1 is about
> a month out, so there's not much time to fix things beforehand.
>
> Looking at YARN-5355 there are also many unresolved subtasks. Should most
> of these be moved out to a new umbrella? I'm wondering what needs to be
> completed before sending the merge vote.
>
> Given that TSv2 is committed for 3.0.0 GA, I'm more willing to flex the
> beta1 release date for this feature than others. Hopefully that won't be
> necessary though :)
>
> Best,
> Andrew
>
> On Wed, Aug 16, 2017 at 10:26 AM, Vrushali Channapattan <
> vrushalic2...@gmail.com> wrote:
>
>> Looks like some of the hyperlinks appear messed up, my apologies,
>> resending
>> the same email with hopefully better looking content:
>>
>> Hi All,
>>
>> I'd like to open a discussion for merging Timeline Service v2 (YARN-5355)
>> to trunk in a few weeks.
>>
>> We have previously completed one merge onto trunk [1] and Timeline Service
>> v2 has been part of Hadoop release 3.0.0-alpha1.
>>
>> Since then, we have been working on extending the capabilities of Timeline
>> Service v2 in a feature branch [2].  There are a few related issues
>> pending
>> that are being actively worked upon and tested. As soon as they are
>> resolved, we plan on starting a merge vote within the next two weeks. The
>> goal is to get this into hadoop3 beta.
>>
>> We have paid close attention to ensure that  once disabled Timeline
>> Service
>> v2 does not impact existing functionality when disabled (by default).
>>
>> At a high level, following are the key features that have been implemented
>> since 3.0.0-alpha1:
>> - Security (via Kerberos Authentication & delegation tokens) [YARN-3053]
>> - Timeline server usability improvements [timeline-server
>> > %20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-5355%20AND%
>> 20component%20%3D%20timelineserver%20AND%20labels%20!%3D%
>> 20atsv2-hbase%20ORDER%20BY%20updated%20ASC%2C%20priority%
>> 20DESC%2C%20created%20ASC>
>> ]
>> - HBase specific improvements [atsv2-hbase
>> > %20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-5355%20AND%
>> 20labels%20%3D%20atsv2-hbase%20ORDER%20BY%20updated%20DESC%
>> 2C%20affectedVersion%20DESC%2C%20priority%20DESC%2C%20created%20ASC>
>> ]
>> - REST API additions and improvements [timeline-reader
>> > %20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-5355%20AND%
>> 20component%20in%20(timelineclient%2C%20timelinere
>> ader)%20ORDER%20BY%20updated%20ASC%2C%20priority%20DESC%2C%
>> 20created%20ASC>
>> ]
>> - Reader side simple authorization via whitelist [YARN-6820]
>>
>> We would love to get your thoughts on this before we open a real voting
>> thread.
>>
>> Special thanks to a team of folks who worked hard and contributed towards
>> this effort via patches, reviews and guidance: Rohith Sharma K S, Varun
>> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
>> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
>>
>> Thanks
>> Vrushali
>> [1] Merge to trunk: http://www.mail-archive.com/yarn-dev@hadoop.apache.
>> org/msg23897.html
>> 
>> [2] feature branch YARN-5355 commits: https://github.com/ap
>> ache/hadoop/commits/YARN-5355
>>
>> 
>>
>>
>>
>>
>>
>>
>> On Wed, Aug 16, 2017 at 10:02 AM, Vrushali Channapattan <
>> vrushal...@gmail.com> wrote:
>>
>> > Hi All
>> >
>> > I’d like to open a discussion for merging Timeline Service v.2
>> (YARN-5355)
>> > to trunk in a few weeks.
>> >
>> > We have previously completed one merge onto trunk [1] and Timeline
>> Service
>> > v2 has been part of Hadoop 

Re: [DISCUSS] Merging YARN-5355 (Timeline Service v.2) to trunk

[Andrew Wang]

Hi Vrushali,

Glad to hear this major dev milestone is nearing completion!

Repeating my request on other merge [DISCUSS] threads, could you comment on
testing and API stability of this merge? Our timeline for beta1 is about a
month out, so there's not much time to fix things beforehand.

Looking at YARN-5355 there are also many unresolved subtasks. Should most
of these be moved out to a new umbrella? I'm wondering what needs to be
completed before sending the merge vote.

Given that TSv2 is committed for 3.0.0 GA, I'm more willing to flex the
beta1 release date for this feature than others. Hopefully that won't be
necessary though :)

Best,
Andrew

On Wed, Aug 16, 2017 at 10:26 AM, Vrushali Channapattan <
vrushalic2...@gmail.com> wrote:

> Looks like some of the hyperlinks appear messed up, my apologies, resending
> the same email with hopefully better looking content:
>
> Hi All,
>
> I'd like to open a discussion for merging Timeline Service v2 (YARN-5355)
> to trunk in a few weeks.
>
> We have previously completed one merge onto trunk [1] and Timeline Service
> v2 has been part of Hadoop release 3.0.0-alpha1.
>
> Since then, we have been working on extending the capabilities of Timeline
> Service v2 in a feature branch [2].  There are a few related issues pending
> that are being actively worked upon and tested. As soon as they are
> resolved, we plan on starting a merge vote within the next two weeks. The
> goal is to get this into hadoop3 beta.
>
> We have paid close attention to ensure that  once disabled Timeline Service
> v2 does not impact existing functionality when disabled (by default).
>
> At a high level, following are the key features that have been implemented
> since 3.0.0-alpha1:
> - Security (via Kerberos Authentication & delegation tokens) [YARN-3053]
> - Timeline server usability improvements [timeline-server
>  project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-
> 5355%20AND%20component%20%3D%20timelineserver%20AND%
> 20labels%20!%3D%20atsv2-hbase%20ORDER%20BY%20updated%20ASC%
> 2C%20priority%20DESC%2C%20created%20ASC>
> ]
> - HBase specific improvements [atsv2-hbase
>  project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-
> 5355%20AND%20labels%20%3D%20atsv2-hbase%20ORDER%20BY%20updated%20DESC%2C%
> 20affectedVersion%20DESC%2C%20priority%20DESC%2C%20created%20ASC>
> ]
> - REST API additions and improvements [timeline-reader
>  project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-
> 5355%20AND%20component%20in%20(timelineclient%2C%
> 20timelinereader)%20ORDER%20BY%20updated%20ASC%2C%20priority%20DESC%2C%
> 20created%20ASC>
> ]
> - Reader side simple authorization via whitelist [YARN-6820]
>
> We would love to get your thoughts on this before we open a real voting
> thread.
>
> Special thanks to a team of folks who worked hard and contributed towards
> this effort via patches, reviews and guidance: Rohith Sharma K S, Varun
> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
>
> Thanks
> Vrushali
> [1] Merge to trunk: http://www.mail-archive.com/yarn-dev@hadoop.apache.
> org/msg23897.html
> [2] feature branch YARN-5355 commits: https://github.com/ap
> ache/hadoop/commits/YARN-5355
> 
>
>
>
>
>
>
> On Wed, Aug 16, 2017 at 10:02 AM, Vrushali Channapattan <
> vrushal...@gmail.com> wrote:
>
> > Hi All
> >
> > I’d like to open a discussion for merging Timeline Service v.2
> (YARN-5355)
> > to trunk in a few weeks.
> >
> > We have previously completed one merge onto trunk [1] and Timeline
> Service
> > v2 has been part of Hadoop release 3.0.0-alpha1.
> >
> > Since then, we have been working on extending the capabilities of
> Timeline
> > Service v2 in a feature branch [2]. There are a few related issues
> pending
> > that are being actively worked upon & tested. As soon as they are
> resolved,
> > we plan on starting a merge vote within the next 2 weeks. The goal is to
> > get this in for Hadoop3 beta.
> >
> > We have paid close attention to ensure that once disabled Timeline
> Service
> > v.2 does not impact existing functionality when disabled (by default).
> >
> > At a high level, following are the key features that have been
> implemented
> > since 3.0.0-alpha1:
> > - Security (via Kerberos Authentication & delegation tokens) at the
> writer
> > [YARN-3053]
> > - Timeline server usability improvements [timeline-server
> >  > project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-
> > 5355%20AND%20component%20%3D%20timelineserver%20AND%
> > 20labels%20%21%3D%20atsv2-hbase%20ORDER%20BY%20updated%
> > 20ASC%2C%20priority%20DESC%2C%20created%20ASC>
> > ]
> > - HBase specific improvements [atsv2-hbase
> > 

Re: Getting close to a vote on a merge of S3Guard., HADOOP-13345

[Andrew Wang]

Hi Steve,

What's the target release vehicle, and the timeline for merging this? The
target date for beta1 is mid-September, so any large code movements make me
nervous.

Could you comment on testing and API stability of this branch? I'm trusting
the judgement of the contributors involved, since there isn't much time to
fix things before beta1.

Best,
Andrew

On Wed, Aug 16, 2017 at 5:25 AM, Steve Loughran 
wrote:

>
> FYI, We're getting ready for a patch to merge the current S3Guard branch,
> HADOOP-13345, via a patch https://issues.apache.org/
> jira/browse/HADOOP-13998
>
> After that's done, we do plan to have a second iteration, work on a
> 0-rename committer (HADOOP-13786) with all the other tuning and
> improvements; We'd add a new uber-JIRA & move stuff over, maybe branch,
> and/or do things patch-by-patch .
>
> Anyway, now is a great time for people to download and play
>
> https://github.com/apache/hadoop/blob/HADOOP-13345/
> hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/s3guard.md
>
> testing this
>
> https://github.com/apache/hadoop/blob/HADOOP-13345/
> hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testing.md
>
> The Inconsistent AWS Client is also something everyone is free to use for
> injecting inconsistencies (and soon faults) into their own apps by way of
> 2-3 config options. Want to know how your code handles S3A being observably
> inconsistent? We'll let you do that.
>
> -Steve
>
>
>

Re: [DISCUSS] Merge yarn-native-services branch into trunk

[Andrew Wang]

Hi Jian,

Hadoop 3.0.0-beta1 is planned for mid-September. If the plan is to merge in
hopefully the next two weeks, that's very, very close to the goal release
date. We've already got a pile of blockers and criticals to resolve before
then.

Could you comment on testing and API stability for this branch? YARN
Federation was run at high scale and did not add new APIs, which provided a
lot of confidence in the merge.

I'll also raise the option of cutting branch-3 or branch-3.0 for the 3.0.0
efforts, and targeting this for 3.1.0.

Best,
Andrew

On Tue, Aug 15, 2017 at 1:56 PM, Jian He  wrote:

> Hi All,
> I would like to bring up the discussion of merging yarn-native-services
> branch into trunk in a few weeks. There are a few issues left under
> YARN-5079 that are being actively worked upon. As soon as they are
> resolved, we plan on start a vote hopefully in next 2 weeks. The goal is to
> get this in for hadoop3 beta.
>
> The major work in this branch include below umbrella jiras:
>  - YARN-5079. A native YARN framework (ApplicationMaster) to migrate and
> orchestrate existing services to YARN either docker or non-docker based.
>  - YARN-4793. A Rest API server for user to deploy a service via a simple
> JSON spec
>  - YARN-4757. Extending today's service registry with a simple DNS service
> to enable users to discover services deployed on YARN
>  - YARN-6419. UI support for native-services on the new YARN UI
> All these new services are optional and have to be explicitly enabled.
>
> Special thanks to a team of folks who worked hard towards this: Billie
> Rinaldi, Gour Saha, Vinod Kumar Vavilapalli, Jonathan Maron, Rohith Sharma
> K S, Sunil G, Akhil PB. This effort could not be possible without their
> ideas and hard work.
>
> Please share your thoughts. Thanks.
>
> Jian
>
>
> -
> To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
>
>

Re: [DISCUSS] Merging YARN-5355 (Timeline Service v.2) to trunk

[Vrushali Channapattan]

Looks like some of the hyperlinks appear messed up, my apologies, resending
the same email with hopefully better looking content:

Hi All,

I'd like to open a discussion for merging Timeline Service v2 (YARN-5355)
to trunk in a few weeks.

We have previously completed one merge onto trunk [1] and Timeline Service
v2 has been part of Hadoop release 3.0.0-alpha1.

Since then, we have been working on extending the capabilities of Timeline
Service v2 in a feature branch [2].  There are a few related issues pending
that are being actively worked upon and tested. As soon as they are
resolved, we plan on starting a merge vote within the next two weeks. The
goal is to get this into hadoop3 beta.

We have paid close attention to ensure that  once disabled Timeline Service
v2 does not impact existing functionality when disabled (by default).

At a high level, following are the key features that have been implemented
since 3.0.0-alpha1:
- Security (via Kerberos Authentication & delegation tokens) [YARN-3053]
- Timeline server usability improvements [timeline-server

]
- HBase specific improvements [atsv2-hbase

]
- REST API additions and improvements [timeline-reader

]
- Reader side simple authorization via whitelist [YARN-6820]

We would love to get your thoughts on this before we open a real voting
thread.

Special thanks to a team of folks who worked hard and contributed towards
this effort via patches, reviews and guidance: Rohith Sharma K S, Varun
Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.

Thanks
Vrushali
[1] Merge to trunk: http://www.mail-archive.com/yarn-dev@hadoop.apache.
org/msg23897.html
[2] feature branch YARN-5355 commits: https://github.com/ap
ache/hadoop/commits/YARN-5355







On Wed, Aug 16, 2017 at 10:02 AM, Vrushali Channapattan <
vrushal...@gmail.com> wrote:

> Hi All
>
> I’d like to open a discussion for merging Timeline Service v.2 (YARN-5355)
> to trunk in a few weeks.
>
> We have previously completed one merge onto trunk [1] and Timeline Service
> v2 has been part of Hadoop release 3.0.0-alpha1.
>
> Since then, we have been working on extending the capabilities of Timeline
> Service v2 in a feature branch [2]. There are a few related issues pending
> that are being actively worked upon & tested. As soon as they are resolved,
> we plan on starting a merge vote within the next 2 weeks. The goal is to
> get this in for Hadoop3 beta.
>
> We have paid close attention to ensure that once disabled Timeline Service
> v.2 does not impact existing functionality when disabled (by default).
>
> At a high level, following are the key features that have been implemented
> since 3.0.0-alpha1:
> - Security (via Kerberos Authentication & delegation tokens) at the writer
> [YARN-3053]
> - Timeline server usability improvements [timeline-server
>  project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-
> 5355%20AND%20component%20%3D%20timelineserver%20AND%
> 20labels%20%21%3D%20atsv2-hbase%20ORDER%20BY%20updated%
> 20ASC%2C%20priority%20DESC%2C%20created%20ASC>
> ]
> - HBase specific improvements [atsv2-hbase
>  project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-
> 5355%20AND%20labels%20%3D%20atsv2-hbase%20ORDER%20BY%20updated%20DESC%2C%
> 20affectedVersion%20DESC%2C%20priority%20DESC%2C%20created%20ASC>
> ]
> - REST API additions & improvements [timeline-reader
>  project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-
> 5355%20AND%20component%20in%20%28timelineclient%2C%
> 20timelinereader%29%20ORDER%20BY%20updated%20ASC%2C%20priority%20DESC%2C%
> 20created%20ASC>
> ]
> - Reader side simple authorization via whitelist [YARN-6820]
>
> We would love to get your thoughts on these and more before we open a real
> voting thread.
>
> Special thanks to a team of folks who worked hard and contributed towards
> this effort with patches, reviews and guidance: Rohith Sharma K S, Varun
> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
> Rottinghuis, Jason Lowe, Jian 

[jira] [Created] (HADOOP-14779) Refactor decryptEncryptedKey in KeyProviderCryptoExtension

[Xiao Chen (JIRA)]

Xiao Chen created HADOOP-14779:
--

 Summary: Refactor decryptEncryptedKey in KeyProviderCryptoExtension
 Key: HADOOP-14779
 URL: https://issues.apache.org/jira/browse/HADOOP-14779
 Project: Hadoop Common
  Issue Type: Improvement
  Components: kms
Affects Versions: 2.6.0
Reporter: Xiao Chen
Assignee: Xiao Chen
Priority: Minor


We could separate out the actual decrypt logic from the 
{{decryptEncryptedKey}}. This enables reencrypt calls to possibly reuse the 
codec.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[DISCUSS] Merging YARN-5355 (Timeline Service v.2) to trunk

[Vrushali Channapattan]

Hi All

I’d like to open a discussion for merging Timeline Service v.2 (YARN-5355)
to trunk in a few weeks.

We have previously completed one merge onto trunk [1] and Timeline Service
v2 has been part of Hadoop release 3.0.0-alpha1.

Since then, we have been working on extending the capabilities of Timeline
Service v2 in a feature branch [2]. There are a few related issues pending
that are being actively worked upon & tested. As soon as they are resolved,
we plan on starting a merge vote within the next 2 weeks. The goal is to
get this in for Hadoop3 beta.

We have paid close attention to ensure that once disabled Timeline Service
v.2 does not impact existing functionality when disabled (by default).

At a high level, following are the key features that have been implemented
since 3.0.0-alpha1:
- Security (via Kerberos Authentication & delegation tokens) at the writer
[YARN-3053]
- Timeline server usability improvements [timeline-server

]
- HBase specific improvements [atsv2-hbase

]
- REST API additions & improvements [timeline-reader

]
- Reader side simple authorization via whitelist [YARN-6820]

We would love to get your thoughts on these and more before we open a real
voting thread.

Special thanks to a team of folks who worked hard and contributed towards
this effort with patches, reviews and guidance: Rohith Sharma K S, Varun
Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.

Thanks!
Vrushali

[1] Merge to trunk: http://www.mail-archive.com/yarn-dev@hadoop.
apache.org/msg23897.html
[2] feature branch YARN-5355 commits: https://github.com/
apache/hadoop/commits/YARN-5355

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

[Apache Jenkins Server]

For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/494/

[Aug 15, 2017 8:48:49 AM] (yqlin) HDFS-11696. Fix warnings from Spotbugs in 
hadoop-hdfs. Contributed by
[Aug 15, 2017 2:38:43 PM] (weichiu) HDFS-12054. 
FSNamesystem#addErasureCodingPolicies should call
[Aug 15, 2017 2:41:43 PM] (weichiu) HDFS-12066. When Namenode is in 
safemode,may not allowed to remove an
[Aug 15, 2017 4:28:44 PM] (sunilg) YARN-5146. Support for Fair Scheduler in new 
YARN UI. Contributed by
[Aug 15, 2017 8:52:48 PM] (nroberts) YARN-7014. Fix off-by-one error causing 
heap corruption (Jason Lowe via
[Aug 15, 2017 10:44:59 PM] (raviprak) HDFS-12301. NN File Browser UI: Navigate 
to a path when enter is pressed
[Aug 15, 2017 11:53:59 PM] (subu) HADOOP-14773. Extend ZKCuratorManager API for 
more reusability. (Íñigo
[Aug 16, 2017 5:06:22 AM] (aajisaka) YARN-6965. Duplicate instantiation in 
FairSchedulerQueueInfo.




-1 overall


The following subsystems voted -1:
findbugs unit


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

FindBugs :

   
module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
 
   Hard coded reference to an absolute pathname in 
org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DockerLinuxContainerRuntime.launchContainer(ContainerRuntimeContext)
 At DockerLinuxContainerRuntime.java:absolute pathname in 
org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DockerLinuxContainerRuntime.launchContainer(ContainerRuntimeContext)
 At DockerLinuxContainerRuntime.java:[line 490] 

Failed junit tests :

   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080 
   hadoop.yarn.server.nodemanager.containermanager.TestContainerManager 
   
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation 
   hadoop.yarn.sls.appmaster.TestAMSimulator 
   hadoop.yarn.sls.nodemanager.TestNMSimulator 
  

   cc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/494/artifact/out/diff-compile-cc-root.txt
  [4.0K]

   javac:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/494/artifact/out/diff-compile-javac-root.txt
  [300K]

   checkstyle:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/494/artifact/out/diff-checkstyle-root.txt
  [17M]

   pylint:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/494/artifact/out/diff-patch-pylint.txt
  [20K]

   shellcheck:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/494/artifact/out/diff-patch-shellcheck.txt
  [20K]

   shelldocs:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/494/artifact/out/diff-patch-shelldocs.txt
  [12K]

   whitespace:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/494/artifact/out/whitespace-eol.txt
  [11M]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/494/artifact/out/whitespace-tabs.txt
  [1.2M]

   findbugs:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/494/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html
  [8.0K]

   javadoc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/494/artifact/out/diff-javadoc-javadoc-root.txt
  [1.9M]

   unit:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/494/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
  [236K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/494/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
  [40K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/494/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
  [64K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/494/artifact/out/patch-unit-hadoop-tools_hadoop-sls.txt
  [16K]

Powered by Apache Yetus 0.6.0-SNAPSHOT   http://yetus.apache.org

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-14778) AzureNativeFileSystemStore.getDirectorySet() to use getTrimmedStrings to get directory listings

[Steve Loughran (JIRA)]

Steve Loughran created HADOOP-14778:
---

 Summary: AzureNativeFileSystemStore.getDirectorySet() to use 
getTrimmedStrings to get directory listings
 Key: HADOOP-14778
 URL: https://issues.apache.org/jira/browse/HADOOP-14778
 Project: Hadoop Common
  Issue Type: Sub-task
  Components: fs/azure
Affects Versions: 2.8.1
Reporter: Steve Loughran
Priority: Minor


{{AzureNativeFileSystemStore.getDirectorySet()}} should switch to 
{{Configuration.getTrimmedStrings()}} so the list of directories get whitespace 
and line endings removed. 

As it stands, all paths need to go one the same line, without whitespace



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-14777) S3Guard premerge changes: java 7 build & test tuning

[Steve Loughran (JIRA)]

Steve Loughran created HADOOP-14777:
---

 Summary: S3Guard premerge changes: java 7 build & test tuning
 Key: HADOOP-14777
 URL: https://issues.apache.org/jira/browse/HADOOP-14777
 Project: Hadoop Common
  Issue Type: Sub-task
  Components: fs/s3, test
Affects Versions: 3.0.0-beta1
Reporter: Steve Loughran
Assignee: Steve Loughran
Priority: Minor


Another set of changes for S3Guard in preparation for merging via HADOOP-13998

* checkstyle issues
* Made Java 7 friendly (indeed, tested applied to branch-2 with some POM 
changes & tested there)
* improve diagnostics on some test failure. This would address HADOOP-14750.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Getting close to a vote on a merge of S3Guard., HADOOP-13345

[Steve Loughran]

FYI, We're getting ready for a patch to merge the current S3Guard branch, 
HADOOP-13345, via a patch https://issues.apache.org/jira/browse/HADOOP-13998

After that's done, we do plan to have a second iteration, work on a 0-rename 
committer (HADOOP-13786) with all the other tuning and improvements; We'd add a 
new uber-JIRA & move stuff over, maybe branch, and/or do things patch-by-patch .

Anyway, now is a great time for people to download and play

https://github.com/apache/hadoop/blob/HADOOP-13345/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/s3guard.md

testing this

https://github.com/apache/hadoop/blob/HADOOP-13345/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testing.md

The Inconsistent AWS Client is also something everyone is free to use for 
injecting inconsistencies (and soon faults) into their own apps by way of 2-3 
config options. Want to know how your code handles S3A being observably 
inconsistent? We'll let you do that.

-Steve