[jira] [Created] (HADOOP-15754) s3guard: testDynamoTableTagging should clear existing config

[Aaron Fabbri (JIRA)]

Aaron Fabbri created HADOOP-15754:
-

 Summary: s3guard: testDynamoTableTagging should clear existing 
config
 Key: HADOOP-15754
 URL: https://issues.apache.org/jira/browse/HADOOP-15754
 Project: Hadoop Common
  Issue Type: Bug
  Components: fs/s3
Affects Versions: 3.2.0
Reporter: Aaron Fabbri


I recently committed HADOOP-14734 which adds support for tagging Dynamo DB 
tables for S3Guard when they are created.

 

Later, when testing another patch, I hit a test failure because I still had a 
tag option set in my test configuration (auth-keys.xml) that was adding my own 
table tag.
{noformat}
[ERROR] 
testDynamoTableTagging(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardToolDynamoDB)
  Time elapsed: 13.384 s  <<< FAILURE!

java.lang.AssertionError: expected:<2> but was:<3>

        at org.junit.Assert.fail(Assert.java:88)

        at org.junit.Assert.failNotEquals(Assert.java:743)

        at org.junit.Assert.assertEquals(Assert.java:118)

        at org.junit.Assert.assertEquals(Assert.java:555)

        at org.junit.Assert.assertEquals(Assert.java:542)

        at 
org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardToolDynamoDB.testDynamoTableTagging(ITestS3GuardToolDynamoDB.java:129)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:498)

        at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)

        at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)

        at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)

        at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)

        at 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)

        at 
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)

        at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)

        at 
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74){noformat}
I think the solution is just to clear any tag.* options set in the 
configuration at the beginning of the test.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-15753) ABFS: support path "abfs://mycluster/file/path"

[Da Zhou (JIRA)]

Da Zhou created HADOOP-15753:


 Summary: ABFS: support path "abfs://mycluster/file/path"
 Key: HADOOP-15753
 URL: https://issues.apache.org/jira/browse/HADOOP-15753
 Project: Hadoop Common
  Issue Type: Sub-task
Reporter: Da Zhou
Assignee: Da Zhou


WASB support path format: "wasb://mycluster/file/path", but ABFS doesn't, which 
caused some issues for customer. I will add support for this path format.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 2.8.5 (RC0)

[Kitti Nánási]

Thanks for working on this Junping!

+1 (non-binding)

- checked out git tag release-2.8.5-RC0
- built from source on Mac OS X 10.13.4, java version 8.0.172-zulu
- deployed on a 3 node cluster
- ran terasort, teragen, teravalidate with success
- executed basic hdfs and dfsadmin commands

Thanks,
Kitti


On Wed, Sep 12, 2018 at 2:37 PM, Yesha Vora  wrote:

> +1 (binding)
>
> Tested the following:
> - launch Pseudo-Distributed 1 node cluster
> - Tested with word count / teragen / terasort / nnbench/ mrbench on yarn
> - Validate yarn UI
> - Test basic yarn cli operations such as -kill, -status, -appStates,
> -appTypes
>
> --
> Yesha Vora
>
>
> 
> From: 俊平堵 
> Sent: Monday, September 10, 2018 5:00 AM
> To: Hadoop Common; Hdfs-dev; mapreduce-...@hadoop.apache.org;
> yarn-...@hadoop.apache.org
> Cc: junpingdu(堵俊平)
> Subject: [VOTE] Release Apache Hadoop 2.8.5 (RC0)
>
> Hi all,
>
>  I've created the first release candidate (RC0) for Apache
> Hadoop 2.8.5. This is our next point release to follow up 2.8.4. It
> includes 33 important fixes and improvements.
>
>
> The RC artifacts are available at:
> http://home.apache.org/~junping_du/hadoop-2.8.5-RC0
>
>
> The RC tag in git is: release-2.8.5-RC0
>
>
>
> The maven artifacts are available via repository.apache.org<
> http://repository.apache.org> at:
>
> https://repository.apache.org/content/repositories/orgapachehadoop-1140
>
>
> Please try the release and vote; the vote will run for the usual 5
> working
> days, ending on 9/15/2018 PST time.
>
>
> Thanks,
>
>
> Junping
>
>
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>
>

Re: [VOTE] Release Apache Hadoop 2.8.5 (RC0)

[Yesha Vora]

+1 (binding)

Tested the following:
- launch Pseudo-Distributed 1 node cluster
- Tested with word count / teragen / terasort / nnbench/ mrbench on yarn 
- Validate yarn UI
- Test basic yarn cli operations such as -kill, -status, -appStates, -appTypes

--
Yesha Vora



From: 俊平堵 
Sent: Monday, September 10, 2018 5:00 AM
To: Hadoop Common; Hdfs-dev; mapreduce-...@hadoop.apache.org; 
yarn-...@hadoop.apache.org
Cc: junpingdu(堵俊平)
Subject: [VOTE] Release Apache Hadoop 2.8.5 (RC0)

Hi all,

 I've created the first release candidate (RC0) for Apache
Hadoop 2.8.5. This is our next point release to follow up 2.8.4. It
includes 33 important fixes and improvements.


The RC artifacts are available at:
http://home.apache.org/~junping_du/hadoop-2.8.5-RC0


The RC tag in git is: release-2.8.5-RC0



The maven artifacts are available via repository.apache.org<
http://repository.apache.org> at:

https://repository.apache.org/content/repositories/orgapachehadoop-1140


Please try the release and vote; the vote will run for the usual 5 working
days, ending on 9/15/2018 PST time.


Thanks,


Junping


-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [Vote] Merge discussion for Node attribute support feature YARN-3409

[Shane Kumpf]

I just committed YARN-8768 to trunk. Thanks for the report and quick fix.

Thanks,
-Shane

On Wed, Sep 12, 2018 at 2:28 PM Naganarasimha Garla <
naganarasimha...@apache.org> wrote:

> Thanks for reporting Sean and Sunil for acting on it. My bad i had used
> alias to compile which was skipping the javadoc !
> Will try to commit as soon as the jenkins results are out.
>
> On Thu, Sep 13, 2018 at 1:18 AM Sunil G  wrote:
>
> > Thanks Sean
> >
> > YARN-8768 is raised and we will get this in very soon.
> >
> > - Sunil
> >
> > On Wed, Sep 12, 2018 at 9:16 PM Sean Mackrory 
> > wrote:
> >
> >> `mvn install` fails on trunk right now due to javadoc error in yarn-api.
> >> Is
> >> that related? If so, can we revert or fix ASAP please?
> >>
> >> On Wed, Sep 12, 2018 at 6:05 AM Naganarasimha Garla <
> >> naganarasimha...@apache.org> wrote:
> >>
> >> > Hi All,
> >> >  Me and Sunil have successfully merged YARN-3409 to the trunk
> >> and
> >> > we have created new umbrella jira YARN-8766 to track the pending jiras
> >> and
> >> > the remaining planned improvement features.
> >> > Thanks all for the support !
> >> >
> >> > Regards,
> >> > + Naga
> >> >
> >> >
> >> > On Wed, Sep 12, 2018 at 7:46 AM Naganarasimha Garla <
> >> > naganarasimha...@apache.org> wrote:
> >> >
> >> > > Hi All,
> >> > >  Voting has been running since 6 days and adding my vote we
> >> have
> >> > 4
> >> > > binding and 2 non binding +1's with no -1's this voting passes and
> we
> >> > will
> >> > > be merging the branch shortly. Thanks for all who participated in
> the
> >> > > discussion and voting thread !
> >> > >
> >> > > Thanks and Regards,
> >> > > + Naga
> >> > >
> >> > > On Mon, Sep 10, 2018 at 2:50 PM Zian Chen 
> >> wrote:
> >> > >
> >> > >> +1 for merge.
> >> > >>
> >> > >> > On Sep 9, 2018, at 10:47 PM, Weiwei Yang 
> >> wrote:
> >> > >> >
> >> > >> > +1 for the merge
> >> > >> >
> >> > >> > On Mon, Sep 10, 2018 at 12:06 PM Rohith Sharma K S <
> >> > >> > rohithsharm...@apache.org> wrote:
> >> > >> >
> >> > >> >> +1 for merge
> >> > >> >>
> >> > >> >> -Rohith Sharma K S
> >> > >> >>
> >> > >> >> On Wed, 5 Sep 2018 at 18:01, Naganarasimha Garla <
> >> > >> >> naganarasimha...@apache.org> wrote:
> >> > >> >>
> >> > >> >>> Hi All,
> >> > >> >>>Thanks for feedback folks, based on the positive
> response
> >> > >> >> starting
> >> > >> >>> a Vote thread for merging YARN-3409 to master.
> >> > >> >>>
> >> > >> >>> Regards,
> >> > >> >>> + Naga & Sunil
> >> > >> >>>
> >> > >> >>> On Wed, 5 Sep 2018 2:51 am Wangda Tan, 
> >> wrote:
> >> > >> >>>
> >> > >>  +1 for the merge, it gonna be a great addition to 3.2.0
> release.
> >> > >> Thanks
> >> > >> >>> to
> >> > >>  everybody for pushing this feature to complete.
> >> > >> 
> >> > >>  Best,
> >> > >>  Wangda
> >> > >> 
> >> > >>  On Tue, Sep 4, 2018 at 8:25 AM Bibinchundatt <
> >> > >> >> bibin.chund...@huawei.com>
> >> > >>  wrote:
> >> > >> 
> >> > >> > +1 for merge. Fetaure would be a good addition to 3.2
> release.
> >> > >> >
> >> > >> > --
> >> > >> > Bibin A Chundatt
> >> > >> > M: +91-9742095715 <+91%2097420%2095715>
> >> > >> > E: bibin.chund...@huawei.com bibin.chund...@huawei.com>
> >> > >> > 2012实验室-印研IT BU分部
> >> > >> > 2012 Laboratories-IT BU Branch Dept.
> >> > >> > From:Naganarasimha Garla
> >> > >> > To:common-dev@hadoop.apache.org,Hdfs-dev,
> >> > yarn-...@hadoop.apache.org
> >> > >> ,
> >> > >> > mapreduce-...@hadoop.apache.org,
> >> > >> > Date:2018-08-29 20:00:44
> >> > >> > Subject:[Discuss] Merge discussion for Node attribute support
> >> > >> feature
> >> > >> > YARN-3409
> >> > >> >
> >> > >> > Hi All,
> >> > >> >
> >> > >> > We would like to hear your thoughts on merging “Node
> Attributes
> >> > >> >> Support
> >> > >> >>> in
> >> > >> > YARN” branch (YARN-3409) [2] into trunk in a few weeks. The
> >> goal
> >> > is
> >> > >> to
> >> > >> >>> get
> >> > >> > it in for HADOOP 3.2.
> >> > >> >
> >> > >> > *Major work happened in this branch*
> >> > >> >
> >> > >> > YARN-6858. Attribute Manager to store and provide node
> >> attributes
> >> > in
> >> > >> >> RM
> >> > >> > YARN-7871. Support Node attributes reporting from NM to RM(
> >> > >> >> distributed
> >> > >> > node attributes)
> >> > >> > YARN-7863. Modify placement constraints to support node
> >> attributes
> >> > >> > YARN-7875. Node Attribute store for storing and recovering
> >> > >> attributes
> >> > >> >
> >> > >> > *Detailed Design:*
> >> > >> >
> >> > >> > Please refer [1] for detailed design document.
> >> > >> >
> >> > >> > *Testing Efforts:*
> >> > >> >
> >> > >> > We did detailed tests for the feature in the last few weeks.
> >> > >> > This feature will be enabled only when Node Attributes
> >> constraints
> >> > >> are
> >> > >> > specified 

Re: [Vote] Merge discussion for Node attribute support feature YARN-3409

[Naganarasimha Garla]

Thanks for reporting Sean and Sunil for acting on it. My bad i had used
alias to compile which was skipping the javadoc !
Will try to commit as soon as the jenkins results are out.

On Thu, Sep 13, 2018 at 1:18 AM Sunil G  wrote:

> Thanks Sean
>
> YARN-8768 is raised and we will get this in very soon.
>
> - Sunil
>
> On Wed, Sep 12, 2018 at 9:16 PM Sean Mackrory 
> wrote:
>
>> `mvn install` fails on trunk right now due to javadoc error in yarn-api.
>> Is
>> that related? If so, can we revert or fix ASAP please?
>>
>> On Wed, Sep 12, 2018 at 6:05 AM Naganarasimha Garla <
>> naganarasimha...@apache.org> wrote:
>>
>> > Hi All,
>> >  Me and Sunil have successfully merged YARN-3409 to the trunk
>> and
>> > we have created new umbrella jira YARN-8766 to track the pending jiras
>> and
>> > the remaining planned improvement features.
>> > Thanks all for the support !
>> >
>> > Regards,
>> > + Naga
>> >
>> >
>> > On Wed, Sep 12, 2018 at 7:46 AM Naganarasimha Garla <
>> > naganarasimha...@apache.org> wrote:
>> >
>> > > Hi All,
>> > >  Voting has been running since 6 days and adding my vote we
>> have
>> > 4
>> > > binding and 2 non binding +1's with no -1's this voting passes and we
>> > will
>> > > be merging the branch shortly. Thanks for all who participated in the
>> > > discussion and voting thread !
>> > >
>> > > Thanks and Regards,
>> > > + Naga
>> > >
>> > > On Mon, Sep 10, 2018 at 2:50 PM Zian Chen 
>> wrote:
>> > >
>> > >> +1 for merge.
>> > >>
>> > >> > On Sep 9, 2018, at 10:47 PM, Weiwei Yang 
>> wrote:
>> > >> >
>> > >> > +1 for the merge
>> > >> >
>> > >> > On Mon, Sep 10, 2018 at 12:06 PM Rohith Sharma K S <
>> > >> > rohithsharm...@apache.org> wrote:
>> > >> >
>> > >> >> +1 for merge
>> > >> >>
>> > >> >> -Rohith Sharma K S
>> > >> >>
>> > >> >> On Wed, 5 Sep 2018 at 18:01, Naganarasimha Garla <
>> > >> >> naganarasimha...@apache.org> wrote:
>> > >> >>
>> > >> >>> Hi All,
>> > >> >>>Thanks for feedback folks, based on the positive response
>> > >> >> starting
>> > >> >>> a Vote thread for merging YARN-3409 to master.
>> > >> >>>
>> > >> >>> Regards,
>> > >> >>> + Naga & Sunil
>> > >> >>>
>> > >> >>> On Wed, 5 Sep 2018 2:51 am Wangda Tan, 
>> wrote:
>> > >> >>>
>> > >>  +1 for the merge, it gonna be a great addition to 3.2.0 release.
>> > >> Thanks
>> > >> >>> to
>> > >>  everybody for pushing this feature to complete.
>> > >> 
>> > >>  Best,
>> > >>  Wangda
>> > >> 
>> > >>  On Tue, Sep 4, 2018 at 8:25 AM Bibinchundatt <
>> > >> >> bibin.chund...@huawei.com>
>> > >>  wrote:
>> > >> 
>> > >> > +1 for merge. Fetaure would be a good addition to 3.2 release.
>> > >> >
>> > >> > --
>> > >> > Bibin A Chundatt
>> > >> > M: +91-9742095715 <+91%2097420%2095715>
>> > >> > E: bibin.chund...@huawei.com
>> > >> > 2012实验室-印研IT BU分部
>> > >> > 2012 Laboratories-IT BU Branch Dept.
>> > >> > From:Naganarasimha Garla
>> > >> > To:common-dev@hadoop.apache.org,Hdfs-dev,
>> > yarn-...@hadoop.apache.org
>> > >> ,
>> > >> > mapreduce-...@hadoop.apache.org,
>> > >> > Date:2018-08-29 20:00:44
>> > >> > Subject:[Discuss] Merge discussion for Node attribute support
>> > >> feature
>> > >> > YARN-3409
>> > >> >
>> > >> > Hi All,
>> > >> >
>> > >> > We would like to hear your thoughts on merging “Node Attributes
>> > >> >> Support
>> > >> >>> in
>> > >> > YARN” branch (YARN-3409) [2] into trunk in a few weeks. The
>> goal
>> > is
>> > >> to
>> > >> >>> get
>> > >> > it in for HADOOP 3.2.
>> > >> >
>> > >> > *Major work happened in this branch*
>> > >> >
>> > >> > YARN-6858. Attribute Manager to store and provide node
>> attributes
>> > in
>> > >> >> RM
>> > >> > YARN-7871. Support Node attributes reporting from NM to RM(
>> > >> >> distributed
>> > >> > node attributes)
>> > >> > YARN-7863. Modify placement constraints to support node
>> attributes
>> > >> > YARN-7875. Node Attribute store for storing and recovering
>> > >> attributes
>> > >> >
>> > >> > *Detailed Design:*
>> > >> >
>> > >> > Please refer [1] for detailed design document.
>> > >> >
>> > >> > *Testing Efforts:*
>> > >> >
>> > >> > We did detailed tests for the feature in the last few weeks.
>> > >> > This feature will be enabled only when Node Attributes
>> constraints
>> > >> are
>> > >> > specified through SchedulingRequest from AM.
>> > >> > Manager implementation will help to store and recover Node
>> > >> Attributes.
>> > >> > This
>> > >> > works with existing placement constraints.
>> > >> >
>> > >> > *Regarding to API stability:*
>> > >> >
>> > >> > All newly added @Public APIs are @Unstable.
>> > >> >
>> > >> > Documentation jira [3] could help to provide detailed
>> > configuration
>> > >> > details. This feature works from 

Re: [Vote] Merge discussion for Node attribute support feature YARN-3409

[Sunil G]

Thanks Sean

YARN-8768 is raised and we will get this in very soon.

- Sunil

On Wed, Sep 12, 2018 at 9:16 PM Sean Mackrory  wrote:

> `mvn install` fails on trunk right now due to javadoc error in yarn-api. Is
> that related? If so, can we revert or fix ASAP please?
>
> On Wed, Sep 12, 2018 at 6:05 AM Naganarasimha Garla <
> naganarasimha...@apache.org> wrote:
>
> > Hi All,
> >  Me and Sunil have successfully merged YARN-3409 to the trunk and
> > we have created new umbrella jira YARN-8766 to track the pending jiras
> and
> > the remaining planned improvement features.
> > Thanks all for the support !
> >
> > Regards,
> > + Naga
> >
> >
> > On Wed, Sep 12, 2018 at 7:46 AM Naganarasimha Garla <
> > naganarasimha...@apache.org> wrote:
> >
> > > Hi All,
> > >  Voting has been running since 6 days and adding my vote we
> have
> > 4
> > > binding and 2 non binding +1's with no -1's this voting passes and we
> > will
> > > be merging the branch shortly. Thanks for all who participated in the
> > > discussion and voting thread !
> > >
> > > Thanks and Regards,
> > > + Naga
> > >
> > > On Mon, Sep 10, 2018 at 2:50 PM Zian Chen 
> wrote:
> > >
> > >> +1 for merge.
> > >>
> > >> > On Sep 9, 2018, at 10:47 PM, Weiwei Yang 
> wrote:
> > >> >
> > >> > +1 for the merge
> > >> >
> > >> > On Mon, Sep 10, 2018 at 12:06 PM Rohith Sharma K S <
> > >> > rohithsharm...@apache.org> wrote:
> > >> >
> > >> >> +1 for merge
> > >> >>
> > >> >> -Rohith Sharma K S
> > >> >>
> > >> >> On Wed, 5 Sep 2018 at 18:01, Naganarasimha Garla <
> > >> >> naganarasimha...@apache.org> wrote:
> > >> >>
> > >> >>> Hi All,
> > >> >>>Thanks for feedback folks, based on the positive response
> > >> >> starting
> > >> >>> a Vote thread for merging YARN-3409 to master.
> > >> >>>
> > >> >>> Regards,
> > >> >>> + Naga & Sunil
> > >> >>>
> > >> >>> On Wed, 5 Sep 2018 2:51 am Wangda Tan, 
> wrote:
> > >> >>>
> > >>  +1 for the merge, it gonna be a great addition to 3.2.0 release.
> > >> Thanks
> > >> >>> to
> > >>  everybody for pushing this feature to complete.
> > >> 
> > >>  Best,
> > >>  Wangda
> > >> 
> > >>  On Tue, Sep 4, 2018 at 8:25 AM Bibinchundatt <
> > >> >> bibin.chund...@huawei.com>
> > >>  wrote:
> > >> 
> > >> > +1 for merge. Fetaure would be a good addition to 3.2 release.
> > >> >
> > >> > --
> > >> > Bibin A Chundatt
> > >> > M: +91-9742095715 <+91%2097420%2095715>
> > >> > E: bibin.chund...@huawei.com
> > >> > 2012实验室-印研IT BU分部
> > >> > 2012 Laboratories-IT BU Branch Dept.
> > >> > From:Naganarasimha Garla
> > >> > To:common-dev@hadoop.apache.org,Hdfs-dev,
> > yarn-...@hadoop.apache.org
> > >> ,
> > >> > mapreduce-...@hadoop.apache.org,
> > >> > Date:2018-08-29 20:00:44
> > >> > Subject:[Discuss] Merge discussion for Node attribute support
> > >> feature
> > >> > YARN-3409
> > >> >
> > >> > Hi All,
> > >> >
> > >> > We would like to hear your thoughts on merging “Node Attributes
> > >> >> Support
> > >> >>> in
> > >> > YARN” branch (YARN-3409) [2] into trunk in a few weeks. The goal
> > is
> > >> to
> > >> >>> get
> > >> > it in for HADOOP 3.2.
> > >> >
> > >> > *Major work happened in this branch*
> > >> >
> > >> > YARN-6858. Attribute Manager to store and provide node
> attributes
> > in
> > >> >> RM
> > >> > YARN-7871. Support Node attributes reporting from NM to RM(
> > >> >> distributed
> > >> > node attributes)
> > >> > YARN-7863. Modify placement constraints to support node
> attributes
> > >> > YARN-7875. Node Attribute store for storing and recovering
> > >> attributes
> > >> >
> > >> > *Detailed Design:*
> > >> >
> > >> > Please refer [1] for detailed design document.
> > >> >
> > >> > *Testing Efforts:*
> > >> >
> > >> > We did detailed tests for the feature in the last few weeks.
> > >> > This feature will be enabled only when Node Attributes
> constraints
> > >> are
> > >> > specified through SchedulingRequest from AM.
> > >> > Manager implementation will help to store and recover Node
> > >> Attributes.
> > >> > This
> > >> > works with existing placement constraints.
> > >> >
> > >> > *Regarding to API stability:*
> > >> >
> > >> > All newly added @Public APIs are @Unstable.
> > >> >
> > >> > Documentation jira [3] could help to provide detailed
> > configuration
> > >> > details. This feature works from end-to-end and we tested this
> in
> > >> our
> > >> > local
> > >> > cluster. Branch code is run against trunk and tracked via [4].
> > >> >
> > >> > We would love to get your thoughts before opening a voting
> thread.
> > >> >
> > >> > Special thanks to a team of folks who worked hard and
> contributed
> > >> >>> towards
> > >> > this efforts including design 

Re: [VOTE] Release Apache Hadoop 2.8.5 (RC0)

[Ajay Kumar]

Built from source. Not sure if I am missing some config but MR test jobs (Pi) 
are failing with error. ("Error: Could not find or load main class 
org.apache.hadoop.mapreduce.v2.app.MRAppMaster".) 
 Anyone else facing same issue?

See below exception in Nodemanager which I haven't seen earlier.
2018-09-11 11:26:31,343 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: 
NodeManager metrics system started
2018-09-11 11:26:31,386 WARN 
org.apache.hadoop.yarn.util.ResourceCalculatorPlugin: 
java.lang.UnsupportedOperationException: Could not determine OS: Failed to 
instantiate default resource calculator.
java.lang.UnsupportedOperationException: Could not determine OS
at org.apache.hadoop.util.SysInfo.newInstance(SysInfo.java:43)
at 
org.apache.hadoop.yarn.util.ResourceCalculatorPlugin.(ResourceCalculatorPlugin.java:41)
at 
org.apache.hadoop.yarn.util.ResourceCalculatorPlugin.getResourceCalculatorPlugin(ResourceCalculatorPlugin.java:191)
at 
org.apache.hadoop.yarn.server.nodemanager.NodeResourceMonitorImpl.serviceInit(NodeResourceMonitorImpl.java:74)
at 
org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at 
org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
at 
org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:357)
at 
org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at 
org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:636)
at 
org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:684)

Thanks,
Ajay

On 9/12/18, 6:31 AM, "Kihwal Lee"  wrote:

+1 (binding)

- Built from source
- Brought up a single node cluster
- Checked basic HDFS functions and checked the UIs
- Ran several simple jobs.


On Mon, Sep 10, 2018 at 7:01 AM 俊平堵  wrote:

> Hi all,
>
>  I've created the first release candidate (RC0) for Apache
> Hadoop 2.8.5. This is our next point release to follow up 2.8.4. It
> includes 33 important fixes and improvements.
>
>
> The RC artifacts are available at:
> http://home.apache.org/~junping_du/hadoop-2.8.5-RC0
>
>
> The RC tag in git is: release-2.8.5-RC0
>
>
>
> The maven artifacts are available via repository.apache.org<
> http://repository.apache.org> at:
>
> https://repository.apache.org/content/repositories/orgapachehadoop-1140
>
>
> Please try the release and vote; the vote will run for the usual 5
> working
> days, ending on 9/15/2018 PST time.
>
>
> Thanks,
>
>
> Junping
>



-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [Vote] Merge discussion for Node attribute support feature YARN-3409

[Sean Mackrory]

`mvn install` fails on trunk right now due to javadoc error in yarn-api. Is
that related? If so, can we revert or fix ASAP please?

On Wed, Sep 12, 2018 at 6:05 AM Naganarasimha Garla <
naganarasimha...@apache.org> wrote:

> Hi All,
>  Me and Sunil have successfully merged YARN-3409 to the trunk and
> we have created new umbrella jira YARN-8766 to track the pending jiras and
> the remaining planned improvement features.
> Thanks all for the support !
>
> Regards,
> + Naga
>
>
> On Wed, Sep 12, 2018 at 7:46 AM Naganarasimha Garla <
> naganarasimha...@apache.org> wrote:
>
> > Hi All,
> >  Voting has been running since 6 days and adding my vote we have
> 4
> > binding and 2 non binding +1's with no -1's this voting passes and we
> will
> > be merging the branch shortly. Thanks for all who participated in the
> > discussion and voting thread !
> >
> > Thanks and Regards,
> > + Naga
> >
> > On Mon, Sep 10, 2018 at 2:50 PM Zian Chen  wrote:
> >
> >> +1 for merge.
> >>
> >> > On Sep 9, 2018, at 10:47 PM, Weiwei Yang  wrote:
> >> >
> >> > +1 for the merge
> >> >
> >> > On Mon, Sep 10, 2018 at 12:06 PM Rohith Sharma K S <
> >> > rohithsharm...@apache.org> wrote:
> >> >
> >> >> +1 for merge
> >> >>
> >> >> -Rohith Sharma K S
> >> >>
> >> >> On Wed, 5 Sep 2018 at 18:01, Naganarasimha Garla <
> >> >> naganarasimha...@apache.org> wrote:
> >> >>
> >> >>> Hi All,
> >> >>>Thanks for feedback folks, based on the positive response
> >> >> starting
> >> >>> a Vote thread for merging YARN-3409 to master.
> >> >>>
> >> >>> Regards,
> >> >>> + Naga & Sunil
> >> >>>
> >> >>> On Wed, 5 Sep 2018 2:51 am Wangda Tan,  wrote:
> >> >>>
> >>  +1 for the merge, it gonna be a great addition to 3.2.0 release.
> >> Thanks
> >> >>> to
> >>  everybody for pushing this feature to complete.
> >> 
> >>  Best,
> >>  Wangda
> >> 
> >>  On Tue, Sep 4, 2018 at 8:25 AM Bibinchundatt <
> >> >> bibin.chund...@huawei.com>
> >>  wrote:
> >> 
> >> > +1 for merge. Fetaure would be a good addition to 3.2 release.
> >> >
> >> > --
> >> > Bibin A Chundatt
> >> > M: +91-9742095715
> >> > E: bibin.chund...@huawei.com
> >> > 2012实验室-印研IT BU分部
> >> > 2012 Laboratories-IT BU Branch Dept.
> >> > From:Naganarasimha Garla
> >> > To:common-dev@hadoop.apache.org,Hdfs-dev,
> yarn-...@hadoop.apache.org
> >> ,
> >> > mapreduce-...@hadoop.apache.org,
> >> > Date:2018-08-29 20:00:44
> >> > Subject:[Discuss] Merge discussion for Node attribute support
> >> feature
> >> > YARN-3409
> >> >
> >> > Hi All,
> >> >
> >> > We would like to hear your thoughts on merging “Node Attributes
> >> >> Support
> >> >>> in
> >> > YARN” branch (YARN-3409) [2] into trunk in a few weeks. The goal
> is
> >> to
> >> >>> get
> >> > it in for HADOOP 3.2.
> >> >
> >> > *Major work happened in this branch*
> >> >
> >> > YARN-6858. Attribute Manager to store and provide node attributes
> in
> >> >> RM
> >> > YARN-7871. Support Node attributes reporting from NM to RM(
> >> >> distributed
> >> > node attributes)
> >> > YARN-7863. Modify placement constraints to support node attributes
> >> > YARN-7875. Node Attribute store for storing and recovering
> >> attributes
> >> >
> >> > *Detailed Design:*
> >> >
> >> > Please refer [1] for detailed design document.
> >> >
> >> > *Testing Efforts:*
> >> >
> >> > We did detailed tests for the feature in the last few weeks.
> >> > This feature will be enabled only when Node Attributes constraints
> >> are
> >> > specified through SchedulingRequest from AM.
> >> > Manager implementation will help to store and recover Node
> >> Attributes.
> >> > This
> >> > works with existing placement constraints.
> >> >
> >> > *Regarding to API stability:*
> >> >
> >> > All newly added @Public APIs are @Unstable.
> >> >
> >> > Documentation jira [3] could help to provide detailed
> configuration
> >> > details. This feature works from end-to-end and we tested this in
> >> our
> >> > local
> >> > cluster. Branch code is run against trunk and tracked via [4].
> >> >
> >> > We would love to get your thoughts before opening a voting thread.
> >> >
> >> > Special thanks to a team of folks who worked hard and contributed
> >> >>> towards
> >> > this efforts including design discussion / patch / reviews, etc.:
> >> >> Weiwei
> >> > Yang, Bibin Chundatt, Wangda Tan, Vinod Kumar Vavilappali,
> >> >> Konstantinos
> >> > Karanasos, Arun Suresh, Varun Saxena, Devaraj Kavali, Lei Guo,
> Chong
> >> >>> Chen.
> >> >
> >> > [1] :
> >> >
> >> >
> >> >>>
> >> >>
> >>
> https://issues.apache.org/jira/secure/attachment/12937633/Node-Attributes-Requirements-Design-doc_v2.pdf
> >> > [2] : 

[jira] [Resolved] (HADOOP-15752) [s3a] testDestroyNoBucket always fails

[Sean Mackrory (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-15752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Mackrory resolved HADOOP-15752.

Resolution: Duplicate

> [s3a] testDestroyNoBucket always fails
> --
>
> Key: HADOOP-15752
> URL: https://issues.apache.org/jira/browse/HADOOP-15752
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: fs/s3
>Reporter: Sean Mackrory
>Priority: Major
>
> ITestS3GuardToolDynamoDB and ITestS3GuardToolLocal both fail the same way:
> {code}
> [INFO] Running org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardToolLocal
> [ERROR] Tests run: 23, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 
> 73.198 s <<< FAILURE! - in 
> org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardToolLocal
> [ERROR] 
> testDestroyNoBucket(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardToolLocal)  
> Time elapsed: 7.048 s  <<< FAILURE!
> java.lang.AssertionError: Expected a java.io.FileNotFoundException to be 
> thrown, but got the result: : 0
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-15752) [s3a] testDestroyNoBucket always fails

[Sean Mackrory (JIRA)]

Sean Mackrory created HADOOP-15752:
--

 Summary: [s3a] testDestroyNoBucket always fails
 Key: HADOOP-15752
 URL: https://issues.apache.org/jira/browse/HADOOP-15752
 Project: Hadoop Common
  Issue Type: Bug
  Components: fs/s3
Reporter: Sean Mackrory


ITestS3GuardToolDynamoDB and ITestS3GuardToolLocal both fail the same way:

{code}
[INFO] Running org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardToolLocal
[ERROR] Tests run: 23, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 73.198 
s <<< FAILURE! - in org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardToolLocal
[ERROR] 
testDestroyNoBucket(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardToolLocal)  
Time elapsed: 7.048 s  <<< FAILURE!
java.lang.AssertionError: Expected a java.io.FileNotFoundException to be 
thrown, but got the result: : 0
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-15751) AWS Data read stack trace in S3a putObjectDirect

[Steve Loughran (JIRA)]

Steve Loughran created HADOOP-15751:
---

 Summary: AWS Data read stack trace in S3a putObjectDirect
 Key: HADOOP-15751
 URL: https://issues.apache.org/jira/browse/HADOOP-15751
 Project: Hadoop Common
  Issue Type: Sub-task
  Components: fs/s3, test
Affects Versions: 3.3.0
Reporter: Steve Loughran


(transient) failure in a put to AWS london from UK Bristol; 120+ miles away 
(I.e not that far). 

Error "Data read has a different length than the expected:" after the PUT

This is happening in a delayed listing path, so I'd suspect fault injection 
-except this is happening deep in the AWS layer. Maybe its an observed 
inconsistency at that layer, one we aren't yet handling.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-15750) remove obsolete S3A test ITestS3ACredentialsInURL

[Steve Loughran (JIRA)]

Steve Loughran created HADOOP-15750:
---

 Summary: remove obsolete S3A test ITestS3ACredentialsInURL
 Key: HADOOP-15750
 URL: https://issues.apache.org/jira/browse/HADOOP-15750
 Project: Hadoop Common
  Issue Type: Sub-task
  Components: fs/s3
Affects Versions: 3.3.0
Reporter: Steve Loughran
Assignee: Steve Loughran


now that we've cut user:pass, need to remove the test ITestS3ACredentialsInURL, 
which fails.

I'd missed this because I've got s3guard enabled in test runs, and it 
automatically skips.  oops.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

[Apache Jenkins Server]

For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/894/

[Sep 11, 2018 4:42:34 AM] (brahma) HDFS-13237. [Documentation] RBF: Mount 
points across multiple
[Sep 11, 2018 5:34:23 AM] (xiao) HDFS-13051. Fix dead lock during async editlog 
rolling if edit queue is
[Sep 11, 2018 3:55:34 PM] (aw) HADOOP-14671. Upgrade Apache Yetus to 0.8.0
[Sep 11, 2018 4:16:33 PM] (jlowe) MAPREDUCE-7135. 
TestTaskAttemptContainerRequest should reset
[Sep 11, 2018 5:19:50 PM] (aengineer) HDDS-222. Remove hdfs command line from 
ozone distribution. Contributed
[Sep 11, 2018 5:44:47 PM] (aengineer) HDDS-431. LeaseManager of CommandWatcher 
is not started. Contributed by
[Sep 11, 2018 6:10:50 PM] (jlowe) MAPREDUCE-7136. TestMRAppMetrics should 
shutdown DefaultMetricsSystem
[Sep 11, 2018 10:15:24 PM] (xyao) HDDS-416. Remove currentPosition from 
ChunkInputStreamEntry. Contributed
[Sep 11, 2018 11:47:33 PM] (aengineer) HDDS-424. Consolidate ozone oz 
parameters to use GNU convention.
[Sep 12, 2018 12:00:04 AM] (aengineer) HDDS-432. Replication of closed 
containers is not working. Contributed
[Sep 12, 2018 12:18:42 AM] (liuml07) HADOOP-14833. Remove s3a user:secret 
authentication. Contributed by
[Sep 12, 2018 1:00:57 AM] (brahma) HDFS-13902. Add JMX, conf and stacks menus 
to the datanode page.




-1 overall


The following subsystems voted -1:
asflicense findbugs hadolint pathlen unit xml


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

XML :

   Parsing Error(s): 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/public/crossdomain.xml
 

FindBugs :

   
module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-submarine
 
   Unread field:FSBasedSubmarineStorageImpl.java:[line 39] 
   Found reliance on default encoding in 
org.apache.hadoop.yarn.submarine.runtimes.yarnservice.YarnServiceJobSubmitter.generateCommandLaunchScript(RunJobParameters,
 TaskType, Component):in 
org.apache.hadoop.yarn.submarine.runtimes.yarnservice.YarnServiceJobSubmitter.generateCommandLaunchScript(RunJobParameters,
 TaskType, Component): new java.io.FileWriter(File) At 
YarnServiceJobSubmitter.java:[line 195] 
   
org.apache.hadoop.yarn.submarine.runtimes.yarnservice.YarnServiceJobSubmitter.generateCommandLaunchScript(RunJobParameters,
 TaskType, Component) may fail to clean up java.io.Writer on checked exception 
Obligation to clean up resource created at YarnServiceJobSubmitter.java:to 
clean up java.io.Writer on checked exception Obligation to clean up resource 
created at YarnServiceJobSubmitter.java:[line 195] is not discharged 
   
org.apache.hadoop.yarn.submarine.runtimes.yarnservice.YarnServiceUtils.getComponentArrayJson(String,
 int, String) concatenates strings using + in a loop At 
YarnServiceUtils.java:using + in a loop At YarnServiceUtils.java:[line 72] 

Failed CTEST tests :

   test_test_libhdfs_threaded_hdfs_static 
   test_libhdfs_threaded_hdfspp_test_shim_static 

Failed junit tests :

   hadoop.util.TestReadWriteDiskValidator 
   hadoop.util.TestBasicDiskValidator 
   hadoop.util.TestDiskCheckerWithDiskIo 
   hadoop.hdfs.TestLeaseRecovery2 
   hadoop.hdfs.client.impl.TestBlockReaderLocal 
   hadoop.yarn.sls.appmaster.TestAMSimulator 
  

   cc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/894/artifact/out/diff-compile-cc-root.txt
  [4.0K]

   javac:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/894/artifact/out/diff-compile-javac-root.txt
  [304K]

   checkstyle:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/894/artifact/out/diff-checkstyle-root.txt
  [17M]

   hadolint:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/894/artifact/out/diff-patch-hadolint.txt
  [4.0K]

   pathlen:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/894/artifact/out/pathlen.txt
  [12K]

   pylint:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/894/artifact/out/diff-patch-pylint.txt
  [24K]

   shellcheck:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/894/artifact/out/diff-patch-shellcheck.txt
  [20K]

   shelldocs:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/894/artifact/out/diff-patch-shelldocs.txt
  [16K]

   whitespace:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/894/artifact/out/whitespace-eol.txt
  [9.4M]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/894/artifact/out/whitespace-tabs.txt
  [1.1M]

   xml:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/894/artifact/out/xml.txt
  [4.0K]

   findbugs:

   

Re: [VOTE] Release Apache Hadoop 2.8.5 (RC0)

[Kihwal Lee]

+1 (binding)

- Built from source
- Brought up a single node cluster
- Checked basic HDFS functions and checked the UIs
- Ran several simple jobs.


On Mon, Sep 10, 2018 at 7:01 AM 俊平堵  wrote:

> Hi all,
>
>  I've created the first release candidate (RC0) for Apache
> Hadoop 2.8.5. This is our next point release to follow up 2.8.4. It
> includes 33 important fixes and improvements.
>
>
> The RC artifacts are available at:
> http://home.apache.org/~junping_du/hadoop-2.8.5-RC0
>
>
> The RC tag in git is: release-2.8.5-RC0
>
>
>
> The maven artifacts are available via repository.apache.org<
> http://repository.apache.org> at:
>
> https://repository.apache.org/content/repositories/orgapachehadoop-1140
>
>
> Please try the release and vote; the vote will run for the usual 5
> working
> days, ending on 9/15/2018 PST time.
>
>
> Thanks,
>
>
> Junping
>

Re: [Vote] Merge discussion for Node attribute support feature YARN-3409

[Naganarasimha Garla]

Hi All,
 Me and Sunil have successfully merged YARN-3409 to the trunk and
we have created new umbrella jira YARN-8766 to track the pending jiras and
the remaining planned improvement features.
Thanks all for the support !

Regards,
+ Naga


On Wed, Sep 12, 2018 at 7:46 AM Naganarasimha Garla <
naganarasimha...@apache.org> wrote:

> Hi All,
>  Voting has been running since 6 days and adding my vote we have 4
> binding and 2 non binding +1's with no -1's this voting passes and we will
> be merging the branch shortly. Thanks for all who participated in the
> discussion and voting thread !
>
> Thanks and Regards,
> + Naga
>
> On Mon, Sep 10, 2018 at 2:50 PM Zian Chen  wrote:
>
>> +1 for merge.
>>
>> > On Sep 9, 2018, at 10:47 PM, Weiwei Yang  wrote:
>> >
>> > +1 for the merge
>> >
>> > On Mon, Sep 10, 2018 at 12:06 PM Rohith Sharma K S <
>> > rohithsharm...@apache.org> wrote:
>> >
>> >> +1 for merge
>> >>
>> >> -Rohith Sharma K S
>> >>
>> >> On Wed, 5 Sep 2018 at 18:01, Naganarasimha Garla <
>> >> naganarasimha...@apache.org> wrote:
>> >>
>> >>> Hi All,
>> >>>Thanks for feedback folks, based on the positive response
>> >> starting
>> >>> a Vote thread for merging YARN-3409 to master.
>> >>>
>> >>> Regards,
>> >>> + Naga & Sunil
>> >>>
>> >>> On Wed, 5 Sep 2018 2:51 am Wangda Tan,  wrote:
>> >>>
>>  +1 for the merge, it gonna be a great addition to 3.2.0 release.
>> Thanks
>> >>> to
>>  everybody for pushing this feature to complete.
>> 
>>  Best,
>>  Wangda
>> 
>>  On Tue, Sep 4, 2018 at 8:25 AM Bibinchundatt <
>> >> bibin.chund...@huawei.com>
>>  wrote:
>> 
>> > +1 for merge. Fetaure would be a good addition to 3.2 release.
>> >
>> > --
>> > Bibin A Chundatt
>> > M: +91-9742095715
>> > E: bibin.chund...@huawei.com
>> > 2012实验室-印研IT BU分部
>> > 2012 Laboratories-IT BU Branch Dept.
>> > From:Naganarasimha Garla
>> > To:common-dev@hadoop.apache.org,Hdfs-dev,yarn-...@hadoop.apache.org
>> ,
>> > mapreduce-...@hadoop.apache.org,
>> > Date:2018-08-29 20:00:44
>> > Subject:[Discuss] Merge discussion for Node attribute support
>> feature
>> > YARN-3409
>> >
>> > Hi All,
>> >
>> > We would like to hear your thoughts on merging “Node Attributes
>> >> Support
>> >>> in
>> > YARN” branch (YARN-3409) [2] into trunk in a few weeks. The goal is
>> to
>> >>> get
>> > it in for HADOOP 3.2.
>> >
>> > *Major work happened in this branch*
>> >
>> > YARN-6858. Attribute Manager to store and provide node attributes in
>> >> RM
>> > YARN-7871. Support Node attributes reporting from NM to RM(
>> >> distributed
>> > node attributes)
>> > YARN-7863. Modify placement constraints to support node attributes
>> > YARN-7875. Node Attribute store for storing and recovering
>> attributes
>> >
>> > *Detailed Design:*
>> >
>> > Please refer [1] for detailed design document.
>> >
>> > *Testing Efforts:*
>> >
>> > We did detailed tests for the feature in the last few weeks.
>> > This feature will be enabled only when Node Attributes constraints
>> are
>> > specified through SchedulingRequest from AM.
>> > Manager implementation will help to store and recover Node
>> Attributes.
>> > This
>> > works with existing placement constraints.
>> >
>> > *Regarding to API stability:*
>> >
>> > All newly added @Public APIs are @Unstable.
>> >
>> > Documentation jira [3] could help to provide detailed configuration
>> > details. This feature works from end-to-end and we tested this in
>> our
>> > local
>> > cluster. Branch code is run against trunk and tracked via [4].
>> >
>> > We would love to get your thoughts before opening a voting thread.
>> >
>> > Special thanks to a team of folks who worked hard and contributed
>> >>> towards
>> > this efforts including design discussion / patch / reviews, etc.:
>> >> Weiwei
>> > Yang, Bibin Chundatt, Wangda Tan, Vinod Kumar Vavilappali,
>> >> Konstantinos
>> > Karanasos, Arun Suresh, Varun Saxena, Devaraj Kavali, Lei Guo, Chong
>> >>> Chen.
>> >
>> > [1] :
>> >
>> >
>> >>>
>> >>
>> https://issues.apache.org/jira/secure/attachment/12937633/Node-Attributes-Requirements-Design-doc_v2.pdf
>> > [2] : https://issues.apache.org/jira/browse/YARN-3409
>> > [3] : https://issues.apache.org/jira/browse/YARN-7865
>> > [4] : https://issues.apache.org/jira/browse/YARN-8718
>> >
>> > Thanks,
>> > + Naga & Sunil Govindan
>> >
>> 
>> >>>
>> >>
>> >
>> >
>> > --
>> > Weiwei Yang
>>
>>

Re: Hadoop 3.2 Release Plan proposal

[Sunil G]

Hi All,

Inline with the original 3.2 communication proposal dated 17th July 2018, I
would like to provide more updates.

We are approaching previously proposed code freeze date (September 14,
2018). So I would like to cut 3.2 branch on 17th Sept and point existing
trunk to 3.3 if there are no issues.

*Current Release Plan:*
Feature freeze date : all features to merge by September 7, 2018.
Code freeze date : blockers/critical only, no improvements and
blocker/critical bug-fixes September 14, 2018.
Release date: September 28, 2018

If any critical/blocker tickets which are targeted to 3.2.0, we need to
backport to 3.2 post branch cut.

Here's an updated 3.2.0 feature status:

1. Merged & Completed features:

- (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning workloads
Initial cut.
- (Uma) HDFS-10285: HDFS Storage Policy Satisfier
- (Sunil) YARN-7494: Multi Node scheduling support in Capacity Scheduler.
- (Chandni/Eric) YARN-7512: Support service upgrade via YARN Service API
and CLI.
- (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
- (Inigo) HDFS-12615: Router-based HDFS federation. Improvement works.

2. Features close to finish:

- (Steve) S3Guard Phase III. Close to commit.
- (Steve) S3a phase V. Close to commit.
- (Steve) Support Windows Azure Storage. Close to commit.

3. Tentative/Cancelled features for 3.2:
- (Rohith) YARN-5742: Serve aggregated logs of historical apps from ATSv2.
Patch in progress.
- (Haibo Chen) YARN-1011: Resource overcommitment. Looks challenging to be
done before Aug 2018.
- (Eric) YARN-7129: Application Catalog for YARN applications. Challenging
as more discussions are on-going.

*Summary of 3.2.0 issues status:*
19 Blocker and Critical issues [1] are open, I am following up with owners
to get status on each of them to get in by Code Freeze date.

[1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in (Blocker,
Critical) AND resolution = Unresolved AND "Target Version/s" = 3.2.0 ORDER
BY priority DESC

Thanks,
Sunil



On Thu, Aug 30, 2018 at 9:59 PM Sunil G  wrote:

> Hi All,
>
> Inline with earlier communication dated 17th July 2018, I would like to
> provide some updates.
>
> We are approaching previously proposed code freeze date (Aug 31).
>
> One of the critical feature Node Attributes feature merge discussion/vote
> is ongoing. Also few other Blocker bugs need a bit more time. With regard
> to this, suggesting to push the feature/code freeze for 2 more weeks to
> accommodate these jiras too.
>
> Proposing Updated changes in plan inline with this:
> Feature freeze date : all features to merge by September 7, 2018.
> Code freeze date : blockers/critical only, no improvements and
>  blocker/critical bug-fixes September 14, 2018.
> Release date: September 28, 2018
>
> If any features in branch which are targeted to 3.2.0, please reply to
> this email thread.
>
> *Here's an updated 3.2.0 feature status:*
>
> 1. Merged & Completed features:
>
> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning workloads
> Initial cut.
> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity Scheduler.
> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN Service API
> and CLI.
>
> 2. Features close to finish:
>
> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN. Merge/Vote
> Ongoing.
> - (Rohith) YARN-5742: Serve aggregated logs of historical apps from ATSv2.
> Patch in progress.
> - (Virajit) HDFS-12615: Router-based HDFS federation. Improvement works.
> - (Steve) S3Guard Phase III, S3a phase V, Support Windows Azure Storage.
> In progress.
>
> 3. Tentative features:
>
> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks challenging to be
> done before Aug 2018.
> - (Eric) YARN-7129: Application Catalog for YARN applications. Challenging
> as more discussions are on-going.
>
> *Summary of 3.2.0 issues status:*
>
> 26 Blocker and Critical issues [1] are open, I am following up with
> owners to get status on each of them to get in by Code Freeze date.
>
> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in (Blocker,
> Critical) AND resolution = Unresolved AND "Target Version/s" = 3.2.0 ORDER
> BY priority DESC
>
> Thanks,
> Sunil
>
> On Tue, Aug 14, 2018 at 10:30 PM Sunil G  wrote:
>
>> Hi All,
>>
>> Thanks for the feedbacks. Inline with earlier communication dated 17th
>> July 2018, I would like to provide some updates.
>>
>> We are approaching previously proposed feature freeze date (Aug 21, about
>> 7 days from today).
>> If any features in branch which are targeted to 3.2.0, please reply to
>> this email thread.
>> Steve has mentioned about the s3 features which will come close to Code
>> Freeze Date (Aug 31st).
>>
>> *Here's an updated 3.2.0 feature status:*
>>
>> 1. Merged & Completed features:
>>
>> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning workloads
>> Initial cut.
>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>>
>> 2. 

[jira] [Created] (HADOOP-15748) S3 listing inconsistency can raise NPE in globber

[Steve Loughran (JIRA)]

Steve Loughran created HADOOP-15748:
---

 Summary: S3 listing inconsistency can raise NPE in globber
 Key: HADOOP-15748
 URL: https://issues.apache.org/jira/browse/HADOOP-15748
 Project: Hadoop Common
  Issue Type: Sub-task
  Components: fs
Affects Versions: 2.8.4, 2.9.1
Reporter: Steve Loughran
Assignee: Steve Loughran


FileSystem Globber does a listStatus(path) and then, if only one element is 
returned, {{getFileStatus(path).isDirectory()}} to see if it is a dir. The way 
getFileStatus() is wrapped, IOEs are downgraded to null

On S3, if the path has had entries deleted, the listing may include files which 
are no longer there, so the getFileStatus(path),isDirectory triggers an NPE

While its wrong to glob against S3 when its being inconsistent, we should at 
least fail gracefully here.

Proposed
# log all IOEs raised in Globber.getFileStatus @ debug
# catch FNFEs and downgrade to warn
# continue

The alternative would be fail fast on FNFE, but that's more traumatic




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-15747) warning about user:pass in URI to explicitly call out Hadoop 3.3 as removal

[Steve Loughran (JIRA)]

Steve Loughran created HADOOP-15747:
---

 Summary: warning about user:pass in URI to explicitly call out 
Hadoop 3.3 as removal
 Key: HADOOP-15747
 URL: https://issues.apache.org/jira/browse/HADOOP-15747
 Project: Hadoop Common
  Issue Type: Sub-task
  Components: fs/s3
Affects Versions: 3.0.3, 3.1.1, 2.9.1
 Environment: Now that HADOOP-14833 has removed user:secret from URIs, 
change the warning printed when people to that to explicitly declare Hadoop 3.3 
as when it will happen. Do the same in the docs
Reporter: Steve Loughran
Assignee: Steve Loughran






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Resolved] (HADOOP-15746) Hadoop: branch-2.8 docker image failed to build

[Akira Ajisaka (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-15746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Akira Ajisaka resolved HADOOP-15746.

   Resolution: Duplicate
Fix Version/s: (was: 2.8.5)
   (was: 2.9.2)

> Hadoop: branch-2.8 docker image failed to build
> ---
>
> Key: HADOOP-15746
> URL: https://issues.apache.org/jira/browse/HADOOP-15746
> Project: Hadoop Common
>  Issue Type: Bug
>Reporter: Weiwei Yang
>Priority: Major
>
> The jenkins build was failing on hadoop branch-2.8 for quite a while, see an 
> example result in YARN-8664. The error
> {noformat}
>  Step 24/31 : RUN pip install pylint
>  ---> Running in b1f0e6c1d209
>  Downloading/unpacking pylint
>  ...
>  setuptools_scm.version.SetuptoolsOutdatedWarning: your setuptools is too old 
> (<12)
>  Complete output from command python setup.py egg_info:
>  /usr/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution 
> option: 'python_requires'
> {noformat}
>  detail output can be seen 
> [here|https://builds.apache.org/job/PreCommit-YARN-Build/21808/console].



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Reopened] (HADOOP-15746) Hadoop: branch-2.8 docker image failed to build

[Akira Ajisaka (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-15746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Akira Ajisaka reopened HADOOP-15746:

  Assignee: (was: Akira Ajisaka)

> Hadoop: branch-2.8 docker image failed to build
> ---
>
> Key: HADOOP-15746
> URL: https://issues.apache.org/jira/browse/HADOOP-15746
> Project: Hadoop Common
>  Issue Type: Bug
>Reporter: Weiwei Yang
>Priority: Major
>
> The jenkins build was failing on hadoop branch-2.8 for quite a while, see an 
> example result in YARN-8664. The error
> {noformat}
>  Step 24/31 : RUN pip install pylint
>  ---> Running in b1f0e6c1d209
>  Downloading/unpacking pylint
>  ...
>  setuptools_scm.version.SetuptoolsOutdatedWarning: your setuptools is too old 
> (<12)
>  Complete output from command python setup.py egg_info:
>  /usr/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution 
> option: 'python_requires'
> {noformat}
>  detail output can be seen 
> [here|https://builds.apache.org/job/PreCommit-YARN-Build/21808/console].



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Resolved] (HADOOP-15746) Hadoop: branch-2.8 docker image failed to build

[Weiwei Yang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-15746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Weiwei Yang resolved HADOOP-15746.
--
   Resolution: Fixed
Fix Version/s: 2.8.5
   2.9.2

> Hadoop: branch-2.8 docker image failed to build
> ---
>
> Key: HADOOP-15746
> URL: https://issues.apache.org/jira/browse/HADOOP-15746
> Project: Hadoop Common
>  Issue Type: Bug
>Reporter: Weiwei Yang
>Assignee: Akira Ajisaka
>Priority: Major
> Fix For: 2.9.2, 2.8.5
>
>
> The jenkins build was failing on hadoop branch-2.8 for quite a while, see an 
> example result in YARN-8664. The error
> {noformat}
>  Step 24/31 : RUN pip install pylint
>  ---> Running in b1f0e6c1d209
>  Downloading/unpacking pylint
>  ...
>  setuptools_scm.version.SetuptoolsOutdatedWarning: your setuptools is too old 
> (<12)
>  Complete output from command python setup.py egg_info:
>  /usr/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution 
> option: 'python_requires'
> {noformat}
>  detail output can be seen 
> [here|https://builds.apache.org/job/PreCommit-YARN-Build/21808/console].



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org