RE: [VOTE] Merge Storage Policy Satisfier (SPS) [HDFS-10285] feature branch to trunk

[Chen, Sammi]

+ 1.   

Thanks for the great work.  Looking forward to see the SPS in the next 
rerelease. 


Bests,
Sammi

-Original Message-
From: Uma Maheswara Rao G [mailto:hadoop@gmail.com] 
Sent: Wednesday, August 1, 2018 2:38 PM
To: hdfs-dev@hadoop.apache.org
Subject: [VOTE] Merge Storage Policy Satisfier (SPS) [HDFS-10285] feature 
branch to trunk

Hi All,



 From the positive responses from JIRA discussion and no objections from below 
DISCUSS thread [1], I am converting it to voting thread.



 Last couple of weeks we spent time on testing the feature and so far it is 
working fine. Surendra uploaded a test report at HDFS-10285:  [2]



 In this phase, we provide to run SPS outside of Namenode only and as a next 
phase we continue to discuss and work on to enable it as Internal SPS as 
explained below. We have got clean QA report on branch and if there are any 
static tool comments triggered later while running this thread, we will make 
sure to fix them before merge. We committed and continue to improve the code on 
trunk. Please refer to HDFS-10285 for discussion details.



 This has been a long effort and we're grateful for the support we've received 
from the community. In particular, thanks to Andrew Wang, Anoop Sam John, Anu 
Engineer, Chris Douglas, Daryn Sharp, Du Jingcheng , Ewan Higgs, Jing Zhao, Kai 
Zheng,  Rakesh R, Ramkrishna , Surendra Singh Lilhore , Thomas Demoor, Uma 
Maheswara Rao G, Vinayakumar, Virajith,  Wei Zhou, Yuanbo Liu. Without these 
members effort, this feature might not have reached to this state.



To start with, here is my +1

It will end on 6th Aug.



Regards,

Uma

[1]  https://s.apache.org/bhyu
[2]  https://s.apache.org/AXvL


On Wed, Jun 27, 2018 at 3:21 PM, Uma Maheswara Rao G 
wrote:

> Hi All,
>
>   After long discussions(offline and on JIRA) on SPS, we came to a 
> conclusion on JIRA(HDFS-10285) that, we will go ahead with External 
> SPS merge in first phase. In this phase process will not be running 
> inside Namenode.
>   We will continue discussion on Internal SPS. Current code base 
> supports both internal and external option. We have review comments 
> for Internal which needs some additional works for analysis and 
> testing etc. We will move Internal SPS work to under HDFS-12226 
> (Follow-on work for SPS in NN) We are working on cleanup task HDFS-13076 for 
> the merge. .
> For more clarity on Internal and External SPS proposal thoughts, 
> please refer to JIRA HDFS-10285.
>
> If there are no objections with this, I will go ahead for voting soon.
>
> Regards,
> Uma
>
> On Fri, Nov 17, 2017 at 3:16 PM, Uma Maheswara Rao G 
>  > wrote:
>
>> Update: We worked on the review comments and additional JIRAs above 
>> mentioned.
>>
>> >1. After the feedbacks from Andrew, Eddy, Xiao in JIRA reviews, we
>> planned to take up the support for recursive API support. HDFS-12291< 
>> https://issues.apache.org/jira/browse/HDFS-12291>
>>
>> We provided the recursive API support now.
>>
>> >2. Xattr optimizations HDFS-12225> he.org/jira/browse/HDFS-12225>
>> Improved this portion as well
>>
>> >3. Few other review comments already fixed and committed HDFS-12214<
>> https://issues.apache.org/jira/browse/HDFS-12214>
>> Fixed the comments.
>>
>> We are continuing to test the feature and working so far well. Also 
>> we uploaded a combined patch and got the good QA report.
>>
>> If there are no further objections, we would like to go for merge 
>> vote tomorrow. Please by default this feature will be disabled.
>>
>> Regards,
>> Uma
>>
>> On Fri, Aug 18, 2017 at 11:27 PM, Gangumalla, Uma < 
>> uma.ganguma...@intel.com> wrote:
>>
>>> Hi Andrew,
>>>
>>> >Great to hear. It'd be nice to define which use cases are met by 
>>> >the
>>> current version of SPS, and which will be handled after the merge.
>>> After the discussions in JIRA, we planned to support recursive API 
>>> as well. The primary use cases we planned was for Hbase. Please 
>>> check next point for use case details.
>>>
>>> >A bit more detail in the design doc on how HBase would use this 
>>> >feature
>>> would also be helpful. Is there an HBase JIRA already?
>>> Please find the usecase details at this comment in JIRA:
>>> https://issues.apache.org/jira/browse/HDFS-10285?focusedComm
>>> entId=16120227=com.atlassian.jira.plugin.system.issueta
>>> bpanels:comment-tabpanel#comment-16120227
>>>
>>> >I also spent some more time with the design doc and posted a few
>>> questions on the JIRA.
>>> Thank you for the reviews.
>>>
>>> To summarize the discussions in JIRA:
>>> 1. After the feedbacks from Andrew, Eddy, Xiao in JIRA reviews, we 
>>> planned to take up the support for recursive API support. 
>>> HDFS-12291< https://issues.apache.org/jira/browse/HDFS-12291> 
>>> (Rakesh started the work on it) 2. Xattr optimizations 
>>> HDFS-12225 (Patch 
>>> available) 3. Few other review comments already fixed and committed 
>>> HDFS-12214< 

Re: [VOTE] Merge Storage Policy Satisfier (SPS) [HDFS-10285] feature branch to trunk

[Jitendra Pandey]

+1

On 7/31/18, 11:38 PM, "Uma Maheswara Rao G"  wrote:

Hi All,



 From the positive responses from JIRA discussion and no objections from
below DISCUSS thread [1], I am converting it to voting thread.



 Last couple of weeks we spent time on testing the feature and so far it is
working fine. Surendra uploaded a test report at HDFS-10285:  [2]



 In this phase, we provide to run SPS outside of Namenode only and as a
next phase we continue to discuss and work on to enable it as Internal SPS
as explained below. We have got clean QA report on branch and if there are
any static tool comments triggered later while running this thread, we will
make sure to fix them before merge. We committed and continue to improve
the code on trunk. Please refer to HDFS-10285 for discussion details.



 This has been a long effort and we're grateful for the support we've
received from the community. In particular, thanks to Andrew Wang, Anoop
Sam John, Anu Engineer, Chris Douglas, Daryn Sharp, Du Jingcheng , Ewan
Higgs, Jing Zhao, Kai Zheng,  Rakesh R, Ramkrishna , Surendra Singh Lilhore
, Thomas Demoor, Uma Maheswara Rao G, Vinayakumar, Virajith,  Wei Zhou,
Yuanbo Liu. Without these members effort, this feature might not have
reached to this state.



To start with, here is my +1

It will end on 6th Aug.



Regards,

Uma

[1]  https://s.apache.org/bhyu
[2]  https://s.apache.org/AXvL


On Wed, Jun 27, 2018 at 3:21 PM, Uma Maheswara Rao G 
wrote:

> Hi All,
>
>   After long discussions(offline and on JIRA) on SPS, we came to a
> conclusion on JIRA(HDFS-10285) that, we will go ahead with External SPS
> merge in first phase. In this phase process will not be running inside
> Namenode.
>   We will continue discussion on Internal SPS. Current code base supports
> both internal and external option. We have review comments for Internal
> which needs some additional works for analysis and testing etc. We will
> move Internal SPS work to under HDFS-12226 (Follow-on work for SPS in NN)
> We are working on cleanup task HDFS-13076 for the merge. .
> For more clarity on Internal and External SPS proposal thoughts, please
> refer to JIRA HDFS-10285.
>
> If there are no objections with this, I will go ahead for voting soon.
>
> Regards,
> Uma
>
> On Fri, Nov 17, 2017 at 3:16 PM, Uma Maheswara Rao G  > wrote:
>
>> Update: We worked on the review comments and additional JIRAs above
>> mentioned.
>>
>> >1. After the feedbacks from Andrew, Eddy, Xiao in JIRA reviews, we
>> planned to take up the support for recursive API support. HDFS-12291<
>> https://issues.apache.org/jira/browse/HDFS-12291>
>>
>> We provided the recursive API support now.
>>
>> >2. Xattr optimizations HDFS-12225> he.org/jira/browse/HDFS-12225>
>> Improved this portion as well
>>
>> >3. Few other review comments already fixed and committed HDFS-12214<
>> https://issues.apache.org/jira/browse/HDFS-12214>
>> Fixed the comments.
>>
>> We are continuing to test the feature and working so far well. Also we
>> uploaded a combined patch and got the good QA report.
>>
>> If there are no further objections, we would like to go for merge vote
>> tomorrow. Please by default this feature will be disabled.
>>
>> Regards,
>> Uma
>>
>> On Fri, Aug 18, 2017 at 11:27 PM, Gangumalla, Uma <
>> uma.ganguma...@intel.com> wrote:
>>
>>> Hi Andrew,
>>>
>>> >Great to hear. It'd be nice to define which use cases are met by the
>>> current version of SPS, and which will be handled after the merge.
>>> After the discussions in JIRA, we planned to support recursive API as
>>> well. The primary use cases we planned was for Hbase. Please check next
>>> point for use case details.
>>>
>>> >A bit more detail in the design doc on how HBase would use this feature
>>> would also be helpful. Is there an HBase JIRA already?
>>> Please find the usecase details at this comment in JIRA:
>>> https://issues.apache.org/jira/browse/HDFS-10285?focusedComm
>>> entId=16120227=com.atlassian.jira.plugin.system.issueta
>>> bpanels:comment-tabpanel#comment-16120227
>>>
>>> >I also spent some more time with the design doc and posted a few
>>> questions on the JIRA.
>>> Thank you for the reviews.
>>>
>>> To summarize the discussions in JIRA:
>>> 1. After the feedbacks from Andrew, Eddy, Xiao in JIRA reviews, we
>>> planned to take up the support for recursive API support. HDFS-12291<
>>> https://issues.apache.org/jira/browse/HDFS-12291> (Rakesh started the
>>> work on it)
>>> 2. Xattr optimizations 

[jira] [Resolved] (HDFS-13803) RBF: Remove FSCK from Router Web UI, because fsck is not supported currently

[Fei Hui (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-13803?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Fei Hui resolved HDFS-13803.

Resolution: Invalid

> RBF: Remove FSCK from Router Web UI, because fsck is not supported currently
> 
>
> Key: HDFS-13803
> URL: https://issues.apache.org/jira/browse/HDFS-13803
> Project: Hadoop HDFS
>  Issue Type: Bug
>Affects Versions: 2.9.1, 3.0.3
>Reporter: Fei Hui
>Priority: Major
>
> When i click FSCK on Router Web UI Utilities, i got errors
> {quote}
> HTTP ERROR 404
> Problem accessing /fsck. Reason:
> NOT_FOUND
> Powered by Jetty://
> {quote}
> I deep into the source code and find that fsck is not supported currently, So 
> i think we should remove FSCK from Router Web UI



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-13803) RBF: Remove FSCK from Router Web UI, because fsck is not supported currently

[Fei Hui (JIRA)]

Fei Hui created HDFS-13803:
--

 Summary: RBF: Remove FSCK from Router Web UI, because fsck is not 
supported currently
 Key: HDFS-13803
 URL: https://issues.apache.org/jira/browse/HDFS-13803
 Project: Hadoop HDFS
  Issue Type: Bug
Affects Versions: 3.0.3, 2.9.1
Reporter: Fei Hui


When i click FSCK on Router Web UI Utilities, i got errors
{quote}
HTTP ERROR 404
Problem accessing /fsck. Reason:

NOT_FOUND
Powered by Jetty://
{quote}
I deep into the source code and find that fsck is not supported currently, So i 
think we should remove FSCK from Router Web UI



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-13802) RBF: Remove FSCK from Router Web UI, because fsck is not supported currently

[Fei Hui (JIRA)]

Fei Hui created HDFS-13802:
--

 Summary: RBF: Remove FSCK from Router Web UI, because fsck is not 
supported currently
 Key: HDFS-13802
 URL: https://issues.apache.org/jira/browse/HDFS-13802
 Project: Hadoop HDFS
  Issue Type: Bug
Affects Versions: 3.0.3, 2.9.1
Reporter: Fei Hui


When i click FSCK on Router Web UI Utilities, i got errors
{quote}
HTTP ERROR 404
Problem accessing /fsck. Reason:

NOT_FOUND
Powered by Jetty://
{quote}
I deep into the source code and find that fsck is not supported currently, So i 
think we should remove FSCK from Router Web UI



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [VOTE] Merge Storage Policy Satisfier (SPS) [HDFS-10285] feature branch to trunk

[Shashikant Banerjee]

+1 (non-binding)

Thanks
Shashi

On 8/8/18, 9:04 AM, "Mukul Kumar Singh"  wrote:

+1, Thanks for the great work.

Thanks,
Mukul

On 08/08/18, 8:55 AM, "Rakesh Radhakrishnan"  wrote:

+1

Thanks,
Rakesh

On Wed, Aug 1, 2018 at 12:08 PM, Uma Maheswara Rao G 

wrote:

> Hi All,
>
>
>
>  From the positive responses from JIRA discussion and no objections 
from
> below DISCUSS thread [1], I am converting it to voting thread.
>
>
>
>  Last couple of weeks we spent time on testing the feature and so far 
it is
> working fine. Surendra uploaded a test report at HDFS-10285:  [2]
>
>
>
>  In this phase, we provide to run SPS outside of Namenode only and as 
a
> next phase we continue to discuss and work on to enable it as 
Internal SPS
> as explained below. We have got clean QA report on branch and if 
there are
> any static tool comments triggered later while running this thread, 
we will
> make sure to fix them before merge. We committed and continue to 
improve
> the code on trunk. Please refer to HDFS-10285 for discussion details.
>
>
>
>  This has been a long effort and we're grateful for the support we've
> received from the community. In particular, thanks to Andrew Wang, 
Anoop
> Sam John, Anu Engineer, Chris Douglas, Daryn Sharp, Du Jingcheng , 
Ewan
> Higgs, Jing Zhao, Kai Zheng,  Rakesh R, Ramkrishna , Surendra Singh 
Lilhore
> , Thomas Demoor, Uma Maheswara Rao G, Vinayakumar, Virajith,  Wei 
Zhou,
> Yuanbo Liu. Without these members effort, this feature might not have
> reached to this state.
>
>
>
> To start with, here is my +1
>
> It will end on 6th Aug.
>
>
>
> Regards,
>
> Uma
>
> [1]  https://s.apache.org/bhyu
> [2]  https://s.apache.org/AXvL
>
>
> On Wed, Jun 27, 2018 at 3:21 PM, Uma Maheswara Rao G 
 >
> wrote:
>
> > Hi All,
> >
> >   After long discussions(offline and on JIRA) on SPS, we came to a
> > conclusion on JIRA(HDFS-10285) that, we will go ahead with External 
SPS
> > merge in first phase. In this phase process will not be running 
inside
> > Namenode.
> >   We will continue discussion on Internal SPS. Current code base 
supports
> > both internal and external option. We have review comments for 
Internal
> > which needs some additional works for analysis and testing etc. We 
will
> > move Internal SPS work to under HDFS-12226 (Follow-on work for SPS 
in NN)
> > We are working on cleanup task HDFS-13076 for the merge. .
> > For more clarity on Internal and External SPS proposal thoughts, 
please
> > refer to JIRA HDFS-10285.
> >
> > If there are no objections with this, I will go ahead for voting 
soon.
> >
> > Regards,
> > Uma
> >
> > On Fri, Nov 17, 2017 at 3:16 PM, Uma Maheswara Rao G <
> hadoop@gmail.com
> > > wrote:
> >
> >> Update: We worked on the review comments and additional JIRAs above
> >> mentioned.
> >>
> >> >1. After the feedbacks from Andrew, Eddy, Xiao in JIRA reviews, we
> >> planned to take up the support for recursive API support. 
HDFS-12291<
> >> https://issues.apache.org/jira/browse/HDFS-12291>
> >>
> >> We provided the recursive API support now.
> >>
> >> >2. Xattr optimizations HDFS-12225 >> he.org/jira/browse/HDFS-12225>
> >> Improved this portion as well
> >>
> >> >3. Few other review comments already fixed and committed 
HDFS-12214<
> >> https://issues.apache.org/jira/browse/HDFS-12214>
> >> Fixed the comments.
> >>
> >> We are continuing to test the feature and working so far well. 
Also we
> >> uploaded a combined patch and got the good QA report.
> >>
> >> If there are no further objections, we would like to go for merge 
vote
> >> tomorrow. Please by default this feature will be disabled.
> >>
> >> Regards,
> >> Uma
> >>
> >> On Fri, Aug 18, 2017 at 11:27 PM, Gangumalla, Uma <
> >> uma.ganguma...@intel.com> wrote:
> >>
> >>> Hi Andrew,
> >>>
> >>> >Great to hear. It'd be nice to define which use cases are met by 
the
> >>> current version of SPS, and which will be handled after the merge.
> >>> After the discussions in JIRA, we planned to support recursive 
API as
> >>> well. The primary use 

Re: [VOTE] Merge Storage Policy Satisfier (SPS) [HDFS-10285] feature branch to trunk

[Mukul Kumar Singh]

+1, Thanks for the great work.

Thanks,
Mukul

On 08/08/18, 8:55 AM, "Rakesh Radhakrishnan"  wrote:

+1

Thanks,
Rakesh

On Wed, Aug 1, 2018 at 12:08 PM, Uma Maheswara Rao G 
wrote:

> Hi All,
>
>
>
>  From the positive responses from JIRA discussion and no objections from
> below DISCUSS thread [1], I am converting it to voting thread.
>
>
>
>  Last couple of weeks we spent time on testing the feature and so far it 
is
> working fine. Surendra uploaded a test report at HDFS-10285:  [2]
>
>
>
>  In this phase, we provide to run SPS outside of Namenode only and as a
> next phase we continue to discuss and work on to enable it as Internal SPS
> as explained below. We have got clean QA report on branch and if there are
> any static tool comments triggered later while running this thread, we 
will
> make sure to fix them before merge. We committed and continue to improve
> the code on trunk. Please refer to HDFS-10285 for discussion details.
>
>
>
>  This has been a long effort and we're grateful for the support we've
> received from the community. In particular, thanks to Andrew Wang, Anoop
> Sam John, Anu Engineer, Chris Douglas, Daryn Sharp, Du Jingcheng , Ewan
> Higgs, Jing Zhao, Kai Zheng,  Rakesh R, Ramkrishna , Surendra Singh 
Lilhore
> , Thomas Demoor, Uma Maheswara Rao G, Vinayakumar, Virajith,  Wei Zhou,
> Yuanbo Liu. Without these members effort, this feature might not have
> reached to this state.
>
>
>
> To start with, here is my +1
>
> It will end on 6th Aug.
>
>
>
> Regards,
>
> Uma
>
> [1]  https://s.apache.org/bhyu
> [2]  https://s.apache.org/AXvL
>
>
> On Wed, Jun 27, 2018 at 3:21 PM, Uma Maheswara Rao G  >
> wrote:
>
> > Hi All,
> >
> >   After long discussions(offline and on JIRA) on SPS, we came to a
> > conclusion on JIRA(HDFS-10285) that, we will go ahead with External SPS
> > merge in first phase. In this phase process will not be running inside
> > Namenode.
> >   We will continue discussion on Internal SPS. Current code base 
supports
> > both internal and external option. We have review comments for Internal
> > which needs some additional works for analysis and testing etc. We will
> > move Internal SPS work to under HDFS-12226 (Follow-on work for SPS in 
NN)
> > We are working on cleanup task HDFS-13076 for the merge. .
> > For more clarity on Internal and External SPS proposal thoughts, please
> > refer to JIRA HDFS-10285.
> >
> > If there are no objections with this, I will go ahead for voting soon.
> >
> > Regards,
> > Uma
> >
> > On Fri, Nov 17, 2017 at 3:16 PM, Uma Maheswara Rao G <
> hadoop@gmail.com
> > > wrote:
> >
> >> Update: We worked on the review comments and additional JIRAs above
> >> mentioned.
> >>
> >> >1. After the feedbacks from Andrew, Eddy, Xiao in JIRA reviews, we
> >> planned to take up the support for recursive API support. HDFS-12291<
> >> https://issues.apache.org/jira/browse/HDFS-12291>
> >>
> >> We provided the recursive API support now.
> >>
> >> >2. Xattr optimizations HDFS-12225 >> he.org/jira/browse/HDFS-12225>
> >> Improved this portion as well
> >>
> >> >3. Few other review comments already fixed and committed HDFS-12214<
> >> https://issues.apache.org/jira/browse/HDFS-12214>
> >> Fixed the comments.
> >>
> >> We are continuing to test the feature and working so far well. Also we
> >> uploaded a combined patch and got the good QA report.
> >>
> >> If there are no further objections, we would like to go for merge vote
> >> tomorrow. Please by default this feature will be disabled.
> >>
> >> Regards,
> >> Uma
> >>
> >> On Fri, Aug 18, 2017 at 11:27 PM, Gangumalla, Uma <
> >> uma.ganguma...@intel.com> wrote:
> >>
> >>> Hi Andrew,
> >>>
> >>> >Great to hear. It'd be nice to define which use cases are met by the
> >>> current version of SPS, and which will be handled after the merge.
> >>> After the discussions in JIRA, we planned to support recursive API as
> >>> well. The primary use cases we planned was for Hbase. Please check 
next
> >>> point for use case details.
> >>>
> >>> >A bit more detail in the design doc on how HBase would use this
> feature
> >>> would also be helpful. Is there an HBase JIRA already?
> >>> Please find the usecase details at this comment in JIRA:
> >>> https://issues.apache.org/jira/browse/HDFS-10285?focusedComm
> >>> entId=16120227=com.atlassian.jira.plugin.system.issueta
> >>> bpanels:comment-tabpanel#comment-16120227
> >>>
> >>> >I also spent some more time with the design doc and posted a few
 

Re: [VOTE] Merge Storage Policy Satisfier (SPS) [HDFS-10285] feature branch to trunk

[Rakesh Radhakrishnan]

+1

Thanks,
Rakesh

On Wed, Aug 1, 2018 at 12:08 PM, Uma Maheswara Rao G 
wrote:

> Hi All,
>
>
>
>  From the positive responses from JIRA discussion and no objections from
> below DISCUSS thread [1], I am converting it to voting thread.
>
>
>
>  Last couple of weeks we spent time on testing the feature and so far it is
> working fine. Surendra uploaded a test report at HDFS-10285:  [2]
>
>
>
>  In this phase, we provide to run SPS outside of Namenode only and as a
> next phase we continue to discuss and work on to enable it as Internal SPS
> as explained below. We have got clean QA report on branch and if there are
> any static tool comments triggered later while running this thread, we will
> make sure to fix them before merge. We committed and continue to improve
> the code on trunk. Please refer to HDFS-10285 for discussion details.
>
>
>
>  This has been a long effort and we're grateful for the support we've
> received from the community. In particular, thanks to Andrew Wang, Anoop
> Sam John, Anu Engineer, Chris Douglas, Daryn Sharp, Du Jingcheng , Ewan
> Higgs, Jing Zhao, Kai Zheng,  Rakesh R, Ramkrishna , Surendra Singh Lilhore
> , Thomas Demoor, Uma Maheswara Rao G, Vinayakumar, Virajith,  Wei Zhou,
> Yuanbo Liu. Without these members effort, this feature might not have
> reached to this state.
>
>
>
> To start with, here is my +1
>
> It will end on 6th Aug.
>
>
>
> Regards,
>
> Uma
>
> [1]  https://s.apache.org/bhyu
> [2]  https://s.apache.org/AXvL
>
>
> On Wed, Jun 27, 2018 at 3:21 PM, Uma Maheswara Rao G  >
> wrote:
>
> > Hi All,
> >
> >   After long discussions(offline and on JIRA) on SPS, we came to a
> > conclusion on JIRA(HDFS-10285) that, we will go ahead with External SPS
> > merge in first phase. In this phase process will not be running inside
> > Namenode.
> >   We will continue discussion on Internal SPS. Current code base supports
> > both internal and external option. We have review comments for Internal
> > which needs some additional works for analysis and testing etc. We will
> > move Internal SPS work to under HDFS-12226 (Follow-on work for SPS in NN)
> > We are working on cleanup task HDFS-13076 for the merge. .
> > For more clarity on Internal and External SPS proposal thoughts, please
> > refer to JIRA HDFS-10285.
> >
> > If there are no objections with this, I will go ahead for voting soon.
> >
> > Regards,
> > Uma
> >
> > On Fri, Nov 17, 2017 at 3:16 PM, Uma Maheswara Rao G <
> hadoop@gmail.com
> > > wrote:
> >
> >> Update: We worked on the review comments and additional JIRAs above
> >> mentioned.
> >>
> >> >1. After the feedbacks from Andrew, Eddy, Xiao in JIRA reviews, we
> >> planned to take up the support for recursive API support. HDFS-12291<
> >> https://issues.apache.org/jira/browse/HDFS-12291>
> >>
> >> We provided the recursive API support now.
> >>
> >> >2. Xattr optimizations HDFS-12225 >> he.org/jira/browse/HDFS-12225>
> >> Improved this portion as well
> >>
> >> >3. Few other review comments already fixed and committed HDFS-12214<
> >> https://issues.apache.org/jira/browse/HDFS-12214>
> >> Fixed the comments.
> >>
> >> We are continuing to test the feature and working so far well. Also we
> >> uploaded a combined patch and got the good QA report.
> >>
> >> If there are no further objections, we would like to go for merge vote
> >> tomorrow. Please by default this feature will be disabled.
> >>
> >> Regards,
> >> Uma
> >>
> >> On Fri, Aug 18, 2017 at 11:27 PM, Gangumalla, Uma <
> >> uma.ganguma...@intel.com> wrote:
> >>
> >>> Hi Andrew,
> >>>
> >>> >Great to hear. It'd be nice to define which use cases are met by the
> >>> current version of SPS, and which will be handled after the merge.
> >>> After the discussions in JIRA, we planned to support recursive API as
> >>> well. The primary use cases we planned was for Hbase. Please check next
> >>> point for use case details.
> >>>
> >>> >A bit more detail in the design doc on how HBase would use this
> feature
> >>> would also be helpful. Is there an HBase JIRA already?
> >>> Please find the usecase details at this comment in JIRA:
> >>> https://issues.apache.org/jira/browse/HDFS-10285?focusedComm
> >>> entId=16120227=com.atlassian.jira.plugin.system.issueta
> >>> bpanels:comment-tabpanel#comment-16120227
> >>>
> >>> >I also spent some more time with the design doc and posted a few
> >>> questions on the JIRA.
> >>> Thank you for the reviews.
> >>>
> >>> To summarize the discussions in JIRA:
> >>> 1. After the feedbacks from Andrew, Eddy, Xiao in JIRA reviews, we
> >>> planned to take up the support for recursive API support. HDFS-12291<
> >>> https://issues.apache.org/jira/browse/HDFS-12291> (Rakesh started the
> >>> work on it)
> >>> 2. Xattr optimizations HDFS-12225 >>> he.org/jira/browse/HDFS-12225> (Patch available)
> >>> 3. Few other review comments already fixed and committed HDFS-12214<
> >>> https://issues.apache.org/jira/browse/HDFS-12214>
> >>>
> >>> For 

[jira] [Created] (HDDS-335) Fix logging for scm events

[Ajay Kumar (JIRA)]

Ajay Kumar created HDDS-335:
---

 Summary: Fix logging for scm events
 Key: HDDS-335
 URL: https://issues.apache.org/jira/browse/HDDS-335
 Project: Hadoop Distributed Data Store
  Issue Type: Bug
Reporter: Ajay Kumar
Assignee: Ajay Kumar


Logs should print event type.

 \{code}java.lang.IllegalArgumentException: No event handler registered for 
event org.apache.hadoop.hdds.server.events.TypedEvent@69464649
 at 
org.apache.hadoop.hdds.server.events.EventQueue.fireEvent(EventQueue.java:116)
 at 
org.apache.hadoop.hdds.scm.server.SCMDatanodeHeartbeatDispatcher.dispatch(SCMDatanodeHeartbeatDispatcher.java:66)
 at 
org.apache.hadoop.hdds.scm.server.SCMDatanodeProtocolServer.sendHeartbeat(SCMDatanodeProtocolServer.java:219)
 at 
org.apache.hadoop.ozone.protocolPB.StorageContainerDatanodeProtocolServerSideTranslatorPB.sendHeartbeat(StorageContainerDatanodeProtocolServerSideTranslatorPB.java:90)
 at 
org.apache.hadoop.hdds.protocol.proto.StorageContainerDatanodeProtocolProtos$StorageContainerDatanodeProtocolService$2.callBlockingMethod(StorageContainerDatanodeProtocolProtos.java:19310)\{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.1.1 - RC0

[Chandni Singh]

Thanks Wangda!

+1 (non-binding)

Tested the following:
- Built from source and ran a single node cluster
- Ran the example pi job
- Launched yarn service sleep example
- Verified upgrade of yarn service sleep

Thanks,
Chandni


On Tue, Aug 7, 2018 at 7:02 PM Suma Shivaprasad <
sumasai.shivapra...@gmail.com> wrote:

> Thanks Wangda!
>
> +1 (non-binding)
>
> Tested the following:
>   - Built from source
>   - Setup single node cluster
>   - Tested Dynamic queues
>   - Tested MR and DS with default, docker runtime
>   - Tested Yarn Services with various restart policies
>
> Thanks
> Suma
>
>
> On Tue, Aug 7, 2018 at 2:45 PM Eric Payne 
> wrote:
>
> > Thanks Wangda for creating this release.
> >
> > +1 (binding)
> > Tested:
> > - Built from source
> > - Deployed to 6-node, multi-tennant, unsecured pseudo cluster with
> > hierarchical queue structure (CS)
> > - Refreshed queue (CS) properties
> > - Intra-queue preemption (CS)
> > - inter-queue preemption (CS)
> > - User weights (CS)
> >
> > Issues:
> > - Inter-queue preemption seems to be preempting unnecessarily (flapping)
> > when the queue balancing feature is enabled. This does not seem to be
> > specific to this release.
> > - The preemption-to-balance-queue-after-satisfied.enabled property seems
> > to always be enabled, but again, that is not specific to this release.
> >
> >
> > Eric
> >
> >
> > On Thursday, August 2, 2018, 1:44:22 PM CDT, Wangda Tan <
> > wheele...@gmail.com> wrote:
> >
> >
> >
> >
> >
> > Hi folks,
> >
> > I've created RC0 for Apache Hadoop 3.1.1. The artifacts are available
> here:
> >
> > http://people.apache.org/~wangda/hadoop-3.1.1-RC0/
> >
> > The RC tag in git is release-3.1.1-RC0:
> > https://github.com/apache/hadoop/commits/release-3.1.1-RC0
> >
> > The maven artifacts are available via repository.apache.org at
> > https://repository.apache.org/content/repositories/orgapachehadoop-1139/
> >
> > You can find my public key at
> > http://svn.apache.org/repos/asf/hadoop/common/dist/KEYS
> >
> > This vote will run 5 days from now.
> >
> > 3.1.1 contains 435 [1] fixed JIRA issues since 3.1.0.
> >
> > I have done testing with a pseudo cluster and distributed shell job. My
> +1
> > to start.
> >
> > Best,
> > Wangda Tan
> >
> > [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.1)
> > ORDER BY priority DESC
> >
> > -
> > To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
> > For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
> >
> >
>

Re: [VOTE] Release Apache Hadoop 3.1.1 - RC0

[Suma Shivaprasad]

Thanks Wangda!

+1 (non-binding)

Tested the following:
  - Built from source
  - Setup single node cluster
  - Tested Dynamic queues
  - Tested MR and DS with default, docker runtime
  - Tested Yarn Services with various restart policies

Thanks
Suma


On Tue, Aug 7, 2018 at 2:45 PM Eric Payne 
wrote:

> Thanks Wangda for creating this release.
>
> +1 (binding)
> Tested:
> - Built from source
> - Deployed to 6-node, multi-tennant, unsecured pseudo cluster with
> hierarchical queue structure (CS)
> - Refreshed queue (CS) properties
> - Intra-queue preemption (CS)
> - inter-queue preemption (CS)
> - User weights (CS)
>
> Issues:
> - Inter-queue preemption seems to be preempting unnecessarily (flapping)
> when the queue balancing feature is enabled. This does not seem to be
> specific to this release.
> - The preemption-to-balance-queue-after-satisfied.enabled property seems
> to always be enabled, but again, that is not specific to this release.
>
>
> Eric
>
>
> On Thursday, August 2, 2018, 1:44:22 PM CDT, Wangda Tan <
> wheele...@gmail.com> wrote:
>
>
>
>
>
> Hi folks,
>
> I've created RC0 for Apache Hadoop 3.1.1. The artifacts are available here:
>
> http://people.apache.org/~wangda/hadoop-3.1.1-RC0/
>
> The RC tag in git is release-3.1.1-RC0:
> https://github.com/apache/hadoop/commits/release-3.1.1-RC0
>
> The maven artifacts are available via repository.apache.org at
> https://repository.apache.org/content/repositories/orgapachehadoop-1139/
>
> You can find my public key at
> http://svn.apache.org/repos/asf/hadoop/common/dist/KEYS
>
> This vote will run 5 days from now.
>
> 3.1.1 contains 435 [1] fixed JIRA issues since 3.1.0.
>
> I have done testing with a pseudo cluster and distributed shell job. My +1
> to start.
>
> Best,
> Wangda Tan
>
> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.1)
> ORDER BY priority DESC
>
> -
> To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
>
>

[jira] [Created] (HDDS-334) Update GettingStarted page to mention details about Ozone GenConf tool

[Dinesh Chitlangia (JIRA)]

Dinesh Chitlangia created HDDS-334:
--

 Summary: Update GettingStarted page to mention details about Ozone 
GenConf tool
 Key: HDDS-334
 URL: https://issues.apache.org/jira/browse/HDDS-334
 Project: Hadoop Distributed Data Store
  Issue Type: Bug
Reporter: Dinesh Chitlangia
Assignee: Dinesh Chitlangia


Add description about Ozone GenConf tool in GettingStarted page



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-333) Create an Ozone Logo

[Anu Engineer (JIRA)]

Anu Engineer created HDDS-333:
-

 Summary: Create an Ozone Logo
 Key: HDDS-333
 URL: https://issues.apache.org/jira/browse/HDDS-333
 Project: Hadoop Distributed Data Store
  Issue Type: Improvement
  Components: Ozone Manager
Reporter: Anu Engineer
 Fix For: 0.2.1


As part of developing Ozone Website and Documentation, It would be nice to have 
an Ozone Logo.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-332) Remove the ability to configure ozone.handler.type

[Dinesh Chitlangia (JIRA)]

Dinesh Chitlangia created HDDS-332:
--

 Summary: Remove the ability to configure ozone.handler.type
 Key: HDDS-332
 URL: https://issues.apache.org/jira/browse/HDDS-332
 Project: Hadoop Distributed Data Store
  Issue Type: Bug
Reporter: Dinesh Chitlangia
Assignee: Dinesh Chitlangia


Based on feedback from [~anu] and [~xyao], logging this Jira to remove the 
ability to configure ozone.handler.type.

Currently, it can be configured with values - distributed or local.

Value 'local' was strictly for internal testing purposes in initial stages. By 
removing the configurability, we can avoid situations where users may end up 
misconfiguring and running into issues.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Apache Hadoop qbt Report: trunk+JDK8 on Windows/x64

[Apache Jenkins Server]

For more details, see https://builds.apache.org/job/hadoop-trunk-win/551/

[Aug 6, 2018 4:37:59 PM] (eyang) YARN-8624. Updated verbiage around entry point 
support.   
[Aug 7, 2018 1:32:51 AM] (yqlin) HDFS-13792. Fix FSN read/write lock metrics 
name. Contributed by Chao
[Aug 7, 2018 9:33:14 AM] (msingh) HDDS-230. ContainerStateMachine should 
implement readStateMachineData
[Aug 7, 2018 10:39:53 AM] (msingh) HDDS-301. ozone command shell does not 
contain subcommand to run ozoneFS




-1 overall


The following subsystems voted -1:
compile mvninstall pathlen unit


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc javac


The following subsystems are considered long running:
(runtime bigger than 1h 00m 00s)
unit


Specific tests:

Failed junit tests :

   hadoop.crypto.key.kms.server.TestKMS 
   hadoop.cli.TestAclCLI 
   hadoop.cli.TestAclCLIWithPosixAclInheritance 
   hadoop.cli.TestCacheAdminCLI 
   hadoop.cli.TestCryptoAdminCLI 
   hadoop.cli.TestDeleteCLI 
   hadoop.cli.TestErasureCodingCLI 
   hadoop.cli.TestHDFSCLI 
   hadoop.cli.TestXAttrCLI 
   hadoop.fs.contract.hdfs.TestHDFSContractAppend 
   hadoop.fs.contract.hdfs.TestHDFSContractConcat 
   hadoop.fs.contract.hdfs.TestHDFSContractCreate 
   hadoop.fs.contract.hdfs.TestHDFSContractDelete 
   hadoop.fs.contract.hdfs.TestHDFSContractGetFileStatus 
   hadoop.fs.contract.hdfs.TestHDFSContractMkdir 
   hadoop.fs.contract.hdfs.TestHDFSContractOpen 
   hadoop.fs.contract.hdfs.TestHDFSContractPathHandle 
   hadoop.fs.contract.hdfs.TestHDFSContractRename 
   hadoop.fs.contract.hdfs.TestHDFSContractRootDirectory 
   hadoop.fs.contract.hdfs.TestHDFSContractSeek 
   hadoop.fs.contract.hdfs.TestHDFSContractSetTimes 
   hadoop.fs.loadGenerator.TestLoadGenerator 
   hadoop.fs.permission.TestStickyBit 
   hadoop.fs.shell.TestHdfsTextCommand 
   hadoop.fs.TestEnhancedByteBufferAccess 
   hadoop.fs.TestFcHdfsCreateMkdir 
   hadoop.fs.TestFcHdfsPermission 
   hadoop.fs.TestFcHdfsSetUMask 
   hadoop.fs.TestGlobPaths 
   hadoop.fs.TestHDFSFileContextMainOperations 
   hadoop.fs.TestHDFSMultipartUploader 
   hadoop.fs.TestHdfsNativeCodeLoader 
   hadoop.fs.TestResolveHdfsSymlink 
   hadoop.fs.TestSWebHdfsFileContextMainOperations 
   hadoop.fs.TestSymlinkHdfsDisable 
   hadoop.fs.TestSymlinkHdfsFileContext 
   hadoop.fs.TestSymlinkHdfsFileSystem 
   hadoop.fs.TestUnbuffer 
   hadoop.fs.TestUrlStreamHandler 
   hadoop.fs.TestWebHdfsFileContextMainOperations 
   hadoop.fs.viewfs.TestViewFileSystemAtHdfsRoot 
   hadoop.fs.viewfs.TestViewFileSystemHdfs 
   hadoop.fs.viewfs.TestViewFileSystemLinkFallback 
   hadoop.fs.viewfs.TestViewFileSystemLinkMergeSlash 
   hadoop.fs.viewfs.TestViewFileSystemWithAcls 
   hadoop.fs.viewfs.TestViewFileSystemWithTruncate 
   hadoop.fs.viewfs.TestViewFileSystemWithXAttrs 
   hadoop.fs.viewfs.TestViewFsAtHdfsRoot 
   hadoop.fs.viewfs.TestViewFsDefaultValue 
   hadoop.fs.viewfs.TestViewFsFileStatusHdfs 
   hadoop.fs.viewfs.TestViewFsHdfs 
   hadoop.fs.viewfs.TestViewFsWithAcls 
   hadoop.fs.viewfs.TestViewFsWithXAttrs 
   hadoop.hdfs.client.impl.TestBlockReaderLocal 
   hadoop.hdfs.client.impl.TestBlockReaderLocalLegacy 
   hadoop.hdfs.client.impl.TestBlockReaderRemote 
   hadoop.hdfs.client.impl.TestClientBlockVerification 
   hadoop.hdfs.crypto.TestHdfsCryptoStreams 
   hadoop.hdfs.protocol.datatransfer.sasl.TestSaslDataTransfer 
   hadoop.hdfs.qjournal.client.TestEpochsAreUnique 
   hadoop.hdfs.qjournal.client.TestQJMWithFaults 
   hadoop.hdfs.qjournal.client.TestQuorumJournalManager 
   hadoop.hdfs.qjournal.server.TestJournal 
   hadoop.hdfs.qjournal.server.TestJournalNode 
   hadoop.hdfs.qjournal.server.TestJournalNodeMXBean 
   hadoop.hdfs.qjournal.server.TestJournalNodeRespectsBindHostKeys 
   hadoop.hdfs.qjournal.server.TestJournalNodeSync 
   hadoop.hdfs.qjournal.TestMiniJournalCluster 
   hadoop.hdfs.qjournal.TestNNWithQJM 
   hadoop.hdfs.qjournal.TestSecureNNWithQJM 
   hadoop.hdfs.security.TestDelegationToken 
   hadoop.hdfs.security.TestDelegationTokenForProxyUser 
   hadoop.hdfs.security.token.block.TestBlockToken 
   hadoop.hdfs.server.balancer.TestBalancer 
   hadoop.hdfs.server.balancer.TestBalancerRPCDelay 
   hadoop.hdfs.server.balancer.TestBalancerWithEncryptedTransfer 
   hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes 
   hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes 
   hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup 
   hadoop.hdfs.server.balancer.TestBalancerWithSaslDataTransfer 
   
hadoop.hdfs.server.blockmanagement.TestAvailableSpaceBlockPlacementPolicy 
   

Re: [VOTE] Release Apache Hadoop 3.1.1 - RC0

[Eric Payne]

Thanks Wangda for creating this release.

+1 (binding)
Tested:
- Built from source
- Deployed to 6-node, multi-tennant, unsecured pseudo cluster with hierarchical 
queue structure (CS)
- Refreshed queue (CS) properties
- Intra-queue preemption (CS)
- inter-queue preemption (CS)
- User weights (CS)

Issues:
- Inter-queue preemption seems to be preempting unnecessarily (flapping) when 
the queue balancing feature is enabled. This does not seem to be specific to 
this release.
- The preemption-to-balance-queue-after-satisfied.enabled property seems to 
always be enabled, but again, that is not specific to this release.


Eric


On Thursday, August 2, 2018, 1:44:22 PM CDT, Wangda Tan  
wrote: 





Hi folks,

I've created RC0 for Apache Hadoop 3.1.1. The artifacts are available here:

http://people.apache.org/~wangda/hadoop-3.1.1-RC0/

The RC tag in git is release-3.1.1-RC0:
https://github.com/apache/hadoop/commits/release-3.1.1-RC0

The maven artifacts are available via repository.apache.org at
https://repository.apache.org/content/repositories/orgapachehadoop-1139/

You can find my public key at
http://svn.apache.org/repos/asf/hadoop/common/dist/KEYS

This vote will run 5 days from now.

3.1.1 contains 435 [1] fixed JIRA issues since 3.1.0.

I have done testing with a pseudo cluster and distributed shell job. My +1
to start.

Best,
Wangda Tan

[1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.1)
ORDER BY priority DESC

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Resolved] (HDDS-331) Fix docker-compose for ozone-perf

[Xiaoyu Yao (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDDS-331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiaoyu Yao resolved HDDS-331.
-
Resolution: Not A Bug

Need to run the init.sh script to copy the jar. Problem is fixed after that. 

> Fix docker-compose for ozone-perf
> -
>
> Key: HDDS-331
> URL: https://issues.apache.org/jira/browse/HDDS-331
> Project: Hadoop Distributed Data Store
>  Issue Type: Bug
>Reporter: Xiaoyu Yao
>Assignee: Xiaoyu Yao
>Priority: Major
>
> OM/SCM/DN exit  with the following failures.
> {code}
> ozoneManager_1  | Error occurred during initialization of VM
> ozoneManager_1  | agent library failed to init: instrument
> ozoneManager_1  | Error opening zip file or JAR manifest missing : 
> /opt/jmxpromo.jar
> ...
> scm_1           | WARNING: /opt/hadoop/logs does not exist. Creating.
> scm_1           | Error opening zip file or JAR manifest missing : 
> /opt/jmxpromo.jar
> scm_1           | Error occurred during initialization of VM
> scm_1           | agent library failed to init: instrument
>  
> datanode_1      | Error opening zip file or JAR manifest missing : 
> /opt/jmxpromo.jar
> datanode_1      | Error occurred during initialization of VM
> datanode_1      | agent library failed to init: instrument
> {code}
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-13801) Fix docker-compose for ozone-perf

[Xiaoyu Yao (JIRA)]

Xiaoyu Yao created HDFS-13801:
-

 Summary: Fix docker-compose for ozone-perf
 Key: HDFS-13801
 URL: https://issues.apache.org/jira/browse/HDFS-13801
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Xiaoyu Yao
Assignee: Xiaoyu Yao


OM/SCM/DN exit  with the following failures.

{code}

ozoneManager_1  | Error occurred during initialization of VM

ozoneManager_1  | agent library failed to init: instrument

ozoneManager_1  | Error opening zip file or JAR manifest missing : 
/opt/jmxpromo.jar

...

scm_1           | WARNING: /opt/hadoop/logs does not exist. Creating.

scm_1           | Error opening zip file or JAR manifest missing : 
/opt/jmxpromo.jar

scm_1           | Error occurred during initialization of VM

scm_1           | agent library failed to init: instrument

 

datanode_1      | Error opening zip file or JAR manifest missing : 
/opt/jmxpromo.jar

datanode_1      | Error occurred during initialization of VM

datanode_1      | agent library failed to init: instrument

{code}

 

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.1.1 - RC0

[Wangda Tan]

Thanks everyone for voting!

With 4 binding votes, 10 non-binding votes and no veto. The vote passed,
I'm going to work on staging the release.

Best,
Wangda


On Tue, Aug 7, 2018 at 12:33 PM Nandakumar Vadivelu <
nvadiv...@hortonworks.com> wrote:

> Thanks for all the hard work Wangda!
> +1
>
> * Verified pgp signatures
> * Verified MD5 checksums
> * Verified both source and binary releases
> * Installed pseudo distributed cluster
> * Verified hdfs filesystem commands
> * Verified hdfs admin commands
> * Verified hdfs snapshot commands
> * Basic sanity check on NN UI
>
> Issue found
> * In "Browse the file system" under "Utilities" tab in NN UI, "Head the
> file" and "Tail the file" are not working.
>
> -Nanda
>
> On 8/3/18, 12:14 AM, "Wangda Tan"  wrote:
>
> Hi folks,
>
> I've created RC0 for Apache Hadoop 3.1.1. The artifacts are available
> here:
>
> http://people.apache.org/~wangda/hadoop-3.1.1-RC0/
>
> The RC tag in git is release-3.1.1-RC0:
> https://github.com/apache/hadoop/commits/release-3.1.1-RC0
>
> The maven artifacts are available via repository.apache.org at
>
> https://repository.apache.org/content/repositories/orgapachehadoop-1139/
>
> You can find my public key at
> http://svn.apache.org/repos/asf/hadoop/common/dist/KEYS
>
> This vote will run 5 days from now.
>
> 3.1.1 contains 435 [1] fixed JIRA issues since 3.1.0.
>
> I have done testing with a pseudo cluster and distributed shell job.
> My +1
> to start.
>
> Best,
> Wangda Tan
>
> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in
> (3.1.1)
> ORDER BY priority DESC
>
>
>

Re: [VOTE] Release Apache Hadoop 3.1.1 - RC0

[Nandakumar Vadivelu]

Thanks for all the hard work Wangda!
+1

* Verified pgp signatures
* Verified MD5 checksums
* Verified both source and binary releases
* Installed pseudo distributed cluster
* Verified hdfs filesystem commands
* Verified hdfs admin commands
* Verified hdfs snapshot commands
* Basic sanity check on NN UI

Issue found
* In "Browse the file system" under "Utilities" tab in NN UI, "Head the file" 
and "Tail the file" are not working.

-Nanda

On 8/3/18, 12:14 AM, "Wangda Tan"  wrote:

Hi folks,

I've created RC0 for Apache Hadoop 3.1.1. The artifacts are available here:

http://people.apache.org/~wangda/hadoop-3.1.1-RC0/

The RC tag in git is release-3.1.1-RC0:
https://github.com/apache/hadoop/commits/release-3.1.1-RC0

The maven artifacts are available via repository.apache.org at
https://repository.apache.org/content/repositories/orgapachehadoop-1139/

You can find my public key at
http://svn.apache.org/repos/asf/hadoop/common/dist/KEYS

This vote will run 5 days from now.

3.1.1 contains 435 [1] fixed JIRA issues since 3.1.0.

I have done testing with a pseudo cluster and distributed shell job. My +1
to start.

Best,
Wangda Tan

[1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.1)
ORDER BY priority DESC

[jira] [Created] (HDFS-13800) Improve the error message when contacting an IPC port via a browser

[Daniel Templeton (JIRA)]

Daniel Templeton created HDFS-13800:
---

 Summary: Improve the error message when contacting an IPC port via 
a browser
 Key: HDFS-13800
 URL: https://issues.apache.org/jira/browse/HDFS-13800
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: hdfs
Affects Versions: 3.0.0
Reporter: Daniel Templeton
Assignee: Daniel Templeton


When I point a browser at {{http://:9000}}, get get back a 404 with 
the following text: {quote}It looks like you are making an HTTP request to a 
Hadoop IPC port. This is not the correct port for the web interface on this 
daemon.{quote}  While accurate, that's not exactly helpful.  It would be worlds 
more useful to include the URL for the web UI in the text.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.1.1 - RC0

[Gabor Bota]

Thanks for the work Wangda Tan!

 +1 (non-binding)

checked out git tag release-3.1.1-RC0
hadoop-aws integration (mvn verify) test run was successful on eu-west-1
built from source on Mac OS X 10.13.6, java version 8.0.181-oracle
deployed on a 3 node cluster
verified pi job (yarn), teragen, terasort and teravalidate

Regards,
  Gabor Bota

On Thu, Aug 2, 2018 at 8:44 PM Wangda Tan  wrote:

> Hi folks,
>
> I've created RC0 for Apache Hadoop 3.1.1. The artifacts are available here:
>
> http://people.apache.org/~wangda/hadoop-3.1.1-RC0/
>
> The RC tag in git is release-3.1.1-RC0:
> https://github.com/apache/hadoop/commits/release-3.1.1-RC0
>
> The maven artifacts are available via repository.apache.org at
> https://repository.apache.org/content/repositories/orgapachehadoop-1139/
>
> You can find my public key at
> http://svn.apache.org/repos/asf/hadoop/common/dist/KEYS
>
> This vote will run 5 days from now.
>
> 3.1.1 contains 435 [1] fixed JIRA issues since 3.1.0.
>
> I have done testing with a pseudo cluster and distributed shell job. My +1
> to start.
>
> Best,
> Wangda Tan
>
> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.1)
> ORDER BY priority DESC
>

Re: [VOTE] Release Apache Hadoop 3.1.1 - RC0

[Shane Kumpf]

+1 (binding) - Thanks for putting this together Wangda.

- Built from source
- Deployed a single node cluster on CentOS 7.4
- Basic dfs command checkout
- Ran dshell, MR pi, and MR sleep with the default and docker runtimes
- Validated the ability to disable the docker runtime
- Validated the ability to disable privileged docker containers
- Successfully launched a docker based YARN service
- Validated basic lookups against Registry DNS

-Shane

On Thu, Aug 2, 2018 at 12:44 PM Wangda Tan  wrote:

> Hi folks,
>
> I've created RC0 for Apache Hadoop 3.1.1. The artifacts are available here:
>
> http://people.apache.org/~wangda/hadoop-3.1.1-RC0/
>
> The RC tag in git is release-3.1.1-RC0:
> https://github.com/apache/hadoop/commits/release-3.1.1-RC0
>
> The maven artifacts are available via repository.apache.org at
> https://repository.apache.org/content/repositories/orgapachehadoop-1139/
>
> You can find my public key at
> http://svn.apache.org/repos/asf/hadoop/common/dist/KEYS
>
> This vote will run 5 days from now.
>
> 3.1.1 contains 435 [1] fixed JIRA issues since 3.1.0.
>
> I have done testing with a pseudo cluster and distributed shell job. My +1
> to start.
>
> Best,
> Wangda Tan
>
> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.1)
> ORDER BY priority DESC
>

Re: [VOTE] Release Apache Hadoop 3.1.1 - RC0

[Craig . Condit]

+1 (non-binding).

Built from tag, spun up single-node cluster, ran basic DFS commands, various MR 
/ YARN jobs with and without Docker.

Craig Condit

On 2018/08/02 18:43:50, Wangda Tan mailto:w...@gmail.com>> 
wrote:
> Hi folks,>
>
> I've created RC0 for Apache Hadoop 3.1.1. The artifacts are available here:>
>
> http://people.apache.org/~wangda/hadoop-3.1.1-RC0/>
>
> The RC tag in git is release-3.1.1-RC0:>
> https://github.com/apache/hadoop/commits/release-3.1.1-RC0>
>
> The maven artifacts are available via repository.apache.org at>
> https://repository.apache.org/content/repositories/orgapachehadoop-1139/>
>
> You can find my public key at>
> http://svn.apache.org/repos/asf/hadoop/common/dist/KEYS>
>
> This vote will run 5 days from now.>
>
> 3.1.1 contains 435 [1] fixed JIRA issues since 3.1.0.>
>
> I have done testing with a pseudo cluster and distributed shell job. My +1>
> to start.>
>
> Best,>
> Wangda Tan>
>
> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.1)>
> ORDER BY priority DESC>
>

Re: [VOTE] Release Apache Hadoop 3.1.1 - RC0

[Ajay Kumar]

 +1 (non-binding)  Thanks for working on this Wangda!!

- built from source on mac, jdk1.8.0_111
- deployed on a 3 node non-secure cluster
- tested basic hdfs cli functionality (put,get, delete,ls etc)
- executed sample MR jobs (wordcount, TestDFSIO read/write)
- Basic sanity check for NN and RM UI

Best,
Ajay

On 8/7/18, 8:51 AM, "Kitti Nánási"  wrote:

Thanks Wangda for working on this!

+1 (non-binding)

- checked out git tag release-3.1.1-RC0
- built from source on Mac OS X 10.13.4, java version 8.0.172-zulu
- deployed on a 3 node cluster
- ran terasort, teragen, teravalidate with success
- executed basic hdfs and dfsadmin commands
- executed unit tests for hadoop-hdfs-project/hadoop-hdfs-client and
 hadoop-hdfs-project/hadoop-- hdfs-httpfs modules that have all passed

Thanks,
Kitti


On Tue, Aug 7, 2018 at 5:23 PM, Brahma Reddy Battula 
wrote:

> +1 ,(binding).
>
> Wangda Tan thanks for driving this.
>
>
> --Built with source.
> --Installed the HA cluster
> --Ran basic hdfs operations through shell/API
> --Ran Sample jobs like pi, wordcount
> --Browsed UI.
>
> On Fri, Aug 3, 2018 at 12:14 AM Wangda Tan  wrote:
>
> > Hi folks,
> >
> > I've created RC0 for Apache Hadoop 3.1.1. The artifacts are available
> here:
> >
> > http://people.apache.org/~wangda/hadoop-3.1.1-RC0/
> >
> > The RC tag in git is release-3.1.1-RC0:
> > https://github.com/apache/hadoop/commits/release-3.1.1-RC0
> >
> > The maven artifacts are available via repository.apache.org at
> > https://repository.apache.org/content/repositories/orgapachehadoop-1139/
> >
> > You can find my public key at
> > http://svn.apache.org/repos/asf/hadoop/common/dist/KEYS
> >
> > This vote will run 5 days from now.
> >
> > 3.1.1 contains 435 [1] fixed JIRA issues since 3.1.0.
> >
> > I have done testing with a pseudo cluster and distributed shell job. My
> +1
> > to start.
> >
> > Best,
> > Wangda Tan
> >
> > [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.1)
> > ORDER BY priority DESC
> >
>



-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.1.1 - RC0

[Kitti Nánási]

Thanks Wangda for working on this!

+1 (non-binding)

- checked out git tag release-3.1.1-RC0
- built from source on Mac OS X 10.13.4, java version 8.0.172-zulu
- deployed on a 3 node cluster
- ran terasort, teragen, teravalidate with success
- executed basic hdfs and dfsadmin commands
- executed unit tests for hadoop-hdfs-project/hadoop-hdfs-client and
 hadoop-hdfs-project/hadoop-- hdfs-httpfs modules that have all passed

Thanks,
Kitti


On Tue, Aug 7, 2018 at 5:23 PM, Brahma Reddy Battula 
wrote:

> +1 ,(binding).
>
> Wangda Tan thanks for driving this.
>
>
> --Built with source.
> --Installed the HA cluster
> --Ran basic hdfs operations through shell/API
> --Ran Sample jobs like pi, wordcount
> --Browsed UI.
>
> On Fri, Aug 3, 2018 at 12:14 AM Wangda Tan  wrote:
>
> > Hi folks,
> >
> > I've created RC0 for Apache Hadoop 3.1.1. The artifacts are available
> here:
> >
> > http://people.apache.org/~wangda/hadoop-3.1.1-RC0/
> >
> > The RC tag in git is release-3.1.1-RC0:
> > https://github.com/apache/hadoop/commits/release-3.1.1-RC0
> >
> > The maven artifacts are available via repository.apache.org at
> > https://repository.apache.org/content/repositories/orgapachehadoop-1139/
> >
> > You can find my public key at
> > http://svn.apache.org/repos/asf/hadoop/common/dist/KEYS
> >
> > This vote will run 5 days from now.
> >
> > 3.1.1 contains 435 [1] fixed JIRA issues since 3.1.0.
> >
> > I have done testing with a pseudo cluster and distributed shell job. My
> +1
> > to start.
> >
> > Best,
> > Wangda Tan
> >
> > [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.1)
> > ORDER BY priority DESC
> >
>

Re: [VOTE] Release Apache Hadoop 3.1.1 - RC0

[Brahma Reddy Battula]

+1 ,(binding).

Wangda Tan thanks for driving this.


--Built with source.
--Installed the HA cluster
--Ran basic hdfs operations through shell/API
--Ran Sample jobs like pi, wordcount
--Browsed UI.

On Fri, Aug 3, 2018 at 12:14 AM Wangda Tan  wrote:

> Hi folks,
>
> I've created RC0 for Apache Hadoop 3.1.1. The artifacts are available here:
>
> http://people.apache.org/~wangda/hadoop-3.1.1-RC0/
>
> The RC tag in git is release-3.1.1-RC0:
> https://github.com/apache/hadoop/commits/release-3.1.1-RC0
>
> The maven artifacts are available via repository.apache.org at
> https://repository.apache.org/content/repositories/orgapachehadoop-1139/
>
> You can find my public key at
> http://svn.apache.org/repos/asf/hadoop/common/dist/KEYS
>
> This vote will run 5 days from now.
>
> 3.1.1 contains 435 [1] fixed JIRA issues since 3.1.0.
>
> I have done testing with a pseudo cluster and distributed shell job. My +1
> to start.
>
> Best,
> Wangda Tan
>
> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.1)
> ORDER BY priority DESC
>

Re: [VOTE] Release Apache Hadoop 3.1.1 - RC0

[Eric Badger]

+1 (non-binding)

- Verified all hashes and checksums
- Built from source on macOS 10.13.6, Java 1.8.0u65
- Deployed a pseudo cluster
- Ran some example jobs


On Tue, Aug 7, 2018 at 8:46 AM, Shashikant Banerjee <
sbaner...@hortonworks.com> wrote:

> +1(non-binding)
>
> * checked out git tag release-3.1.1-RC0
> * built from source
> * deployed on a single node cluster
> * executed basic dfs commands
> * executed some snapshot commands
>
> Thank you very much the work Wangda.
>
> Thanks
> Shashi
>
> On 8/7/18, 6:43 PM, "Elek, Marton"  wrote:
>
>
> +1 (non-binding)
>
> 1. Built from the source package.
> 2. Checked the signature
> 3. Started docker based pseudo cluster and smoketested some basic
> functionality (hdfs cli, ec cli, viewfs, yarn examples, spark word
> count
> job)
>
> Thank you very much the work Wangda.
> Marton
>
>
> On 08/02/2018 08:43 PM, Wangda Tan wrote:
> > Hi folks,
> >
> > I've created RC0 for Apache Hadoop 3.1.1. The artifacts are
> available here:
> >
> > http://people.apache.org/~wangda/hadoop-3.1.1-RC0/
> >
> > The RC tag in git is release-3.1.1-RC0:
> > https://github.com/apache/hadoop/commits/release-3.1.1-RC0
> >
> > The maven artifacts are available via repository.apache.org at
> > https://repository.apache.org/content/repositories/
> orgapachehadoop-1139/
> >
> > You can find my public key at
> > http://svn.apache.org/repos/asf/hadoop/common/dist/KEYS
> >
> > This vote will run 5 days from now.
> >
> > 3.1.1 contains 435 [1] fixed JIRA issues since 3.1.0.
> >
> > I have done testing with a pseudo cluster and distributed shell job.
> My +1
> > to start.
> >
> > Best,
> > Wangda Tan
> >
> > [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in
> (3.1.1)
> > ORDER BY priority DESC
> >
>
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>
>
>
>
> -
> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
>
>

Re: 回复：[VOTE] Merge Storage Policy Satisfier (SPS) [HDFS-10285] feature branch to trunk

[Nandakumar Vadivelu]

+1 for the merge. Thanks for the work Uma et al.

-Nanda

On 8/7/18, 7:35 PM, "Vinayakumar B"  wrote:

+1

Great work guys.
-Vinay

On Tue, 7 Aug 2018, 7:20 pm Anu Engineer,  wrote:

> +1, Sorry for the late vote. Thanks for the perseverance and seeing this
> thru.
>
> --Anu
>
>
> On 8/7/18, 1:04 AM, "郑锴(铁杰)"  wrote:
>
> +1 for the work to be in. Thanks Uma and folks for the hard taking!
>
> When it's in, I'd suggest we use a more general name for the new
> daemon service. It'd be good to evolve and support more self-running admin
> functionalities incubated first there before doing it directly in 
NameNode.
>
> Regards,
> Kai
> --
> 发件人：Uma Maheswara Rao G 
> 发送时间：2018年8月1日(星期三) 14:38
> 收件人：hdfs-dev 
> 主 题：[VOTE] Merge Storage Policy Satisfier (SPS) [HDFS-10285] feature
> branch to trunk
>
> Hi All,
>
>
>
>  From the positive responses from JIRA discussion and no objections
> from
> below DISCUSS thread [1], I am converting it to voting thread.
>
>
>
>  Last couple of weeks we spent time on testing the feature and so far
> it is
> working fine. Surendra uploaded a test report at HDFS-10285:  [2]
>
>
>
>  In this phase, we provide to run SPS outside of Namenode only and as 
a
> next phase we continue to discuss and work on to enable it as Internal
> SPS
> as explained below. We have got clean QA report on branch and if there
> are
> any static tool comments triggered later while running this thread, we
> will
> make sure to fix them before merge. We committed and continue to
> improve
> the code on trunk. Please refer to HDFS-10285 for discussion details.
>
>
>
>  This has been a long effort and we're grateful for the support we've
> received from the community. In particular, thanks to Andrew Wang,
> Anoop
> Sam John, Anu Engineer, Chris Douglas, Daryn Sharp, Du Jingcheng , 
Ewan
> Higgs, Jing Zhao, Kai Zheng,  Rakesh R, Ramkrishna , Surendra Singh
> Lilhore
> , Thomas Demoor, Uma Maheswara Rao G, Vinayakumar, Virajith,  Wei 
Zhou,
> Yuanbo Liu. Without these members effort, this feature might not have
> reached to this state.
>
>
>
> To start with, here is my +1
>
> It will end on 6th Aug.
>
>
>
> Regards,
>
> Uma
>
> [1]  https://s.apache.org/bhyu
> [2]  https://s.apache.org/AXvL
>
>
> On Wed, Jun 27, 2018 at 3:21 PM, Uma Maheswara Rao G <
> hadoop@gmail.com>
> wrote:
>
> > Hi All,
> >
> >   After long discussions(offline and on JIRA) on SPS, we came to a
> > conclusion on JIRA(HDFS-10285) that, we will go ahead with External
> SPS
> > merge in first phase. In this phase process will not be running
> inside
> > Namenode.
> >   We will continue discussion on Internal SPS. Current code base
> supports
> > both internal and external option. We have review comments for
> Internal
> > which needs some additional works for analysis and testing etc. We
> will
> > move Internal SPS work to under HDFS-12226 (Follow-on work for SPS
> in NN)
> > We are working on cleanup task HDFS-13076 for the merge. .
> > For more clarity on Internal and External SPS proposal thoughts,
> please
> > refer to JIRA HDFS-10285.
> >
> > If there are no objections with this, I will go ahead for voting
> soon.
> >
> > Regards,
> > Uma
> >
> > On Fri, Nov 17, 2017 at 3:16 PM, Uma Maheswara Rao G <
> hadoop@gmail.com
> > > wrote:
> >
> >> Update: We worked on the review comments and additional JIRAs above
> >> mentioned.
> >>
> >> >1. After the feedbacks from Andrew, Eddy, Xiao in JIRA reviews, we
> >> planned to take up the support for recursive API support.
> HDFS-12291<
> >> https://issues.apache.org/jira/browse/HDFS-12291>
> >>
> >> We provided the recursive API support now.
> >>
> >> >2. Xattr optimizations HDFS-12225 >> he.org/jira/browse/HDFS-12225>
> >> Improved this portion as well
> >>
> >> >3. Few other review comments already fixed and committed
> HDFS-12214<
> >> https://issues.apache.org/jira/browse/HDFS-12214>
> >> Fixed the comments.
> >>
> >> We are continuing to test the feature and working so far well. Also
> we
> >> uploaded a combined patch and got the good QA report.

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

[Apache Jenkins Server]

For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/861/

[Aug 6, 2018 4:37:59 PM] (eyang) YARN-8624. Updated verbiage around entry point 
support.   
[Aug 7, 2018 1:32:51 AM] (yqlin) HDFS-13792. Fix FSN read/write lock metrics 
name. Contributed by Chao




-1 overall


The following subsystems voted -1:
asflicense findbugs pathlen unit xml


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

Failed CTEST tests :

   test_test_libhdfs_threaded_hdfs_static 
   test_libhdfs_threaded_hdfspp_test_shim_static 

Failed junit tests :

   hadoop.security.TestRaceWhenRelogin 
   hadoop.hdfs.TestPersistBlocks 
   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure 
   hadoop.hdfs.client.impl.TestBlockReaderLocal 
   hadoop.hdfs.web.TestWebHdfsTimeouts 
   hadoop.yarn.server.resourcemanager.TestRMEmbeddedElector 
   hadoop.yarn.applications.distributedshell.TestDistributedShell 
   hadoop.mapred.TestMRTimelineEventHandling 
  

   cc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/861/artifact/out/diff-compile-cc-root.txt
  [4.0K]

   javac:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/861/artifact/out/diff-compile-javac-root.txt
  [332K]

   checkstyle:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/861/artifact/out/diff-checkstyle-root.txt
  [4.0K]

   pathlen:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/861/artifact/out/pathlen.txt
  [12K]

   pylint:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/861/artifact/out/diff-patch-pylint.txt
  [24K]

   shellcheck:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/861/artifact/out/diff-patch-shellcheck.txt
  [20K]

   shelldocs:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/861/artifact/out/diff-patch-shelldocs.txt
  [16K]

   whitespace:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/861/artifact/out/whitespace-eol.txt
  [9.4M]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/861/artifact/out/whitespace-tabs.txt
  [1.1M]

   xml:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/861/artifact/out/xml.txt
  [4.0K]

   findbugs:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/861/artifact/out/branch-findbugs-hadoop-hdds_client.txt
  [56K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/861/artifact/out/branch-findbugs-hadoop-hdds_container-service.txt
  [52K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/861/artifact/out/branch-findbugs-hadoop-hdds_framework.txt
  [12K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/861/artifact/out/branch-findbugs-hadoop-hdds_server-scm.txt
  [56K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/861/artifact/out/branch-findbugs-hadoop-hdds_tools.txt
  [16K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/861/artifact/out/branch-findbugs-hadoop-ozone_client.txt
  [4.0K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/861/artifact/out/branch-findbugs-hadoop-ozone_common.txt
  [28K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/861/artifact/out/branch-findbugs-hadoop-ozone_objectstore-service.txt
  [4.0K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/861/artifact/out/branch-findbugs-hadoop-ozone_ozone-manager.txt
  [4.0K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/861/artifact/out/branch-findbugs-hadoop-ozone_ozonefs.txt
  [8.0K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/861/artifact/out/branch-findbugs-hadoop-ozone_tools.txt
  [4.0K]

   javadoc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/861/artifact/out/diff-javadoc-javadoc-root.txt
  [760K]

   CTEST:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/861/artifact/out/patch-hadoop-hdfs-project_hadoop-hdfs-native-client-ctest.txt
  [116K]

   unit:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/861/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt
  [168K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/861/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
  [432K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/861/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-native-client.txt
  [112K]
   

Re: 回复：[VOTE] Merge Storage Policy Satisfier (SPS) [HDFS-10285] feature branch to trunk

[Vinayakumar B]

+1

Great work guys.
-Vinay

On Tue, 7 Aug 2018, 7:20 pm Anu Engineer,  wrote:

> +1, Sorry for the late vote. Thanks for the perseverance and seeing this
> thru.
>
> --Anu
>
>
> On 8/7/18, 1:04 AM, "郑锴(铁杰)"  wrote:
>
> +1 for the work to be in. Thanks Uma and folks for the hard taking!
>
> When it's in, I'd suggest we use a more general name for the new
> daemon service. It'd be good to evolve and support more self-running admin
> functionalities incubated first there before doing it directly in NameNode.
>
> Regards,
> Kai
> --
> 发件人：Uma Maheswara Rao G 
> 发送时间：2018年8月1日(星期三) 14:38
> 收件人：hdfs-dev 
> 主 题：[VOTE] Merge Storage Policy Satisfier (SPS) [HDFS-10285] feature
> branch to trunk
>
> Hi All,
>
>
>
>  From the positive responses from JIRA discussion and no objections
> from
> below DISCUSS thread [1], I am converting it to voting thread.
>
>
>
>  Last couple of weeks we spent time on testing the feature and so far
> it is
> working fine. Surendra uploaded a test report at HDFS-10285:  [2]
>
>
>
>  In this phase, we provide to run SPS outside of Namenode only and as a
> next phase we continue to discuss and work on to enable it as Internal
> SPS
> as explained below. We have got clean QA report on branch and if there
> are
> any static tool comments triggered later while running this thread, we
> will
> make sure to fix them before merge. We committed and continue to
> improve
> the code on trunk. Please refer to HDFS-10285 for discussion details.
>
>
>
>  This has been a long effort and we're grateful for the support we've
> received from the community. In particular, thanks to Andrew Wang,
> Anoop
> Sam John, Anu Engineer, Chris Douglas, Daryn Sharp, Du Jingcheng , Ewan
> Higgs, Jing Zhao, Kai Zheng,  Rakesh R, Ramkrishna , Surendra Singh
> Lilhore
> , Thomas Demoor, Uma Maheswara Rao G, Vinayakumar, Virajith,  Wei Zhou,
> Yuanbo Liu. Without these members effort, this feature might not have
> reached to this state.
>
>
>
> To start with, here is my +1
>
> It will end on 6th Aug.
>
>
>
> Regards,
>
> Uma
>
> [1]  https://s.apache.org/bhyu
> [2]  https://s.apache.org/AXvL
>
>
> On Wed, Jun 27, 2018 at 3:21 PM, Uma Maheswara Rao G <
> hadoop@gmail.com>
> wrote:
>
> > Hi All,
> >
> >   After long discussions(offline and on JIRA) on SPS, we came to a
> > conclusion on JIRA(HDFS-10285) that, we will go ahead with External
> SPS
> > merge in first phase. In this phase process will not be running
> inside
> > Namenode.
> >   We will continue discussion on Internal SPS. Current code base
> supports
> > both internal and external option. We have review comments for
> Internal
> > which needs some additional works for analysis and testing etc. We
> will
> > move Internal SPS work to under HDFS-12226 (Follow-on work for SPS
> in NN)
> > We are working on cleanup task HDFS-13076 for the merge. .
> > For more clarity on Internal and External SPS proposal thoughts,
> please
> > refer to JIRA HDFS-10285.
> >
> > If there are no objections with this, I will go ahead for voting
> soon.
> >
> > Regards,
> > Uma
> >
> > On Fri, Nov 17, 2017 at 3:16 PM, Uma Maheswara Rao G <
> hadoop@gmail.com
> > > wrote:
> >
> >> Update: We worked on the review comments and additional JIRAs above
> >> mentioned.
> >>
> >> >1. After the feedbacks from Andrew, Eddy, Xiao in JIRA reviews, we
> >> planned to take up the support for recursive API support.
> HDFS-12291<
> >> https://issues.apache.org/jira/browse/HDFS-12291>
> >>
> >> We provided the recursive API support now.
> >>
> >> >2. Xattr optimizations HDFS-12225 >> he.org/jira/browse/HDFS-12225>
> >> Improved this portion as well
> >>
> >> >3. Few other review comments already fixed and committed
> HDFS-12214<
> >> https://issues.apache.org/jira/browse/HDFS-12214>
> >> Fixed the comments.
> >>
> >> We are continuing to test the feature and working so far well. Also
> we
> >> uploaded a combined patch and got the good QA report.
> >>
> >> If there are no further objections, we would like to go for merge
> vote
> >> tomorrow. Please by default this feature will be disabled.
> >>
> >> Regards,
> >> Uma
> >>
> >> On Fri, Aug 18, 2017 at 11:27 PM, Gangumalla, Uma <
> >> uma.ganguma...@intel.com> wrote:
> >>
> >>> Hi Andrew,
> >>>
> >>> >Great to hear. It'd be nice to define which use cases are met by
> the
> >>> current version of SPS, and which will be handled after the merge.
> >>> After the discussions in JIRA, we planned to support recursive API
> as
> >>> well. The primary use cases we planned was for Hbase. Please check
> next

Re: 回复：[VOTE] Merge Storage Policy Satisfier (SPS) [HDFS-10285] feature branch to trunk

[Anu Engineer]

+1, Sorry for the late vote. Thanks for the perseverance and seeing this thru. 

--Anu


On 8/7/18, 1:04 AM, "郑锴(铁杰)"  wrote:

+1 for the work to be in. Thanks Uma and folks for the hard taking!

When it's in, I'd suggest we use a more general name for the new daemon 
service. It'd be good to evolve and support more self-running admin 
functionalities incubated first there before doing it directly in NameNode.

Regards,
Kai
--
发件人：Uma Maheswara Rao G 
发送时间：2018年8月1日(星期三) 14:38
收件人：hdfs-dev 
主　题：[VOTE] Merge Storage Policy Satisfier (SPS) [HDFS-10285] feature branch 
to trunk

Hi All,



 From the positive responses from JIRA discussion and no objections from
below DISCUSS thread [1], I am converting it to voting thread.



 Last couple of weeks we spent time on testing the feature and so far it is
working fine. Surendra uploaded a test report at HDFS-10285:  [2]



 In this phase, we provide to run SPS outside of Namenode only and as a
next phase we continue to discuss and work on to enable it as Internal SPS
as explained below. We have got clean QA report on branch and if there are
any static tool comments triggered later while running this thread, we will
make sure to fix them before merge. We committed and continue to improve
the code on trunk. Please refer to HDFS-10285 for discussion details.



 This has been a long effort and we're grateful for the support we've
received from the community. In particular, thanks to Andrew Wang, Anoop
Sam John, Anu Engineer, Chris Douglas, Daryn Sharp, Du Jingcheng , Ewan
Higgs, Jing Zhao, Kai Zheng,  Rakesh R, Ramkrishna , Surendra Singh Lilhore
, Thomas Demoor, Uma Maheswara Rao G, Vinayakumar, Virajith,  Wei Zhou,
Yuanbo Liu. Without these members effort, this feature might not have
reached to this state.



To start with, here is my +1

It will end on 6th Aug.



Regards,

Uma

[1]  https://s.apache.org/bhyu
[2]  https://s.apache.org/AXvL


On Wed, Jun 27, 2018 at 3:21 PM, Uma Maheswara Rao G 
wrote:

> Hi All,
>
>   After long discussions(offline and on JIRA) on SPS, we came to a
> conclusion on JIRA(HDFS-10285) that, we will go ahead with External SPS
> merge in first phase. In this phase process will not be running inside
> Namenode.
>   We will continue discussion on Internal SPS. Current code base supports
> both internal and external option. We have review comments for Internal
> which needs some additional works for analysis and testing etc. We will
> move Internal SPS work to under HDFS-12226 (Follow-on work for SPS in NN)
> We are working on cleanup task HDFS-13076 for the merge. .
> For more clarity on Internal and External SPS proposal thoughts, please
> refer to JIRA HDFS-10285.
>
> If there are no objections with this, I will go ahead for voting soon.
>
> Regards,
> Uma
>
> On Fri, Nov 17, 2017 at 3:16 PM, Uma Maheswara Rao G  > wrote:
>
>> Update: We worked on the review comments and additional JIRAs above
>> mentioned.
>>
>> >1. After the feedbacks from Andrew, Eddy, Xiao in JIRA reviews, we
>> planned to take up the support for recursive API support. HDFS-12291<
>> https://issues.apache.org/jira/browse/HDFS-12291>
>>
>> We provided the recursive API support now.
>>
>> >2. Xattr optimizations HDFS-12225> he.org/jira/browse/HDFS-12225>
>> Improved this portion as well
>>
>> >3. Few other review comments already fixed and committed HDFS-12214<
>> https://issues.apache.org/jira/browse/HDFS-12214>
>> Fixed the comments.
>>
>> We are continuing to test the feature and working so far well. Also we
>> uploaded a combined patch and got the good QA report.
>>
>> If there are no further objections, we would like to go for merge vote
>> tomorrow. Please by default this feature will be disabled.
>>
>> Regards,
>> Uma
>>
>> On Fri, Aug 18, 2017 at 11:27 PM, Gangumalla, Uma <
>> uma.ganguma...@intel.com> wrote:
>>
>>> Hi Andrew,
>>>
>>> >Great to hear. It'd be nice to define which use cases are met by the
>>> current version of SPS, and which will be handled after the merge.
>>> After the discussions in JIRA, we planned to support recursive API as
>>> well. The primary use cases we planned was for Hbase. Please check next
>>> point for use case details.
>>>
>>> >A bit more detail in the design doc on how HBase would use this feature
>>> would also be helpful. Is there an HBase JIRA already?
>>> Please find the usecase details at this comment in JIRA:
>>> 

Re: [VOTE] Release Apache Hadoop 3.1.1 - RC0

[Shashikant Banerjee]

+1(non-binding)

* checked out git tag release-3.1.1-RC0
* built from source 
* deployed on a single node cluster
* executed basic dfs commands
* executed some snapshot commands

Thank you very much the work Wangda.

Thanks
Shashi

On 8/7/18, 6:43 PM, "Elek, Marton"  wrote:


+1 (non-binding)

1. Built from the source package.
2. Checked the signature
3. Started docker based pseudo cluster and smoketested some basic 
functionality (hdfs cli, ec cli, viewfs, yarn examples, spark word count 
job)

Thank you very much the work Wangda.
Marton


On 08/02/2018 08:43 PM, Wangda Tan wrote:
> Hi folks,
> 
> I've created RC0 for Apache Hadoop 3.1.1. The artifacts are available 
here:
> 
> http://people.apache.org/~wangda/hadoop-3.1.1-RC0/
> 
> The RC tag in git is release-3.1.1-RC0:
> https://github.com/apache/hadoop/commits/release-3.1.1-RC0
> 
> The maven artifacts are available via repository.apache.org at
> https://repository.apache.org/content/repositories/orgapachehadoop-1139/
> 
> You can find my public key at
> http://svn.apache.org/repos/asf/hadoop/common/dist/KEYS
> 
> This vote will run 5 days from now.
> 
> 3.1.1 contains 435 [1] fixed JIRA issues since 3.1.0.
> 
> I have done testing with a pseudo cluster and distributed shell job. My +1
> to start.
> 
> Best,
> Wangda Tan
> 
> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.1)
> ORDER BY priority DESC
> 

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org




-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.1.1 - RC0

[Elek, Marton]

+1 (non-binding)

1. Built from the source package.
2. Checked the signature
3. Started docker based pseudo cluster and smoketested some basic 
functionality (hdfs cli, ec cli, viewfs, yarn examples, spark word count 
job)


Thank you very much the work Wangda.
Marton


On 08/02/2018 08:43 PM, Wangda Tan wrote:

Hi folks,

I've created RC0 for Apache Hadoop 3.1.1. The artifacts are available here:

http://people.apache.org/~wangda/hadoop-3.1.1-RC0/

The RC tag in git is release-3.1.1-RC0:
https://github.com/apache/hadoop/commits/release-3.1.1-RC0

The maven artifacts are available via repository.apache.org at
https://repository.apache.org/content/repositories/orgapachehadoop-1139/

You can find my public key at
http://svn.apache.org/repos/asf/hadoop/common/dist/KEYS

This vote will run 5 days from now.

3.1.1 contains 435 [1] fixed JIRA issues since 3.1.0.

I have done testing with a pseudo cluster and distributed shell job. My +1
to start.

Best,
Wangda Tan

[1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.1)
ORDER BY priority DESC



-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.1.1 - RC0

[Zsolt Venczel]

Thanks for your efforts Wangda!

+1 (non-binding)

* checked out git tag release-3.1.1-RC0
* built from source on Ubuntu 16.04, java 8u161-oracle
* deployed on a 4 node cluster
* executed snapshot commands: create, diff, checked file presence in
.snapshot directory
* execute map-reduce examples: teragen, terasort, teravalidate
* checked out web UI: file browser, snapshots
* executed unit tests for hadoop-hdfs-project/hadoop-hdfs module that have
all passed

Best regards,
Zsolt

On Tue, Aug 7, 2018 at 12:28 PM Sunil G  wrote:

> Thanks Wangda for the initiative.
> +1 for this RC.
> I have tested this RC built from source file.
>
>- Also ran few MR apps and verified both new YARN UI and old RM UI.
>- Tested Application priority and timeout
>- Inter Queue and Intra Queue Preemption cases were also verified
>- Tested basic placement constraints with DS.
>- Tested NodeLabel scenarios.
>- Tested new YARN UI with ATS v2
>
> - Sunil
>
> On Fri, Aug 3, 2018 at 12:14 AM Wangda Tan  wrote:
>
> > Hi folks,
> >
> > I've created RC0 for Apache Hadoop 3.1.1. The artifacts are available
> here:
> >
> > http://people.apache.org/~wangda/hadoop-3.1.1-RC0/
> >
> > The RC tag in git is release-3.1.1-RC0:
> > https://github.com/apache/hadoop/commits/release-3.1.1-RC0
> >
> > The maven artifacts are available via repository.apache.org at
> > https://repository.apache.org/content/repositories/orgapachehadoop-1139/
> >
> > You can find my public key at
> > http://svn.apache.org/repos/asf/hadoop/common/dist/KEYS
> >
> > This vote will run 5 days from now.
> >
> > 3.1.1 contains 435 [1] fixed JIRA issues since 3.1.0.
> >
> > I have done testing with a pseudo cluster and distributed shell job. My
> +1
> > to start.
> >
> > Best,
> > Wangda Tan
> >
> > [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.1)
> > ORDER BY priority DESC
> >
>

Re: [VOTE] Release Apache Hadoop 3.1.1 - RC0

[Sunil G]

Thanks Wangda for the initiative.
+1 for this RC.
I have tested this RC built from source file.

   - Also ran few MR apps and verified both new YARN UI and old RM UI.
   - Tested Application priority and timeout
   - Inter Queue and Intra Queue Preemption cases were also verified
   - Tested basic placement constraints with DS.
   - Tested NodeLabel scenarios.
   - Tested new YARN UI with ATS v2

- Sunil

On Fri, Aug 3, 2018 at 12:14 AM Wangda Tan  wrote:

> Hi folks,
>
> I've created RC0 for Apache Hadoop 3.1.1. The artifacts are available here:
>
> http://people.apache.org/~wangda/hadoop-3.1.1-RC0/
>
> The RC tag in git is release-3.1.1-RC0:
> https://github.com/apache/hadoop/commits/release-3.1.1-RC0
>
> The maven artifacts are available via repository.apache.org at
> https://repository.apache.org/content/repositories/orgapachehadoop-1139/
>
> You can find my public key at
> http://svn.apache.org/repos/asf/hadoop/common/dist/KEYS
>
> This vote will run 5 days from now.
>
> 3.1.1 contains 435 [1] fixed JIRA issues since 3.1.0.
>
> I have done testing with a pseudo cluster and distributed shell job. My +1
> to start.
>
> Best,
> Wangda Tan
>
> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.1)
> ORDER BY priority DESC
>

Re: [VOTE] Release Apache Hadoop 3.1.1 - RC0

[Rohith Sharma K S]

+1 (binding)

- Built from source and deployed 2 node RM HA non-secure cluster
- ATSv2 enabled for profile hbase.profile=2.0. Enabled both ats 1.5 and 2.0.
- Ran sample MR and DS applications
- Verified for RM HA, Work preserving restart, NM work preserving restart.
- Verified jobs for application priority and application timeout.
- Accessed UI2 for all web pages. Verified for UI2+ATSv2 integration web
pages.

-Rohith Sharma K S

On Fri, 3 Aug 2018 at 00:14, Wangda Tan  wrote:

> Hi folks,
>
> I've created RC0 for Apache Hadoop 3.1.1. The artifacts are available here:
>
> http://people.apache.org/~wangda/hadoop-3.1.1-RC0/
>
> The RC tag in git is release-3.1.1-RC0:
> https://github.com/apache/hadoop/commits/release-3.1.1-RC0
>
> The maven artifacts are available via repository.apache.org at
> https://repository.apache.org/content/repositories/orgapachehadoop-1139/
>
> You can find my public key at
> http://svn.apache.org/repos/asf/hadoop/common/dist/KEYS
>
> This vote will run 5 days from now.
>
> 3.1.1 contains 435 [1] fixed JIRA issues since 3.1.0.
>
> I have done testing with a pseudo cluster and distributed shell job. My +1
> to start.
>
> Best,
> Wangda Tan
>
> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.1)
> ORDER BY priority DESC
>

回复：[VOTE] Merge Storage Policy Satisfier (SPS) [HDFS-10285] feature branch to trunk

[郑锴(铁杰)]

+1 for the work to be in. Thanks Uma and folks for the hard taking!

When it's in, I'd suggest we use a more general name for the new daemon 
service. It'd be good to evolve and support more self-running admin 
functionalities incubated first there before doing it directly in NameNode.

Regards,
Kai
--
发件人：Uma Maheswara Rao G 
发送时间：2018年8月1日(星期三) 14:38
收件人：hdfs-dev 
主　题：[VOTE] Merge Storage Policy Satisfier (SPS) [HDFS-10285] feature branch to 
trunk

Hi All,



 From the positive responses from JIRA discussion and no objections from
below DISCUSS thread [1], I am converting it to voting thread.



 Last couple of weeks we spent time on testing the feature and so far it is
working fine. Surendra uploaded a test report at HDFS-10285:  [2]



 In this phase, we provide to run SPS outside of Namenode only and as a
next phase we continue to discuss and work on to enable it as Internal SPS
as explained below. We have got clean QA report on branch and if there are
any static tool comments triggered later while running this thread, we will
make sure to fix them before merge. We committed and continue to improve
the code on trunk. Please refer to HDFS-10285 for discussion details.



 This has been a long effort and we're grateful for the support we've
received from the community. In particular, thanks to Andrew Wang, Anoop
Sam John, Anu Engineer, Chris Douglas, Daryn Sharp, Du Jingcheng , Ewan
Higgs, Jing Zhao, Kai Zheng,  Rakesh R, Ramkrishna , Surendra Singh Lilhore
, Thomas Demoor, Uma Maheswara Rao G, Vinayakumar, Virajith,  Wei Zhou,
Yuanbo Liu. Without these members effort, this feature might not have
reached to this state.



To start with, here is my +1

It will end on 6th Aug.



Regards,

Uma

[1]  https://s.apache.org/bhyu
[2]  https://s.apache.org/AXvL


On Wed, Jun 27, 2018 at 3:21 PM, Uma Maheswara Rao G 
wrote:

> Hi All,
>
>   After long discussions(offline and on JIRA) on SPS, we came to a
> conclusion on JIRA(HDFS-10285) that, we will go ahead with External SPS
> merge in first phase. In this phase process will not be running inside
> Namenode.
>   We will continue discussion on Internal SPS. Current code base supports
> both internal and external option. We have review comments for Internal
> which needs some additional works for analysis and testing etc. We will
> move Internal SPS work to under HDFS-12226 (Follow-on work for SPS in NN)
> We are working on cleanup task HDFS-13076 for the merge. .
> For more clarity on Internal and External SPS proposal thoughts, please
> refer to JIRA HDFS-10285.
>
> If there are no objections with this, I will go ahead for voting soon.
>
> Regards,
> Uma
>
> On Fri, Nov 17, 2017 at 3:16 PM, Uma Maheswara Rao G  > wrote:
>
>> Update: We worked on the review comments and additional JIRAs above
>> mentioned.
>>
>> >1. After the feedbacks from Andrew, Eddy, Xiao in JIRA reviews, we
>> planned to take up the support for recursive API support. HDFS-12291<
>> https://issues.apache.org/jira/browse/HDFS-12291>
>>
>> We provided the recursive API support now.
>>
>> >2. Xattr optimizations HDFS-12225> he.org/jira/browse/HDFS-12225>
>> Improved this portion as well
>>
>> >3. Few other review comments already fixed and committed HDFS-12214<
>> https://issues.apache.org/jira/browse/HDFS-12214>
>> Fixed the comments.
>>
>> We are continuing to test the feature and working so far well. Also we
>> uploaded a combined patch and got the good QA report.
>>
>> If there are no further objections, we would like to go for merge vote
>> tomorrow. Please by default this feature will be disabled.
>>
>> Regards,
>> Uma
>>
>> On Fri, Aug 18, 2017 at 11:27 PM, Gangumalla, Uma <
>> uma.ganguma...@intel.com> wrote:
>>
>>> Hi Andrew,
>>>
>>> >Great to hear. It'd be nice to define which use cases are met by the
>>> current version of SPS, and which will be handled after the merge.
>>> After the discussions in JIRA, we planned to support recursive API as
>>> well. The primary use cases we planned was for Hbase. Please check next
>>> point for use case details.
>>>
>>> >A bit more detail in the design doc on how HBase would use this feature
>>> would also be helpful. Is there an HBase JIRA already?
>>> Please find the usecase details at this comment in JIRA:
>>> https://issues.apache.org/jira/browse/HDFS-10285?focusedComm
>>> entId=16120227=com.atlassian.jira.plugin.system.issueta
>>> bpanels:comment-tabpanel#comment-16120227
>>>
>>> >I also spent some more time with the design doc and posted a few
>>> questions on the JIRA.
>>> Thank you for the reviews.
>>>
>>> To summarize the discussions in JIRA:
>>> 1. After the feedbacks from Andrew, Eddy, Xiao in JIRA reviews, we
>>> planned to take up the support for recursive API support. HDFS-12291<
>>> https://issues.apache.org/jira/browse/HDFS-12291> (Rakesh started the
>>> work on it)
>>> 2. Xattr optimizations HDFS-12225>> 

[jira] [Resolved] (HDFS-13074) Ozone File System

[Elek, Marton (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-13074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Elek, Marton resolved HDFS-13074.
-
Resolution: Fixed

Resolving as all of the sub jiras are resolved.

> Ozone File System
> -
>
> Key: HDFS-13074
> URL: https://issues.apache.org/jira/browse/HDFS-13074
> Project: Hadoop HDFS
>  Issue Type: New Feature
>  Components: hdfs
>Reporter: Jitendra Nath Pandey
>Assignee: Jitendra Nath Pandey
>Priority: Major
>
> This jira splits out the Ozone's Key Value namespace out of HDFS-7240, 
> leaving that jira to focus on the block layer Hadoop Distributed Storage 
> Layer (HDSL).  HDFS-10419 focuses on the traditional hierarchical 
> namespace/NN on top of HDFS while this jira focuses on a flat Key-Value 
> namespace call ed Ozone FS on top HDSL. 
> [~owen.omalley] suggested the split in HDFS-7240 in this comment.
> Ozone provides two APIs:
>  * A KV API
>  * A Hadoop compatible FS (Haddop FileSystem and Hadoop FileContext), on top 
> of KV API.
> Ozone FS serves the following purpose
>  * It helps test the new storage layer (HDSL)
>  * It can be directly used by applications (such. Hive, Spark)  that are 
> ready for cloud's native storage systems that use a KV namespace such as S3, 
> Azure ADLS.
> Ozone's namespace server scales well because of following reasons:
>  * It keeps only the working set of metadata in memory - this scales the 
> Key-Value namespace.
>  * Ozone does not need to keep the equivalent of the block-map; instead the 
> corresponding container-to-location mapping resides in in the SCM - this 
> leaves additional free space for caching the metadata. This benefit is also 
> available to NN that adapts to the new block-storage container layer as 
> explained in Evolving NN using new block container layer attached in 
> HDFS-10419.
>  * It does not have a single global lock - this will scale well against a 
> large number of concurrent clients/rpcs. 
>  * It can be partitioned/sharded much more easily because it is a flat 
> namespace. Indeed a natural partitioning level is the bucket. Partitioning 
> would further scale the number of clients/rpcs and also the Key-value 
> namespace.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

RE: [VOTE] Merge Storage Policy Satisfier (SPS) [HDFS-10285] feature branch to trunk

[Brahma Reddy Battula]

+1, Great work Guys. 




Brahma Reddy Battula
 

-Original Message-
From: Uma Maheswara Rao G [mailto:hadoop@gmail.com] 
Sent: 01 August 2018 12:08
To: hdfs-dev@hadoop.apache.org
Subject: [VOTE] Merge Storage Policy Satisfier (SPS) [HDFS-10285] feature 
branch to trunk

Hi All,



 From the positive responses from JIRA discussion and no objections from below 
DISCUSS thread [1], I am converting it to voting thread.



 Last couple of weeks we spent time on testing the feature and so far it is 
working fine. Surendra uploaded a test report at HDFS-10285:  [2]



 In this phase, we provide to run SPS outside of Namenode only and as a next 
phase we continue to discuss and work on to enable it as Internal SPS as 
explained below. We have got clean QA report on branch and if there are any 
static tool comments triggered later while running this thread, we will make 
sure to fix them before merge. We committed and continue to improve the code on 
trunk. Please refer to HDFS-10285 for discussion details.



 This has been a long effort and we're grateful for the support we've received 
from the community. In particular, thanks to Andrew Wang, Anoop Sam John, Anu 
Engineer, Chris Douglas, Daryn Sharp, Du Jingcheng , Ewan Higgs, Jing Zhao, Kai 
Zheng,  Rakesh R, Ramkrishna , Surendra Singh Lilhore , Thomas Demoor, Uma 
Maheswara Rao G, Vinayakumar, Virajith,  Wei Zhou, Yuanbo Liu. Without these 
members effort, this feature might not have reached to this state.



To start with, here is my +1

It will end on 6th Aug.



Regards,

Uma

[1]  https://s.apache.org/bhyu
[2]  https://s.apache.org/AXvL


On Wed, Jun 27, 2018 at 3:21 PM, Uma Maheswara Rao G 
wrote:

> Hi All,
>
>   After long discussions(offline and on JIRA) on SPS, we came to a 
> conclusion on JIRA(HDFS-10285) that, we will go ahead with External 
> SPS merge in first phase. In this phase process will not be running 
> inside Namenode.
>   We will continue discussion on Internal SPS. Current code base 
> supports both internal and external option. We have review comments 
> for Internal which needs some additional works for analysis and 
> testing etc. We will move Internal SPS work to under HDFS-12226 
> (Follow-on work for SPS in NN) We are working on cleanup task HDFS-13076 for 
> the merge. .
> For more clarity on Internal and External SPS proposal thoughts, 
> please refer to JIRA HDFS-10285.
>
> If there are no objections with this, I will go ahead for voting soon.
>
> Regards,
> Uma
>
> On Fri, Nov 17, 2017 at 3:16 PM, Uma Maheswara Rao G 
>  > wrote:
>
>> Update: We worked on the review comments and additional JIRAs above 
>> mentioned.
>>
>> >1. After the feedbacks from Andrew, Eddy, Xiao in JIRA reviews, we
>> planned to take up the support for recursive API support. HDFS-12291< 
>> https://issues.apache.org/jira/browse/HDFS-12291>
>>
>> We provided the recursive API support now.
>>
>> >2. Xattr optimizations HDFS-12225> he.org/jira/browse/HDFS-12225>
>> Improved this portion as well
>>
>> >3. Few other review comments already fixed and committed HDFS-12214<
>> https://issues.apache.org/jira/browse/HDFS-12214>
>> Fixed the comments.
>>
>> We are continuing to test the feature and working so far well. Also 
>> we uploaded a combined patch and got the good QA report.
>>
>> If there are no further objections, we would like to go for merge 
>> vote tomorrow. Please by default this feature will be disabled.
>>
>> Regards,
>> Uma
>>
>> On Fri, Aug 18, 2017 at 11:27 PM, Gangumalla, Uma < 
>> uma.ganguma...@intel.com> wrote:
>>
>>> Hi Andrew,
>>>
>>> >Great to hear. It'd be nice to define which use cases are met by 
>>> >the
>>> current version of SPS, and which will be handled after the merge.
>>> After the discussions in JIRA, we planned to support recursive API 
>>> as well. The primary use cases we planned was for Hbase. Please 
>>> check next point for use case details.
>>>
>>> >A bit more detail in the design doc on how HBase would use this 
>>> >feature
>>> would also be helpful. Is there an HBase JIRA already?
>>> Please find the usecase details at this comment in JIRA:
>>> https://issues.apache.org/jira/browse/HDFS-10285?focusedComm
>>> entId=16120227=com.atlassian.jira.plugin.system.issueta
>>> bpanels:comment-tabpanel#comment-16120227
>>>
>>> >I also spent some more time with the design doc and posted a few
>>> questions on the JIRA.
>>> Thank you for the reviews.
>>>
>>> To summarize the discussions in JIRA:
>>> 1. After the feedbacks from Andrew, Eddy, Xiao in JIRA reviews, we 
>>> planned to take up the support for recursive API support. 
>>> HDFS-12291< https://issues.apache.org/jira/browse/HDFS-12291> 
>>> (Rakesh started the work on it) 2. Xattr optimizations 
>>> HDFS-12225 (Patch 
>>> available) 3. Few other review comments already fixed and committed 
>>> HDFS-12214< https://issues.apache.org/jira/browse/HDFS-12214>
>>>
>>> For tracking the 

RE: [VOTE] Merge Storage Policy Satisfier (SPS) [HDFS-10285] feature branch to trunk

[surendra lilhore]

+1, looking forward to seeing SPS in coming releases.


Regards,
Surendra

-Original Message-
From: Uma Maheswara Rao G [mailto:hadoop@gmail.com] 
Sent: 01 August 2018 14:38
To: hdfs-dev@hadoop.apache.org
Subject: [VOTE] Merge Storage Policy Satisfier (SPS) [HDFS-10285] feature 
branch to trunk

Hi All,



 From the positive responses from JIRA discussion and no objections from below 
DISCUSS thread [1], I am converting it to voting thread.



 Last couple of weeks we spent time on testing the feature and so far it is 
working fine. Surendra uploaded a test report at HDFS-10285:  [2]



 In this phase, we provide to run SPS outside of Namenode only and as a next 
phase we continue to discuss and work on to enable it as Internal SPS as 
explained below. We have got clean QA report on branch and if there are any 
static tool comments triggered later while running this thread, we will make 
sure to fix them before merge. We committed and continue to improve the code on 
trunk. Please refer to HDFS-10285 for discussion details.



 This has been a long effort and we're grateful for the support we've received 
from the community. In particular, thanks to Andrew Wang, Anoop Sam John, Anu 
Engineer, Chris Douglas, Daryn Sharp, Du Jingcheng , Ewan Higgs, Jing Zhao, Kai 
Zheng,  Rakesh R, Ramkrishna , Surendra Singh Lilhore , Thomas Demoor, Uma 
Maheswara Rao G, Vinayakumar, Virajith,  Wei Zhou, Yuanbo Liu. Without these 
members effort, this feature might not have reached to this state.



To start with, here is my +1

It will end on 6th Aug.



Regards,

Uma

[1]  https://s.apache.org/bhyu
[2]  https://s.apache.org/AXvL


On Wed, Jun 27, 2018 at 3:21 PM, Uma Maheswara Rao G 
wrote:

> Hi All,
>
>   After long discussions(offline and on JIRA) on SPS, we came to a 
> conclusion on JIRA(HDFS-10285) that, we will go ahead with External 
> SPS merge in first phase. In this phase process will not be running 
> inside Namenode.
>   We will continue discussion on Internal SPS. Current code base 
> supports both internal and external option. We have review comments 
> for Internal which needs some additional works for analysis and 
> testing etc. We will move Internal SPS work to under HDFS-12226 
> (Follow-on work for SPS in NN) We are working on cleanup task HDFS-13076 for 
> the merge. .
> For more clarity on Internal and External SPS proposal thoughts, 
> please refer to JIRA HDFS-10285.
>
> If there are no objections with this, I will go ahead for voting soon.
>
> Regards,
> Uma
>
> On Fri, Nov 17, 2017 at 3:16 PM, Uma Maheswara Rao G 
>  > wrote:
>
>> Update: We worked on the review comments and additional JIRAs above 
>> mentioned.
>>
>> >1. After the feedbacks from Andrew, Eddy, Xiao in JIRA reviews, we
>> planned to take up the support for recursive API support. HDFS-12291< 
>> https://issues.apache.org/jira/browse/HDFS-12291>
>>
>> We provided the recursive API support now.
>>
>> >2. Xattr optimizations HDFS-12225> he.org/jira/browse/HDFS-12225>
>> Improved this portion as well
>>
>> >3. Few other review comments already fixed and committed HDFS-12214<
>> https://issues.apache.org/jira/browse/HDFS-12214>
>> Fixed the comments.
>>
>> We are continuing to test the feature and working so far well. Also 
>> we uploaded a combined patch and got the good QA report.
>>
>> If there are no further objections, we would like to go for merge 
>> vote tomorrow. Please by default this feature will be disabled.
>>
>> Regards,
>> Uma
>>
>> On Fri, Aug 18, 2017 at 11:27 PM, Gangumalla, Uma < 
>> uma.ganguma...@intel.com> wrote:
>>
>>> Hi Andrew,
>>>
>>> >Great to hear. It'd be nice to define which use cases are met by 
>>> >the
>>> current version of SPS, and which will be handled after the merge.
>>> After the discussions in JIRA, we planned to support recursive API 
>>> as well. The primary use cases we planned was for Hbase. Please 
>>> check next point for use case details.
>>>
>>> >A bit more detail in the design doc on how HBase would use this 
>>> >feature
>>> would also be helpful. Is there an HBase JIRA already?
>>> Please find the usecase details at this comment in JIRA:
>>> https://issues.apache.org/jira/browse/HDFS-10285?focusedComm
>>> entId=16120227=com.atlassian.jira.plugin.system.issueta
>>> bpanels:comment-tabpanel#comment-16120227
>>>
>>> >I also spent some more time with the design doc and posted a few
>>> questions on the JIRA.
>>> Thank you for the reviews.
>>>
>>> To summarize the discussions in JIRA:
>>> 1. After the feedbacks from Andrew, Eddy, Xiao in JIRA reviews, we 
>>> planned to take up the support for recursive API support. 
>>> HDFS-12291< https://issues.apache.org/jira/browse/HDFS-12291> 
>>> (Rakesh started the work on it) 2. Xattr optimizations 
>>> HDFS-12225 (Patch 
>>> available) 3. Few other review comments already fixed and committed 
>>> HDFS-12214< https://issues.apache.org/jira/browse/HDFS-12214>
>>>