Re: About 2.7.4 Release

[Konstantin Shvachko]

Hey guys,

Looks like we are done with blockers for Apache Hadoop 2.7.4 release.
https://issues.apache.org/jira/issues/?filter=12340814

I just committed HDFS-11896
<https://issues.apache.org/jira/browse/HDFS-11896>, and decided not to wait
for HDFS-11576 <https://issues.apache.org/jira/browse/HDFS-11576>, see Jira
comment.
Thanks Vinod for pointing out HDFS-11742
<https://issues.apache.org/jira/browse/HDFS-11742>. It was thoroughly
tested on a small cluster and Kihwal committed it last week, thanks.

Will start building initial RC. Please refrain from committing to
branch-2.7 for some time.

Thank you everybody for contributing.
--Konst

On Thu, Jul 20, 2017 at 12:07 PM, Vinod Kumar Vavilapalli <
vino...@apache.org> wrote:

> Thanks for taking 2.7.4 over Konstantin!
>
> Regarding rolling RC next week, I still see that there are 4 blocker /
> critical tickets targeted for 2.7.4: https://issues.apache.
> org/jira/issues/?jql=project%20in%20(HDFS%2C%20MAPREDUCE%
> 2C%20HADOOP%2C%20YARN)%20AND%20priority%20in%20(Blocker%2C%
> 20Critical)%20AND%20resolution%20%3D%20Unresolved%20AND%20%
> 22Target%20Version%2Fs%22%20%3D%202.7.4
> <https://issues.apache.org/jira/issues/?jql=project%20in%20(HDFS,%20MAPREDUCE,%20HADOOP,%20YARN)%20AND%20priority%20in%20(Blocker,%20Critical)%20AND%20resolution%20=%20Unresolved%20AND%20%22Target%20Version/s%22%20=%202.7.4>
> .
>
> We should get closure on them. https://issues.apache.
> org/jira/browse/HDFS-11742 definitely was something that was deemed a
> blocker for 2.8.2, not sure about 2.7.4.
>
> I’m ‘back’ - let me know if you need any help.
>
> Thanks
> +Vinod
>
> On Jul 13, 2017, at 5:45 PM, Konstantin Shvachko <shv.had...@gmail.com>
> wrote:
>
> Hi everybody.
>
> We have been doing some internal testing of Hadoop 2.7.4. The testing is
> going well.
> Did not find any major issues on our workloads.
> Used an internal tool called Dynamometer to check NameNode performance on
> real cluster traces. Good.
> Overall test cluster performance looks good.
> Some more testing is still going on.
>
> I plan to build an RC next week. If there are no objection.
>
> Thanks,
> --Konst
>
> On Thu, Jun 15, 2017 at 4:42 PM, Konstantin Shvachko <shv.had...@gmail.com
> >
> wrote:
>
> Hey guys.
>
> An update on 2.7.4 progress.
> We are down to 4 blockers. There is some work remaining on those.
> https://issues.apache.org/jira/browse/HDFS-11896?filter=12340814
> Would be good if people could follow up on review comments.
>
> I looked through nightly Jenkins build results for 2.7.4 both on Apache
> Jenkins and internal.
> Some test fail intermittently, but there no consistent failures. I filed
> HDFS-11985 to track some of them.
> https://issues.apache.org/jira/browse/HDFS-11985
> I do not currently consider these failures as blockers. LMK if some of
> them are.
>
> We started internal testing of branch-2.7 on one of our smallish (100+
> nodes) test clusters.
> Will update on the results.
>
> There is a plan to enable BigTop for 2.7.4 testing.
>
> Akira, Brahma thank you for setting up a wiki page for 2.7.4 release.
> Thank you everybody for contributing to this effort.
>
> Regards,
> --Konstantin
>
>
> On Tue, May 30, 2017 at 12:08 AM, Akira Ajisaka <aajis...@apache.org>
> wrote:
>
> Sure.
> If you want to edit the wiki, please tell me your ASF confluence account.
>
> -Akira
>
> On 2017/05/30 15:31, Rohith Sharma K S wrote:
>
> Couple of more JIRAs need to be back ported for 2.7.4 release. These will
> solve RM HA unstability issues.
> https://issues.apache.org/jira/browse/YARN-5333
> https://issues.apache.org/jira/browse/YARN-5988
> https://issues.apache.org/jira/browse/YARN-6304
>
> I will raise a JIRAs to back port it.
>
> @Akira , could  you help to add these JIRAs into wiki?
>
> Thanks & Regards
> Rohith Sharma K S
>
> On 29 May 2017 at 12:19, Akira Ajisaka <aajis...@apache.org> wrote:
>
> Created a page for 2.7.4 release.
>
> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.7.4
>
> If you want to edit this wiki, please ping me.
>
> Regards,
> Akira
>
>
> On 2017/05/23 4:42, Brahma Reddy Battula wrote:
>
> Hi Konstantin Shvachko
>
>
>
> how about creating a wiki page for 2.7.4 release status like 2.8 and
> trunk in following link.??
>
>
> https://cwiki.apache.org/confluence/display/HADOOP
>
>
> 
> From: Konstantin Shvachko <shv.had...@gmail.com>
> Sent: Saturday, May 13, 2017 3:58 AM
> To: Akira Ajisaka
> Cc: Hadoop Common; Hdfs-dev; mapreduce-...@hadoop.apache.org;
> yarn-dev@hadoop.apache.org
> Subject: Re: About

Re: LinkedIn Dynamometer Tool (was About 2.7.4 Release)

[Erik Krogen]

Hi Sanjay,

Actually I was not aware of that work… This seems to be a better way of 
achieving some of the same things we do externally to the DN process. I will 
look into reimplementing some parts on top of this; seems it should just 
require some very small extensions to DataNodeCluster. Thank you very much for 
the pointer!

Erik

On 7/21/17, 11:01 AM, "sanjay Radia" <sanjayo...@gmail.com> wrote:

Erik
  Great stuff. 
BTW did you build on top of the “simulated data nodes” in HDFS which has a 
way to storing only the length of data (but not real data)? That work allowed 
supplementing  with a matching editsLog for the NN. Your approach of using a 
real image has the advantage of being able to replay traces from audit logs.
(Ref 
https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DataNodeCluster.java)

thanks

sanjay
> On Jul 20, 2017, at 10:42 AM, Erik Krogen <ekro...@linkedin.com.INVALID> 
wrote:
> 
> forking off of the 2.7.4 release thread to answer this question about
> Dynamometer
> 
> Dynamometer is a tool developed at LinkedIn for scale testing HDFS,
> specifically the NameNode. We have been using it for some time now and 
have
> recently been making some enhancements to ease of use and reproducibility.
> We hope to post a blog post sometime in the not-too-distant future, and
> also to open source it. I can provide some details here given that we have
> been leveraging it as part of our 2.7.4 release / upgrade process (in
> addition to previous upgrades).
> 
> The basic idea is to get full-scale black-box testing of the HDFS NN while
> using significantly less (~10%) hardware than a real cluster of that size
> would require. We use real NN images from our at-scale clusters paired 
with
> some logic to fake out DNs into thinking they are storing data when they
> are not, allowing us to stuff more DNs onto each machine. Since we use a
> real image, we can replay real traces (collected from audit logs) to
> compare actual production performance vs. performance on this simulated
> cluster (with additional tuning, different version, etc.). We leverage 
YARN
> to manage setting up this cluster and to replay the traces.
> 
> Happy to answer questions.
> 
> Erik
> 
> On Wed, Jul 19, 2017 at 5:05 PM, Konstantin Shvachko 
<shv.had...@gmail.com>
> wrote:
> 
>> Hi Tianyi,
>> 
>> Glad you are interested in Dynamometer. Erik (CC-ed) is actively working
>> on this project right now, I'll let him elaborate.
>> Erik, you should probably respond on Apache dev list, as I think it could
>> be interesting for other people as well, asince we planned to open source
>> it. You can fork the "About 2.7.4 Release" thread with a new subject and
>> give some details about Dynamometer there.
>> 
>> Thanks,
>> --Konstantin
>> 
>> On Wed, Jul 19, 2017 at 1:40 AM, 何天一 <hetia...@bytedance.com> wrote:
>> 
>>> Hi, Shavachko.
>>> 
>>> You mentioned an internal tool called Dynamometer to test NameNode
>>> performance earlier in the 2.7.4 release thread.
>>> I wonder if you could share some ideas behind the tool. Or is there a
>>> plan to bring Dynamometer to open source community?
>>> 
>>> Thanks.
>>> 
>>> BR,
>>> Tianyi
>>> 
>>> On Fri, Jul 14, 2017 at 8:45 AM Konstantin Shvachko 
<shv.had...@gmail.com>
>>> wrote:
>>> 
>>>> Hi everybody.
>>>> 
>>>> We have been doing some internal testing of Hadoop 2.7.4. The testing 
is
>>>> going well.
>>>> Did not find any major issues on our workloads.
>>>> Used an internal tool called Dynamometer to check NameNode performance 
on
>>>> real cluster traces. Good.
>>>> Overall test cluster performance looks good.
>>>> Some more testing is still going on.
>>>> 
>>>> I plan to build an RC next week. If there are no objection.
>>>> 
>>>> Thanks,
>>>> --Konst
>>>> 
>>>> On Thu, Jun 15, 2017 at 4:42 PM, Konstantin Shvachko <
>>>> shv.had...@gmail.com>
>>>> wrote:
>>>> 
>>>>> Hey guys.
>>>>> 
>>>>> An update on 2.7.4 progress.
>>>>> We are down

Re: LinkedIn Dynamometer Tool (was About 2.7.4 Release)

[sanjay Radia]

Erik
  Great stuff. 
BTW did you build on top of the “simulated data nodes” in HDFS which has a way 
to storing only the length of data (but not real data)? That work allowed 
supplementing  with a matching editsLog for the NN. Your approach of using a 
real image has the advantage of being able to replay traces from audit logs.
(Ref 
https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DataNodeCluster.java)

thanks

sanjay
> On Jul 20, 2017, at 10:42 AM, Erik Krogen <ekro...@linkedin.com.INVALID> 
> wrote:
> 
> forking off of the 2.7.4 release thread to answer this question about
> Dynamometer
> 
> Dynamometer is a tool developed at LinkedIn for scale testing HDFS,
> specifically the NameNode. We have been using it for some time now and have
> recently been making some enhancements to ease of use and reproducibility.
> We hope to post a blog post sometime in the not-too-distant future, and
> also to open source it. I can provide some details here given that we have
> been leveraging it as part of our 2.7.4 release / upgrade process (in
> addition to previous upgrades).
> 
> The basic idea is to get full-scale black-box testing of the HDFS NN while
> using significantly less (~10%) hardware than a real cluster of that size
> would require. We use real NN images from our at-scale clusters paired with
> some logic to fake out DNs into thinking they are storing data when they
> are not, allowing us to stuff more DNs onto each machine. Since we use a
> real image, we can replay real traces (collected from audit logs) to
> compare actual production performance vs. performance on this simulated
> cluster (with additional tuning, different version, etc.). We leverage YARN
> to manage setting up this cluster and to replay the traces.
> 
> Happy to answer questions.
> 
> Erik
> 
> On Wed, Jul 19, 2017 at 5:05 PM, Konstantin Shvachko <shv.had...@gmail.com>
> wrote:
> 
>> Hi Tianyi,
>> 
>> Glad you are interested in Dynamometer. Erik (CC-ed) is actively working
>> on this project right now, I'll let him elaborate.
>> Erik, you should probably respond on Apache dev list, as I think it could
>> be interesting for other people as well, asince we planned to open source
>> it. You can fork the "About 2.7.4 Release" thread with a new subject and
>> give some details about Dynamometer there.
>> 
>> Thanks,
>> --Konstantin
>> 
>> On Wed, Jul 19, 2017 at 1:40 AM, 何天一 <hetia...@bytedance.com> wrote:
>> 
>>> Hi, Shavachko.
>>> 
>>> You mentioned an internal tool called Dynamometer to test NameNode
>>> performance earlier in the 2.7.4 release thread.
>>> I wonder if you could share some ideas behind the tool. Or is there a
>>> plan to bring Dynamometer to open source community?
>>> 
>>> Thanks.
>>> 
>>> BR,
>>> Tianyi
>>> 
>>> On Fri, Jul 14, 2017 at 8:45 AM Konstantin Shvachko <shv.had...@gmail.com>
>>> wrote:
>>> 
>>>> Hi everybody.
>>>> 
>>>> We have been doing some internal testing of Hadoop 2.7.4. The testing is
>>>> going well.
>>>> Did not find any major issues on our workloads.
>>>> Used an internal tool called Dynamometer to check NameNode performance on
>>>> real cluster traces. Good.
>>>> Overall test cluster performance looks good.
>>>> Some more testing is still going on.
>>>> 
>>>> I plan to build an RC next week. If there are no objection.
>>>> 
>>>> Thanks,
>>>> --Konst
>>>> 
>>>> On Thu, Jun 15, 2017 at 4:42 PM, Konstantin Shvachko <
>>>> shv.had...@gmail.com>
>>>> wrote:
>>>> 
>>>>> Hey guys.
>>>>> 
>>>>> An update on 2.7.4 progress.
>>>>> We are down to 4 blockers. There is some work remaining on those.
>>>>> https://issues.apache.org/jira/browse/HDFS-11896?filter=12340814
>>>>> Would be good if people could follow up on review comments.
>>>>> 
>>>>> I looked through nightly Jenkins build results for 2.7.4 both on Apache
>>>>> Jenkins and internal.
>>>>> Some test fail intermittently, but there no consistent failures. I
>>>> filed
>>>>> HDFS-11985 to track some of them.
>>>>> https://issues.apache.org/jira/browse/HDFS-11985
>>>>> I do not currently consider these failures as blockers. LMK if some of
>>>>> them are.
>>>>> 
>>&g

Re: About 2.7.4 Release

[Vinod Kumar Vavilapalli]

Thanks for taking 2.7.4 over Konstantin!

Regarding rolling RC next week, I still see that there are 4 blocker / critical 
tickets targeted for 2.7.4: 
https://issues.apache.org/jira/issues/?jql=project%20in%20(HDFS%2C%20MAPREDUCE%2C%20HADOOP%2C%20YARN)%20AND%20priority%20in%20(Blocker%2C%20Critical)%20AND%20resolution%20%3D%20Unresolved%20AND%20%22Target%20Version%2Fs%22%20%3D%202.7.4
 
<https://issues.apache.org/jira/issues/?jql=project%20in%20(HDFS,%20MAPREDUCE,%20HADOOP,%20YARN)%20AND%20priority%20in%20(Blocker,%20Critical)%20AND%20resolution%20=%20Unresolved%20AND%20%22Target%20Version/s%22%20=%202.7.4>.

We should get closure on them. https://issues.apache.org/jira/browse/HDFS-11742 
<https://issues.apache.org/jira/browse/HDFS-11742> definitely was something 
that was deemed a blocker for 2.8.2, not sure about 2.7.4.

I’m ‘back’ - let me know if you need any help.

Thanks
+Vinod

> On Jul 13, 2017, at 5:45 PM, Konstantin Shvachko <shv.had...@gmail.com> wrote:
> 
> Hi everybody.
> 
> We have been doing some internal testing of Hadoop 2.7.4. The testing is
> going well.
> Did not find any major issues on our workloads.
> Used an internal tool called Dynamometer to check NameNode performance on
> real cluster traces. Good.
> Overall test cluster performance looks good.
> Some more testing is still going on.
> 
> I plan to build an RC next week. If there are no objection.
> 
> Thanks,
> --Konst
> 
> On Thu, Jun 15, 2017 at 4:42 PM, Konstantin Shvachko <shv.had...@gmail.com>
> wrote:
> 
>> Hey guys.
>> 
>> An update on 2.7.4 progress.
>> We are down to 4 blockers. There is some work remaining on those.
>> https://issues.apache.org/jira/browse/HDFS-11896?filter=12340814
>> Would be good if people could follow up on review comments.
>> 
>> I looked through nightly Jenkins build results for 2.7.4 both on Apache
>> Jenkins and internal.
>> Some test fail intermittently, but there no consistent failures. I filed
>> HDFS-11985 to track some of them.
>> https://issues.apache.org/jira/browse/HDFS-11985
>> I do not currently consider these failures as blockers. LMK if some of
>> them are.
>> 
>> We started internal testing of branch-2.7 on one of our smallish (100+
>> nodes) test clusters.
>> Will update on the results.
>> 
>> There is a plan to enable BigTop for 2.7.4 testing.
>> 
>> Akira, Brahma thank you for setting up a wiki page for 2.7.4 release.
>> Thank you everybody for contributing to this effort.
>> 
>> Regards,
>> --Konstantin
>> 
>> 
>> On Tue, May 30, 2017 at 12:08 AM, Akira Ajisaka <aajis...@apache.org>
>> wrote:
>> 
>>> Sure.
>>> If you want to edit the wiki, please tell me your ASF confluence account.
>>> 
>>> -Akira
>>> 
>>> On 2017/05/30 15:31, Rohith Sharma K S wrote:
>>> 
>>>> Couple of more JIRAs need to be back ported for 2.7.4 release. These will
>>>> solve RM HA unstability issues.
>>>> https://issues.apache.org/jira/browse/YARN-5333
>>>> https://issues.apache.org/jira/browse/YARN-5988
>>>> https://issues.apache.org/jira/browse/YARN-6304
>>>> 
>>>> I will raise a JIRAs to back port it.
>>>> 
>>>> @Akira , could  you help to add these JIRAs into wiki?
>>>> 
>>>> Thanks & Regards
>>>> Rohith Sharma K S
>>>> 
>>>> On 29 May 2017 at 12:19, Akira Ajisaka <aajis...@apache.org> wrote:
>>>> 
>>>> Created a page for 2.7.4 release.
>>>>> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.7.4
>>>>> 
>>>>> If you want to edit this wiki, please ping me.
>>>>> 
>>>>> Regards,
>>>>> Akira
>>>>> 
>>>>> 
>>>>> On 2017/05/23 4:42, Brahma Reddy Battula wrote:
>>>>> 
>>>>> Hi Konstantin Shvachko
>>>>>> 
>>>>>> 
>>>>>> how about creating a wiki page for 2.7.4 release status like 2.8 and
>>>>>> trunk in following link.??
>>>>>> 
>>>>>> 
>>>>>> https://cwiki.apache.org/confluence/display/HADOOP
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> From: Konstantin Shvachko <shv.had...@gmail.com>
>>>>>> Sent: Saturday, May 13, 2017 3:58 AM
>>>>>> To: Akira Ajisaka
>>>>>> Cc: Hadoop Common; Hdfs-dev; mapreduce-...@hadoop.apache.org;
>>>&

Re: LinkedIn Dynamometer Tool (was About 2.7.4 Release)

[Anu Engineer]

Hi Erik,

Looking forward to the release of this tool. Thank you very much for the 
contribution.

Had a couple of questions about how the tool works.

1. Would you be able to provide the traces along with this tool? In other 
words, would I be able to use this out of the box, or do I have to build up 
traces myself? 

2. Could you explain how the “fake out DNs into thinking they are storing data” 
— works? Or I can be patient and read your blog post too.

Thanks
Anu






On 7/20/17, 10:42 AM, "Erik Krogen" <ekro...@linkedin.com.INVALID> wrote:

>forking off of the 2.7.4 release thread to answer this question about
>Dynamometer
>
>Dynamometer is a tool developed at LinkedIn for scale testing HDFS,
>specifically the NameNode. We have been using it for some time now and have
>recently been making some enhancements to ease of use and reproducibility.
>We hope to post a blog post sometime in the not-too-distant future, and
>also to open source it. I can provide some details here given that we have
>been leveraging it as part of our 2.7.4 release / upgrade process (in
>addition to previous upgrades).
>
>The basic idea is to get full-scale black-box testing of the HDFS NN while
>using significantly less (~10%) hardware than a real cluster of that size
>would require. We use real NN images from our at-scale clusters paired with
>some logic to fake out DNs into thinking they are storing data when they
>are not, allowing us to stuff more DNs onto each machine. Since we use a
>real image, we can replay real traces (collected from audit logs) to
>compare actual production performance vs. performance on this simulated
>cluster (with additional tuning, different version, etc.). We leverage YARN
>to manage setting up this cluster and to replay the traces.
>
>Happy to answer questions.
>
>Erik
>
>On Wed, Jul 19, 2017 at 5:05 PM, Konstantin Shvachko <shv.had...@gmail.com>
>wrote:
>
>> Hi Tianyi,
>>
>> Glad you are interested in Dynamometer. Erik (CC-ed) is actively working
>> on this project right now, I'll let him elaborate.
>> Erik, you should probably respond on Apache dev list, as I think it could
>> be interesting for other people as well, asince we planned to open source
>> it. You can fork the "About 2.7.4 Release" thread with a new subject and
>> give some details about Dynamometer there.
>>
>> Thanks,
>> --Konstantin
>>
>> On Wed, Jul 19, 2017 at 1:40 AM, 何天一 <hetia...@bytedance.com> wrote:
>>
>>> Hi, Shavachko.
>>>
>>> You mentioned an internal tool called Dynamometer to test NameNode
>>> performance earlier in the 2.7.4 release thread.
>>> I wonder if you could share some ideas behind the tool. Or is there a
>>> plan to bring Dynamometer to open source community?
>>>
>>> Thanks.
>>>
>>> BR,
>>> Tianyi
>>>
>>> On Fri, Jul 14, 2017 at 8:45 AM Konstantin Shvachko <shv.had...@gmail.com>
>>> wrote:
>>>
>>>> Hi everybody.
>>>>
>>>> We have been doing some internal testing of Hadoop 2.7.4. The testing is
>>>> going well.
>>>> Did not find any major issues on our workloads.
>>>> Used an internal tool called Dynamometer to check NameNode performance on
>>>> real cluster traces. Good.
>>>> Overall test cluster performance looks good.
>>>> Some more testing is still going on.
>>>>
>>>> I plan to build an RC next week. If there are no objection.
>>>>
>>>> Thanks,
>>>> --Konst
>>>>
>>>> On Thu, Jun 15, 2017 at 4:42 PM, Konstantin Shvachko <
>>>> shv.had...@gmail.com>
>>>> wrote:
>>>>
>>>> > Hey guys.
>>>> >
>>>> > An update on 2.7.4 progress.
>>>> > We are down to 4 blockers. There is some work remaining on those.
>>>> > https://issues.apache.org/jira/browse/HDFS-11896?filter=12340814
>>>> > Would be good if people could follow up on review comments.
>>>> >
>>>> > I looked through nightly Jenkins build results for 2.7.4 both on Apache
>>>> > Jenkins and internal.
>>>> > Some test fail intermittently, but there no consistent failures. I
>>>> filed
>>>> > HDFS-11985 to track some of them.
>>>> > https://issues.apache.org/jira/browse/HDFS-11985
>>>> > I do not currently consider these failures as blockers. LMK if some of
>>>> > them are.
>>>> >
>>>> > We started internal testing of branch-2.7 on o

Re: About 2.7.4 Release

[Konstantin Shvachko]

Hi everybody.

We have been doing some internal testing of Hadoop 2.7.4. The testing is
going well.
Did not find any major issues on our workloads.
Used an internal tool called Dynamometer to check NameNode performance on
real cluster traces. Good.
Overall test cluster performance looks good.
Some more testing is still going on.

I plan to build an RC next week. If there are no objection.

Thanks,
--Konst

On Thu, Jun 15, 2017 at 4:42 PM, Konstantin Shvachko <shv.had...@gmail.com>
wrote:

> Hey guys.
>
> An update on 2.7.4 progress.
> We are down to 4 blockers. There is some work remaining on those.
> https://issues.apache.org/jira/browse/HDFS-11896?filter=12340814
> Would be good if people could follow up on review comments.
>
> I looked through nightly Jenkins build results for 2.7.4 both on Apache
> Jenkins and internal.
> Some test fail intermittently, but there no consistent failures. I filed
> HDFS-11985 to track some of them.
> https://issues.apache.org/jira/browse/HDFS-11985
> I do not currently consider these failures as blockers. LMK if some of
> them are.
>
> We started internal testing of branch-2.7 on one of our smallish (100+
> nodes) test clusters.
> Will update on the results.
>
> There is a plan to enable BigTop for 2.7.4 testing.
>
> Akira, Brahma thank you for setting up a wiki page for 2.7.4 release.
> Thank you everybody for contributing to this effort.
>
> Regards,
> --Konstantin
>
>
> On Tue, May 30, 2017 at 12:08 AM, Akira Ajisaka <aajis...@apache.org>
> wrote:
>
>> Sure.
>> If you want to edit the wiki, please tell me your ASF confluence account.
>>
>> -Akira
>>
>> On 2017/05/30 15:31, Rohith Sharma K S wrote:
>>
>>> Couple of more JIRAs need to be back ported for 2.7.4 release. These will
>>> solve RM HA unstability issues.
>>> https://issues.apache.org/jira/browse/YARN-5333
>>> https://issues.apache.org/jira/browse/YARN-5988
>>> https://issues.apache.org/jira/browse/YARN-6304
>>>
>>> I will raise a JIRAs to back port it.
>>>
>>> @Akira , could  you help to add these JIRAs into wiki?
>>>
>>> Thanks & Regards
>>> Rohith Sharma K S
>>>
>>> On 29 May 2017 at 12:19, Akira Ajisaka <aajis...@apache.org> wrote:
>>>
>>> Created a page for 2.7.4 release.
>>>> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.7.4
>>>>
>>>> If you want to edit this wiki, please ping me.
>>>>
>>>> Regards,
>>>> Akira
>>>>
>>>>
>>>> On 2017/05/23 4:42, Brahma Reddy Battula wrote:
>>>>
>>>> Hi Konstantin Shvachko
>>>>>
>>>>>
>>>>> how about creating a wiki page for 2.7.4 release status like 2.8 and
>>>>> trunk in following link.??
>>>>>
>>>>>
>>>>> https://cwiki.apache.org/confluence/display/HADOOP
>>>>>
>>>>>
>>>>> 
>>>>> From: Konstantin Shvachko <shv.had...@gmail.com>
>>>>> Sent: Saturday, May 13, 2017 3:58 AM
>>>>> To: Akira Ajisaka
>>>>> Cc: Hadoop Common; Hdfs-dev; mapreduce-...@hadoop.apache.org;
>>>>> yarn-dev@hadoop.apache.org
>>>>> Subject: Re: About 2.7.4 Release
>>>>>
>>>>> Latest update on the links and filters. Here is the correct link for
>>>>> the
>>>>> filter:
>>>>> https://issues.apache.org/jira/secure/IssueNavigator.jspa?
>>>>> requestId=12340814
>>>>>
>>>>> Also updated: https://s.apache.org/Dzg4
>>>>>
>>>>> Had to do some Jira debugging. Sorry for confusion.
>>>>>
>>>>> Thanks,
>>>>> --Konstantin
>>>>>
>>>>> On Wed, May 10, 2017 at 2:30 PM, Konstantin Shvachko <
>>>>> shv.had...@gmail.com>
>>>>> wrote:
>>>>>
>>>>> Hey Akira,
>>>>>
>>>>>>
>>>>>> I didn't have private filters. Most probably Jira caches something.
>>>>>> Your filter is in the right direction, but for some reason it lists
>>>>>> only
>>>>>> 22 issues, while mine has 29.
>>>>>> It misses e.g. YARN-5543 <https://issues.apache.org/jir
>>>>>> a/browse/YARN-5543>
>>>>>> .
>>>>>>
>>>>>> Anyways, I created a Jira filter now "Hadoop 2.7.4 

Re: About 2.7.4 Release

[Konstantin Shvachko]

Hey guys.

An update on 2.7.4 progress.
We are down to 4 blockers. There is some work remaining on those.
https://issues.apache.org/jira/browse/HDFS-11896?filter=12340814
Would be good if people could follow up on review comments.

I looked through nightly Jenkins build results for 2.7.4 both on Apache
Jenkins and internal.
Some test fail intermittently, but there no consistent failures. I filed
HDFS-11985 to track some of them.
https://issues.apache.org/jira/browse/HDFS-11985
I do not currently consider these failures as blockers. LMK if some of them
are.

We started internal testing of branch-2.7 on one of our smallish (100+
nodes) test clusters.
Will update on the results.

There is a plan to enable BigTop for 2.7.4 testing.

Akira, Brahma thank you for setting up a wiki page for 2.7.4 release.
Thank you everybody for contributing to this effort.

Regards,
--Konstantin


On Tue, May 30, 2017 at 12:08 AM, Akira Ajisaka <aajis...@apache.org> wrote:

> Sure.
> If you want to edit the wiki, please tell me your ASF confluence account.
>
> -Akira
>
> On 2017/05/30 15:31, Rohith Sharma K S wrote:
>
>> Couple of more JIRAs need to be back ported for 2.7.4 release. These will
>> solve RM HA unstability issues.
>> https://issues.apache.org/jira/browse/YARN-5333
>> https://issues.apache.org/jira/browse/YARN-5988
>> https://issues.apache.org/jira/browse/YARN-6304
>>
>> I will raise a JIRAs to back port it.
>>
>> @Akira , could  you help to add these JIRAs into wiki?
>>
>> Thanks & Regards
>> Rohith Sharma K S
>>
>> On 29 May 2017 at 12:19, Akira Ajisaka <aajis...@apache.org> wrote:
>>
>> Created a page for 2.7.4 release.
>>> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.7.4
>>>
>>> If you want to edit this wiki, please ping me.
>>>
>>> Regards,
>>> Akira
>>>
>>>
>>> On 2017/05/23 4:42, Brahma Reddy Battula wrote:
>>>
>>> Hi Konstantin Shvachko
>>>>
>>>>
>>>> how about creating a wiki page for 2.7.4 release status like 2.8 and
>>>> trunk in following link.??
>>>>
>>>>
>>>> https://cwiki.apache.org/confluence/display/HADOOP
>>>>
>>>>
>>>> 
>>>> From: Konstantin Shvachko <shv.had...@gmail.com>
>>>> Sent: Saturday, May 13, 2017 3:58 AM
>>>> To: Akira Ajisaka
>>>> Cc: Hadoop Common; Hdfs-dev; mapreduce-...@hadoop.apache.org;
>>>> yarn-dev@hadoop.apache.org
>>>> Subject: Re: About 2.7.4 Release
>>>>
>>>> Latest update on the links and filters. Here is the correct link for the
>>>> filter:
>>>> https://issues.apache.org/jira/secure/IssueNavigator.jspa?
>>>> requestId=12340814
>>>>
>>>> Also updated: https://s.apache.org/Dzg4
>>>>
>>>> Had to do some Jira debugging. Sorry for confusion.
>>>>
>>>> Thanks,
>>>> --Konstantin
>>>>
>>>> On Wed, May 10, 2017 at 2:30 PM, Konstantin Shvachko <
>>>> shv.had...@gmail.com>
>>>> wrote:
>>>>
>>>> Hey Akira,
>>>>
>>>>>
>>>>> I didn't have private filters. Most probably Jira caches something.
>>>>> Your filter is in the right direction, but for some reason it lists
>>>>> only
>>>>> 22 issues, while mine has 29.
>>>>> It misses e.g. YARN-5543 <https://issues.apache.org/jir
>>>>> a/browse/YARN-5543>
>>>>> .
>>>>>
>>>>> Anyways, I created a Jira filter now "Hadoop 2.7.4 release blockers",
>>>>> shared it with "everybody", and updated my link to point to that
>>>>> filter.
>>>>> So
>>>>> you can use any of the three methods below to get the correct list:
>>>>> 1. Go to https://s.apache.org/Dzg4
>>>>> 2. Go to the filter via
>>>>> https://issues.apache.org/jira/issues?filter=12340814
>>>>>or by finding "Hadoop 2.7.4 release blockers" filter in the jira
>>>>> 3. On Advanced issues search page paste this:
>>>>> project in (HDFS, HADOOP, YARN, MAPREDUCE) AND labels = release-blocker
>>>>> AND "Target Version/s" = 2.7.4
>>>>>
>>>>> Hope this solves the confusion for which issues are included.
>>>>> Please LMK if it doesn't, as it is important.
>>>>>
>>>>> T

Re: About 2.7.4 Release

[Akira Ajisaka]

Sure.
If you want to edit the wiki, please tell me your ASF confluence account.

-Akira

On 2017/05/30 15:31, Rohith Sharma K S wrote:

Couple of more JIRAs need to be back ported for 2.7.4 release. These will
solve RM HA unstability issues.
https://issues.apache.org/jira/browse/YARN-5333
https://issues.apache.org/jira/browse/YARN-5988
https://issues.apache.org/jira/browse/YARN-6304

I will raise a JIRAs to back port it.

@Akira , could  you help to add these JIRAs into wiki?

Thanks & Regards
Rohith Sharma K S

On 29 May 2017 at 12:19, Akira Ajisaka <aajis...@apache.org> wrote:


Created a page for 2.7.4 release.
https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.7.4

If you want to edit this wiki, please ping me.

Regards,
Akira


On 2017/05/23 4:42, Brahma Reddy Battula wrote:


Hi Konstantin Shvachko


how about creating a wiki page for 2.7.4 release status like 2.8 and
trunk in following link.??


https://cwiki.apache.org/confluence/display/HADOOP



From: Konstantin Shvachko <shv.had...@gmail.com>
Sent: Saturday, May 13, 2017 3:58 AM
To: Akira Ajisaka
Cc: Hadoop Common; Hdfs-dev; mapreduce-...@hadoop.apache.org;
yarn-dev@hadoop.apache.org
Subject: Re: About 2.7.4 Release

Latest update on the links and filters. Here is the correct link for the
filter:
https://issues.apache.org/jira/secure/IssueNavigator.jspa?
requestId=12340814

Also updated: https://s.apache.org/Dzg4

Had to do some Jira debugging. Sorry for confusion.

Thanks,
--Konstantin

On Wed, May 10, 2017 at 2:30 PM, Konstantin Shvachko <
shv.had...@gmail.com>
wrote:

Hey Akira,


I didn't have private filters. Most probably Jira caches something.
Your filter is in the right direction, but for some reason it lists only
22 issues, while mine has 29.
It misses e.g. YARN-5543 <https://issues.apache.org/jir
a/browse/YARN-5543>
.

Anyways, I created a Jira filter now "Hadoop 2.7.4 release blockers",
shared it with "everybody", and updated my link to point to that filter.
So
you can use any of the three methods below to get the correct list:
1. Go to https://s.apache.org/Dzg4
2. Go to the filter via
https://issues.apache.org/jira/issues?filter=12340814
   or by finding "Hadoop 2.7.4 release blockers" filter in the jira
3. On Advanced issues search page paste this:
project in (HDFS, HADOOP, YARN, MAPREDUCE) AND labels = release-blocker
AND "Target Version/s" = 2.7.4

Hope this solves the confusion for which issues are included.
Please LMK if it doesn't, as it is important.

Thanks,
--Konstantin

On Tue, May 9, 2017 at 9:58 AM, Akira Ajisaka <aajis...@apache.org>
wrote:

Hi Konstantin,


Thank you for volunteering as release manager!

Actually the original link works fine: https://s.apache.org/Dzg4



I couldn't see the link. Maybe is it private filter?

Here is a link I generated: https://s.apache.org/ehKy
This filter includes resolved issue and excludes fixversion == 2.7.4

Thanks and Regards,
Akira

On 2017/05/08 19:20, Konstantin Shvachko wrote:

Hi Brahma Reddy Battula,


Actually the original link works fine: https://s.apache.org/Dzg4
Your link excludes closed and resolved issues, which needs backporting,
and
which we cannot reopen, as discussed in this thread earlier.

Looked through the issues you proposed:

HDFS-9311 <https://issues.apache.org/jira/browse/HDFS-9311>
Seems like a new feature. It helps failover to standby node when
primary
is
under heavy load, but it introduces new APIs, addresses, config
parameters.
And needs at least one follow up jira.
Looks like a backward compatible change, though.
Did you have a chance to run it in production?

+1 on
HDFS-10987 <https://issues.apache.org/jira/browse/HDFS-10987>


[HDFS-10987] Make Decommission less expensive when lot of ...<

https://issues.apache.org/jira/browse/HDFS-10987>
issues.apache.org
When user want to decommission a node which having 50M blocks ,it could
hold the namesystem lock for long time.We've seen it is taking 36 sec. As
we knew during this ...



HDFS-9902 <https://issues.apache.org/jira/browse/HDFS-9902>



[HDFS-9902] Support different values of dfs.datanode.du ...<

https://issues.apache.org/jira/browse/HDFS-9902>
issues.apache.org
Now Hadoop support different storage type for DISK, SSD, ARCHIVE and
RAM_DISK, but they share one configuration dfs.datanode.du.reserved. The
DISK size may be several ...



HDFS-8312 <https://issues.apache.org/jira/browse/HDFS-8312>



Trash does not descent into child directories to check for ...<

https://issues.apache.org/jira/browse/HDFS-8312>
issues.apache.org
HDFS trash does not descent into child directory to check if user has
permission to delete files. For example: Run the following command to
initialize directory ...



HADOOP-14100 <https://issues.apache.org/jira/browse/HADOOP-14100>



Upgrade Jsch jar to latest version to fix vulnerability in ...<

https://is

Re: About 2.7.4 Release

[Rohith Sharma K S]

Couple of more JIRAs need to be back ported for 2.7.4 release. These will
solve RM HA unstability issues.
https://issues.apache.org/jira/browse/YARN-5333
https://issues.apache.org/jira/browse/YARN-5988
https://issues.apache.org/jira/browse/YARN-6304

I will raise a JIRAs to back port it.

@Akira , could  you help to add these JIRAs into wiki?

Thanks & Regards
Rohith Sharma K S

On 29 May 2017 at 12:19, Akira Ajisaka <aajis...@apache.org> wrote:

> Created a page for 2.7.4 release.
> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.7.4
>
> If you want to edit this wiki, please ping me.
>
> Regards,
> Akira
>
>
> On 2017/05/23 4:42, Brahma Reddy Battula wrote:
>
>> Hi Konstantin Shvachko
>>
>>
>> how about creating a wiki page for 2.7.4 release status like 2.8 and
>> trunk in following link.??
>>
>>
>> https://cwiki.apache.org/confluence/display/HADOOP
>>
>>
>> 
>> From: Konstantin Shvachko <shv.had...@gmail.com>
>> Sent: Saturday, May 13, 2017 3:58 AM
>> To: Akira Ajisaka
>> Cc: Hadoop Common; Hdfs-dev; mapreduce-...@hadoop.apache.org;
>> yarn-dev@hadoop.apache.org
>> Subject: Re: About 2.7.4 Release
>>
>> Latest update on the links and filters. Here is the correct link for the
>> filter:
>> https://issues.apache.org/jira/secure/IssueNavigator.jspa?
>> requestId=12340814
>>
>> Also updated: https://s.apache.org/Dzg4
>>
>> Had to do some Jira debugging. Sorry for confusion.
>>
>> Thanks,
>> --Konstantin
>>
>> On Wed, May 10, 2017 at 2:30 PM, Konstantin Shvachko <
>> shv.had...@gmail.com>
>> wrote:
>>
>> Hey Akira,
>>>
>>> I didn't have private filters. Most probably Jira caches something.
>>> Your filter is in the right direction, but for some reason it lists only
>>> 22 issues, while mine has 29.
>>> It misses e.g. YARN-5543 <https://issues.apache.org/jir
>>> a/browse/YARN-5543>
>>> .
>>>
>>> Anyways, I created a Jira filter now "Hadoop 2.7.4 release blockers",
>>> shared it with "everybody", and updated my link to point to that filter.
>>> So
>>> you can use any of the three methods below to get the correct list:
>>> 1. Go to https://s.apache.org/Dzg4
>>> 2. Go to the filter via
>>> https://issues.apache.org/jira/issues?filter=12340814
>>>or by finding "Hadoop 2.7.4 release blockers" filter in the jira
>>> 3. On Advanced issues search page paste this:
>>> project in (HDFS, HADOOP, YARN, MAPREDUCE) AND labels = release-blocker
>>> AND "Target Version/s" = 2.7.4
>>>
>>> Hope this solves the confusion for which issues are included.
>>> Please LMK if it doesn't, as it is important.
>>>
>>> Thanks,
>>> --Konstantin
>>>
>>> On Tue, May 9, 2017 at 9:58 AM, Akira Ajisaka <aajis...@apache.org>
>>> wrote:
>>>
>>> Hi Konstantin,
>>>>
>>>> Thank you for volunteering as release manager!
>>>>
>>>> Actually the original link works fine: https://s.apache.org/Dzg4
>>>>>
>>>> I couldn't see the link. Maybe is it private filter?
>>>>
>>>> Here is a link I generated: https://s.apache.org/ehKy
>>>> This filter includes resolved issue and excludes fixversion == 2.7.4
>>>>
>>>> Thanks and Regards,
>>>> Akira
>>>>
>>>> On 2017/05/08 19:20, Konstantin Shvachko wrote:
>>>>
>>>> Hi Brahma Reddy Battula,
>>>>>
>>>>> Actually the original link works fine: https://s.apache.org/Dzg4
>>>>> Your link excludes closed and resolved issues, which needs backporting,
>>>>> and
>>>>> which we cannot reopen, as discussed in this thread earlier.
>>>>>
>>>>> Looked through the issues you proposed:
>>>>>
>>>>> HDFS-9311 <https://issues.apache.org/jira/browse/HDFS-9311>
>>>>> Seems like a new feature. It helps failover to standby node when
>>>>> primary
>>>>> is
>>>>> under heavy load, but it introduces new APIs, addresses, config
>>>>> parameters.
>>>>> And needs at least one follow up jira.
>>>>> Looks like a backward compatible change, though.
>>>>> Did you have a chance to run it in production?
>>>>>
>>>>> +1 on
>>>>> HDFS

Re: About 2.7.4 Release

[Akira Ajisaka]

Created a page for 2.7.4 release.
https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.7.4

If you want to edit this wiki, please ping me.

Regards,
Akira

On 2017/05/23 4:42, Brahma Reddy Battula wrote:

Hi Konstantin Shvachko


how about creating a wiki page for 2.7.4 release status like 2.8 and trunk in 
following link.??


https://cwiki.apache.org/confluence/display/HADOOP



From: Konstantin Shvachko <shv.had...@gmail.com>
Sent: Saturday, May 13, 2017 3:58 AM
To: Akira Ajisaka
Cc: Hadoop Common; Hdfs-dev; mapreduce-...@hadoop.apache.org; 
yarn-dev@hadoop.apache.org
Subject: Re: About 2.7.4 Release

Latest update on the links and filters. Here is the correct link for the
filter:
https://issues.apache.org/jira/secure/IssueNavigator.jspa?requestId=12340814

Also updated: https://s.apache.org/Dzg4

Had to do some Jira debugging. Sorry for confusion.

Thanks,
--Konstantin

On Wed, May 10, 2017 at 2:30 PM, Konstantin Shvachko <shv.had...@gmail.com>
wrote:


Hey Akira,

I didn't have private filters. Most probably Jira caches something.
Your filter is in the right direction, but for some reason it lists only
22 issues, while mine has 29.
It misses e.g. YARN-5543 <https://issues.apache.org/jira/browse/YARN-5543>
.

Anyways, I created a Jira filter now "Hadoop 2.7.4 release blockers",
shared it with "everybody", and updated my link to point to that filter. So
you can use any of the three methods below to get the correct list:
1. Go to https://s.apache.org/Dzg4
2. Go to the filter via
https://issues.apache.org/jira/issues?filter=12340814
   or by finding "Hadoop 2.7.4 release blockers" filter in the jira
3. On Advanced issues search page paste this:
project in (HDFS, HADOOP, YARN, MAPREDUCE) AND labels = release-blocker
AND "Target Version/s" = 2.7.4

Hope this solves the confusion for which issues are included.
Please LMK if it doesn't, as it is important.

Thanks,
--Konstantin

On Tue, May 9, 2017 at 9:58 AM, Akira Ajisaka <aajis...@apache.org> wrote:


Hi Konstantin,

Thank you for volunteering as release manager!


Actually the original link works fine: https://s.apache.org/Dzg4

I couldn't see the link. Maybe is it private filter?

Here is a link I generated: https://s.apache.org/ehKy
This filter includes resolved issue and excludes fixversion == 2.7.4

Thanks and Regards,
Akira

On 2017/05/08 19:20, Konstantin Shvachko wrote:


Hi Brahma Reddy Battula,

Actually the original link works fine: https://s.apache.org/Dzg4
Your link excludes closed and resolved issues, which needs backporting,
and
which we cannot reopen, as discussed in this thread earlier.

Looked through the issues you proposed:

HDFS-9311 <https://issues.apache.org/jira/browse/HDFS-9311>
Seems like a new feature. It helps failover to standby node when primary
is
under heavy load, but it introduces new APIs, addresses, config
parameters.
And needs at least one follow up jira.
Looks like a backward compatible change, though.
Did you have a chance to run it in production?

+1 on
HDFS-10987 <https://issues.apache.org/jira/browse/HDFS-10987>

[HDFS-10987] Make Decommission less expensive when lot of 
...<https://issues.apache.org/jira/browse/HDFS-10987>
issues.apache.org
When user want to decommission a node which having 50M blocks ,it could hold 
the namesystem lock for long time.We've seen it is taking 36 sec. As we knew 
during this ...




HDFS-9902 <https://issues.apache.org/jira/browse/HDFS-9902>

[HDFS-9902] Support different values of dfs.datanode.du 
...<https://issues.apache.org/jira/browse/HDFS-9902>
issues.apache.org
Now Hadoop support different storage type for DISK, SSD, ARCHIVE and RAM_DISK, 
but they share one configuration dfs.datanode.du.reserved. The DISK size may be 
several ...




HDFS-8312 <https://issues.apache.org/jira/browse/HDFS-8312>

Trash does not descent into child directories to check for 
...<https://issues.apache.org/jira/browse/HDFS-8312>
issues.apache.org
HDFS trash does not descent into child directory to check if user has 
permission to delete files. For example: Run the following command to 
initialize directory ...




HADOOP-14100 <https://issues.apache.org/jira/browse/HADOOP-14100>

Upgrade Jsch jar to latest version to fix vulnerability in 
...<https://issues.apache.org/jira/browse/HADOOP-14100>
issues.apache.org
Recently there was on vulnerability reported on jsch library. Its fixed in 
latest 0.1.54 version before CVE was made public. 
https://cve.mitre.org/cgi-bin/cvename.cgi ...





Added them to 2.7.4 release. You should see them via the above link now.
Would be good if you could attach backport patches for some of them?

Appreciate your help,
--Konstantin

On Mon, May 8, 2017 at 8:39 AM, Brahma Reddy Battula <
brahmareddy.batt...@huawei.com> wrote:



Looks following link is not correct..

https://s.apache.org/Dzg4

It should be 

Re: About 2.7.4 Release

[Brahma Reddy Battula]

Hi Konstantin Shvachko


how about creating a wiki page for 2.7.4 release status like 2.8 and trunk in 
following link.??


https://cwiki.apache.org/confluence/display/HADOOP



From: Konstantin Shvachko <shv.had...@gmail.com>
Sent: Saturday, May 13, 2017 3:58 AM
To: Akira Ajisaka
Cc: Hadoop Common; Hdfs-dev; mapreduce-...@hadoop.apache.org; 
yarn-dev@hadoop.apache.org
Subject: Re: About 2.7.4 Release

Latest update on the links and filters. Here is the correct link for the
filter:
https://issues.apache.org/jira/secure/IssueNavigator.jspa?requestId=12340814

Also updated: https://s.apache.org/Dzg4

Had to do some Jira debugging. Sorry for confusion.

Thanks,
--Konstantin

On Wed, May 10, 2017 at 2:30 PM, Konstantin Shvachko <shv.had...@gmail.com>
wrote:

> Hey Akira,
>
> I didn't have private filters. Most probably Jira caches something.
> Your filter is in the right direction, but for some reason it lists only
> 22 issues, while mine has 29.
> It misses e.g. YARN-5543 <https://issues.apache.org/jira/browse/YARN-5543>
> .
>
> Anyways, I created a Jira filter now "Hadoop 2.7.4 release blockers",
> shared it with "everybody", and updated my link to point to that filter. So
> you can use any of the three methods below to get the correct list:
> 1. Go to https://s.apache.org/Dzg4
> 2. Go to the filter via
> https://issues.apache.org/jira/issues?filter=12340814
>or by finding "Hadoop 2.7.4 release blockers" filter in the jira
> 3. On Advanced issues search page paste this:
> project in (HDFS, HADOOP, YARN, MAPREDUCE) AND labels = release-blocker
> AND "Target Version/s" = 2.7.4
>
> Hope this solves the confusion for which issues are included.
> Please LMK if it doesn't, as it is important.
>
> Thanks,
> --Konstantin
>
> On Tue, May 9, 2017 at 9:58 AM, Akira Ajisaka <aajis...@apache.org> wrote:
>
>> Hi Konstantin,
>>
>> Thank you for volunteering as release manager!
>>
>> > Actually the original link works fine: https://s.apache.org/Dzg4
>> I couldn't see the link. Maybe is it private filter?
>>
>> Here is a link I generated: https://s.apache.org/ehKy
>> This filter includes resolved issue and excludes fixversion == 2.7.4
>>
>> Thanks and Regards,
>> Akira
>>
>> On 2017/05/08 19:20, Konstantin Shvachko wrote:
>>
>>> Hi Brahma Reddy Battula,
>>>
>>> Actually the original link works fine: https://s.apache.org/Dzg4
>>> Your link excludes closed and resolved issues, which needs backporting,
>>> and
>>> which we cannot reopen, as discussed in this thread earlier.
>>>
>>> Looked through the issues you proposed:
>>>
>>> HDFS-9311 <https://issues.apache.org/jira/browse/HDFS-9311>
>>> Seems like a new feature. It helps failover to standby node when primary
>>> is
>>> under heavy load, but it introduces new APIs, addresses, config
>>> parameters.
>>> And needs at least one follow up jira.
>>> Looks like a backward compatible change, though.
>>> Did you have a chance to run it in production?
>>>
>>> +1 on
>>> HDFS-10987 <https://issues.apache.org/jira/browse/HDFS-10987>
[HDFS-10987] Make Decommission less expensive when lot of 
...<https://issues.apache.org/jira/browse/HDFS-10987>
issues.apache.org
When user want to decommission a node which having 50M blocks ,it could hold 
the namesystem lock for long time.We've seen it is taking 36 sec. As we knew 
during this ...



>>> HDFS-9902 <https://issues.apache.org/jira/browse/HDFS-9902>
[HDFS-9902] Support different values of dfs.datanode.du 
...<https://issues.apache.org/jira/browse/HDFS-9902>
issues.apache.org
Now Hadoop support different storage type for DISK, SSD, ARCHIVE and RAM_DISK, 
but they share one configuration dfs.datanode.du.reserved. The DISK size may be 
several ...



>>> HDFS-8312 <https://issues.apache.org/jira/browse/HDFS-8312>
Trash does not descent into child directories to check for 
...<https://issues.apache.org/jira/browse/HDFS-8312>
issues.apache.org
HDFS trash does not descent into child directory to check if user has 
permission to delete files. For example: Run the following command to 
initialize directory ...



>>> HADOOP-14100 <https://issues.apache.org/jira/browse/HADOOP-14100>
Upgrade Jsch jar to latest version to fix vulnerability in 
...<https://issues.apache.org/jira/browse/HADOOP-14100>
issues.apache.org
Recently there was on vulnerability reported on jsch library. Its fixed in 
latest 0.1.54 version before CVE was made public. 
https://cve.mitre.org/cgi-bin/cvename.cgi ...



>>>
>>> Ad

Re: About 2.7.4 Release

[Konstantin Shvachko]

Latest update on the links and filters. Here is the correct link for the
filter:
https://issues.apache.org/jira/secure/IssueNavigator.jspa?requestId=12340814

Also updated: https://s.apache.org/Dzg4

Had to do some Jira debugging. Sorry for confusion.

Thanks,
--Konstantin

On Wed, May 10, 2017 at 2:30 PM, Konstantin Shvachko <shv.had...@gmail.com>
wrote:

> Hey Akira,
>
> I didn't have private filters. Most probably Jira caches something.
> Your filter is in the right direction, but for some reason it lists only
> 22 issues, while mine has 29.
> It misses e.g. YARN-5543 <https://issues.apache.org/jira/browse/YARN-5543>
> .
>
> Anyways, I created a Jira filter now "Hadoop 2.7.4 release blockers",
> shared it with "everybody", and updated my link to point to that filter. So
> you can use any of the three methods below to get the correct list:
> 1. Go to https://s.apache.org/Dzg4
> 2. Go to the filter via
> https://issues.apache.org/jira/issues?filter=12340814
>or by finding "Hadoop 2.7.4 release blockers" filter in the jira
> 3. On Advanced issues search page paste this:
> project in (HDFS, HADOOP, YARN, MAPREDUCE) AND labels = release-blocker
> AND "Target Version/s" = 2.7.4
>
> Hope this solves the confusion for which issues are included.
> Please LMK if it doesn't, as it is important.
>
> Thanks,
> --Konstantin
>
> On Tue, May 9, 2017 at 9:58 AM, Akira Ajisaka <aajis...@apache.org> wrote:
>
>> Hi Konstantin,
>>
>> Thank you for volunteering as release manager!
>>
>> > Actually the original link works fine: https://s.apache.org/Dzg4
>> I couldn't see the link. Maybe is it private filter?
>>
>> Here is a link I generated: https://s.apache.org/ehKy
>> This filter includes resolved issue and excludes fixversion == 2.7.4
>>
>> Thanks and Regards,
>> Akira
>>
>> On 2017/05/08 19:20, Konstantin Shvachko wrote:
>>
>>> Hi Brahma Reddy Battula,
>>>
>>> Actually the original link works fine: https://s.apache.org/Dzg4
>>> Your link excludes closed and resolved issues, which needs backporting,
>>> and
>>> which we cannot reopen, as discussed in this thread earlier.
>>>
>>> Looked through the issues you proposed:
>>>
>>> HDFS-9311 <https://issues.apache.org/jira/browse/HDFS-9311>
>>> Seems like a new feature. It helps failover to standby node when primary
>>> is
>>> under heavy load, but it introduces new APIs, addresses, config
>>> parameters.
>>> And needs at least one follow up jira.
>>> Looks like a backward compatible change, though.
>>> Did you have a chance to run it in production?
>>>
>>> +1 on
>>> HDFS-10987 <https://issues.apache.org/jira/browse/HDFS-10987>
>>> HDFS-9902 <https://issues.apache.org/jira/browse/HDFS-9902>
>>> HDFS-8312 <https://issues.apache.org/jira/browse/HDFS-8312>
>>> HADOOP-14100 <https://issues.apache.org/jira/browse/HADOOP-14100>
>>>
>>> Added them to 2.7.4 release. You should see them via the above link now.
>>> Would be good if you could attach backport patches for some of them?
>>>
>>> Appreciate your help,
>>> --Konstantin
>>>
>>> On Mon, May 8, 2017 at 8:39 AM, Brahma Reddy Battula <
>>> brahmareddy.batt...@huawei.com> wrote:
>>>
>>>
>>>> Looks following link is not correct..
>>>>
>>>> https://s.apache.org/Dzg4
>>>>
>>>> It should be like following..?
>>>>
>>>> https://s.apache.org/wi3U
>>>>
>>>>
>>>> Apart from Konstantin mentioned,Following also good to go..? let me know
>>>> your thoughts on this.
>>>>
>>>> For Large Cluster:
>>>> =
>>>>
>>>> https://issues.apache.org/jira/browse/HDFS-9311===Life Line Protocol
>>>> https://issues.apache.org/jira/browse/HDFS-10987===Deecommission
>>>> Expensive when lot's of blocks are present
>>>>
>>>> https://issues.apache.org/jira/browse/HDFS-9902===
>>>> "dfs.datanode.du.reserved"  per Storage Type
>>>>
>>>> For Security:
>>>> =
>>>> https://issues.apache.org/jira/browse/HDFS-8312===Trash does not
>>>> descent
>>>> into child directories to check for permission
>>>> https://issues.apache.org/jira/browse/HADOOP-14100===Upgrade Jsch jar
>>>> to
>>>> latest version to fix vulnerability i

Re: About 2.7.4 Release

[Konstantin Shvachko]

Hey Akira,

I didn't have private filters. Most probably Jira caches something.
Your filter is in the right direction, but for some reason it lists only 22
issues, while mine has 29.
It misses e.g. YARN-5543 <https://issues.apache.org/jira/browse/YARN-5543>.

Anyways, I created a Jira filter now "Hadoop 2.7.4 release blockers",
shared it with "everybody", and updated my link to point to that filter. So
you can use any of the three methods below to get the correct list:
1. Go to https://s.apache.org/Dzg4
2. Go to the filter via
https://issues.apache.org/jira/issues?filter=12340814
   or by finding "Hadoop 2.7.4 release blockers" filter in the jira
3. On Advanced issues search page paste this:
project in (HDFS, HADOOP, YARN, MAPREDUCE) AND labels = release-blocker AND
"Target Version/s" = 2.7.4

Hope this solves the confusion for which issues are included.
Please LMK if it doesn't, as it is important.

Thanks,
--Konstantin

On Tue, May 9, 2017 at 9:58 AM, Akira Ajisaka <aajis...@apache.org> wrote:

> Hi Konstantin,
>
> Thank you for volunteering as release manager!
>
> > Actually the original link works fine: https://s.apache.org/Dzg4
> I couldn't see the link. Maybe is it private filter?
>
> Here is a link I generated: https://s.apache.org/ehKy
> This filter includes resolved issue and excludes fixversion == 2.7.4
>
> Thanks and Regards,
> Akira
>
> On 2017/05/08 19:20, Konstantin Shvachko wrote:
>
>> Hi Brahma Reddy Battula,
>>
>> Actually the original link works fine: https://s.apache.org/Dzg4
>> Your link excludes closed and resolved issues, which needs backporting,
>> and
>> which we cannot reopen, as discussed in this thread earlier.
>>
>> Looked through the issues you proposed:
>>
>> HDFS-9311 <https://issues.apache.org/jira/browse/HDFS-9311>
>> Seems like a new feature. It helps failover to standby node when primary
>> is
>> under heavy load, but it introduces new APIs, addresses, config
>> parameters.
>> And needs at least one follow up jira.
>> Looks like a backward compatible change, though.
>> Did you have a chance to run it in production?
>>
>> +1 on
>> HDFS-10987 <https://issues.apache.org/jira/browse/HDFS-10987>
>> HDFS-9902 <https://issues.apache.org/jira/browse/HDFS-9902>
>> HDFS-8312 <https://issues.apache.org/jira/browse/HDFS-8312>
>> HADOOP-14100 <https://issues.apache.org/jira/browse/HADOOP-14100>
>>
>> Added them to 2.7.4 release. You should see them via the above link now.
>> Would be good if you could attach backport patches for some of them?
>>
>> Appreciate your help,
>> --Konstantin
>>
>> On Mon, May 8, 2017 at 8:39 AM, Brahma Reddy Battula <
>> brahmareddy.batt...@huawei.com> wrote:
>>
>>
>>> Looks following link is not correct..
>>>
>>> https://s.apache.org/Dzg4
>>>
>>> It should be like following..?
>>>
>>> https://s.apache.org/wi3U
>>>
>>>
>>> Apart from Konstantin mentioned,Following also good to go..? let me know
>>> your thoughts on this.
>>>
>>> For Large Cluster:
>>> =
>>>
>>> https://issues.apache.org/jira/browse/HDFS-9311===Life Line Protocol
>>> https://issues.apache.org/jira/browse/HDFS-10987===Deecommission
>>> Expensive when lot's of blocks are present
>>>
>>> https://issues.apache.org/jira/browse/HDFS-9902===
>>> "dfs.datanode.du.reserved"  per Storage Type
>>>
>>> For Security:
>>> =
>>> https://issues.apache.org/jira/browse/HDFS-8312===Trash does not descent
>>> into child directories to check for permission
>>> https://issues.apache.org/jira/browse/HADOOP-14100===Upgrade Jsch jar to
>>> latest version to fix vulnerability in old versions
>>>
>>>
>>>
>>> Regards
>>> Brahma Reddy Battula
>>>
>>> -Original Message-
>>> From: Erik Krogen [mailto:ekro...@linkedin.com.INVALID]
>>> Sent: 06 May 2017 02:40
>>> To: Konstantin Shvachko
>>> Cc: Zhe Zhang; Hadoop Common; Hdfs-dev; mapreduce-...@hadoop.apache.org;
>>> yarn-dev@hadoop.apache.org
>>> Subject: Re: About 2.7.4 Release
>>>
>>> List LGTM Konstantin!
>>>
>>> Let's say that we will only create a new tracking JIRA for patches which
>>> do not backport cleanly, to avoid having too many lying around. Otherwise
>>> we can directly attach to old ticket. If a clean backport does happen to
>>> break a test the nightly 

Re: About 2.7.4 Release

[Konstantin Shvachko]

Hi Brahma Reddy Battula,

Actually the original link works fine: https://s.apache.org/Dzg4
Your link excludes closed and resolved issues, which needs backporting, and
which we cannot reopen, as discussed in this thread earlier.

Looked through the issues you proposed:

HDFS-9311 <https://issues.apache.org/jira/browse/HDFS-9311>
Seems like a new feature. It helps failover to standby node when primary is
under heavy load, but it introduces new APIs, addresses, config parameters.
And needs at least one follow up jira.
Looks like a backward compatible change, though.
Did you have a chance to run it in production?

+1 on
HDFS-10987 <https://issues.apache.org/jira/browse/HDFS-10987>
HDFS-9902 <https://issues.apache.org/jira/browse/HDFS-9902>
HDFS-8312 <https://issues.apache.org/jira/browse/HDFS-8312>
HADOOP-14100 <https://issues.apache.org/jira/browse/HADOOP-14100>

Added them to 2.7.4 release. You should see them via the above link now.
Would be good if you could attach backport patches for some of them?

Appreciate your help,
--Konstantin

On Mon, May 8, 2017 at 8:39 AM, Brahma Reddy Battula <
brahmareddy.batt...@huawei.com> wrote:

>
> Looks following link is not correct..
>
> https://s.apache.org/Dzg4
>
> It should be like following..?
>
> https://s.apache.org/wi3U
>
>
> Apart from Konstantin mentioned,Following also good to go..? let me know
> your thoughts on this.
>
> For Large Cluster:
> =
>
> https://issues.apache.org/jira/browse/HDFS-9311===Life Line Protocol
> https://issues.apache.org/jira/browse/HDFS-10987===Deecommission
> Expensive when lot's of blocks are present
>
> https://issues.apache.org/jira/browse/HDFS-9902===
> "dfs.datanode.du.reserved"  per Storage Type
>
> For Security:
> =
> https://issues.apache.org/jira/browse/HDFS-8312===Trash does not descent
> into child directories to check for permission
> https://issues.apache.org/jira/browse/HADOOP-14100===Upgrade Jsch jar to
> latest version to fix vulnerability in old versions
>
>
>
> Regards
> Brahma Reddy Battula
>
> -Original Message-
> From: Erik Krogen [mailto:ekro...@linkedin.com.INVALID]
> Sent: 06 May 2017 02:40
> To: Konstantin Shvachko
> Cc: Zhe Zhang; Hadoop Common; Hdfs-dev; mapreduce-...@hadoop.apache.org;
> yarn-dev@hadoop.apache.org
> Subject: Re: About 2.7.4 Release
>
> List LGTM Konstantin!
>
> Let's say that we will only create a new tracking JIRA for patches which
> do not backport cleanly, to avoid having too many lying around. Otherwise
> we can directly attach to old ticket. If a clean backport does happen to
> break a test the nightly build will help us catch it.
>
> Erik
>
> On Thu, May 4, 2017 at 7:21 PM, Konstantin Shvachko <shv.had...@gmail.com>
> wrote:
>
> > Great Zhe. Let's monitor the build.
> >
> > I marked all jiras I knew of for inclusion into 2.7.4 as I described
> > before.
> > Target Version/s: 2.7.4
> > Label: release-blocker
> >
> > Here is the link to the list: https://s.apache.org/Dzg4 Please let me
> > know if I missed anything.
> > And feel free to pick up any. Most of backports are pretty
> > straightforward, but not all.
> >
> > We can create tracking jiras for backporting if you need to run
> > Jenkins on the patch (and since Allen does not allow reopening them).
> > But I think the final patch should be attached to the original jira.
> > Otherwise history will be hard to follow.
> >
> > Thanks,
> > --Konstantin
> >
> > On Wed, May 3, 2017 at 4:53 PM, Zhe Zhang <z...@apache.org> wrote:
> >
> > > Thanks for volunteering as RM Konstantin! The plan LGTM.
> > >
> > > I've created a nightly Jenkins job for branch-2.7 (unit tests):
> > > https://builds.apache.org/job/Hadoop-branch2.7-nightly/
> > >
> > > On Wed, May 3, 2017 at 12:42 AM Konstantin Shvachko <
> > shv.had...@gmail.com>
> > > wrote:
> > >
> > >> Hey guys,
> > >>
> > >> I and a few of my colleagues would like to help here and move 2.7.4
> > >> release forward. A few points in this regard.
> > >>
> > >> 1. Reading through this thread since March 1 I see that Vinod
> > >> hinted on managing the release. Vinod, if you still want the job /
> > >> have bandwidth will be happy to work with you.
> > >> Otherwise I am glad to volunteer as the release manager.
> > >>
> > >> 2. In addition to current blockers and criticals, I would like to
> > propose
> > >> a
> > >> few issues to be included in the release, see the list below. Tho

RE: About 2.7.4 Release

[Brahma Reddy Battula]

Looks following link is not correct.. 

https://s.apache.org/Dzg4

It should be like following..?

https://s.apache.org/wi3U


Apart from Konstantin mentioned,Following also good to go..? let me know your 
thoughts on this.

For Large Cluster:
=

https://issues.apache.org/jira/browse/HDFS-9311===Life Line Protocol
https://issues.apache.org/jira/browse/HDFS-10987===Deecommission Expensive when 
lot's of blocks are present

https://issues.apache.org/jira/browse/HDFS-9902=== "dfs.datanode.du.reserved"  
per Storage Type

For Security:
=
https://issues.apache.org/jira/browse/HDFS-8312===Trash does not descent into 
child directories to check for permission
https://issues.apache.org/jira/browse/HADOOP-14100===Upgrade Jsch jar to latest 
version to fix vulnerability in old versions



Regards
Brahma Reddy Battula

-Original Message-
From: Erik Krogen [mailto:ekro...@linkedin.com.INVALID] 
Sent: 06 May 2017 02:40
To: Konstantin Shvachko
Cc: Zhe Zhang; Hadoop Common; Hdfs-dev; mapreduce-...@hadoop.apache.org; 
yarn-dev@hadoop.apache.org
Subject: Re: About 2.7.4 Release

List LGTM Konstantin!

Let's say that we will only create a new tracking JIRA for patches which do not 
backport cleanly, to avoid having too many lying around. Otherwise we can 
directly attach to old ticket. If a clean backport does happen to break a test 
the nightly build will help us catch it.

Erik

On Thu, May 4, 2017 at 7:21 PM, Konstantin Shvachko <shv.had...@gmail.com>
wrote:

> Great Zhe. Let's monitor the build.
>
> I marked all jiras I knew of for inclusion into 2.7.4 as I described 
> before.
> Target Version/s: 2.7.4
> Label: release-blocker
>
> Here is the link to the list: https://s.apache.org/Dzg4 Please let me 
> know if I missed anything.
> And feel free to pick up any. Most of backports are pretty 
> straightforward, but not all.
>
> We can create tracking jiras for backporting if you need to run 
> Jenkins on the patch (and since Allen does not allow reopening them).
> But I think the final patch should be attached to the original jira.
> Otherwise history will be hard to follow.
>
> Thanks,
> --Konstantin
>
> On Wed, May 3, 2017 at 4:53 PM, Zhe Zhang <z...@apache.org> wrote:
>
> > Thanks for volunteering as RM Konstantin! The plan LGTM.
> >
> > I've created a nightly Jenkins job for branch-2.7 (unit tests):
> > https://builds.apache.org/job/Hadoop-branch2.7-nightly/
> >
> > On Wed, May 3, 2017 at 12:42 AM Konstantin Shvachko <
> shv.had...@gmail.com>
> > wrote:
> >
> >> Hey guys,
> >>
> >> I and a few of my colleagues would like to help here and move 2.7.4 
> >> release forward. A few points in this regard.
> >>
> >> 1. Reading through this thread since March 1 I see that Vinod 
> >> hinted on managing the release. Vinod, if you still want the job / 
> >> have bandwidth will be happy to work with you.
> >> Otherwise I am glad to volunteer as the release manager.
> >>
> >> 2. In addition to current blockers and criticals, I would like to
> propose
> >> a
> >> few issues to be included in the release, see the list below. Those 
> >> are mostly bug fixes and optimizations, which we already have in 
> >> our
> internal
> >> branch and run in production. Plus one minor feature "node 
> >> labeling", which we found very handy, when you have heterogeneous 
> >> environments and mixed workloads, like MR and Spark.
> >>
> >> 3. For marking issues for the release I propose to
> >>  - set the target version to 2.7.4, and
> >>  - add a new label "release-blocker"
> >> That way we will know issues targeted for the release without 
> >> reopening them for backports.
> >>
> >> 4. I see quite a few people are interested in the release. With all 
> >> the help I think we can target to release by the end of May.
> >>
> >> Other things include fixing CHANGES.txt and fixing Jenkins build 
> >> for
> 2.7.4
> >> branch.
> >>
> >> Thanks,
> >> --Konstantin
> >>
> >> ==  List of issue for 2.7.4  ===
> >> -- Backports
> >> HADOOP-12975 <https://issues.apache.org/jira/browse/HADOOP-12975>. 
> >> Add
> du
> >> jitters
> >> HDFS-9710 <https://issues.apache.org/jira/browse/HDFS-9710>. IBR
> batching
> >> HDFS-10715 <https://issues.apache.org/jira/browse/HDFS-10715>. NPE 
> >> when applying AvailableSpaceBlockPlacementPolicy
> >> HDFS-2538 <https://issues.apache.org/jira/browse/HDFS-2538>. fsck
> re

Re: About 2.7.4 Release

[Konstantin Shvachko]

Great Zhe. Let's monitor the build.

I marked all jiras I knew of for inclusion into 2.7.4 as I described before.
Target Version/s: 2.7.4
Label: release-blocker

Here is the link to the list: https://s.apache.org/Dzg4
Please let me know if I missed anything.
And feel free to pick up any. Most of backports are pretty straightforward,
but not all.

We can create tracking jiras for backporting if you need to run Jenkins on
the patch (and since Allen does not allow reopening them).
But I think the final patch should be attached to the original jira.
Otherwise history will be hard to follow.

Thanks,
--Konstantin

On Wed, May 3, 2017 at 4:53 PM, Zhe Zhang  wrote:

> Thanks for volunteering as RM Konstantin! The plan LGTM.
>
> I've created a nightly Jenkins job for branch-2.7 (unit tests):
> https://builds.apache.org/job/Hadoop-branch2.7-nightly/
>
> On Wed, May 3, 2017 at 12:42 AM Konstantin Shvachko 
> wrote:
>
>> Hey guys,
>>
>> I and a few of my colleagues would like to help here and move 2.7.4
>> release
>> forward. A few points in this regard.
>>
>> 1. Reading through this thread since March 1 I see that Vinod hinted on
>> managing the release. Vinod, if you still want the job / have bandwidth
>> will be happy to work with you.
>> Otherwise I am glad to volunteer as the release manager.
>>
>> 2. In addition to current blockers and criticals, I would like to propose
>> a
>> few issues to be included in the release, see the list below. Those are
>> mostly bug fixes and optimizations, which we already have in our internal
>> branch and run in production. Plus one minor feature "node labeling",
>> which
>> we found very handy, when you have heterogeneous environments and mixed
>> workloads, like MR and Spark.
>>
>> 3. For marking issues for the release I propose to
>>  - set the target version to 2.7.4, and
>>  - add a new label "release-blocker"
>> That way we will know issues targeted for the release without reopening
>> them for backports.
>>
>> 4. I see quite a few people are interested in the release. With all the
>> help I think we can target to release by the end of May.
>>
>> Other things include fixing CHANGES.txt and fixing Jenkins build for 2.7.4
>> branch.
>>
>> Thanks,
>> --Konstantin
>>
>> ==  List of issue for 2.7.4  ===
>> -- Backports
>> HADOOP-12975 . Add du
>> jitters
>> HDFS-9710 . IBR batching
>> HDFS-10715 . NPE when
>> applying AvailableSpaceBlockPlacementPolicy
>> HDFS-2538 . fsck removal
>> of dot printing
>> HDFS-8131 .
>> space-balanced
>> policy for balancer
>> HDFS-8549 . abort
>> balancer
>> if upgrade in progress
>> HDFS-9412 . skip small
>> blocks in getBlocks
>>
>> YARN-1471 . SLS
>> simulator
>> YARN-4302 . SLS
>> YARN-4367 . SLS
>> YARN-4612 . SLS
>>
>> - Node labeling
>> MAPREDUCE-6304 
>> YARN-2943 
>> YARN-4109 
>> YARN-4140 
>> YARN-4250 
>> YARN-4925 
>>
> --
> Zhe Zhang
> Apache Hadoop Committer
> http://zhe-thoughts.github.io/about/ | @oldcap
>

Re: About 2.7.4 Release

[Zhe Zhang]

Thanks for volunteering as RM Konstantin! The plan LGTM.

I've created a nightly Jenkins job for branch-2.7 (unit tests):
https://builds.apache.org/job/Hadoop-branch2.7-nightly/

On Wed, May 3, 2017 at 12:42 AM Konstantin Shvachko 
wrote:

> Hey guys,
>
> I and a few of my colleagues would like to help here and move 2.7.4 release
> forward. A few points in this regard.
>
> 1. Reading through this thread since March 1 I see that Vinod hinted on
> managing the release. Vinod, if you still want the job / have bandwidth
> will be happy to work with you.
> Otherwise I am glad to volunteer as the release manager.
>
> 2. In addition to current blockers and criticals, I would like to propose a
> few issues to be included in the release, see the list below. Those are
> mostly bug fixes and optimizations, which we already have in our internal
> branch and run in production. Plus one minor feature "node labeling", which
> we found very handy, when you have heterogeneous environments and mixed
> workloads, like MR and Spark.
>
> 3. For marking issues for the release I propose to
>  - set the target version to 2.7.4, and
>  - add a new label "release-blocker"
> That way we will know issues targeted for the release without reopening
> them for backports.
>
> 4. I see quite a few people are interested in the release. With all the
> help I think we can target to release by the end of May.
>
> Other things include fixing CHANGES.txt and fixing Jenkins build for 2.7.4
> branch.
>
> Thanks,
> --Konstantin
>
> ==  List of issue for 2.7.4  ===
> -- Backports
> HADOOP-12975 . Add du
> jitters
> HDFS-9710 . IBR batching
> HDFS-10715 . NPE when
> applying AvailableSpaceBlockPlacementPolicy
> HDFS-2538 . fsck removal
> of dot printing
> HDFS-8131 .
> space-balanced
> policy for balancer
> HDFS-8549 . abort
> balancer
> if upgrade in progress
> HDFS-9412 . skip small
> blocks in getBlocks
>
> YARN-1471 . SLS simulator
> YARN-4302 . SLS
> YARN-4367 . SLS
> YARN-4612 . SLS
>
> - Node labeling
> MAPREDUCE-6304 
> YARN-2943 
> YARN-4109 
> YARN-4140 
> YARN-4250 
> YARN-4925 
>
-- 
Zhe Zhang
Apache Hadoop Committer
http://zhe-thoughts.github.io/about/ | @oldcap

Re: About 2.7.4 Release

[Konstantin Shvachko]

Hey guys,

I and a few of my colleagues would like to help here and move 2.7.4 release
forward. A few points in this regard.

1. Reading through this thread since March 1 I see that Vinod hinted on
managing the release. Vinod, if you still want the job / have bandwidth
will be happy to work with you.
Otherwise I am glad to volunteer as the release manager.

2. In addition to current blockers and criticals, I would like to propose a
few issues to be included in the release, see the list below. Those are
mostly bug fixes and optimizations, which we already have in our internal
branch and run in production. Plus one minor feature "node labeling", which
we found very handy, when you have heterogeneous environments and mixed
workloads, like MR and Spark.

3. For marking issues for the release I propose to
 - set the target version to 2.7.4, and
 - add a new label "release-blocker"
That way we will know issues targeted for the release without reopening
them for backports.

4. I see quite a few people are interested in the release. With all the
help I think we can target to release by the end of May.

Other things include fixing CHANGES.txt and fixing Jenkins build for 2.7.4
branch.

Thanks,
--Konstantin

==  List of issue for 2.7.4  ===
-- Backports
HADOOP-12975 . Add du
jitters
HDFS-9710 . IBR batching
HDFS-10715 . NPE when
applying AvailableSpaceBlockPlacementPolicy
HDFS-2538 . fsck removal
of dot printing
HDFS-8131 . space-balanced
policy for balancer
HDFS-8549 . abort balancer
if upgrade in progress
HDFS-9412 . skip small
blocks in getBlocks

YARN-1471 . SLS simulator
YARN-4302 . SLS
YARN-4367 . SLS
YARN-4612 . SLS

- Node labeling
MAPREDUCE-6304 
YARN-2943 
YARN-4109 
YARN-4140 
YARN-4250 
YARN-4925 

Re: About 2.7.4 Release

[Andrew Wang]

Can we wait for 2.7.4 first? There are still backports happening to
branch-2.7. After that, there shouldn't be many backports since both 2.8.x
and 2.7.x will be up-to-date with what's in 3.0.0-alpha1 and 3.0.0-alpha2.


On Tue, May 2, 2017 at 4:07 PM, Allen Wittenauer 
wrote:

>
> Is there any reason to not Close -alpha1+resolved state JIRAs?  It's been
> quite a while and those definitely should not getting re-opened anymore.
> What about -alpha2's that are also resolved?

Re: About 2.7.4 Release

[Allen Wittenauer]

Is there any reason to not Close -alpha1+resolved state JIRAs?  It's been quite 
a while and those definitely should not getting re-opened anymore.  What about 
-alpha2's that are also resolved?
-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: About 2.7.4 Release

[Andrew Wang]

On Mon, May 1, 2017 at 3:44 PM, Allen Wittenauer 
wrote:

>
> > On May 1, 2017, at 2:27 PM, Andrew Wang 
> wrote:
> > I believe I asked about this on dev-yetus a while back. I'd prefer that
> the presence of the fix version be sufficient to indicate whether a JIRA is
> included in a release branch. Yetus requires that the JIRA be resolved as
> "Fixed" to show up, which is why we are in our current situation.
>
> We can't do this because Hadoop is the only one that I've seen
> that sets Fix version at close time.  Everyone else is setting fix version
> in place of target (which is a custom field, iirc).
>
> Let's see if I can revive the discussion over on a yetus list/jira. I
think it's easier to add a new flag to Yetus than changing the Hadoop JIRA
workflow, and it seems like this issue is becoming more acute.

Re: About 2.7.4 Release

[Allen Wittenauer]

> On May 1, 2017, at 2:27 PM, Andrew Wang  wrote:
> I believe I asked about this on dev-yetus a while back. I'd prefer that the 
> presence of the fix version be sufficient to indicate whether a JIRA is 
> included in a release branch. Yetus requires that the JIRA be resolved as 
> "Fixed" to show up, which is why we are in our current situation.

We can't do this because Hadoop is the only one that I've seen that 
sets Fix version at close time.  Everyone else is setting fix version in place 
of target (which is a custom field, iirc).



-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: About 2.7.4 Release

[Andrew Wang]

I didn't close JIRAs after the 3.0.0-alpha1 or alpha2 releases since
closing makes the JIRAs read-only. This makes it more annoying to backport
to older releases and for concurrent releases in general.

I believe I asked about this on dev-yetus a while back. I'd prefer that the
presence of the fix version be sufficient to indicate whether a JIRA is
included in a release branch. Yetus requires that the JIRA be resolved as
"Fixed" to show up, which is why we are in our current situation.

On Thu, Apr 27, 2017 at 12:47 AM, Akira Ajisaka  wrote:

> Thanks Allen for the additional information.
>
> > At one point, JIRA was configured to refuse re-opening after a release
> is cut.
>
> In the past, release manager closed the tickets and the process is written
> in the wiki: https://wiki.apache.org/hadoop/HowToRelease
>
> > 10. In JIRA, close issues resolved in the release. Disable mail
> notifications for this bulk change.
>
> Therefore, let's close them.
>
> On 2017/04/27 3:01, Allen Wittenauer wrote:
>
>>
>> On Apr 25, 2017, at 12:35 AM, Akira Ajisaka  wrote:
>>>
 Maybe we should create a jira to track this?

>>>
>>> I think now either way (reopen or create) is fine.
>>>
>>> Release doc maker creates change logs by fetching information from JIRA,
>>> so reopening the tickets should be avoided when a release process is in
>>> progress.
>>>
>>>
>> Keep in mind that the release documentation is part of the build
>> process.  Users who are doing their own builds will have incomplete
>> documentation if we keep re-opening JIRAs after a release.  At one point,
>> JIRA was configured to refuse re-opening after a release is cut.  I'm not
>> sure why it stopped doing that, but it might be time to see if we can
>> re-enable that functionality.
>>
>>
>> -
>> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
>> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>>
>>
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>
>

Re: About 2.7.4 Release

[Akira Ajisaka]

Thanks Allen for the additional information.

> At one point, JIRA was configured to refuse re-opening after a 
release is cut.


In the past, release manager closed the tickets and the process is 
written in the wiki: https://wiki.apache.org/hadoop/HowToRelease


> 10. In JIRA, close issues resolved in the release. Disable mail 
notifications for this bulk change.


Therefore, let's close them.

On 2017/04/27 3:01, Allen Wittenauer wrote:



On Apr 25, 2017, at 12:35 AM, Akira Ajisaka  wrote:

Maybe we should create a jira to track this?


I think now either way (reopen or create) is fine.

Release doc maker creates change logs by fetching information from JIRA, so 
reopening the tickets should be avoided when a release process is in progress.



Keep in mind that the release documentation is part of the build 
process.  Users who are doing their own builds will have incomplete 
documentation if we keep re-opening JIRAs after a release.  At one point, JIRA 
was configured to refuse re-opening after a release is cut.  I'm not sure why 
it stopped doing that, but it might be time to see if we can re-enable that 
functionality.


-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org



-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: About 2.7.4 Release

[Allen Wittenauer]

> On Apr 25, 2017, at 12:35 AM, Akira Ajisaka  wrote:
> > Maybe we should create a jira to track this?
> 
> I think now either way (reopen or create) is fine.
> 
> Release doc maker creates change logs by fetching information from JIRA, so 
> reopening the tickets should be avoided when a release process is in progress.
> 

Keep in mind that the release documentation is part of the build 
process.  Users who are doing their own builds will have incomplete 
documentation if we keep re-opening JIRAs after a release.  At one point, JIRA 
was configured to refuse re-opening after a release is cut.  I'm not sure why 
it stopped doing that, but it might be time to see if we can re-enable that 
functionality.


-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: About 2.7.4 Release

[Akira Ajisaka]

Okay. If you file a jira and attach a patch, I'll review it.

-Akira

On 2017/04/25 22:15, Brahma Reddy Battula wrote:

Looks Following Jira's are not updated in CHANGES.txt


HADOOP-14066,HDFS-11608,HADOOP-14293,HDFS-11628,YARN-6274,YARN-6152,HADOOP-13119,HDFS-10733,HADOOP-13958,HDFS-11280,YARN-6024.

May be we can raise one Jira to track this..?


--Brahma Reddy Battula

-Original Message-
From: Akira Ajisaka [mailto:aajis...@apache.org]
Sent: 25 April 2017 15:36
To: Haohui Mai
Cc: Brahma Reddy Battula; Andrew Wang; Sangjin Lee; Vinod Kumar Vavilapalli; 
Marton Elek; Hadoop Common; yarn-dev@hadoop.apache.org; Hdfs-dev; 
mapreduce-...@hadoop.apache.org
Subject: Re: About 2.7.4 Release

 > It would be great to backport HDFS-9710 to 2.7.4 as this is one of the  > 
critical fixes on scalability.

Sounds good.

 > Maybe we should create a jira to track this?

I think now either way (reopen or create) is fine.

Release doc maker creates change logs by fetching information from JIRA, so 
reopening the tickets should be avoided when a release process is in progress.

The issue HDFS-9710 (and HDFS-9726) have been fixed in 2.8.0 and
3.0.0-alpha1 and both versions have been released, so reopening this issue does 
not affect the release doc maker.

-Akira

On 2017/04/25 16:21, Haohui Mai wrote:

It would be great to backport HDFS-9710 to 2.7.4 as this is one of the
critical fixes on scalability. Maybe we should create a jira to track
this?

~Haohui

On Tue, Apr 25, 2017 at 12:06 AM, Akira Ajisaka <aajis...@apache.org> wrote:

Ping

I too can help with the release process.

Now there are 0 blocker and 6 critical issues targeted for 2.7.4.
https://s.apache.org/HsIu

If there are critical/blocker issues that need to be fixed in
branch-2.7, please set Target Version/s to 2.7.4. That way the issues
can be found by the above query.

I'll check if there are conflicts among JIRA, git commit log, and the
change logs.

Regards,
Akira


On 2017/04/18 15:40, Brahma Reddy Battula wrote:


Hi All

Any update on 2.7.4 ..?  Gentle Remainder!! Let me know anything I
can help on this..



Regards
Brahma Reddy Battula

-Original Message-
From: Andrew Wang [mailto:andrew.w...@cloudera.com]
Sent: 08 March 2017 04:22
To: Sangjin Lee
Cc: Marton Elek; Hadoop Common; yarn-dev@hadoop.apache.org;
Hdfs-dev; mapreduce-...@hadoop.apache.org
Subject: Re: About 2.7.4 Release

Our release steps are documented on the wiki:

2.6/2.7:

https://wiki.apache.org/hadoop/HowToReleasePreDSBCR

2.8+:
https://wiki.apache.org/hadoop/HowToRelease

I think given the push toward 2.8 and 3.0, there's less interest in
streamlining the 2.6 and 2.7 release processes. CHANGES.txt is the
biggest pain, and that's fixed in 2.8+.

Current pain points for 2.8+ include:

# fixing up JIRA versions and the release notes, though I somewhat
addressed this with the versions script for 3.x # making and staging
an RC and sending the vote email still requires a lot of manual
steps # publishing the release is also quite manual

I think the RC issues can be attacked with enough scripting. Steve
had an ant file that automated a lot of this for slider. I think
it'd be nice to have a nightly Jenkins job that builds an RC, since
I've spent a day or two for each 3.x alpha fixing build issues.

Publishing can be attacked via a mix of scripting and revamping the
darned website. Forrest is pretty bad compared to the newer static
site generators out there (e.g. need to write XML instead of
markdown, it's hard to review a staging site because of all the
absolute links, hard to customize, did I mention XML?), and the look
and feel of the site is from the 00s. We don't actually have that
much site content, so it should be possible to migrate to a new system.

On Tue, Mar 7, 2017 at 9:13 AM, Sangjin Lee <sj...@apache.org> wrote:


I don't think there should be any linkage between releasing 2.8.0
and 2.7.4. If we have a volunteer for releasing 2.7.4, we should go
full speed ahead. We still need a volunteer from a PMC member or a
committer as some tasks may require certain privileges, but I don't
think it precludes working with others to close down the release.

I for one would like to see more frequent releases, and being able
to automate release steps more would go a long way.

On Tue, Mar 7, 2017 at 2:16 AM, Marton Elek <me...@hortonworks.com>
wrote:


Is there any reason to wait for 2.8 with 2.7.4?

Unfortunately the previous  thread about release cadence has been
ended without final decision. But if I understood well, there was
more or less


an


agreement about that it would be great to achieve more frequent
releases, if possible (with or without written rules and EOL policy).

I personally prefer to be more closer to the scheduling part of
the
proposal:

"A minor release on the latest major line should be every 6
months, and a maintenance release on a minor release (as there may
be concurrently maintained minor releases) every 2 months&quo

RE: About 2.7.4 Release

[Brahma Reddy Battula]

Looks Following Jira's are not updated in CHANGES.txt


HADOOP-14066,HDFS-11608,HADOOP-14293,HDFS-11628,YARN-6274,YARN-6152,HADOOP-13119,HDFS-10733,HADOOP-13958,HDFS-11280,YARN-6024.

May be we can raise one Jira to track this..?


--Brahma Reddy Battula

-Original Message-
From: Akira Ajisaka [mailto:aajis...@apache.org] 
Sent: 25 April 2017 15:36
To: Haohui Mai
Cc: Brahma Reddy Battula; Andrew Wang; Sangjin Lee; Vinod Kumar Vavilapalli; 
Marton Elek; Hadoop Common; yarn-dev@hadoop.apache.org; Hdfs-dev; 
mapreduce-...@hadoop.apache.org
Subject: Re: About 2.7.4 Release

 > It would be great to backport HDFS-9710 to 2.7.4 as this is one of the  > 
 > critical fixes on scalability.

Sounds good.

 > Maybe we should create a jira to track this?

I think now either way (reopen or create) is fine.

Release doc maker creates change logs by fetching information from JIRA, so 
reopening the tickets should be avoided when a release process is in progress.

The issue HDFS-9710 (and HDFS-9726) have been fixed in 2.8.0 and
3.0.0-alpha1 and both versions have been released, so reopening this issue does 
not affect the release doc maker.

-Akira

On 2017/04/25 16:21, Haohui Mai wrote:
> It would be great to backport HDFS-9710 to 2.7.4 as this is one of the 
> critical fixes on scalability. Maybe we should create a jira to track 
> this?
>
> ~Haohui
>
> On Tue, Apr 25, 2017 at 12:06 AM, Akira Ajisaka <aajis...@apache.org> wrote:
>> Ping
>>
>> I too can help with the release process.
>>
>> Now there are 0 blocker and 6 critical issues targeted for 2.7.4.
>> https://s.apache.org/HsIu
>>
>> If there are critical/blocker issues that need to be fixed in 
>> branch-2.7, please set Target Version/s to 2.7.4. That way the issues 
>> can be found by the above query.
>>
>> I'll check if there are conflicts among JIRA, git commit log, and the 
>> change logs.
>>
>> Regards,
>> Akira
>>
>>
>> On 2017/04/18 15:40, Brahma Reddy Battula wrote:
>>>
>>> Hi All
>>>
>>> Any update on 2.7.4 ..?  Gentle Remainder!! Let me know anything I 
>>> can help on this..
>>>
>>>
>>>
>>> Regards
>>> Brahma Reddy Battula
>>>
>>> -Original Message-
>>> From: Andrew Wang [mailto:andrew.w...@cloudera.com]
>>> Sent: 08 March 2017 04:22
>>> To: Sangjin Lee
>>> Cc: Marton Elek; Hadoop Common; yarn-dev@hadoop.apache.org; 
>>> Hdfs-dev; mapreduce-...@hadoop.apache.org
>>> Subject: Re: About 2.7.4 Release
>>>
>>> Our release steps are documented on the wiki:
>>>
>>> 2.6/2.7:
>>>
>>> https://wiki.apache.org/hadoop/HowToReleasePreDSBCR
>>>
>>> 2.8+:
>>> https://wiki.apache.org/hadoop/HowToRelease
>>>
>>> I think given the push toward 2.8 and 3.0, there's less interest in 
>>> streamlining the 2.6 and 2.7 release processes. CHANGES.txt is the 
>>> biggest pain, and that's fixed in 2.8+.
>>>
>>> Current pain points for 2.8+ include:
>>>
>>> # fixing up JIRA versions and the release notes, though I somewhat 
>>> addressed this with the versions script for 3.x # making and staging 
>>> an RC and sending the vote email still requires a lot of manual 
>>> steps # publishing the release is also quite manual
>>>
>>> I think the RC issues can be attacked with enough scripting. Steve 
>>> had an ant file that automated a lot of this for slider. I think 
>>> it'd be nice to have a nightly Jenkins job that builds an RC, since 
>>> I've spent a day or two for each 3.x alpha fixing build issues.
>>>
>>> Publishing can be attacked via a mix of scripting and revamping the 
>>> darned website. Forrest is pretty bad compared to the newer static 
>>> site generators out there (e.g. need to write XML instead of 
>>> markdown, it's hard to review a staging site because of all the 
>>> absolute links, hard to customize, did I mention XML?), and the look 
>>> and feel of the site is from the 00s. We don't actually have that 
>>> much site content, so it should be possible to migrate to a new system.
>>>
>>> On Tue, Mar 7, 2017 at 9:13 AM, Sangjin Lee <sj...@apache.org> wrote:
>>>
>>>> I don't think there should be any linkage between releasing 2.8.0 
>>>> and 2.7.4. If we have a volunteer for releasing 2.7.4, we should go 
>>>> full speed ahead. We still need a volunteer from a PMC member or a 
>>>> committer as some tasks may require certain privileges, but I don't 
>>>> think 

Re: About 2.7.4 Release

[Akira Ajisaka]

> It would be great to backport HDFS-9710 to 2.7.4 as this is one of the
> critical fixes on scalability.

Sounds good.

> Maybe we should create a jira to track this?

I think now either way (reopen or create) is fine.

Release doc maker creates change logs by fetching information from JIRA, 
so reopening the tickets should be avoided when a release process is in 
progress.


The issue HDFS-9710 (and HDFS-9726) have been fixed in 2.8.0 and 
3.0.0-alpha1 and both versions have been released, so reopening this 
issue does not affect the release doc maker.


-Akira

On 2017/04/25 16:21, Haohui Mai wrote:

It would be great to backport HDFS-9710 to 2.7.4 as this is one of the
critical fixes on scalability. Maybe we should create a jira to track
this?

~Haohui

On Tue, Apr 25, 2017 at 12:06 AM, Akira Ajisaka <aajis...@apache.org> wrote:

Ping

I too can help with the release process.

Now there are 0 blocker and 6 critical issues targeted for 2.7.4.
https://s.apache.org/HsIu

If there are critical/blocker issues that need to be fixed in branch-2.7,
please set Target Version/s to 2.7.4. That way the issues can be found by
the above query.

I'll check if there are conflicts among JIRA, git commit log, and the change
logs.

Regards,
Akira


On 2017/04/18 15:40, Brahma Reddy Battula wrote:


Hi All

Any update on 2.7.4 ..?  Gentle Remainder!! Let me know anything I can
help on this..



Regards
Brahma Reddy Battula

-Original Message-
From: Andrew Wang [mailto:andrew.w...@cloudera.com]
Sent: 08 March 2017 04:22
To: Sangjin Lee
Cc: Marton Elek; Hadoop Common; yarn-dev@hadoop.apache.org; Hdfs-dev;
mapreduce-...@hadoop.apache.org
Subject: Re: About 2.7.4 Release

Our release steps are documented on the wiki:

2.6/2.7:

https://wiki.apache.org/hadoop/HowToReleasePreDSBCR

2.8+:
https://wiki.apache.org/hadoop/HowToRelease

I think given the push toward 2.8 and 3.0, there's less interest in
streamlining the 2.6 and 2.7 release processes. CHANGES.txt is the biggest
pain, and that's fixed in 2.8+.

Current pain points for 2.8+ include:

# fixing up JIRA versions and the release notes, though I somewhat
addressed this with the versions script for 3.x # making and staging an RC
and sending the vote email still requires a lot of manual steps # publishing
the release is also quite manual

I think the RC issues can be attacked with enough scripting. Steve had an
ant file that automated a lot of this for slider. I think it'd be nice to
have a nightly Jenkins job that builds an RC, since I've spent a day or two
for each 3.x alpha fixing build issues.

Publishing can be attacked via a mix of scripting and revamping the darned
website. Forrest is pretty bad compared to the newer static site generators
out there (e.g. need to write XML instead of markdown, it's hard to review a
staging site because of all the absolute links, hard to customize, did I
mention XML?), and the look and feel of the site is from the 00s. We don't
actually have that much site content, so it should be possible to migrate to
a new system.

On Tue, Mar 7, 2017 at 9:13 AM, Sangjin Lee <sj...@apache.org> wrote:


I don't think there should be any linkage between releasing 2.8.0 and
2.7.4. If we have a volunteer for releasing 2.7.4, we should go full
speed ahead. We still need a volunteer from a PMC member or a
committer as some tasks may require certain privileges, but I don't
think it precludes working with others to close down the release.

I for one would like to see more frequent releases, and being able to
automate release steps more would go a long way.

On Tue, Mar 7, 2017 at 2:16 AM, Marton Elek <me...@hortonworks.com>
wrote:


Is there any reason to wait for 2.8 with 2.7.4?

Unfortunately the previous  thread about release cadence has been
ended without final decision. But if I understood well, there was
more or less


an


agreement about that it would be great to achieve more frequent
releases, if possible (with or without written rules and EOL policy).

I personally prefer to be more closer to the scheduling part of the
proposal:

"A minor release on the latest major line should be every 6 months,
and a maintenance release on a minor release (as there may be
concurrently maintained minor releases) every 2 months".

I don't know what is the hardest part of creating new
minor/maintenance releases. But if the problems are technical
(smoketesting, unit tests,


old


release script, anything else) I would be happy to do any task for
new maintenance releases (or more frequent releases).

Regards,
Marton



From: Akira Ajisaka <aajis...@apache.org>
Sent: Tuesday, March 07, 2017 7:34 AM
To: Brahma Reddy Battula; Hadoop Common; yarn-dev@hadoop.apache.org;
Hdfs-dev; mapreduce-...@hadoop.apache.org
Subject: Re: About 2.7.4 Release

Probably 2.8.0 will be released soon.
https://issues.apache.org/jira/browse/HADOOP-13866?
focusedCommentId=15898379=com.atlassian.jira.
pl

Re: About 2.7.4 Release

[Haohui Mai]

It would be great to backport HDFS-9710 to 2.7.4 as this is one of the
critical fixes on scalability. Maybe we should create a jira to track
this?

~Haohui

On Tue, Apr 25, 2017 at 12:06 AM, Akira Ajisaka <aajis...@apache.org> wrote:
> Ping
>
> I too can help with the release process.
>
> Now there are 0 blocker and 6 critical issues targeted for 2.7.4.
> https://s.apache.org/HsIu
>
> If there are critical/blocker issues that need to be fixed in branch-2.7,
> please set Target Version/s to 2.7.4. That way the issues can be found by
> the above query.
>
> I'll check if there are conflicts among JIRA, git commit log, and the change
> logs.
>
> Regards,
> Akira
>
>
> On 2017/04/18 15:40, Brahma Reddy Battula wrote:
>>
>> Hi All
>>
>> Any update on 2.7.4 ..?  Gentle Remainder!! Let me know anything I can
>> help on this..
>>
>>
>>
>> Regards
>> Brahma Reddy Battula
>>
>> -Original Message-
>> From: Andrew Wang [mailto:andrew.w...@cloudera.com]
>> Sent: 08 March 2017 04:22
>> To: Sangjin Lee
>> Cc: Marton Elek; Hadoop Common; yarn-dev@hadoop.apache.org; Hdfs-dev;
>> mapreduce-...@hadoop.apache.org
>> Subject: Re: About 2.7.4 Release
>>
>> Our release steps are documented on the wiki:
>>
>> 2.6/2.7:
>>
>> https://wiki.apache.org/hadoop/HowToReleasePreDSBCR
>>
>> 2.8+:
>> https://wiki.apache.org/hadoop/HowToRelease
>>
>> I think given the push toward 2.8 and 3.0, there's less interest in
>> streamlining the 2.6 and 2.7 release processes. CHANGES.txt is the biggest
>> pain, and that's fixed in 2.8+.
>>
>> Current pain points for 2.8+ include:
>>
>> # fixing up JIRA versions and the release notes, though I somewhat
>> addressed this with the versions script for 3.x # making and staging an RC
>> and sending the vote email still requires a lot of manual steps # publishing
>> the release is also quite manual
>>
>> I think the RC issues can be attacked with enough scripting. Steve had an
>> ant file that automated a lot of this for slider. I think it'd be nice to
>> have a nightly Jenkins job that builds an RC, since I've spent a day or two
>> for each 3.x alpha fixing build issues.
>>
>> Publishing can be attacked via a mix of scripting and revamping the darned
>> website. Forrest is pretty bad compared to the newer static site generators
>> out there (e.g. need to write XML instead of markdown, it's hard to review a
>> staging site because of all the absolute links, hard to customize, did I
>> mention XML?), and the look and feel of the site is from the 00s. We don't
>> actually have that much site content, so it should be possible to migrate to
>> a new system.
>>
>> On Tue, Mar 7, 2017 at 9:13 AM, Sangjin Lee <sj...@apache.org> wrote:
>>
>>> I don't think there should be any linkage between releasing 2.8.0 and
>>> 2.7.4. If we have a volunteer for releasing 2.7.4, we should go full
>>> speed ahead. We still need a volunteer from a PMC member or a
>>> committer as some tasks may require certain privileges, but I don't
>>> think it precludes working with others to close down the release.
>>>
>>> I for one would like to see more frequent releases, and being able to
>>> automate release steps more would go a long way.
>>>
>>> On Tue, Mar 7, 2017 at 2:16 AM, Marton Elek <me...@hortonworks.com>
>>> wrote:
>>>
>>>> Is there any reason to wait for 2.8 with 2.7.4?
>>>>
>>>> Unfortunately the previous  thread about release cadence has been
>>>> ended without final decision. But if I understood well, there was
>>>> more or less
>>>
>>> an
>>>>
>>>> agreement about that it would be great to achieve more frequent
>>>> releases, if possible (with or without written rules and EOL policy).
>>>>
>>>> I personally prefer to be more closer to the scheduling part of the
>>>> proposal:
>>>>
>>>> "A minor release on the latest major line should be every 6 months,
>>>> and a maintenance release on a minor release (as there may be
>>>> concurrently maintained minor releases) every 2 months".
>>>>
>>>> I don't know what is the hardest part of creating new
>>>> minor/maintenance releases. But if the problems are technical
>>>> (smoketesting, unit tests,
>>>
>>> old
>>>>
>>>> release script, anything else) I would be happy to do any task 

Re: About 2.7.4 Release

[Akira Ajisaka]

Ping

I too can help with the release process.

Now there are 0 blocker and 6 critical issues targeted for 2.7.4.
https://s.apache.org/HsIu

If there are critical/blocker issues that need to be fixed in 
branch-2.7, please set Target Version/s to 2.7.4. That way the issues 
can be found by the above query.


I'll check if there are conflicts among JIRA, git commit log, and the 
change logs.


Regards,
Akira

On 2017/04/18 15:40, Brahma Reddy Battula wrote:

Hi All

Any update on 2.7.4 ..?  Gentle Remainder!! Let me know anything I can help on 
this..



Regards
Brahma Reddy Battula

-Original Message-
From: Andrew Wang [mailto:andrew.w...@cloudera.com]
Sent: 08 March 2017 04:22
To: Sangjin Lee
Cc: Marton Elek; Hadoop Common; yarn-dev@hadoop.apache.org; Hdfs-dev; 
mapreduce-...@hadoop.apache.org
Subject: Re: About 2.7.4 Release

Our release steps are documented on the wiki:

2.6/2.7:

https://wiki.apache.org/hadoop/HowToReleasePreDSBCR

2.8+:
https://wiki.apache.org/hadoop/HowToRelease

I think given the push toward 2.8 and 3.0, there's less interest in 
streamlining the 2.6 and 2.7 release processes. CHANGES.txt is the biggest 
pain, and that's fixed in 2.8+.

Current pain points for 2.8+ include:

# fixing up JIRA versions and the release notes, though I somewhat addressed 
this with the versions script for 3.x # making and staging an RC and sending 
the vote email still requires a lot of manual steps # publishing the release is 
also quite manual

I think the RC issues can be attacked with enough scripting. Steve had an ant 
file that automated a lot of this for slider. I think it'd be nice to have a 
nightly Jenkins job that builds an RC, since I've spent a day or two for each 
3.x alpha fixing build issues.

Publishing can be attacked via a mix of scripting and revamping the darned 
website. Forrest is pretty bad compared to the newer static site generators out 
there (e.g. need to write XML instead of markdown, it's hard to review a 
staging site because of all the absolute links, hard to customize, did I 
mention XML?), and the look and feel of the site is from the 00s. We don't 
actually have that much site content, so it should be possible to migrate to a 
new system.

On Tue, Mar 7, 2017 at 9:13 AM, Sangjin Lee <sj...@apache.org> wrote:


I don't think there should be any linkage between releasing 2.8.0 and
2.7.4. If we have a volunteer for releasing 2.7.4, we should go full
speed ahead. We still need a volunteer from a PMC member or a
committer as some tasks may require certain privileges, but I don't
think it precludes working with others to close down the release.

I for one would like to see more frequent releases, and being able to
automate release steps more would go a long way.

On Tue, Mar 7, 2017 at 2:16 AM, Marton Elek <me...@hortonworks.com> wrote:


Is there any reason to wait for 2.8 with 2.7.4?

Unfortunately the previous  thread about release cadence has been
ended without final decision. But if I understood well, there was
more or less

an

agreement about that it would be great to achieve more frequent
releases, if possible (with or without written rules and EOL policy).

I personally prefer to be more closer to the scheduling part of the
proposal:

"A minor release on the latest major line should be every 6 months,
and a maintenance release on a minor release (as there may be
concurrently maintained minor releases) every 2 months".

I don't know what is the hardest part of creating new
minor/maintenance releases. But if the problems are technical
(smoketesting, unit tests,

old

release script, anything else) I would be happy to do any task for
new maintenance releases (or more frequent releases).

Regards,
Marton



From: Akira Ajisaka <aajis...@apache.org>
Sent: Tuesday, March 07, 2017 7:34 AM
To: Brahma Reddy Battula; Hadoop Common; yarn-dev@hadoop.apache.org;
Hdfs-dev; mapreduce-...@hadoop.apache.org
Subject: Re: About 2.7.4 Release

Probably 2.8.0 will be released soon.
https://issues.apache.org/jira/browse/HADOOP-13866?
focusedCommentId=15898379=com.atlassian.jira.
plugin.system.issuetabpanels:comment-tabpanel#comment-15898379

I'm thinking 2.7.4 release process starts after 2.8.0 release, so
2.7.4 will be released in April or May. (hopefully)

Thoughts?

Regards,
Akira

On 2017/03/01 21:01, Brahma Reddy Battula wrote:

Hi All

It has been six months for branch-2.7 release.. is there any near
plan

for 2.7.4..?



Thanks
Brahma Reddy Battula





- To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org




- To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: commo

RE: About 2.7.4 Release

[Brahma Reddy Battula]

Hi All

Any update on 2.7.4 ..?  Gentle Remainder!! Let me know anything I can help on 
this..



Regards
Brahma Reddy Battula

-Original Message-
From: Andrew Wang [mailto:andrew.w...@cloudera.com] 
Sent: 08 March 2017 04:22
To: Sangjin Lee
Cc: Marton Elek; Hadoop Common; yarn-dev@hadoop.apache.org; Hdfs-dev; 
mapreduce-...@hadoop.apache.org
Subject: Re: About 2.7.4 Release

Our release steps are documented on the wiki:

2.6/2.7:

https://wiki.apache.org/hadoop/HowToReleasePreDSBCR

2.8+:
https://wiki.apache.org/hadoop/HowToRelease

I think given the push toward 2.8 and 3.0, there's less interest in 
streamlining the 2.6 and 2.7 release processes. CHANGES.txt is the biggest 
pain, and that's fixed in 2.8+.

Current pain points for 2.8+ include:

# fixing up JIRA versions and the release notes, though I somewhat addressed 
this with the versions script for 3.x # making and staging an RC and sending 
the vote email still requires a lot of manual steps # publishing the release is 
also quite manual

I think the RC issues can be attacked with enough scripting. Steve had an ant 
file that automated a lot of this for slider. I think it'd be nice to have a 
nightly Jenkins job that builds an RC, since I've spent a day or two for each 
3.x alpha fixing build issues.

Publishing can be attacked via a mix of scripting and revamping the darned 
website. Forrest is pretty bad compared to the newer static site generators out 
there (e.g. need to write XML instead of markdown, it's hard to review a 
staging site because of all the absolute links, hard to customize, did I 
mention XML?), and the look and feel of the site is from the 00s. We don't 
actually have that much site content, so it should be possible to migrate to a 
new system.

On Tue, Mar 7, 2017 at 9:13 AM, Sangjin Lee <sj...@apache.org> wrote:

> I don't think there should be any linkage between releasing 2.8.0 and 
> 2.7.4. If we have a volunteer for releasing 2.7.4, we should go full 
> speed ahead. We still need a volunteer from a PMC member or a 
> committer as some tasks may require certain privileges, but I don't 
> think it precludes working with others to close down the release.
>
> I for one would like to see more frequent releases, and being able to 
> automate release steps more would go a long way.
>
> On Tue, Mar 7, 2017 at 2:16 AM, Marton Elek <me...@hortonworks.com> wrote:
>
> > Is there any reason to wait for 2.8 with 2.7.4?
> >
> > Unfortunately the previous  thread about release cadence has been 
> > ended without final decision. But if I understood well, there was 
> > more or less
> an
> > agreement about that it would be great to achieve more frequent 
> > releases, if possible (with or without written rules and EOL policy).
> >
> > I personally prefer to be more closer to the scheduling part of the
> > proposal:
> >
> > "A minor release on the latest major line should be every 6 months, 
> > and a maintenance release on a minor release (as there may be 
> > concurrently maintained minor releases) every 2 months".
> >
> > I don't know what is the hardest part of creating new 
> > minor/maintenance releases. But if the problems are technical 
> > (smoketesting, unit tests,
> old
> > release script, anything else) I would be happy to do any task for 
> > new maintenance releases (or more frequent releases).
> >
> > Regards,
> > Marton
> >
> >
> > 
> > From: Akira Ajisaka <aajis...@apache.org>
> > Sent: Tuesday, March 07, 2017 7:34 AM
> > To: Brahma Reddy Battula; Hadoop Common; yarn-dev@hadoop.apache.org; 
> > Hdfs-dev; mapreduce-...@hadoop.apache.org
> > Subject: Re: About 2.7.4 Release
> >
> > Probably 2.8.0 will be released soon.
> > https://issues.apache.org/jira/browse/HADOOP-13866?
> > focusedCommentId=15898379=com.atlassian.jira.
> > plugin.system.issuetabpanels:comment-tabpanel#comment-15898379
> >
> > I'm thinking 2.7.4 release process starts after 2.8.0 release, so 
> > 2.7.4 will be released in April or May. (hopefully)
> >
> > Thoughts?
> >
> > Regards,
> > Akira
> >
> > On 2017/03/01 21:01, Brahma Reddy Battula wrote:
> > > Hi All
> > >
> > > It has been six months for branch-2.7 release.. is there any near 
> > > plan
> > for 2.7.4..?
> > >
> > >
> > > Thanks
> > > Brahma Reddy Battula
> > >
> > >
> >
> > 
> > - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> > For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
> >
> >
> >
> > 
> > - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> > For additional commands, e-mail: common-dev-h...@hadoop.apache.org
> >
> >
>

Re: About 2.7.4 Release

[Allen Wittenauer]

> On Mar 8, 2017, at 1:54 PM, Allen Wittenauer  
> wrote:
> 
>   This is already possible:
>   * don’t use —asfrelease
>   * use —sign, —native, and, if appropriate for your platform, 
> —docker and —dockercache


Oh yeah, I forgot about this:

https://effectivemachines.com/2016/08/16/building-your-own-apache-hadoop-distribution/



-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: About 2.7.4 Release

[Allen Wittenauer]

> On Mar 8, 2017, at 10:55 AM, Marton Elek  wrote:
> 
> I think the main point here is the testing of the release script, not the 
> creation of the official release.

… except the Hadoop PMC was doing exactly this from 2.3.0 up until 
recently. Which means we have a few years worth of releases that are 
effectively untrustworthy despite being signed.  One of the (many) reasons I 
rewrote the release process was to get Hadoop back in line with ASF policy.  
Given the massive turn over in committers, I don’t want us to repeat the same 
mistakes (like we usually do).

> I think there should be an option to configure the release tool to use a 
> forked github repo and/or a private playground nexus instead of official 
> apache repos. In this case it would be easy to test regularly the tool, even 
> by a non-committer (or even from Jenkins). But it would be just a smoketest 
> of the release script…

This is already possible:
* don’t use —asfrelease
* use —sign, —native, and, if appropriate for your platform, 
—docker and —dockercache


-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: About 2.7.4 Release

[Marton Elek]

I think the main point here is the testing of the release script, not the 
creation of the official release.

I think there should be an option to configure the release tool to use a forked 
github repo and/or a private playground nexus instead of official apache repos. 
In this case it would be easy to test regularly the tool, even by a 
non-committer (or even from Jenkins). But it would be just a smoketest of the 
release script...

Marton
   

From: Allen Wittenauer <a...@effectivemachines.com>
Sent: Wednesday, March 08, 2017 2:24 AM
To: Andrew Wang
Cc: Hadoop Common; yarn-dev@hadoop.apache.org; Hdfs-dev; 
mapreduce-...@hadoop.apache.org
Subject: Re: About 2.7.4 Release

> On Mar 7, 2017, at 2:51 PM, Andrew Wang <andrew.w...@cloudera.com> wrote:
> I think it'd be nice to
> have a nightly Jenkins job that builds an RC,

Just a reminder that any such build cannot be used for an actual 
release:

http://www.apache.org/legal/release-policy.html#owned-controlled-hardware



-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org



-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: About 2.7.4 Release

[Marton Elek]

Thank you very much for your feedback. I am opening the following JIRAs:

1. Create dev-support scripts to do the bulk jira updates required by the 
releases (check remaining jiras, update fix versions, etc.)

2. Create a 'wizzard' like script which guide through the release process (all 
the steps from the wiki pages, not just a build. But it may be an extension of 
the existing script):

  Goals:
  * It would work even without the apache infrastructure: with custom 
configuration (forked repositories/alternative nexus), it would be possible to 
test the scripts even by a non-commiter.  
  * every step which could be automated should be scripted (create git 
branches, build,...). if something could be not automated there an explanation 
could be printed out, and wait for confirmation
  * Before dangerous steps (eg. bulk jira update) we can ask for confirmation 
and explain what will be happened (eg. the following jira items will be 
changed: ) 
  * The run should be idempontent (and there should be an option to continue 
the release from any steps).   

3. Migrate the forrest based home page to a use a modern static site generator.

  Goals: * existing links should work (or at least redirected)
 * It should be easy to add more content required by a release 
automatically

4. It's not about the release, but I think the current maven site theme also 
could be updated to use a (more modern) theme, which could be similar to the 
main site from step 3.

Let me know if you have any other suggestion for actionable items. Or comment 
the Jiras if you have more specific requirements.

Marton

ps: Vinod, I will contact with you, soon.




From: Vinod Kumar Vavilapalli <vino...@apache.org>
Sent: Tuesday, March 07, 2017 11:58 PM
To: Sangjin Lee
Cc: Marton Elek; common-...@hadoop.apache.org; yarn-dev@hadoop.apache.org; 
Hdfs-dev; mapreduce-...@hadoop.apache.org
Subject: Re: About 2.7.4 Release

I was planning to take this up, celebrating my return from my paternity leave 
of absence for quite a while.

Marton, let me know if you do want to take this up instead and we can work 
together.

Thanks
+Vinod

> On Mar 7, 2017, at 9:13 AM, Sangjin Lee <sj...@apache.org> wrote:
>
> If we have a volunteer for releasing 2.7.4, we should go full speed
> ahead. We still need a volunteer from a PMC member or a committer as some
> tasks may require certain privileges, but I don't think it precludes
> working with others to close down the release.


-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: About 2.7.4 Release

[Allen Wittenauer]

> On Mar 7, 2017, at 2:51 PM, Andrew Wang  wrote:
> I think it'd be nice to
> have a nightly Jenkins job that builds an RC,

Just a reminder that any such build cannot be used for an actual 
release:

http://www.apache.org/legal/release-policy.html#owned-controlled-hardware



-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: About 2.7.4 Release

[Vinod Kumar Vavilapalli]

I was planning to take this up, celebrating my return from my paternity leave 
of absence for quite a while.

Marton, let me know if you do want to take this up instead and we can work 
together.

Thanks
+Vinod

> On Mar 7, 2017, at 9:13 AM, Sangjin Lee  wrote:
> 
> If we have a volunteer for releasing 2.7.4, we should go full speed
> ahead. We still need a volunteer from a PMC member or a committer as some
> tasks may require certain privileges, but I don't think it precludes
> working with others to close down the release.

Re: About 2.7.4 Release

[Andrew Wang]

Our release steps are documented on the wiki:

2.6/2.7:

https://wiki.apache.org/hadoop/HowToReleasePreDSBCR

2.8+:
https://wiki.apache.org/hadoop/HowToRelease

I think given the push toward 2.8 and 3.0, there's less interest in
streamlining the 2.6 and 2.7 release processes. CHANGES.txt is the biggest
pain, and that's fixed in 2.8+.

Current pain points for 2.8+ include:

# fixing up JIRA versions and the release notes, though I somewhat
addressed this with the versions script for 3.x
# making and staging an RC and sending the vote email still requires a lot
of manual steps
# publishing the release is also quite manual

I think the RC issues can be attacked with enough scripting. Steve had an
ant file that automated a lot of this for slider. I think it'd be nice to
have a nightly Jenkins job that builds an RC, since I've spent a day or two
for each 3.x alpha fixing build issues.

Publishing can be attacked via a mix of scripting and revamping the darned
website. Forrest is pretty bad compared to the newer static site generators
out there (e.g. need to write XML instead of markdown, it's hard to review
a staging site because of all the absolute links, hard to customize, did I
mention XML?), and the look and feel of the site is from the 00s. We don't
actually have that much site content, so it should be possible to migrate
to a new system.

On Tue, Mar 7, 2017 at 9:13 AM, Sangjin Lee <sj...@apache.org> wrote:

> I don't think there should be any linkage between releasing 2.8.0 and
> 2.7.4. If we have a volunteer for releasing 2.7.4, we should go full speed
> ahead. We still need a volunteer from a PMC member or a committer as some
> tasks may require certain privileges, but I don't think it precludes
> working with others to close down the release.
>
> I for one would like to see more frequent releases, and being able to
> automate release steps more would go a long way.
>
> On Tue, Mar 7, 2017 at 2:16 AM, Marton Elek <me...@hortonworks.com> wrote:
>
> > Is there any reason to wait for 2.8 with 2.7.4?
> >
> > Unfortunately the previous  thread about release cadence has been ended
> > without final decision. But if I understood well, there was more or less
> an
> > agreement about that it would be great to achieve more frequent releases,
> > if possible (with or without written rules and EOL policy).
> >
> > I personally prefer to be more closer to the scheduling part of the
> > proposal:
> >
> > "A minor release on the latest major line should be every 6 months, and a
> > maintenance release on a minor release (as there may be concurrently
> > maintained minor releases) every 2 months".
> >
> > I don't know what is the hardest part of creating new minor/maintenance
> > releases. But if the problems are technical (smoketesting, unit tests,
> old
> > release script, anything else) I would be happy to do any task for new
> > maintenance releases (or more frequent releases).
> >
> > Regards,
> > Marton
> >
> >
> > 
> > From: Akira Ajisaka <aajis...@apache.org>
> > Sent: Tuesday, March 07, 2017 7:34 AM
> > To: Brahma Reddy Battula; Hadoop Common; yarn-dev@hadoop.apache.org;
> > Hdfs-dev; mapreduce-...@hadoop.apache.org
> > Subject: Re: About 2.7.4 Release
> >
> > Probably 2.8.0 will be released soon.
> > https://issues.apache.org/jira/browse/HADOOP-13866?
> > focusedCommentId=15898379=com.atlassian.jira.
> > plugin.system.issuetabpanels:comment-tabpanel#comment-15898379
> >
> > I'm thinking 2.7.4 release process starts after 2.8.0 release,
> > so 2.7.4 will be released in April or May. (hopefully)
> >
> > Thoughts?
> >
> > Regards,
> > Akira
> >
> > On 2017/03/01 21:01, Brahma Reddy Battula wrote:
> > > Hi All
> > >
> > > It has been six months for branch-2.7 release.. is there any near plan
> > for 2.7.4..?
> > >
> > >
> > > Thanks
> > > Brahma Reddy Battula
> > >
> > >
> >
> > -
> > To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> > For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
> >
> >
> >
> > -
> > To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> > For additional commands, e-mail: common-dev-h...@hadoop.apache.org
> >
> >
>

Re: About 2.7.4 Release

[Sangjin Lee]

I don't think there should be any linkage between releasing 2.8.0 and
2.7.4. If we have a volunteer for releasing 2.7.4, we should go full speed
ahead. We still need a volunteer from a PMC member or a committer as some
tasks may require certain privileges, but I don't think it precludes
working with others to close down the release.

I for one would like to see more frequent releases, and being able to
automate release steps more would go a long way.

On Tue, Mar 7, 2017 at 2:16 AM, Marton Elek <me...@hortonworks.com> wrote:

> Is there any reason to wait for 2.8 with 2.7.4?
>
> Unfortunately the previous  thread about release cadence has been ended
> without final decision. But if I understood well, there was more or less an
> agreement about that it would be great to achieve more frequent releases,
> if possible (with or without written rules and EOL policy).
>
> I personally prefer to be more closer to the scheduling part of the
> proposal:
>
> "A minor release on the latest major line should be every 6 months, and a
> maintenance release on a minor release (as there may be concurrently
> maintained minor releases) every 2 months".
>
> I don't know what is the hardest part of creating new minor/maintenance
> releases. But if the problems are technical (smoketesting, unit tests, old
> release script, anything else) I would be happy to do any task for new
> maintenance releases (or more frequent releases).
>
> Regards,
> Marton
>
>
> 
> From: Akira Ajisaka <aajis...@apache.org>
> Sent: Tuesday, March 07, 2017 7:34 AM
> To: Brahma Reddy Battula; Hadoop Common; yarn-dev@hadoop.apache.org;
> Hdfs-dev; mapreduce-...@hadoop.apache.org
> Subject: Re: About 2.7.4 Release
>
> Probably 2.8.0 will be released soon.
> https://issues.apache.org/jira/browse/HADOOP-13866?
> focusedCommentId=15898379=com.atlassian.jira.
> plugin.system.issuetabpanels:comment-tabpanel#comment-15898379
>
> I'm thinking 2.7.4 release process starts after 2.8.0 release,
> so 2.7.4 will be released in April or May. (hopefully)
>
> Thoughts?
>
> Regards,
> Akira
>
> On 2017/03/01 21:01, Brahma Reddy Battula wrote:
> > Hi All
> >
> > It has been six months for branch-2.7 release.. is there any near plan
> for 2.7.4..?
> >
> >
> > Thanks
> > Brahma Reddy Battula
> >
> >
>
> -
> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
>
>
>
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>
>

Re: About 2.7.4 Release

[Marton Elek]

Is there any reason to wait for 2.8 with 2.7.4?

Unfortunately the previous  thread about release cadence has been ended without 
final decision. But if I understood well, there was more or less an agreement 
about that it would be great to achieve more frequent releases, if possible 
(with or without written rules and EOL policy).

I personally prefer to be more closer to the scheduling part of the proposal:

"A minor release on the latest major line should be every 6 months, and a 
maintenance release on a minor release (as there may be concurrently
maintained minor releases) every 2 months".

I don't know what is the hardest part of creating new minor/maintenance 
releases. But if the problems are technical (smoketesting, unit tests, old 
release script, anything else) I would be happy to do any task for new 
maintenance releases (or more frequent releases).

Regards,
Marton

 

From: Akira Ajisaka <aajis...@apache.org>
Sent: Tuesday, March 07, 2017 7:34 AM
To: Brahma Reddy Battula; Hadoop Common; yarn-dev@hadoop.apache.org; Hdfs-dev; 
mapreduce-...@hadoop.apache.org
Subject: Re: About 2.7.4 Release

Probably 2.8.0 will be released soon.
https://issues.apache.org/jira/browse/HADOOP-13866?focusedCommentId=15898379=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15898379

I'm thinking 2.7.4 release process starts after 2.8.0 release,
so 2.7.4 will be released in April or May. (hopefully)

Thoughts?

Regards,
Akira

On 2017/03/01 21:01, Brahma Reddy Battula wrote:
> Hi All
>
> It has been six months for branch-2.7 release.. is there any near plan for 
> 2.7.4..?
>
>
> Thanks
> Brahma Reddy Battula
>
>

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org



-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: About 2.7.4 Release

[Akira Ajisaka]

Probably 2.8.0 will be released soon.
https://issues.apache.org/jira/browse/HADOOP-13866?focusedCommentId=15898379=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15898379

I'm thinking 2.7.4 release process starts after 2.8.0 release,
so 2.7.4 will be released in April or May. (hopefully)

Thoughts?

Regards,
Akira

On 2017/03/01 21:01, Brahma Reddy Battula wrote:

Hi All

It has been six months for branch-2.7 release.. is there any near plan for 
2.7.4..?


Thanks
Brahma Reddy Battula




-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

About 2.7.4 Release

[Brahma Reddy Battula]

Hi All

It has been six months for branch-2.7 release.. is there any near plan for 
2.7.4..?


Thanks
Brahma Reddy Battula