[jira] [Created] (HADOOP-18127) Backport HADOOP-13055 into branch-2.10

[Konstantin Shvachko (Jira)]

Konstantin Shvachko created HADOOP-18127:


 Summary: Backport HADOOP-13055 into branch-2.10
 Key: HADOOP-18127
 URL: https://issues.apache.org/jira/browse/HADOOP-18127
 Project: Hadoop Common
  Issue Type: Sub-task
  Components: viewfs
Reporter: Konstantin Shvachko


HADOOP-13055 introduce linkMergeSlash and linkFallback for ViewFileSystem. 
Would be good to backport it to branch-2.10



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Resolved] (HADOOP-17999) No-op implementation of setWriteChecksum and setVerifyChecksum in ViewFileSystem

[Konstantin Shvachko (Jira)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-17999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Konstantin Shvachko resolved HADOOP-17999.
--
Fix Version/s: 3.4.0
   2.10.2
   3.2.4
   3.3.3
 Hadoop Flags: Reviewed
   Resolution: Fixed

I just committed this to all supported branches. Thank you [~abhishekd].

> No-op implementation of setWriteChecksum and setVerifyChecksum in 
> ViewFileSystem
> 
>
> Key: HADOOP-17999
> URL: https://issues.apache.org/jira/browse/HADOOP-17999
> Project: Hadoop Common
>  Issue Type: Bug
>Reporter: Abhishek Das
>Assignee: Abhishek Das
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.4.0, 2.10.2, 3.2.4, 3.3.3
>
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> Currently setVerifyChecksum and setWriteChecksum initializes all target file 
> systems which causes delay in hadoop shell copy commands such as get, put, 
> copyFromLocal etc.
> This also eventually causes OOM.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Resolved] (HADOOP-16532) Fix TestViewFsTrash to use the correct homeDir.

[Konstantin Shvachko (Jira)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-16532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Konstantin Shvachko resolved HADOOP-16532.
--
Fix Version/s: 3.2.4
   3.3.2
   2.10.2
   3.4.0
 Hadoop Flags: Reviewed
   Resolution: Fixed

I just committed this. Thank you [~xinglin].

> Fix TestViewFsTrash to use the correct homeDir.
> ---
>
> Key: HADOOP-16532
> URL: https://issues.apache.org/jira/browse/HADOOP-16532
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: fs, test
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Assignee: Xing Lin
>Priority: Minor
>  Labels: pull-request-available
> Fix For: 3.4.0, 2.10.2, 3.3.2, 3.2.4
>
>  Time Spent: 2h
>  Remaining Estimate: 0h
>
> the test {{TestViewFsTrash}} uses the .Trash directory under the current 
> user's home dir so
> * fails in test setups which block writing to it (jenkins)
> * fails when users have real trash in there
> * may fail if there are parallel test runs.
> the home dir should be under some test path of the build.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Resolved] (HADOOP-17819) Add extensions to ProtobufRpcEngine RequestHeaderProto

[Konstantin Shvachko (Jira)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-17819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Konstantin Shvachko resolved HADOOP-17819.
--
Fix Version/s: 3.2.3
   2.10.2
   3.4.0
 Hadoop Flags: Reviewed
 Assignee: Hector Sandoval Chaverri
   Resolution: Fixed

I just committed this. Thank you [~hchaverri]

> Add extensions to ProtobufRpcEngine RequestHeaderProto
> --
>
> Key: HADOOP-17819
> URL: https://issues.apache.org/jira/browse/HADOOP-17819
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: common
>Reporter: Hector Sandoval Chaverri
>Assignee: Hector Sandoval Chaverri
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.4.0, 2.10.2, 3.2.3
>
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> The header used in ProtobufRpcEngine messages doesn't allow for new 
> properties to be added by child classes. We can add a range of extensions 
> that can be useful for proto classes that need to extend RequestHeaderProto.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Resolved] (HADOOP-17028) ViewFS should initialize target filesystems lazily

[Konstantin Shvachko (Jira)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-17028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Konstantin Shvachko resolved HADOOP-17028.
--
Fix Version/s: 3.3.2
   3.2.3
   2.10.2
   3.4.0
   3.1.5
 Hadoop Flags: Reviewed
   Resolution: Fixed

> ViewFS should initialize target filesystems lazily
> --
>
> Key: HADOOP-17028
> URL: https://issues.apache.org/jira/browse/HADOOP-17028
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: client-mounts, fs, viewfs
>Affects Versions: 3.2.1
>Reporter: Uma Maheswara Rao G
>Assignee: Abhishek Das
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.1.5, 3.4.0, 2.10.2, 3.2.3, 3.3.2
>
>  Time Spent: 7h 50m
>  Remaining Estimate: 0h
>
> Currently viewFS initialize all configured target filesystems when 
> viewfs#init itself.
> Some target file system initialization involve creating heavy objects and 
> proxy connections. Ex: DistributedFileSystem#initialize will create DFSClient 
> object which will create proxy connections to NN etc.
> For example: if ViewFS configured with 10 target fs with hdfs uri and 2 
> targets with s3a.
> If one of the client only work with s3a target, But ViewFS will initialize 
> all targets irrespective of what clients interested to work with. That means, 
> here client will create 10 DFS initializations and 2 s3a initializations. Its 
> unnecessary to have DFS initialization here. So, it will be a good idea to 
> initialize the target fs only when first time usage call come to particular 
> target fs scheme. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-17501) Fix logging typo in ShutdownHookManager

[Konstantin Shvachko (Jira)]

Konstantin Shvachko created HADOOP-17501:


 Summary: Fix logging typo in ShutdownHookManager
 Key: HADOOP-17501
 URL: https://issues.apache.org/jira/browse/HADOOP-17501
 Project: Hadoop Common
  Issue Type: Improvement
  Components: common
Reporter: Konstantin Shvachko


Three log messages in {{ShutdownHookManager}} have a typo, saying 
"ShutdownHookManger". SHould be "ShutdownHookManager"



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-17477) [SBN read] Implement msync() for ViewFS

[Konstantin Shvachko (Jira)]

Konstantin Shvachko created HADOOP-17477:


 Summary: [SBN read] Implement msync() for ViewFS
 Key: HADOOP-17477
 URL: https://issues.apache.org/jira/browse/HADOOP-17477
 Project: Hadoop Common
  Issue Type: Improvement
  Components: viewfs
Reporter: Konstantin Shvachko


We should implement {{FileSystem.mscyn()}} for {{ViewFS}} and 
{{ViewFileSystem}}.
It should call msync() on all volumes for which observer reads are enabled and 
{{dfs.namenode.state.context.enabled = true}}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: Regarding HDFS-15567. HDFS should expose msync() API to allow downstream applications call it explicitly

[Konstantin Shvachko]

Hi Steve,

I created HDFS-15751 <https://issues.apache.org/jira/browse/HDFS-15751> for
documenting msync API.
Would appreciate your suggestions.

Stay safe,
--Konstantin

On Mon, Dec 21, 2020 at 5:19 AM Steve Loughran  wrote:

>
>
> On Fri, 18 Dec 2020 at 23:29, Konstantin Shvachko 
> wrote:
>
>> Hey Steve,
>>
>> Thanks for the references. I was reading but still need to understand how
>> exactly this applies to msync.
>>
>
> mainly: pull it up and it becomes part of the broader API, so needs to be
> specified in a way which can be understood by users and for implementors of
> others stores: to give their own stores the same semantics.
>
> What does the HDFS one do?
>
>
>
>> Will come up with a plan and post it on a new jira.
>> Will make sure to create it under HADOOP and ping Hadoop Common list for
>> visibility.
>>
>>
> thanks
>
>
>> You are right about ViewFS. The impl should make sure it calls msync() on
>> all mount points that enabled observer reads.
>>
>>
> That's the kind of issue this process aims to resolve. Another is to
> identify where we have HDFS-layer "quirks" and at least document them (e.g.
> how hdfs streams are thread safe, rename isn't Posix, ...) and list what we
> know breaks if you don't re-implement
>

Re: Regarding HDFS-15567. HDFS should expose msync() API to allow downstream applications call it explicitly

[Konstantin Shvachko]

Hey Steve,

Thanks for the references. I was reading but still need to understand how
exactly this applies to msync.
Will come up with a plan and post it on a new jira.
Will make sure to create it under HADOOP and ping Hadoop Common list for
visibility.

You are right about ViewFS. The impl should make sure it calls msync() on
all mount points that enabled observer reads.

--Konst

On Tue, Dec 15, 2020 at 11:15 AM Steve Loughran  wrote:

>
>
> On Sun, 13 Dec 2020 at 21:08, Konstantin Shvachko 
> wrote:
>
>> Hi Steve,
>>
>> I am not sure I fully understand what is broken here. It is not an
>> incompatible change, right?
>>
>
> The issue is that the FileSystem/FileContext APIs are something we have to
> maintain ~forever, so every API change needs to be
>
> - something we are happy with being there for the life of the class
> - defined strictly enough that people implementing other filesystems can
> re-implement without having to reverse-engineer HDFS and then conclude
> "that is what they meant to do". That's with a bit of
> - and with an AbstractFileSystemContractTest for implementors.
>
> That's it: define, specify, add a contract test rather than just something
> for HDFS.
>
>
>
>> Could you please explain what you think the process is.
>> Would be best if you could share a link to a document describing it.
>> I would be glad to follow up with tests and documentation that are needed.
>>
>>
> ideally,
> hadoop-common-project/hadoop-common/src/site/markdown/filesystem/extending.md
>
> Pulling up something from hdfs is different from saying "hey, new rename
> API!", but it's still time to actually define what it does so that not only
> can other people like me reimplement it, but to actually define it well
> enough that we can all see when there's a regression.
>
> Equally important: is there a way to test that it works?
>
> We've been using hasPathCapabilities() to probe for an FS having a given
> feature down a path; the idea is to let code check upfront for a feature
> before having to call it and catching an exception,
>
> We can add that for an API even if it has shipped already. For example,
> here is a PR to do exactly that for Syncable
> https://github.com/apache/hadoop/pull/2102
>
>
>
>> As you can see I proposed multiple solutions to the problem in the jira.
>> Seemed nobody was objecting, so I chose one and explained why.
>> I believe we call it lazy consensus.
>>
>
> I'm happy with lazy consensus, but can you involve more people? In
> particular. i was filed in an HDFS JIRA so it didn't surface in
> hadoop-common.
>
> If you'd done a HADOOP- JIRA "pull up msync into FileSystem API" or even
> just a note to hadoop-common saying "we need to do this" that would have
> been enough to start a discussion.
>
>  As it is I only noticed after some rebasing with a "hang on a minute.
> here's a new method who's behaviour doesn't seem to have defined other than
> 'whatever hdfs does'". Which, if you've ever tried to work out what
> rename() does, you'll recognise as danger.
>
> Anyway, to finish off, have a look at the extending.md doc and just add a
> new method definition in the filesystem.md saying what it is meant to do.
>
> Now: what about viewfs? maybe: For all mounted fileystems which declare
> their support, call msync()? Or just "call it and swallow all exceptions?"
>
>
>> Stay safe,
>>
>
>
> yeah: )
>
>> --Konstantin
>>
>
>>>

Re: Regarding HDFS-15567. HDFS should expose msync() API to allow downstream applications call it explicitly

[Konstantin Shvachko]

Hey Xiaoqiao,

HDFS-14272 was committed to all branches up to 2.10. The jira versions were
not updated properly.
I'll ping Chen for an update. He committed it in May.

Stay safe,
--Konstantin

On Tue, Dec 15, 2020 at 1:21 AM Xiaoqiao He  wrote:

> Hi All,
>
>
>> I'm just curious why this is included in the 3.2.2 release? HDFS-15567 is
>> tagged with 3.2.3 and the corresponding HDFS-14272 on server side is tagged
>> with 3.3.0.
>
>
> Have checked the fix version tag, I found there are 8 issues which do not
> include branch-3.2.2 correctly or both branch-3.2.2 and branch-3.2.3
> missed. And have updated them manually. Please have a look. Thanks.
> HADOOP-15691
> HDFS-15464
> HDFS-15478
> HDFS-15567
> HDFS-15574
> HDFS-15583
> HDFS-15628
> YARN-10430
>
> Regards,
> - He Xiaoqiao
>
> On Mon, Dec 14, 2020 at 5:08 AM Konstantin Shvachko 
> wrote:
>
>> Hi Steve,
>>
>> I am not sure I fully understand what is broken here. It is not an
>> incompatible change, right?
>> Could you please explain what you think the process is.
>> Would be best if you could share a link to a document describing it.
>> I would be glad to follow up with tests and documentation that are needed.
>>
>> As you can see I proposed multiple solutions to the problem in the jira.
>> Seemed nobody was objecting, so I chose one and explained why.
>> I believe we call it lazy consensus.
>>
>> Stay safe,
>> --Konstantin
>>
>> On Sun, Dec 13, 2020 at 10:22 AM Chao Sun  wrote:
>>
>>> > This is an API where it'd be ok to have a no-op if not implemented,
>>> correct? Or is there an requirement like Syncable that specific
>>> guarantees
>>> are met？
>>>
>>> Yes I think it's ok to leave it as no-op for other non-HDFS FS impls: it
>>> is
>>> only used by HDFS standby reads so far.
>>>
>>>
>>>
>>> On Sun, Dec 13, 2020 at 4:58 AM Steve Loughran 
>>> wrote:
>>>
>>> > This isn't worth holding up the RC. We'd just add something to the
>>> > release notes "use with caution". And if we can get what the API does
>>> > defined in a way which works, it shouldn't need changing.
>>> >
>>> > (which reminds me, I do need to check that RC out, don't I?)
>>> >
>>> > On Sun, 13 Dec 2020 at 09:00, Xiaoqiao He 
>>> wrote:
>>> >
>>> >> Thanks Steve very much for your discussion here.
>>> >>
>>> >> Leave some comments inline. Will focus on this thread to wait for the
>>> >> final
>>> >> conclusion to decide if we should prepare another release candidate of
>>> >> 3.2.2.
>>> >> Thanks Steve and Chao again for your warm discussions.
>>> >>
>>> >> On Sat, Dec 12, 2020 at 7:18 PM Steve Loughran
>>> >> 
>>> >> wrote:
>>> >>
>>> >> > Maybe it's not in the release; it's certainly in the 3.2 branch.
>>> Will
>>> >> check
>>> >> > further. If it's in the release I was thinking of adding a warning
>>> in
>>> >> the
>>> >> > notes "unstable API"; stable if invoked from DFSClient
>>> >>
>>> >> On Fri, 11 Dec 2020 at 18:21, Chao Sun  wrote:
>>> >> >
>>> >> > > I'm just curious why this is included in the 3.2.2 release?
>>> >> HDFS-15567 is
>>> >> > > tagged with 3.2.3 and the corresponding HDFS-14272 on server side
>>> is
>>> >> > tagged
>>> >> > > with 3.3.0.
>>> >> >
>>> >>
>>> >> Just checked that HDFS-15567 has been involved in Hadoop-3.2.2 RC4.
>>> IIRC,
>>> >> I
>>> >> have cut branch-3.2.2 in early October, at that time branch-3.2.3 has
>>> >> created but source code not freeze completely because several blocked
>>> >> issues reported and code freeze has done about mid October. Some
>>> issues
>>> >> which are tagged with 3.2.3 has also been involved in 3.2.2 during
>>> >> that period, include HDFS-15567. I will check them later, and make
>>> sure
>>> >> that we have mark the correct tags.
>>> >>
>>> >>
>>> >> > >
>>> >> > > > If it goes into FS/FC, what does it do for a viewfs with >1
>>> mounted
>>> >> > HDFS?
>>> >

Re: Regarding HDFS-15567. HDFS should expose msync() API to allow downstream applications call it explicitly

[Konstantin Shvachko]

Hi Steve,

I am not sure I fully understand what is broken here. It is not an
incompatible change, right?
Could you please explain what you think the process is.
Would be best if you could share a link to a document describing it.
I would be glad to follow up with tests and documentation that are needed.

As you can see I proposed multiple solutions to the problem in the jira.
Seemed nobody was objecting, so I chose one and explained why.
I believe we call it lazy consensus.

Stay safe,
--Konstantin

On Sun, Dec 13, 2020 at 10:22 AM Chao Sun  wrote:

> > This is an API where it'd be ok to have a no-op if not implemented,
> correct? Or is there an requirement like Syncable that specific guarantees
> are met？
>
> Yes I think it's ok to leave it as no-op for other non-HDFS FS impls: it is
> only used by HDFS standby reads so far.
>
>
>
> On Sun, Dec 13, 2020 at 4:58 AM Steve Loughran 
> wrote:
>
> > This isn't worth holding up the RC. We'd just add something to the
> > release notes "use with caution". And if we can get what the API does
> > defined in a way which works, it shouldn't need changing.
> >
> > (which reminds me, I do need to check that RC out, don't I?)
> >
> > On Sun, 13 Dec 2020 at 09:00, Xiaoqiao He  wrote:
> >
> >> Thanks Steve very much for your discussion here.
> >>
> >> Leave some comments inline. Will focus on this thread to wait for the
> >> final
> >> conclusion to decide if we should prepare another release candidate of
> >> 3.2.2.
> >> Thanks Steve and Chao again for your warm discussions.
> >>
> >> On Sat, Dec 12, 2020 at 7:18 PM Steve Loughran
> >> 
> >> wrote:
> >>
> >> > Maybe it's not in the release; it's certainly in the 3.2 branch. Will
> >> check
> >> > further. If it's in the release I was thinking of adding a warning in
> >> the
> >> > notes "unstable API"; stable if invoked from DFSClient
> >>
> >> On Fri, 11 Dec 2020 at 18:21, Chao Sun  wrote:
> >> >
> >> > > I'm just curious why this is included in the 3.2.2 release?
> >> HDFS-15567 is
> >> > > tagged with 3.2.3 and the corresponding HDFS-14272 on server side is
> >> > tagged
> >> > > with 3.3.0.
> >> >
> >>
> >> Just checked that HDFS-15567 has been involved in Hadoop-3.2.2 RC4.
> IIRC,
> >> I
> >> have cut branch-3.2.2 in early October, at that time branch-3.2.3 has
> >> created but source code not freeze completely because several blocked
> >> issues reported and code freeze has done about mid October. Some issues
> >> which are tagged with 3.2.3 has also been involved in 3.2.2 during
> >> that period, include HDFS-15567. I will check them later, and make sure
> >> that we have mark the correct tags.
> >>
> >>
> >> > >
> >> > > > If it goes into FS/FC, what does it do for a viewfs with >1
> mounted
> >> > HDFS?
> >> > > Should it take path, msync(path) so that viewFS knows where to
> forward
> >> > it?
> >> > >
> >> > > The API shouldn't take any path - for viewFS I think it should call
> >> this
> >> > on
> >> > > all the child file systems. It might also need to handle the case
> >> where
> >> > > some downstream clusters support this capability while others don't.
> >> > >
> >> >
> >> > That's an extra bit of work for ViewFS then. It should probe for
> >> capability
> >> > and invoke as/when supported.
> >> >
> >> > >
> >> > > > Options
> >> > > 1. I roll HDFS-15567 back "please be follow process"
> >> > > 2. Someone does a followup patch with specification and contract
> test,
> >> > view
> >> > > FS. Add even more to the java
> >> > > 3. We do as per HADOOP-16898 into an MSyncable interface and then
> >> > > FileSystem & HDFS can implement. ViewFS and filterFS still need to
> >> pass
> >> > > through.
> >> > >
> >> > > I'm slightly in favor of the hasPathCapabilities approach and make
> >> this a
> >> > > mixin where FS impls can optionally support. Happy to hear what
> others
> >> > > think.
> >> > >
> >> >
> >> > Mixins are great when FC and FS can both implement; makes it easier to
> >> code
> >> > against either. All the filtering/aggregating FS's will have to
> >> implement
> >> > it, which means that presence of the interface doesn't guarantee
> >> support.
> >> >
> >> > This is an API where it'd be ok to have a no-op if not implemented,
> >> > correct? Or is there an requirement like Syncable that specific
> >> guarantees
> >> > are met?
> >> >
> >> > >
> >> > > Chao
> >> > >
> >> > >
> >> > > On Fri, Dec 11, 2020 at 9:00 AM Steve Loughran
> >> >  >> > > >
> >> > > wrote:
> >> > >
> >> > > > Silence from the  HDFS team
> >> > > >
> >> > > >
> >> > > > Hadoop 3.2.2 is in an RC; it has the new FS API call. I really
> don't
> >> > want
> >> > > > to veto the release just because someone pulled up a method
> without
> >> > doing
> >> > > > the due diligence.
> >> >
> >>
> >> Thanks Steve started this discussion here. I agree to roll back
> HDFS-15567
> >> if there are still some incompatible issues not resolved completely. And
> >> release will not be the blocked things here, I would like to prepare
> >> 

Re: [VOTE] Moving Ozone to a separated Apache project

[Konstantin Shvachko]

+1

Stay safe,
--Konstantin

On Thu, Sep 24, 2020 at 10:59 PM Elek, Marton  wrote:

> Hi all,
>
> Thank you for all the feedback and requests,
>
> As we discussed in the previous thread(s) [1], Ozone is proposed to be a
> separated Apache Top Level Project (TLP)
>
> The proposal with all the details, motivation and history is here:
>
>
> https://cwiki.apache.org/confluence/display/HADOOP/Ozone+Hadoop+subproject+to+Apache+TLP+proposal
>
> This voting runs for 7 days and will be concluded at 2nd of October, 6AM
> GMT.
>
> Thanks,
> Marton Elek
>
> [1]:
>
> https://lists.apache.org/thread.html/rc6c79463330b3e993e24a564c6817aca1d290f186a1206c43ff0436a%40%3Chdfs-dev.hadoop.apache.org%3E
>
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>
>

[jira] [Created] (HADOOP-17026) TestRefreshCallQueue is failing due to changed CallQueue constructor

[Konstantin Shvachko (Jira)]

Konstantin Shvachko created HADOOP-17026:


 Summary: TestRefreshCallQueue is failing due to changed CallQueue 
constructor
 Key: HADOOP-17026
 URL: https://issues.apache.org/jira/browse/HADOOP-17026
 Project: Hadoop Common
  Issue Type: Bug
  Components: test
Reporter: Konstantin Shvachko






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [DISCUSS] Making 2.10 the last minor 2.x release

[Konstantin Shvachko]

in
>>>   - Committers have an extra branch (5 vs 4 total branches) to commit
>>> patches to if they should go all the way back to 2.10.x
>>> - It is less necessary to move to 3.x
>>>
>>> So on the one hand you get added stability in fewer features being
>>> committed to 2.10.x, but then on the other you get fewer bug fixes being
>>> committed. In a perfect world, we wouldn't have to make this tradeoff.
>>> But
>>> we don't live in a perfect world and committers will make mistakes either
>>> because of lack of knowledge or simply because they made a mistake. If we
>>> have a branch-2, committers will forget, not know to, or choose not to
>>> (for
>>> whatever reason) commit valid bug fixes back all the way to branch-2.10.
>>> If
>>> we don't have a branch-2, committers who want their borderline risky
>>> feature in the 2.x line will err on the side of putting it into
>>> branch-2.10
>>> instead of proposing the creation of a branch-2. Clearly I have made
>>> quite
>>> a few assumptions here based on my own experiences, so I would like to
>>> hear
>>> if others have similar or opposing views.
>>>
>>> As far as 3.x goes, to me it seems like some of the reasoning for killing
>>> branch-2 is due to an effort to push the community towards 3.x. This is
>>> why
>>> I have added movement to 3.x as both a pro and a con. As a community
>>> trying
>>> to move forward, keeping as many companies on similar branches as
>>> possible
>>> is a good way to make sure the code is well-tested. However, from a
>>> stability point of view, moving to 3.x is still scary and being able to
>>> stay on 2.x until you are comfortable to move is very nice. The 2.10.0
>>> bridge release effort has been very good at making it possible for people
>>> to move from 2.x in 3.x, but the diff between 2.x and 3.x is so large
>>> that
>>> it is reasonable for companies to want to be extra cautious with 3.x due
>>> to
>>> potential performance degradation at large scale.
>>>
>>> A question I'm pondering is what happens when we move to Java 11 and
>>> someone is still on 2.x? If they want to backport HADOOP-15338
>>> <https://issues.apache.org/jira/browse/HADOOP-15338> for Java 11
>>> support to
>>> 2.x, surely not everyone is going to want that (at least not
>>> immediately).
>>> The 2.10 documentation states, "The JVM requirements will not change
>>> across
>>> point releases within the same minor release except if the JVM version
>>> under question becomes unsupported" [1], so this would warrant a 2.11
>>> release until Java 8 becomes unsupported (though one could argue that it
>>> is
>>> already unsupported since Oracle is no longer giving public Java 8
>>> update).
>>> If we don't keep branch-2 around now, would a Java 11 backport be the
>>> catalyst for a branch-2 revival?
>>>
>>> Not sure if this really leads to any sort of answer from me on whether or
>>> not we should keep branch-2 alive, but these are the things that I am
>>> weighing in my mind. For me, the bigger problem beyond having branch-2 or
>>> not is committers not being on the same page with where they should
>>> commit
>>> their patches.
>>>
>>> Eric
>>>
>>> [1]
>>>
>>> https://hadoop.apache.org/docs/r2.10.0/hadoop-project-dist/hadoop-common/Compatibility.html
>>> [2]
>>>
>>> https://hadoop.apache.org/docs/r3.0.0/hadoop-project-dist/hadoop-common/Compatibility.html
>>>
>>> On Tue, Nov 19, 2019 at 2:49 PM epa...@apache.org 
>>> wrote:
>>>
>>> > Hi Konstantin,
>>> >
>>> > Sure, I understand those concerns. On the other hand, I worry about the
>>> > stability of 2.10, since we will be on it for a couple of years at
>>> least.
>>> > I worry
>>> >  that some committers may want to put new features into a branch 2
>>> release,
>>> >  and without a branch-2, they will go directly into 2.10. Since we
>>> don't
>>> > always
>>> >  catch corner cases or performance problems for some time (usually not
>>> > until
>>> >  the release is deployed to a busy, 4-thousand node cluster), it may be
>>> > very
>>> >  difficult to back out those changes.
>>> >
>>> > It sounds like I'm in the minority here, so I'm not ni

Re: [DISCUSS] Making 2.10 the last minor 2.x release

[Konstantin Shvachko]

Hi Eric,

We had a long discussion on this list regarding making the 2.10 release the
last of branch-2 releases. We intended 2.10 as a bridge release between
Hadoop 2 and 3. We may have bug-fix releases or 2.10, but 2.11 is not in
the picture right now, and many people may object this idea.

I understand Jonathan's proposal as an attempt to
1. eliminate confusion which branches people should commit their back-ports
to
2. save engineering effort committing to more branches than necessary

"Branches are cheap" as our founder used to say. If we ever decide to
release 2.11 we can resurrect the branch.
Until then I am in favor of Jonathan's proposal +1.

Thanks,
--Konstantin


On Mon, Nov 18, 2019 at 10:41 AM Jonathan Hung  wrote:

> Thanks Eric for the comments - regarding your concerns, I feel the pros
> outweigh the cons. To me, the chances of patch releases on 2.10.x are much
> higher than a new 2.11 minor release. (There didn't seem to be many people
> outside of our company who expressed interest in getting new features to
> branch-2 prior to the 2.10.0 release.) Even now, a few weeks after 2.10.0
> release, there's 29 patches that have gone into branch-2 and 9 in
> branch-2.10, so it's already diverged quite a bit.
>
> In any case, we can always reverse this decision if we really need to, by
> recreating branch-2. But this proposal would reduce a lot of confusion IMO.
>
> Jonathan Hung
>
>
> On Fri, Nov 15, 2019 at 11:41 AM epa...@apache.org 
> wrote:
>
> > Thanks Jonathan for opening the discussion.
> >
> > I am not in favor of this proposal. 2.10 was very recently released, and
> > moving to 2.10 will take some time for the community. It seems premature
> to
> > make a decision at this point that there will never be a need for a 2.11
> > release.
> >
> > -Eric
> >
> >
> >  On Thursday, November 14, 2019, 8:51:59 PM CST, Jonathan Hung <
> > jyhung2...@gmail.com> wrote:
> >
> > Hi folks,
> >
> > Given the release of 2.10.0, and the fact that it's intended to be a
> bridge
> > release to Hadoop 3.x [1], I'm proposing we make 2.10.x the last minor
> > release line in branch-2. Currently, the main issue is that there's many
> > fixes going into branch-2 (the theoretical 2.11.0) that's not going into
> > branch-2.10 (which will become 2.10.1), so the fixes in branch-2 will
> > likely never see the light of day unless they are backported to
> > branch-2.10.
> >
> > To do this, I propose we:
> >
> >   - Delete branch-2.10
> >   - Rename branch-2 to branch-2.10
> >   - Set version in the new branch-2.10 to 2.10.1-SNAPSHOT
> >
> > This way we get all the current branch-2 fixes into the 2.10.x release
> > line. Then the commit chain will look like: trunk -> branch-3.2 ->
> > branch-3.1 -> branch-2.10 -> branch-2.9 -> branch-2.8
> >
> > Thoughts?
> >
> > Jonathan Hung
> >
> > [1]
> https://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg29479.html
> >
>

Re: [VOTE] Release Apache Hadoop 2.10.0 (RC1)

[Konstantin Shvachko]

+1 on RC1

- Verified signatures
- Verified maven artifacts on Nexus for sources
- Checked rat reports
- Checked documentation
- Checked packaging contents
- Built from sources on RHEL 7 box
- Ran unit tests for new HDFS features with Java 8

Thanks,
--Konstantin

On Tue, Oct 22, 2019 at 2:55 PM Jonathan Hung  wrote:

> Hi folks,
>
> This is the second release candidate for the first release of Apache Hadoop
> 2.10 line. It contains 362 fixes/improvements since 2.9 [1]. It includes
> features such as:
>
> - User-defined resource types
> - Native GPU support as a schedulable resource type
> - Consistent reads from standby node
> - Namenode port based selective encryption
> - Improvements related to rolling upgrade support from 2.x to 3.x
> - Cost based fair call queue
>
> The RC1 artifacts are at: http://home.apache.org/~jhung/hadoop-2.10.0-RC1/
>
> RC tag is release-2.10.0-RC1.
>
> The maven artifacts are hosted here:
> https://repository.apache.org/content/repositories/orgapachehadoop-1243/
>
> My public key is available here:
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
> The vote will run for 5 weekdays, until Tuesday, October 29 at 3:00 pm PDT.
>
> Thanks,
> Jonathan Hung
>
> [1]
>
> https://issues.apache.org/jira/issues/?jql=project%20in%20(HDFS%2C%20YARN%2C%20HADOOP%2C%20MAPREDUCE)%20AND%20resolution%20%3D%20Fixed%20AND%20fixVersion%20%3D%202.10.0%20AND%20fixVersion%20not%20in%20(2.9.2%2C%202.9.1%2C%202.9.0)
>

Re: [VOTE] Release Apache Hadoop 2.10.0 (RC0)

[Konstantin Shvachko]

+1 on RC0.
- Verified signatures
- Built from sources
- Ran unit tests for new features
- Checked artifacts on Nexus, made sure the sources are present.

Thanks
--Konstantin


On Wed, Oct 16, 2019 at 6:01 PM Jonathan Hung  wrote:

> Hi folks,
>
> This is the first release candidate for the first release of Apache Hadoop
> 2.10 line. It contains 361 fixes/improvements since 2.9 [1]. It includes
> features such as:
>
> - User-defined resource types
> - Native GPU support as a schedulable resource type
> - Consistent reads from standby node
> - Namenode port based selective encryption
> - Improvements related to rolling upgrade support from 2.x to 3.x
>
> The RC0 artifacts are at: http://home.apache.org/~jhung/hadoop-2.10.0-RC0/
>
> RC tag is release-2.10.0-RC0.
>
> The maven artifacts are hosted here:
> https://repository.apache.org/content/repositories/orgapachehadoop-1241/
>
> My public key is available here:
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
> The vote will run for 5 weekdays, until Wednesday, October 23 at 6:00 pm
> PDT.
>
> Thanks,
> Jonathan Hung
>
> [1]
>
> https://issues.apache.org/jira/issues/?jql=project%20in%20(HDFS%2C%20YARN%2C%20HADOOP%2C%20MAPREDUCE)%20AND%20resolution%20%3D%20Fixed%20AND%20fixVersion%20%3D%202.10.0%20AND%20fixVersion%20not%20in%20(2.9.2%2C%202.9.1%2C%202.9.0)
>

Re: [VOTE] Merge YARN-8200 to branch-2 and branch-3.0

[Konstantin Shvachko]

+1 for the merge.

We probably should not bother with branch-3.0 merge since it's been voted
EOL.

Thanks,
--Konstantin

On Thu, Aug 22, 2019 at 4:43 PM Jonathan Hung  wrote:

> Hi folks,
>
> As per [1], starting a vote to merge YARN-8200 (and YARN-8200.branch3)
> feature branch to branch-2 (and branch-3.0).
>
> Vote runs for 7 days, to Thursday, Aug 29 5:00PM PDT.
>
> Thanks.
>
> [1]
>
> http://mail-archives.apache.org/mod_mbox/hadoop-yarn-dev/201908.mbox/%3cCAHzWLgcX7f5Tr3q=csrqgysvpdf7mh-iu17femgx89dhr+1...@mail.gmail.com%3e
>
> Jonathan Hung
>

Re: [VOTE] Mark 2.6, 2.7, 3.0 release lines EOL

[Konstantin Shvachko]

+1 for the proposal.

I thought we already EOL-ed 2.6 though some time ago.

Thanks,
--Konstantin

On Tue, Aug 20, 2019 at 8:03 PM Wangda Tan  wrote:

> Hi all,
>
> This is a vote thread to mark any versions smaller than 2.7 (inclusive),
> and 3.0 EOL. This is based on discussions of [1]
>
> This discussion runs for 7 days and will conclude on Aug 28 Wed.
>
> Please feel free to share your thoughts.
>
> Thanks,
> Wangda
>
> [1]
>
> http://mail-archives.apache.org/mod_mbox/hadoop-yarn-dev/201908.mbox/%3cCAD++eC=ou-tit1faob-dbecqe6ht7ede7t1dyra2p1yinpe...@mail.gmail.com%3e
> ,
>

Re: Make EOL branches to public?

[Konstantin Shvachko]

I would also suggest making an explicit commit to the branch stating it is
EOL.

Thanks,
--Konstantin

On Tue, Aug 20, 2019 at 7:59 PM Wangda Tan  wrote:

> Thank you all for suggestions. Let me send a vote email to mark 2.6, 2.7,
> 3.0 EOL.
>
> - Wangda
>
> On Wed, Aug 21, 2019 at 9:34 AM Akira Ajisaka  wrote:
>
> > +1
> >
> > Thank you for the discussion.
> >
> > -Akira
> >
> > On Wed, Aug 21, 2019 at 5:51 AM Wei-Chiu Chuang 
> > wrote:
> > >
> > > +1
> > > I feel like one year of inactivity is a good sign that the community is
> > not
> > > interested in the branch any more.
> > >
> > > On Fri, Aug 16, 2019 at 3:14 AM Wangda Tan 
> wrote:
> > >
> > > > Hi folks,
> > > >
> > > > Want to hear your thoughts about what we should do to make some
> > branches
> > > > EOL. We discussed a couple of times before in dev lists and PMC list.
> > > > However, we couldn't get a formal process of EOL. According to the
> > > > discussion. It is hard to decide it based on time like "After 1st
> > release,
> > > > EOL in 2 years". Because community members still want to maintain it
> > and
> > > > developers still want to get a newer version released.
> > > >
> > > > However, without a public place to figure out which release will be
> > EOL, it
> > > > is very hard for users to choose the right releases to upgrade and
> > develop.
> > > >
> > > > So I want to propose to make an agreement about making a public EOL
> > wiki
> > > > page and create a process to EOL a release:
> > > >
> > > > The process I'm thinking is very simple: If no volunteer to do a
> > > > maintenance release in a short to mid-term (like 3 months to 1 or 1.5
> > > > year). We will claim a release is EOL. After EOL community can still
> > choose
> > > > to do a security-only release.
> > > >
> > > > Here's a list which I can think about:
> > > >
> > > > 1) 2.6.x (Or any release older than 2.6) (Last released at Oct 2016)
> > > > 2) 2.7.x (Last released at Apr 2018)
> > > > 4) 3.0.x (Last released at May 2018)
> > > >
> > > > Thoughts?
> > > >
> > > > Thanks,
> > > > Wangda
> > > >
> >
>

Re: [DISCUSS] Release 3.0.4 or branch-3.0 EOL

[Konstantin Shvachko]

Hey Wei-Chiu,

Thanks for sharing your plans on Hadoop 3. Good to know.

1. So I understand people don't care about Hadoop 3.0 any more.
I see we don't even list this line on the Releases page:
https://hadoop.apache.org/releases.html
I think it is fair to EOL 3.0 and stop making changes to it.

2. Sounds like there are still plans for Hadoop 3.1. So let's try to be
more consistent with backporting fixes into branch-3.1 along with
branch-3.2.

Do we have a Wiki or a page reflecting active and EOL releases?

Thanks,
--Konstantin

On Fri, Aug 9, 2019 at 4:06 PM Wei-Chiu Chuang  wrote:

> Erik,
> I've just got this message now. Something funky is going on with my
> workplace mailbox.
>
> With my Apache's hat on, I find it too costly to maintain a 3.0.x release
> line: my personal time to try to backport commits, resolving conflicts.
> Additionally, we would be forced to make additional 3.0.x releases if a new
> CVE is discovered.
>
> With my Cloudera's hat on, the last downtream release, CDH6.3.0 is the last
> release on Apache Hadoop 3.0.x. The next release, CDP 1.0 is on 3.1.x, and
> at some point, may rebase onto 3.2.x. The timeline to rebase onto 3.2 is
> unclear (at least 6 months out), and so I am still actively maintaining
> this branch as much as possible.
>
> That is to say, assuming Cloudera's the only corporate sponsor for
> branch-3.0, this branch is all but dead.
>
> On another note, I recently learned that Apache Spark 3 will only support
> Hadoop 3.2, skipping Hadoop 3.0/3.1. I don't know how you guys think, but I
> feel like that'd expedite the death of branch-3.1, which is why we are
> thinking about a 3.2 rebase in CDP.
>
> On Fri, Aug 9, 2019 at 3:31 PM Erik Krogen  wrote:
>
> > I'd like to revive this discussion. Today I see committers variously
> > skipping backports to branch-3.0 and/or branch-3.1 (when backporting to
> > branch-2, for example) and I'm concerned that we will have divergence
> > between what commits are present in which branches.
> >
> > Wei-Chiu, is Cloudera still rebasing on top of branch-3.1? If so then it
> > seems we should be continuing to actively maintain it and not skipping
> any
> > backports here.
> >
> > > If HDFS-13596 is fixed in branch-3.1/3.2, will it be possible to
> > > rolling upgrade from Hadoop 2 to Hadoop 3.1/3.2? If the answer is yes,
> > > I'm thinking we can drop 3.0 and consider 3.1+ only.
> >
> >
> > I agree on this front. I don't see any reason that we can't encourage
> > upgrades from 2.x to 3.1+.
> >
> > Should we initiate a vote to officially EOL the 3.0 line?
> >
> > Thanks
> > Erik
> >
> > P.S.: Sorry if you received this email twice, I was informed by members
> of
> > this team that my original email was not received by most folks.
> >
> > On Sun, May 26, 2019 at 11:56 PM Akira Ajisaka 
> > wrote:
> >
> > > Thank you for your feedback,
> > >
> > > > IMHO, I'd love to see HDFS-13596 <
> > > https://issues.apache.org/jira/browse/HDFS-13596> fixed to make it
> > > possible to rolling upgrade from Hadoop 2 to Hadoop 3.0
> > >
> > > If HDFS-13596 is fixed in branch-3.1/3.2, will it be possible to
> > > rolling upgrade from Hadoop 2 to Hadoop 3.1/3.2? If the answer is yes,
> > > I'm thinking we can drop 3.0 and consider 3.1+ only.
> > >
> > > Thanks,
> > > Akira
> > >
> > > On Wed, May 22, 2019 at 4:39 AM Ajay Kumar 
> > > wrote:
> > > >
> > > > +1 for keeping no of active branches low, specially when it is not
> used
> > > actively.
> > > >
> > > > On Mon, May 20, 2019 at 3:33 AM Steve Loughran
> > >  wrote:
> > > >>
> > > >> I've been intermittently backporting things to it, but like you say:
> > not
> > > >> getting much active use.
> > > >>
> > > >>
> > > >>1. I'd like to view the 3.1 branch as the main "ready to play
> with"
> > > >>Hadoop 3.x release, with 3.2 some new features.
> > > >>2. I'm planning on backporting some of the hadoop 3.x ABFS and
> S3A
> > > work
> > > >>to that 3.x release, and for S3A, some to 3.1.x (and indeed,
> maybe
> > we
> > > >>should do the abfs connector, After all: it's not going to cause
> > any
> > > >>regressions).
> > > >>3. The hadoop-aws changes will include HADOOP-16117, AWS SDK
> > update -
> > > >>Sean Mackrory has been advocating "keep all active branches
> current
> > > with
> > > >>the AWS SDKs", and I've come to agree. The testing there didn't
> > find
> > > any
> > > >>regressions, which was a pleasant surprise.
> > > >>4. For S3A, a big patch has just gone in, HADOOP-16085, which
> adds
> > > etag
> > > >>and version columns to the S3Guard DDB tables, and uses these at
> > > load time.
> > > >>Even if we don't backport that patch to the 3.1 line, it makes
> > sense
> > > for
> > > >>all 3.1.x clients to be updating the DB with the relevant columns
> > as
> > > they
> > > >>write it, so that on a mixed-client deployment, everyone keeps
> the
> > > table up
> > > >>to date
> > > >>
> > > >>
> > > >> As usual, help welcome, 

Re: HDFS Scalability Limit?

[Konstantin Shvachko]

Hi Wei-Chiu,

We run similar Hadoop installations as Kihwal describes. Thanks for sharing
Kihwal.
Our clusters are also not Federated.
So far the growth rate is the same as we reported in our talk last year
(slide #5) 2x per year:
https://www.slideshare.net/Hadoop_Summit/scaling-hadoop-at-linkedin-107176757

We track three main metrics: Total space used, Number of Objects (Files+
blocks), and Number of Tasks per day.
I found that the number of nodes is mostly irrelevant in measuring cluster
size, since the nodes are very diverse in configuration and are constantly
upgrading, so you may have the same #nodes, but much more drives, cores,
RAM on each of them - a bigger cluster.

I do not see 200 GB heap size as a limit. We ran Dynamometer experiments
with a bigger heap fitting 1 billion files and blocks. It should be doable,
but we may hit other scalability limits when we get to so many objects. See
Erik's talk discussing the experiments and solutions:
https://www.slideshare.net/xkrogen/hadoop-meetup-jan-2019-dynamometer-and-a-case-study-in-namenode-gc

Well Hadoop scalability has always been a moving target for me. I don't
think you can set it in stone once and for all.

Thanks,
--Konstantin

On Sat, Jun 15, 2019 at 5:20 PM Wei-Chiu Chuang  wrote:

> Thank you, Kihwal for the insightful comments!
>
> As I understand it, Yahoo's ops team has a good control of application
> behavior. I tend to be conservative in terms of number of files and
> heap size. We don't have such luxury, and our customers have a wide
> spectrum of workloads and features (e.g., snapshots, data at-rest
> encryption, Impala).
>
> Yes -- decomm/recomm is a pain, and I am working with my colleague,
> @Stephen
> O'Donnell  , to address this problem. Have you
> tried maintenance mode? It's in Hadoop 2.9 but a number of decomm/recomm
> needs are alleviated by maintenance mode.
>
> I know Twitter is a big user of maintenance mode, and I'm wondering if
> Twitter folks can offer some experience with it at large scale. CDH
> supports maintenance mode, but our users don't seem to be quite familiar
> with it. Are there issues that were dealt with, but not reported in the
> JIRA? Basically, I'd like to know the operational complexity of this
> feature at large scale.
>
> On Thu, Jun 13, 2019 at 4:00 PM Kihwal Lee  .invalid>
> wrote:
>
> > Hi Wei-Chiu,
> >
> > We have experience with 5,000 - 6,000 node clusters.  Although it
> ran/runs
> > fine, any heavy hitter activities such as decommissioning needed to be
> > carefully planned.   In terms of files and blocks, we have multiple
> > clusters running stable with over 500M files and blocks.  Some at over
> 800M
> > with the max heap at 256GB. It can probably go higher, but we haven't
> done
> > performance testing & optimizations beyond 256GB yet.  All our clusters
> are
> > un-federated. Funny how the feature was developed in Yahoo! and ended up
> > not being used here. :)  We have a cluster with about 180PB of
> provisioned
> > space. Many clusters are using over 100PB in their steady state.  We
> don't
> > run datanodes too dense, so can't tell what the per-datanode limit is.
> >
> > Thanks and 73
> > Kihwal
> >
> > On Thu, Jun 13, 2019 at 1:57 PM Wei-Chiu Chuang 
> > wrote:
> >
> >> Hi community,
> >>
> >> I am currently drafting a HDFS scalability guideline doc, and I'd like
> to
> >> understand any data points regarding HDFS scalability limit. I'd like to
> >> share it publicly eventually.
> >>
> >> As an example, through my workplace, and through community chatters, I
> am
> >> aware that HDFS is capable of operating at the following scale:
> >>
> >> Number of DataNodes:
> >> Unfederated: I can reasonably believe a single HDFS NameNode can manage
> up
> >> to 4000 DataNodes. Is there any one who would like to share an even
> larger
> >> cluster?
> >>
> >> Federated: I am aware of one federated HDFS cluster composed of 20,000
> >> DataNodes. JD.com
> >> <
> >>
> https://conferences.oreilly.com/strata/strata-eu-2018/public/schedule/detail/64692
> >> >
> >> has a 15K DN cluster and 210PB total capacity. I suspect it's a
> federated
> >> HDFS cluster.
> >>
> >> Number of blocks & files:
> >> 500 million files seems to be the upper limit at this point. At
> >> this
> >> scale NameNode consumes around 200GB heap, and my experience told me any
> >> number beyond 200GB is unstable. But at some point I recalled some one
> >> mentioned a 400GB NN heap.
> >>
> >> Amount of Data:
> >> I am aware a few clusters more than 100PB in size (federated or not) --
> >> Uber, Yahoo Japan, JD.com.
> >>
> >> Number of Volumes in a DataNode:
> >> DataNodes with 24 volumes is known to work reasonably well. If DataNode
> is
> >> used for archival use cases, a DN can have up to 48 volumes. This is
> >> certainly hardware dependent, but if I know where the current limit is,
> I
> >> can start optimizing the software.
> >>
> >> Total disk space:
> >> CDH
> >> <
> >>
> 

Re: [VOTE] Moving branch-2 precommit/nightly test builds to java 8

[Konstantin Shvachko]

+1 Makes sense to me.

Thanks,
--Konst

On Mon, Feb 4, 2019 at 6:14 PM Jonathan Hung  wrote:

> Hello,
>
> Starting a vote based on the discuss thread [1] for moving branch-2
> precommit/nightly test builds to openjdk8. After this change, the test
> phase for precommit builds [2] and branch-2 nightly build [3] will run on
> openjdk8. To maintain source compatibility, these builds will still run
> their compile phase for branch-2 on openjdk7 as they do now (in addition to
> compiling on openjdk8).
>
> Vote will run for three business days until Thursday Feb 7 6:00PM PDT.
>
> [1]
>
> https://lists.apache.org/thread.html/7e6fb28fc67560f83a2eb62752df35a8d58d86b2a3df4cacb5d738ca@%3Ccommon-dev.hadoop.apache.org%3E
>
> [2]
> https://builds.apache.org/view/H-L/view/Hadoop/job/PreCommit-HADOOP-Build/
> https://builds.apache.org/view/H-L/view/Hadoop/job/PreCommit-HDFS-Build/
> https://builds.apache.org/view/H-L/view/Hadoop/job/PreCommit-YARN-Build/
>
> https://builds.apache.org/view/H-L/view/Hadoop/job/PreCommit-MAPREDUCE-Build/
>
> [3]
>
> https://builds.apache.org/view/H-L/view/Hadoop/job/hadoop-qbt-branch2-java7-linux-x86/
>
> Jonathan Hung
>

Re: [DISCUSS] Moving branch-2 to java 8

[Konstantin Shvachko]

Just to make sure we are on the same page, as the subject of this thread is
too generic and confusing.
*The proposal is to move branch-2 Jenkins builds such as precommit to run
tests on openJDK-8.*
We do not want to break Java 7 source compatibility. The sources and
releases will still depend on Java 7.
We don't see test failures discussed in HADOOP-15711 when we run them
locally with Oracle Java 7.

Thanks,
--Konst

On Fri, Feb 1, 2019 at 12:44 PM Jonathan Hung  wrote:

> Thanks Vinod and Steve, agreed about java7 compile compatibility. At least
> for now, we should be able to maintain java7 source compatibility and run
> tests on java8. There's a test run here:
> https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86-jhung/46/
> which calls a java8 specific API, installs both openjdk7/openjdk8 in the
> dockerfile, compiles on both versions, and tests on just java8 (via
>
> --multijdkdirs=/usr/lib/jvm/java-7-openjdk-amd64,/usr/lib/jvm/java-8-openjdk-amd64
> and --multijdktests=compile). If we eventually decide it's too much of a
> pain to maintain java7 source compatibility we can do that at a later
> point.
>
> Also based on discussion with others in the community at the contributors
> meetup this past Wednesday, seems we are generally in favor of testing
> against java8. I'll start a vote soon.
>
> Jonathan Hung
>
>
> On Tue, Jan 29, 2019 at 4:11 AM Steve Loughran 
> wrote:
>
> > branch-2 is the JDK 7 branch, but for a long time I (and presumably
> > others) have relied on jenkins to keep us honest by doing that build and
> > test
> >
> > right now, we can't do that any more, due to jdk7 bugs which will never
> be
> > fixed by oracle, or at least, not in a public release.
> >
> > If we can still do the compile in java 7 language and link to java 7 JDK,
> > then that bit of the release is good -then java 8 can be used for that
> test
> >
> > Ultimately, we're going to be forced onto java 8 just because all our
> > dependencies have moved onto it, and some CVE will force us to move.
> >
> > At which point, I think its time to declare branch-2 dead. It's had a
> > great life, but trying to keep java 7 support alive isn't sustainable.
> Not
> > just in this testing, but
> > cherrypicking patches back gets more and more difficult -branch-3 has
> > moved on in both use of java 8 language, and in the codebase in general.
> >
> > > On 28 Jan 2019, at 20:18, Vinod Kumar Vavilapalli 
> > wrote:
> > >
> > > The community made a decision long time ago that we'd like to keep the
> > compatibility & so tie branch-2 to Java 7, but do Java 8+ only work on
> 3.x.
> > >
> > > I always assumed that most (all?) downstream users build branch-2 on
> JDK
> > 7 only, can anyone confirm? If so, there may be an easier way to address
> > these test issues.
> > >
> > > +Vinod
> > >
> > >> On Jan 28, 2019, at 11:24 AM, Jonathan Hung 
> > wrote:
> > >>
> > >> Hi folks,
> > >>
> > >> Forking a discussion based on HADOOP-15711. To summarize, there are
> > issues
> > >> with branch-2 tests running on java 7 (openjdk) which don't exist on
> > java
> > >> 8. From our testing, the build can pass with openjdk 8.
> > >>
> > >> For branch-3, the work to move the build to use java 8 was done in
> > >> HADOOP-14816 as part of the Dockerfile OS version change. HADOOP-16053
> > was
> > >> filed to backport this OS version change to branch-2 (but without the
> > java
> > >> 7 -> java 8 change). So my proposal is to also make the java 7 ->
> java 8
> > >> version change in branch-2.
> > >>
> > >> As mentioned in HADOOP-15711, the main issue is around source and
> binary
> > >> compatibility. I don't currently have a great answer, but one initial
> > >> thought is to build source/binary against java 7 to ensure
> compatibility
> > >> and run the rest of the build as java 8.
> > >>
> > >> Thoughts?
> > >>
> > >> Jonathan Hung
> > >
> > >
> > > -
> > > To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
> > > For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
> > >
> >
> >
>

Re: [Result] [VOTE - 2] Merge HDFS-12943 branch to trunk - Consistent Reads from Standby

[Konstantin Shvachko]

I just merged the feature branch into trunk.
Look for [SBN read] prefix in commits.
Thank you everybody for working on this.

--Konstantin

On Fri, Dec 21, 2018 at 6:34 PM Brahma Reddy Battula <
brahmareddy.batt...@huawei.com> wrote:

> My late +1. Really it's useful feature.. Great work.
>
> -Original Message-
> From: Konstantin Shvachko [mailto:shv.had...@gmail.com]
> Sent: Saturday, December 22, 2018 6:48 AM
> To: Hadoop Common ; hdfs-dev <
> hdfs-...@hadoop.apache.org>
> Cc: mapreduce-...@hadoop.apache.org; yarn-...@hadoop.apache.org
> Subject: [Result] [VOTE - 2] Merge HDFS-12943 branch to trunk - Consistent
> Reads from Standby
>
> Obviously +1 from me.
>
> With four binding +1s, two non-binding +1s, and no -1s this vote passes.
> Thank you folks for working on the feature and for voting.
> Will do the merge in bit.
>
> Thanks,
> --Konst
>
> On Fri, Dec 14, 2018 at 6:16 PM Konstantin Shvachko 
> wrote:
>
> > Hi Hadoop developers,
> >
> > I would like to propose to merge to trunk the feature branch
> > HDFS-12943 for Consistent Reads from Standby Node. The feature is
> > intended to scale read RPC workloads. On large clusters reads comprise
> > 95% of all RPCs to the NameNode. We should be able to accommodate
> > higher overall RPC workloads (up to 4x by some estimates) by adding
> multiple ObserverNodes.
> >
> > The main functionality has been implemented see sub-tasks of HDFS-12943.
> > We followed up with the test plan. Testing was done on two independent
> > clusters (see HDFS-14058 and HDFS-14059) with security enabled.
> > We ran standard HDFS commands, MR jobs, admin commands including
> > manual failover.
> > We know of one cluster running this feature in production.
> >
> > Since the previous vote we addressed Daryn's concern (see HDFS-13873),
> > added documentation for the new feature, and fixed a few other jiras.
> >
> > I attached a unified patch to the umbrella jira for the review.
> > Please vote on this thread. The vote will run for 7 days until Wed Dec
> 21.
> >
> > Thanks,
> > --Konstantin
> >
>

[Result] [VOTE - 2] Merge HDFS-12943 branch to trunk - Consistent Reads from Standby

[Konstantin Shvachko]

Obviously +1 from me.

With four binding +1s, two non-binding +1s, and no -1s this vote passes.
Thank you folks for working on the feature and for voting.
Will do the merge in bit.

Thanks,
--Konst

On Fri, Dec 14, 2018 at 6:16 PM Konstantin Shvachko 
wrote:

> Hi Hadoop developers,
>
> I would like to propose to merge to trunk the feature branch HDFS-12943
> for Consistent Reads from Standby Node. The feature is intended to scale
> read RPC workloads. On large clusters reads comprise 95% of all RPCs to the
> NameNode. We should be able to accommodate higher overall RPC workloads (up
> to 4x by some estimates) by adding multiple ObserverNodes.
>
> The main functionality has been implemented see sub-tasks of HDFS-12943.
> We followed up with the test plan. Testing was done on two independent
> clusters (see HDFS-14058 and HDFS-14059) with security enabled.
> We ran standard HDFS commands, MR jobs, admin commands including manual
> failover.
> We know of one cluster running this feature in production.
>
> Since the previous vote we addressed Daryn's concern (see HDFS-13873),
> added documentation for the new feature, and fixed a few other jiras.
>
> I attached a unified patch to the umbrella jira for the review.
> Please vote on this thread. The vote will run for 7 days until Wed Dec 21.
>
> Thanks,
> --Konstantin
>

[VOTE - 2] Merge HDFS-12943 branch to trunk - Consistent Reads from Standby

[Konstantin Shvachko]

Hi Hadoop developers,

I would like to propose to merge to trunk the feature branch HDFS-12943 for
Consistent Reads from Standby Node. The feature is intended to scale read
RPC workloads. On large clusters reads comprise 95% of all RPCs to the
NameNode. We should be able to accommodate higher overall RPC workloads (up
to 4x by some estimates) by adding multiple ObserverNodes.

The main functionality has been implemented see sub-tasks of HDFS-12943.
We followed up with the test plan. Testing was done on two independent
clusters (see HDFS-14058 and HDFS-14059) with security enabled.
We ran standard HDFS commands, MR jobs, admin commands including manual
failover.
We know of one cluster running this feature in production.

Since the previous vote we addressed Daryn's concern (see HDFS-13873),
added documentation for the new feature, and fixed a few other jiras.

I attached a unified patch to the umbrella jira for the review.
Please vote on this thread. The vote will run for 7 days until Wed Dec 21.

Thanks,
--Konstantin

Re: [VOTE] Merge HDFS-12943 branch to trunk - Consistent Reads from Standby

[Konstantin Shvachko]

Hi Yongjun,

Good suggestion. This is essentially what HDFS-13873 is implementing to
mitigate the concern.

Thanks,
--Konstantin

On Wed, Dec 12, 2018 at 10:35 PM Yongjun Zhang  wrote:

> Hi Konstantin,
>
> Thanks for addressing my other question about failover.
>
> Some thought to share about the suggestion Daryn made.  Seems we could try
> this: let ObserverNode throws an RetriableException back to client saying
> it has not reached the transaction ID to serve the client yet, maybe even
> include the transaction ID gap information in the exception, then when the
> client received the RetriableException, it can decide whether the continue
> to send the request to the observer node again, or to the active NN when
> the gap is too big.
>
> Though saving another RPC would help the performance with the current
> implementation, I expect the above mentioned exception only happens
> infrequently, so the performance won't be too bad, plus the client has a
> chance to try ANN when knowing that the observer is too behind at extreme
> case.
>
> I wonder how different the performance is between these two approaches in
> cluster with real workload.
>
> Comments?
>
> --Yongjun
>
> On Fri, Dec 7, 2018 at 4:10 PM Konstantin Shvachko 
> wrote:
>
>> Hi Daryn,
>>
>> Wanted to backup Chen's earlier response to your concerns about rotating
>> calls in the call queue.
>> Our design
>> 1. targets directly the livelock problem by rejecting calls on the
>> Observer
>> that are not likely to be responded in timely matter: HDFS-13873.
>> 2. The call queue rotation is only done on Observers, and never on the
>> active NN, so it stays free of attacks like you suggest.
>>
>> If this is a satisfactory mitigation for the problem could you please
>> reconsider your -1, so that people could continue voting on this thread.
>>
>> Thanks,
>> --Konst
>>
>> On Thu, Dec 6, 2018 at 10:38 AM Daryn Sharp  wrote:
>>
>> > -1 pending additional info.  After a cursory scan, I have serious
>> concerns
>> > regarding the design.  This seems like a feature that should have been
>> > purely implemented in hdfs w/o touching the common IPC layer.
>> >
>> > The biggest issue in the alignment context.  It's purpose appears to be
>> > for allowing handlers to reinsert calls back into the call queue.
>> That's
>> > completely unacceptable.  A buggy or malicious client can easily cause
>> > livelock in the IPC layer with handlers only looping on calls that never
>> > satisfy the condition.  Why is this not implemented via
>> RetriableExceptions?
>> >
>> > On Thu, Dec 6, 2018 at 1:24 AM Yongjun Zhang
>> 
>> > wrote:
>> >
>> >> Great work guys.
>> >>
>> >> Wonder if we can elaborate what's impact of not having #2 fixed, and
>> why
>> >> #2
>> >> is not needed for the feature to complete?
>> >> 2. Need to fix automatic failover with ZKFC. Currently it does not
>> doesn't
>> >> know about ObserverNodes trying to convert them to SBNs.
>> >>
>> >> Thanks.
>> >> --Yongjun
>> >>
>> >>
>> >> On Wed, Dec 5, 2018 at 5:27 PM Konstantin Shvachko <
>> shv.had...@gmail.com>
>> >> wrote:
>> >>
>> >> > Hi Hadoop developers,
>> >> >
>> >> > I would like to propose to merge to trunk the feature branch
>> HDFS-12943
>> >> for
>> >> > Consistent Reads from Standby Node. The feature is intended to scale
>> >> read
>> >> > RPC workloads. On large clusters reads comprise 95% of all RPCs to
>> the
>> >> > NameNode. We should be able to accommodate higher overall RPC
>> workloads
>> >> (up
>> >> > to 4x by some estimates) by adding multiple ObserverNodes.
>> >> >
>> >> > The main functionality has been implemented see sub-tasks of
>> HDFS-12943.
>> >> > We followed up with the test plan. Testing was done on two
>> independent
>> >> > clusters (see HDFS-14058 and HDFS-14059) with security enabled.
>> >> > We ran standard HDFS commands, MR jobs, admin commands including
>> manual
>> >> > failover.
>> >> > We know of one cluster running this feature in production.
>> >> >
>> >> > There are a few outstanding issues:
>> >> > 1. Need to provide proper documentation - a user guide for the new
>> >> feature
>> >> > 2. Need to fix automatic failover with ZKFC. Currently it does not
>> >> doesn't
>> >> > know about ObserverNodes trying to convert them to SBNs.
>> >> > 3. Scale testing and performance fine-tuning
>> >> > 4. As testing progresses, we continue fixing non-critical bugs like
>> >> > HDFS-14116.
>> >> >
>> >> > I attached a unified patch to the umbrella jira for the review and
>> >> Jenkins
>> >> > build.
>> >> > Please vote on this thread. The vote will run for 7 days until Wed
>> Dec
>> >> 12.
>> >> >
>> >> > Thanks,
>> >> > --Konstantin
>> >> >
>> >>
>> >
>> >
>> > --
>> >
>> > Daryn
>> >
>>
>

[Result] [VOTE] Merge HDFS-12943 branch to trunk - Consistent Reads from Standby

[Konstantin Shvachko]

This vote failed due to Daryn Sharp's veto.
The concern is being addressed by HDFS-13873. I will start a new vote once
this is committed.

Note for Daryn. Your non-responsive handling of the veto makes a bad
precedence and is a bad example of communication on the lists from a
respected member of this community. Please check your availability for
followup discussions if you choose to get involved with important decisions.

On Fri, Dec 7, 2018 at 4:10 PM Konstantin Shvachko 
wrote:

> Hi Daryn,
>
> Wanted to backup Chen's earlier response to your concerns about rotating
> calls in the call queue.
> Our design
> 1. targets directly the livelock problem by rejecting calls on the
> Observer that are not likely to be responded in timely matter: HDFS-13873.
> 2. The call queue rotation is only done on Observers, and never on the
> active NN, so it stays free of attacks like you suggest.
>
> If this is a satisfactory mitigation for the problem could you please
> reconsider your -1, so that people could continue voting on this thread.
>
> Thanks,
> --Konst
>
> On Thu, Dec 6, 2018 at 10:38 AM Daryn Sharp  wrote:
>
>> -1 pending additional info.  After a cursory scan, I have serious
>> concerns regarding the design.  This seems like a feature that should have
>> been purely implemented in hdfs w/o touching the common IPC layer.
>>
>> The biggest issue in the alignment context.  It's purpose appears to be
>> for allowing handlers to reinsert calls back into the call queue.  That's
>> completely unacceptable.  A buggy or malicious client can easily cause
>> livelock in the IPC layer with handlers only looping on calls that never
>> satisfy the condition.  Why is this not implemented via RetriableExceptions?
>>
>> On Thu, Dec 6, 2018 at 1:24 AM Yongjun Zhang 
>> wrote:
>>
>>> Great work guys.
>>>
>>> Wonder if we can elaborate what's impact of not having #2 fixed, and why
>>> #2
>>> is not needed for the feature to complete?
>>> 2. Need to fix automatic failover with ZKFC. Currently it does not
>>> doesn't
>>> know about ObserverNodes trying to convert them to SBNs.
>>>
>>> Thanks.
>>> --Yongjun
>>>
>>>
>>> On Wed, Dec 5, 2018 at 5:27 PM Konstantin Shvachko >> >
>>> wrote:
>>>
>>> > Hi Hadoop developers,
>>> >
>>> > I would like to propose to merge to trunk the feature branch
>>> HDFS-12943 for
>>> > Consistent Reads from Standby Node. The feature is intended to scale
>>> read
>>> > RPC workloads. On large clusters reads comprise 95% of all RPCs to the
>>> > NameNode. We should be able to accommodate higher overall RPC
>>> workloads (up
>>> > to 4x by some estimates) by adding multiple ObserverNodes.
>>> >
>>> > The main functionality has been implemented see sub-tasks of
>>> HDFS-12943.
>>> > We followed up with the test plan. Testing was done on two independent
>>> > clusters (see HDFS-14058 and HDFS-14059) with security enabled.
>>> > We ran standard HDFS commands, MR jobs, admin commands including manual
>>> > failover.
>>> > We know of one cluster running this feature in production.
>>> >
>>> > There are a few outstanding issues:
>>> > 1. Need to provide proper documentation - a user guide for the new
>>> feature
>>> > 2. Need to fix automatic failover with ZKFC. Currently it does not
>>> doesn't
>>> > know about ObserverNodes trying to convert them to SBNs.
>>> > 3. Scale testing and performance fine-tuning
>>> > 4. As testing progresses, we continue fixing non-critical bugs like
>>> > HDFS-14116.
>>> >
>>> > I attached a unified patch to the umbrella jira for the review and
>>> Jenkins
>>> > build.
>>> > Please vote on this thread. The vote will run for 7 days until Wed Dec
>>> 12.
>>> >
>>> > Thanks,
>>> > --Konstantin
>>> >
>>>
>>
>>
>> --
>>
>> Daryn
>>
>

Re: [VOTE] Merge HDFS-12943 branch to trunk - Consistent Reads from Standby

[Konstantin Shvachko]

Hi Daryn,

Wanted to backup Chen's earlier response to your concerns about rotating
calls in the call queue.
Our design
1. targets directly the livelock problem by rejecting calls on the Observer
that are not likely to be responded in timely matter: HDFS-13873.
2. The call queue rotation is only done on Observers, and never on the
active NN, so it stays free of attacks like you suggest.

If this is a satisfactory mitigation for the problem could you please
reconsider your -1, so that people could continue voting on this thread.

Thanks,
--Konst

On Thu, Dec 6, 2018 at 10:38 AM Daryn Sharp  wrote:

> -1 pending additional info.  After a cursory scan, I have serious concerns
> regarding the design.  This seems like a feature that should have been
> purely implemented in hdfs w/o touching the common IPC layer.
>
> The biggest issue in the alignment context.  It's purpose appears to be
> for allowing handlers to reinsert calls back into the call queue.  That's
> completely unacceptable.  A buggy or malicious client can easily cause
> livelock in the IPC layer with handlers only looping on calls that never
> satisfy the condition.  Why is this not implemented via RetriableExceptions?
>
> On Thu, Dec 6, 2018 at 1:24 AM Yongjun Zhang 
> wrote:
>
>> Great work guys.
>>
>> Wonder if we can elaborate what's impact of not having #2 fixed, and why
>> #2
>> is not needed for the feature to complete?
>> 2. Need to fix automatic failover with ZKFC. Currently it does not doesn't
>> know about ObserverNodes trying to convert them to SBNs.
>>
>> Thanks.
>> --Yongjun
>>
>>
>> On Wed, Dec 5, 2018 at 5:27 PM Konstantin Shvachko 
>> wrote:
>>
>> > Hi Hadoop developers,
>> >
>> > I would like to propose to merge to trunk the feature branch HDFS-12943
>> for
>> > Consistent Reads from Standby Node. The feature is intended to scale
>> read
>> > RPC workloads. On large clusters reads comprise 95% of all RPCs to the
>> > NameNode. We should be able to accommodate higher overall RPC workloads
>> (up
>> > to 4x by some estimates) by adding multiple ObserverNodes.
>> >
>> > The main functionality has been implemented see sub-tasks of HDFS-12943.
>> > We followed up with the test plan. Testing was done on two independent
>> > clusters (see HDFS-14058 and HDFS-14059) with security enabled.
>> > We ran standard HDFS commands, MR jobs, admin commands including manual
>> > failover.
>> > We know of one cluster running this feature in production.
>> >
>> > There are a few outstanding issues:
>> > 1. Need to provide proper documentation - a user guide for the new
>> feature
>> > 2. Need to fix automatic failover with ZKFC. Currently it does not
>> doesn't
>> > know about ObserverNodes trying to convert them to SBNs.
>> > 3. Scale testing and performance fine-tuning
>> > 4. As testing progresses, we continue fixing non-critical bugs like
>> > HDFS-14116.
>> >
>> > I attached a unified patch to the umbrella jira for the review and
>> Jenkins
>> > build.
>> > Please vote on this thread. The vote will run for 7 days until Wed Dec
>> 12.
>> >
>> > Thanks,
>> > --Konstantin
>> >
>>
>
>
> --
>
> Daryn
>

Re: [VOTE] Merge HDFS-12943 branch to trunk - Consistent Reads from Standby

[Konstantin Shvachko]

Hi Yongjun,

Automatic failover sure needs to be fixed (see HDFS-14130 and HDFS-13182).
Along with all other outstanding issues. We plan to continue this on trunk.
The feature is usable now without this issues (see HDFS-14067).
And we would like to get it in, so that people could have early access,
and so that newly developed features were aware of this functionality.
Let us know if you have other suggestions.

Thanks,
--Konstantin

On Wed, Dec 5, 2018 at 11:24 PM Yongjun Zhang  wrote:

> Great work guys.
>
> Wonder if we can elaborate what's impact of not having #2 fixed, and why
> #2 is not needed for the feature to complete?
> 2. Need to fix automatic failover with ZKFC. Currently it does not doesn't
> know about ObserverNodes trying to convert them to SBNs.
>
> Thanks.
> --Yongjun
>
>
> On Wed, Dec 5, 2018 at 5:27 PM Konstantin Shvachko 
> wrote:
>
>> Hi Hadoop developers,
>>
>> I would like to propose to merge to trunk the feature branch HDFS-12943
>> for
>> Consistent Reads from Standby Node. The feature is intended to scale read
>> RPC workloads. On large clusters reads comprise 95% of all RPCs to the
>> NameNode. We should be able to accommodate higher overall RPC workloads
>> (up
>> to 4x by some estimates) by adding multiple ObserverNodes.
>>
>> The main functionality has been implemented see sub-tasks of HDFS-12943.
>> We followed up with the test plan. Testing was done on two independent
>> clusters (see HDFS-14058 and HDFS-14059) with security enabled.
>> We ran standard HDFS commands, MR jobs, admin commands including manual
>> failover.
>> We know of one cluster running this feature in production.
>>
>> There are a few outstanding issues:
>> 1. Need to provide proper documentation - a user guide for the new feature
>> 2. Need to fix automatic failover with ZKFC. Currently it does not doesn't
>> know about ObserverNodes trying to convert them to SBNs.
>> 3. Scale testing and performance fine-tuning
>> 4. As testing progresses, we continue fixing non-critical bugs like
>> HDFS-14116.
>>
>> I attached a unified patch to the umbrella jira for the review and Jenkins
>> build.
>> Please vote on this thread. The vote will run for 7 days until Wed Dec 12.
>>
>> Thanks,
>> --Konstantin
>>
>

[VOTE] Merge HDFS-12943 branch to trunk - Consistent Reads from Standby

[Konstantin Shvachko]

Hi Hadoop developers,

I would like to propose to merge to trunk the feature branch HDFS-12943 for
Consistent Reads from Standby Node. The feature is intended to scale read
RPC workloads. On large clusters reads comprise 95% of all RPCs to the
NameNode. We should be able to accommodate higher overall RPC workloads (up
to 4x by some estimates) by adding multiple ObserverNodes.

The main functionality has been implemented see sub-tasks of HDFS-12943.
We followed up with the test plan. Testing was done on two independent
clusters (see HDFS-14058 and HDFS-14059) with security enabled.
We ran standard HDFS commands, MR jobs, admin commands including manual
failover.
We know of one cluster running this feature in production.

There are a few outstanding issues:
1. Need to provide proper documentation - a user guide for the new feature
2. Need to fix automatic failover with ZKFC. Currently it does not doesn't
know about ObserverNodes trying to convert them to SBNs.
3. Scale testing and performance fine-tuning
4. As testing progresses, we continue fixing non-critical bugs like
HDFS-14116.

I attached a unified patch to the umbrella jira for the review and Jenkins
build.
Please vote on this thread. The vote will run for 7 days until Wed Dec 12.

Thanks,
--Konstantin

Re: [DISCUSS] Hadoop RPC encryption performance improvements

[Konstantin Shvachko]

Hi Wei-Chiu,

Thanks for starting the thread and summarizing the problem. Sorry for slow
response.
We've been looking at the encrypted performance as well and are interested
in this effort.
We ran some benchmarks locally. Our benchmarks also showed substantial
penalty for turning on wire encryption on rpc.
Although it was less drastic - more in the range of -40%. But we ran a
different benchmark NNThroughputBenchmark, and we ran it on 2.6 last year.
Could have published the results, but need to rerun on more recent versions.

Three points from me on this discussion:

1. We should settle on the benchmarking tools.
For development RPCCallBenchmark is good as it measures directly the
improvement on the RPC layer. But for external consumption it is more
important to know about e.g. NameNode RPCs performance. So we probably
should run both benchmarks.
2. SASL vs SSL.
Since current implementation is based on SASL, I think it would make sense
to make improvements in this direction. I assume switching to SSL would
require changes in configuration. Not sure if it will be compatible, since
we don't have the details. At this point I would go with HADOOP-10768.
Given all (Daryn's) concerns are addressed.
3. Performance improvement expectations.
Ideally we want to have < 10% penalty for encrypted communication. Anything
over 30% will probably have very limited usability. And there is the gray
area in between, which could be mitigated by allowing mixed encrypted and
un-encrypted RPCs on the single NameNode like in HDFS-13566.

Thanks,
--Konstantin

On Wed, Oct 31, 2018 at 7:39 AM Daryn Sharp  wrote:

> Various KMS tasks have been delaying my RPC encryption work – which is 2nd
> on TODO list.  It's becoming a top priority for us so I'll try my best to
> get a preliminary netty server patch (sans TLS) up this week if that helps.
>
> The two cited jiras had some critical flaws.  Skimming my comments, both
> use blocking IO (obvious nonstarter).  HADOOP-10768 is a hand rolled
> TLS-like encryption which I don't feel is something the community can or
> should maintain from a security standpoint.
>
> Daryn
>
> On Wed, Oct 31, 2018 at 8:43 AM Wei-Chiu Chuang 
> wrote:
>
> > Ping. Any one? Cloudera is interested in moving forward with the RPC
> > encryption improvements, but I just like to get a consensus which
> approach
> > to go with.
> >
> > Otherwise I'll pick HADOOP-10768 since it's ready for commit, and I've
> > spent time on testing it.
> >
> > On Thu, Oct 25, 2018 at 11:04 AM Wei-Chiu Chuang 
> > wrote:
> >
> > > Folks,
> > >
> > > I would like to invite all to discuss the various Hadoop RPC encryption
> > > performance improvements. As you probably know, Hadoop RPC encryption
> > > currently relies on Java SASL, and have _really_ bad performance (in
> > terms
> > > of number of RPCs per second, around 15~20% of the one without SASL)
> > >
> > > There have been some attempts to address this, most notably,
> HADOOP-10768
> > >  (Optimize Hadoop
> > RPC
> > > encryption performance) and HADOOP-13836
> > >  (Securing Hadoop
> > RPC
> > > using SSL). But it looks like both attempts have not been progressing.
> > >
> > > During the recent Hadoop contributor meetup, Daryn Sharp mentioned he's
> > > working on another approach that leverages Netty for its SSL
> encryption,
> > > and then integrate Netty with Hadoop RPC so that Hadoop RPC
> automatically
> > > benefits from netty's SSL encryption performance.
> > >
> > > So there are at least 3 attempts to address this issue as I see it. Do
> we
> > > have a consensus that:
> > > 1. this is an important problem
> > > 2. which approach we want to move forward with
> > >
> > > --
> > > A very happy Hadoop contributor
> > >
> >
> >
> > --
> > A very happy Hadoop contributor
> >
>
>
> --
>
> Daryn
>

Re: Hadoop 3.2 Release Plan proposal

[Konstantin Shvachko]

Another thing is that I see a bunch of jiras under HDFS-8707, which don't
have the Fix Version field listing 3.2, some have it just empty.
This means they will not be populated into release notes.

Thanks,
--Konstantin

On Thu, Oct 25, 2018 at 7:59 AM Sunil G  wrote:

> Thanks Konstantin for pointing out.
> As 3.2 is pretty much on RC level, its better we try to find a good
> solution to this issue.
>
> I ll follow up on this in the jira.
>
> - Sunil
>
> On Thu, Oct 25, 2018 at 11:35 AM Konstantin Shvachko 
> wrote:
>
>> I've tried to attract attention to an incompatibility issue through the
>> jira, but it didn't work. So pitching in in this thread.
>> https://issues.apache.org/jira/browse/HDFS-12026
>> It introduced binary incompatibility, which will prevent people from
>> upgrading from 3.1 to 3.2.
>> I think it can get messy if we release anything with this feature.
>>
>> Thanks,
>> --Konstantin
>>
>> On Mon, Oct 22, 2018 at 5:01 AM Steve Loughran 
>> wrote:
>>
>>> its in.
>>>
>>> good catch!
>>>
>>> On 20 Oct 2018, at 01:35, Wei-Chiu Chuang >> weic...@cloudera.com>> wrote:
>>>
>>> Thanks Sunil G for driving the release,
>>> I filed HADOOP-15866<https://issues.apache.org/jira/browse/HADOOP-15866>
>>> for a compat fix. If any one has cycle please review it, as I think it is
>>> needed for 3.2.0.
>>>
>>> On Thu, Oct 18, 2018 at 4:43 AM Sunil G >> sun...@apache.org>> wrote:
>>> Hi Folks,
>>>
>>> As we previously communicated for 3.2.0 release, we have delayed due to
>>> few
>>> blockers in our gate.
>>>
>>> I just cut branch-3.2.0 for release purpose. branch-3.2 will be open for
>>> all bug fixes.
>>>
>>> - Sunil
>>>
>>>
>>> On Tue, Oct 16, 2018 at 8:59 AM Sunil G >> sun...@apache.org>> wrote:
>>>
>>> > Hi Folks,
>>> >
>>> > We are now close to RC as other blocker issues are now merged to trunk
>>> and
>>> > branch-3.2. Last 2 critical issues are closer to merge and will be
>>> > committed in few hours.
>>> > With this, I will be creating 3.2.0 branch today and will go ahead
>>> with RC
>>> > related process.
>>> >
>>> > - Sunil
>>> >
>>> > On Mon, Oct 15, 2018 at 11:43 PM Jonathan Bender >> <mailto:jonben...@stripe.com>>
>>> > wrote:
>>> >
>>> >> Hello, were there any updates around the 3.2.0 RC timing? All I see in
>>> >> the current blockers are related to the new Submarine subproject,
>>> wasn't
>>> >> sure if that is what is holding things up.
>>> >>
>>> >> Cheers,
>>> >> Jon
>>> >>
>>> >> On Tue, Oct 2, 2018 at 7:13 PM, Sunil G >> sun...@apache.org>> wrote:
>>> >>
>>> >>> Thanks Robert and Haibo for quickly correcting same.
>>> >>> Sigh, I somehow missed one file while committing the change. Sorry
>>> for
>>> >>> the
>>> >>> trouble.
>>> >>>
>>> >>> - Sunil
>>> >>>
>>> >>> On Wed, Oct 3, 2018 at 5:22 AM Robert Kanter >> <mailto:rkan...@cloudera.com>>
>>> >>> wrote:
>>> >>>
>>> >>> > Looks like there's two that weren't updated:
>>> >>> > >> [115] 16:32 : hadoop-common (trunk) :: grep "3.2.0-SNAPSHOT" .
>>> -r
>>> >>> > --include=pom.xml
>>> >>> > ./hadoop-project/pom.xml:
>>> >>> >
>>> 3.2.0-SNAPSHOT
>>> >>> > ./pom.xml:3.2.0-SNAPSHOT
>>> >>> >
>>> >>> > I've just pushed in an addendum commit to fix those.
>>> >>> > In the future, please make sure to do a sanity compile when
>>> updating
>>> >>> poms.
>>> >>> >
>>> >>> > thanks
>>> >>> > - Robert
>>> >>> >
>>> >>> > On Tue, Oct 2, 2018 at 11:44 AM Aaron Fabbri
>>> >>> mailto:fab...@cloudera.com.invalid>>
>>> >>> > wrote:
>>> >>> >
>>> >>> >> Trunk is not building for me.. Did you miss a 3.2.0-SNAPSHOT in
>>> the
>>> >>> >>

Re: Hadoop 3.2 Release Plan proposal

[Konstantin Shvachko]

I've tried to attract attention to an incompatibility issue through the
jira, but it didn't work. So pitching in in this thread.
https://issues.apache.org/jira/browse/HDFS-12026
It introduced binary incompatibility, which will prevent people from
upgrading from 3.1 to 3.2.
I think it can get messy if we release anything with this feature.

Thanks,
--Konstantin

On Mon, Oct 22, 2018 at 5:01 AM Steve Loughran 
wrote:

> its in.
>
> good catch!
>
> On 20 Oct 2018, at 01:35, Wei-Chiu Chuang  weic...@cloudera.com>> wrote:
>
> Thanks Sunil G for driving the release,
> I filed HADOOP-15866
> for a compat fix. If any one has cycle please review it, as I think it is
> needed for 3.2.0.
>
> On Thu, Oct 18, 2018 at 4:43 AM Sunil G  sun...@apache.org>> wrote:
> Hi Folks,
>
> As we previously communicated for 3.2.0 release, we have delayed due to few
> blockers in our gate.
>
> I just cut branch-3.2.0 for release purpose. branch-3.2 will be open for
> all bug fixes.
>
> - Sunil
>
>
> On Tue, Oct 16, 2018 at 8:59 AM Sunil G  sun...@apache.org>> wrote:
>
> > Hi Folks,
> >
> > We are now close to RC as other blocker issues are now merged to trunk
> and
> > branch-3.2. Last 2 critical issues are closer to merge and will be
> > committed in few hours.
> > With this, I will be creating 3.2.0 branch today and will go ahead with
> RC
> > related process.
> >
> > - Sunil
> >
> > On Mon, Oct 15, 2018 at 11:43 PM Jonathan Bender  >
> > wrote:
> >
> >> Hello, were there any updates around the 3.2.0 RC timing? All I see in
> >> the current blockers are related to the new Submarine subproject, wasn't
> >> sure if that is what is holding things up.
> >>
> >> Cheers,
> >> Jon
> >>
> >> On Tue, Oct 2, 2018 at 7:13 PM, Sunil G  sun...@apache.org>> wrote:
> >>
> >>> Thanks Robert and Haibo for quickly correcting same.
> >>> Sigh, I somehow missed one file while committing the change. Sorry for
> >>> the
> >>> trouble.
> >>>
> >>> - Sunil
> >>>
> >>> On Wed, Oct 3, 2018 at 5:22 AM Robert Kanter  >
> >>> wrote:
> >>>
> >>> > Looks like there's two that weren't updated:
> >>> > >> [115] 16:32 : hadoop-common (trunk) :: grep "3.2.0-SNAPSHOT" . -r
> >>> > --include=pom.xml
> >>> > ./hadoop-project/pom.xml:
> >>> > 3.2.0-SNAPSHOT
> >>> > ./pom.xml:3.2.0-SNAPSHOT
> >>> >
> >>> > I've just pushed in an addendum commit to fix those.
> >>> > In the future, please make sure to do a sanity compile when updating
> >>> poms.
> >>> >
> >>> > thanks
> >>> > - Robert
> >>> >
> >>> > On Tue, Oct 2, 2018 at 11:44 AM Aaron Fabbri
> >>> mailto:fab...@cloudera.com.invalid>>
> >>> > wrote:
> >>> >
> >>> >> Trunk is not building for me.. Did you miss a 3.2.0-SNAPSHOT in the
> >>> >> top-level pom.xml?
> >>> >>
> >>> >>
> >>> >> On Tue, Oct 2, 2018 at 10:16 AM Sunil G  sun...@apache.org>> wrote:
> >>> >>
> >>> >> > Hi All
> >>> >> >
> >>> >> > As mentioned in earlier mail, I have cut branch-3.2 and reset
> trunk
> >>> to
> >>> >> > 3.3.0-SNAPSHOT. I will share the RC details sooner once all
> >>> necessary
> >>> >> > patches are pulled into branch-3.2.
> >>> >> >
> >>> >> > Thank You
> >>> >> > - Sunil
> >>> >> >
> >>> >> >
> >>> >> > On Mon, Sep 24, 2018 at 2:00 PM Sunil G  > wrote:
> >>> >> >
> >>> >> > > Hi All
> >>> >> > >
> >>> >> > > We are now down to the last Blocker and HADOOP-15407 is merged
> to
> >>> >> trunk.
> >>> >> > > Thanks for the support.
> >>> >> > >
> >>> >> > > *Plan for RC*
> >>> >> > > 3.2 branch cut and reset trunk : *25th Tuesday*
> >>> >> > > RC0 for 3.2: *28th Friday*
> >>> >> > >
> >>> >> > > Thank You
> >>> >> > > Sunil
> >>> >> > >
> >>> >> > >
> >>> >> > > On Mon, Sep 17, 2018 at 3:21 PM Sunil G  >
> >>> wrote:
> >>> >> > >
> >>> >> > >> Hi All
> >>> >> > >>
> >>> >> > >> We are down to 3 Blockers and 4 Critical now. Thanks all of you
> >>> for
> >>> >> > >> helping in this. I am following up on these tickets, once its
> >>> closed
> >>> >> we
> >>> >> > >> will cut the 3.2 branch.
> >>> >> > >>
> >>> >> > >> Thanks
> >>> >> > >> Sunil Govindan
> >>> >> > >>
> >>> >> > >>
> >>> >> > >> On Wed, Sep 12, 2018 at 5:10 PM Sunil G  >
> >>> wrote:
> >>> >> > >>
> >>> >> > >>> Hi All,
> >>> >> > >>>
> >>> >> > >>> Inline with the original 3.2 communication proposal dated 17th
> >>> July
> >>> >> > >>> 2018, I would like to provide more updates.
> >>> >> > >>>
> >>> >> > >>> We are approaching previously proposed code freeze date
> >>> (September
> >>> >> 14,
> >>> >> > >>> 2018). So I would like to cut 3.2 branch on 17th Sept and
> point
> >>> >> > existing
> >>> >> > >>> trunk to 3.3 if there are no issues.
> >>> >> > >>>
> >>> >> > >>> *Current Release Plan:*
> >>> >> > >>> Feature freeze date : all features to merge by September 7,
> >>> 2018.
> >>> >> > >>> Code freeze date : blockers/critical only, no improvements and
> >>> >> > >>> 

[RESULT] [VOTE] Release Apache Hadoop 2.7.6 (RC0)

[Konstantin Shvachko]

Hi everybody,

With 4 binding and 4 non-binding +1s and no -1s the vote for Apache Release
2.7.6 passes.
Thank you everybody for contributing to the release, testing, and voting.

Binding +1s
Zhe Zhang
Brahma Reddy Battula
Jason Lowe
Konstantin Shvachko

Non-binding +1s
Chen Liang
Erik Krogen
Takanobu Asanuma
Ajay Kumar



On Mon, Apr 9, 2018 at 4:14 PM, Konstantin Shvachko <shv.had...@gmail.com>
wrote:

> Hi everybody,
>
> This is the next dot release of Apache Hadoop 2.7 line. The previous one
> 2.7.5 was released on December 14, 2017.
> Release 2.7.6 includes critical bug fixes and optimizations. See more
> details in Release Note:
> http://home.apache.org/~shv/hadoop-2.7.6-RC0/releasenotes.html
>
> The RC0 is available at: http://home.apache.org/~shv/hadoop-2.7.6-RC0/
>
> Please give it a try and vote on this thread. The vote will run for 5
> days ending 04/16/2018.
>
> My up to date public key is available from:
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
> Thanks,
> --Konstantin
>

Re: [VOTE] Release Apache Hadoop 2.7.6 (RC0)

[Konstantin Shvachko]

My formal +1 for 2.7.6 RC0

Thanks,
--Konstantin

On Mon, Apr 9, 2018 at 4:14 PM, Konstantin Shvachko <shv.had...@gmail.com>
wrote:

> Hi everybody,
>
> This is the next dot release of Apache Hadoop 2.7 line. The previous one
> 2.7.5 was released on December 14, 2017.
> Release 2.7.6 includes critical bug fixes and optimizations. See more
> details in Release Note:
> http://home.apache.org/~shv/hadoop-2.7.6-RC0/releasenotes.html
>
> The RC0 is available at: http://home.apache.org/~shv/hadoop-2.7.6-RC0/
>
> Please give it a try and vote on this thread. The vote will run for 5
> days ending 04/16/2018.
>
> My up to date public key is available from:
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
> Thanks,
> --Konstantin
>

Re: [VOTE] Release Apache Hadoop 3.0.2 (RC0)

[Konstantin Shvachko]

A note to release managers. As discussed in
https://issues.apache.org/jira/browse/HADOOP-15205
We are producing release artifacts without sources jars. See e.g.
https://repository.apache.org/content/repositories/releases/org/apache/hadoop/hadoop-common/3.0.1/
Based on what is staged on Nexus for 3.0.2 (RC0) I see the next release
will not have sources as well.
https://repository.apache.org/#stagingRepositories
I believe this has something to do with maven deployment stage, potentially
maven-source-plugin.
This is similar for all releases now, and I believe it should be fixed.

Thanks,
--Konstantin

On Tue, Apr 10, 2018 at 8:32 AM, Ajay Kumar 
wrote:

> Thanks Lie for working on this.
>
>- Downloaded src tarball and verified checksums
>- Built from src on mac with java 1.8.0_111
>- Built a pseudo distributed hdfs cluster
>- Run test mr jobs (pi, dfsio,wordcount
>- Verified basic hdfs operations
>- Basic validation for webui
>
> ** I checked maven artifacts and it seems source jars are not there
> (checked hadoop-hdfs , hadoop-client). Not sure if they are required for
> release.
>
>
> On 4/9/18, 4:19 PM, "Xiao Chen"  wrote:
>
> Thanks Eddy for the effort!
>
> +1 (binding)
>
>- Downloaded src tarball and verified checksums
>- Built from src
>- Started a pseudo distributed hdfs cluster
>- Verified basic hdfs operations work
>- Sanity checked logs / webui
>
> Best,
> -Xiao
>
>
> On Mon, Apr 9, 2018 at 11:28 AM, Eric Payne  invalid>
> wrote:
>
> > Thanks a lot for working to produce this release.
> >
> > +1 (binding)
> > Tested the following:
> > - built from source and installed on 6-node pseudo-cluster
> > - tested Capacity Scheduler FairOrderingPolicy and
> FifoOrderingPolicy to
> > determine that capacity was assigned as expected in each case
> > - tested user weights with FifoOrderingPolicy to ensure that weights
> were
> > assigned to users as expected.
> >
> > Eric Payne
> >
> >
> >
> >
> >
> >
> > On Friday, April 6, 2018, 1:17:10 PM CDT, Lei Xu 
> wrote:
> >
> >
> >
> >
> >
> > Hi, All
> >
> > I've created release candidate RC-0 for Apache Hadoop 3.0.2.
> >
> > Please note: this is an amendment for Apache Hadoop 3.0.1 release to
> > fix shaded jars in apache maven repository. The codebase of 3.0.2
> > release is the same as 3.0.1.  New bug fixes will be included in
> > Apache Hadoop 3.0.3 instead.
> >
> > The release page is:
> > https://cwiki.apache.org/confluence/display/HADOOP/
> Hadoop+3.0+Release
> >
> > New RC is available at: http://home.apache.org/~lei/
> hadoop-3.0.2-RC0/
> >
> > The git tag is release-3.0.2-RC0, and the latest commit is
> > 5c141f7c0f24c12cb8704a6ccc1ff8ec991f41ee
> >
> > The maven artifacts are available at
> > https://repository.apache.org/content/repositories/
> orgapachehadoop-1096/
> >
> > Please try the release, especially, *verify the maven artifacts*,
> and vote.
> >
> > The vote will run 5 days, ending 4/11/2018.
> >
> > Thanks for everyone who helped to spot the error and proposed fixes!
> >
> > 
> -
> > To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
> > For additional commands, e-mail: mapreduce-dev-help@hadoop.
> apache.org
> >
> >
> > 
> -
> > To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> > For additional commands, e-mail: common-dev-h...@hadoop.apache.org
> >
> >
>
>
>
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>

Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)

[Konstantin Shvachko]

A note to release managers. As discussed in
https://issues.apache.org/jira/browse/HADOOP-15205
We are producing release artifacts without sources jars. See e.g.
https://repository.apache.org/content/repositories/releases/org/apache/hadoop/hadoop-common/3.1.0/
I believe this has something to do with maven deployment stage, potentially
maven-source-plugin.
This is similar for all releases now, and I believe it should be fixed.

Thanks,
--Konstantin

On Fri, Apr 6, 2018 at 2:01 PM, Wangda Tan  wrote:

> Thanks guys for the additional votes! I just sent out announcement email.
>
> Best,
> Wangda
>
> On Fri, Apr 6, 2018 at 2:32 AM, 俊平堵  wrote:
>
> > Thanks Wangda for the great work! Sorry for my late coming +1 (binding),
> > based on:
> >
> > - Verified signatures
> >
> > - Verified checksums for source and binary artifacts
> >
> > - Built from source
> >
> > - Deployed a single node cluster
> >
> > - Verified web UIs, include Namenode, RM, etc.
> >
> > * Tried shell commands of HDFS and YARN
> >
> > * Ran sample MR jobs, include PI, Sleep, Terasort, etc.
> >
> >
> > Thanks,
> >
> >
> > Junping
> >
> >
> >
> > Wangda Tan 于2018年3月30日 周五下午12:15写道：
> >
> >> Hi folks,
> >>
> >> Thanks to the many who helped with this release since Dec 2017 [1].
> We've
> >> created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
> >>
> >> http://people.apache.org/~wangda/hadoop-3.1.0-RC1
> >>
> >> The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
> >> 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
> >>
> >> The maven artifacts are available via repository.apache.org at
> >> https://repository.apache.org/content/repositories/
> orgapachehadoop-1090/
> >> This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
> >>
> >> 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
> >> include the first class GPU/FPGA support on YARN, Native services,
> Support
> >> rich placement constraints in YARN, S3-related enhancements, allow HDFS
> >> block replicas to be provided by an external storage system, etc.
> >>
> >> For 3.1.0 RC0 vote discussion, please see [3].
> >>
> >> We’d like to use this as a starting release for 3.1.x [1], depending on
> >> how
> >> it goes, get it stabilized and potentially use a 3.1.1 in several weeks
> as
> >> the stable release.
> >>
> >> We have done testing with a pseudo cluster:
> >> - Ran distributed job.
> >> - GPU scheduling/isolation.
> >> - Placement constraints (intra-application anti-affinity) by using
> >> distributed shell.
> >>
> >> My +1 to start.
> >>
> >> Best,
> >> Wangda/Vinod
> >>
> >> [1]
> >> https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
> >> c9e19e559aedc5ebedb4ca08c8@%3Cyarn-dev.hadoop.apache.org%3E
> >> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
> >> AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER
> BY
> >> fixVersion ASC
> >> [3]
> >> https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
> >> 2d4061f26f83547e41d0983ea6@%3Cyarn-dev.hadoop.apache.org%3E
> >>
> >
>

[VOTE] Release Apache Hadoop 2.7.6 (RC0)

[Konstantin Shvachko]

Hi everybody,

This is the next dot release of Apache Hadoop 2.7 line. The previous one 2.7.5
was released on December 14, 2017.
Release 2.7.6 includes critical bug fixes and optimizations. See more
details in Release Note:
http://home.apache.org/~shv/hadoop-2.7.6-RC0/releasenotes.html

The RC0 is available at: http://home.apache.org/~shv/hadoop-2.7.6-RC0/

Please give it a try and vote on this thread. The vote will run for 5 days
ending 04/16/2018.

My up to date public key is available from:
https://dist.apache.org/repos/dist/release/hadoop/common/KEYS

Thanks,
--Konstantin

Re: [DISCUSS] Apache Hadoop 2.7.6 Release Plan

[Konstantin Shvachko]

Just heads up, working on the release candidate now.
It's been a while, I know. But we had some blockers.

Thanks,
--Konstantin

On Thu, Feb 22, 2018 at 5:59 PM, Konstantin Shvachko <shv.had...@gmail.com>
wrote:

> Planning to do a maintenance 2.7.6 release with close to 40 changes since
> the last release.
> If there are no objections I'll start preparations.
>
> Please let me know if there are blockers:
> https://issues.apache.org/jira/browse/HDFS-12641?filter=12343253
>
> Thanks,
> --Konstantin
>

[jira] [Resolved] (HADOOP-15325) Make Configuration#getPasswordFromCredentialsProvider() a public API

[Konstantin Shvachko (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-15325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Konstantin Shvachko resolved HADOOP-15325.
--
Resolution: Implemented

This has been incorporated into HADOOP-12862.

> Make Configuration#getPasswordFromCredentialsProvider() a public API
> 
>
> Key: HADOOP-15325
> URL: https://issues.apache.org/jira/browse/HADOOP-15325
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: conf
>Affects Versions: 2.6.0
>Reporter: Wei-Chiu Chuang
>Assignee: Zsolt Venczel
>Priority: Major
>
> HADOOP-10607 added a public API Configuration.getPassword() which reads 
> passwords from credential provider and then falls back to reading from 
> configuration if one is not available.
> This API has been used throughout Hadoop codebase and downstream 
> applications. It is understandable for old password configuration keys to 
> fallback to configuration to maintain backward compatibility. But for new 
> configuration passwords that don't have legacy, there should be an option to 
> _not_ fallback, because storing passwords in configuration is considered a 
> bad security practice.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [VOTE] Merging branch HDFS-7240 to trunk

[Konstantin Shvachko]

The proposal to add it as a subproject of Hadoop makes sense to me. Thank
you Owen.
I am glad to have a path for scaling HDFS further, especially as it enters
areas like IoT and self-driving cars, where storage requirements are huge.

I am not very fond of the name HDSL, though. "Storage Layer" sounds too
generic.
May be something more descriptive, like HDDS / HDSS (Hadoop Dynamically
Distributed/Scaling Storage).
We can discuss this in the jira HDFS-10419.

Thanks,
--Konstantin

On Fri, Mar 16, 2018 at 3:17 PM, Sanjay Radia 
wrote:

> Owen,
> Thanks for your proposal.
> While I would have prefered to have HDSL in HDFS and also to be part
> of Hadoop releases  for the reasons stated earlier in this thread,
> I am willing to accept your proposal as a compromise to move this forward.
>
> Jitendra, Anu, Daryn, Andrew, Konstantine your thoughts?
>
> Thanks
>
> Sanjay
>
>
> On Mar 14, 2018, at 1:50 PM, Owen O'Malley > wrote:
>
> This discussion seems to have died down coming closer consensus without a
> resolution.
>
> I'd like to propose the following compromise:
>
> * HDSL become a subproject of Hadoop.
> * HDSL will release separately from Hadoop. Hadoop releases will not
> contain HDSL and vice versa.
> * HDSL will get its own jira instance so that the release tags stay
> separate.
> * On trunk (as opposed to release branches) HDSL will be a separate module
> in Hadoop's source tree. This will enable the HDSL to work on their trunk
> and the Hadoop trunk without making releases for every change.
> * Hadoop's trunk will only build HDSL if a non-default profile is enabled.
> * When Hadoop creates a release branch, the RM will delete the HDSL module
> from the branch.
> * HDSL will have their own Yetus checks and won't cause failures in the
> Hadoop patch check.
>
> I think this accomplishes most of the goals of encouraging HDSL development
> while minimizing the potential for disruption of HDFS development.
>
> Thoughts? Andrew, Jitendra, & Sanjay?
>
> Thanks,
>   Owen
>
>

Re: [DISCUSS] 2.9+ stabilization branch

[Konstantin Shvachko]

Thanks Subru for initiating the thread about GPU support.
I think the path of taking 2.9 as a base for 2.10 and adding new resource
types into it is quite reasonable.
That way we can combine stabilization effort on 2.9 with GPUs.

Arun, upgrading Java is probably a separate topic.
We should discuss it on a separate followup thread if we agree to add GPU
support into 2.10.

Andrew, we actually ran a small 3.0 cluster to experiment with Tensorflow
on YARN with gpu resources. It worked well! Therefore the interest.
Although given the breadth (and the quantity) of our use cases it is
infeasible to jump directly to 3.0, as Jonathan explained.
A transitional stage such as 2.10 will be required. Probably the same for
many other big-cluster folks.
It would be great if people who run different hadoop versions <= 2.8 can
converge at 2.10 bridge, to help cross over to 3.
GPU support would be a serious catalyst for us to move forward, which I
also heard from other organizations interested in ML.

Thanks,
--Konstantin

On Tue, Feb 27, 2018 at 1:28 PM, Andrew Wang 
wrote:

> Hi Arun/Subru,
>
> Bumping the minimum Java version is a major change, and incompatible for
> users who are unable to upgrade their JVM version. We're beyond the EOL for
> Java 7, but as we know from our experience with Java 6, there are plenty of
> users who stick on old Java versions. Bumping the Java version also makes
> backports more difficult, and we're still maintaining a number of older 2.x
> releases. I think this is too big for a minor release, particularly when we
> have 3.x as an option that fully supports Java 8.
>
> What's the rationale for bumping it here?
>
> I'm also curious if there are known issues with 3.x that we can fix to make
> 3.x upgrades smoother. I would prefer improving the upgrade experience to
> backporting major features to 2.x since 3.x is meant to be the delivery
> vehicle for new features beyond the ones named here.
>
> Best,
> Andrew
>
> On Tue, Feb 27, 2018 at 11:01 AM, Arun Suresh  wrote:
>
> > Hello folks
> >
> > We also think this bridging release opens up an opportunity to bump the
> > java version in branch-2 to java 8.
> > Would really love to hear thoughts on that.
> >
> > Cheers
> > -Arun/Subru
> >
> >
> > On Mon, Feb 26, 2018 at 5:18 PM, Jonathan Hung 
> > wrote:
> >
> > > Hi Subru,
> > >
> > > Thanks for starting the discussion.
> > >
> > > We (LinkedIn) have an immediate need for resource types and native GPU
> > > support. Given we are running 2.7 on our main clusters, we decided to
> > avoid
> > > deploying hadoop 3.x on our machine learning clusters (and having to
> > > support two very different hadoop versions). Since for us there is
> > > considerable risk and work involved in upgrading to hadoop 3, I think
> > > having a branch-2.10 bridge release for porting important hadoop 3
> > features
> > > to branch-2 is a good idea.
> > >
> > > Thanks,
> > >
> > >
> > > Jonathan Hung
> > >
> > > On Mon, Feb 26, 2018 at 2:37 PM, Subru Krishnan 
> > wrote:
> > >
> > > > Folks,
> > > >
> > > > We (i.e. Microsoft) have started stabilization of 2.9 for our
> > production
> > > > deployment. During planning, we realized that we need to backport 3.x
> > > > features to support GPUs (and more resource types like network IO)
> > > natively
> > > > as part of the upgrade. We'd like to share that work with the
> > community.
> > > >
> > > > Instead of stabilizing the base release and cherry-picking fixes back
> > to
> > > > Apache, we want to work publicly and push fixes directly into
> > > > trunk/.../branch-2 for a stable 2.10.0 release. Our goal is to
> create a
> > > > bridge release for our production clusters to the 3.x series and to
> > > address
> > > > scalability problems in large clusters (N*10k nodes). As we find
> > issues,
> > > we
> > > > will file JIRAs and track resolution of significant
> regressions/faults
> > in
> > > > wiki. Moreover, LinkedIn also has committed plans for a production
> > > > deployment of the same branch. We welcome broad participation,
> > > particularly
> > > > since we'll be stabilizing relatively new features.
> > > >
> > > > The exact list of features we would like to backport in YARN are:
> > > >
> > > >- Support for Resource types [1][2]
> > > >- Native support for GPUs[3]
> > > >- Absolute Resource configuration in CapacityScheduler [4]
> > > >
> > > >
> > > > With regards to HDFS, we are currently looking at mainly fixes to
> > Router
> > > > based Federation and Windows specific fixes which should anyways flow
> > > > normally.
> > > >
> > > > Thoughts?
> > > >
> > > > Thanks,
> > > > Subru/Arun
> > > >
> > > > [1] https://www.mail-archive.com/yarn-dev@hadoop.apache.org/
> > > msg27786.html
> > > > [2] https://www.mail-archive.com/yarn-dev@hadoop.apache.org/
> > > msg28281.html
> > > > [3] https://issues.apache.org/jira/browse/YARN-6223
> > > > [4] 

[DISCUSS] Apache Hadoop 2.7.6 Release Plan

[Konstantin Shvachko]

Planning to do a maintenance 2.7.6 release with close to 40 changes since
the last release.
If there are no objections I'll start preparations.

Please let me know if there are blockers:
https://issues.apache.org/jira/browse/HDFS-12641?filter=12343253

Thanks,
--Konstantin

Re: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk

[Konstantin Shvachko]

Hi Sanjay,

With respect to Ozone my two main concerns were:
1. Wether Ozone can help scaling out the namespace service in handling
higher RPC workloads.
I think we came to common conclusion that using Ozone as a block management
layer is a reasonable path to scaling HDFS.
The discussions are in-progress and probably will go on for a while.
2. Releasing Hadoop with components that don't have security. I was asking
for a design doc at a minimum.
I think the document Anu published addresses it.

Thanks,
Konstantin

On Tue, Feb 20, 2018 at 3:06 PM, sanjay Radia  wrote:

>
>
>
> Konstantine
>
>
> Thanks for your feedback and comments over the last few months.  Have we
> addressed all your  issues and concerns?
>
> sanjay
>
>
> > On Feb 13, 2018, at 6:28 PM, sanjay Radia  wrote:
> >
> > Sorry the formatting got messed by my email client.  Here it is again
> >
> >
> > Dear
> > Hadoop Community Members,
> >
> >  We had multiple community discussions, a few meetings in smaller groups
> and also jira discussions with respect to this thread. We express our
> gratitude for participation and valuable comments.
> >
> > The key questions raised were following
> > 1) How the new block storage layer and OzoneFS benefit HDFS and we were
> asked to chalk out a roadmap towards the goal of a scalable namenode
> working with the new storage layer
> > 2) We were asked to provide a security design
> > 3)There were questions around stability given ozone brings in a large
> body of code.
> > 4) Why can’t they be separate projects forever or merged in when
> production ready?
> >
> > We have responded to all the above questions with detailed explanations
> and answers on the jira as well as in the discussions. We believe that
> should sufficiently address community’s concerns.
> >
> > Please see the summary below:
> >
> > 1) The new code base benefits HDFS scaling and a roadmap has been
> provided.
> >
> > Summary:
> > - New block storage layer addresses the scalability of the block layer.
> We have shown how existing NN can be connected to the new block layer and
> its benefits. We have shown 2 milestones, 1st milestone is much simpler
> than 2nd milestone while giving almost the same scaling benefits.
> Originally we had proposed simply milestone 2 and the community felt that
> removing the FSN/BM lock was was a fair amount of work and a simpler
> solution would be useful
> > - We provide a new K-V namespace called Ozone FS with
> FileSystem/FileContext plugins to allow the users to use the new system.
> BTW Hive and Spark work very well on KV-namespaces on the cloud. This will
> facilitate stabilizing the new block layer.
> > - The new block layer has a new netty based protocol engine in the
> Datanode which, when stabilized, can be used by  the old hdfs block layer.
> See details below on sharing of code.
> >
> >
> > 2) Stability impact on the existing HDFS code base and code separation.
> The new block layer and the OzoneFS are in modules that are separate from
> old HDFS code - currently there are no calls from HDFS into Ozone except
> for DN starting the new block  layer module if configured to do so. It does
> not add instability (the instability argument has been raised many times).
> Over time as we share code, we will ensure that the old HDFS continues to
> remains stable. (for example we plan to stabilize the new netty based
> protocol engine in the new block layer before sharing it with HDFS’s old
> block layer)
> >
> >
> > 3) In the short term and medium term, the new system and HDFS  will be
> used side-by-side by users. Side by-side usage in the short term for
> testing and side-by-side in the medium term for actual production use till
> the new system has feature parity with old HDFS. During this time, sharing
> the DN daemon and admin functions between the two systems is operationally
> important:
> > - Sharing DN daemon to avoid additional operational daemon lifecycle
> management
> > - Common decommissioning of the daemon and DN: One place to decommission
> for a node and its storage.
> > - Replacing failed disks and internal balancing capacity across disks -
> this needs to be done for both the current HDFS blocks and the new
> block-layer blocks.
> > - Balancer: we would like use the same balancer and provide a common way
> to balance and common management of the bandwidth used for balancing
> > - Security configuration setup - reuse existing set up for DNs rather
> then a new one for an independent cluster.
> >
> >
> > 4) Need to easily share the block layer code between the two systems
> when used side-by-side. Areas where sharing code is desired over time:
> > - Sharing new block layer’s  new netty based protocol engine for old
> HDFS DNs (a long time sore issue for HDFS block layer).
> > - Shallow data copy from old system to new system is practical only if
> within same project and daemon otherwise have to deal with security setting
> and coordinations across 

[jira] [Reopened] (HADOOP-12751) While using kerberos Hadoop incorrectly assumes names with '@' to be non-simple

[Konstantin Shvachko (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-12751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Konstantin Shvachko reopened HADOOP-12751:
--

> While using kerberos Hadoop incorrectly assumes names with '@' to be 
> non-simple
> ---
>
> Key: HADOOP-12751
> URL: https://issues.apache.org/jira/browse/HADOOP-12751
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: security
>Affects Versions: 2.7.2
> Environment: kerberos
>Reporter: Bolke de Bruin
>Assignee: Bolke de Bruin
>Priority: Critical
>  Labels: kerberos
> Fix For: 2.8.0, 3.0.0-alpha1
>
> Attachments: 0001-HADOOP-12751-leave-user-validation-to-os.patch, 
> 0001-Remove-check-for-user-name-characters-and.patch, 
> 0002-HADOOP-12751-leave-user-validation-to-os.patch, 
> 0003-HADOOP-12751-leave-user-validation-to-os.patch, 
> 0004-HADOOP-12751-leave-user-validation-to-os.patch, 
> 0005-HADOOP-12751-leave-user-validation-to-os.patch, 
> 0006-HADOOP-12751-leave-user-validation-to-os.patch, 
> 0007-HADOOP-12751-leave-user-validation-to-os.patch, 
> 0007-HADOOP-12751-leave-user-validation-to-os.patch, 
> 0008-HADOOP-12751-leave-user-validation-to-os.patch, 
> 0008-HADOOP-12751-leave-user-validation-to-os.patch, HADOOP-12751-009.patch
>
>
> In the scenario of a trust between two directories, eg. FreeIPA (ipa.local) 
> and Active Directory (ad.local) users can be made available on the OS level 
> by something like sssd. The trusted users will be of the form 'user@ad.local' 
> while other users are will not contain the domain. Executing 'id -Gn 
> user@ad.local' will successfully return the groups the user belongs to if 
> configured correctly. 
> However, it is assumed by Hadoop that users of the format with '@' cannot be 
> correct. This code is in KerberosName.java and seems to be a validator if the 
> 'auth_to_local' rules are applied correctly.
> In my opinion this should be removed or changed to a different kind of check 
> or maybe logged as a warning while still proceeding, as the current behavior 
> limits integration possibilities with other standard tools.
> Workaround are difficult to apply (by having a rewrite by system tools to for 
> example user_ad_local) due to down stream consequences.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[RESULT] [VOTE] Release Apache Hadoop 2.7.5 (RC1)

[Konstantin Shvachko]

Updating again to reflect all votes.

Correction:
With 8 binding and 4 non-binding +1s and no -1s the vote for Apache Release
2.7.5 passes.
Thank you everybody for contributing to the release, testing it, and voting.

Binding +1s
Kihwal Lee
Jason Lowe
John Zhuge
Rohith Sharma K S
Eric Payne
Zhe Zhang
Konstantin Shvachko
Naganarasimha Garla

Non-binding +1s
Erik Krogen
Brahma Reddy Battula
Eric Badger
Jonathan Hung



>> On Thu, Dec 7, 2017 at 7:22 PM, Konstantin Shvachko <shv.had...@gmail.com
>> > wrote:
>>
>>> Hi everybody,
>>>
>>> I updated CHANGES.txt and fixed documentation links.
>>> Also committed  MAPREDUCE-6165, which fixes a consistently failing test.
>>>
>>> This is RC1 for the next dot release of Apache Hadoop 2.7 line. The
>>> previous one 2.7.4 was release August 4, 2017.
>>> Release 2.7.5 includes critical bug fixes and optimizations. See more
>>> details in Release Note:
>>> http://home.apache.org/~shv/hadoop-2.7.5-RC1/releasenotes.html
>>>
>>> The RC0 is available at: http://home.apache.org/~shv/hadoop-2.7.5-RC1/
>>>
>>> Please give it a try and vote on this thread. The vote will run for 5
>>> days ending 12/13/2017.
>>>
>>> My up to date public key is available from:
>>> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>>>
>>> Thanks,
>>> --Konstantin
>>>
>>
>>
>

[RESULT] [VOTE] Release Apache Hadoop 2.7.5 (RC1)

[Konstantin Shvachko]

Correction:
With 7 binding and 4 non-binding +1s and no -1s the vote for Apache Release
2.7.5 passes.
Thank you everybody for contributing to the release, testing it, and voting.

Binding +1s
Kihwal Lee
Jason Lowe
John Zhuge
Rohith Sharma K S
Eric Payne
Zhe Zhang
Konstantin Shvachko

Non-binding +1s
Erik Krogen
Brahma Reddy Battula
Eric Badger
Jonathan Hung


On Fri, Dec 15, 2017 at 8:25 AM, Konstantin Shvachko <shv.had...@gmail.com>
wrote:

> Hi everybody,
>
> With 6 binding and 4 non-binding +1s and no -1s the vote for Apache
> Release 2.7.5 passes.
> Thank you everybody for contributing to the release, testing it, and
> voting.
>
> Binding +1s
> Kihwal Lee
> Jason Lowe
> John Zhuge
> Rohith Sharma K S
> Eric Payne
> Zhe Zhang
>
> Non-binding +1s
> Erik Krogen
> Brahma Reddy Battula
> Eric Badger
> Jonathan Hung
>
>
> On Thu, Dec 7, 2017 at 7:22 PM, Konstantin Shvachko <shv.had...@gmail.com>
> wrote:
>
>> Hi everybody,
>>
>> I updated CHANGES.txt and fixed documentation links.
>> Also committed  MAPREDUCE-6165, which fixes a consistently failing test.
>>
>> This is RC1 for the next dot release of Apache Hadoop 2.7 line. The
>> previous one 2.7.4 was release August 4, 2017.
>> Release 2.7.5 includes critical bug fixes and optimizations. See more
>> details in Release Note:
>> http://home.apache.org/~shv/hadoop-2.7.5-RC1/releasenotes.html
>>
>> The RC0 is available at: http://home.apache.org/~shv/hadoop-2.7.5-RC1/
>>
>> Please give it a try and vote on this thread. The vote will run for 5
>> days ending 12/13/2017.
>>
>> My up to date public key is available from:
>> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>>
>> Thanks,
>> --Konstantin
>>
>
>

[RESULT] [VOTE] Release Apache Hadoop 2.7.5 (RC1)

[Konstantin Shvachko]

Hi everybody,

With 6 binding and 4 non-binding +1s and no -1s the vote for Apache Release
2.7.5 passes.
Thank you everybody for contributing to the release, testing it, and voting.

Binding +1s
Kihwal Lee
Jason Lowe
John Zhuge
Rohith Sharma K S
Eric Payne
Zhe Zhang

Non-binding +1s
Erik Krogen
Brahma Reddy Battula
Eric Badger
Jonathan Hung


On Thu, Dec 7, 2017 at 7:22 PM, Konstantin Shvachko <shv.had...@gmail.com>
wrote:

> Hi everybody,
>
> I updated CHANGES.txt and fixed documentation links.
> Also committed  MAPREDUCE-6165, which fixes a consistently failing test.
>
> This is RC1 for the next dot release of Apache Hadoop 2.7 line. The
> previous one 2.7.4 was release August 4, 2017.
> Release 2.7.5 includes critical bug fixes and optimizations. See more
> details in Release Note:
> http://home.apache.org/~shv/hadoop-2.7.5-RC1/releasenotes.html
>
> The RC0 is available at: http://home.apache.org/~shv/hadoop-2.7.5-RC1/
>
> Please give it a try and vote on this thread. The vote will run for 5
> days ending 12/13/2017.
>
> My up to date public key is available from:
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
> Thanks,
> --Konstantin
>

Re: [VOTE] Release Apache Hadoop 2.7.5 (RC1)

[Konstantin Shvachko]

Here is my formal +1.

Thanks,
--Konstantin

On Thu, Dec 7, 2017 at 7:22 PM, Konstantin Shvachko <shv.had...@gmail.com>
wrote:

> Hi everybody,
>
> I updated CHANGES.txt and fixed documentation links.
> Also committed  MAPREDUCE-6165, which fixes a consistently failing test.
>
> This is RC1 for the next dot release of Apache Hadoop 2.7 line. The
> previous one 2.7.4 was release August 4, 2017.
> Release 2.7.5 includes critical bug fixes and optimizations. See more
> details in Release Note:
> http://home.apache.org/~shv/hadoop-2.7.5-RC1/releasenotes.html
>
> The RC0 is available at: http://home.apache.org/~shv/hadoop-2.7.5-RC1/
>
> Please give it a try and vote on this thread. The vote will run for 5
> days ending 12/13/2017.
>
> My up to date public key is available from:
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
> Thanks,
> --Konstantin
>

Re: [VOTE] Release Apache Hadoop 2.8.3 (RC0)

[Konstantin Shvachko]

Downloaded again, now the checksums look good. Sorry my fault

Thanks,
--Konstantin

On Mon, Dec 11, 2017 at 5:03 PM, Junping Du <j...@hortonworks.com> wrote:

> Hi Konstantin,
>
>  Thanks for verification and comments. I was verifying your example
> below but found it is actually matched:
>
>
> *jduMBP:hadoop-2.8.3 jdu$ md5 ~/Downloads/hadoop-2.8.3-src.tar.gz*
> *MD5 (/Users/jdu/Downloads/hadoop-2.8.3-src.tar.gz) =
> e53d04477b85e8b58ac0a26468f04736*
>
> What's your md5 checksum for given source tar ball?
>
>
> Thanks,
>
>
> Junping
>
>
> --
> *From:* Konstantin Shvachko <shv.had...@gmail.com>
> *Sent:* Saturday, December 9, 2017 11:06 AM
> *To:* Junping Du
> *Cc:* common-dev@hadoop.apache.org; hdfs-...@hadoop.apache.org;
> mapreduce-...@hadoop.apache.org; yarn-...@hadoop.apache.org
> *Subject:* Re: [VOTE] Release Apache Hadoop 2.8.3 (RC0)
>
> Hey Junping,
>
> Could you pls upload mds relative to the tar.gz etc. files rather than
> their full path
> /build/source/target/artifacts/hadoop-2.8.3-src.tar.gz:
>MD5 = E5 3D 04 47 7B 85 E8 B5  8A C0 A2 64 68 F0 47 36
>
> Otherwise mds don't match for me.
>
> Thanks,
> --Konstantin
>
> On Tue, Dec 5, 2017 at 1:58 AM, Junping Du <j...@hortonworks.com> wrote:
>
>> Hi all,
>>  I've created the first release candidate (RC0) for Apache Hadoop
>> 2.8.3. This is our next maint release to follow up 2.8.2. It includes 79
>> important fixes and improvements.
>>
>>   The RC artifacts are available at: http://home.apache.org/~junpin
>> g_du/hadoop-2.8.3-RC0
>>
>>   The RC tag in git is: release-2.8.3-RC0
>>
>>   The maven artifacts are available via repository.apache.org at:
>> https://repository.apache.org/content/repositories/orgapachehadoop-1072
>>
>>   Please try the release and vote; the vote will run for the usual 5
>> working days, ending on 12/12/2017 PST time.
>>
>> Thanks,
>>
>> Junping
>>
>
>

Re: [VOTE] Release Apache Hadoop 2.8.3 (RC0)

[Konstantin Shvachko]

Hey Junping,

Could you pls upload mds relative to the tar.gz etc. files rather than
their full path
/build/source/target/artifacts/hadoop-2.8.3-src.tar.gz:
   MD5 = E5 3D 04 47 7B 85 E8 B5  8A C0 A2 64 68 F0 47 36

Otherwise mds don't match for me.

Thanks,
--Konstantin

On Tue, Dec 5, 2017 at 1:58 AM, Junping Du  wrote:

> Hi all,
>  I've created the first release candidate (RC0) for Apache Hadoop
> 2.8.3. This is our next maint release to follow up 2.8.2. It includes 79
> important fixes and improvements.
>
>   The RC artifacts are available at: http://home.apache.org/~
> junping_du/hadoop-2.8.3-RC0
>
>   The RC tag in git is: release-2.8.3-RC0
>
>   The maven artifacts are available via repository.apache.org at:
> https://repository.apache.org/content/repositories/orgapachehadoop-1072
>
>   Please try the release and vote; the vote will run for the usual 5
> working days, ending on 12/12/2017 PST time.
>
> Thanks,
>
> Junping
>

[VOTE] Release Apache Hadoop 2.7.5 (RC1)

[Konstantin Shvachko]

Hi everybody,

I updated CHANGES.txt and fixed documentation links.
Also committed  MAPREDUCE-6165, which fixes a consistently failing test.

This is RC1 for the next dot release of Apache Hadoop 2.7 line. The
previous one 2.7.4 was release August 4, 2017.
Release 2.7.5 includes critical bug fixes and optimizations. See more
details in Release Note:
http://home.apache.org/~shv/hadoop-2.7.5-RC1/releasenotes.html

The RC0 is available at: http://home.apache.org/~shv/hadoop-2.7.5-RC1/

Please give it a try and vote on this thread. The vote will run for 5 days
ending 12/13/2017.

My up to date public key is available from:
https://dist.apache.org/repos/dist/release/hadoop/common/KEYS

Thanks,
--Konstantin

CANCELED: VOTE] Release Apache Hadoop 2.7.5 (RC0)

[Konstantin Shvachko]

Decided to cancel this release candidate.
Found that 9 commits in 2.7.5 are missing or misplaced in CHANGES.txt.
HADOOP-14867, HADOOP-14919, HDFS-12578, HDFS-12596, HDFS-8829,
MAPREDUCE-6937, MAPREDUCE-6975, YARN-6959, YARN-7084
Will take care of CHANGES.txt and documentation in the next RC.

Thanks,
--Konstantin

On Thu, Dec 7, 2017 at 10:17 AM, Konstantin Shvachko <shv.had...@gmail.com>
wrote:

> Hey Nagarasimha,
>
> Good find. thanks.
> Indeed we need those as they are linked from the side bar.
> I'll make sure to add CHANGES.txt when publishing new documentation.
>
> Thanks,
> --Konstantin
>
>
> On Wed, Dec 6, 2017 at 4:02 PM, Naganarasimha Garla <
> naganarasimha...@apache.org> wrote:
>
>> Thanks for the release Konstantin.
>>
>> Verified the following:
>> - Downloaded the tar on Ubuntu and verified the signatures
>> - Deployed pseudo cluster
>> - Sanity checks
>> - Basic hdfs operations
>> - Spark PyWordcount & few MR jobs
>> - Accessed most of the web UI's
>>
>> when accessing the docs(from the tar) was able to notice :
>> -  Release Notes, Common, HDFS, MapReduce Changes showing file
>> not found
>> -  I observed that changes for all components were not available
>> for 2.7.4 as well (http://hadoop.apache.org/docs
>> /r2.7.4/hadoop-project-dist/hadoop-common/CHANGES.txt)
>>
>> So not sure whether its missed or not required, else everything else is
>> fine.
>>
>> Regards,
>> + Naga
>>
>>
>> On Tue, Dec 5, 2017 at 2:50 PM, Brahma Reddy Battula <
>> brahmareddy.batt...@huawei.com> wrote:
>>
>>> +1  (non-binding), thanks Konstantin for driving this.
>>>
>>>
>>> --Built from the source
>>> --Installed 3 Node HA Cluster
>>> --Ran basic shell commands
>>> --Verified append/snapshot/truncate
>>> --Ran sample jobs like pi,wordcount
>>>
>>>
>>> Looks follow commits are missed in changes.txt.
>>>
>>> MAPREDUCE-6975
>>> HADOOP-14919
>>> HDFS-12596
>>> YARN-7084
>>> HADOOP-14881
>>> HADOOP-14827
>>> HDFS-12832
>>>
>>>
>>> --Brahma Reddy Battula
>>>
>>> -Original Message-
>>> From: Konstantin Shvachko [mailto:shv.had...@gmail.com]
>>> Sent: 02 December 2017 10:13
>>> To: common-dev@hadoop.apache.org; hdfs-...@hadoop.apache.org;
>>> mapreduce-...@hadoop.apache.org; yarn-...@hadoop.apache.org
>>> Subject: VOTE] Release Apache Hadoop 2.7.5 (RC0)
>>>
>>> Hi everybody,
>>>
>>> This is the next dot release of Apache Hadoop 2.7 line. The previous one
>>> 2.7.4 was release August 4, 2017.
>>> Release 2.7.5 includes critical bug fixes and optimizations. See more
>>> details in Release Note:
>>> http://home.apache.org/~shv/hadoop-2.7.5-RC0/releasenotes.html
>>>
>>> The RC0 is available at: http://home.apache.org/~shv/hadoop-2.7.5-RC0/
>>>
>>> Please give it a try and vote on this thread. The vote will run for 5
>>> days ending 12/08/2017.
>>>
>>> My up to date public key is available from:
>>> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>>>
>>> Thanks,
>>> --Konstantin
>>>
>>
>>
>

Re: VOTE] Release Apache Hadoop 2.7.5 (RC0)

[Konstantin Shvachko]

Hey Brahma,

Yes, I see some commits did not update CHANGES.txt. I see 2 options
1. Commit CHANGES.txt to 2.7.6, so that they are not lost, and make sure
they are available while publishing documentation for the release.
2. Update CHANGES.txt on 2.7.5, build and put RC1 for another vote.

LMK what people think?

Thanks,
--Konstantin

On Wed, Dec 6, 2017 at 2:46 PM, Erik Krogen <ekro...@linkedin.com> wrote:

> +1 (non-binding)
>
> - Verified signatures, MD5, RMD160, SHA* for bin and src tarballs
> - Built from source on macOS 10.12.6 and RHEL 6.6
> - Ran local HDFS cluster, ran basic commands, verified read and write
> capability.
> - Ran 3000 node cluster via Dynamometer and do not see significant
> performance variation from 2.7.4 expectations
>
> @Brahma, I was able to find HDFS-12831, HADOOP-14881, and HADOOP-14827 in
> CHANGES.txt, but agree with you on the others listed. I was, however, able
> to find all of them in the linked releasenotes.html.
>
> Thanks Konstantin!
>
> Erik
>
> On 12/4/17, 10:50 PM, "Brahma Reddy Battula" <brahmareddy.battula@huawei.
> com> wrote:
>
> +1  (non-binding), thanks Konstantin for driving this.
>
>
> --Built from the source
> --Installed 3 Node HA Cluster
> --Ran basic shell commands
> --Verified append/snapshot/truncate
> --Ran sample jobs like pi,wordcount
>
>
> Looks follow commits are missed in changes.txt.
>
> MAPREDUCE-6975
> HADOOP-14919
> HDFS-12596
> YARN-7084
> HADOOP-14881
> HADOOP-14827
> HDFS-12832
>
>
> --Brahma Reddy Battula
>
> -Original Message-
> From: Konstantin Shvachko [mailto:shv.had...@gmail.com]
> Sent: 02 December 2017 10:13
> To: common-dev@hadoop.apache.org; hdfs-...@hadoop.apache.org;
> mapreduce-...@hadoop.apache.org; yarn-...@hadoop.apache.org
> Subject: VOTE] Release Apache Hadoop 2.7.5 (RC0)
>
> Hi everybody,
>
> This is the next dot release of Apache Hadoop 2.7 line. The previous
> one
> 2.7.4 was release August 4, 2017.
> Release 2.7.5 includes critical bug fixes and optimizations. See more
> details in Release Note:
> http://home.apache.org/~shv/hadoop-2.7.5-RC0/releasenotes.html
>
> The RC0 is available at: http://home.apache.org/~shv/hadoop-2.7.5-RC0/
>
> Please give it a try and vote on this thread. The vote will run for 5
> days ending 12/08/2017.
>
> My up to date public key is available from:
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
> Thanks,
> --Konstantin
>
>
>

Re: VOTE] Release Apache Hadoop 2.7.5 (RC0)

[Konstantin Shvachko]

Hey Nagarasimha,

Good find. thanks.
Indeed we need those as they are linked from the side bar.
I'll make sure to add CHANGES.txt when publishing new documentation.

Thanks,
--Konstantin


On Wed, Dec 6, 2017 at 4:02 PM, Naganarasimha Garla <
naganarasimha...@apache.org> wrote:

> Thanks for the release Konstantin.
>
> Verified the following:
> - Downloaded the tar on Ubuntu and verified the signatures
> - Deployed pseudo cluster
> - Sanity checks
> - Basic hdfs operations
> - Spark PyWordcount & few MR jobs
> - Accessed most of the web UI's
>
> when accessing the docs(from the tar) was able to notice :
> -  Release Notes, Common, HDFS, MapReduce Changes showing file not
> found
> -  I observed that changes for all components were not available
> for 2.7.4 as well (http://hadoop.apache.org/docs/r2.7.4/hadoop-project-
> dist/hadoop-common/CHANGES.txt)
>
> So not sure whether its missed or not required, else everything else is
> fine.
>
> Regards,
> + Naga
>
>
> On Tue, Dec 5, 2017 at 2:50 PM, Brahma Reddy Battula <
> brahmareddy.batt...@huawei.com> wrote:
>
>> +1  (non-binding), thanks Konstantin for driving this.
>>
>>
>> --Built from the source
>> --Installed 3 Node HA Cluster
>> --Ran basic shell commands
>> --Verified append/snapshot/truncate
>> --Ran sample jobs like pi,wordcount
>>
>>
>> Looks follow commits are missed in changes.txt.
>>
>> MAPREDUCE-6975
>> HADOOP-14919
>> HDFS-12596
>> YARN-7084
>> HADOOP-14881
>> HADOOP-14827
>> HDFS-12832
>>
>>
>> --Brahma Reddy Battula
>>
>> -Original Message-
>> From: Konstantin Shvachko [mailto:shv.had...@gmail.com]
>> Sent: 02 December 2017 10:13
>> To: common-dev@hadoop.apache.org; hdfs-...@hadoop.apache.org;
>> mapreduce-...@hadoop.apache.org; yarn-...@hadoop.apache.org
>> Subject: VOTE] Release Apache Hadoop 2.7.5 (RC0)
>>
>> Hi everybody,
>>
>> This is the next dot release of Apache Hadoop 2.7 line. The previous one
>> 2.7.4 was release August 4, 2017.
>> Release 2.7.5 includes critical bug fixes and optimizations. See more
>> details in Release Note:
>> http://home.apache.org/~shv/hadoop-2.7.5-RC0/releasenotes.html
>>
>> The RC0 is available at: http://home.apache.org/~shv/hadoop-2.7.5-RC0/
>>
>> Please give it a try and vote on this thread. The vote will run for 5
>> days ending 12/08/2017.
>>
>> My up to date public key is available from:
>> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>>
>> Thanks,
>> --Konstantin
>>
>
>

VOTE] Release Apache Hadoop 2.7.5 (RC0)

[Konstantin Shvachko]

Hi everybody,

This is the next dot release of Apache Hadoop 2.7 line. The previous one
2.7.4 was release August 4, 2017.
Release 2.7.5 includes critical bug fixes and optimizations. See more
details in Release Note:
http://home.apache.org/~shv/hadoop-2.7.5-RC0/releasenotes.html

The RC0 is available at: http://home.apache.org/~shv/hadoop-2.7.5-RC0/

Please give it a try and vote on this thread. The vote will run for 5 days
ending 12/08/2017.

My up to date public key is available from:
https://dist.apache.org/repos/dist/release/hadoop/common/KEYS

Thanks,
--Konstantin

Re: [VOTE] Release Apache Hadoop 2.9.0 (RC3)

[Konstantin Shvachko]

Thanks for the links, Arun. I see I missed this part.
Glad the decision was conscious.
Thanks for the release.
--Konstantin

On Wed, Nov 22, 2017 at 11:39 AM, Arun Suresh <arun.sur...@gmail.com> wrote:

> Hello Konstantin
>
> We actually did discuss a bit about this:
> https://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg28407.html
> https://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg28403.html
>
> We finally decided to add the following disclaimer on the release page (
> https://hadoop.apache.org/releases.html):
> "Please note: Although this release has been tested on fairly large
> clusters, production users can wait for a subsequent point release which
> will contain fixes from further stabilization and downstream adoption."
>
> Hope this suffices.
>
>
>
>
> On Wed, Nov 22, 2017 at 10:30 AM, Konstantin Shvachko <
> shv.had...@gmail.com> wrote:
>
>> Hey guys,
>>
>> I don't think this has been discussed, pardon if it was.
>> As it stands today hadoop 2.9.0 is marked as stable release. Isn't that
>> deceptive for users?
>> Not to diminish the quality and not to understate the effort, which was
>> huge and very much appreciated.
>> But it is the first in the series, with quite a few experimental features,
>> and big delta from the last stable, ...
>>
>> Thanks,
>> --Konstantin
>>
>>
>> On Fri, Nov 17, 2017 at 1:56 PM, Subru Krishnan <su...@apache.org> wrote:
>>
>> > Wrapping up the vote with my +1.
>> >
>> > Deployed RC3 on a federated YARN cluster with 6 sub-clusters:
>> > - ran multiple sample jobs
>> > - enabled opportunistic containers and submitted more samples
>> > - configured HDFS federation and reran jobs.
>> >
>> >
>> > With 13 binding +1s and 7 non-binding +1s and no +/-1s, pleased to
>> announce
>> > the vote is passed successfully.
>> >
>> > Thanks to the many of you who contributed to the release and made this
>> > possible and to everyone in this thread who took the time/effort to
>> > validate and vote!
>> >
>> > We’ll push the release bits and send out an announcement for 2.9.0 soon.
>> >
>> > Cheers,
>> > Subru
>> >
>> > On Fri, Nov 17, 2017 at 12:41 PM Eric Payne <eric.payne1...@yahoo.com>
>> > wrote:
>> >
>> > > Thanks Arun and Subru for the hard work on this release.
>> > >
>> > > +1 (binding)
>> > >
>> > > Built from source and stood up a pseudo cluster with 4 NMs
>> > >
>> > > Tested the following:
>> > >
>> > > o User limits are honored during In-queue preemption
>> > >
>> > > o Priorities are honored during In-queue preemption
>> > >
>> > > o Can kill applications from the command line
>> > >
>> > > o Users with different weights are assigned resources proportional to
>> > > their weights.
>> > >
>> > > Thanks,
>> > > -Eric Payne
>> > >
>> > >
>> > > --
>> > > *From:* Arun Suresh <asur...@apache.org>
>> > > *To:* yarn-...@hadoop.apache.org; mapreduce-...@hadoop.apache.org;
>> > Hadoop
>> > > Common <common-dev@hadoop.apache.org>; Hdfs-dev <
>> > > hdfs-...@hadoop.apache.org>
>> > > *Cc:* Subramaniam Krishnan <su...@apache.org>
>> > > *Sent:* Monday, November 13, 2017 6:10 PM
>> > >
>> > > *Subject:* [VOTE] Release Apache Hadoop 2.9.0 (RC3)
>> > >
>> > > Hi Folks,
>> > >
>> > > Apache Hadoop 2.9.0 is the first release of Hadoop 2.9 line and will
>> be
>> > the
>> > > starting release for Apache Hadoop 2.9.x line - it includes 30 New
>> > Features
>> > > with 500+ subtasks, 407 Improvements, 790 Bug fixes new fixed issues
>> > since
>> > > 2.8.2.
>> > >
>> > > More information about the 2.9.0 release plan can be found here:
>> > > *
>> > > https://cwiki.apache.org/confluence/display/HADOOP/
>> > Roadmap#Roadmap-Version2.9
>> > > <
>> > > https://cwiki.apache.org/confluence/display/HADOOP/
>> > Roadmap#Roadmap-Version2.9
>> > > >*
>> > >
>> > > New RC is available at: *
>> > > https://home.apache.org/~asuresh/hadoop-2.9.0-RC3/
>> > > <https://home.apache.org/~asuresh/hadoop-2.9.0-RC3/>*
>> > >
>> > > The RC tag in git is: release-2.9.0-RC3, and the latest commit id is:
>> > > 756ebc8394e473ac25feac05fa493f6d612e6c50.
>> > >
>> > > The maven artifacts are available via repository.apache.org at:
>> > > <
>> > > https://www.google.com/url?q=https%3A%2F%2Frepository.
>> > apache.org%2Fcontent%2Frepositories%2Forgapachehadoop-1066=D&
>> > sntz=1=AFQjCNFcern4uingMV_sEreko_zeLlgdlg
>> > > >*https://repository.apache.org/content/repositories/
>> > orgapachehadoop-1068/
>> > > <https://repository.apache.org/content/repositories/
>> > orgapachehadoop-1068/
>> > > >*
>> > >
>> > > We are carrying over the votes from the previous RC given that the
>> delta
>> > is
>> > > the license fix.
>> > >
>> > > Given the above - we are also going to stick with the original
>> deadline
>> > for
>> > > the vote : ending on Friday 17th November 2017 2pm PT time.
>> > >
>> > > Thanks,
>> > > -Arun/Subru
>> > >
>> > >
>> > >
>> >
>>
>
>

Re: [VOTE] Release Apache Hadoop 2.9.0 (RC3)

[Konstantin Shvachko]

Hey guys,

I don't think this has been discussed, pardon if it was.
As it stands today hadoop 2.9.0 is marked as stable release. Isn't that
deceptive for users?
Not to diminish the quality and not to understate the effort, which was
huge and very much appreciated.
But it is the first in the series, with quite a few experimental features,
and big delta from the last stable, ...

Thanks,
--Konstantin


On Fri, Nov 17, 2017 at 1:56 PM, Subru Krishnan  wrote:

> Wrapping up the vote with my +1.
>
> Deployed RC3 on a federated YARN cluster with 6 sub-clusters:
> - ran multiple sample jobs
> - enabled opportunistic containers and submitted more samples
> - configured HDFS federation and reran jobs.
>
>
> With 13 binding +1s and 7 non-binding +1s and no +/-1s, pleased to announce
> the vote is passed successfully.
>
> Thanks to the many of you who contributed to the release and made this
> possible and to everyone in this thread who took the time/effort to
> validate and vote!
>
> We’ll push the release bits and send out an announcement for 2.9.0 soon.
>
> Cheers,
> Subru
>
> On Fri, Nov 17, 2017 at 12:41 PM Eric Payne 
> wrote:
>
> > Thanks Arun and Subru for the hard work on this release.
> >
> > +1 (binding)
> >
> > Built from source and stood up a pseudo cluster with 4 NMs
> >
> > Tested the following:
> >
> > o User limits are honored during In-queue preemption
> >
> > o Priorities are honored during In-queue preemption
> >
> > o Can kill applications from the command line
> >
> > o Users with different weights are assigned resources proportional to
> > their weights.
> >
> > Thanks,
> > -Eric Payne
> >
> >
> > --
> > *From:* Arun Suresh 
> > *To:* yarn-...@hadoop.apache.org; mapreduce-...@hadoop.apache.org;
> Hadoop
> > Common ; Hdfs-dev <
> > hdfs-...@hadoop.apache.org>
> > *Cc:* Subramaniam Krishnan 
> > *Sent:* Monday, November 13, 2017 6:10 PM
> >
> > *Subject:* [VOTE] Release Apache Hadoop 2.9.0 (RC3)
> >
> > Hi Folks,
> >
> > Apache Hadoop 2.9.0 is the first release of Hadoop 2.9 line and will be
> the
> > starting release for Apache Hadoop 2.9.x line - it includes 30 New
> Features
> > with 500+ subtasks, 407 Improvements, 790 Bug fixes new fixed issues
> since
> > 2.8.2.
> >
> > More information about the 2.9.0 release plan can be found here:
> > *
> > https://cwiki.apache.org/confluence/display/HADOOP/
> Roadmap#Roadmap-Version2.9
> > <
> > https://cwiki.apache.org/confluence/display/HADOOP/
> Roadmap#Roadmap-Version2.9
> > >*
> >
> > New RC is available at: *
> > https://home.apache.org/~asuresh/hadoop-2.9.0-RC3/
> > *
> >
> > The RC tag in git is: release-2.9.0-RC3, and the latest commit id is:
> > 756ebc8394e473ac25feac05fa493f6d612e6c50.
> >
> > The maven artifacts are available via repository.apache.org at:
> > <
> > https://www.google.com/url?q=https%3A%2F%2Frepository.
> apache.org%2Fcontent%2Frepositories%2Forgapachehadoop-1066=D&
> sntz=1=AFQjCNFcern4uingMV_sEreko_zeLlgdlg
> > >*https://repository.apache.org/content/repositories/
> orgapachehadoop-1068/
> >  orgapachehadoop-1068/
> > >*
> >
> > We are carrying over the votes from the previous RC given that the delta
> is
> > the license fix.
> >
> > Given the above - we are also going to stick with the original deadline
> for
> > the vote : ending on Friday 17th November 2017 2pm PT time.
> >
> > Thanks,
> > -Arun/Subru
> >
> >
> >
>

Re: Apache Hadoop 2.8.3 Release Plan

[Konstantin Shvachko]

I would consider these two blockers for 2.8.3 as they crash NN:

https://issues.apache.org/jira/browse/HDFS-12638
https://issues.apache.org/jira/browse/HDFS-12832

Thanks,
--Konstantin

On Tue, Nov 21, 2017 at 11:16 AM, Junping Du  wrote:

> Thanks Andrew and Wangda for comments!
>
> To me, an improvement with 17 patches is not a big problem as this is
> self-contained and I didn't see a single line of delete/update on existing
> code - well, arguably, patches with only adding code can also have big
> impact but not likely the case here.
>
> While the dependency discussions on HADOOP-14964 are still going on, I
> will leave the decision to JIRA discussion based on which approach we will
> choose(shaded?) and impact. If we cannot make consensus in short term,
> probably we have to miss this in 2.8.3 release.
>
>
> Okay. Last call for blocker/critical fixes landing on branch-2.8.3. RC0
> will get cut-off shortly.
>
>
>
> Thanks,
>
>
> Junping
>
>
> 
> From: Wangda Tan 
> Sent: Tuesday, November 21, 2017 10:52 AM
> To: Andrew Wang
> Cc: Junping Du; Zheng, Kai; common-dev@hadoop.apache.org;
> hdfs-...@hadoop.apache.org; mapreduce-...@hadoop.apache.org;
> yarn-...@hadoop.apache.org
> Subject: Re: Apache Hadoop 2.8.3 Release Plan
>
> Thanks Junping for driving this.
>
> For the bug fix vs. improvement, it is actually very hard to define,
> improvement could be self-contained and useful, bug fix could be dangerous
> in some cases. To me, If an improvement fixed some existing use case, and
> the fix is self-contained. I will be open to bring such fix to maintenance
> release. For example, in 2.8.2, we back ported CapacityScheduler intra
> queue preemption https://issues.apache.org/jira/browse/YARN-2113. It is a
> big change in terms of patch size, but since it fixes broken use case
> (balance user usage under Capacity Scheduler leaf queue), we backported it
> to 2.8.2 after thorough tests and validations by Yahoo.
>
> I'm not quite familiar with HADOOP-14964, I will leave the decision to
> committers who know more about the field.
>
> Just my 2 cents.
>
> Regards,
> Wangda
>
>
> On Tue, Nov 21, 2017 at 10:21 AM, Andrew Wang  mailto:andrew.w...@cloudera.com>> wrote:
> The Aliyun OSS code isn't a small improvement. If you look at Sammi's
> comment
>  focusedCommentId=16247085=com.atlassian.jira.
> plugin.system.issuetabpanels:comment-tabpanel#comment-16247085>,
> it's
> a 17 patch series that is being backported in one shot. What we're talking
> about is equivalent to merging a feature branch in a maintenance release. I
> see that Kai and Chris are having a discussion about the dependency
> changes, which indicates this is not a zero-risk change either. We really
> should not be changing dependency versions in a maintenance unless it's
> because of a bug.
>
> It's unfortunate from a timing perspective that this missed 2.9.0, but I
> still think it should wait for the next minor. Merging a feature into a
> maintenance release sets the wrong precedent.
>
> Best,
> Andrew
>
> On Tue, Nov 21, 2017 at 1:08 AM, Junping Du  u...@hortonworks.com>> wrote:
>
> > Thanks Kai for calling out this feature/improvement for attention and
> > Andrew for comments.
> >
> >
> > While I agree that maintenance release should focus on important bug fix
> > only, I doubt we have strict rules to disallow any features/improvements
> to
> > land on maint release especially when those are small footprint or low
> > impact on existing code/features. In practice, we indeed has 77 new
> > features/improvements in latest 2.7.3 and 2.7.4 release.
> >
> >
> > Back to HADOOP-14964, I did a quick check and it looks like case here
> > belongs to self-contained improvement that has very low impact on
> existing
> > code base, so I am OK with the improvement get landed on branch-2.8 in
> case
> > it is well reviewed and tested.
> >
> >
> > However, as RM of branch-2.8, I have two concerns to accept it in our
> > 2.8.3 release:
> >
> > 1. Timing - as I mentioned in beginning, the main purpose of 2.8.3 are
> for
> > several critical bug fixes and we should target to release it very soon -
> > my current plan is to cut RC out within this week inline with waiting
> > for 3.0.0 vote closing. Can this improvement be well tested against
> > branch-2.8.3 within this strictly timeline? It seems a bit rush unless we
> > have strong commitment on test plan and activities in such a tight time.
> >
> >
> > 2. Upgrading - I haven't heard we settle down the plan of releasing this
> > feature in 2.9.1 release - though I saw some discussions are going on
> > at HADOOP-14964. Assume 2.8.3 is released ahead of 2.9.1 and it includes
> > this improvement, then users consuming this feature/improvement have no
> 2.9
> > release to upgrade or forcefully upgrade with regression. We may need a

Re: [DISCUSS] Apache Hadoop 2.7.5 Release Plan

[Konstantin Shvachko]

I created a filter to track the release issues.
https://issues.apache.org/jira/issues/?filter=12342617

You should set Target version = 2.7.5 and Label issue "release-blocker"

Thanks,
--Konstantin

On Fri, Nov 17, 2017 at 8:48 AM, Wei-Chiu Chuang <weic...@cloudera.com>
wrote:

> Hi Konstantin,
> Thanks for initiating the release effort.
>
> I am marking HDFS-12641 <https://issues.apache.org/jira/browse/HDFS-12641> as
> a blocker for Hadoop 2.7.5 because during our internal testing for CDH we
> found a regression in HDFS-11445 that was fixed by HDFS-11755 (technically
> not a real regression since HDFS-11755 was committed before HDFS-11445).
> The regression results in bogus corrupt block reports. It is not clear to
> me if the same behavior is in Apache Hadoop, but since the later
> (HDFS-11755) is currently Hadoop 2.8.x and above, I would want to be more
> cautious about it.
>
> On Thu, Nov 16, 2017 at 5:20 PM, Konstantin Shvachko <shv.had...@gmail.com
> > wrote:
>
>> Hi developers,
>>
>> We have accumulated about 30 commits on branch-2.7. Those are mostly
>> valuable bug fixes, minor optimizations and test corrections. I would like
>> to propose to make a quick maintenance release 2.7.5.
>>
>> If there are no objections I'll start preparations.
>>
>> Thanks,
>> --Konstantin
>>
>
>
>
> --
> A very happy Clouderan
>

[DISCUSS] Apache Hadoop 2.7.5 Release Plan

[Konstantin Shvachko]

Hi developers,

We have accumulated about 30 commits on branch-2.7. Those are mostly
valuable bug fixes, minor optimizations and test corrections. I would like
to propose to make a quick maintenance release 2.7.5.

If there are no objections I'll start preparations.

Thanks,
--Konstantin

Re: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk

[Konstantin Shvachko]

Hi Sanjay,

Read your doc. I clearly see the value of Ozone with your use cases, but I
agree with Stack and others the question why it should be a part of Hadoop
isn't clear. More details in the jira:

https://issues.apache.org/jira/browse/HDFS-7240?focusedCommentId=16239313=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16239313

Thanks,
--Konstantin

On Fri, Nov 3, 2017 at 1:56 PM, sanjay Radia <sanjayo...@gmail.com> wrote:

> Konstantine,
>  Thanks for your comments, questions and feedback. I have attached a
> document to the HDFS-7240 jira
>  that explains a design for scaling HDFS and how Ozone paves the way
> towards the full solution.
>
>
> https://issues.apache.org/jira/secure/attachment/
> 12895963/HDFS%20Scalability%20and%20Ozone.pdf
>
>
> sanjay
>
>
>
>
> On Oct 28, 2017, at 2:00 PM, Konstantin Shvachko <shv.had...@gmail.com>
> wrote:
>
> Hey guys,
>
> It is an interesting question whether Ozone should be a part of Hadoop.
> There are two main reasons why I think it should not.
>
> 1. With close to 500 sub-tasks, with 6 MB of code changes, and with a
> sizable community behind, it looks to me like a whole new project.
> It is essentially a new storage system, with different (than HDFS)
> architecture, separate S3-like APIs. This is really great - the World sure
> needs more distributed file systems. But it is not clear why Ozone should
> co-exist with HDFS under the same roof.
>
> 2. Ozone is probably just the first step in rebuilding HDFS under a new
> architecture. With the next steps presumably being HDFS-10419 and
> HDFS-8.
> The design doc for the new architecture has never been published. I can
> only assume based on some presentations and personal communications that
> the idea is to use Ozone as a block storage, and re-implement NameNode, so
> that it stores only a partial namesapce in memory, while the bulk of it
> (cold data) is persisted to a local storage.
> Such architecture makes me wonder if it solves Hadoop's main problems.
> There are two main limitations in HDFS:
>  a. The throughput of Namespace operations. Which is limited by the number
> of RPCs the NameNode can handle
>  b. The number of objects (files + blocks) the system can maintain. Which
> is limited by the memory size of the NameNode.
> The RPC performance (a) is more important for Hadoop scalability than the
> object count (b). The read RPCs being the main priority.
> The new architecture targets the object count problem, but in the expense
> of the RPC throughput. Which seems to be a wrong resolution of the
> tradeoff.
> Also based on the use patterns on our large clusters we read up to 90% of
> the data we write, so cold data is a small fraction and most of it must be
> cached.
>
> To summarize:
> - Ozone is a big enough system to deserve its own project.
> - The architecture that Ozone leads to does not seem to solve the intrinsic
> problems of current HDFS.
>
> I will post my opinion in the Ozone jira. Should be more convenient to
> discuss it there for further reference.
>
> Thanks,
> --Konstantin
>
>
>
> On Wed, Oct 18, 2017 at 6:54 PM, Yang Weiwei <cheersy...@hotmail.com>
> wrote:
>
> Hello everyone,
>
>
> I would like to start this thread to discuss merging Ozone (HDFS-7240) to
> trunk. This feature implements an object store which can co-exist with
> HDFS. Ozone is disabled by default. We have tested Ozone with cluster sizes
> varying from 1 to 100 data nodes.
>
>
>
> The merge payload includes the following:
>
>  1.  All services, management scripts
>  2.  Object store APIs, exposed via both REST and RPC
>  3.  Master service UIs, command line interfaces
>  4.  Pluggable pipeline Integration
>  5.  Ozone File System (Hadoop compatible file system implementation,
> passes all FileSystem contract tests)
>  6.  Corona - a load generator for Ozone.
>  7.  Essential documentation added to Hadoop site.
>  8.  Version specific Ozone Documentation, accessible via service UI.
>  9.  Docker support for ozone, which enables faster development cycles.
>
>
> To build Ozone and run ozone using docker, please follow instructions in
> this wiki page. https://cwiki.apache.org/confl
> uence/display/HADOOP/Dev+cluster+with+docker.
>
>
> We have built a passionate and diverse community to drive this feature
> development. As a team, we have achieved significant progress in past 3
> years since first JIRA for HDFS-7240 was opened on Oct 2014. So far, we
> have resolved almost 400 JIRAs by 20+ contributors/committers from
> different countries and affiliations. We also want to thank the large
> number of community members who were supportive of our efforts and
> contr

Re: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk

[Konstantin Shvachko]

Hey guys,

It is an interesting question whether Ozone should be a part of Hadoop.
There are two main reasons why I think it should not.

1. With close to 500 sub-tasks, with 6 MB of code changes, and with a
sizable community behind, it looks to me like a whole new project.
It is essentially a new storage system, with different (than HDFS)
architecture, separate S3-like APIs. This is really great - the World sure
needs more distributed file systems. But it is not clear why Ozone should
co-exist with HDFS under the same roof.

2. Ozone is probably just the first step in rebuilding HDFS under a new
architecture. With the next steps presumably being HDFS-10419 and
HDFS-8.
The design doc for the new architecture has never been published. I can
only assume based on some presentations and personal communications that
the idea is to use Ozone as a block storage, and re-implement NameNode, so
that it stores only a partial namesapce in memory, while the bulk of it
(cold data) is persisted to a local storage.
Such architecture makes me wonder if it solves Hadoop's main problems.
There are two main limitations in HDFS:
  a. The throughput of Namespace operations. Which is limited by the number
of RPCs the NameNode can handle
  b. The number of objects (files + blocks) the system can maintain. Which
is limited by the memory size of the NameNode.
The RPC performance (a) is more important for Hadoop scalability than the
object count (b). The read RPCs being the main priority.
The new architecture targets the object count problem, but in the expense
of the RPC throughput. Which seems to be a wrong resolution of the tradeoff.
Also based on the use patterns on our large clusters we read up to 90% of
the data we write, so cold data is a small fraction and most of it must be
cached.

To summarize:
- Ozone is a big enough system to deserve its own project.
- The architecture that Ozone leads to does not seem to solve the intrinsic
problems of current HDFS.

I will post my opinion in the Ozone jira. Should be more convenient to
discuss it there for further reference.

Thanks,
--Konstantin



On Wed, Oct 18, 2017 at 6:54 PM, Yang Weiwei  wrote:

> Hello everyone,
>
>
> I would like to start this thread to discuss merging Ozone (HDFS-7240) to
> trunk. This feature implements an object store which can co-exist with
> HDFS. Ozone is disabled by default. We have tested Ozone with cluster sizes
> varying from 1 to 100 data nodes.
>
>
>
> The merge payload includes the following:
>
>   1.  All services, management scripts
>   2.  Object store APIs, exposed via both REST and RPC
>   3.  Master service UIs, command line interfaces
>   4.  Pluggable pipeline Integration
>   5.  Ozone File System (Hadoop compatible file system implementation,
> passes all FileSystem contract tests)
>   6.  Corona - a load generator for Ozone.
>   7.  Essential documentation added to Hadoop site.
>   8.  Version specific Ozone Documentation, accessible via service UI.
>   9.  Docker support for ozone, which enables faster development cycles.
>
>
> To build Ozone and run ozone using docker, please follow instructions in
> this wiki page. https://cwiki.apache.org/confl
> uence/display/HADOOP/Dev+cluster+with+docker.
>
>
> We have built a passionate and diverse community to drive this feature
> development. As a team, we have achieved significant progress in past 3
> years since first JIRA for HDFS-7240 was opened on Oct 2014. So far, we
> have resolved almost 400 JIRAs by 20+ contributors/committers from
> different countries and affiliations. We also want to thank the large
> number of community members who were supportive of our efforts and
> contributed ideas and participated in the design of ozone.
>
>
> Please share your thoughts, thanks!
>
>
> -- Weiwei Yang
>


On Wed, Oct 18, 2017 at 6:54 PM, Yang Weiwei  wrote:

> Hello everyone,
>
>
> I would like to start this thread to discuss merging Ozone (HDFS-7240) to
> trunk. This feature implements an object store which can co-exist with
> HDFS. Ozone is disabled by default. We have tested Ozone with cluster sizes
> varying from 1 to 100 data nodes.
>
>
>
> The merge payload includes the following:
>
>   1.  All services, management scripts
>   2.  Object store APIs, exposed via both REST and RPC
>   3.  Master service UIs, command line interfaces
>   4.  Pluggable pipeline Integration
>   5.  Ozone File System (Hadoop compatible file system implementation,
> passes all FileSystem contract tests)
>   6.  Corona - a load generator for Ozone.
>   7.  Essential documentation added to Hadoop site.
>   8.  Version specific Ozone Documentation, accessible via service UI.
>   9.  Docker support for ozone, which enables faster development cycles.
>
>
> To build Ozone and run ozone using docker, please follow instructions in
> this wiki page. https://cwiki.apache.org/confluence/display/HADOOP/Dev+
> cluster+with+docker.
>
>
> We have built a passionate and 

Re: [VOTE] Merge feature branch YARN-5734 (API based scheduler configuration) to trunk, branch-3.0, branch-2

[Konstantin Shvachko]

+1 (binding)

I was following this effort closely from the side.
The feature proved to be very useful based on our internal use of it.
I think it will be beneficial for the community as well.
Thanks for working on this Jonathan, Wangda, and everybody involved.

--Konstantin

On Mon, Oct 2, 2017 at 11:09 AM, Jonathan Hung <jyhung2...@gmail.com> wrote:

> Hi all,
>
> From discussion at [1], I'd like to start a vote to merge feature branch
> YARN-5734 to trunk, branch-3.0, and branch-2. Vote will be 7 days, ending
> Monday Oct 9 at 11:00AM PDT.
>
> This branch adds a framework to the scheduler to allow scheduler
> configuration mutation on the fly, including a REST and CLI interface, and
> an interface for the scheduler configuration backing store. Currently the
> capacity scheduler implements this framework.
>
> Umbrella is here (YARN-5734
> <https://issues.apache.org/jira/browse/YARN-5734>), jenkins build is here
> (
> YARN-7241 <https://issues.apache.org/jira/browse/YARN-7241>). All required
> tasks for this feature are committed. Since this feature changes RM only,
> we have tested this on a local RM setup with a suite of configuration
> changes with no issue so far.
>
> Key points:
> - The feature is turned off by default, and must be explicitly configured
> to turn on. When turned off, the behavior reverts back to the original file
> based mechanism for changing scheduler configuration (i.e. yarn rmadmin
> -refreshQueues).
> - The framework was designed in a way to be extendable to other schedulers
> (most notably FairScheduler).
> - A pluggable ACL policy (YARN-5949
> <https://issues.apache.org/jira/browse/YARN-5949>) allows admins
> fine-grained control for who can change what configurations.
> - The configuration storage backend is also pluggable. Currently an
> in-memory, leveldb, and zookeeper implementation are supported.
>
> There were 15 subtasks completed for this feature.
>
> Huge thanks to everyone who helped with reviews, commits, guidance, and
> technical discussion/design, including Carlo Curino, Xuan Gong, Subru
> Krishnan, Min Shen, Konstantin Shvachko, Carl Steinbach, Wangda Tan, Vinod
> Kumar Vavilapalli, Suja Viswesan, Zhe Zhang, Ye Zhou.
>
> [1]
> http://mail-archives.apache.org/mod_mbox/hadoop-yarn-dev/201709.mbox/%
> 3CCAHzWLgfEAgczjcEOUCg-03ma3ROtO=pkec9dpggyx9rzf3n...@mail.gmail.com%3E
>
> Jonathan Hung
>

[RESULT] [VOTE] Release Apache Hadoop 2.7.4 (RC0)

[Konstantin Shvachko]

My formal vote
+1 (binding)

I am glad to summarize that
with 7 binding and 13 non-binding +1s and no -1s the vote for Apache
Release 2.7.4 passes.
Thank you everybody for contributing to the release, testing it, and voting.

Binding +1s (7)
Zhe Zhang
Jason Lowe
Eric Payne
Sunil G
Akira Ajisaka
Chris Douglas
Konstantin Shvachko

Non-binding +1s (13)
John Zhuge
Surendra Lilhore
Masatake Iwasaki
Hanisha Koneru
Chen Liang
Fnu Ajay Kumar
Brahma Reddy Battula
Edwina Lu
Ye Zhou
Eric Badger
Mingliang Liu
Kuhu Shukla
Erik Krogen

Thanks,
--Konstantin

On Sat, Jul 29, 2017 at 4:29 PM, Konstantin Shvachko <shv.had...@gmail.com>
wrote:

> Hi everybody,
>
> Here is the next release of Apache Hadoop 2.7 line. The previous stable
> release 2.7.3 was available since 25 August, 2016.
> Release 2.7.4 includes 264 issues fixed after release 2.7.3, which are
> critical bug fixes and major optimizations. See more details in Release
> Note:
> http://home.apache.org/~shv/hadoop-2.7.4-RC0/releasenotes.html
>
> The RC0 is available at: http://home.apache.org/~shv/hadoop-2.7.4-RC0/
>
> Please give it a try and vote on this thread. The vote will run for 5 days
> ending 08/04/2017.
>
> Please note that my up to date public key are available from:
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> Please don't forget to refresh the page if you've been there recently.
> There are other place on Apache sites, which may contain my outdated key.
>
> Thanks,
> --Konstantin
>

Re: [VOTE] Release Apache Hadoop 2.7.4 (RC0)

[Konstantin Shvachko]

Hi Wei-Chi,

There is bunch of performance jiras in Hadoop.
This one is not marked as blocker and seems to be hard to reproduce.
And I see that you are still discussing the problem and the solution with
Daryn in the jira.
I don't think it should block the release.
If this or anything else will prove to be a serious problem we can follow
up with a maintenance release containing hotfixes only.

Thanks,
--Konst

On Fri, Aug 4, 2017 at 9:24 AM, Wei-Chiu Chuang <weic...@cloudera.com>
wrote:

> Hi,
> I'm sorry coming to this vote late.
> Daryn mentioned in HDFS-12136
> <https://issues.apache.org/jira/browse/HDFS-12136> that HDFS-11160
> <https://issues.apache.org/jira/browse/HDFS-11160> has a performance
> regression at DataNode due to the way it locks dataset lock.
>
> HDFS-11160 is in Hadoop 2.7.4. Would it be critical enough to warrant a
> stop for release?
>
> I myself can't reproduce the performance regression (assuming it only
> occurs under extreme workload). Would Daryn or other Yahoo folks comment?
>
> On Thu, Aug 3, 2017 at 10:31 PM, Akira Ajisaka <aajis...@apache.org>
> wrote:
>
>> +1 (binding)
>>
>> - Verified the checksum and the signature of the source tarball
>> - Built from source with CentOS 7.2 and OpenJDK 1.8.0_141
>> - Built Hive 2.1.0/2.3.0 and Tez 0.8.5/0.9.0 with Hadoop 2.7.4 artifacts
>> - Built single node cluster and ran some Hive on Tez queries successfully
>>
>> Regards,
>> Akira
>>
>>
>> On 2017/08/04 0:25, Kuhu Shukla wrote:
>>
>>> +1 (non-binding)
>>>
>>> 1. Verified signatures and digests.
>>> 2. Built source.
>>> 3. Installed on a pseudo-distributed cluster.
>>> 4. Ran sample MR jobs and Tez example jobs like orderedwordcount
>>> successfully.
>>>
>>> Thank you Konstantin and others for this release.
>>>
>>> Regards,
>>> Kuhu
>>>
>>>
>>>
>>> On Thursday, August 3, 2017, 7:19:07 AM CDT, Sunil G <sun...@apache.org>
>>> wrote:
>>>
>>>
>>> Thanks Konstantin
>>>
>>> +1 (binding)
>>>
>>> 1. Build tar ball from source package
>>> 2. Ran basic MR jobs and verified UI.
>>> 3. Enabled node labels and ran sleep job. Works fine.
>>> 4. Verified CLI commands related to node labels and its working fine.
>>> 5. RM WorkPreserving restart cases are also verified, and looks fine
>>>
>>> Thanks
>>> Sunil
>>>
>>>
>>>
>>> On Sun, Jul 30, 2017 at 4:59 AM Konstantin Shvachko <
>>> shv.had...@gmail.com>
>>> wrote:
>>>
>>> Hi everybody,
>>>>
>>>> Here is the next release of Apache Hadoop 2.7 line. The previous stable
>>>> release 2.7.3 was available since 25 August, 2016.
>>>> Release 2.7.4 includes 264 issues fixed after release 2.7.3, which are
>>>> critical bug fixes and major optimizations. See more details in Release
>>>> Note:
>>>> http://home.apache.org/~shv/hadoop-2.7.4-RC0/releasenotes.html
>>>>
>>>> The RC0 is available at: http://home.apache.org/~shv/hadoop-2.7.4-RC0/
>>>>
>>>> Please give it a try and vote on this thread. The vote will run for 5
>>>> days
>>>> ending 08/04/2017.
>>>>
>>>> Please note that my up to date public key are available from:
>>>> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>>>> Please don't forget to refresh the page if you've been there recently.
>>>> There are other place on Apache sites, which may contain my outdated
>>>> key.
>>>>
>>>> Thanks,
>>>> --Konstantin
>>>>
>>>>
>> -
>> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
>> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
>>
>>
>
>
> --
> A very happy Clouderan
>

Re: [VOTE] Release Apache Hadoop 2.7.4 (RC0)

[Konstantin Shvachko]

Hi Akira,

The artifacts are on Nexus in Staging Repositories:
https://repository.apache.org/#stagingRepositories
or
https://repository.apache.org/content/repositories/orgapachehadoop-1061/

They are not released yet, which happens once the release is approved.
See in Publishing section #5
https://wiki.apache.org/hadoop/HowToReleasePreDSBCR#Publishing

Thanks,
--Konst

On Wed, Aug 2, 2017 at 11:36 PM, Akira Ajisaka <aajis...@apache.org> wrote:

> Thanks Konstantin for the work!
> I have a question: Where are the maven artifacts deployed?
>
> -Akira
>
> On 2017/07/30 8:29, Konstantin Shvachko wrote:
>
>> Hi everybody,
>>
>> Here is the next release of Apache Hadoop 2.7 line. The previous stable
>> release 2.7.3 was available since 25 August, 2016.
>> Release 2.7.4 includes 264 issues fixed after release 2.7.3, which are
>> critical bug fixes and major optimizations. See more details in Release
>> Note:
>> http://home.apache.org/~shv/hadoop-2.7.4-RC0/releasenotes.html
>>
>> The RC0 is available at: http://home.apache.org/~shv/hadoop-2.7.4-RC0/
>>
>> Please give it a try and vote on this thread. The vote will run for 5 days
>> ending 08/04/2017.
>>
>> Please note that my up to date public key are available from:
>> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>> Please don't forget to refresh the page if you've been there recently.
>> There are other place on Apache sites, which may contain my outdated key.
>>
>> Thanks,
>> --Konstantin
>>
>>

Re: Are binary artifacts are part of a release?

[Konstantin Shvachko]

It does not. Just adding historical references, as Andrew raised the
question.

On Mon, Jul 31, 2017 at 7:38 PM, Allen Wittenauer <a...@effectivemachines.com>
wrote:

>
> ... that doesn't contradict anything I said.
>
> > On Jul 31, 2017, at 7:23 PM, Konstantin Shvachko <shv.had...@gmail.com>
> wrote:
> >
> > The issue was discussed on several occasions in the past.
> > Took me a while to dig this out as an example:
> > http://mail-archives.apache.org/mod_mbox/hadoop-general/
> 20.mbox/%3C4EB0827C.6040204%40apache.org%3E
> >
> > Doug Cutting:
> > "Folks should not primarily evaluate binaries when voting. The ASF
> primarily produces and publishes source-code
> > so voting artifacts should be optimized for evaluation of that."
> >
> > Thanks,
> > --Konst
> >
> > On Mon, Jul 31, 2017 at 4:51 PM, Allen Wittenauer <
> a...@effectivemachines.com> wrote:
> >
> > > On Jul 31, 2017, at 4:18 PM, Andrew Wang <andrew.w...@cloudera.com>
> wrote:
> > >
> > > Forking this off to not distract from release activities.
> > >
> > > I filed https://issues.apache.org/jira/browse/LEGAL-323 to get
> clarity on the matter. I read the entire webpage, and it could be improved
> one way or the other.
> >
> >
> > IANAL, my read has always lead me to believe:
> >
> > * An artifact is anything that is uploaded to dist.a.o
> and repository.a.o
> > * A release consists of one or more artifacts ("Releases
> are, by definition, anything that is published beyond the group that owns
> it. In our case, that means any publication outside the group of people on
> the product dev list.")
> > * One of those artifacts MUST be source
> > * (insert voting rules here)
> > * They must be built on a machine in control of the RM
> > * There are no exceptions for alpha, nightly, etc
> > * (various other requirements)
> >
> > i.e., release != artifact  it's more like release =
> artifact * n .
> >
> > Do you have to have binaries?  No (e.g., Apache SpamAssassin has
> no binaries to create).  But if you place binaries in dist.a.o or
> repository.a.o, they are effectively part of your release and must follow
> the same rules.  (Votes, etc.)
> >
> >
>
>

Re: Are binary artifacts are part of a release?

[Konstantin Shvachko]

The issue was discussed on several occasions in the past.
Took me a while to dig this out as an example:
http://mail-archives.apache.org/mod_mbox/hadoop-general/20.mbox/%3C4EB0827C.6040204%40apache.org%3E

Doug Cutting:
"Folks should not primarily evaluate binaries when voting. The ASF
primarily produces and publishes source-code
so voting artifacts should be optimized for evaluation of that."

Thanks,
--Konst

On Mon, Jul 31, 2017 at 4:51 PM, Allen Wittenauer 
wrote:

>
> > On Jul 31, 2017, at 4:18 PM, Andrew Wang 
> wrote:
> >
> > Forking this off to not distract from release activities.
> >
> > I filed https://issues.apache.org/jira/browse/LEGAL-323 to get clarity
> on the matter. I read the entire webpage, and it could be improved one way
> or the other.
>
>
> IANAL, my read has always lead me to believe:
>
> * An artifact is anything that is uploaded to dist.a.o and
> repository.a.o
> * A release consists of one or more artifacts ("Releases
> are, by definition, anything that is published beyond the group that owns
> it. In our case, that means any publication outside the group of people on
> the product dev list.")
> * One of those artifacts MUST be source
> * (insert voting rules here)
> * They must be built on a machine in control of the RM
> * There are no exceptions for alpha, nightly, etc
> * (various other requirements)
>
> i.e., release != artifact  it's more like release =
> artifact * n .
>
> Do you have to have binaries?  No (e.g., Apache SpamAssassin has
> no binaries to create).  But if you place binaries in dist.a.o or
> repository.a.o, they are effectively part of your release and must follow
> the same rules.  (Votes, etc.)
>
>

Re: [VOTE] Release Apache Hadoop 2.7.4 (RC0)

[Konstantin Shvachko]

Uploaded new binaries hadoop-2.7.4-RC0.tar.gz, which adds lib/native/.
Same place: http://home.apache.org/~shv/hadoop-2.7.4-RC0/

Thanks,
--Konstantin

On Mon, Jul 31, 2017 at 3:56 PM, Chris Douglas <cdoug...@apache.org> wrote:

> On Mon, Jul 31, 2017 at 3:02 PM, Konstantin Shvachko
> <shv.had...@gmail.com> wrote:
> > For the packaging, here is the exact phrasing from the sited
> release-policy
> > document relevant to binaries:
> > "As a convenience to users that might not have the appropriate tools to
> > build a compiled version of the source, binary/bytecode packages MAY be
> > distributed alongside official Apache releases. In all such cases, the
> > binary/bytecode package MUST have the same version number as the source
> > release and MUST only add binary/bytecode files that are the result of
> > compiling that version of the source code release and its dependencies."
> > I don't think my binary package violates any of these.
>
> +1 The PMC VOTE applies to source code, only. If someone wants to
> rebuild the binary tarball with native libs and replace this one,
> that's fine.
>
> My reading of the above is that source code must be distributed with
> binaries, not that we omit the source code from binary releases... -C
>
> > But I'll upload an additional tar.gz with native bits and no src, as you
> > guys requested.
> > Will keep it as RC0 as there is no source code change and it comes from
> the
> > same build.
> > Hope this is satisfactory.
> >
> > Thanks,
> > --Konstantin
> >
> > On Mon, Jul 31, 2017 at 1:53 PM, Andrew Wang <andrew.w...@cloudera.com>
> > wrote:
> >
> >> I agree with Brahma on the two issues flagged (having src in the binary
> >> tarball, missing native libs). These are regressions from prior
> releases.
> >>
> >> As an aside, "we release binaries as a convenience" doesn't relax the
> >> quality bar. The binaries are linked on our website and distributed
> through
> >> official Apache channels. They have to adhere to Apache release
> >> requirements. And, most users consume our work via Maven dependencies,
> >> which are binary artifacts.
> >>
> >> http://www.apache.org/legal/release-policy.html goes into this in more
> >> detail. A release must minimally include source packages, and can also
> >> include binary artifacts.
> >>
> >> Best,
> >> Andrew
> >>
> >> On Mon, Jul 31, 2017 at 12:30 PM, Konstantin Shvachko <
> >> shv.had...@gmail.com> wrote:
> >>
> >>> To avoid any confusion in this regard. I built RC0 manually in
> compliance
> >>> with Apache release policy
> >>> http://www.apache.org/legal/release-policy.html
> >>> I edited the HowToReleasePreDSBCR page to make sure people don't use
> >>> Jenkins option for building.
> >>>
> >>> A side note. This particular build is broken anyways, so no worries
> there.
> >>> I think though it would be useful to have it working for testing and
> as a
> >>> packaging standard.
> >>>
> >>> Thanks,
> >>> --Konstantin
> >>>
> >>> On Mon, Jul 31, 2017 at 11:40 AM, Allen Wittenauer <
> >>> a...@effectivemachines.com
> >>> > wrote:
> >>>
> >>> >
> >>> > > On Jul 31, 2017, at 11:20 AM, Konstantin Shvachko <
> >>> shv.had...@gmail.com>
> >>> > wrote:
> >>> > >
> >>> > > https://wiki.apache.org/hadoop/HowToReleasePreDSBCR
> >>> >
> >>> > FYI:
> >>> >
> >>> > If you are using ASF Jenkins to create an ASF release
> >>> > artifact, it's pretty much an automatic vote failure as any such
> >>> release is
> >>> > in violation of ASF policy.
> >>> >
> >>> >
> >>>
> >>
> >>
>

Re: [VOTE] Release Apache Hadoop 2.7.4 (RC0)

[Konstantin Shvachko]

Maven dependencies should be fine.

For the packaging, here is the exact phrasing from the sited release-policy
document relevant to binaries:
"As a convenience to users that might not have the appropriate tools to
build a compiled version of the source, binary/bytecode packages MAY be
distributed alongside official Apache releases. In all such cases, the
binary/bytecode package MUST have the same version number as the source
release and MUST only add binary/bytecode files that are the result of
compiling that version of the source code release and its dependencies."
I don't think my binary package violates any of these.

But I'll upload an additional tar.gz with native bits and no src, as you
guys requested.
Will keep it as RC0 as there is no source code change and it comes from the
same build.
Hope this is satisfactory.

Thanks,
--Konstantin

On Mon, Jul 31, 2017 at 1:53 PM, Andrew Wang <andrew.w...@cloudera.com>
wrote:

> I agree with Brahma on the two issues flagged (having src in the binary
> tarball, missing native libs). These are regressions from prior releases.
>
> As an aside, "we release binaries as a convenience" doesn't relax the
> quality bar. The binaries are linked on our website and distributed through
> official Apache channels. They have to adhere to Apache release
> requirements. And, most users consume our work via Maven dependencies,
> which are binary artifacts.
>
> http://www.apache.org/legal/release-policy.html goes into this in more
> detail. A release must minimally include source packages, and can also
> include binary artifacts.
>
> Best,
> Andrew
>
> On Mon, Jul 31, 2017 at 12:30 PM, Konstantin Shvachko <
> shv.had...@gmail.com> wrote:
>
>> To avoid any confusion in this regard. I built RC0 manually in compliance
>> with Apache release policy
>> http://www.apache.org/legal/release-policy.html
>> I edited the HowToReleasePreDSBCR page to make sure people don't use
>> Jenkins option for building.
>>
>> A side note. This particular build is broken anyways, so no worries there.
>> I think though it would be useful to have it working for testing and as a
>> packaging standard.
>>
>> Thanks,
>> --Konstantin
>>
>> On Mon, Jul 31, 2017 at 11:40 AM, Allen Wittenauer <
>> a...@effectivemachines.com
>> > wrote:
>>
>> >
>> > > On Jul 31, 2017, at 11:20 AM, Konstantin Shvachko <
>> shv.had...@gmail.com>
>> > wrote:
>> > >
>> > > https://wiki.apache.org/hadoop/HowToReleasePreDSBCR
>> >
>> > FYI:
>> >
>> > If you are using ASF Jenkins to create an ASF release
>> > artifact, it's pretty much an automatic vote failure as any such
>> release is
>> > in violation of ASF policy.
>> >
>> >
>>
>
>

Re: [VOTE] Release Apache Hadoop 2.7.4 (RC0)

[Konstantin Shvachko]

To avoid any confusion in this regard. I built RC0 manually in compliance
with Apache release policy
http://www.apache.org/legal/release-policy.html
I edited the HowToReleasePreDSBCR page to make sure people don't use
Jenkins option for building.

A side note. This particular build is broken anyways, so no worries there.
I think though it would be useful to have it working for testing and as a
packaging standard.

Thanks,
--Konstantin

On Mon, Jul 31, 2017 at 11:40 AM, Allen Wittenauer <a...@effectivemachines.com
> wrote:

>
> > On Jul 31, 2017, at 11:20 AM, Konstantin Shvachko <shv.had...@gmail.com>
> wrote:
> >
> > https://wiki.apache.org/hadoop/HowToReleasePreDSBCR
>
> FYI:
>
> If you are using ASF Jenkins to create an ASF release
> artifact, it's pretty much an automatic vote failure as any such release is
> in violation of ASF policy.
>
>

Re: [VOTE] Release Apache Hadoop 2.7.4 (RC0)

[Konstantin Shvachko]

Hi John,

Thank you for checking and voting.
As far as I know test failures on 2.7.4 are intermittent. We have a jira
for that
https://issues.apache.org/jira/browse/HDFS-11985
but decided it should not block the release.
The "dr,who" thing is a configuration issue. This page may be helpful:
http://hadoop.apache.org/docs/stable/hadoop-hdfs-httpfs/ServerSetup.html

Thanks,
--Konstantin

On Sun, Jul 30, 2017 at 11:24 PM, John Zhuge <john.zh...@gmail.com> wrote:

> Hi Konstantin,
>
> Thanks a lot for the effort to prepare the 2.7.4-RC0 release!
>
> +1 (non-binding)
>
>- Verified checksums and signatures of all tarballs
>- Built source with native, Java 1.8.0_131-b11 on Mac OS X 10.12.6
>- Verified cloud connectors:
>   - All S3A integration tests
>- Deployed both binary and built source to a pseudo cluster, passed
>the following sanity tests in insecure, SSL, and SSL+Kerberos mode:
>   - HDFS basic and ACL
>   - DistCp basic
>   - MapReduce wordcount (only failed in SSL+Kerberos mode for binary
>   tarball, probably unrelated)
>   - KMS and HttpFS basic
>   - Balancer start/stop
>
> Shall we worry this test failures? Likely fixed by
> https://issues.apache.org/jira/browse/HADOOP-13707.
>
>- Got “curl: (22) The requested URL returned error: 403 User dr.who is
>unauthorized to access this page.” when accessing NameNode web servlet
>/jmx, /conf, /logLevel, and /stacks. It passed in branch-2.8.
>
>
> On Sat, Jul 29, 2017 at 4:29 PM, Konstantin Shvachko <shv.had...@gmail.com
> > wrote:
>
>> Hi everybody,
>>
>> Here is the next release of Apache Hadoop 2.7 line. The previous stable
>> release 2.7.3 was available since 25 August, 2016.
>> Release 2.7.4 includes 264 issues fixed after release 2.7.3, which are
>> critical bug fixes and major optimizations. See more details in Release
>> Note:
>> http://home.apache.org/~shv/hadoop-2.7.4-RC0/releasenotes.html
>>
>> The RC0 is available at: http://home.apache.org/~shv/hadoop-2.7.4-RC0/
>>
>> Please give it a try and vote on this thread. The vote will run for 5 days
>> ending 08/04/2017.
>>
>> Please note that my up to date public key are available from:
>> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>> Please don't forget to refresh the page if you've been there recently.
>> There are other place on Apache sites, which may contain my outdated key.
>>
>> Thanks,
>> --Konstantin
>>
>
>
>
> --
> John
>

Re: [VOTE] Release Apache Hadoop 2.7.4 (RC0)

[Konstantin Shvachko]

Hi Brahma Reddy Battula,

Formally Apache releases sources. We provide binaries as a reference for
convenience.
The release instructions for Hadoop 2.7 line at
https://wiki.apache.org/hadoop/HowToReleasePreDSBCR
don't give much guidance on how to actually build and package the binary
tarball.
I intentionally included sources as the main content of the release.
And did not include native binaries, which very much depend on the
environment.
LMK if binary packaging is a blocker for you.

Thanks,
--Konstantin

On Sun, Jul 30, 2017 at 7:45 PM, Brahma Reddy Battula <
brahmareddy.batt...@huawei.com> wrote:

> Thanks konstantin.
>
> Native files (lib/native) is missed
> And src folder exists in tar ball.
>
> Please check the same.
>
> Host1:/home/Brahma/Release/hadoop-2.7.4 # ll
> total 144
> -rw-r--r--  1 20415 messagebus 86424 Jul 30 02:54 LICENSE.txt
> -rw-r--r--  1 20415 messagebus 14978 Jul 30 02:54 NOTICE.txt
> -rw-r--r--  1 20415 messagebus  1366 Jul 30 02:54 README.txt
> drwxr-xr-x  2 20415 messagebus  4096 Jul 30 02:54 bin
> drwxr-xr-x  3 20415 messagebus  4096 Jul 30 02:54 etc
> -rw-r--r--  1 20415 messagebus  1683 Jul 30 03:15 hadoop-client.list
> drwxr-xr-x  2 20415 messagebus  4096 Jul 30 02:54 include
> drwxr-xr-x  2 20415 messagebus  4096 Jul 30 02:54 libexec
> drwxr-xr-x  2 20415 messagebus  4096 Jul 30 02:54 sbin
> drwxr-xr-x  4 20415 messagebus  4096 Jul 30 02:54 share
> drwxr-xr-x 19 20415 messagebus  4096 Jul 30 03:01 src
>
>
>
> --Brahma Reddy Battula
>
> -Original Message-
> From: Konstantin Shvachko [mailto:shv.had...@gmail.com]
> Sent: 30 July 2017 07:29
> To: common-dev@hadoop.apache.org; hdfs-...@hadoop.apache.org;
> mapreduce-...@hadoop.apache.org; yarn-...@hadoop.apache.org
> Subject: [VOTE] Release Apache Hadoop 2.7.4 (RC0)
>
> Hi everybody,
>
> Here is the next release of Apache Hadoop 2.7 line. The previous stable
> release 2.7.3 was available since 25 August, 2016.
> Release 2.7.4 includes 264 issues fixed after release 2.7.3, which are
> critical bug fixes and major optimizations. See more details in Release
> Note:
> http://home.apache.org/~shv/hadoop-2.7.4-RC0/releasenotes.html
>
> The RC0 is available at: http://home.apache.org/~shv/hadoop-2.7.4-RC0/
>
> Please give it a try and vote on this thread. The vote will run for 5 days
> ending 08/04/2017.
>
> Please note that my up to date public key are available from:
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> Please don't forget to refresh the page if you've been there recently.
> There are other place on Apache sites, which may contain my outdated key.
>
> Thanks,
> --Konstantin
>

[VOTE] Release Apache Hadoop 2.7.4 (RC0)

[Konstantin Shvachko]

Hi everybody,

Here is the next release of Apache Hadoop 2.7 line. The previous stable
release 2.7.3 was available since 25 August, 2016.
Release 2.7.4 includes 264 issues fixed after release 2.7.3, which are
critical bug fixes and major optimizations. See more details in Release
Note:
http://home.apache.org/~shv/hadoop-2.7.4-RC0/releasenotes.html

The RC0 is available at: http://home.apache.org/~shv/hadoop-2.7.4-RC0/

Please give it a try and vote on this thread. The vote will run for 5 days
ending 08/04/2017.

Please note that my up to date public key are available from:
https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
Please don't forget to refresh the page if you've been there recently.
There are other place on Apache sites, which may contain my outdated key.

Thanks,
--Konstantin

Re: Moved remaining 16 Jiras targeted for version 2.7.4 to 2.7.5

[Konstantin Shvachko]

And some more.

On Thu, Jul 27, 2017 at 5:11 PM, Konstantin Shvachko <shv.had...@gmail.com>
wrote:

> Thanks,
> --Konst
>

Moved remaining 16 Jiras targeted for version 2.7.4 to 2.7.5

[Konstantin Shvachko]

Thanks,
--Konst

Re: About 2.7.4 Release

[Konstantin Shvachko]

Hey guys,

Looks like we are done with blockers for Apache Hadoop 2.7.4 release.
https://issues.apache.org/jira/issues/?filter=12340814

I just committed HDFS-11896
<https://issues.apache.org/jira/browse/HDFS-11896>, and decided not to wait
for HDFS-11576 <https://issues.apache.org/jira/browse/HDFS-11576>, see Jira
comment.
Thanks Vinod for pointing out HDFS-11742
<https://issues.apache.org/jira/browse/HDFS-11742>. It was thoroughly
tested on a small cluster and Kihwal committed it last week, thanks.

Will start building initial RC. Please refrain from committing to
branch-2.7 for some time.

Thank you everybody for contributing.
--Konst

On Thu, Jul 20, 2017 at 12:07 PM, Vinod Kumar Vavilapalli <
vino...@apache.org> wrote:

> Thanks for taking 2.7.4 over Konstantin!
>
> Regarding rolling RC next week, I still see that there are 4 blocker /
> critical tickets targeted for 2.7.4: https://issues.apache.
> org/jira/issues/?jql=project%20in%20(HDFS%2C%20MAPREDUCE%
> 2C%20HADOOP%2C%20YARN)%20AND%20priority%20in%20(Blocker%2C%
> 20Critical)%20AND%20resolution%20%3D%20Unresolved%20AND%20%
> 22Target%20Version%2Fs%22%20%3D%202.7.4
> <https://issues.apache.org/jira/issues/?jql=project%20in%20(HDFS,%20MAPREDUCE,%20HADOOP,%20YARN)%20AND%20priority%20in%20(Blocker,%20Critical)%20AND%20resolution%20=%20Unresolved%20AND%20%22Target%20Version/s%22%20=%202.7.4>
> .
>
> We should get closure on them. https://issues.apache.
> org/jira/browse/HDFS-11742 definitely was something that was deemed a
> blocker for 2.8.2, not sure about 2.7.4.
>
> I’m ‘back’ - let me know if you need any help.
>
> Thanks
> +Vinod
>
> On Jul 13, 2017, at 5:45 PM, Konstantin Shvachko <shv.had...@gmail.com>
> wrote:
>
> Hi everybody.
>
> We have been doing some internal testing of Hadoop 2.7.4. The testing is
> going well.
> Did not find any major issues on our workloads.
> Used an internal tool called Dynamometer to check NameNode performance on
> real cluster traces. Good.
> Overall test cluster performance looks good.
> Some more testing is still going on.
>
> I plan to build an RC next week. If there are no objection.
>
> Thanks,
> --Konst
>
> On Thu, Jun 15, 2017 at 4:42 PM, Konstantin Shvachko <shv.had...@gmail.com
> >
> wrote:
>
> Hey guys.
>
> An update on 2.7.4 progress.
> We are down to 4 blockers. There is some work remaining on those.
> https://issues.apache.org/jira/browse/HDFS-11896?filter=12340814
> Would be good if people could follow up on review comments.
>
> I looked through nightly Jenkins build results for 2.7.4 both on Apache
> Jenkins and internal.
> Some test fail intermittently, but there no consistent failures. I filed
> HDFS-11985 to track some of them.
> https://issues.apache.org/jira/browse/HDFS-11985
> I do not currently consider these failures as blockers. LMK if some of
> them are.
>
> We started internal testing of branch-2.7 on one of our smallish (100+
> nodes) test clusters.
> Will update on the results.
>
> There is a plan to enable BigTop for 2.7.4 testing.
>
> Akira, Brahma thank you for setting up a wiki page for 2.7.4 release.
> Thank you everybody for contributing to this effort.
>
> Regards,
> --Konstantin
>
>
> On Tue, May 30, 2017 at 12:08 AM, Akira Ajisaka <aajis...@apache.org>
> wrote:
>
> Sure.
> If you want to edit the wiki, please tell me your ASF confluence account.
>
> -Akira
>
> On 2017/05/30 15:31, Rohith Sharma K S wrote:
>
> Couple of more JIRAs need to be back ported for 2.7.4 release. These will
> solve RM HA unstability issues.
> https://issues.apache.org/jira/browse/YARN-5333
> https://issues.apache.org/jira/browse/YARN-5988
> https://issues.apache.org/jira/browse/YARN-6304
>
> I will raise a JIRAs to back port it.
>
> @Akira , could  you help to add these JIRAs into wiki?
>
> Thanks & Regards
> Rohith Sharma K S
>
> On 29 May 2017 at 12:19, Akira Ajisaka <aajis...@apache.org> wrote:
>
> Created a page for 2.7.4 release.
>
> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.7.4
>
> If you want to edit this wiki, please ping me.
>
> Regards,
> Akira
>
>
> On 2017/05/23 4:42, Brahma Reddy Battula wrote:
>
> Hi Konstantin Shvachko
>
>
>
> how about creating a wiki page for 2.7.4 release status like 2.8 and
> trunk in following link.??
>
>
> https://cwiki.apache.org/confluence/display/HADOOP
>
>
> 
> From: Konstantin Shvachko <shv.had...@gmail.com>
> Sent: Saturday, May 13, 2017 3:58 AM
> To: Akira Ajisaka
> Cc: Hadoop Common; Hdfs-dev; mapreduce-...@hadoop.apache.org;
> yarn-...@hadoop.apache.org
> Subject: Re: About

[jira] [Created] (HADOOP-14689) Dummy jira for testing barnch-2.7 build.

[Konstantin Shvachko (JIRA)]

Konstantin Shvachko created HADOOP-14689:


 Summary: Dummy jira for testing barnch-2.7 build.
 Key: HADOOP-14689
 URL: https://issues.apache.org/jira/browse/HADOOP-14689
 Project: Hadoop Common
  Issue Type: Task
Reporter: Konstantin Shvachko
Assignee: Konstantin Shvachko


Will add dummy patches to test Jenkins PreCommit-HADOOP-Build failures on 
branch-2.7.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Resolved] (HADOOP-14686) Branch-2.7 .gitignore is out of date

[Konstantin Shvachko (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-14686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Konstantin Shvachko resolved HADOOP-14686.
--
  Resolution: Fixed
Hadoop Flags: Reviewed

Also updated CHANGES.txt.
I just committed this to branch 2.7. Thank you [~busbey].

> Branch-2.7 .gitignore is out of date
> 
>
> Key: HADOOP-14686
> URL: https://issues.apache.org/jira/browse/HADOOP-14686
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: build, precommit
>Affects Versions: 2.7.4
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Blocker
> Fix For: 2.7.4
>
> Attachments: HADOOP-14686-branch-2.7.v0.patch, 
> HADOOP-14686-branch-2.7.v1.patch
>
>
> .gitignore is out of date on branch-2.7, which is causing issues in precommit 
> checks for that branch.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Reopened] (HADOOP-14686) Branch-2.7 .gitignore is out of date

[Konstantin Shvachko (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-14686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Konstantin Shvachko reopened HADOOP-14686:
--

Will have to revert this, because it also removed {{contract-test-options.xml}} 
from .gitignore, which could be considered as incompatible change for 2.7.4.

> Branch-2.7 .gitignore is out of date
> 
>
> Key: HADOOP-14686
> URL: https://issues.apache.org/jira/browse/HADOOP-14686
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: build, precommit
>Affects Versions: 2.7.4
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Blocker
> Fix For: 2.7.4
>
> Attachments: HADOOP-14686-branch-2.7.v0.patch
>
>
> .gitignore is out of date on branch-2.7, which is causing issues in precommit 
> checks for that branch.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: Pre-Commit build is failing

[Konstantin Shvachko]

Hi Yetus developers,

We cannot build Hadoop branch-2.7 anymore. Here is a recent example of a
failed build:
https://builds.apache.org/job/PreCommit-HDFS-Build/20409/console

It seems the build is failing because Yetus cannot apply the patch from the
jira.

ERROR: HDFS-11896 does not apply to branch-2.7.

As far as I understand this is Yetus problem. Probably in 0.3.0.
I can apply this patch successfully, but Yetus test-patch.sh script clearly
failed to apply. Cannot say why because Yetus does not report it.
I also ran Hadoop's test-patch.sh script locally and it passed successfully
on branch-2.7.

Could anybody please take a look and help fixing the build.
This would be very helpful for the release (2.7.4) process.

Thanks,
--Konst

On Mon, Jul 24, 2017 at 10:41 PM, Konstantin Shvachko <shv.had...@gmail.com>
wrote:

> Or should we backport the entire HADOOP-11917
> <https://issues.apache.org/jira/browse/HADOOP-11917> ?
>
> Thanks,
> --Konst
>
> On Mon, Jul 24, 2017 at 6:56 PM, Konstantin Shvachko <shv.had...@gmail.com
> > wrote:
>
>> Allen,
>>
>> Should we add "patchprocess/" to .gitignore, is that the problem for 2.7?
>>
>> Thanks,
>> --Konstantin
>>
>> On Fri, Jul 21, 2017 at 6:24 PM, Konstantin Shvachko <
>> shv.had...@gmail.com> wrote:
>>
>>> What stuff? Is there a jira?
>>> It did work like a week ago. Is it a new Yetus requirement.
>>> Anyways I can commit a change to fix the build on our side.
>>> Just need to know what is missing.
>>>
>>> Thanks,
>>> --Konst
>>>
>>> On Fri, Jul 21, 2017 at 5:50 PM, Allen Wittenauer <
>>> a...@effectivemachines.com> wrote:
>>>
>>>>
>>>> > On Jul 21, 2017, at 5:46 PM, Konstantin Shvachko <
>>>> shv.had...@gmail.com> wrote:
>>>> >
>>>> > + d...@yetus.apache.org
>>>> >
>>>> > Guys, could you please take a look. Seems like Yetus problem with
>>>> > pre-commit build for branch-2.7.
>>>>
>>>>
>>>> branch-2.7 is missing stuff in .gitignore.
>>>
>>>
>>>
>>
>

Re: Pre-Commit build is failing

[Konstantin Shvachko]

Or should we backport the entire HADOOP-11917
<https://issues.apache.org/jira/browse/HADOOP-11917> ?

Thanks,
--Konst

On Mon, Jul 24, 2017 at 6:56 PM, Konstantin Shvachko <shv.had...@gmail.com>
wrote:

> Allen,
>
> Should we add "patchprocess/" to .gitignore, is that the problem for 2.7?
>
> Thanks,
> --Konstantin
>
> On Fri, Jul 21, 2017 at 6:24 PM, Konstantin Shvachko <shv.had...@gmail.com
> > wrote:
>
>> What stuff? Is there a jira?
>> It did work like a week ago. Is it a new Yetus requirement.
>> Anyways I can commit a change to fix the build on our side.
>> Just need to know what is missing.
>>
>> Thanks,
>> --Konst
>>
>> On Fri, Jul 21, 2017 at 5:50 PM, Allen Wittenauer <
>> a...@effectivemachines.com> wrote:
>>
>>>
>>> > On Jul 21, 2017, at 5:46 PM, Konstantin Shvachko <shv.had...@gmail.com>
>>> wrote:
>>> >
>>> > + d...@yetus.apache.org
>>> >
>>> > Guys, could you please take a look. Seems like Yetus problem with
>>> > pre-commit build for branch-2.7.
>>>
>>>
>>> branch-2.7 is missing stuff in .gitignore.
>>
>>
>>
>

Re: Pre-Commit build is failing

[Konstantin Shvachko]

Allen,

Should we add "patchprocess/" to .gitignore, is that the problem for 2.7?

Thanks,
--Konstantin

On Fri, Jul 21, 2017 at 6:24 PM, Konstantin Shvachko <shv.had...@gmail.com>
wrote:

> What stuff? Is there a jira?
> It did work like a week ago. Is it a new Yetus requirement.
> Anyways I can commit a change to fix the build on our side.
> Just need to know what is missing.
>
> Thanks,
> --Konst
>
> On Fri, Jul 21, 2017 at 5:50 PM, Allen Wittenauer <
> a...@effectivemachines.com> wrote:
>
>>
>> > On Jul 21, 2017, at 5:46 PM, Konstantin Shvachko <shv.had...@gmail.com>
>> wrote:
>> >
>> > + d...@yetus.apache.org
>> >
>> > Guys, could you please take a look. Seems like Yetus problem with
>> > pre-commit build for branch-2.7.
>>
>>
>> branch-2.7 is missing stuff in .gitignore.
>
>
>

[jira] [Created] (HADOOP-14676) Wrong default value for "fs.du.interval"

[Konstantin Shvachko (JIRA)]

Konstantin Shvachko created HADOOP-14676:


 Summary: Wrong default value for "fs.du.interval"
 Key: HADOOP-14676
 URL: https://issues.apache.org/jira/browse/HADOOP-14676
 Project: Hadoop Common
  Issue Type: Bug
  Components: common, conf, fs
Affects Versions: 2.6.1
Reporter: Konstantin Shvachko


According to {{core-default.xml}} the default value of {{fs.du.interval = 60 
sec}}. But the implementation of {{DF}} substitutes 3 sec instead. The problem 
is that {{DF}} uses outdated constant {{DF.DF_INTERVAL_DEFAULT}} instead of the 
correct one {{CommonConfigurationKeysPublic.FS_DU_INTERVAL_DEFAULT}}.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: Pre-Commit build is failing

[Konstantin Shvachko]

What stuff? Is there a jira?
It did work like a week ago. Is it a new Yetus requirement.
Anyways I can commit a change to fix the build on our side.
Just need to know what is missing.

Thanks,
--Konst

On Fri, Jul 21, 2017 at 5:50 PM, Allen Wittenauer <a...@effectivemachines.com>
wrote:

>
> > On Jul 21, 2017, at 5:46 PM, Konstantin Shvachko <shv.had...@gmail.com>
> wrote:
> >
> > + d...@yetus.apache.org
> >
> > Guys, could you please take a look. Seems like Yetus problem with
> > pre-commit build for branch-2.7.
>
>
> branch-2.7 is missing stuff in .gitignore.

Re: Pre-Commit build is failing

[Konstantin Shvachko]

+ d...@yetus.apache.org

Guys, could you please take a look. Seems like Yetus problem with
pre-commit build for branch-2.7.

Thanks,
--Konstantin

On Thu, Jul 20, 2017 at 7:19 PM, Brahma Reddy Battula <
brahmareddy.batt...@huawei.com> wrote:

> Looks this problem is in only branc-2.7..
>
>
> --Brahma Reddy Battula
>
> From: Brahma Reddy Battula
> Sent: 21 July 2017 09:36
> To: common-dev@hadoop.apache.org; hdfs-...@hadoop.apache.org
> Subject: Pre-Commit build is failing
> Importance: High
>
> Looks pre-commit build is failing with following error.
>
>
> /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-
> Build/patchprocess/apache-yetus-a444ed1/precommit/core.d/00-yetuslib.sh:
> line 87: /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-
> Build/patchprocess/patch-dryrun.log: No such file or directory
> /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-
> Build/patchprocess/apache-yetus-a444ed1/precommit/core.d/00-yetuslib.sh:
> line 98: /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-
> Build/patchprocess/patch-dryrun.log: No such file or directory
> /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-
> Build/patchprocess/apache-yetus-a444ed1/precommit/core.d/00-yetuslib.sh:
> line 87: /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-
> Build/patchprocess/patch-dryrun.log: No such file or directory
> /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-
> Build/patchprocess/apache-yetus-a444ed1/precommit/core.d/00-yetuslib.sh:
> line 98: /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-
> Build/patchprocess/patch-dryrun.log: No such file or directory
>
>
>
> Reference :
>
> https://builds.apache.org/view/PreCommit%20Builds/job/
> PreCommit-HDFS-Build/20362/console
>
>
>
>
> --Brahma Reddy Battula
>
>

Re: About 2.7.4 Release

[Konstantin Shvachko]

Hi everybody.

We have been doing some internal testing of Hadoop 2.7.4. The testing is
going well.
Did not find any major issues on our workloads.
Used an internal tool called Dynamometer to check NameNode performance on
real cluster traces. Good.
Overall test cluster performance looks good.
Some more testing is still going on.

I plan to build an RC next week. If there are no objection.

Thanks,
--Konst

On Thu, Jun 15, 2017 at 4:42 PM, Konstantin Shvachko <shv.had...@gmail.com>
wrote:

> Hey guys.
>
> An update on 2.7.4 progress.
> We are down to 4 blockers. There is some work remaining on those.
> https://issues.apache.org/jira/browse/HDFS-11896?filter=12340814
> Would be good if people could follow up on review comments.
>
> I looked through nightly Jenkins build results for 2.7.4 both on Apache
> Jenkins and internal.
> Some test fail intermittently, but there no consistent failures. I filed
> HDFS-11985 to track some of them.
> https://issues.apache.org/jira/browse/HDFS-11985
> I do not currently consider these failures as blockers. LMK if some of
> them are.
>
> We started internal testing of branch-2.7 on one of our smallish (100+
> nodes) test clusters.
> Will update on the results.
>
> There is a plan to enable BigTop for 2.7.4 testing.
>
> Akira, Brahma thank you for setting up a wiki page for 2.7.4 release.
> Thank you everybody for contributing to this effort.
>
> Regards,
> --Konstantin
>
>
> On Tue, May 30, 2017 at 12:08 AM, Akira Ajisaka <aajis...@apache.org>
> wrote:
>
>> Sure.
>> If you want to edit the wiki, please tell me your ASF confluence account.
>>
>> -Akira
>>
>> On 2017/05/30 15:31, Rohith Sharma K S wrote:
>>
>>> Couple of more JIRAs need to be back ported for 2.7.4 release. These will
>>> solve RM HA unstability issues.
>>> https://issues.apache.org/jira/browse/YARN-5333
>>> https://issues.apache.org/jira/browse/YARN-5988
>>> https://issues.apache.org/jira/browse/YARN-6304
>>>
>>> I will raise a JIRAs to back port it.
>>>
>>> @Akira , could  you help to add these JIRAs into wiki?
>>>
>>> Thanks & Regards
>>> Rohith Sharma K S
>>>
>>> On 29 May 2017 at 12:19, Akira Ajisaka <aajis...@apache.org> wrote:
>>>
>>> Created a page for 2.7.4 release.
>>>> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.7.4
>>>>
>>>> If you want to edit this wiki, please ping me.
>>>>
>>>> Regards,
>>>> Akira
>>>>
>>>>
>>>> On 2017/05/23 4:42, Brahma Reddy Battula wrote:
>>>>
>>>> Hi Konstantin Shvachko
>>>>>
>>>>>
>>>>> how about creating a wiki page for 2.7.4 release status like 2.8 and
>>>>> trunk in following link.??
>>>>>
>>>>>
>>>>> https://cwiki.apache.org/confluence/display/HADOOP
>>>>>
>>>>>
>>>>> 
>>>>> From: Konstantin Shvachko <shv.had...@gmail.com>
>>>>> Sent: Saturday, May 13, 2017 3:58 AM
>>>>> To: Akira Ajisaka
>>>>> Cc: Hadoop Common; Hdfs-dev; mapreduce-...@hadoop.apache.org;
>>>>> yarn-...@hadoop.apache.org
>>>>> Subject: Re: About 2.7.4 Release
>>>>>
>>>>> Latest update on the links and filters. Here is the correct link for
>>>>> the
>>>>> filter:
>>>>> https://issues.apache.org/jira/secure/IssueNavigator.jspa?
>>>>> requestId=12340814
>>>>>
>>>>> Also updated: https://s.apache.org/Dzg4
>>>>>
>>>>> Had to do some Jira debugging. Sorry for confusion.
>>>>>
>>>>> Thanks,
>>>>> --Konstantin
>>>>>
>>>>> On Wed, May 10, 2017 at 2:30 PM, Konstantin Shvachko <
>>>>> shv.had...@gmail.com>
>>>>> wrote:
>>>>>
>>>>> Hey Akira,
>>>>>
>>>>>>
>>>>>> I didn't have private filters. Most probably Jira caches something.
>>>>>> Your filter is in the right direction, but for some reason it lists
>>>>>> only
>>>>>> 22 issues, while mine has 29.
>>>>>> It misses e.g. YARN-5543 <https://issues.apache.org/jir
>>>>>> a/browse/YARN-5543>
>>>>>> .
>>>>>>
>>>>>> Anyways, I created a Jira filter now "Hadoop 2.7.4 

Re: About 2.7.4 Release

[Konstantin Shvachko]

Hey guys.

An update on 2.7.4 progress.
We are down to 4 blockers. There is some work remaining on those.
https://issues.apache.org/jira/browse/HDFS-11896?filter=12340814
Would be good if people could follow up on review comments.

I looked through nightly Jenkins build results for 2.7.4 both on Apache
Jenkins and internal.
Some test fail intermittently, but there no consistent failures. I filed
HDFS-11985 to track some of them.
https://issues.apache.org/jira/browse/HDFS-11985
I do not currently consider these failures as blockers. LMK if some of them
are.

We started internal testing of branch-2.7 on one of our smallish (100+
nodes) test clusters.
Will update on the results.

There is a plan to enable BigTop for 2.7.4 testing.

Akira, Brahma thank you for setting up a wiki page for 2.7.4 release.
Thank you everybody for contributing to this effort.

Regards,
--Konstantin


On Tue, May 30, 2017 at 12:08 AM, Akira Ajisaka <aajis...@apache.org> wrote:

> Sure.
> If you want to edit the wiki, please tell me your ASF confluence account.
>
> -Akira
>
> On 2017/05/30 15:31, Rohith Sharma K S wrote:
>
>> Couple of more JIRAs need to be back ported for 2.7.4 release. These will
>> solve RM HA unstability issues.
>> https://issues.apache.org/jira/browse/YARN-5333
>> https://issues.apache.org/jira/browse/YARN-5988
>> https://issues.apache.org/jira/browse/YARN-6304
>>
>> I will raise a JIRAs to back port it.
>>
>> @Akira , could  you help to add these JIRAs into wiki?
>>
>> Thanks & Regards
>> Rohith Sharma K S
>>
>> On 29 May 2017 at 12:19, Akira Ajisaka <aajis...@apache.org> wrote:
>>
>> Created a page for 2.7.4 release.
>>> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.7.4
>>>
>>> If you want to edit this wiki, please ping me.
>>>
>>> Regards,
>>> Akira
>>>
>>>
>>> On 2017/05/23 4:42, Brahma Reddy Battula wrote:
>>>
>>> Hi Konstantin Shvachko
>>>>
>>>>
>>>> how about creating a wiki page for 2.7.4 release status like 2.8 and
>>>> trunk in following link.??
>>>>
>>>>
>>>> https://cwiki.apache.org/confluence/display/HADOOP
>>>>
>>>>
>>>> 
>>>> From: Konstantin Shvachko <shv.had...@gmail.com>
>>>> Sent: Saturday, May 13, 2017 3:58 AM
>>>> To: Akira Ajisaka
>>>> Cc: Hadoop Common; Hdfs-dev; mapreduce-...@hadoop.apache.org;
>>>> yarn-...@hadoop.apache.org
>>>> Subject: Re: About 2.7.4 Release
>>>>
>>>> Latest update on the links and filters. Here is the correct link for the
>>>> filter:
>>>> https://issues.apache.org/jira/secure/IssueNavigator.jspa?
>>>> requestId=12340814
>>>>
>>>> Also updated: https://s.apache.org/Dzg4
>>>>
>>>> Had to do some Jira debugging. Sorry for confusion.
>>>>
>>>> Thanks,
>>>> --Konstantin
>>>>
>>>> On Wed, May 10, 2017 at 2:30 PM, Konstantin Shvachko <
>>>> shv.had...@gmail.com>
>>>> wrote:
>>>>
>>>> Hey Akira,
>>>>
>>>>>
>>>>> I didn't have private filters. Most probably Jira caches something.
>>>>> Your filter is in the right direction, but for some reason it lists
>>>>> only
>>>>> 22 issues, while mine has 29.
>>>>> It misses e.g. YARN-5543 <https://issues.apache.org/jir
>>>>> a/browse/YARN-5543>
>>>>> .
>>>>>
>>>>> Anyways, I created a Jira filter now "Hadoop 2.7.4 release blockers",
>>>>> shared it with "everybody", and updated my link to point to that
>>>>> filter.
>>>>> So
>>>>> you can use any of the three methods below to get the correct list:
>>>>> 1. Go to https://s.apache.org/Dzg4
>>>>> 2. Go to the filter via
>>>>> https://issues.apache.org/jira/issues?filter=12340814
>>>>>or by finding "Hadoop 2.7.4 release blockers" filter in the jira
>>>>> 3. On Advanced issues search page paste this:
>>>>> project in (HDFS, HADOOP, YARN, MAPREDUCE) AND labels = release-blocker
>>>>> AND "Target Version/s" = 2.7.4
>>>>>
>>>>> Hope this solves the confusion for which issues are included.
>>>>> Please LMK if it doesn't, as it is important.
>>>>>
>>>>> T

[jira] [Resolved] (HADOOP-14446) Backport HADOOP-7851 to branch 2.7

[Konstantin Shvachko (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-14446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Konstantin Shvachko resolved HADOOP-14446.
--
  Resolution: Fixed
Hadoop Flags: Reviewed

I just committed this to branch-2, 2.8, 2.7. Thank you [~elgoiri] for backport.

> Backport HADOOP-7851 to branch 2.7
> --
>
> Key: HADOOP-14446
> URL: https://issues.apache.org/jira/browse/HADOOP-14446
> Project: Hadoop Common
>  Issue Type: Bug
>Reporter: Inigo Goiri
>Assignee: Inigo Goiri
> Attachments: HADOOP-7851-branch-2.7.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: About 2.7.4 Release

[Konstantin Shvachko]

Latest update on the links and filters. Here is the correct link for the
filter:
https://issues.apache.org/jira/secure/IssueNavigator.jspa?requestId=12340814

Also updated: https://s.apache.org/Dzg4

Had to do some Jira debugging. Sorry for confusion.

Thanks,
--Konstantin

On Wed, May 10, 2017 at 2:30 PM, Konstantin Shvachko <shv.had...@gmail.com>
wrote:

> Hey Akira,
>
> I didn't have private filters. Most probably Jira caches something.
> Your filter is in the right direction, but for some reason it lists only
> 22 issues, while mine has 29.
> It misses e.g. YARN-5543 <https://issues.apache.org/jira/browse/YARN-5543>
> .
>
> Anyways, I created a Jira filter now "Hadoop 2.7.4 release blockers",
> shared it with "everybody", and updated my link to point to that filter. So
> you can use any of the three methods below to get the correct list:
> 1. Go to https://s.apache.org/Dzg4
> 2. Go to the filter via
> https://issues.apache.org/jira/issues?filter=12340814
>or by finding "Hadoop 2.7.4 release blockers" filter in the jira
> 3. On Advanced issues search page paste this:
> project in (HDFS, HADOOP, YARN, MAPREDUCE) AND labels = release-blocker
> AND "Target Version/s" = 2.7.4
>
> Hope this solves the confusion for which issues are included.
> Please LMK if it doesn't, as it is important.
>
> Thanks,
> --Konstantin
>
> On Tue, May 9, 2017 at 9:58 AM, Akira Ajisaka <aajis...@apache.org> wrote:
>
>> Hi Konstantin,
>>
>> Thank you for volunteering as release manager!
>>
>> > Actually the original link works fine: https://s.apache.org/Dzg4
>> I couldn't see the link. Maybe is it private filter?
>>
>> Here is a link I generated: https://s.apache.org/ehKy
>> This filter includes resolved issue and excludes fixversion == 2.7.4
>>
>> Thanks and Regards,
>> Akira
>>
>> On 2017/05/08 19:20, Konstantin Shvachko wrote:
>>
>>> Hi Brahma Reddy Battula,
>>>
>>> Actually the original link works fine: https://s.apache.org/Dzg4
>>> Your link excludes closed and resolved issues, which needs backporting,
>>> and
>>> which we cannot reopen, as discussed in this thread earlier.
>>>
>>> Looked through the issues you proposed:
>>>
>>> HDFS-9311 <https://issues.apache.org/jira/browse/HDFS-9311>
>>> Seems like a new feature. It helps failover to standby node when primary
>>> is
>>> under heavy load, but it introduces new APIs, addresses, config
>>> parameters.
>>> And needs at least one follow up jira.
>>> Looks like a backward compatible change, though.
>>> Did you have a chance to run it in production?
>>>
>>> +1 on
>>> HDFS-10987 <https://issues.apache.org/jira/browse/HDFS-10987>
>>> HDFS-9902 <https://issues.apache.org/jira/browse/HDFS-9902>
>>> HDFS-8312 <https://issues.apache.org/jira/browse/HDFS-8312>
>>> HADOOP-14100 <https://issues.apache.org/jira/browse/HADOOP-14100>
>>>
>>> Added them to 2.7.4 release. You should see them via the above link now.
>>> Would be good if you could attach backport patches for some of them?
>>>
>>> Appreciate your help,
>>> --Konstantin
>>>
>>> On Mon, May 8, 2017 at 8:39 AM, Brahma Reddy Battula <
>>> brahmareddy.batt...@huawei.com> wrote:
>>>
>>>
>>>> Looks following link is not correct..
>>>>
>>>> https://s.apache.org/Dzg4
>>>>
>>>> It should be like following..?
>>>>
>>>> https://s.apache.org/wi3U
>>>>
>>>>
>>>> Apart from Konstantin mentioned,Following also good to go..? let me know
>>>> your thoughts on this.
>>>>
>>>> For Large Cluster:
>>>> =
>>>>
>>>> https://issues.apache.org/jira/browse/HDFS-9311===Life Line Protocol
>>>> https://issues.apache.org/jira/browse/HDFS-10987===Deecommission
>>>> Expensive when lot's of blocks are present
>>>>
>>>> https://issues.apache.org/jira/browse/HDFS-9902===
>>>> "dfs.datanode.du.reserved"  per Storage Type
>>>>
>>>> For Security:
>>>> =
>>>> https://issues.apache.org/jira/browse/HDFS-8312===Trash does not
>>>> descent
>>>> into child directories to check for permission
>>>> https://issues.apache.org/jira/browse/HADOOP-14100===Upgrade Jsch jar
>>>> to
>>>> latest version to fix vulnerability i

Re: About 2.7.4 Release

[Konstantin Shvachko]

Hey Akira,

I didn't have private filters. Most probably Jira caches something.
Your filter is in the right direction, but for some reason it lists only 22
issues, while mine has 29.
It misses e.g. YARN-5543 <https://issues.apache.org/jira/browse/YARN-5543>.

Anyways, I created a Jira filter now "Hadoop 2.7.4 release blockers",
shared it with "everybody", and updated my link to point to that filter. So
you can use any of the three methods below to get the correct list:
1. Go to https://s.apache.org/Dzg4
2. Go to the filter via
https://issues.apache.org/jira/issues?filter=12340814
   or by finding "Hadoop 2.7.4 release blockers" filter in the jira
3. On Advanced issues search page paste this:
project in (HDFS, HADOOP, YARN, MAPREDUCE) AND labels = release-blocker AND
"Target Version/s" = 2.7.4

Hope this solves the confusion for which issues are included.
Please LMK if it doesn't, as it is important.

Thanks,
--Konstantin

On Tue, May 9, 2017 at 9:58 AM, Akira Ajisaka <aajis...@apache.org> wrote:

> Hi Konstantin,
>
> Thank you for volunteering as release manager!
>
> > Actually the original link works fine: https://s.apache.org/Dzg4
> I couldn't see the link. Maybe is it private filter?
>
> Here is a link I generated: https://s.apache.org/ehKy
> This filter includes resolved issue and excludes fixversion == 2.7.4
>
> Thanks and Regards,
> Akira
>
> On 2017/05/08 19:20, Konstantin Shvachko wrote:
>
>> Hi Brahma Reddy Battula,
>>
>> Actually the original link works fine: https://s.apache.org/Dzg4
>> Your link excludes closed and resolved issues, which needs backporting,
>> and
>> which we cannot reopen, as discussed in this thread earlier.
>>
>> Looked through the issues you proposed:
>>
>> HDFS-9311 <https://issues.apache.org/jira/browse/HDFS-9311>
>> Seems like a new feature. It helps failover to standby node when primary
>> is
>> under heavy load, but it introduces new APIs, addresses, config
>> parameters.
>> And needs at least one follow up jira.
>> Looks like a backward compatible change, though.
>> Did you have a chance to run it in production?
>>
>> +1 on
>> HDFS-10987 <https://issues.apache.org/jira/browse/HDFS-10987>
>> HDFS-9902 <https://issues.apache.org/jira/browse/HDFS-9902>
>> HDFS-8312 <https://issues.apache.org/jira/browse/HDFS-8312>
>> HADOOP-14100 <https://issues.apache.org/jira/browse/HADOOP-14100>
>>
>> Added them to 2.7.4 release. You should see them via the above link now.
>> Would be good if you could attach backport patches for some of them?
>>
>> Appreciate your help,
>> --Konstantin
>>
>> On Mon, May 8, 2017 at 8:39 AM, Brahma Reddy Battula <
>> brahmareddy.batt...@huawei.com> wrote:
>>
>>
>>> Looks following link is not correct..
>>>
>>> https://s.apache.org/Dzg4
>>>
>>> It should be like following..?
>>>
>>> https://s.apache.org/wi3U
>>>
>>>
>>> Apart from Konstantin mentioned,Following also good to go..? let me know
>>> your thoughts on this.
>>>
>>> For Large Cluster:
>>> =
>>>
>>> https://issues.apache.org/jira/browse/HDFS-9311===Life Line Protocol
>>> https://issues.apache.org/jira/browse/HDFS-10987===Deecommission
>>> Expensive when lot's of blocks are present
>>>
>>> https://issues.apache.org/jira/browse/HDFS-9902===
>>> "dfs.datanode.du.reserved"  per Storage Type
>>>
>>> For Security:
>>> =
>>> https://issues.apache.org/jira/browse/HDFS-8312===Trash does not descent
>>> into child directories to check for permission
>>> https://issues.apache.org/jira/browse/HADOOP-14100===Upgrade Jsch jar to
>>> latest version to fix vulnerability in old versions
>>>
>>>
>>>
>>> Regards
>>> Brahma Reddy Battula
>>>
>>> -Original Message-
>>> From: Erik Krogen [mailto:ekro...@linkedin.com.INVALID]
>>> Sent: 06 May 2017 02:40
>>> To: Konstantin Shvachko
>>> Cc: Zhe Zhang; Hadoop Common; Hdfs-dev; mapreduce-...@hadoop.apache.org;
>>> yarn-...@hadoop.apache.org
>>> Subject: Re: About 2.7.4 Release
>>>
>>> List LGTM Konstantin!
>>>
>>> Let's say that we will only create a new tracking JIRA for patches which
>>> do not backport cleanly, to avoid having too many lying around. Otherwise
>>> we can directly attach to old ticket. If a clean backport does happen to
>>> break a test the nightly 

Re: About 2.7.4 Release

[Konstantin Shvachko]

Hi Brahma Reddy Battula,

Actually the original link works fine: https://s.apache.org/Dzg4
Your link excludes closed and resolved issues, which needs backporting, and
which we cannot reopen, as discussed in this thread earlier.

Looked through the issues you proposed:

HDFS-9311 <https://issues.apache.org/jira/browse/HDFS-9311>
Seems like a new feature. It helps failover to standby node when primary is
under heavy load, but it introduces new APIs, addresses, config parameters.
And needs at least one follow up jira.
Looks like a backward compatible change, though.
Did you have a chance to run it in production?

+1 on
HDFS-10987 <https://issues.apache.org/jira/browse/HDFS-10987>
HDFS-9902 <https://issues.apache.org/jira/browse/HDFS-9902>
HDFS-8312 <https://issues.apache.org/jira/browse/HDFS-8312>
HADOOP-14100 <https://issues.apache.org/jira/browse/HADOOP-14100>

Added them to 2.7.4 release. You should see them via the above link now.
Would be good if you could attach backport patches for some of them?

Appreciate your help,
--Konstantin

On Mon, May 8, 2017 at 8:39 AM, Brahma Reddy Battula <
brahmareddy.batt...@huawei.com> wrote:

>
> Looks following link is not correct..
>
> https://s.apache.org/Dzg4
>
> It should be like following..?
>
> https://s.apache.org/wi3U
>
>
> Apart from Konstantin mentioned,Following also good to go..? let me know
> your thoughts on this.
>
> For Large Cluster:
> =
>
> https://issues.apache.org/jira/browse/HDFS-9311===Life Line Protocol
> https://issues.apache.org/jira/browse/HDFS-10987===Deecommission
> Expensive when lot's of blocks are present
>
> https://issues.apache.org/jira/browse/HDFS-9902===
> "dfs.datanode.du.reserved"  per Storage Type
>
> For Security:
> =
> https://issues.apache.org/jira/browse/HDFS-8312===Trash does not descent
> into child directories to check for permission
> https://issues.apache.org/jira/browse/HADOOP-14100===Upgrade Jsch jar to
> latest version to fix vulnerability in old versions
>
>
>
> Regards
> Brahma Reddy Battula
>
> -Original Message-
> From: Erik Krogen [mailto:ekro...@linkedin.com.INVALID]
> Sent: 06 May 2017 02:40
> To: Konstantin Shvachko
> Cc: Zhe Zhang; Hadoop Common; Hdfs-dev; mapreduce-...@hadoop.apache.org;
> yarn-...@hadoop.apache.org
> Subject: Re: About 2.7.4 Release
>
> List LGTM Konstantin!
>
> Let's say that we will only create a new tracking JIRA for patches which
> do not backport cleanly, to avoid having too many lying around. Otherwise
> we can directly attach to old ticket. If a clean backport does happen to
> break a test the nightly build will help us catch it.
>
> Erik
>
> On Thu, May 4, 2017 at 7:21 PM, Konstantin Shvachko <shv.had...@gmail.com>
> wrote:
>
> > Great Zhe. Let's monitor the build.
> >
> > I marked all jiras I knew of for inclusion into 2.7.4 as I described
> > before.
> > Target Version/s: 2.7.4
> > Label: release-blocker
> >
> > Here is the link to the list: https://s.apache.org/Dzg4 Please let me
> > know if I missed anything.
> > And feel free to pick up any. Most of backports are pretty
> > straightforward, but not all.
> >
> > We can create tracking jiras for backporting if you need to run
> > Jenkins on the patch (and since Allen does not allow reopening them).
> > But I think the final patch should be attached to the original jira.
> > Otherwise history will be hard to follow.
> >
> > Thanks,
> > --Konstantin
> >
> > On Wed, May 3, 2017 at 4:53 PM, Zhe Zhang <z...@apache.org> wrote:
> >
> > > Thanks for volunteering as RM Konstantin! The plan LGTM.
> > >
> > > I've created a nightly Jenkins job for branch-2.7 (unit tests):
> > > https://builds.apache.org/job/Hadoop-branch2.7-nightly/
> > >
> > > On Wed, May 3, 2017 at 12:42 AM Konstantin Shvachko <
> > shv.had...@gmail.com>
> > > wrote:
> > >
> > >> Hey guys,
> > >>
> > >> I and a few of my colleagues would like to help here and move 2.7.4
> > >> release forward. A few points in this regard.
> > >>
> > >> 1. Reading through this thread since March 1 I see that Vinod
> > >> hinted on managing the release. Vinod, if you still want the job /
> > >> have bandwidth will be happy to work with you.
> > >> Otherwise I am glad to volunteer as the release manager.
> > >>
> > >> 2. In addition to current blockers and criticals, I would like to
> > propose
> > >> a
> > >> few issues to be included in the release, see the list below. Tho

Re: About 2.7.4 Release

[Konstantin Shvachko]

Great Zhe. Let's monitor the build.

I marked all jiras I knew of for inclusion into 2.7.4 as I described before.
Target Version/s: 2.7.4
Label: release-blocker

Here is the link to the list: https://s.apache.org/Dzg4
Please let me know if I missed anything.
And feel free to pick up any. Most of backports are pretty straightforward,
but not all.

We can create tracking jiras for backporting if you need to run Jenkins on
the patch (and since Allen does not allow reopening them).
But I think the final patch should be attached to the original jira.
Otherwise history will be hard to follow.

Thanks,
--Konstantin

On Wed, May 3, 2017 at 4:53 PM, Zhe Zhang <z...@apache.org> wrote:

> Thanks for volunteering as RM Konstantin! The plan LGTM.
>
> I've created a nightly Jenkins job for branch-2.7 (unit tests):
> https://builds.apache.org/job/Hadoop-branch2.7-nightly/
>
> On Wed, May 3, 2017 at 12:42 AM Konstantin Shvachko <shv.had...@gmail.com>
> wrote:
>
>> Hey guys,
>>
>> I and a few of my colleagues would like to help here and move 2.7.4
>> release
>> forward. A few points in this regard.
>>
>> 1. Reading through this thread since March 1 I see that Vinod hinted on
>> managing the release. Vinod, if you still want the job / have bandwidth
>> will be happy to work with you.
>> Otherwise I am glad to volunteer as the release manager.
>>
>> 2. In addition to current blockers and criticals, I would like to propose
>> a
>> few issues to be included in the release, see the list below. Those are
>> mostly bug fixes and optimizations, which we already have in our internal
>> branch and run in production. Plus one minor feature "node labeling",
>> which
>> we found very handy, when you have heterogeneous environments and mixed
>> workloads, like MR and Spark.
>>
>> 3. For marking issues for the release I propose to
>>  - set the target version to 2.7.4, and
>>  - add a new label "release-blocker"
>> That way we will know issues targeted for the release without reopening
>> them for backports.
>>
>> 4. I see quite a few people are interested in the release. With all the
>> help I think we can target to release by the end of May.
>>
>> Other things include fixing CHANGES.txt and fixing Jenkins build for 2.7.4
>> branch.
>>
>> Thanks,
>> --Konstantin
>>
>> ==  List of issue for 2.7.4  ===
>> -- Backports
>> HADOOP-12975 <https://issues.apache.org/jira/browse/HADOOP-12975>. Add du
>> jitters
>> HDFS-9710 <https://issues.apache.org/jira/browse/HDFS-9710>. IBR batching
>> HDFS-10715 <https://issues.apache.org/jira/browse/HDFS-10715>. NPE when
>> applying AvailableSpaceBlockPlacementPolicy
>> HDFS-2538 <https://issues.apache.org/jira/browse/HDFS-2538>. fsck removal
>> of dot printing
>> HDFS-8131 <https://issues.apache.org/jira/browse/HDFS-8131>.
>> space-balanced
>> policy for balancer
>> HDFS-8549 <https://issues.apache.org/jira/browse/HDFS-8549>. abort
>> balancer
>> if upgrade in progress
>> HDFS-9412 <https://issues.apache.org/jira/browse/HDFS-9412>. skip small
>> blocks in getBlocks
>>
>> YARN-1471 <https://issues.apache.org/jira/browse/YARN-1471>. SLS
>> simulator
>> YARN-4302 <https://issues.apache.org/jira/browse/YARN-4302>. SLS
>> YARN-4367 <https://issues.apache.org/jira/browse/YARN-4367>. SLS
>> YARN-4612 <https://issues.apache.org/jira/browse/YARN-4612>. SLS
>>
>> - Node labeling
>> MAPREDUCE-6304 <https://issues.apache.org/jira/browse/MAPREDUCE-6304>
>> YARN-2943 <https://issues.apache.org/jira/browse/YARN-2943>
>> YARN-4109 <https://issues.apache.org/jira/browse/YARN-4109>
>> YARN-4140 <https://issues.apache.org/jira/browse/YARN-4140>
>> YARN-4250 <https://issues.apache.org/jira/browse/YARN-4250>
>> YARN-4925 <https://issues.apache.org/jira/browse/YARN-4925>
>>
> --
> Zhe Zhang
> Apache Hadoop Committer
> http://zhe-thoughts.github.io/about/ | @oldcap
>

Re: About 2.7.4 Release

[Konstantin Shvachko]

Hey guys,

I and a few of my colleagues would like to help here and move 2.7.4 release
forward. A few points in this regard.

1. Reading through this thread since March 1 I see that Vinod hinted on
managing the release. Vinod, if you still want the job / have bandwidth
will be happy to work with you.
Otherwise I am glad to volunteer as the release manager.

2. In addition to current blockers and criticals, I would like to propose a
few issues to be included in the release, see the list below. Those are
mostly bug fixes and optimizations, which we already have in our internal
branch and run in production. Plus one minor feature "node labeling", which
we found very handy, when you have heterogeneous environments and mixed
workloads, like MR and Spark.

3. For marking issues for the release I propose to
 - set the target version to 2.7.4, and
 - add a new label "release-blocker"
That way we will know issues targeted for the release without reopening
them for backports.

4. I see quite a few people are interested in the release. With all the
help I think we can target to release by the end of May.

Other things include fixing CHANGES.txt and fixing Jenkins build for 2.7.4
branch.

Thanks,
--Konstantin

==  List of issue for 2.7.4  ===
-- Backports
HADOOP-12975 . Add du
jitters
HDFS-9710 . IBR batching
HDFS-10715 . NPE when
applying AvailableSpaceBlockPlacementPolicy
HDFS-2538 . fsck removal
of dot printing
HDFS-8131 . space-balanced
policy for balancer
HDFS-8549 . abort balancer
if upgrade in progress
HDFS-9412 . skip small
blocks in getBlocks

YARN-1471 . SLS simulator
YARN-4302 . SLS
YARN-4367 . SLS
YARN-4612 . SLS

- Node labeling
MAPREDUCE-6304 
YARN-2943 
YARN-4109 
YARN-4140 
YARN-4250 
YARN-4925 

[jira] [Created] (HADOOP-13902) Update documentation to reflect IPv6 Hadoop cluster setup

[Konstantin Shvachko (JIRA)]

Konstantin Shvachko created HADOOP-13902:


 Summary: Update documentation to reflect IPv6 Hadoop cluster setup
 Key: HADOOP-13902
 URL: https://issues.apache.org/jira/browse/HADOOP-13902
 Project: Hadoop Common
  Issue Type: Sub-task
  Components: documentation
Reporter: Konstantin Shvachko


Document IPv6 cluster setup.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [DISCUSS] Release numbering semantics with concurrent (>2) releases [Was Setting JIRA fix versions for 3.0.0 releases]

[Konstantin Shvachko]

On Thu, Aug 4, 2016 at 11:20 AM, Andrew Wang <andrew.w...@cloudera.com>
wrote:

> Hi Konst, thanks for commenting,
>
> On Wed, Aug 3, 2016 at 11:29 PM, Konstantin Shvachko <shv.had...@gmail.com
> > wrote:
>
>> 1. I probably missed something but I didn't get it how "alpha"s made
>> their way into release numbers again. This was discussed on several
>> occasions and I thought the common perception was to use just three level
>> numbers for release versioning and avoid branding them.
>> It is particularly confusing to have 3.0.0-alpha1 and 3.0.0-alpha2. What
>> is alphaX - fourth level? I think releasing 3.0.0 and setting trunk to
>> 3.1.0 would be perfectly in line with our current release practices.
>>
>
> We discussed release numbering a while ago when discussing the release
> plan for 3.0.0, and agreed on this scheme. "-alphaX" is essentially a
> fourth level as you say, but the intent is to only use it (and "-betaX") in
> the leadup to 3.0.0.
>
> The goal here is clarity for end users, since most other enterprise
> software uses a a.0.0 version to denote the GA of a new major version. Same
> for a.b.0 for a new minor version, though we haven't talked about that yet.
> The alphaX and betaX scheme also shares similarity to release versioning of
> other enterprise software.
>

As you remember we did this (alpha, beta) for Hadoop-2 and I don't think it
went well with user perception.
Say release 2.0.5-alpha turned out to be quite good even though still
branded "alpha", while 2.2 was not and not branded.
We should move a release to stable, when people ran it and agree it is GA
worthy. Otherwise you never know.


>
>> 2. I do not see any confusions with releasing 2.8.0 after 3.0.0.
>> The release number is not intended to reflect historical release
>> sequence, but rather the point in the source tree, which it was branched
>> off. So one can release 2.8, 2.9, etc. after or before 3.0.
>>
>
> As described earlier in this thread, the issue here is setting the fix
> versions such that the changelog is a useful diff from a previous version,
> and also clear about what changes are present in each branch. If we do not
> order a specific 2.x before 3.0, then we don't know what 2.x to diff from.
>

So the problem is in determining the latest commit, which was not present
in the last release, when the last release bears higher number than the one
being released.
Interesting problem. Don't have a strong opinion on that. I guess it's OK
to have overlapping in changelogs.
As long as we keep following the rule that commits should be made to trunk
first and them propagated to lower branches until the target branch is
reached.


>
>> 3. I agree that current 3.0.0 branch can be dropped and re-cut. We may
>> think of another rule that if a release branch is not released in 3 month
>> it should be abandoned. Which is applicable to branch 2.8.0 and it is too
>> much work syncing it with branch-2.
>>
>> Time-based rules are tough here. I'd prefer we continue to leave this up
> to release managers. If you think we should recut branch-2.8, recommend
> pinging Vinod and discussing on a new thread.
>

Not recut, but abandon 2.8.0. And Vinod (or anybody who volunteers to RM)
can recut  from the desired point.
People were committing to branch-2 and branch-2.8 for months. And they are
out of sync anyways. So what's the point of the extra commit.
Probably still a different thread.

Thanks,
--Konst

Re: [DISCUSS] Release numbering semantics with concurrent (>2) releases [Was Setting JIRA fix versions for 3.0.0 releases]

[Konstantin Shvachko]

1. I probably missed something but I didn't get it how "alpha"s made their
way into release numbers again. This was discussed on several occasions and
I thought the common perception was to use just three level numbers for
release versioning and avoid branding them.
It is particularly confusing to have 3.0.0-alpha1 and 3.0.0-alpha2. What is
alphaX - fourth level? I think releasing 3.0.0 and setting trunk to 3.1.0
would be perfectly in line with our current release practices.

2. I do not see any confusions with releasing 2.8.0 after 3.0.0.
The release number is not intended to reflect historical release sequence,
but rather the point in the source tree, which it was branched off. So one
can release 2.8, 2.9, etc. after or before 3.0.

3. I agree that current 3.0.0 branch can be dropped and re-cut. We may
think of another rule that if a release branch is not released in 3 month
it should be abandoned. Which is applicable to branch 2.8.0 and it is too
much work syncing it with branch-2.

Thanks,
--Konstantin