Re: [DISCUSS] Should flush decisions be made based on data size (key-value only) or based on heap size (including metadata overhead)?

2017-07-05 Thread ramkrishna vasudevan 
>>I'm not sure whether we should rollback, but if stick on current policy
there should be more documents, metrics (monitoring heap/data occupancy
separately) and log message refinements, etc.

Agree to this.

>>Sounds like we should be doing the former, heap occupancy
Stack, so do you mean we need to roll back this new change in trunk? The
background is https://issues.apache.org/jira/browse/HBASE-16747.

Regards
Ram


On Thu, Jul 6, 2017 at 8:40 AM, Yu Li  wrote:

> We've also observed more blocking updates happening with the new policy
> (flush decision made on data size), but could work-around it by reducing
> the hbase.hregion.memstore.flush.size setting. The advantage of current
> policy is we could control the flushed file size more accurately, but
> meanwhile losing some "compatibility" (requires configuration updating
> during rolling upgrade).
>
> I'm not sure whether we should rollback, but if stick on current policy
> there should be more documents, metrics (monitoring heap/data occupancy
> separately) and log message refinements, etc. Attaching some of the logs we
> observed, which is pretty confusing w/o knowing the details of
> implementation:
>
> 2017-07-03 16:11:54,724 INFO
>  [B.defaultRpcServer.handler=182,queue=11,port=16020]
> regionserver.MemStoreFlusher: Blocking updates on
> hadoop0528.et2.tbsite.net,16020,1497336978160:
> global memstore heapsize 7.2 G is >= than blocking 7.2 G size
> 2017-07-03 16:11:54,754 INFO
>  [B.defaultRpcServer.handler=186,queue=15,port=16020]
> regionserver.MemStoreFlusher: Blocking updates on
> hadoop0528.et2.tbsite.net,16020,1497336978160:
> global memstore heapsize 7.2 G is >= than blocking 7.2 G size
> 2017-07-03 16:11:57,571 INFO  [MemStoreFlusher.0]
> regionserver.MemStoreFlusher: Flush of region
> mainv7_main_result_c,1496,1499062935573.02adfa7cbdc606dce5b79a516e16492a.
> due to global heap pressure. Total Memstore size=3.2 G, Region memstore
> size=331.4 M
> 2017-07-03 16:11:57,571 WARN
>  [B.defaultRpcServer.handler=49,queue=11,port=16020]
> regionserver.MemStoreFlusher: Memstore is above high water mark and block
> 2892ms
>
> Best Regards,
> Yu
>
> On 6 July 2017 at 00:56, Stack  wrote:
>
> > On Wed, Jul 5, 2017 at 6:30 AM, Eshcar Hillel
>  > >
> > wrote:
> >
> > > Hi All,
> > > I opened a new Jira https://issues.apache.org/jira/browse/HBASE-18294
> to
> > > discuss this question.
> > > Flush decisions are taken at the region level and also at the region
> > > server level - there is the question of when to trigger a flush and
> then
> > > which region/store to flush.Regions track both their data size
> (key-value
> > > size only) and their total heap occupancy (including index and
> additional
> > > metadata).One option (which was the past policy) is to trigger flushes
> > and
> > > choose flush subjects based on regions heap size - this gives a better
> > > estimation for sysadmin of how many regions can a RS carry.Another
> option
> > > (which is the current policy) is to look at the data size - this gives
> a
> > > better estimation of the size of the files that are created by the
> flush.
> > >
> >
> >
> > Sounds like we should be doing the former, heap occupancy. An
> > OutOfMemoryException puts a nail in any benefit other accountings might
> > have.
> >
> > St.Ack
> >
> >
> >
> > > I see this is as critical to HBase performance and usability, namely
> > > meeting the user expectation from the system, hence I would like to
> hear
> > as
> > > many voices as possible.Please join the discussion in the Jira and let
> us
> > > know what you think.
> > > Thanks,Eshcar
> > >
> > >
> >
>

[jira] [Resolved] (HBASE-18325) Disable flakey TestMasterProcedureWalLease

2017-07-05 Thread stack (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HBASE-18325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack resolved HBASE-18325.
---
   Resolution: Fixed
 Assignee: stack
Fix Version/s: 2.0.0

Disable test that is failing 80% of time according to flakey test dashboard. 
Need to fix it.  HBASE-18326

> Disable flakey TestMasterProcedureWalLease
> --
>
> Key: HBASE-18325
> URL: https://issues.apache.org/jira/browse/HBASE-18325
> Project: HBase
>  Issue Type: Bug
>  Components: test
>Reporter: stack
>Assignee: stack
> Fix For: 2.0.0
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HBASE-18326) Fix and reenable TestMasterProcedureWalLease

2017-07-05 Thread stack (JIRA) 
stack created HBASE-18326:
-

 Summary: Fix and reenable TestMasterProcedureWalLease
 Key: HBASE-18326
 URL: https://issues.apache.org/jira/browse/HBASE-18326
 Project: HBase
  Issue Type: Sub-task
Reporter: stack
Priority: Blocker


Fix and reenable flakey important test.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: [DISCUSS] Should flush decisions be made based on data size (key-value only) or based on heap size (including metadata overhead)?

2017-07-05 Thread Yu Li 
We've also observed more blocking updates happening with the new policy
(flush decision made on data size), but could work-around it by reducing
the hbase.hregion.memstore.flush.size setting. The advantage of current
policy is we could control the flushed file size more accurately, but
meanwhile losing some "compatibility" (requires configuration updating
during rolling upgrade).

I'm not sure whether we should rollback, but if stick on current policy
there should be more documents, metrics (monitoring heap/data occupancy
separately) and log message refinements, etc. Attaching some of the logs we
observed, which is pretty confusing w/o knowing the details of
implementation:

2017-07-03 16:11:54,724 INFO
 [B.defaultRpcServer.handler=182,queue=11,port=16020]
regionserver.MemStoreFlusher: Blocking updates on
hadoop0528.et2.tbsite.net,16020,1497336978160:
global memstore heapsize 7.2 G is >= than blocking 7.2 G size
2017-07-03 16:11:54,754 INFO
 [B.defaultRpcServer.handler=186,queue=15,port=16020]
regionserver.MemStoreFlusher: Blocking updates on
hadoop0528.et2.tbsite.net,16020,1497336978160:
global memstore heapsize 7.2 G is >= than blocking 7.2 G size
2017-07-03 16:11:57,571 INFO  [MemStoreFlusher.0]
regionserver.MemStoreFlusher: Flush of region
mainv7_main_result_c,1496,1499062935573.02adfa7cbdc606dce5b79a516e16492a.
due to global heap pressure. Total Memstore size=3.2 G, Region memstore
size=331.4 M
2017-07-03 16:11:57,571 WARN
 [B.defaultRpcServer.handler=49,queue=11,port=16020]
regionserver.MemStoreFlusher: Memstore is above high water mark and block
2892ms

Best Regards,
Yu

On 6 July 2017 at 00:56, Stack  wrote:

> On Wed, Jul 5, 2017 at 6:30 AM, Eshcar Hillel  >
> wrote:
>
> > Hi All,
> > I opened a new Jira https://issues.apache.org/jira/browse/HBASE-18294 to
> > discuss this question.
> > Flush decisions are taken at the region level and also at the region
> > server level - there is the question of when to trigger a flush and then
> > which region/store to flush.Regions track both their data size (key-value
> > size only) and their total heap occupancy (including index and additional
> > metadata).One option (which was the past policy) is to trigger flushes
> and
> > choose flush subjects based on regions heap size - this gives a better
> > estimation for sysadmin of how many regions can a RS carry.Another option
> > (which is the current policy) is to look at the data size - this gives a
> > better estimation of the size of the files that are created by the flush.
> >
>
>
> Sounds like we should be doing the former, heap occupancy. An
> OutOfMemoryException puts a nail in any benefit other accountings might
> have.
>
> St.Ack
>
>
>
> > I see this is as critical to HBase performance and usability, namely
> > meeting the user expectation from the system, hence I would like to hear
> as
> > many voices as possible.Please join the discussion in the Jira and let us
> > know what you think.
> > Thanks,Eshcar
> >
> >
>

[jira] [Created] (HBASE-18325) Disable flakey TestMasterProcedureWalLease

2017-07-05 Thread stack (JIRA) 
stack created HBASE-18325:
-

 Summary: Disable flakey TestMasterProcedureWalLease
 Key: HBASE-18325
 URL: https://issues.apache.org/jira/browse/HBASE-18325
 Project: HBase
  Issue Type: Bug
  Components: test
Reporter: stack






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Re: [ANNOUNCE] Devaraj Das joins the Apache HBase PMC

2017-07-05 Thread Guanghao Zhang 
Congratulations!

2017-07-06 10:52 GMT+08:00 Yu Li :

> Congratulations Devaraj!
>
> Best Regards,
> Yu
>
> On 6 July 2017 at 10:37, Chunhui Shen  wrote:
>
> > Congratulations!
> >
> >
> > At 2017-07-06 09:52:17, "Allan Yang"  wrote:
> > >Congratulations, Devaraj !
> > >
> > >Best Regards
> > >Allan Yang
> > >
> > >2017-07-06 9:44 GMT+08:00 Pankaj kr :
> > >
> > >> Congratulations Devaraj..!!
> > >>
> > >> Thanks & Regards,
> > >> Pankaj
> > >>
> > >> HUAWEI TECHNOLOGIES CO.LTD.
> > >> Huawei Tecnologies India Pvt. Ltd.
> > >> Near EPIP Industrial Area, Kundalahalli Village
> > >> Whitefield, Bangalore-560066
> > >> www.huawei.com
> > >> 
> > >> -
> > >> This e-mail and its attachments contain confidential information from
> > >> HUAWEI, which
> > >> is intended only for the person or entity whose address is listed
> above.
> > >> Any use of the
> > >> information contained herein in any way (including, but not limited
> to,
> > >> total or partial
> > >> disclosure, reproduction, or dissemination) by persons other than the
> > >> intended
> > >> recipient(s) is prohibited. If you receive this e-mail in error,
> please
> > >> notify the sender by
> > >> phone or email immediately and delete it!
> > >>
> > >>
> > >> -Original Message-
> > >> From: Josh Elser [mailto:els...@apache.org]
> > >> Sent: Thursday, July 06, 2017 12:27 AM
> > >> To: dev@hbase.apache.org; u...@hbase.apache.org
> > >> Subject: [ANNOUNCE] Devaraj Das joins the Apache HBase PMC
> > >>
> > >> I'm pleased to announce yet another PMC addition in the form of
> Devaraj
> > >> Das. One of the "old guard" in the broader Hadoop umbrella, he's also
> a
> > >> long-standing member in our community. We all look forward to the
> > continued
> > >> contributions and project leadership.
> > >>
> > >> Please join me in welcoming Devaraj!
> > >>
> > >> - Josh (on behalf of the PMC)
> > >>
> >
>

Re: Re: [ANNOUNCE] Devaraj Das joins the Apache HBase PMC

2017-07-05 Thread Yu Li 
Congratulations Devaraj!

Best Regards,
Yu

On 6 July 2017 at 10:37, Chunhui Shen  wrote:

> Congratulations!
>
>
> At 2017-07-06 09:52:17, "Allan Yang"  wrote:
> >Congratulations, Devaraj !
> >
> >Best Regards
> >Allan Yang
> >
> >2017-07-06 9:44 GMT+08:00 Pankaj kr :
> >
> >> Congratulations Devaraj..!!
> >>
> >> Thanks & Regards,
> >> Pankaj
> >>
> >> HUAWEI TECHNOLOGIES CO.LTD.
> >> Huawei Tecnologies India Pvt. Ltd.
> >> Near EPIP Industrial Area, Kundalahalli Village
> >> Whitefield, Bangalore-560066
> >> www.huawei.com
> >> 
> >> -
> >> This e-mail and its attachments contain confidential information from
> >> HUAWEI, which
> >> is intended only for the person or entity whose address is listed above.
> >> Any use of the
> >> information contained herein in any way (including, but not limited to,
> >> total or partial
> >> disclosure, reproduction, or dissemination) by persons other than the
> >> intended
> >> recipient(s) is prohibited. If you receive this e-mail in error, please
> >> notify the sender by
> >> phone or email immediately and delete it!
> >>
> >>
> >> -Original Message-
> >> From: Josh Elser [mailto:els...@apache.org]
> >> Sent: Thursday, July 06, 2017 12:27 AM
> >> To: dev@hbase.apache.org; u...@hbase.apache.org
> >> Subject: [ANNOUNCE] Devaraj Das joins the Apache HBase PMC
> >>
> >> I'm pleased to announce yet another PMC addition in the form of Devaraj
> >> Das. One of the "old guard" in the broader Hadoop umbrella, he's also a
> >> long-standing member in our community. We all look forward to the
> continued
> >> contributions and project leadership.
> >>
> >> Please join me in welcoming Devaraj!
> >>
> >> - Josh (on behalf of the PMC)
> >>
>

Re:Re: [ANNOUNCE] Devaraj Das joins the Apache HBase PMC

2017-07-05 Thread Chunhui Shen 
Congratulations!


At 2017-07-06 09:52:17, "Allan Yang"  wrote:
>Congratulations, Devaraj !
>
>Best Regards
>Allan Yang
>
>2017-07-06 9:44 GMT+08:00 Pankaj kr :
>
>> Congratulations Devaraj..!!
>>
>> Thanks & Regards,
>> Pankaj
>>
>> HUAWEI TECHNOLOGIES CO.LTD.
>> Huawei Tecnologies India Pvt. Ltd.
>> Near EPIP Industrial Area, Kundalahalli Village
>> Whitefield, Bangalore-560066
>> www.huawei.com
>> 
>> -
>> This e-mail and its attachments contain confidential information from
>> HUAWEI, which
>> is intended only for the person or entity whose address is listed above.
>> Any use of the
>> information contained herein in any way (including, but not limited to,
>> total or partial
>> disclosure, reproduction, or dissemination) by persons other than the
>> intended
>> recipient(s) is prohibited. If you receive this e-mail in error, please
>> notify the sender by
>> phone or email immediately and delete it!
>>
>>
>> -Original Message-
>> From: Josh Elser [mailto:els...@apache.org]
>> Sent: Thursday, July 06, 2017 12:27 AM
>> To: dev@hbase.apache.org; u...@hbase.apache.org
>> Subject: [ANNOUNCE] Devaraj Das joins the Apache HBase PMC
>>
>> I'm pleased to announce yet another PMC addition in the form of Devaraj
>> Das. One of the "old guard" in the broader Hadoop umbrella, he's also a
>> long-standing member in our community. We all look forward to the continued
>> contributions and project leadership.
>>
>> Please join me in welcoming Devaraj!
>>
>> - Josh (on behalf of the PMC)
>>

Re: [ANNOUNCE] Devaraj Das joins the Apache HBase PMC

2017-07-05 Thread Allan Yang 
Congratulations, Devaraj !

Best Regards
Allan Yang

2017-07-06 9:44 GMT+08:00 Pankaj kr :

> Congratulations Devaraj..!!
>
> Thanks & Regards,
> Pankaj
>
> HUAWEI TECHNOLOGIES CO.LTD.
> Huawei Tecnologies India Pvt. Ltd.
> Near EPIP Industrial Area, Kundalahalli Village
> Whitefield, Bangalore-560066
> www.huawei.com
> 
> -
> This e-mail and its attachments contain confidential information from
> HUAWEI, which
> is intended only for the person or entity whose address is listed above.
> Any use of the
> information contained herein in any way (including, but not limited to,
> total or partial
> disclosure, reproduction, or dissemination) by persons other than the
> intended
> recipient(s) is prohibited. If you receive this e-mail in error, please
> notify the sender by
> phone or email immediately and delete it!
>
>
> -Original Message-
> From: Josh Elser [mailto:els...@apache.org]
> Sent: Thursday, July 06, 2017 12:27 AM
> To: dev@hbase.apache.org; u...@hbase.apache.org
> Subject: [ANNOUNCE] Devaraj Das joins the Apache HBase PMC
>
> I'm pleased to announce yet another PMC addition in the form of Devaraj
> Das. One of the "old guard" in the broader Hadoop umbrella, he's also a
> long-standing member in our community. We all look forward to the continued
> contributions and project leadership.
>
> Please join me in welcoming Devaraj!
>
> - Josh (on behalf of the PMC)
>

RE: [ANNOUNCE] Devaraj Das joins the Apache HBase PMC

2017-07-05 Thread Pankaj kr 
Congratulations Devaraj..!! 

Thanks & Regards,
Pankaj

HUAWEI TECHNOLOGIES CO.LTD.
Huawei Tecnologies India Pvt. Ltd.
Near EPIP Industrial Area, Kundalahalli Village
Whitefield, Bangalore-560066
www.huawei.com
-
This e-mail and its attachments contain confidential information from HUAWEI, 
which 
is intended only for the person or entity whose address is listed above. Any 
use of the 
information contained herein in any way (including, but not limited to, total 
or partial 
disclosure, reproduction, or dissemination) by persons other than the intended 
recipient(s) is prohibited. If you receive this e-mail in error, please notify 
the sender by 
phone or email immediately and delete it!


-Original Message-
From: Josh Elser [mailto:els...@apache.org] 
Sent: Thursday, July 06, 2017 12:27 AM
To: dev@hbase.apache.org; u...@hbase.apache.org
Subject: [ANNOUNCE] Devaraj Das joins the Apache HBase PMC

I'm pleased to announce yet another PMC addition in the form of Devaraj Das. 
One of the "old guard" in the broader Hadoop umbrella, he's also a 
long-standing member in our community. We all look forward to the continued 
contributions and project leadership.

Please join me in welcoming Devaraj!

- Josh (on behalf of the PMC)

Re: [ANNOUNCE] Devaraj Das joins the Apache HBase PMC

2017-07-05 Thread Jingcheng Du 
Congratulations!!!

2017-07-06 5:52 GMT+08:00 Jerry He :

> Congrats, Devaraj !
>
>
> Thanks,
>
> Jerry
>
> On Wed, Jul 5, 2017 at 1:36 PM, Nick Dimiduk  wrote:
> > Congratulations Devaraj!
> >
> > On Wed, Jul 5, 2017 at 9:27 AM, Josh Elser  wrote:
> >
> >> I'm pleased to announce yet another PMC addition in the form of Devaraj
> >> Das. One of the "old guard" in the broader Hadoop umbrella, he's also a
> >> long-standing member in our community. We all look forward to the
> continued
> >> contributions and project leadership.
> >>
> >> Please join me in welcoming Devaraj!
> >>
> >> - Josh (on behalf of the PMC)
> >>
>

[jira] [Reopened] (HBASE-18107) [AMv2] Remove DispatchMergingRegionsRequest & DispatchMergingRegions

2017-07-05 Thread Yi Liang (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HBASE-18107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yi Liang reopened HBASE-18107:
--
  Assignee: Yi Liang

> [AMv2] Remove DispatchMergingRegionsRequest & DispatchMergingRegions
> 
>
> Key: HBASE-18107
> URL: https://issues.apache.org/jira/browse/HBASE-18107
> Project: HBase
>  Issue Type: Sub-task
>  Components: Region Assignment
>Affects Versions: 2.0.0
>Reporter: stack
>Assignee: Yi Liang
> Fix For: 2.0.0
>
>
> They don't align with how we have named the Split equivalents; i.e. 
> SplitRegion (so should be MergeRegion...). They probably have these awkward 
> names because the obvious slots are occupied... so this may not be fixable 
> but filing issue anyways.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HBASE-18324) [hbase-thirdparty] Tooling to prevent commits that mistakenly make use of unshaded protobuf/guava/etc.

2017-07-05 Thread stack (JIRA) 
stack created HBASE-18324:
-

 Summary: [hbase-thirdparty] Tooling to prevent commits that 
mistakenly make use of unshaded protobuf/guava/etc.
 Key: HBASE-18324
 URL: https://issues.apache.org/jira/browse/HBASE-18324
 Project: HBase
  Issue Type: Bug
Reporter: stack


Chatting w/ [~mdrob], he brought up the question of what if a dev makes use of 
unshaded protobuf or guava? Afterall, the old jars (pb2.5 and hadoops guava12.0 
or whatever) are still on the CLASSPATH because upstream depends on them.

We could add a check as part of prebuild. It would complain if use of 
com.google instead of org.apache.hadoop.hbase.shaded.com.google. But even then, 
there are cases where com.google is legit (coprocessor endpoints) so it would 
have to let these pass.

TODO.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: [ANNOUNCE] Devaraj Das joins the Apache HBase PMC

2017-07-05 Thread Jerry He 
Congrats, Devaraj !


Thanks,

Jerry

On Wed, Jul 5, 2017 at 1:36 PM, Nick Dimiduk  wrote:
> Congratulations Devaraj!
>
> On Wed, Jul 5, 2017 at 9:27 AM, Josh Elser  wrote:
>
>> I'm pleased to announce yet another PMC addition in the form of Devaraj
>> Das. One of the "old guard" in the broader Hadoop umbrella, he's also a
>> long-standing member in our community. We all look forward to the continued
>> contributions and project leadership.
>>
>> Please join me in welcoming Devaraj!
>>
>> - Josh (on behalf of the PMC)
>>

[jira] [Resolved] (HBASE-15349) Update surefire version to 2.19.1

2017-07-05 Thread stack (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HBASE-15349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack resolved HBASE-15349.
---
Resolution: Implemented

Resolving as implemented. Was fixed by

author Peter Somogyi  Thu Jun 29 15:37:22 2017 +0200
committer Michael Stack  Mon Jul 3 19:43:42 2017 -0700

HBASE-18264 Update pom plugins

Update plugins in main and subprojects
Unified versions to use variable instead of direct values

> Update surefire version to 2.19.1
> -
>
> Key: HBASE-15349
> URL: https://issues.apache.org/jira/browse/HBASE-15349
> Project: HBase
>  Issue Type: Improvement
>Reporter: Appy
>Assignee: Appy
> Fix For: 2.0.0
>
> Attachments: HBASE-15349.patch
>
>
> So that new properties like surefire.excludesFile and includesFile can be 
> used to easily exclude/include flaky tests.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (HBASE-18255) Time-Delayed HBase Performance Degradation with Java 7

2017-07-05 Thread Josh Elser (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HBASE-18255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Josh Elser resolved HBASE-18255.

  Resolution: Fixed
Hadoop Flags: Reviewed

Pushed to all 1.x branches. Thanks for the quick patch, Vlad!

> Time-Delayed HBase Performance Degradation with Java 7
> --
>
> Key: HBASE-18255
> URL: https://issues.apache.org/jira/browse/HBASE-18255
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.3.1, 1.2.6, 1.1.11
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Fix For: 1.4.0, 1.3.2, 1.2.7, 1.1.12
>
> Attachments: HBASE-18255-branch-1.x.v1.patch
>
>
> The good summary of the issue and provided resolution can be found in this 
> article:
> https://community.hortonworks.com/articles/105802/time-delayed-hbase-performance-degradation-with-ja.html
> In a few words, due to internal JVM 7 bug (which has been addressed only in 
> Java 8), HotSpot code cache can become full and after that ALL JIT 
> compilations get suspended indefinitely.  The default value for code cache 
> size in JVM 7 is quite low: 48MB. It is recommended to increase this value at 
> least to 256MB (default in JVM 8).
> This BUG affects only 1.x 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (HBASE-16825) Backup mega patch: Review 52748 work

2017-07-05 Thread Vladimir Rodionov (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HBASE-16825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vladimir Rodionov resolved HBASE-16825.
---
Resolution: Fixed

> Backup mega patch: Review 52748 work
> 
>
> Key: HBASE-16825
> URL: https://issues.apache.org/jira/browse/HBASE-16825
> Project: HBase
>  Issue Type: Task
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Fix For: 2.0.0
>
>
> This ticket to address comments/issues raised in RB:
> https://reviews.apache.org/r/52748



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (HBASE-17533) NPE in backup create session

2017-07-05 Thread Vladimir Rodionov (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HBASE-17533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vladimir Rodionov resolved HBASE-17533.
---
Resolution: Invalid

{code}
/hbase backup create "full" file:///tmp/
{code}
Backup does not support local file system as destination.

> NPE in backup create session
> 
>
> Key: HBASE-17533
> URL: https://issues.apache.org/jira/browse/HBASE-17533
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Vladimir Rodionov
> Fix For: HBASE-7912
>
>
> When executing backup in a fresh cluster, I got NPEs: 
> HW10676:hbase-backup-phase-1$ bin/hbase backup create "full" file:///tmp/ 
> 2017-01-12 16:18:10,633 WARN [main] util.NativeCodeLoader: Unable to load 
> native-hadoop library for your platform... using builtin-java classes where 
> applicable
> 2017-01-12 16:18:11,545 INFO [main] impl.BackupManager: Full backup all the 
> tables available in the cluster: [backup.system]
> 2017-01-12 16:18:11,549 INFO [main] impl.TableBackupClient: Backup 
> backup_1484266691104 started at 1484266691548.
> 2017-01-12 16:18:11,756 INFO [main] impl.FullTableBackupClient: Execute roll 
> log procedure for full backup ...
> 2017-01-12 16:18:12,138 WARN [main] util.BackupServerUtil: Can not 
> parsefile:/var/folders/h5/bbxg50c12r5bhsd9hf1n9hlrgp/T/hbase-enis/hbase/oldWALs/10.22.7.15%2C63190%2C1484265627048.1484265631575
> java.lang.NullPointerException
> at 
> org.apache.hadoop.hbase.backup.util.BackupServerUtil$1.accept(BackupServerUtil.java:370)
> at 
> org.apache.hadoop.hbase.backup.util.BackupClientUtil.getFiles(BackupClientUtil.java:172)
> at 
> org.apache.hadoop.hbase.backup.util.BackupServerUtil.getWALFilesOlderThan(BackupServerUtil.java:379)
> at 
> org.apache.hadoop.hbase.backup.impl.FullTableBackupClient.execute(FullTableBackupClient.java:137)
> at 
> org.apache.hadoop.hbase.backup.impl.BackupAdminImpl.backupTables(BackupAdminImpl.java:522)
> at 
> org.apache.hadoop.hbase.backup.impl.BackupCommands$CreateCommand.execute(BackupCommands.java:251)
> at 
> org.apache.hadoop.hbase.backup.BackupDriver.parseAndRun(BackupDriver.java:126)
> at org.apache.hadoop.hbase.backup.BackupDriver.doWork(BackupDriver.java:155)
> at org.apache.hadoop.hbase.backup.BackupDriver.run(BackupDriver.java:188)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.hbase.backup.BackupDriver.main(BackupDriver.java:163)
> 2017-01-12 16:18:12,157 WARN [main] util.BackupServerUtil: Can not



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (HBASE-17535) Remove backup manifest duplicates from backup file system

2017-07-05 Thread Vladimir Rodionov (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HBASE-17535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vladimir Rodionov resolved HBASE-17535.
---
Resolution: Duplicate

This is going to be fixed in HBASE-14135. Closing as duplicate

> Remove backup manifest duplicates from backup file system
> -
>
> Key: HBASE-17535
> URL: https://issues.apache.org/jira/browse/HBASE-17535
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Fix For: 2.0.0
>
>
> It seems that in the backup directory layout, we are serializing the 
> backupimage in all of the tables like this: 
> /tmp/backup_1484268153482//default/t1/.backup.manifest
> /tmp/backup_1484268153482//default/t2/.backup.manifest
> /tmp/backup_1484268153482//default/t3/.backup.manifest
> however the contents are the same, and backupImage itself is a container for 
> multi-table backup. What is the reason that we are serializing this many 
> times? It is even under WALs/ dir for incremental backups.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: [ANNOUNCE] Devaraj Das joins the Apache HBase PMC

2017-07-05 Thread Nick Dimiduk 
Congratulations Devaraj!

On Wed, Jul 5, 2017 at 9:27 AM, Josh Elser  wrote:

> I'm pleased to announce yet another PMC addition in the form of Devaraj
> Das. One of the "old guard" in the broader Hadoop umbrella, he's also a
> long-standing member in our community. We all look forward to the continued
> contributions and project leadership.
>
> Please join me in welcoming Devaraj!
>
> - Josh (on behalf of the PMC)
>

Re: [ANNOUNCE] Devaraj Das joins the Apache HBase PMC

2017-07-05 Thread Stephen Jiang 
Congratulations!

Thanks
Stephen

On Wed, Jul 5, 2017 at 11:00 AM, Karan Mehta  wrote:

> Congrats Devraj!
>
> On Wed, Jul 5, 2017 at 10:59 AM ramkrishna vasudevan <
> ramkrishna.s.vasude...@gmail.com> wrote:
>
> > Congratulations, Devaraj !!!
> >
> > On Wed, Jul 5, 2017 at 10:13 PM, Andrew Purtell 
> > wrote:
> >
> > > Congratulations, Devaraj!
> > >
> > > On Wed, Jul 5, 2017 at 9:27 AM, Josh Elser  wrote:
> > >
> > > > I'm pleased to announce yet another PMC addition in the form of
> Devaraj
> > > > Das. One of the "old guard" in the broader Hadoop umbrella, he's
> also a
> > > > long-standing member in our community. We all look forward to the
> > > continued
> > > > contributions and project leadership.
> > > >
> > > > Please join me in welcoming Devaraj!
> > > >
> > > > - Josh (on behalf of the PMC)
> > > >
> > >
> > >
> > >
> > > --
> > > Best regards,
> > > Andrew
> > >
> > > Words like orphans lost among the crosstalk, meaning torn from truth's
> > > decrepit hands
> > >- A23, Crosstalk
> > >
> >
>

Re: [ANNOUNCE] Devaraj Das joins the Apache HBase PMC

2017-07-05 Thread Karan Mehta 
Congrats Devraj!

On Wed, Jul 5, 2017 at 10:59 AM ramkrishna vasudevan <
ramkrishna.s.vasude...@gmail.com> wrote:

> Congratulations, Devaraj !!!
>
> On Wed, Jul 5, 2017 at 10:13 PM, Andrew Purtell 
> wrote:
>
> > Congratulations, Devaraj!
> >
> > On Wed, Jul 5, 2017 at 9:27 AM, Josh Elser  wrote:
> >
> > > I'm pleased to announce yet another PMC addition in the form of Devaraj
> > > Das. One of the "old guard" in the broader Hadoop umbrella, he's also a
> > > long-standing member in our community. We all look forward to the
> > continued
> > > contributions and project leadership.
> > >
> > > Please join me in welcoming Devaraj!
> > >
> > > - Josh (on behalf of the PMC)
> > >
> >
> >
> >
> > --
> > Best regards,
> > Andrew
> >
> > Words like orphans lost among the crosstalk, meaning torn from truth's
> > decrepit hands
> >- A23, Crosstalk
> >
>

Re: [ANNOUNCE] Devaraj Das joins the Apache HBase PMC

2017-07-05 Thread ramkrishna vasudevan 
Congratulations, Devaraj !!!

On Wed, Jul 5, 2017 at 10:13 PM, Andrew Purtell  wrote:

> Congratulations, Devaraj!
>
> On Wed, Jul 5, 2017 at 9:27 AM, Josh Elser  wrote:
>
> > I'm pleased to announce yet another PMC addition in the form of Devaraj
> > Das. One of the "old guard" in the broader Hadoop umbrella, he's also a
> > long-standing member in our community. We all look forward to the
> continued
> > contributions and project leadership.
> >
> > Please join me in welcoming Devaraj!
> >
> > - Josh (on behalf of the PMC)
> >
>
>
>
> --
> Best regards,
> Andrew
>
> Words like orphans lost among the crosstalk, meaning torn from truth's
> decrepit hands
>- A23, Crosstalk
>

Re: [DISCUSS] Should flush decisions be made based on data size (key-value only) or based on heap size (including metadata overhead)?

2017-07-05 Thread Stack 
On Wed, Jul 5, 2017 at 6:30 AM, Eshcar Hillel 
wrote:

> Hi All,
> I opened a new Jira https://issues.apache.org/jira/browse/HBASE-18294 to
> discuss this question.
> Flush decisions are taken at the region level and also at the region
> server level - there is the question of when to trigger a flush and then
> which region/store to flush.Regions track both their data size (key-value
> size only) and their total heap occupancy (including index and additional
> metadata).One option (which was the past policy) is to trigger flushes and
> choose flush subjects based on regions heap size - this gives a better
> estimation for sysadmin of how many regions can a RS carry.Another option
> (which is the current policy) is to look at the data size - this gives a
> better estimation of the size of the files that are created by the flush.
>


Sounds like we should be doing the former, heap occupancy. An
OutOfMemoryException puts a nail in any benefit other accountings might
have.

St.Ack



> I see this is as critical to HBase performance and usability, namely
> meeting the user expectation from the system, hence I would like to hear as
> many voices as possible.Please join the discussion in the Jira and let us
> know what you think.
> Thanks,Eshcar
>
>

Re: [ANNOUNCE] Devaraj Das joins the Apache HBase PMC

2017-07-05 Thread Andrew Purtell 
Congratulations, Devaraj!

On Wed, Jul 5, 2017 at 9:27 AM, Josh Elser  wrote:

> I'm pleased to announce yet another PMC addition in the form of Devaraj
> Das. One of the "old guard" in the broader Hadoop umbrella, he's also a
> long-standing member in our community. We all look forward to the continued
> contributions and project leadership.
>
> Please join me in welcoming Devaraj!
>
> - Josh (on behalf of the PMC)
>



-- 
Best regards,
Andrew

Words like orphans lost among the crosstalk, meaning torn from truth's
decrepit hands
   - A23, Crosstalk

Re: [ANNOUNCE] Chunhui Shen joins the Apache HBase PMC

2017-07-05 Thread Ashish Singhi 
Congratulations!

Sent from my iPhone

> On 04-Jul-2017, at 10:54 AM, Yu Li  wrote:
> 
> On behalf of the Apache HBase PMC I am pleased to announce that Chunhui Shen
> has accepted our invitation to become a PMC member on the Apache
> HBase project. He has been an active contributor to HBase for past many
> years. Looking forward for many more contributions from him.
> 
> Please join me in welcoming Chunhui to the HBase PMC!
> 
> Best Regards,
> Yu

Re: [ANNOUNCE] Devaraj Das joins the Apache HBase PMC

2017-07-05 Thread Ashish Singhi 
Congratulations!

Sent from my iPhone

> On 05-Jul-2017, at 9:57 PM, Josh Elser  wrote:
> 
> I'm pleased to announce yet another PMC addition in the form of Devaraj Das. 
> One of the "old guard" in the broader Hadoop umbrella, he's also a 
> long-standing member in our community. We all look forward to the continued 
> contributions and project leadership.
> 
> Please join me in welcoming Devaraj!
> 
> - Josh (on behalf of the PMC)

Re: [ANNOUNCE] Devaraj Das joins the Apache HBase PMC

2017-07-05 Thread Ted Yu 
Congratulations, Devaraj !

On Wed, Jul 5, 2017 at 9:27 AM, Josh Elser  wrote:

> I'm pleased to announce yet another PMC addition in the form of Devaraj
> Das. One of the "old guard" in the broader Hadoop umbrella, he's also a
> long-standing member in our community. We all look forward to the continued
> contributions and project leadership.
>
> Please join me in welcoming Devaraj!
>
> - Josh (on behalf of the PMC)
>

[ANNOUNCE] Devaraj Das joins the Apache HBase PMC

2017-07-05 Thread Josh Elser 
I'm pleased to announce yet another PMC addition in the form of Devaraj 
Das. One of the "old guard" in the broader Hadoop umbrella, he's also a 
long-standing member in our community. We all look forward to the 
continued contributions and project leadership.


Please join me in welcoming Devaraj!

- Josh (on behalf of the PMC)

[jira] [Resolved] (HBASE-18240) Add hbase-thirdparty, a project with hbase utility including an hbase-shaded-thirdparty module with guava, netty, etc.

2017-07-05 Thread stack (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HBASE-18240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack resolved HBASE-18240.
---
  Resolution: Fixed
Hadoop Flags: Reviewed

We release hbase-thirdparty-1.0.0 (See '[RESULT][VOTE] hbase-thirdparty 
project, first release candidate 1.0.0RC0' thread -- no fancy ANNOUNCE for an 
internal lib).  Adjacent issues under this parent are about moving to use this 
new lib for guava, nettty, etc.

> Add hbase-thirdparty, a project with hbase utility including an 
> hbase-shaded-thirdparty module with guava, netty, etc.
> --
>
> Key: HBASE-18240
> URL: https://issues.apache.org/jira/browse/HBASE-18240
> Project: HBase
>  Issue Type: Sub-task
>  Components: dependencies, shading
>Reporter: stack
>Assignee: stack
> Fix For: 2.0.0
>
> Attachments: HBASE-18240.master.001.patch, hbase-auxillary.tgz
>
>
> This issue is about adding a new related project to host hbase auxillary 
> utility. In this new project, the first thing we'd add is a module to host 
> shaded versions of third party libraries.
> This task comes of discussion held here 
> http://apache-hbase.679495.n3.nabble.com/DISCUSS-More-Shading-td4083025.html 
> where one conclusion of the DISCUSSION was "... pushing this part forward 
> with some code is the next logical step. Seems to be consensus about taking 
> our known internal dependencies and performing this shade magic."



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

RE: [ANNOUNCE] Chunhui Shen joins the Apache HBase PMC

2017-07-05 Thread QI Congyun 
Congratulations, Chunhui.

-Original Message-
From: Jerry He [mailto:jerry...@gmail.com] 
Sent: Wednesday, July 05, 2017 1:02 PM
To: dev@hbase.apache.org; u...@hbase.apache.org
Subject: Re: [ANNOUNCE] Chunhui Shen joins the Apache HBase PMC

Congrats,  Chunhui!

Thanks

Jerry

On Tue, Jul 4, 2017 at 8:37 PM Anoop John  wrote:

> Congrats Chunhui..
>
> On Wed, Jul 5, 2017 at 6:55 AM, Pankaj kr  wrote:
> > Congratulations Chunhui..!!
> >
> > Regards,
> > Pankaj
> >
> >
> > -Original Message-
> > From: Yu Li [mailto:car...@gmail.com]
> > Sent: Tuesday, July 04, 2017 1:24 PM
> > To: dev@hbase.apache.org; Hbase-User
> > Subject: [ANNOUNCE] Chunhui Shen joins the Apache HBase PMC
> >
> > On behalf of the Apache HBase PMC I am pleased to announce that 
> > Chunhui
> Shen has accepted our invitation to become a PMC member on the Apache 
> HBase project. He has been an active contributor to HBase for past many years.
> Looking forward for many more contributions from him.
> >
> > Please join me in welcoming Chunhui to the HBase PMC!
> >
> > Best Regards,
> > Yu
>

[DISCUSS] Should flush decisions be made based on data size (key-value only) or based on heap size (including metadata overhead)?

2017-07-05 Thread Eshcar Hillel 
Hi All,
I opened a new Jira https://issues.apache.org/jira/browse/HBASE-18294 to 
discuss this question.
Flush decisions are taken at the region level and also at the region server 
level - there is the question of when to trigger a flush and then which 
region/store to flush.Regions track both their data size (key-value size only) 
and their total heap occupancy (including index and additional metadata).One 
option (which was the past policy) is to trigger flushes and choose flush 
subjects based on regions heap size - this gives a better estimation for 
sysadmin of how many regions can a RS carry.Another option (which is the 
current policy) is to look at the data size - this gives a better estimation of 
the size of the files that are created by the flush.  
I see this is as critical to HBase performance and usability, namely meeting 
the user expectation from the system, hence I would like to hear as many voices 
as possible.Please join the discussion in the Jira and let us know what you 
think.
Thanks,Eshcar

Re: [ANNOUNCE] Chunhui Shen joins the Apache HBase PMC

2017-07-05 Thread Phil Yang 
Congratulations!

Thanks,
Phil


2017-07-05 13:05 GMT+08:00 ramkrishna vasudevan <
ramkrishna.s.vasude...@gmail.com>:

> Congratulations !!!
>
> On Wed, Jul 5, 2017 at 10:33 AM, QI Congyun  cn
> > wrote:
>
> > Congratulations, Chunhui.
> >
> > -Original Message-
> > From: Jerry He [mailto:jerry...@gmail.com]
> > Sent: Wednesday, July 05, 2017 1:02 PM
> > To: dev@hbase.apache.org; u...@hbase.apache.org
> > Subject: Re: [ANNOUNCE] Chunhui Shen joins the Apache HBase PMC
> >
> > Congrats,  Chunhui!
> >
> > Thanks
> >
> > Jerry
> >
> > On Tue, Jul 4, 2017 at 8:37 PM Anoop John  wrote:
> >
> > > Congrats Chunhui..
> > >
> > > On Wed, Jul 5, 2017 at 6:55 AM, Pankaj kr 
> wrote:
> > > > Congratulations Chunhui..!!
> > > >
> > > > Regards,
> > > > Pankaj
> > > >
> > > >
> > > > -Original Message-
> > > > From: Yu Li [mailto:car...@gmail.com]
> > > > Sent: Tuesday, July 04, 2017 1:24 PM
> > > > To: dev@hbase.apache.org; Hbase-User
> > > > Subject: [ANNOUNCE] Chunhui Shen joins the Apache HBase PMC
> > > >
> > > > On behalf of the Apache HBase PMC I am pleased to announce that
> > > > Chunhui
> > > Shen has accepted our invitation to become a PMC member on the Apache
> > > HBase project. He has been an active contributor to HBase for past many
> > years.
> > > Looking forward for many more contributions from him.
> > > >
> > > > Please join me in welcoming Chunhui to the HBase PMC!
> > > >
> > > > Best Regards,
> > > > Yu
> > >
> >
>

[jira] [Created] (HBASE-18323) Remove multiple ACLs for the same user in kerberos

2017-07-05 Thread Shibin Zhang (JIRA) 
Shibin Zhang created HBASE-18323:


 Summary: Remove multiple ACLs for the same user in kerberos
 Key: HBASE-18323
 URL: https://issues.apache.org/jira/browse/HBASE-18323
 Project: HBase
  Issue Type: Bug
Affects Versions: 3.0.0
Reporter: Shibin Zhang
Priority: Critical


When deploy hbase in kerberos way ,there will be multiple acls in znode :
'world,'anyone
: r
'sasl,'hbase
: cdrwa
'sasl,'hbase
: cdrwa

I also see the related issue and apply the patch, like  
https://issues.apache.org/jira/browse/HBASE-17717 
but in my environment ,this situation still appear,

After dig into the code , i found the reason in source code  ZKUtil.createAcl  
is

 if (zkw.isClientReadable(node)) {
LOG.error("isSecureZooKeeper user: clientReadable");
acls.addAll(Ids.CREATOR_ALL_ACL);
acls.addAll(Ids.READ_ACL_UNSAFE);
  } else {
LOG.error("isSecureZooKeeper user: clientReadable no");
acls.addAll(Ids.CREATOR_ALL_ACL);
  } 


  acls.addAll(Ids.CREATOR_ALL_ACL);   
  
  Id AUTH_IDS = new Id("auth", "");

ArrayList CREATOR_ALL_ACL = new ArrayList(Collections.singletonList(new 
ACL(31, AUTH_IDS)));


AUTH_IDS   with  "auth " will result  current connection auth user  add to 
znode acl ,
so it will appear multiple acls for same users.


I think this line of code  we can remove  :  acls.addAll(Ids.CREATOR_ALL_ACL);  
 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)