[ANNOUNCE] New HBase committer Lars Francke

[Lars George]

On behalf of the Apache HBase PMC, I am pleased to announce that Lars
Francke has accepted the PMC's invitation to become a committer on the
project.

We appreciate all of Lars' great work thus far and look forward to
continued involvement.

Please join me in congratulating LarsF! (Opting to use last name
initials as we now have three Lars' as committers)

--
Best regards,
LarsG

HBase Backup Mode

[Lars George]

Hi,

I stumbled across this:
hbase-server/src/main/java/org/apache/hadoop/hbase/backup/example/HFileArchiveManager.java

I am wondering what the state of this is. Also, why is this in an
_example_ package, if it is usable code. Should be in `util` or so
instead? If it isn't really used at all or just really an example as
it states, should this be in GitHub instead?

Let me know what you think.

Best,
Lars

Max queue size in replication

[Lars George]

Hi,

I am looking at the replication code, and it has this line:

this.queueSizePerGroup = this.conf.getInt("hbase.regionserver.maxlogs", 32);

This drives the queue size accepting more WAL files. But we have a
different computation now, using the heap size instead, so often have
a maxlogs > 32 in practice, while the config key is not set anymore.
Should this not be aligned?

Lars

Re: Tags class using wrong length?

[Lars George]

Gotcha, sorry for the noise. I documented properly in the upcoming HBase book. 
:)

Sent from my iPhone

> On 7. Aug 2017, at 07:02, ramkrishna vasudevan 
> <ramkrishna.s.vasude...@gmail.com> wrote:
> 
> I think the layout of tags is missing now in the javadoc. May be it got
> missed or moved to some other place?
> I remember we had a layout explaining the tag structure then this code is
> much easier to read this code.
> 
> As Chia-Ping said  is the
> layout.
> So from the KeyValue lay out we extract the tag part which in itself has a
> tag length to represent the complete set of tags.
> 
> From the tags offset and tags length from the KV we extract individual tags
> in that KV.
> 
> For eg
> See TagUtil#asList
> 
> {code}
> List tags = new ArrayList<>();
>int pos = offset;
>while (pos < offset + length) {
>  int tagLen = Bytes.readAsInt(b, pos, TAG_LENGTH_SIZE);
>  tags.add(new ArrayBackedTag(b, pos, tagLen + TAG_LENGTH_SIZE));
>  pos += TAG_LENGTH_SIZE + tagLen;
>}
>return tags;
> {code}
> 
> Regards
> Ram
> 
> 
> 
> 
>> On Mon, Aug 7, 2017 at 3:25 AM, Ted Yu <yuzhih...@gmail.com> wrote:
>> 
>> The byte following the tag length (a short) is the tag type.
>> 
>> The current code is correct.
>> 
>> On Sun, Aug 6, 2017 at 5:40 AM, Chia-Ping Tsai <chia7...@apache.org>
>> wrote:
>> 
>>> According to the following code:
>>>  public ArrayBackedTag(byte tagType, byte[] tag) {
>>>int tagLength = tag.length + TYPE_LENGTH_SIZE;
>>>if (tagLength > MAX_TAG_LENGTH) {
>>>  throw new IllegalArgumentException(
>>>  "Invalid tag data being passed. Its length can not exceed " +
>>> MAX_TAG_LENGTH);
>>>}
>>>length = TAG_LENGTH_SIZE + tagLength;
>>>bytes = new byte[length];
>>>int pos = Bytes.putAsShort(bytes, 0, tagLength);
>>>pos = Bytes.putByte(bytes, pos, tagType);
>>>Bytes.putBytes(bytes, pos, tag, 0, tag.length);
>>>this.type = tagType;
>>>  }
>>> The layout of the byte array should be:
>>> |tag legnth (2 bytes)|tag type(1 byte)|tag|
>>> 
>>> It seems to me that the "bytes[offset + TYPE_LENGTH_SIZE]" is correct.
>>> 
>>>> On 2017-08-06 16:35, Lars George <lars.geo...@gmail.com> wrote:
>>>> Hi,
>>>> 
>>>> I found this reading through tags in 1.3, but checked in trunk as
>>>> well. There is this code:
>>>> 
>>>>  public ArrayBackedTag(byte[] bytes, int offset, int length) {
>>>>if (length > MAX_TAG_LENGTH) {
>>>>  throw new IllegalArgumentException(
>>>>  "Invalid tag data being passed. Its length can not exceed "
>>>> + MAX_TAG_LENGTH);
>>>>}
>>>>this.bytes = bytes;
>>>>this.offset = offset;
>>>>this.length = length;
>>>>this.type = bytes[offset + TAG_LENGTH_SIZE];
>>>>  }
>>>> 
>>>> I am concerned about the last line of the code, using the wrong
>> constant?
>>>> 
>>>>  public final static int TYPE_LENGTH_SIZE = Bytes.SIZEOF_BYTE;
>>>>  public final static int TAG_LENGTH_SIZE = Bytes.SIZEOF_SHORT;
>>>> 
>>>> Should this not read
>>>> 
>>>>this.type = bytes[offset + TYPE_LENGTH_SIZE];
>>>> 
>>>> Would this not read the type from the wrong place in the array?
>>>> 
>>>> Cheers,
>>>> Lars
>>>> 
>>> 
>> 

Re: Tags class using wrong length?

[Lars George]

You mean I am right and it should be the other constant being used?
Sorry, just wanting to confirm.


On Sun, Aug 6, 2017 at 2:40 PM, Chia-Ping Tsai <chia7...@apache.org> wrote:
> According to the following code:
>   public ArrayBackedTag(byte tagType, byte[] tag) {
> int tagLength = tag.length + TYPE_LENGTH_SIZE;
> if (tagLength > MAX_TAG_LENGTH) {
>   throw new IllegalArgumentException(
>   "Invalid tag data being passed. Its length can not exceed " + 
> MAX_TAG_LENGTH);
> }
> length = TAG_LENGTH_SIZE + tagLength;
> bytes = new byte[length];
> int pos = Bytes.putAsShort(bytes, 0, tagLength);
> pos = Bytes.putByte(bytes, pos, tagType);
> Bytes.putBytes(bytes, pos, tag, 0, tag.length);
> this.type = tagType;
>   }
> The layout of the byte array should be:
> |tag legnth (2 bytes)|tag type(1 byte)|tag|
>
> It seems to me that the "bytes[offset + TYPE_LENGTH_SIZE]" is correct.
>
> On 2017-08-06 16:35, Lars George <lars.geo...@gmail.com> wrote:
>> Hi,
>>
>> I found this reading through tags in 1.3, but checked in trunk as
>> well. There is this code:
>>
>>   public ArrayBackedTag(byte[] bytes, int offset, int length) {
>> if (length > MAX_TAG_LENGTH) {
>>   throw new IllegalArgumentException(
>>   "Invalid tag data being passed. Its length can not exceed "
>> + MAX_TAG_LENGTH);
>> }
>> this.bytes = bytes;
>> this.offset = offset;
>> this.length = length;
>> this.type = bytes[offset + TAG_LENGTH_SIZE];
>>   }
>>
>> I am concerned about the last line of the code, using the wrong constant?
>>
>>   public final static int TYPE_LENGTH_SIZE = Bytes.SIZEOF_BYTE;
>>   public final static int TAG_LENGTH_SIZE = Bytes.SIZEOF_SHORT;
>>
>> Should this not read
>>
>> this.type = bytes[offset + TYPE_LENGTH_SIZE];
>>
>> Would this not read the type from the wrong place in the array?
>>
>> Cheers,
>> Lars
>>

Re: Bug in FIFOCompactionPolicy pre-checks?

[Lars George]

Cool, thanks Vlad. Filed https://issues.apache.org/jira/browse/HBASE-18526

On Fri, Aug 4, 2017 at 7:53 PM, Vladimir Rodionov
<vladrodio...@gmail.com> wrote:
> Yes, file a JIRA, Lars
>
> I will take a look
>
> -Vlad
>
>
> On Thu, Aug 3, 2017 at 11:41 PM, Lars George <lars.geo...@gmail.com> wrote:
>
>> Hi,
>>
>> See https://issues.apache.org/jira/browse/HBASE-14468
>>
>> It adds this check to {{HMaster.checkCompactionPolicy()}}:
>>
>> {code}
>> // 1. Check TTL
>> if (hcd.getTimeToLive() == HColumnDescriptor.DEFAULT_TTL) {
>>   message = "Default TTL is not supported for FIFO compaction";
>>   throw new IOException(message);
>> }
>>
>> // 2. Check min versions
>> if (hcd.getMinVersions() > 0) {
>>   message = "MIN_VERSION > 0 is not supported for FIFO compaction";
>>   throw new IOException(message);
>> }
>>
>> // 3. blocking file count
>> String sbfc = htd.getConfigurationValue(HStore.BLOCKING_STOREFILES_KEY);
>> if (sbfc != null) {
>>   blockingFileCount = Integer.parseInt(sbfc);
>> }
>> if (blockingFileCount < 1000) {
>>   message =
>>   "blocking file count '" + HStore.BLOCKING_STOREFILES_KEY + "' "
>> + blockingFileCount
>>   + " is below recommended minimum of 1000";
>>   throw new IOException(message);
>> }
>> {code}
>>
>> Why does it only check the blocking file count on the HTD level, while
>> others are check on the HCD level? Doing this for example fails
>> because of it:
>>
>> {noformat}
>> hbase(main):008:0> create 'ttltable', { NAME => 'cf1', TTL => 300,
>> CONFIGURATION => { 'hbase.hstore.defaultengine.compactionpolicy.class'
>> => 'org.apache.hadoop.hbase.regionserver.compactions.
>> FIFOCompactionPolicy',
>> 'hbase.hstore.blockingStoreFiles' => 2000 } }
>>
>> ERROR: org.apache.hadoop.hbase.DoNotRetryIOException: blocking file
>> count 'hbase.hstore.blockingStoreFiles' 10 is below recommended
>> minimum of 1000 Set hbase.table.sanity.checks to false at conf or
>> table descriptor if you want to bypass sanity checks
>> at org.apache.hadoop.hbase.master.HMaster.warnOrThrowExceptionForFailure
>> (HMaster.java:1782)
>> at org.apache.hadoop.hbase.master.HMaster.sanityCheckTableDescriptor(
>> HMaster.java:1663)
>> at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1545)
>> at org.apache.hadoop.hbase.master.MasterRpcServices.
>> createTable(MasterRpcServices.java:469)
>> at org.apache.hadoop.hbase.protobuf.generated.
>> MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:58549)
>> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2339)
>> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123)
>> at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(
>> RpcExecutor.java:188)
>> at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(
>> RpcExecutor.java:168)
>> Caused by: java.io.IOException: blocking file count
>> 'hbase.hstore.blockingStoreFiles' 10 is below recommended minimum of
>> 1000
>> at org.apache.hadoop.hbase.master.HMaster.checkCompactionPolicy(HMaster.
>> java:1773)
>> at org.apache.hadoop.hbase.master.HMaster.sanityCheckTableDescriptor(
>> HMaster.java:1661)
>> ... 7 more
>> {noformat}
>>
>> That should work on the column family level, right? Shall I file a JIRA?
>>
>> Cheers,
>> Lars
>>

[jira] [Created] (HBASE-18526) FIFOCompactionPolicy pre-check uses wrong scope

[Lars George (JIRA)]

Lars George created HBASE-18526:
---

 Summary: FIFOCompactionPolicy pre-check uses wrong scope
 Key: HBASE-18526
 URL: https://issues.apache.org/jira/browse/HBASE-18526
 Project: HBase
  Issue Type: Bug
  Components: master
Affects Versions: 1.3.1
Reporter: Lars George


See https://issues.apache.org/jira/browse/HBASE-14468

It adds this check to {{HMaster.checkCompactionPolicy()}}:

{code}
// 1. Check TTL
if (hcd.getTimeToLive() == HColumnDescriptor.DEFAULT_TTL) {
  message = "Default TTL is not supported for FIFO compaction";
  throw new IOException(message);
}

// 2. Check min versions
if (hcd.getMinVersions() > 0) {
  message = "MIN_VERSION > 0 is not supported for FIFO compaction";
  throw new IOException(message);
}

// 3. blocking file count
String sbfc = htd.getConfigurationValue(HStore.BLOCKING_STOREFILES_KEY);
if (sbfc != null) {
  blockingFileCount = Integer.parseInt(sbfc);
}
if (blockingFileCount < 1000) {
  message =
  "blocking file count '" + HStore.BLOCKING_STOREFILES_KEY + "' "
+ blockingFileCount
  + " is below recommended minimum of 1000";
  throw new IOException(message);
}
{code}

Why does it only check the blocking file count on the HTD level, while
others are check on the HCD level? Doing this for example fails
because of it:

{noformat}
hbase(main):008:0> create 'ttltable', { NAME => 'cf1', TTL => 300,
CONFIGURATION => { 'hbase.hstore.defaultengine.compactionpolicy.class'
=> 'org.apache.hadoop.hbase.regionserver.compactions.FIFOCompactionPolicy',
'hbase.hstore.blockingStoreFiles' => 2000 } }

ERROR: org.apache.hadoop.hbase.DoNotRetryIOException: blocking file
count 'hbase.hstore.blockingStoreFiles' 10 is below recommended
minimum of 1000 Set hbase.table.sanity.checks to false at conf or
table descriptor if you want to bypass sanity checks
at 
org.apache.hadoop.hbase.master.HMaster.warnOrThrowExceptionForFailure(HMaster.java:1782)
at 
org.apache.hadoop.hbase.master.HMaster.sanityCheckTableDescriptor(HMaster.java:1663)
at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1545)
at 
org.apache.hadoop.hbase.master.MasterRpcServices.createTable(MasterRpcServices.java:469)
at 
org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:58549)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2339)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168)
Caused by: java.io.IOException: blocking file count
'hbase.hstore.blockingStoreFiles' 10 is below recommended minimum of
1000
at 
org.apache.hadoop.hbase.master.HMaster.checkCompactionPolicy(HMaster.java:1773)
at 
org.apache.hadoop.hbase.master.HMaster.sanityCheckTableDescriptor(HMaster.java:1661)
... 7 more
{noformat}

The check should be performed on the column family level instead.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Tags class using wrong length?

[Lars George]

Hi,

I found this reading through tags in 1.3, but checked in trunk as
well. There is this code:

  public ArrayBackedTag(byte[] bytes, int offset, int length) {
if (length > MAX_TAG_LENGTH) {
  throw new IllegalArgumentException(
  "Invalid tag data being passed. Its length can not exceed "
+ MAX_TAG_LENGTH);
}
this.bytes = bytes;
this.offset = offset;
this.length = length;
this.type = bytes[offset + TAG_LENGTH_SIZE];
  }

I am concerned about the last line of the code, using the wrong constant?

  public final static int TYPE_LENGTH_SIZE = Bytes.SIZEOF_BYTE;
  public final static int TAG_LENGTH_SIZE = Bytes.SIZEOF_SHORT;

Should this not read

this.type = bytes[offset + TYPE_LENGTH_SIZE];

Would this not read the type from the wrong place in the array?

Cheers,
Lars

Bug in FIFOCompactionPolicy pre-checks?

[Lars George]

Hi,

See https://issues.apache.org/jira/browse/HBASE-14468

It adds this check to {{HMaster.checkCompactionPolicy()}}:

{code}
// 1. Check TTL
if (hcd.getTimeToLive() == HColumnDescriptor.DEFAULT_TTL) {
  message = "Default TTL is not supported for FIFO compaction";
  throw new IOException(message);
}

// 2. Check min versions
if (hcd.getMinVersions() > 0) {
  message = "MIN_VERSION > 0 is not supported for FIFO compaction";
  throw new IOException(message);
}

// 3. blocking file count
String sbfc = htd.getConfigurationValue(HStore.BLOCKING_STOREFILES_KEY);
if (sbfc != null) {
  blockingFileCount = Integer.parseInt(sbfc);
}
if (blockingFileCount < 1000) {
  message =
  "blocking file count '" + HStore.BLOCKING_STOREFILES_KEY + "' "
+ blockingFileCount
  + " is below recommended minimum of 1000";
  throw new IOException(message);
}
{code}

Why does it only check the blocking file count on the HTD level, while
others are check on the HCD level? Doing this for example fails
because of it:

{noformat}
hbase(main):008:0> create 'ttltable', { NAME => 'cf1', TTL => 300,
CONFIGURATION => { 'hbase.hstore.defaultengine.compactionpolicy.class'
=> 'org.apache.hadoop.hbase.regionserver.compactions.FIFOCompactionPolicy',
'hbase.hstore.blockingStoreFiles' => 2000 } }

ERROR: org.apache.hadoop.hbase.DoNotRetryIOException: blocking file
count 'hbase.hstore.blockingStoreFiles' 10 is below recommended
minimum of 1000 Set hbase.table.sanity.checks to false at conf or
table descriptor if you want to bypass sanity checks
at 
org.apache.hadoop.hbase.master.HMaster.warnOrThrowExceptionForFailure(HMaster.java:1782)
at 
org.apache.hadoop.hbase.master.HMaster.sanityCheckTableDescriptor(HMaster.java:1663)
at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1545)
at 
org.apache.hadoop.hbase.master.MasterRpcServices.createTable(MasterRpcServices.java:469)
at 
org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:58549)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2339)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168)
Caused by: java.io.IOException: blocking file count
'hbase.hstore.blockingStoreFiles' 10 is below recommended minimum of
1000
at 
org.apache.hadoop.hbase.master.HMaster.checkCompactionPolicy(HMaster.java:1773)
at 
org.apache.hadoop.hbase.master.HMaster.sanityCheckTableDescriptor(HMaster.java:1661)
... 7 more
{noformat}

That should work on the column family level, right? Shall I file a JIRA?

Cheers,
Lars

[jira] [Created] (HBASE-18473) VC.listLabels() erroneously closes any connection

[Lars George (JIRA)]

Lars George created HBASE-18473:
---

 Summary: VC.listLabels() erroneously closes any connection
 Key: HBASE-18473
 URL: https://issues.apache.org/jira/browse/HBASE-18473
 Project: HBase
  Issue Type: Bug
  Components: Client
Affects Versions: 1.1.11, 1.2.6, 1.3.1
Reporter: Lars George


In HBASE-13358 the {{VisibilityClient.listLabels()}} was amended to take in a 
connection from the caller, which totally makes sense. But the patch forgot to 
remove the unconditional call to {{connection.close()}} in the {{finally}} 
block:

{code}
finally {
  if (table != null) {
table.close();
  }
  if (connection != null) {
connection.close();
  }
}
{code}

Remove the second {{if}} completely.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: user_permission output

[Lars George]

OK, found that no parameter is treated as asking for the ACLs table. Maybe the 
shell command should say so? Just a minor nit though, there are bigger tofu 
blocks to fry. 

Sent from my iPhone

> On 21. Jul 2017, at 13:33, Lars George <lars.geo...@gmail.com> wrote:
> 
> Hi,
> 
> I am running the shell's "user_permission" command without any
> parameter, and with a ".*" wildcard epxression and get two different
> results back:
> 
> hbase(main):003:0> user_permission
> User
> Namespace,Table,Family,Qualifier:Permission
> hbasebook  hbase,hbase:acl,,:
> [Permission: actions=READ,WRITE,EXEC,CREATE,ADMIN]
> hbase  hbase,hbase:acl,,:
> [Permission: actions=READ,WRITE,EXEC,CREATE,ADMIN]
> 2 row(s) in 0.5110 seconds
> 
> hbase(main):005:0> user_permission ".*"
> User
> Namespace,Table,Family,Qualifier:Permission
> hbasebook  hbase,hbase:acl,,:
> [Permission: actions=READ,WRITE,EXEC,CREATE,ADMIN]
> hbasebook  default,testtable,,:
> [Permission: actions=READ,WRITE,EXEC,CREATE,ADMIN]
> 2 row(s) in 0.4880 seconds
> 
> See how the second row differs? What is the proper behaviour of each
> command? Should it not give the same output?
> 
> Cheers,
> Lars

Re: Ident String

[Lars George]

Don't think it is necessary, does not to any harm. 

Sent from my iPhone

> On 21. Jul 2017, at 00:10, Stack <st...@duboce.net> wrote:
> 
>> On Thu, Jul 20, 2017 at 9:20 PM, Lars George <lars.geo...@gmail.com> wrote:
>> 
>> Ah ok, I missed that HBASE_IDENT_STR is used to personalize the log and
>> pid file names. Hadoop does the same, so that makes sense. But the
>> -Dhbase.id.str is not used, neither is it in Hadoop. No worries, just
>> wanted to see if anyone had an idea if that was ever used. Does not seem to
>> be the case.
>> 
>> 
> Should we remove it then boss?
> St.Ack
> 
> 
> 
>> Cheers,
>> Lars
>> 
>> Sent from my iPad
>> 
>>> On 19. Jul 2017, at 22:42, Lars George <lars.geo...@gmail.com> wrote:
>>> 
>>> It dates back to Hadoop:
>>> 
>>> https://github.com/apache/hbase/commit/24b065cc91f7bcdab25fc363469965
>> 7ac2f27104
>>> 
>>> See this https://github.com/apache/hadoop/blame/trunk/hadoop-
>> common-project/hadoop-common/src/main/conf/hadoop-env.sh#L202
>>> 
>>> It is used there for logs (according to the comment, haven't checked if
>> it really does). Are we doing that? Will check...
>>> 
>>> 
>>> Sent from my iPad
>>> 
>>>> On 19. Jul 2017, at 16:43, Stack <st...@duboce.net> wrote:
>>>> 
>>>> It shows when you do a long processing listing. It is a convention from
>>>> hadoop. It does similar. Here's a few example snippets Lars:
>>>> 
>>>> For hbase master process listing, you'll find this in the long line...:
>>>> -Dproc_master
>>>> 
>>>> Etc:
>>>> 
>>>> -Dproc_zookeeper
>>>> 
>>>> -Dproc_resourcemanager
>>>> 
>>>> St.Ack
>>>> 
>>>> 
>>>>> On Fri, Jul 14, 2017 at 11:55 AM, Lars George <lars.geo...@gmail.com>
>> wrote:
>>>>> 
>>>>> Hi,
>>>>> 
>>>>> Was coming across `HBASE_IDENT_STR` (in hbase-env.sh) which sets
>>>>> `hbase.id.str` as a command line parameter to the daemon. But I cannot
>>>>> find where that is used. Could someone point me in the right
>>>>> direction?
>>>>> 
>>>>> Cheers,
>>>>> Lars
>>>>> 
>> 

HBASE-11344

[Lars George]

Hi,

I was looking at https://issues.apache.org/jira/browse/HBASE-11344
which in JIRA is flagged as 0.99 and 2.0. The comments talk about 0.98
and also 1.x. But looking at the commit logs, I cannot see this apart
from 2.0. Has this never been committed anywhere else? Should the JIRA
be adjusted, or should this be committed to the other branches?

I think this is a useful feature even on 1.x as we have no ACLs for
the UI and being able to hide the sensitive keys is a good thing.

Thoughts?

Lars

user_permission output

[Lars George]

Hi,

I am running the shell's "user_permission" command without any
parameter, and with a ".*" wildcard epxression and get two different
results back:

hbase(main):003:0> user_permission
User
Namespace,Table,Family,Qualifier:Permission
 hbasebook  hbase,hbase:acl,,:
[Permission: actions=READ,WRITE,EXEC,CREATE,ADMIN]
 hbase  hbase,hbase:acl,,:
[Permission: actions=READ,WRITE,EXEC,CREATE,ADMIN]
2 row(s) in 0.5110 seconds

hbase(main):005:0> user_permission ".*"
User
Namespace,Table,Family,Qualifier:Permission
 hbasebook  hbase,hbase:acl,,:
[Permission: actions=READ,WRITE,EXEC,CREATE,ADMIN]
 hbasebook  default,testtable,,:
[Permission: actions=READ,WRITE,EXEC,CREATE,ADMIN]
2 row(s) in 0.4880 seconds

See how the second row differs? What is the proper behaviour of each
command? Should it not give the same output?

Cheers,
Lars

Re: Ident String

[Lars George]

Ah ok, I missed that HBASE_IDENT_STR is used to personalize the log and pid 
file names. Hadoop does the same, so that makes sense. But the -Dhbase.id.str 
is not used, neither is it in Hadoop. No worries, just wanted to see if anyone 
had an idea if that was ever used. Does not seem to be the case.

Cheers,
Lars

Sent from my iPad

> On 19. Jul 2017, at 22:42, Lars George <lars.geo...@gmail.com> wrote:
> 
> It dates back to Hadoop:
> 
> https://github.com/apache/hbase/commit/24b065cc91f7bcdab25fc3634699657ac2f27104
> 
> See this 
> https://github.com/apache/hadoop/blame/trunk/hadoop-common-project/hadoop-common/src/main/conf/hadoop-env.sh#L202
> 
> It is used there for logs (according to the comment, haven't checked if it 
> really does). Are we doing that? Will check...
> 
> 
> Sent from my iPad
> 
>> On 19. Jul 2017, at 16:43, Stack <st...@duboce.net> wrote:
>> 
>> It shows when you do a long processing listing. It is a convention from
>> hadoop. It does similar. Here's a few example snippets Lars:
>> 
>> For hbase master process listing, you'll find this in the long line...:
>> -Dproc_master
>> 
>> Etc:
>> 
>> -Dproc_zookeeper
>> 
>> -Dproc_resourcemanager
>> 
>> St.Ack
>> 
>> 
>>> On Fri, Jul 14, 2017 at 11:55 AM, Lars George <lars.geo...@gmail.com> wrote:
>>> 
>>> Hi,
>>> 
>>> Was coming across `HBASE_IDENT_STR` (in hbase-env.sh) which sets
>>> `hbase.id.str` as a command line parameter to the daemon. But I cannot
>>> find where that is used. Could someone point me in the right
>>> direction?
>>> 
>>> Cheers,
>>> Lars
>>> 

Re: Ident String

[Lars George]

It dates back to Hadoop:

https://github.com/apache/hbase/commit/24b065cc91f7bcdab25fc3634699657ac2f27104

See this 
https://github.com/apache/hadoop/blame/trunk/hadoop-common-project/hadoop-common/src/main/conf/hadoop-env.sh#L202

It is used there for logs (according to the comment, haven't checked if it 
really does). Are we doing that? Will check...


Sent from my iPad

> On 19. Jul 2017, at 16:43, Stack <st...@duboce.net> wrote:
> 
> It shows when you do a long processing listing. It is a convention from
> hadoop. It does similar. Here's a few example snippets Lars:
> 
> For hbase master process listing, you'll find this in the long line...:
> -Dproc_master
> 
> Etc:
> 
> -Dproc_zookeeper
> 
> -Dproc_resourcemanager
> 
> St.Ack
> 
> 
>> On Fri, Jul 14, 2017 at 11:55 AM, Lars George <lars.geo...@gmail.com> wrote:
>> 
>> Hi,
>> 
>> Was coming across `HBASE_IDENT_STR` (in hbase-env.sh) which sets
>> `hbase.id.str` as a command line parameter to the daemon. But I cannot
>> find where that is used. Could someone point me in the right
>> direction?
>> 
>> Cheers,
>> Lars
>> 

Re: Ident String

[Lars George]

Thanks Stack, but you are talking about something else methinks. I know of the 
"-Dproc_" convention. What I am asking about is different, it is 
settable by the user in the hbase-env.sh and is added to the command line as 
"-Dhbase.id.str=...". I have no idea who is using it and why. Makes sense?

Sent from my iPad

> On 19. Jul 2017, at 16:43, Stack <st...@duboce.net> wrote:
> 
> It shows when you do a long processing listing. It is a convention from
> hadoop. It does similar. Here's a few example snippets Lars:
> 
> For hbase master process listing, you'll find this in the long line...:
> -Dproc_master
> 
> Etc:
> 
> -Dproc_zookeeper
> 
> -Dproc_resourcemanager
> 
> St.Ack
> 
> 
>> On Fri, Jul 14, 2017 at 11:55 AM, Lars George <lars.geo...@gmail.com> wrote:
>> 
>> Hi,
>> 
>> Was coming across `HBASE_IDENT_STR` (in hbase-env.sh) which sets
>> `hbase.id.str` as a command line parameter to the daemon. But I cannot
>> find where that is used. Could someone point me in the right
>> direction?
>> 
>> Cheers,
>> Lars
>> 

Re: Ident String

[Lars George]

Bueller? Anyone?

:)

On Fri, Jul 14, 2017 at 12:55 PM, Lars George <lars.geo...@gmail.com> wrote:
> Hi,
>
> Was coming across `HBASE_IDENT_STR` (in hbase-env.sh) which sets
> `hbase.id.str` as a command line parameter to the daemon. But I cannot
> find where that is used. Could someone point me in the right
> direction?
>
> Cheers,
> Lars

Re: RegionServer info page content

[Lars George]

I created https://issues.apache.org/jira/browse/HBASE-18388


On Fri, Jul 14, 2017 at 11:12 PM, Jean-Marc Spaggiari
<jean-m...@spaggiari.org> wrote:
> Good to know, thanks for the details ;)
>
> JM
>
> 2017-07-14 16:51 GMT-04:00 Lars George <lars.geo...@gmail.com>:
>
>> Hey JM,
>>
>> It is not a "random ID" that a region has, but an MD5 of the prefix,
>> including the table name, the startkey, and the time the region was created
>> (that part is also missing from the current description. I am not sure when
>> that description ever was correct. Also, the hbase:meta uses a different
>> hash, Jenkins, to encode the prefix and the missing trailing dot character
>> is indicating that.
>>
>> I will create a JIRA.
>>
>> Lars
>>
>> Sent from my iPad
>>
>> > On 14. Jul 2017, at 17:43, Jean-Marc Spaggiari <jean-m...@spaggiari.org>
>> wrote:
>> >
>> > Hi Lars,
>> >
>> > I'm not sure to get that's wrong :( Can you provide more dtails?
>> >
>> > JMS
>> >
>> > 2017-07-14 11:33 GMT-04:00 Lars George <lars.geo...@gmail.com>:
>> >
>> >> Hi,
>> >>
>> >> On the RS status page it has this:
>> >>
>> >> "Region names are made of the containing table's name, a comma, the
>> >> start key, a comma, and a randomly generated region id. To illustrate,
>> >> the region named domains,apache.org,5464829424211263407 is party to
>> >> the table domains, has an id of 5464829424211263407 and the first key
>> >> in the region is apache.org."
>> >>
>> >> This makes no sense whatsoever today. Did we all stop reading the info
>> >> sections on pages?
>> >>
>> >> Any objections me creating a JIRA?
>> >>
>> >> Cheers,
>> >> Lars
>> >>
>>

[jira] [Created] (HBASE-18388) Fix description on region page, explaining what a region name is made of

[Lars George (JIRA)]

Lars George created HBASE-18388:
---

 Summary: Fix description on region page, explaining what a region 
name is made of
 Key: HBASE-18388
 URL: https://issues.apache.org/jira/browse/HBASE-18388
 Project: HBase
  Issue Type: Improvement
  Components: master, regionserver, UI
Affects Versions: 2.0.0-alpha-1, 1.3.1
Reporter: Lars George
Priority: Minor


In the {{RegionListTmpl.jamon}} we have this:

{code}
Region names are made of the containing table's name, a comma,
the start key, a comma, and a randomly generated region id.  To illustrate,
the region named
domains,apache.org,5464829424211263407 is party to the table
domains, has an id of 5464829424211263407 and the first 
key
in the region is apache.org.The hbase:meta 'table' is 
an internal
system table (or a 'catalog' table in db-speak).
The hbase:meta table keeps a list of all regions in the system. The empty 
key is used to denote
table start and table end.  A region with an empty start key is the first 
region in a table.
If a region has both an empty start key and an empty end key, it's the only 
region in the
table. See http://hbase.org;>HBase Home for further 
explication.
{code}

This is wrong and worded oddly. What needs to be fixed facts wise is:

- Region names contain (separated by commas) the full table name (including the 
namespace), the start key, the time the region was created, and finally a dot 
with an MD5 hash of everything before the dot. For example: 
{{test,,1499410125885.1544f69aeaf787755caa11d3567a9621.}}
- The trailing dot is to distinguish legacy region names (like those used by 
the {{hbase:meta}} table)
- The MD5 hash is used as the directory name within the HBase storage 
directories
- The names for the meta table use a Jenkins hash instead, also leaving out the 
trailing dot, for example {{hbase:meta,,1.1588230740}}. The time is always set 
to {{1}}.
- The start key is printed in safe characters, escaping unprintable characters
- The link to the HBase home page to explain more is useless and should be 
removed.
- Also, for region replicas, the replica ID is inserted into the name, like so 
{{replicatable,,1486289678486_0001.3e8b7655299b21b3038ff8d39062467f.}}, see the 
{{_0001}} part.

As for the wording, I would just make this all flow a little better, that "is 
party of" sounds weird to me (IMHO).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HBASE-18387) [Thrift] Make principal configurable in DemoClient.java

[Lars George (JIRA)]

Lars George created HBASE-18387:
---

 Summary: [Thrift] Make principal configurable in DemoClient.java
 Key: HBASE-18387
 URL: https://issues.apache.org/jira/browse/HBASE-18387
 Project: HBase
  Issue Type: Improvement
Reporter: Lars George
Priority: Minor


In the Thrift1 demo client we have this code:

{code}
transport = new TSaslClientTransport("GSSAPI", null,
  "hbase", // Thrift server user name, should be an authorized proxy user.
  host, // Thrift server domain
  saslProperties, null, transport);
{code}

This will only work when the Thrift server is started with the {{hbase}} 
principal. Often this may deviate, for example I am using {{hbase-thrift}} to 
separate the names from those of backend servers. 

What we need is either an additional command line option to specify the name, 
or a property that can be set with -D and can be passed at runtime. I prefer 
the former, as the latter is making this a little convoluted.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: RegionServer info page content

[Lars George]

Hey JM,

It is not a "random ID" that a region has, but an MD5 of the prefix, including 
the table name, the startkey, and the time the region was created (that part is 
also missing from the current description. I am not sure when that description 
ever was correct. Also, the hbase:meta uses a different hash, Jenkins, to 
encode the prefix and the missing trailing dot character is indicating that.

I will create a JIRA.

Lars 

Sent from my iPad

> On 14. Jul 2017, at 17:43, Jean-Marc Spaggiari <jean-m...@spaggiari.org> 
> wrote:
> 
> Hi Lars,
> 
> I'm not sure to get that's wrong :( Can you provide more dtails?
> 
> JMS
> 
> 2017-07-14 11:33 GMT-04:00 Lars George <lars.geo...@gmail.com>:
> 
>> Hi,
>> 
>> On the RS status page it has this:
>> 
>> "Region names are made of the containing table's name, a comma, the
>> start key, a comma, and a randomly generated region id. To illustrate,
>> the region named domains,apache.org,5464829424211263407 is party to
>> the table domains, has an id of 5464829424211263407 and the first key
>> in the region is apache.org."
>> 
>> This makes no sense whatsoever today. Did we all stop reading the info
>> sections on pages?
>> 
>> Any objections me creating a JIRA?
>> 
>> Cheers,
>> Lars
>> 

RegionServer info page content

[Lars George]

Hi,

On the RS status page it has this:

"Region names are made of the containing table's name, a comma, the
start key, a comma, and a randomly generated region id. To illustrate,
the region named domains,apache.org,5464829424211263407 is party to
the table domains, has an id of 5464829424211263407 and the first key
in the region is apache.org."

This makes no sense whatsoever today. Did we all stop reading the info
sections on pages?

Any objections me creating a JIRA?

Cheers,
Lars

[jira] [Created] (HBASE-18382) [Thrift] Add transport type info to info server

[Lars George (JIRA)]

Lars George created HBASE-18382:
---

 Summary: [Thrift] Add transport type info to info server 
 Key: HBASE-18382
 URL: https://issues.apache.org/jira/browse/HBASE-18382
 Project: HBase
  Issue Type: Improvement
  Components: Thrift
Reporter: Lars George
Priority: Minor


It would be really helpful to know if the Thrift server was started using the 
HTTP or binary transport. Any additional info, like QOP settings for SASL etc. 
would be great too. Right now the UI is very limited and shows {{true/false}} 
for, for example, {{Compact Transport}}. It'd suggest to change this to show 
something more useful like this:

{noformat}
Thrift Impl Type: non-blocking
Protocol: Binary
Transport: Framed
QOP: Authentication & Confidential
{noformat}}

or

{noformat}
Protocol: Binary + HTTP
Transport: Standard
QOP: none
{noformat}}




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Ident String

[Lars George]

Hi,

Was coming across `HBASE_IDENT_STR` (in hbase-env.sh) which sets
`hbase.id.str` as a command line parameter to the daemon. But I cannot
find where that is used. Could someone point me in the right
direction?

Cheers,
Lars

Service-level AuthZ

[Lars George]

Hi,

We have SLAs, implementing the Hadoop classes, in HBase that allow to
grant or revoke global access to RPC services. What we do _not_ have
is the matching `refreshServiceAcl` method, including the wiring into
the shell script to trigger it. Is there a reason why? Should we not
have the same to be able to refresh at runtime?

We could also use our existing `updateConfiguration()` for that,
assuming there is no specific reason to have it separate. If we use
our existing call, it should be simple to add.

Thoughts?

Cheers,
Lars

Re: Assigning regions after restart

[Lars George]

JIRA creation is done: https://issues.apache.org/jira/browse/HBASE-17791

I will have a look into it over the next few days, maybe I can come up
with a patch.

On Wed, Mar 15, 2017 at 6:14 AM, Stack <st...@duboce.net> wrote:
> File a blocker please Lars. I'm pretty sure the boolean on whether we are
> doing a recovery or not has been there a long time so yeah, a single server
> recovery could throw us off, but you make a practical point, that one
> server should not destroy locality over the cluster.
>
> St.Ack
>
> On Tue, Mar 14, 2017 at 7:09 AM, Lars George <lars.geo...@gmail.com> wrote:
>
>> Wait, HBASE-15251 is not enough methinks. The checks added help, but
>> are not covering all the possible edge cases. In particular, say a
>> node really fails, why not just reassign the few regions it did hold
>> and leave all the others where they are? Seems insane as it is.
>>
>> On Tue, Mar 14, 2017 at 2:24 PM, Lars George <lars.geo...@gmail.com>
>> wrote:
>> > Doh, https://issues.apache.org/jira/browse/HBASE-15251 addresses this
>> > (though I am not sure exactly how, see below). This should be
>> > backported to all 1.x branches!
>> >
>> > As for the patch, I see this
>> >
>> >  if (!failover) {
>> >// Fresh cluster startup.
>> > -  LOG.info("Clean cluster startup. Assigning user regions");
>> > +  LOG.info("Clean cluster startup. Don't reassign user regions");
>> >assignAllUserRegions(allRegions);
>> > +} else {
>> > +  LOG.info("Failover! Reassign user regions");
>> >      }
>> >
>> > Would be interesting to see how that log message is actually
>> > reassigning the user regions. I would have assumed the
>> > "assignAllUserRegions()" would have to be called.
>> >
>> >
>> > On Tue, Mar 14, 2017 at 2:21 PM, Lars George <lars.geo...@gmail.com>
>> wrote:
>> >> Looking at the code more... it seems the issue is here
>> >>
>> >> In AssignmentManager.processDeadServersAndRegionsInTransition():
>> >>
>> >> ...
>> >> failoverCleanupDone();
>> >> if (!failover) {
>> >>   // Fresh cluster startup.
>> >>   LOG.info("Clean cluster startup. Assigning user regions");
>> >>   assignAllUserRegions(allRegions);
>> >> }
>> >> ...
>> >>
>> >> As soon as a single node failed, failover is set to true. So no
>> >> assignment is done. Later on the servers are recovered from the
>> >> "crash" (which a clean restart is as well), and then all unassigned
>> >> regions (all of them now, since they are unassigned earlier) are
>> >> assigned round-robin.
>> >>
>> >>
>> >>
>> >> On Tue, Mar 14, 2017 at 1:54 PM, Lars George <lars.geo...@gmail.com>
>> wrote:
>> >>> Hi,
>> >>>
>> >>> I had this happened at multiple clusters recently where after the
>> >>> restart the locality dropped from close to or exactly 100% down to
>> >>> single digits. The reason is that all regions were completely shuffled
>> >>> and reassigned to random servers. Upon reading the (yet again
>> >>> non-trivial) assignment code, I found that a single server missing
>> >>> will trigger a full "recovery" of all servers, which includes a drop
>> >>> of all previous assignments and random new assignment.
>> >>>
>> >>> This is just terrible! In addition, I also assumed that - at least the
>> >>> StochasticLoadBalancer - is checking which node holds most of the data
>> >>> of a region locality wise and picks that server. But that is _not_ the
>> >>> case! It just spreads everything seemingly randomly across the
>> >>> servers.
>> >>>
>> >>> To me this is a big regression (or straight bug) given that a single
>> >>> server out of, for example, hundreds could trigger that and destroy
>> >>> the locality completely. Running a major compaction is not an approach
>> >>> for many reasons.
>> >>>
>> >>> This used to work better, why that regression?
>> >>>
>> >>> Lars
>>

[jira] [Created] (HBASE-17791) Locality should not be affected for non-faulty region servers at startup

[Lars George (JIRA)]

Lars George created HBASE-17791:
---

 Summary: Locality should not be affected for non-faulty region 
servers at startup
 Key: HBASE-17791
 URL: https://issues.apache.org/jira/browse/HBASE-17791
 Project: HBase
  Issue Type: Improvement
  Components: Balancer, Region Assignment
Affects Versions: 1.1.8, 1.2.4, 1.0.3, 1.3.0
Reporter: Lars George
Priority: Blocker


We seem to have an issue with store file locality as soon as a single server is 
missing or faulty upon restart. HBASE-15251 is addressing a subset of the 
problem, ensuring that some remaining files or an empty server do not trigger a 
full recovery, but is missing further, similar cases. Especially, the case 
where a server fails to report in before the active master is finished waiting 
for them to do so, or where servers have been decomissioned (but not removed 
from the {{regionservers}} file), and finally the true case of a dead node.

In case a single node is faulty, the user regions are _not_ assigned as saved 
in the {{hbase:meta}} table, but completely randomized in a round-robin 
fashion. An additional factor is that in this case the regions are _not_ 
assigned to the best matching node (the one with a copy of the data locally), 
but to any node, leaving the locality in shambles.

What is also bad, if the {{hbase.hstore.min.locality.to.skip.major.compact}} 
property is left at the default {{0.0f}}, then an older region that had no 
writes since the last major compaction happened is just skipped (as expected, 
usually) and locality stays bad as-is. All reads for those aged-out regions 
will be network reads. But in any event, having to run a major compaction after 
a restart is not good anyways.

The issue is the code in 
{{AssignmentManager.processDeadServersAndRegionsInTransition()}}, which is 
handed a list of dead servers. But it immediately sets the {{failover}} flag 
and the code

{code}
failoverCleanupDone();
if (!failover) {
  // Fresh cluster startup.
  LOG.info("Clean cluster startup. Don't reassign user regions");
  assignAllUserRegions(allRegions);
} else {
  LOG.info("Failover! Reassign user regions");
}
{code}

is not triggering the assignment of the regions to those servers that are still 
present and have all their region data local. What should happen is that only 
the missing regions are reassigned, just like in the case of a server failing 
while the cluster is running.




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: Assigning regions after restart

[Lars George]

Wait, HBASE-15251 is not enough methinks. The checks added help, but
are not covering all the possible edge cases. In particular, say a
node really fails, why not just reassign the few regions it did hold
and leave all the others where they are? Seems insane as it is.

On Tue, Mar 14, 2017 at 2:24 PM, Lars George <lars.geo...@gmail.com> wrote:
> Doh, https://issues.apache.org/jira/browse/HBASE-15251 addresses this
> (though I am not sure exactly how, see below). This should be
> backported to all 1.x branches!
>
> As for the patch, I see this
>
>  if (!failover) {
>// Fresh cluster startup.
> -  LOG.info("Clean cluster startup. Assigning user regions");
> +  LOG.info("Clean cluster startup. Don't reassign user regions");
>assignAllUserRegions(allRegions);
> +} else {
> +  LOG.info("Failover! Reassign user regions");
>  }
>
> Would be interesting to see how that log message is actually
> reassigning the user regions. I would have assumed the
> "assignAllUserRegions()" would have to be called.
>
>
> On Tue, Mar 14, 2017 at 2:21 PM, Lars George <lars.geo...@gmail.com> wrote:
>> Looking at the code more... it seems the issue is here
>>
>> In AssignmentManager.processDeadServersAndRegionsInTransition():
>>
>> ...
>> failoverCleanupDone();
>> if (!failover) {
>>   // Fresh cluster startup.
>>   LOG.info("Clean cluster startup. Assigning user regions");
>>   assignAllUserRegions(allRegions);
>> }
>> ...
>>
>> As soon as a single node failed, failover is set to true. So no
>> assignment is done. Later on the servers are recovered from the
>> "crash" (which a clean restart is as well), and then all unassigned
>> regions (all of them now, since they are unassigned earlier) are
>> assigned round-robin.
>>
>>
>>
>> On Tue, Mar 14, 2017 at 1:54 PM, Lars George <lars.geo...@gmail.com> wrote:
>>> Hi,
>>>
>>> I had this happened at multiple clusters recently where after the
>>> restart the locality dropped from close to or exactly 100% down to
>>> single digits. The reason is that all regions were completely shuffled
>>> and reassigned to random servers. Upon reading the (yet again
>>> non-trivial) assignment code, I found that a single server missing
>>> will trigger a full "recovery" of all servers, which includes a drop
>>> of all previous assignments and random new assignment.
>>>
>>> This is just terrible! In addition, I also assumed that - at least the
>>> StochasticLoadBalancer - is checking which node holds most of the data
>>> of a region locality wise and picks that server. But that is _not_ the
>>> case! It just spreads everything seemingly randomly across the
>>> servers.
>>>
>>> To me this is a big regression (or straight bug) given that a single
>>> server out of, for example, hundreds could trigger that and destroy
>>> the locality completely. Running a major compaction is not an approach
>>> for many reasons.
>>>
>>> This used to work better, why that regression?
>>>
>>> Lars

[jira] [Resolved] (HBASE-14129) If any regionserver gets shutdown uncleanly during full cluster restart, locality looks to be lost

[Lars George (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lars George resolved HBASE-14129.
-
Resolution: Won't Fix

Closing as "won't fix" as the hardcoded flag is too intrusive. The cluster 
should be able to handle this by fixing the logic in the {{AssignmentManager}}.

> If any regionserver gets shutdown uncleanly during full cluster restart, 
> locality looks to be lost
> --
>
> Key: HBASE-14129
> URL: https://issues.apache.org/jira/browse/HBASE-14129
> Project: HBase
>  Issue Type: Bug
>Reporter: churro morales
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-14129.patch
>
>
> We were doing a cluster restart the other day.  Some regionservers did not 
> shut down cleanly.  Upon restart our locality went from 99% to 5%.  Upon 
> looking at the AssignmentManager.joinCluster() code it calls 
> AssignmentManager.processDeadServersAndRegionsInTransition().
> If the failover flag gets set for any reason it seems we don't call 
> assignAllUserRegions().  Then it looks like the balancer does the work in 
> assigning those regions, we don't use a locality aware balancer and we lost 
> our region locality.
> I don't have a solid grasp on the reasoning for these checks but there could 
> be some potential workarounds here.
> 1. After shutting down your cluster, move your WALs aside (replay later).  
> 2. Clean up your zNodes 
> That seems to work, but requires a lot of manual labor.  Another solution 
> which I prefer would be to have a flag for ./start-hbase.sh --clean 
> If we start master with that flag then we do a check in 
> AssignmentManager.processDeadServersAndRegionsInTransition()  thus if this 
> flag is set we call: assignAllUserRegions() regardless of the failover state.
> I have a patch for the later solution, that is if I am understanding the 
> logic correctly.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: Assigning regions after restart

[Lars George]

Looking at the code more... it seems the issue is here

In AssignmentManager.processDeadServersAndRegionsInTransition():

...
failoverCleanupDone();
if (!failover) {
  // Fresh cluster startup.
  LOG.info("Clean cluster startup. Assigning user regions");
  assignAllUserRegions(allRegions);
}
...

As soon as a single node failed, failover is set to true. So no
assignment is done. Later on the servers are recovered from the
"crash" (which a clean restart is as well), and then all unassigned
regions (all of them now, since they are unassigned earlier) are
assigned round-robin.



On Tue, Mar 14, 2017 at 1:54 PM, Lars George <lars.geo...@gmail.com> wrote:
> Hi,
>
> I had this happened at multiple clusters recently where after the
> restart the locality dropped from close to or exactly 100% down to
> single digits. The reason is that all regions were completely shuffled
> and reassigned to random servers. Upon reading the (yet again
> non-trivial) assignment code, I found that a single server missing
> will trigger a full "recovery" of all servers, which includes a drop
> of all previous assignments and random new assignment.
>
> This is just terrible! In addition, I also assumed that - at least the
> StochasticLoadBalancer - is checking which node holds most of the data
> of a region locality wise and picks that server. But that is _not_ the
> case! It just spreads everything seemingly randomly across the
> servers.
>
> To me this is a big regression (or straight bug) given that a single
> server out of, for example, hundreds could trigger that and destroy
> the locality completely. Running a major compaction is not an approach
> for many reasons.
>
> This used to work better, why that regression?
>
> Lars

Assigning regions after restart

[Lars George]

Hi,

I had this happened at multiple clusters recently where after the
restart the locality dropped from close to or exactly 100% down to
single digits. The reason is that all regions were completely shuffled
and reassigned to random servers. Upon reading the (yet again
non-trivial) assignment code, I found that a single server missing
will trigger a full "recovery" of all servers, which includes a drop
of all previous assignments and random new assignment.

This is just terrible! In addition, I also assumed that - at least the
StochasticLoadBalancer - is checking which node holds most of the data
of a region locality wise and picks that server. But that is _not_ the
case! It just spreads everything seemingly randomly across the
servers.

To me this is a big regression (or straight bug) given that a single
server out of, for example, hundreds could trigger that and destroy
the locality completely. Running a major compaction is not an approach
for many reasons.

This used to work better, why that regression?

Lars

Re: Assigning regions after restart

[Lars George]

Doh, https://issues.apache.org/jira/browse/HBASE-15251 addresses this
(though I am not sure exactly how, see below). This should be
backported to all 1.x branches!

As for the patch, I see this

 if (!failover) {
   // Fresh cluster startup.
-  LOG.info("Clean cluster startup. Assigning user regions");
+  LOG.info("Clean cluster startup. Don't reassign user regions");
   assignAllUserRegions(allRegions);
+} else {
+  LOG.info("Failover! Reassign user regions");
 }

Would be interesting to see how that log message is actually
reassigning the user regions. I would have assumed the
"assignAllUserRegions()" would have to be called.


On Tue, Mar 14, 2017 at 2:21 PM, Lars George <lars.geo...@gmail.com> wrote:
> Looking at the code more... it seems the issue is here
>
> In AssignmentManager.processDeadServersAndRegionsInTransition():
>
> ...
> failoverCleanupDone();
> if (!failover) {
>   // Fresh cluster startup.
>   LOG.info("Clean cluster startup. Assigning user regions");
>   assignAllUserRegions(allRegions);
> }
> ...
>
> As soon as a single node failed, failover is set to true. So no
> assignment is done. Later on the servers are recovered from the
> "crash" (which a clean restart is as well), and then all unassigned
> regions (all of them now, since they are unassigned earlier) are
> assigned round-robin.
>
>
>
> On Tue, Mar 14, 2017 at 1:54 PM, Lars George <lars.geo...@gmail.com> wrote:
>> Hi,
>>
>> I had this happened at multiple clusters recently where after the
>> restart the locality dropped from close to or exactly 100% down to
>> single digits. The reason is that all regions were completely shuffled
>> and reassigned to random servers. Upon reading the (yet again
>> non-trivial) assignment code, I found that a single server missing
>> will trigger a full "recovery" of all servers, which includes a drop
>> of all previous assignments and random new assignment.
>>
>> This is just terrible! In addition, I also assumed that - at least the
>> StochasticLoadBalancer - is checking which node holds most of the data
>> of a region locality wise and picks that server. But that is _not_ the
>> case! It just spreads everything seemingly randomly across the
>> servers.
>>
>> To me this is a big regression (or straight bug) given that a single
>> server out of, for example, hundreds could trigger that and destroy
>> the locality completely. Running a major compaction is not an approach
>> for many reasons.
>>
>> This used to work better, why that regression?
>>
>> Lars

[jira] [Created] (HBASE-17774) Improve locality info in table details page

[Lars George (JIRA)]

Lars George created HBASE-17774:
---

 Summary: Improve locality info in table details page
 Key: HBASE-17774
 URL: https://issues.apache.org/jira/browse/HBASE-17774
 Project: HBase
  Issue Type: Improvement
  Components: UI
Affects Versions: 1.3.0
Reporter: Lars George


The {{table.jsp}} page does list the locality info of each region, but is 
missing some vital other details, which are:

- An extra column listing store and store file counts (could be two separate 
columns too like in Master and RS UI, but a single column saves space, for 
example "3/6", meaning three stores and six store files in total
- Hide locality "0.0" if the stores are all empty, i.e. no store file is 
present. It makes little sense penalizing an empty store when it comes to the 
total locality of the table.
- Make region name clickable like on RS status page, linking to {{region.jsp}} 
page showing store details.
- Summary row at the end, showing the total locality (see note above), number 
stores, number of store files. That is, compute the _real_ locality, which is 
considering only the actual store files.

I also wish we had some simple but effective charts/diagrams here, like a 
heatmap for data distribution within the table (since we have all of this 
info), or another heatmap for current request distribution. Strong +1 to add 
that here too.

I have reasoned about this at customers way too often to not have this improved 
somehow. For the enterprise level admin, a good UI goes a long way!



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: HDFS Balancer

[Lars George]

Yeah, my thoughts exactly... though thanks Harsh for taking action to
clean up the documentation! Good on you.

On Thu, Mar 9, 2017 at 11:01 AM, Jean-Marc Spaggiari
<jean-m...@spaggiari.org> wrote:
> So there is no way to use the pinning feature without having to use the
> favored nodes option? :(
>
> Le 2017-03-08 6:13 AM, "Harsh J" <ha...@cloudera.com> a écrit :
>
>> Hey Lars!,
>>
>> I was on a similar line of investigation today, and I've filed
>> https://issues.apache.org/jira/browse/HBASE-17760 to change the text. The
>> pinning part of the text is relevant, but the command part isn't. In
>> addition, you'd need to manually use the FavoredNodeLoadBalancer work to
>> actually get HBase to apply pinning to its writes by passing around proper
>> favored-node hint hostnames. I've also linked past and future relevant work
>> JIRAs to that one.
>>
>> Stumbled on this email when searching some follow-throughs, thought I'd
>> drop a note.
>>
>> On Tue, 7 Mar 2017 at 20:18 Ted Yu <yuzhih...@gmail.com> wrote:
>>
>> > bq. how that - apparently wrong - information came about
>> >
>> > Maybe Sean / Misty can give some context.
>> >
>> > Cheers
>> >
>> > On Tue, Mar 7, 2017 at 6:37 AM, Lars George <lars.geo...@gmail.com>
>> wrote:
>> >
>> > > Hey Ted,
>> > >
>> > > Thanks Cpt. Obvious :)
>> > >
>> > > I know how to use "blame" or git log how to find the JIRA, but what I
>> was
>> > > after is how that - apparently wrong - information came about. And if
>> it
>> > is
>> > > wrong, what _is_ the current status of this feature.
>> > >
>> > > I do believe this is an important operational piece as it helps with
>> > > rearranging clusters. Since it seems to still be missing, I am
>> wondering
>> > > what needs to be done here.
>> > >
>> > > Makes sense?
>> > >
>> > > Lars
>> > >
>> > > Sent from my iPhone
>> > >
>> > > > On 6 Mar 2017, at 19:50, Ted Yu <yuzhih...@gmail.com> wrote:
>> > > >
>> > > > w.r.t. the first question, the quoted paragraph came from:
>> > > >
>> > > > HBASE-15332 Document how to take advantage of HDFS-6133 in HBase
>> > > >
>> > > >> On Mon, Mar 6, 2017 at 6:38 PM, Lars George <lars.geo...@gmail.com>
>> > > wrote:
>> > > >>
>> > > >> Hi,
>> > > >>
>> > > >> I am trying to grok what came out of all these issues about the HDFS
>> > > >> balancer and being able to avoid it destroying HBase locality. There
>> > > >> is this https://issues.apache.org/jira/browse/HBASE-13021 from JM,
>> > and
>> > > >> the book http://hbase.apache.org/book.html#_hbase_and_hdfs refers
>> to
>> > > >> https://issues.apache.org/jira/browse/HDFS-6133, stating:
>> > > >>
>> > > >> "HDFS-6133 provides the ability to exclude a given directory from
>> the
>> > > >> HDFS load balancer, by setting the dfs.datanode.block-pinning.
>> enabled
>> > > >> property to true in your HDFS configuration and running the
>> following
>> > > >> hdfs command:
>> > > >>
>> > > >> $ sudo -u hdfs hdfs balancer -exclude /hbase"
>> > > >>
>> > > >> I checked the Balancer class in 2.7.2 and it does not have that
>> > > >> support, i.e. being able to exclude a path, it can only exclude
>> hosts.
>> > > >> That is also clear from HDFS-6133, which adds favoured nodes, but
>> not
>> > > >> being able to exclude paths (which would be nice).
>> > > >>
>> > > >> HBASE-13021 mentions that this works in tandem with the HBase
>> favored
>> > > >> node feature, but that makes it much more complicated since you have
>> > > >> to pin individual regions to nodes, instead of doing that wholesale.
>> > > >>
>> > > >> Where does the above in the HBase book come from, and what is the
>> > > >> current state as far as you know?
>> > > >>
>> > > >> Cheers,
>> > > >> Lars
>> > > >>
>> > >
>> >
>>

[jira] [Resolved] (HBASE-17635) enable_table_replication script cannot handle replication scope

[Lars George (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lars George resolved HBASE-17635.
-
Resolution: Duplicate

Sorry, this was opened just a few weeks earlier in HBASE-17460, closing this 
JIRA as a dupe.

> enable_table_replication script cannot handle replication scope
> ---
>
> Key: HBASE-17635
> URL: https://issues.apache.org/jira/browse/HBASE-17635
> Project: HBase
>  Issue Type: Bug
>  Components: Replication
>Affects Versions: 1.3.1
>    Reporter: Lars George
>
> When you add a peer, then enable a table for replication using 
> {{enable_table_replication}}, the script will create the table on the peer 
> cluster, but with one difference:
> _Master Cluster_:
> {noformat}
> hbase(main):027:0> describe 'testtable'
> Table testtable is ENABLED
>   
> 
> testtable 
>   
> 
> COLUMN FAMILIES DESCRIPTION   
>   
> 
> {NAME => 'cf1', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'ROW', 
> REPLICATION_SCOPE => '1', VERSIONS => '1', COMPRESSION => 'NONE', 
> MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE',
>  BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}
>   
> 
> 1 row(s) in 0.0700 seconds
> {noformat}
> _Peer Cluster_:
> {noformat}
> hbase(main):003:0> describe 'testtable'
> Table testtable is ENABLED
>   
> 
> testtable 
>   
> 
> COLUMN FAMILIES DESCRIPTION   
>   
> 
> {NAME => 'cf1', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'ROW', 
> REPLICATION_SCOPE => '0', COMPRESSION => 'NONE', VERSIONS => '1', TTL => 
> 'FOREVER', MIN_VERSIONS => '0', KEEP_DELETED_CELLS => 'FALSE',
>  BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}
>   
> 
> 1 row(s) in 0.1260 seconds
> {noformat}
> Note that the replication scope is different. Removing the peer, adding it 
> again and enabling the table gives this now:
> {noformat}
> hbase(main):026:0> enable_table_replication 'testtable'
> ERROR: Table testtable exists in peer cluster 1, but the table descriptors 
> are not same when compared with source cluster. Thus can not enable the 
> table's replication switch.
> {noformat}
> That is dumb, as it was the same script that enabled the replication scope in 
> the first place. It should skip that particular attribute when comparing the 
> cluster schemas.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: HDFS Balancer

[Lars George]

Hey Ted,

Thanks Cpt. Obvious :)

I know how to use "blame" or git log how to find the JIRA, but what I was after 
is how that - apparently wrong - information came about. And if it is wrong, 
what _is_ the current status of this feature. 

I do believe this is an important operational piece as it helps with 
rearranging clusters. Since it seems to still be missing, I am wondering what 
needs to be done here. 

Makes sense?

Lars

Sent from my iPhone

> On 6 Mar 2017, at 19:50, Ted Yu <yuzhih...@gmail.com> wrote:
> 
> w.r.t. the first question, the quoted paragraph came from:
> 
> HBASE-15332 Document how to take advantage of HDFS-6133 in HBase
> 
>> On Mon, Mar 6, 2017 at 6:38 PM, Lars George <lars.geo...@gmail.com> wrote:
>> 
>> Hi,
>> 
>> I am trying to grok what came out of all these issues about the HDFS
>> balancer and being able to avoid it destroying HBase locality. There
>> is this https://issues.apache.org/jira/browse/HBASE-13021 from JM, and
>> the book http://hbase.apache.org/book.html#_hbase_and_hdfs refers to
>> https://issues.apache.org/jira/browse/HDFS-6133, stating:
>> 
>> "HDFS-6133 provides the ability to exclude a given directory from the
>> HDFS load balancer, by setting the dfs.datanode.block-pinning.enabled
>> property to true in your HDFS configuration and running the following
>> hdfs command:
>> 
>> $ sudo -u hdfs hdfs balancer -exclude /hbase"
>> 
>> I checked the Balancer class in 2.7.2 and it does not have that
>> support, i.e. being able to exclude a path, it can only exclude hosts.
>> That is also clear from HDFS-6133, which adds favoured nodes, but not
>> being able to exclude paths (which would be nice).
>> 
>> HBASE-13021 mentions that this works in tandem with the HBase favored
>> node feature, but that makes it much more complicated since you have
>> to pin individual regions to nodes, instead of doing that wholesale.
>> 
>> Where does the above in the HBase book come from, and what is the
>> current state as far as you know?
>> 
>> Cheers,
>> Lars
>> 

HDFS Balancer

[Lars George]

Hi,

I am trying to grok what came out of all these issues about the HDFS
balancer and being able to avoid it destroying HBase locality. There
is this https://issues.apache.org/jira/browse/HBASE-13021 from JM, and
the book http://hbase.apache.org/book.html#_hbase_and_hdfs refers to
https://issues.apache.org/jira/browse/HDFS-6133, stating:

"HDFS-6133 provides the ability to exclude a given directory from the
HDFS load balancer, by setting the dfs.datanode.block-pinning.enabled
property to true in your HDFS configuration and running the following
hdfs command:

$ sudo -u hdfs hdfs balancer -exclude /hbase"

I checked the Balancer class in 2.7.2 and it does not have that
support, i.e. being able to exclude a path, it can only exclude hosts.
That is also clear from HDFS-6133, which adds favoured nodes, but not
being able to exclude paths (which would be nice).

HBASE-13021 mentions that this works in tandem with the HBase favored
node feature, but that makes it much more complicated since you have
to pin individual regions to nodes, instead of doing that wholesale.

Where does the above in the HBase book come from, and what is the
current state as far as you know?

Cheers,
Lars

Client SSL Authentication

[Lars George]

Hi,

We have support for two-way authentication over TLS in the HttpServer
class, but never use it, nor have a config property that could set it.
Hadoop has the same option but exposes it via config. Should we not do
the same?

Lars

[jira] [Created] (HBASE-17636) Fix speling [sic] error in enable replication script output

[Lars George (JIRA)]

Lars George created HBASE-17636:
---

 Summary: Fix speling [sic] error in enable replication script 
output
 Key: HBASE-17636
 URL: https://issues.apache.org/jira/browse/HBASE-17636
 Project: HBase
  Issue Type: Bug
  Components: Replication
Affects Versions: 1.3.1
Reporter: Lars George


When enabling the replication for a table:

{noformat}
hbase(main):012:0> enable_table_replication 'repltest'
0 row(s) in 7.6080 seconds
The replication swith of table 'repltest' successfully enabled
{noformat}

See {{swith}} as opposed to {{switch}}. Also, that sentence is somewhat too 
complicated. Better is maybe {{Replication for table  successfully 
enabled.}}?




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (HBASE-17635) enable_table_replication script cannot handle replication scope

[Lars George (JIRA)]

Lars George created HBASE-17635:
---

 Summary: enable_table_replication script cannot handle replication 
scope
 Key: HBASE-17635
 URL: https://issues.apache.org/jira/browse/HBASE-17635
 Project: HBase
  Issue Type: Bug
  Components: Replication
Affects Versions: 1.3.1
Reporter: Lars George


When you add a peer, then enable a table for replication using 
{{enable_table_replication}}, the script will create the table on the peer 
cluster, but with one difference:

_Master Cluster_:

{noformat}
hbase(main):027:0> describe 'testtable'
Table testtable is ENABLED  


testtable   


COLUMN FAMILIES DESCRIPTION 


{NAME => 'cf1', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'ROW', 
REPLICATION_SCOPE => '1', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS 
=> '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE',
 BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}  


1 row(s) in 0.0700 seconds
{noformat}

_Peer Cluster_:
{noformat}
hbase(main):003:0> describe 'testtable'
Table testtable is ENABLED  


testtable   


COLUMN FAMILIES DESCRIPTION 


{NAME => 'cf1', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'ROW', 
REPLICATION_SCOPE => '0', COMPRESSION => 'NONE', VERSIONS => '1', TTL => 
'FOREVER', MIN_VERSIONS => '0', KEEP_DELETED_CELLS => 'FALSE',
 BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}  


1 row(s) in 0.1260 seconds
{noformat}

Note that the replication scope is different. Removing the peer, adding it 
again and enabling the table gives this now:

{noformat}
hbase(main):026:0> enable_table_replication 'testtable'

ERROR: Table testtable exists in peer cluster 1, but the table descriptors are 
not same when compared with source cluster. Thus can not enable the table's 
replication switch.
{noformat}

That is dumb, as it was the same script that enabled the replication scope in 
the first place. It should skip that particular attribute when comparing the 
cluster schemas.




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (HBASE-17632) Modify example health script to work on CentOS 6 etc.

[Lars George (JIRA)]

Lars George created HBASE-17632:
---

 Summary: Modify example health script to work on CentOS 6 etc.
 Key: HBASE-17632
 URL: https://issues.apache.org/jira/browse/HBASE-17632
 Project: HBase
  Issue Type: Bug
  Components: master, regionserver
Reporter: Lars George






--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (HBASE-17631) Canary interval too low

[Lars George (JIRA)]

Lars George created HBASE-17631:
---

 Summary: Canary interval too low
 Key: HBASE-17631
 URL: https://issues.apache.org/jira/browse/HBASE-17631
 Project: HBase
  Issue Type: Bug
  Components: canary
Affects Versions: 1.3.1
Reporter: Lars George


The interval currently is {{6000}} milliseconds, or six seconds, which makes 
little sense to test that often in succession. We should set the default to at 
least 60 seconds, or even every 5 minutes?



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (HBASE-17630) Health Script not shutting down server process with certain script behavior

[Lars George (JIRA)]

Lars George created HBASE-17630:
---

 Summary: Health Script not shutting down server process with 
certain script behavior
 Key: HBASE-17630
 URL: https://issues.apache.org/jira/browse/HBASE-17630
 Project: HBase
  Issue Type: Bug
  Components: master, regionserver
Affects Versions: 1.3.1
Reporter: Lars George


As discussed on dev@...

I tried the supplied {{healthcheck.sh}}, but did not have {{snmpd}} running.
That caused the script to take a long time to error out, which exceed
the 10 seconds the check was meant to run. That resets the check and
it keeps reporting the error, but never stops the servers:

{noformat}
2017-02-04 05:55:08,962 INFO
[regionserver/slave-1.internal.larsgeorge.com/10.0.10.10:16020]
hbase.HealthCheckChore: Health Check Chore runs every 10sec
2017-02-04 05:55:08,975 INFO
[regionserver/slave-1.internal.larsgeorge.com/10.0.10.10:16020]
hbase.HealthChecker: HealthChecker initialized with script at
/opt/hbase/bin/healthcheck.sh, timeout=6

...

2017-02-04 05:55:50,435 INFO
[slave-1.internal.larsgeorge.com,16020,1486216506007_ChoreService_1]
hbase.HealthCheckChore: Health status at 412837hrs, 55mins, 50sec :
ERROR check link, OK: disks ok,

2017-02-04 05:55:50,436 INFO
[slave-1.internal.larsgeorge.com,16020,1486216506007_ChoreService_1]
hbase.ScheduledChore: Chore: CompactionChecker missed its start time
2017-02-04 05:55:50,437 INFO
[slave-1.internal.larsgeorge.com,16020,1486216506007_ChoreService_1]
hbase.ScheduledChore: Chore:
slave-1.internal.larsgeorge.com,16020,1486216506007-MemstoreFlusherChore
missed its start time
2017-02-04 05:55:50,438 INFO
[slave-1.internal.larsgeorge.com,16020,1486216506007_ChoreService_2]
hbase.ScheduledChore: Chore: HealthChecker missed its start time
2017-02-04 05:56:20,522 INFO
[slave-1.internal.larsgeorge.com,16020,1486216506007_ChoreService_2]
hbase.HealthCheckChore: Health status at 412837hrs, 56mins, 20sec :
ERROR check link, OK: disks ok,

2017-02-04 05:56:20,523 INFO
[slave-1.internal.larsgeorge.com,16020,1486216506007_ChoreService_2]
hbase.ScheduledChore: Chore: HealthChecker missed its start time
2017-02-04 05:56:50,600 INFO
[slave-1.internal.larsgeorge.com,16020,1486216506007_ChoreService_2]
hbase.HealthCheckChore: Health status at 412837hrs, 56mins, 50sec :
ERROR check link, OK: disks ok,

2017-02-04 05:56:50,600 INFO
[slave-1.internal.larsgeorge.com,16020,1486216506007_ChoreService_2]
hbase.ScheduledChore: Chore: HealthChecker missed its start time
2017-02-04 05:57:20,681 INFO
[slave-1.internal.larsgeorge.com,16020,1486216506007_ChoreService_1]
hbase.HealthCheckChore: Health status at 412837hrs, 57mins, 20sec :
ERROR check link, OK: disks ok,

2017-02-04 05:57:20,681 INFO
[slave-1.internal.larsgeorge.com,16020,1486216506007_ChoreService_1]
hbase.ScheduledChore: Chore: HealthChecker missed its start time
2017-02-04 05:57:50,763 INFO
[slave-1.internal.larsgeorge.com,16020,1486216506007_ChoreService_1]
hbase.HealthCheckChore: Health status at 412837hrs, 57mins, 50sec :
ERROR check link, OK: disks ok,

2017-02-04 05:57:50,764 INFO
[slave-1.internal.larsgeorge.com,16020,1486216506007_ChoreService_1]
hbase.ScheduledChore: Chore: HealthChecker missed its start time
2017-02-04 05:58:20,844 INFO
[slave-1.internal.larsgeorge.com,16020,1486216506007_ChoreService_1]
hbase.HealthCheckChore: Health status at 412837hrs, 58mins, 20sec :
ERROR check link, OK: disks ok,

2017-02-04 05:58:20,844 INFO
[slave-1.internal.larsgeorge.com,16020,1486216506007_ChoreService_1]
hbase.ScheduledChore: Chore: HealthChecker missed its start time
2017-02-04 05:58:50,923 INFO
[slave-1.internal.larsgeorge.com,16020,1486216506007_ChoreService_1]
hbase.HealthCheckChore: Health status at 412837hrs, 58mins, 50sec :
ERROR check link, OK: disks ok,

2017-02-04 05:58:50,923 INFO
[slave-1.internal.larsgeorge.com,16020,1486216506007_ChoreService_1]
hbase.ScheduledChore: Chore: HealthChecker missed its start time
2017-02-04 05:59:21,017 INFO
[slave-1.internal.larsgeorge.com,16020,1486216506007_ChoreService_2]
hbase.HealthCheckChore: Health status at 412837hrs, 59mins, 21sec :
ERROR check link, OK: disks ok,

2017-02-04 05:59:21,018 INFO
[slave-1.internal.larsgeorge.com,16020,1486216506007_ChoreService_2]
hbase.ScheduledChore: Chore: HealthChecker missed its start time
{noformat}

We need to fix the handling of the timeout of the health check script and ho 
the chore is treating that to shut down the server process. The current 
settings of check frequency and timeout overlap and cause the above.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: Health Script does not stop region server

[Lars George]

Will do, thanks


On Sun, Feb 5, 2017 at 3:57 PM, Ted Yu <yuzhih...@gmail.com> wrote:
> Yesterday I tried snmpwalk on CentOS as well - same behavior.
>
> Lars:
> Can you file a JIRA to fix the bug ?
>
> Thanks
>
> On Sun, Feb 5, 2017 at 2:22 AM, Lars George <lars.geo...@gmail.com> wrote:
>
>> Hi Ted,
>>
>> This does not work on Mac as provided. I tried on a CentOS 6 machine,
>> and had to install net-snmp and net-snmp-utils, plus start the snmpd
>> to make it time out quicker. But even even there the snmpwalk return
>> nothing, making the script fail.
>>
>> Anyhow, the snmpwalk failing after the retries is just an example of
>> what can happen if the health check script takes too long to fail. The
>> bottom line is that it does _not_ stop the server as expected as our
>> check in the code is reset because of the chore's delay. That is a bug
>> methinks.
>>
>> Or, in other words, when I fixed the snmpwalk to come back quickly as
>> explained above, the error was caught in time and the server stopped
>> as expected.
>>
>> Makes sense?
>>
>> Lars
>>
>> On Sat, Feb 4, 2017 at 4:30 PM, Ted Yu <yuzhih...@gmail.com> wrote:
>> > Running the command from the script locally (on Mac):
>> >
>> > $ /usr/bin/snmpwalk -t 5 -Oe  -Oq  -Os -v 1 -c public localhost if
>> > Timeout: No Response from localhost
>> > $ echo $?
>> > 1
>> >
>> > Looks like the script should parse the output from snmpwalk and provide
>> > some hint if unexpected result is reported.
>> >
>> > Cheers
>> >
>> > On Sat, Feb 4, 2017 at 6:40 AM, Lars George <lars.geo...@gmail.com>
>> wrote:
>> >
>> >> Hi,
>> >>
>> >> I tried the supplied `healthcheck.sh`, but did not have snmpd running.
>> >> That caused the script to take a long time to error out, which exceed
>> >> the 10 seconds the check was meant to run. That resets the check and
>> >> it keeps reporting the error, but never stops the servers:
>> >>
>> >> 2017-02-04 05:55:08,962 INFO
>> >> [regionserver/slave-1.internal.larsgeorge.com/10.0.10.10:16020]
>> >> hbase.HealthCheckChore: Health Check Chore runs every 10sec
>> >> 2017-02-04 05:55:08,975 INFO
>> >> [regionserver/slave-1.internal.larsgeorge.com/10.0.10.10:16020]
>> >> hbase.HealthChecker: HealthChecker initialized with script at
>> >> /opt/hbase/bin/healthcheck.sh, timeout=6
>> >>
>> >> ...
>> >>
>> >> 2017-02-04 05:55:50,435 INFO
>> >> [slave-1.internal.larsgeorge.com,16020,1486216506007_ChoreService_1]
>> >> hbase.HealthCheckChore: Health status at 412837hrs, 55mins, 50sec :
>> >> ERROR check link, OK: disks ok,
>> >>
>> >> 2017-02-04 05:55:50,436 INFO
>> >> [slave-1.internal.larsgeorge.com,16020,1486216506007_ChoreService_1]
>> >> hbase.ScheduledChore: Chore: CompactionChecker missed its start time
>> >> 2017-02-04 05:55:50,437 INFO
>> >> [slave-1.internal.larsgeorge.com,16020,1486216506007_ChoreService_1]
>> >> hbase.ScheduledChore: Chore:
>> >> slave-1.internal.larsgeorge.com,16020,1486216506007-
>> MemstoreFlusherChore
>> >> missed its start time
>> >> 2017-02-04 05:55:50,438 INFO
>> >> [slave-1.internal.larsgeorge.com,16020,1486216506007_ChoreService_2]
>> >> hbase.ScheduledChore: Chore: HealthChecker missed its start time
>> >> 2017-02-04 05:56:20,522 INFO
>> >> [slave-1.internal.larsgeorge.com,16020,1486216506007_ChoreService_2]
>> >> hbase.HealthCheckChore: Health status at 412837hrs, 56mins, 20sec :
>> >> ERROR check link, OK: disks ok,
>> >>
>> >> 2017-02-04 05:56:20,523 INFO
>> >> [slave-1.internal.larsgeorge.com,16020,1486216506007_ChoreService_2]
>> >> hbase.ScheduledChore: Chore: HealthChecker missed its start time
>> >> 2017-02-04 05:56:50,600 INFO
>> >> [slave-1.internal.larsgeorge.com,16020,1486216506007_ChoreService_2]
>> >> hbase.HealthCheckChore: Health status at 412837hrs, 56mins, 50sec :
>> >> ERROR check link, OK: disks ok,
>> >>
>> >> 2017-02-04 05:56:50,600 INFO
>> >> [slave-1.internal.larsgeorge.com,16020,1486216506007_ChoreService_2]
>> >> hbase.ScheduledChore: Chore: HealthChecker missed its start time
>> >> 2017-02-04 05:57:20,681 INFO
>> >> [slave-1.internal.larsgeorge.com,16020,14862165060

Re: Canary Test Tool and write sniffing

[Lars George]

Please keep in mind we are talking about two issues here:

1) The short default interval time, and
2) the issue that the canary table regions might not be on all servers.

Anyone here that tried write sniffing on a current cluster with the
SLB and saw it work?

Best,
Lars


On Mon, Feb 6, 2017 at 10:38 PM, Enis Söztutar <e...@apache.org> wrote:
> Open an issue?
> Enis
>
> On Mon, Feb 6, 2017 at 9:39 AM, Stack <st...@duboce.net> wrote:
>
>> On Sun, Feb 5, 2017 at 2:25 AM, Lars George <lars.geo...@gmail.com> wrote:
>>
>> > The next example is wrong too, claiming to show 60 secs, while it
>> > shows 600 secs (the default value as well).
>> >
>> > The question is still, what is a good value for intervals? Anyone here
>> > that uses the Canary that would like to chime in?
>> >
>> >
>> I was hanging out with a user where on a mid-sized cluster with Canary
>> running with defaults, the regionserver carrying meta was 100% CPU because
>> of all the requests from Canary doing repeated full-table Scans.
>>
>> 6 seconds is too short. Seems like a typo that should be 60seconds. It is
>> not as though the Canary is going to do anything about it if it finds
>> something wrong.
>>
>> S
>>
>>
>>
>>
>> > On Sat, Feb 4, 2017 at 5:40 PM, Ted Yu <yuzhih...@gmail.com> wrote:
>> > > Brief search on HBASE-4393 didn't reveal why the interval was
>> shortened.
>> > >
>> > > If you read the first paragraph of:
>> > > http://hbase.apache.org/book.html#_run_canary_test_as_daemon_mode
>> > >
>> > > possibly the reasoning was that canary would exit upon seeing some
>> error
>> > > (the first time).
>> > >
>> > > BTW There was a mismatch in the description for this command: (5
>> seconds
>> > > vs. 5 milliseconds)
>> > >
>> > > ${HBASE_HOME}/bin/hbase canary -daemon -interval 5 -f false
>> > >
>> > >
>> > > On Sat, Feb 4, 2017 at 8:21 AM, Lars George <lars.geo...@gmail.com>
>> > wrote:
>> > >
>> > >> Oh right, Ted. An earlier patch attached to the JIRA had 60 secs, the
>> > >> last one has 6 secs. Am I reading this right? It hands 6000 into the
>> > >> Thread.sleep() call, which takes millisecs. So that makes 6 secs
>> > >> between checks, which seems super short, no? I might just dull here.
>> > >>
>> > >> On Sat, Feb 4, 2017 at 5:00 PM, Ted Yu <yuzhih...@gmail.com> wrote:
>> > >> > For the default interval , if you were looking at:
>> > >> >
>> > >> >   private static final long DEFAULT_INTERVAL = 6000;
>> > >> >
>> > >> > The above was from:
>> > >> >
>> > >> > HBASE-4393 Implement a canary monitoring program
>> > >> >
>> > >> > which was integrated on Tue Apr 24 07:20:16 2012
>> > >> >
>> > >> > FYI
>> > >> >
>> > >> > On Sat, Feb 4, 2017 at 4:06 AM, Lars George <lars.geo...@gmail.com>
>> > >> wrote:
>> > >> >
>> > >> >> Also, the default interval used to be 60 secs, but is now 6 secs.
>> > Does
>> > >> >> that make sense? Seems awfully short for a default, assuming you
>> have
>> > >> >> many regions or servers.
>> > >> >>
>> > >> >> On Sat, Feb 4, 2017 at 11:54 AM, Lars George <
>> lars.geo...@gmail.com>
>> > >> >> wrote:
>> > >> >> > Hi,
>> > >> >> >
>> > >> >> > Looking at the Canary tool, it tries to ensure that all canary
>> test
>> > >> >> > table regions are spread across all region servers. If that is
>> not
>> > the
>> > >> >> > case, it calls:
>> > >> >> >
>> > >> >> > if (numberOfCoveredServers < numberOfServers) {
>> > >> >> >   admin.balancer();
>> > >> >> > }
>> > >> >> >
>> > >> >> > I doubt this will help with the StochasticLoadBalancer, which is
>> > known
>> > >> >> > to consider per-table balancing as one of many factors. In
>> > practice,
>> > >> >> > the SLB will most likely _not_ distribute the canary regions
>> > >> >> > sufficiently, leaving gap in the check. Switching on the
>> per-table
>> > >> >> > option is discouraged against to let it do its thing.
>> > >> >> >
>> > >> >> > Just pointing it out for vetting.
>> > >> >> >
>> > >> >> > Lars
>> > >> >>
>> > >>
>> >
>>

Re: Canary Test Tool and write sniffing

[Lars George]

The next example is wrong too, claiming to show 60 secs, while it
shows 600 secs (the default value as well).

The question is still, what is a good value for intervals? Anyone here
that uses the Canary that would like to chime in?

On Sat, Feb 4, 2017 at 5:40 PM, Ted Yu <yuzhih...@gmail.com> wrote:
> Brief search on HBASE-4393 didn't reveal why the interval was shortened.
>
> If you read the first paragraph of:
> http://hbase.apache.org/book.html#_run_canary_test_as_daemon_mode
>
> possibly the reasoning was that canary would exit upon seeing some error
> (the first time).
>
> BTW There was a mismatch in the description for this command: (5 seconds
> vs. 5 milliseconds)
>
> ${HBASE_HOME}/bin/hbase canary -daemon -interval 5 -f false
>
>
> On Sat, Feb 4, 2017 at 8:21 AM, Lars George <lars.geo...@gmail.com> wrote:
>
>> Oh right, Ted. An earlier patch attached to the JIRA had 60 secs, the
>> last one has 6 secs. Am I reading this right? It hands 6000 into the
>> Thread.sleep() call, which takes millisecs. So that makes 6 secs
>> between checks, which seems super short, no? I might just dull here.
>>
>> On Sat, Feb 4, 2017 at 5:00 PM, Ted Yu <yuzhih...@gmail.com> wrote:
>> > For the default interval , if you were looking at:
>> >
>> >   private static final long DEFAULT_INTERVAL = 6000;
>> >
>> > The above was from:
>> >
>> > HBASE-4393 Implement a canary monitoring program
>> >
>> > which was integrated on Tue Apr 24 07:20:16 2012
>> >
>> > FYI
>> >
>> > On Sat, Feb 4, 2017 at 4:06 AM, Lars George <lars.geo...@gmail.com>
>> wrote:
>> >
>> >> Also, the default interval used to be 60 secs, but is now 6 secs. Does
>> >> that make sense? Seems awfully short for a default, assuming you have
>> >> many regions or servers.
>> >>
>> >> On Sat, Feb 4, 2017 at 11:54 AM, Lars George <lars.geo...@gmail.com>
>> >> wrote:
>> >> > Hi,
>> >> >
>> >> > Looking at the Canary tool, it tries to ensure that all canary test
>> >> > table regions are spread across all region servers. If that is not the
>> >> > case, it calls:
>> >> >
>> >> > if (numberOfCoveredServers < numberOfServers) {
>> >> >   admin.balancer();
>> >> > }
>> >> >
>> >> > I doubt this will help with the StochasticLoadBalancer, which is known
>> >> > to consider per-table balancing as one of many factors. In practice,
>> >> > the SLB will most likely _not_ distribute the canary regions
>> >> > sufficiently, leaving gap in the check. Switching on the per-table
>> >> > option is discouraged against to let it do its thing.
>> >> >
>> >> > Just pointing it out for vetting.
>> >> >
>> >> > Lars
>> >>
>>

Re: Health Script does not stop region server

[Lars George]

Hi Ted,

This does not work on Mac as provided. I tried on a CentOS 6 machine,
and had to install net-snmp and net-snmp-utils, plus start the snmpd
to make it time out quicker. But even even there the snmpwalk return
nothing, making the script fail.

Anyhow, the snmpwalk failing after the retries is just an example of
what can happen if the health check script takes too long to fail. The
bottom line is that it does _not_ stop the server as expected as our
check in the code is reset because of the chore's delay. That is a bug
methinks.

Or, in other words, when I fixed the snmpwalk to come back quickly as
explained above, the error was caught in time and the server stopped
as expected.

Makes sense?

Lars

On Sat, Feb 4, 2017 at 4:30 PM, Ted Yu <yuzhih...@gmail.com> wrote:
> Running the command from the script locally (on Mac):
>
> $ /usr/bin/snmpwalk -t 5 -Oe  -Oq  -Os -v 1 -c public localhost if
> Timeout: No Response from localhost
> $ echo $?
> 1
>
> Looks like the script should parse the output from snmpwalk and provide
> some hint if unexpected result is reported.
>
> Cheers
>
> On Sat, Feb 4, 2017 at 6:40 AM, Lars George <lars.geo...@gmail.com> wrote:
>
>> Hi,
>>
>> I tried the supplied `healthcheck.sh`, but did not have snmpd running.
>> That caused the script to take a long time to error out, which exceed
>> the 10 seconds the check was meant to run. That resets the check and
>> it keeps reporting the error, but never stops the servers:
>>
>> 2017-02-04 05:55:08,962 INFO
>> [regionserver/slave-1.internal.larsgeorge.com/10.0.10.10:16020]
>> hbase.HealthCheckChore: Health Check Chore runs every 10sec
>> 2017-02-04 05:55:08,975 INFO
>> [regionserver/slave-1.internal.larsgeorge.com/10.0.10.10:16020]
>> hbase.HealthChecker: HealthChecker initialized with script at
>> /opt/hbase/bin/healthcheck.sh, timeout=6
>>
>> ...
>>
>> 2017-02-04 05:55:50,435 INFO
>> [slave-1.internal.larsgeorge.com,16020,1486216506007_ChoreService_1]
>> hbase.HealthCheckChore: Health status at 412837hrs, 55mins, 50sec :
>> ERROR check link, OK: disks ok,
>>
>> 2017-02-04 05:55:50,436 INFO
>> [slave-1.internal.larsgeorge.com,16020,1486216506007_ChoreService_1]
>> hbase.ScheduledChore: Chore: CompactionChecker missed its start time
>> 2017-02-04 05:55:50,437 INFO
>> [slave-1.internal.larsgeorge.com,16020,1486216506007_ChoreService_1]
>> hbase.ScheduledChore: Chore:
>> slave-1.internal.larsgeorge.com,16020,1486216506007-MemstoreFlusherChore
>> missed its start time
>> 2017-02-04 05:55:50,438 INFO
>> [slave-1.internal.larsgeorge.com,16020,1486216506007_ChoreService_2]
>> hbase.ScheduledChore: Chore: HealthChecker missed its start time
>> 2017-02-04 05:56:20,522 INFO
>> [slave-1.internal.larsgeorge.com,16020,1486216506007_ChoreService_2]
>> hbase.HealthCheckChore: Health status at 412837hrs, 56mins, 20sec :
>> ERROR check link, OK: disks ok,
>>
>> 2017-02-04 05:56:20,523 INFO
>> [slave-1.internal.larsgeorge.com,16020,1486216506007_ChoreService_2]
>> hbase.ScheduledChore: Chore: HealthChecker missed its start time
>> 2017-02-04 05:56:50,600 INFO
>> [slave-1.internal.larsgeorge.com,16020,1486216506007_ChoreService_2]
>> hbase.HealthCheckChore: Health status at 412837hrs, 56mins, 50sec :
>> ERROR check link, OK: disks ok,
>>
>> 2017-02-04 05:56:50,600 INFO
>> [slave-1.internal.larsgeorge.com,16020,1486216506007_ChoreService_2]
>> hbase.ScheduledChore: Chore: HealthChecker missed its start time
>> 2017-02-04 05:57:20,681 INFO
>> [slave-1.internal.larsgeorge.com,16020,1486216506007_ChoreService_1]
>> hbase.HealthCheckChore: Health status at 412837hrs, 57mins, 20sec :
>> ERROR check link, OK: disks ok,
>>
>> 2017-02-04 05:57:20,681 INFO
>> [slave-1.internal.larsgeorge.com,16020,1486216506007_ChoreService_1]
>> hbase.ScheduledChore: Chore: HealthChecker missed its start time
>> 2017-02-04 05:57:50,763 INFO
>> [slave-1.internal.larsgeorge.com,16020,1486216506007_ChoreService_1]
>> hbase.HealthCheckChore: Health status at 412837hrs, 57mins, 50sec :
>> ERROR check link, OK: disks ok,
>>
>> 2017-02-04 05:57:50,764 INFO
>> [slave-1.internal.larsgeorge.com,16020,1486216506007_ChoreService_1]
>> hbase.ScheduledChore: Chore: HealthChecker missed its start time
>> 2017-02-04 05:58:20,844 INFO
>> [slave-1.internal.larsgeorge.com,16020,1486216506007_ChoreService_1]
>> hbase.HealthCheckChore: Health status at 412837hrs, 58mins, 20sec :
>> ERROR check link, OK: disks ok,
>>
>> 2017-02-04 05:58:20,844 INFO
>> [slave-1.internal.larsgeorge.com,16020,1486216506007_ChoreService_1

Re: Canary Test Tool and write sniffing

[Lars George]

Oh right, Ted. An earlier patch attached to the JIRA had 60 secs, the
last one has 6 secs. Am I reading this right? It hands 6000 into the
Thread.sleep() call, which takes millisecs. So that makes 6 secs
between checks, which seems super short, no? I might just dull here.

On Sat, Feb 4, 2017 at 5:00 PM, Ted Yu <yuzhih...@gmail.com> wrote:
> For the default interval , if you were looking at:
>
>   private static final long DEFAULT_INTERVAL = 6000;
>
> The above was from:
>
> HBASE-4393 Implement a canary monitoring program
>
> which was integrated on Tue Apr 24 07:20:16 2012
>
> FYI
>
> On Sat, Feb 4, 2017 at 4:06 AM, Lars George <lars.geo...@gmail.com> wrote:
>
>> Also, the default interval used to be 60 secs, but is now 6 secs. Does
>> that make sense? Seems awfully short for a default, assuming you have
>> many regions or servers.
>>
>> On Sat, Feb 4, 2017 at 11:54 AM, Lars George <lars.geo...@gmail.com>
>> wrote:
>> > Hi,
>> >
>> > Looking at the Canary tool, it tries to ensure that all canary test
>> > table regions are spread across all region servers. If that is not the
>> > case, it calls:
>> >
>> > if (numberOfCoveredServers < numberOfServers) {
>> >   admin.balancer();
>> > }
>> >
>> > I doubt this will help with the StochasticLoadBalancer, which is known
>> > to consider per-table balancing as one of many factors. In practice,
>> > the SLB will most likely _not_ distribute the canary regions
>> > sufficiently, leaving gap in the check. Switching on the per-table
>> > option is discouraged against to let it do its thing.
>> >
>> > Just pointing it out for vetting.
>> >
>> > Lars
>>

Health Script does not stop region server

[Lars George]

Hi,

I tried the supplied `healthcheck.sh`, but did not have snmpd running.
That caused the script to take a long time to error out, which exceed
the 10 seconds the check was meant to run. That resets the check and
it keeps reporting the error, but never stops the servers:

2017-02-04 05:55:08,962 INFO
[regionserver/slave-1.internal.larsgeorge.com/10.0.10.10:16020]
hbase.HealthCheckChore: Health Check Chore runs every 10sec
2017-02-04 05:55:08,975 INFO
[regionserver/slave-1.internal.larsgeorge.com/10.0.10.10:16020]
hbase.HealthChecker: HealthChecker initialized with script at
/opt/hbase/bin/healthcheck.sh, timeout=6

...

2017-02-04 05:55:50,435 INFO
[slave-1.internal.larsgeorge.com,16020,1486216506007_ChoreService_1]
hbase.HealthCheckChore: Health status at 412837hrs, 55mins, 50sec :
ERROR check link, OK: disks ok,

2017-02-04 05:55:50,436 INFO
[slave-1.internal.larsgeorge.com,16020,1486216506007_ChoreService_1]
hbase.ScheduledChore: Chore: CompactionChecker missed its start time
2017-02-04 05:55:50,437 INFO
[slave-1.internal.larsgeorge.com,16020,1486216506007_ChoreService_1]
hbase.ScheduledChore: Chore:
slave-1.internal.larsgeorge.com,16020,1486216506007-MemstoreFlusherChore
missed its start time
2017-02-04 05:55:50,438 INFO
[slave-1.internal.larsgeorge.com,16020,1486216506007_ChoreService_2]
hbase.ScheduledChore: Chore: HealthChecker missed its start time
2017-02-04 05:56:20,522 INFO
[slave-1.internal.larsgeorge.com,16020,1486216506007_ChoreService_2]
hbase.HealthCheckChore: Health status at 412837hrs, 56mins, 20sec :
ERROR check link, OK: disks ok,

2017-02-04 05:56:20,523 INFO
[slave-1.internal.larsgeorge.com,16020,1486216506007_ChoreService_2]
hbase.ScheduledChore: Chore: HealthChecker missed its start time
2017-02-04 05:56:50,600 INFO
[slave-1.internal.larsgeorge.com,16020,1486216506007_ChoreService_2]
hbase.HealthCheckChore: Health status at 412837hrs, 56mins, 50sec :
ERROR check link, OK: disks ok,

2017-02-04 05:56:50,600 INFO
[slave-1.internal.larsgeorge.com,16020,1486216506007_ChoreService_2]
hbase.ScheduledChore: Chore: HealthChecker missed its start time
2017-02-04 05:57:20,681 INFO
[slave-1.internal.larsgeorge.com,16020,1486216506007_ChoreService_1]
hbase.HealthCheckChore: Health status at 412837hrs, 57mins, 20sec :
ERROR check link, OK: disks ok,

2017-02-04 05:57:20,681 INFO
[slave-1.internal.larsgeorge.com,16020,1486216506007_ChoreService_1]
hbase.ScheduledChore: Chore: HealthChecker missed its start time
2017-02-04 05:57:50,763 INFO
[slave-1.internal.larsgeorge.com,16020,1486216506007_ChoreService_1]
hbase.HealthCheckChore: Health status at 412837hrs, 57mins, 50sec :
ERROR check link, OK: disks ok,

2017-02-04 05:57:50,764 INFO
[slave-1.internal.larsgeorge.com,16020,1486216506007_ChoreService_1]
hbase.ScheduledChore: Chore: HealthChecker missed its start time
2017-02-04 05:58:20,844 INFO
[slave-1.internal.larsgeorge.com,16020,1486216506007_ChoreService_1]
hbase.HealthCheckChore: Health status at 412837hrs, 58mins, 20sec :
ERROR check link, OK: disks ok,

2017-02-04 05:58:20,844 INFO
[slave-1.internal.larsgeorge.com,16020,1486216506007_ChoreService_1]
hbase.ScheduledChore: Chore: HealthChecker missed its start time
2017-02-04 05:58:50,923 INFO
[slave-1.internal.larsgeorge.com,16020,1486216506007_ChoreService_1]
hbase.HealthCheckChore: Health status at 412837hrs, 58mins, 50sec :
ERROR check link, OK: disks ok,

2017-02-04 05:58:50,923 INFO
[slave-1.internal.larsgeorge.com,16020,1486216506007_ChoreService_1]
hbase.ScheduledChore: Chore: HealthChecker missed its start time
2017-02-04 05:59:21,017 INFO
[slave-1.internal.larsgeorge.com,16020,1486216506007_ChoreService_2]
hbase.HealthCheckChore: Health status at 412837hrs, 59mins, 21sec :
ERROR check link, OK: disks ok,

2017-02-04 05:59:21,018 INFO
[slave-1.internal.larsgeorge.com,16020,1486216506007_ChoreService_2]
hbase.ScheduledChore: Chore: HealthChecker missed its start time

That seems like a bug, no?

Lars

Re: Canary Test Tool and write sniffing

[Lars George]

Also, the default interval used to be 60 secs, but is now 6 secs. Does
that make sense? Seems awfully short for a default, assuming you have
many regions or servers.

On Sat, Feb 4, 2017 at 11:54 AM, Lars George <lars.geo...@gmail.com> wrote:
> Hi,
>
> Looking at the Canary tool, it tries to ensure that all canary test
> table regions are spread across all region servers. If that is not the
> case, it calls:
>
> if (numberOfCoveredServers < numberOfServers) {
>   admin.balancer();
> }
>
> I doubt this will help with the StochasticLoadBalancer, which is known
> to consider per-table balancing as one of many factors. In practice,
> the SLB will most likely _not_ distribute the canary regions
> sufficiently, leaving gap in the check. Switching on the per-table
> option is discouraged against to let it do its thing.
>
> Just pointing it out for vetting.
>
> Lars

Canary Test Tool and write sniffing

[Lars George]

Hi,

Looking at the Canary tool, it tries to ensure that all canary test
table regions are spread across all region servers. If that is not the
case, it calls:

if (numberOfCoveredServers < numberOfServers) {
  admin.balancer();
}

I doubt this will help with the StochasticLoadBalancer, which is known
to consider per-table balancing as one of many factors. In practice,
the SLB will most likely _not_ distribute the canary regions
sufficiently, leaving gap in the check. Switching on the per-table
option is discouraged against to let it do its thing.

Just pointing it out for vetting.

Lars

Re: copy_table_desc.rb script

[Lars George]

Duh, my bad, JM you are right, I missed to type the leading slashes...
then a little better error handling may be nice. Not sure, you guys
think this warrants a JIRA?

On Fri, Feb 3, 2017 at 3:52 PM, Ted Yu <yuzhih...@gmail.com> wrote:
> Looking at the script:
>
> c1.set(HConstants::ZOOKEEPER_ZNODE_PARENT, parts1[2])
>
> It seems some validation should be added before setting the znode parent.
>
> On Fri, Feb 3, 2017 at 5:47 AM, Jean-Marc Spaggiari <jean-m...@spaggiari.org
>> wrote:
>
>> Lars, I have not tested, but should it not be localhost:2181:/hbase
>> instead? (Note the / in the front of HBase)
>>
>> JMS
>>
>> 2017-02-03 7:52 GMT-05:00 Lars George <lars.geo...@gmail.com>:
>>
>> > I have two local instances with two different ZK ports. Note that the
>> > error occurs already at the first call to HAdmin in the script.
>> > Replication works find btw, it is just this script that bails.
>> >
>> > On Fri, Feb 3, 2017 at 1:49 PM, Ted Yu <yuzhih...@gmail.com> wrote:
>> > > Note the port number 2182 in the second parameter.
>> > >
>> > > Do you have two quorums running on the same machine ?
>> > >
>> > >> On Feb 3, 2017, at 3:05 AM, Lars George <lars.geo...@gmail.com>
>> wrote:
>> > >>
>> > >> Hi,
>> > >>
>> > >> trying to run the `copy_table_desc.rb` script from the
>> > >> `bin/replication` directory gives this:
>> > >>
>> > >> ```
>> > >> $ bin/hbase org.jruby.Main bin/replication/copy_tables_desc.rb
>> > >> localhost:2181:hbase localhost:2182:hbase-2 testtable4
>> > >> ...
>> > >> NativeException: java.io.IOException:
>> > >> java.lang.reflect.InvocationTargetException
>> > >>  (root) at bin/replication/copy_tables_desc.rb:81
>> > >> ```
>> > >>
>> > >> All my 5min attempts of coercing the script kiddie JRuby code to
>> > >> output debug info was futile, hence leaving this here for vetting :)
>> > >>
>> > >> Lars
>> >
>>

Re: copy_table_desc.rb script

[Lars George]

I have two local instances with two different ZK ports. Note that the
error occurs already at the first call to HAdmin in the script.
Replication works find btw, it is just this script that bails.

On Fri, Feb 3, 2017 at 1:49 PM, Ted Yu <yuzhih...@gmail.com> wrote:
> Note the port number 2182 in the second parameter.
>
> Do you have two quorums running on the same machine ?
>
>> On Feb 3, 2017, at 3:05 AM, Lars George <lars.geo...@gmail.com> wrote:
>>
>> Hi,
>>
>> trying to run the `copy_table_desc.rb` script from the
>> `bin/replication` directory gives this:
>>
>> ```
>> $ bin/hbase org.jruby.Main bin/replication/copy_tables_desc.rb
>> localhost:2181:hbase localhost:2182:hbase-2 testtable4
>> ...
>> NativeException: java.io.IOException:
>> java.lang.reflect.InvocationTargetException
>>  (root) at bin/replication/copy_tables_desc.rb:81
>> ```
>>
>> All my 5min attempts of coercing the script kiddie JRuby code to
>> output debug info was futile, hence leaving this here for vetting :)
>>
>> Lars

copy_table_desc.rb script

[Lars George]

Hi,

trying to run the `copy_table_desc.rb` script from the
`bin/replication` directory gives this:

```
$ bin/hbase org.jruby.Main bin/replication/copy_tables_desc.rb
localhost:2181:hbase localhost:2182:hbase-2 testtable4
...
NativeException: java.io.IOException:
java.lang.reflect.InvocationTargetException
  (root) at bin/replication/copy_tables_desc.rb:81
```

All my 5min attempts of coercing the script kiddie JRuby code to
output debug info was futile, hence leaving this here for vetting :)

Lars

Re: HBase Service Level Authorization

[Lars George]

I can see how this all started:

public class HBasePolicyProvider extends PolicyProvider {
  protected static Service[] services = {
  new Service("security.client.protocol.acl", HRegionInterface.class),
  new Service("security.admin.protocol.acl", HMasterInterface.class),
  new Service("security.masterregion.protocol.acl",
HMasterRegionInterface.class)
  };

There naming made sense. But over time this got changed 3-4 times and
added the newer interface or, at last, the protobuf services. But the
current ACL name to service name is a mystery to me. Maybe we can
start the discussion there?

Lars

On Tue, Jan 24, 2017 at 8:36 AM, Lars George <lars.geo...@gmail.com> wrote:
> Hi,
>
> Looking at our `PolicyProvider` implementation I see this:
>
> /**
>  * Implementation of secure Hadoop policy provider for mapping
>  * protocol interfaces to hbase-policy.xml entries.
>  */
> @InterfaceAudience.Private
> public class HBasePolicyProvider extends PolicyProvider {
>   protected final static Service[] services = {
>   new Service("security.client.protocol.acl",
> ClientService.BlockingInterface.class),
>   new Service("security.client.protocol.acl",
> AdminService.BlockingInterface.class),
>   new Service("security.admin.protocol.acl",
> MasterService.BlockingInterface.class),
>   new Service("security.masterregion.protocol.acl",
> RegionServerStatusService.BlockingInterface.class)
>   };
>
> How does that all make sense? Both the `ClientService` and
> `AdminService` are controlled by the `security.client.protocol.acl`
> setting, allowing (or disallowing) access to the two client side APIs.
> But then there is the `security.admin.protocol.acl` for the
> `MasterService` that controls another half of the administrative
> calls?
>
> And then the last of them, which I presume by looking at the provided
> methods that it covers the Master-RegionServer communication. It is
> controlled by `security.masterregion.protocol.acl`... did we run out
> of characters to not name that at least
> `security.masterregionserver.protocol.acl`?
>
> Can anyone explain what we are doing here? And why?
>
> Best,
> Lars

HBase Service Level Authorization

[Lars George]

Hi,

Looking at our `PolicyProvider` implementation I see this:

/**
 * Implementation of secure Hadoop policy provider for mapping
 * protocol interfaces to hbase-policy.xml entries.
 */
@InterfaceAudience.Private
public class HBasePolicyProvider extends PolicyProvider {
  protected final static Service[] services = {
  new Service("security.client.protocol.acl",
ClientService.BlockingInterface.class),
  new Service("security.client.protocol.acl",
AdminService.BlockingInterface.class),
  new Service("security.admin.protocol.acl",
MasterService.BlockingInterface.class),
  new Service("security.masterregion.protocol.acl",
RegionServerStatusService.BlockingInterface.class)
  };

How does that all make sense? Both the `ClientService` and
`AdminService` are controlled by the `security.client.protocol.acl`
setting, allowing (or disallowing) access to the two client side APIs.
But then there is the `security.admin.protocol.acl` for the
`MasterService` that controls another half of the administrative
calls?

And then the last of them, which I presume by looking at the provided
methods that it covers the Master-RegionServer communication. It is
controlled by `security.masterregion.protocol.acl`... did we run out
of characters to not name that at least
`security.masterregionserver.protocol.acl`?

Can anyone explain what we are doing here? And why?

Best,
Lars

Re: Merge and HMerge

[Lars George]

I think that makes sense. The tool with its custom code dates back to where we 
had no built in version. I am all for removing all of the tools and leave the 
API call only. That is the same for an admin then compared to calling flush or 
split. 

No?

Lars

Sent from my iPhone

On 15 Jan 2017, at 04:25, Stephen Jiang <syuanjiang...@gmail.com> wrote:

>> If you remove the util.Merge tool, how then does an operator ask for a merge
> in its absence?
> 
> We have a shell command to merge region.  In the past, it calls the same RS
> side code.  I don't think there is a need to have util.Merge (even if we
> really want, we can ask this utility to call HBaseAdmin.mergeRegions, which
> is the same path from the merge command through 'hbase shell').
> 
> Thanks
> Stephen
> 
>> On Fri, Jan 13, 2017 at 11:29 PM, Stack <st...@duboce.net> wrote:
>> 
>> On Fri, Jan 13, 2017 at 7:16 PM, Stephen Jiang <syuanjiang...@gmail.com>
>> wrote:
>> 
>>> Revive this thread
>>> 
>>> I am in the process of removing Region Server side merge (and split)
>>> transaction code in master branch; as now we have merge (and split)
>>> procedure(s) from master doing the same thing.
>>> 
>>> 
>> Good (Issue?)
>> 
>> 
>>> The Merge tool depends on RS-side merge code.  I'd like to use this
>> chance
>>> to remove the util.Merge tool.  This is for 2.0 and up releases only.
>>> Deprecation does not work here; as keeping the RS-side merge code would
>>> have duplicate logic in source code and make the new Assignment manager
>>> code more complicated.
>>> 
>>> 
>> Could util.Merge be changed to ask the Master run the merge (via AMv2)?
>> 
>> If you remove the util.Merge tool, how then does an operator ask for a
>> merge in its absence?
>> 
>> Thanks Stephen
>> 
>> S
>> 
>> 
>>> Please let me know whether you have objection.
>>> 
>>> Thanks
>>> Stephen
>>> 
>>> PS.  I could deprecated HMerge code if anyone is really using it.  It has
>>> its own logic and standalone (supposed to dangerously work offline and
>>> merge more than 2 regions - the util.Merge and shell not support these
>>> functionality for now).
>>> 
>>> On Wed, Nov 16, 2016 at 11:04 AM, Enis Söztutar <enis@gmail.com>
>>> wrote:
>>> 
>>>> @Appy what is not clear from above?
>>>> 
>>>> I think we should get rid of both Merge and HMerge.
>>>> 
>>>> We should not have any tool which will work in offline mode by going
>> over
>>>> the HDFS data. Seems very brittle to be broken when things get changed.
>>>> Only use case I can think of is that somehow you end up with a lot of
>>>> regions and you cannot bring the cluster back up because of OOMs, etc
>> and
>>>> you have to reduce the number of regions in offline mode. However, we
>> did
>>>> not see this kind of thing in any of our customers for the last couple
>> of
>>>> years so far.
>>>> 
>>>> I think we should seriously look into improving normalizer and enabling
>>>> that by default for all the tables. Ideally, normalizer should be
>> running
>>>> much more frequently, and should be configured with higher-level goals
>>> and
>>>> heuristics. Like on average how many regions per node, etc and should
>> be
>>>> looking at the global state (like the balancer) to decide on split /
>>> merge
>>>> points.
>>>> 
>>>> Enis
>>>> 
>>>> On Wed, Nov 16, 2016 at 1:17 AM, Apekshit Sharma <a...@cloudera.com>
>>>> wrote:
>>>> 
>>>>> bq. HMerge can merge multiple regions by going over the list of
>>>>> regions and checking
>>>>> their sizes.
>>>>> bq. But both of these tools (Merge and HMerge) are very dangerous
>>>>> 
>>>>> I came across HMerge and it looks like dead code. Isn't referenced
>> from
>>>>> anywhere except one test. (This is what lars also pointed out in the
>>>> first
>>>>> email too).
>>>>> It would make perfect sense if it was a tool or was being referenced
>>> from
>>>>> somewhere, but with lack of either of that, am a bit confused here.
>>>>> @Enis, you seem to know everything about them, please educate me.
>>>>> Thanks
>>>>> - Appy
>>>>> 
&

Draining mode and graceful decommissioning

[Lars George]

Hi,

This is along the lines of https://issues.apache.org/jira/browse/HBASE-10367

Since we have draining mode, should we not in the meantime at least
replace the balancer stop/start step in graceful_stop.sh with simply
setting the draining mode for the server? This would less impact any
other balancer work.

BTW, I tried on 1.3 to figure the state of the balancer handling the
draining mode flag for a node, and it seems to do the right thing (see
https://www.evernote.com/l/ACHWQmzwekZN9LIVcInLjviefhwngwuhqJE) - are
there any concerns that it does not? Enis (and Stack in the original
JIRA) mentioned something about it only being implemented in one part,
but not the other?

Cheers,
Lars

[jira] [Created] (HBASE-17399) region_mover.rb uses different default filename, also needs slight change

[Lars George (JIRA)]

Lars George created HBASE-17399:
---

 Summary: region_mover.rb uses different default filename, also 
needs slight change
 Key: HBASE-17399
 URL: https://issues.apache.org/jira/browse/HBASE-17399
 Project: HBase
  Issue Type: Bug
Affects Versions: 1.3.0
Reporter: Lars George


The command line help prints:

{noformat}
-f, --filename=FILE   File to save regions list into unloading, \
or read from loading; default /tmp/hostname:port
{noformat}

while in reality, the code does this:

{code}
def getFilename(options, targetServer, port)
  filename = options[:file]
  if not filename
filename = "/tmp/" + ENV['USER'] + targetServer + ":" + port
  end
  return filename
end
{code}

An example for a generated file is:

{noformat}
/tmp/larsgeorgeslave-3.internal.larsgeorge.com\:16020
{noformat}

I suggest we fix the command line help explanation. But first, we should also 
fix how the name is generated, *adding* a divider between the user name and the 
host name, and also change how the port is attached to the host name. Currently 
this results in a rather strange {{\:}} which could be hard to handle. Maybe we 
simply use an exclamation mark or hash for both? For example:

{noformat}
/tmp/larsgeorge!slave-3.internal.larsgeorge.com!16020
/tmp/larsgeorge#slave-3.internal.larsgeorge.com#16020
{noformat}




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: update_all_config and masters

[Lars George]

I also added this for convenience:
https://issues.apache.org/jira/browse/HBASE-17391

On Thu, Dec 29, 2016 at 4:32 PM, Lars George <lars.geo...@gmail.com> wrote:
> Done: https://issues.apache.org/jira/browse/HBASE-17390
>
>
>
> On Thu, Dec 29, 2016 at 4:05 PM, Ted Yu <yuzhih...@gmail.com> wrote:
>> Good finding.
>>
>> getServers() only returns the live servers. getMaster() and 
>> getBackupMasters()
>> should also be called.
>>
>> Lars:
>> Mind creating a JIRA ?
>>
>> On Thu, Dec 29, 2016 at 6:46 AM, Lars George <lars.geo...@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> It seems that the update command is only iterating over the region servers:
>>>
>>> public void updateConfiguration() throws IOException {
>>>   for (ServerName server : this.getClusterStatus().getServers()) {
>>> updateConfiguration(server);
>>>   }
>>> }
>>>
>>> This leaves out the masters, which you have to trigger manually. In
>>> addition the real master name is hard to find, as the UI is stripping
>>> of the port and timestamp part. I had to use zk_dump to get the
>>> current master server name to send an explicit "update_config" to it.
>>>
>>> Should the "update_all_config" not include the masters? I would say so.
>>>
>>> Thoughts?
>>>
>>> Cheers,
>>> Lars
>>>

[jira] [Created] (HBASE-17391) [Shell] Add shell command to get list of servers, with filters

[Lars George (JIRA)]

Lars George created HBASE-17391:
---

 Summary: [Shell] Add shell command to get list of servers, with 
filters
 Key: HBASE-17391
 URL: https://issues.apache.org/jira/browse/HBASE-17391
 Project: HBase
  Issue Type: Improvement
  Components: shell
Affects Versions: 1.3.0
Reporter: Lars George


For some operations, for example calling {{update_config}}, the user needs to 
specify the full server name. For region servers that is easier to find, but 
not so much for the master (using {{zk_dump}} works but is noisy). It woould be 
good to add a utility call that lists the servers, preferably with an optional 
filter (a regexp, server type, or globbing style format) that allows to whittle 
down the potentially long is of servers. For example:

{noformat}
hbase(main):001:0> list_servers "master"
master-1.internal.larsgeorge.com,16000,1483018890074

hbase(main):002:0> list_servers "rs"
slave-1.internal.larsgeorge.com,16020,1482996572051
slave-3.internal.larsgeorge.com,16020,1482996572481
slave-2.internal.larsgeorge.com,16020,1482996570909

hbase(main):003:0> list_servers "rs:s.*\.com.*"
slave-1.internal.larsgeorge.com,16020,1482996572051
slave-3.internal.larsgeorge.com,16020,1482996572481
slave-2.internal.larsgeorge.com,16020,1482996570909

hbase(main):004:0> list_servers ":.*160?0.*"
master-1.internal.larsgeorge.com,16000,1483018890074
slave-1.internal.larsgeorge.com,16020,1482996572051
slave-3.internal.larsgeorge.com,16020,1482996572481
slave-2.internal.larsgeorge.com,16020,1482996570909
{noformat}

I could imagine to have {{master}}, {{backup-master}}, {{rs}}, and maybe even 
{{zk}} too. The optional regexp shown uses a colon as a divider. This combines 
the "by-type", and using a filter. Example #4 skips the type and only is using 
the filter.

Of course, you could also implement this differently, say with two 
parameters... just suggesting.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-17390) Online update of configuration for all servers leaves out masters

[Lars George (JIRA)]

Lars George created HBASE-17390:
---

 Summary: Online update of configuration for all servers leaves out 
masters
 Key: HBASE-17390
 URL: https://issues.apache.org/jira/browse/HBASE-17390
 Project: HBase
  Issue Type: Bug
Affects Versions: 1.3.0
Reporter: Lars George


Looking at the admin API and this method

{code}
  public void updateConfiguration() throws IOException {
for (ServerName server : this.getClusterStatus().getServers()) {
  updateConfiguration(server);
}
  }
{code}

you can see that it calls {{getServers()}} which only returns the region 
servers.

What is missing is also calling on {{getMaster()}} and {{getBackupMasters()}} 
to also send them a signal.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: update_all_config and masters

[Lars George]

Done: https://issues.apache.org/jira/browse/HBASE-17390



On Thu, Dec 29, 2016 at 4:05 PM, Ted Yu <yuzhih...@gmail.com> wrote:
> Good finding.
>
> getServers() only returns the live servers. getMaster() and getBackupMasters()
> should also be called.
>
> Lars:
> Mind creating a JIRA ?
>
> On Thu, Dec 29, 2016 at 6:46 AM, Lars George <lars.geo...@gmail.com> wrote:
>
>> Hi,
>>
>> It seems that the update command is only iterating over the region servers:
>>
>> public void updateConfiguration() throws IOException {
>>   for (ServerName server : this.getClusterStatus().getServers()) {
>> updateConfiguration(server);
>>   }
>> }
>>
>> This leaves out the masters, which you have to trigger manually. In
>> addition the real master name is hard to find, as the UI is stripping
>> of the port and timestamp part. I had to use zk_dump to get the
>> current master server name to send an explicit "update_config" to it.
>>
>> Should the "update_all_config" not include the masters? I would say so.
>>
>> Thoughts?
>>
>> Cheers,
>> Lars
>>

update_all_config and masters

[Lars George]

Hi,

It seems that the update command is only iterating over the region servers:

public void updateConfiguration() throws IOException {
  for (ServerName server : this.getClusterStatus().getServers()) {
updateConfiguration(server);
  }
}

This leaves out the masters, which you have to trigger manually. In
addition the real master name is hard to find, as the UI is stripping
of the port and timestamp part. I had to use zk_dump to get the
current master server name to send an explicit "update_config" to it.

Should the "update_all_config" not include the masters? I would say so.

Thoughts?

Cheers,
Lars

Status of HBASE-4755?

[Lars George]

Hi,

I am wondering where https://issues.apache.org/jira/browse/HBASE-4755
is up to? Seeing this lasting many years now, why is it not closed? It
seems the balancer is in place.

What worries me is that the tool to recompute a missing region server
location after a failover is having a comment like this (see the TODO
below):

/**
 * A tool that is used for manipulating and viewing favored nodes information
 * for regions. Run with -h to get a list of the options
 */
@InterfaceAudience.Private
// TODO: Remove? Unused. Partially implemented only.
public class RegionPlacementMaintainer {

If everything is in place but the recomputation, which should be
automated as discussed in the JIRA years ago, what is the hold up? Why
do we have such a feature lingering away?

Or let me ask the other way around: who is actively using this
feature, and how are they doing it?

Cheers,
Lars

Balancer Cutoff

[Lars George]

Hi,

In HMaster.java we have this code (added, mostly, in HBASE-3422):

private int getBalancerCutoffTime() {
  int balancerCutoffTime =
getConfiguration().getInt("hbase.balancer.max.balancing", -1);
  if (balancerCutoffTime == -1) {
// No time period set so create one
int balancerPeriod =
  getConfiguration().getInt("hbase.balancer.period", 30);
balancerCutoffTime = balancerPeriod;
// If nonsense period, set it to balancerPeriod
if (balancerCutoffTime <= 0) balancerCutoffTime = balancerPeriod;
  }
  return balancerCutoffTime;
}

Looking at these lines in particular, it seems that HBASE-8119 has
added the unconditional line, make the conditional obsolete:

balancerCutoffTime = balancerPeriod;
// If nonsense period, set it to balancerPeriod
if (balancerCutoffTime <= 0) balancerCutoffTime = balancerPeriod;

Shall I just create a JIRA, patch and remove the conditional assignment?

And since the cut off time is not set, it defaults to the same 5mins
as the balancer period resolves to, making this basically a continuous
process, as the chore is calling balance every 5 mins, and it then
runs for 5 mins before stopping. How does this address the issue
reported in HBASE-3422?

Best,
Lars

[jira] [Created] (HBASE-16815) Low scan ratio in RPC queue tuning triggers divide by zero exception

[Lars George (JIRA)]

Lars George created HBASE-16815:
---

 Summary: Low scan ratio in RPC queue tuning triggers divide by 
zero exception
 Key: HBASE-16815
 URL: https://issues.apache.org/jira/browse/HBASE-16815
 Project: HBase
  Issue Type: Bug
  Components: regionserver, rpc
Affects Versions: 2.0.0, 1.3.0
Reporter: Lars George


Trying the following settings:

{noformat}

  hbase.ipc.server.callqueue.handler.factor
  0.5


  hbase.ipc.server.callqueue.read.ratio
  0.5


  hbase.ipc.server.callqueue.scan.ratio
  0.1

{noformat}

With 30 default handlers, this means 15 queues. Further, it means 8 write 
queues and 7 read queues. 10% of that is {{0.7}} which is then floor'ed to 
{{0}}. The debug log confirms it, as the tertiary check omits the scan details 
when they are zero:

{noformat}
2016-10-12 12:50:27,305 INFO  [main] ipc.SimpleRpcScheduler: Using fifo as user 
call queue, count=15
2016-10-12 12:50:27,311 DEBUG [main] ipc.RWQueueRpcExecutor: FifoRWQ.default 
writeQueues=7 writeHandlers=15 readQueues=8 readHandlers=14
{noformat}

But the code in {{RWQueueRpcExecutor}} calls {{RpcExecutor.startHandler()}} 
nevertheless and that does this:

{code}
for (int i = 0; i < numHandlers; i++) {
  final int index = qindex + (i % qsize);
  String name = "RpcServer." + threadPrefix + ".handler=" + handlers.size() 
+ ",queue=" +
  index + ",port=" + port;
{code}

The modulo triggers then 

{noformat}
2016-10-12 11:41:22,810 ERROR [main] master.HMasterCommandLine: Master exiting
java.lang.RuntimeException: Failed construction of Master: class 
org.apache.hadoop.hbase.master.HMasterCommandLine$LocalHMaster
at 
org.apache.hadoop.hbase.util.JVMClusterUtil.createMasterThread(JVMClusterUtil.java:145)
at 
org.apache.hadoop.hbase.LocalHBaseCluster.addMaster(LocalHBaseCluster.java:220)
at 
org.apache.hadoop.hbase.LocalHBaseCluster.(LocalHBaseCluster.java:155)
at 
org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:222)
at 
org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:137)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at 
org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126)
at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:2524)
Caused by: java.lang.ArithmeticException: / by zero
at 
org.apache.hadoop.hbase.ipc.RpcExecutor.startHandlers(RpcExecutor.java:125)
at 
org.apache.hadoop.hbase.ipc.RWQueueRpcExecutor.startHandlers(RWQueueRpcExecutor.java:178)
at org.apache.hadoop.hbase.ipc.RpcExecutor.start(RpcExecutor.java:78)
at 
org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.start(SimpleRpcScheduler.java:272)
at org.apache.hadoop.hbase.ipc.RpcServer.start(RpcServer.java:2212)
at 
org.apache.hadoop.hbase.regionserver.RSRpcServices.start(RSRpcServices.java:1143)
at 
org.apache.hadoop.hbase.regionserver.HRegionServer.(HRegionServer.java:615)
at org.apache.hadoop.hbase.master.HMaster.(HMaster.java:396)
at 
org.apache.hadoop.hbase.master.HMasterCommandLine$LocalHMaster.(HMasterCommandLine.java:312)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at 
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at 
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
at 
org.apache.hadoop.hbase.util.JVMClusterUtil.createMasterThread(JVMClusterUtil.java:140)
... 7 more
{noformat}

That causes the server to not even start. I would suggest we either skip the 
{{startHandler()}} call altogether, or make it zero aware.

Another possible option is to reserve at least _one_ scan handler/queue when 
the scan ratio is greater than zero, but only of there is more than one read 
handler/queue to begin with. Otherwise the scan handler/queue should be zero 
and share the one read handler/queue.

Makes sense?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Merge and HMerge

[Lars George]

Just to save you from searching:
https://issues.apache.org/jira/browse/HBASE-8219?focusedCommentId=13617529=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13617529

No one replied to Enis it seems?

On Wed, Sep 28, 2016 at 4:32 PM, Lars George <lars.geo...@gmail.com> wrote:
> Hey,
>
> Sorry to resurrect this old thread, but working on the book update, I
> came across the same today, i.e. we have Merge and HMerge. I tried and
> Merge works fine now. It is also the only one of the two flagged as
> being a tool. Should HMerge be removed? At least deprecated?
>
> Cheers,
> Lars
>
>
> On Thu, Jul 7, 2011 at 2:03 AM, Ted Yu <yuzhih...@gmail.com> wrote:
>>>> there is already an issue to do this but not revamp of these Merge
>> classes
>> I guess the issue is HBASE-1621
>>
>> On Wed, Jul 6, 2011 at 2:28 PM, Stack <st...@duboce.net> wrote:
>>
>>> Yeah, can you file an issue Lars.  This stuff is ancient and needs to
>>> be redone AND redone so we can do merging while table is online (there
>>> is already an issue to do this but not revamp of these Merge classes).
>>>  The unit tests for Merge are also all junit3 and do whacky stuff to
>>> put up multiple regions.  This should be redone too (they are often
>>> first thing broke when major change and putting them back together is
>>> a headache since they do not follow the usual pattern).
>>>
>>> St.Ack
>>>
>>> On Sun, Jul 3, 2011 at 12:38 AM, Lars George <lars.geo...@gmail.com>
>>> wrote:
>>> > Hi Ted,
>>> >
>>> > The log is from an earlier attempt, I tried this a few times. This is all
>>> local, after rm'ing the /hbase. So the files are all pretty empty, but since
>>> I put data in I was assuming it should work. Once you gotten into this
>>> state, you also get funny error messages in the shell:
>>> >
>>> > hbase(main):001:0> list
>>> > TABLE
>>> > 11/07/03 09:36:21 INFO ipc.HBaseRPC: Using
>>> org.apache.hadoop.hbase.ipc.WritableRpcEngine for
>>> org.apache.hadoop.hbase.ipc.HMasterInterface
>>> >
>>> > ERROR: undefined method `map' for nil:NilClass
>>> >
>>> > Here is some help for this command:
>>> > List all tables in hbase. Optional regular expression parameter could
>>> > be used to filter the output. Examples:
>>> >
>>> >  hbase> list
>>> >  hbase> list 'abc.*'
>>> >
>>> >
>>> > hbase(main):002:0>
>>> >
>>> > I am assuming this is collateral, but why? The UI works but the table is
>>> gone too.
>>> >
>>> > Lars
>>> >
>>> > On Jul 2, 2011, at 10:55 PM, Ted Yu wrote:
>>> >
>>> >> There is TestMergeTool which tests Merge.
>>> >>
>>> >> From the log you provided, I got a little confused as why
>>> >> 'testtable,row-20,1309613053987.23a35ac696bdf4a8023dcc4c5b8419e0.'
>>> didn't
>>> >> appear in your command line or the output from .META. scanning.
>>> >>
>>> >> On Sat, Jul 2, 2011 at 10:36 AM, Lars George <lars.geo...@gmail.com>
>>> wrote:
>>> >>
>>> >>> Hi,
>>> >>>
>>> >>> These two seem both in a bit of a weird state: HMerge is scoped package
>>> >>> local, therefore no one but the package can call the merge()
>>> functions...
>>> >>> and no one does that but the unit test. But it would be good to have
>>> this on
>>> >>> the CLI and shell as a command (and in the shell maybe with a
>>> confirmation
>>> >>> message?), but it is not available AFAIK.
>>> >>>
>>> >>> HMerge can merge regions of tables that are disabled. It also merges
>>> all
>>> >>> that qualify, i.e. where the merged region is less than or equal of
>>> half the
>>> >>> configured max file size.
>>> >>>
>>> >>> Merge on the other hand does have a main(), so can be invoked:
>>> >>>
>>> >>> $ hbase org.apache.hadoop.hbase.util.Merge
>>> >>> Usage: bin/hbase merge   
>>> >>>
>>> >>> Note how the help insinuates that you can use it as a tool, but that is
>>> not
>>> >>> correct. Also, it only merges two given regions, and the cluster must
>>> be
>&g

Re: Merge and HMerge

[Lars George]

Hey,

Sorry to resurrect this old thread, but working on the book update, I
came across the same today, i.e. we have Merge and HMerge. I tried and
Merge works fine now. It is also the only one of the two flagged as
being a tool. Should HMerge be removed? At least deprecated?

Cheers,
Lars


On Thu, Jul 7, 2011 at 2:03 AM, Ted Yu <yuzhih...@gmail.com> wrote:
>>> there is already an issue to do this but not revamp of these Merge
> classes
> I guess the issue is HBASE-1621
>
> On Wed, Jul 6, 2011 at 2:28 PM, Stack <st...@duboce.net> wrote:
>
>> Yeah, can you file an issue Lars.  This stuff is ancient and needs to
>> be redone AND redone so we can do merging while table is online (there
>> is already an issue to do this but not revamp of these Merge classes).
>>  The unit tests for Merge are also all junit3 and do whacky stuff to
>> put up multiple regions.  This should be redone too (they are often
>> first thing broke when major change and putting them back together is
>> a headache since they do not follow the usual pattern).
>>
>> St.Ack
>>
>> On Sun, Jul 3, 2011 at 12:38 AM, Lars George <lars.geo...@gmail.com>
>> wrote:
>> > Hi Ted,
>> >
>> > The log is from an earlier attempt, I tried this a few times. This is all
>> local, after rm'ing the /hbase. So the files are all pretty empty, but since
>> I put data in I was assuming it should work. Once you gotten into this
>> state, you also get funny error messages in the shell:
>> >
>> > hbase(main):001:0> list
>> > TABLE
>> > 11/07/03 09:36:21 INFO ipc.HBaseRPC: Using
>> org.apache.hadoop.hbase.ipc.WritableRpcEngine for
>> org.apache.hadoop.hbase.ipc.HMasterInterface
>> >
>> > ERROR: undefined method `map' for nil:NilClass
>> >
>> > Here is some help for this command:
>> > List all tables in hbase. Optional regular expression parameter could
>> > be used to filter the output. Examples:
>> >
>> >  hbase> list
>> >  hbase> list 'abc.*'
>> >
>> >
>> > hbase(main):002:0>
>> >
>> > I am assuming this is collateral, but why? The UI works but the table is
>> gone too.
>> >
>> > Lars
>> >
>> > On Jul 2, 2011, at 10:55 PM, Ted Yu wrote:
>> >
>> >> There is TestMergeTool which tests Merge.
>> >>
>> >> From the log you provided, I got a little confused as why
>> >> 'testtable,row-20,1309613053987.23a35ac696bdf4a8023dcc4c5b8419e0.'
>> didn't
>> >> appear in your command line or the output from .META. scanning.
>> >>
>> >> On Sat, Jul 2, 2011 at 10:36 AM, Lars George <lars.geo...@gmail.com>
>> wrote:
>> >>
>> >>> Hi,
>> >>>
>> >>> These two seem both in a bit of a weird state: HMerge is scoped package
>> >>> local, therefore no one but the package can call the merge()
>> functions...
>> >>> and no one does that but the unit test. But it would be good to have
>> this on
>> >>> the CLI and shell as a command (and in the shell maybe with a
>> confirmation
>> >>> message?), but it is not available AFAIK.
>> >>>
>> >>> HMerge can merge regions of tables that are disabled. It also merges
>> all
>> >>> that qualify, i.e. where the merged region is less than or equal of
>> half the
>> >>> configured max file size.
>> >>>
>> >>> Merge on the other hand does have a main(), so can be invoked:
>> >>>
>> >>> $ hbase org.apache.hadoop.hbase.util.Merge
>> >>> Usage: bin/hbase merge   
>> >>>
>> >>> Note how the help insinuates that you can use it as a tool, but that is
>> not
>> >>> correct. Also, it only merges two given regions, and the cluster must
>> be
>> >>> shut down (only the HBase daemons). So that is a step back.
>> >>>
>> >>> What is worse is that I cannot get it to work. I tried in the shell:
>> >>>
>> >>> hbase(main):001:0> create 'testtable', 'colfam1',  {SPLITS =>
>> >>> ['row-10','row-20','row-30','row-40','row-50']}
>> >>> 0 row(s) in 0.2640 seconds
>> >>>
>> >>> hbase(main):002:0> for i in '0'..'9' do for j in '0'..'9' do put
>> >>> 'testtable', "row-#{i}#{j}", "colfam1:#{j}", "#{j}" end end
>> >>> 0 row(s) in

Re: DiffKeyDeltaEncoder not fully efficient?

[Lars George]

You are right Ted, I misread that. Thanks for the second set of eyes!
Disregard this email :)

On Sun, Sep 18, 2016 at 7:09 PM, Ted Yu <yuzhih...@gmail.com> wrote:
> In compressSingleKeyValue(), we compute the number of bytes to store
> timestamp:
>
>   timestampFitsInBytes = ByteBufferUtils.longFitsIn(timestamp);
>
> Then the number of bytes to store the diff between the timestamp of
> previous cell and current timestamp is computed:
>
>   diffTimestampFitsInBytes = ByteBufferUtils.longFitsIn(diffTimestamp);
>
> After that FLAG_TIMESTAMP_IS_DIFF would be set if it is more efficient to
> store the diff:
>
>   if (diffTimestampFitsInBytes < timestampFitsInBytes) {
>
> flag |= (diffTimestampFitsInBytes - 1) << SHIFT_TIMESTAMP_LENGTH;
>
> flag |= FLAG_TIMESTAMP_IS_DIFF;
>
> This should achieve good efficiency.
>
> Cheers
>
> On Sun, Sep 18, 2016 at 9:37 AM, Lars George <lars.geo...@gmail.com> wrote:
>
>> Hi,
>>
>> Reading the key encoder code and noticed this in the DiffKeyDeltaEncoder:
>>
>> if ((flag & FLAG_TIMESTAMP_IS_DIFF) == 0) {
>>   ByteBufferUtils.putLong(out, timestamp, timestampFitsInBytes);
>> } else {
>>   ByteBufferUtils.putLong(out, diffTimestamp, diffTimestampFitsInBytes);
>> }
>>
>> So if the timestamp is the same as the one from the previous cell, we
>> store it again in its full fidelity? This makes no sense as we omit
>> otherwise all identical fields. I would assume this is a mistake?
>>
>> In the decoding we do this:
>>
>> // handle timestamp
>> int timestampFitsInBytes =
>> ((flag & MASK_TIMESTAMP_LENGTH) >>> SHIFT_TIMESTAMP_LENGTH) + 1;
>> long timestamp = ByteBufferUtils.readLong(source, timestampFitsInBytes);
>> if ((flag & FLAG_TIMESTAMP_SIGN) != 0) {
>>   timestamp = -timestamp;
>> }
>> if ((flag & FLAG_TIMESTAMP_IS_DIFF) != 0) {
>>   timestamp = state.timestamp - timestamp;
>> }
>> buffer.putLong(timestamp);
>>
>> This could be changed to simply use the state.timestamp if the value
>> is the same, no? Right now we would add six bytes for the cell
>> timestamp when they are identical.
>>
>> If they are not, we store delta time, so I guess in practice this does
>> not happen too often, that we have the same time, unless they are
>> client supplied, say for a row to emulate transactions.
>>
>> Should I open a ticket?
>>
>> Cheers,
>> Lars
>>

DiffKeyDeltaEncoder not fully efficient?

[Lars George]

Hi,

Reading the key encoder code and noticed this in the DiffKeyDeltaEncoder:

if ((flag & FLAG_TIMESTAMP_IS_DIFF) == 0) {
  ByteBufferUtils.putLong(out, timestamp, timestampFitsInBytes);
} else {
  ByteBufferUtils.putLong(out, diffTimestamp, diffTimestampFitsInBytes);
}

So if the timestamp is the same as the one from the previous cell, we
store it again in its full fidelity? This makes no sense as we omit
otherwise all identical fields. I would assume this is a mistake?

In the decoding we do this:

// handle timestamp
int timestampFitsInBytes =
((flag & MASK_TIMESTAMP_LENGTH) >>> SHIFT_TIMESTAMP_LENGTH) + 1;
long timestamp = ByteBufferUtils.readLong(source, timestampFitsInBytes);
if ((flag & FLAG_TIMESTAMP_SIGN) != 0) {
  timestamp = -timestamp;
}
if ((flag & FLAG_TIMESTAMP_IS_DIFF) != 0) {
  timestamp = state.timestamp - timestamp;
}
buffer.putLong(timestamp);

This could be changed to simply use the state.timestamp if the value
is the same, no? Right now we would add six bytes for the cell
timestamp when they are identical.

If they are not, we store delta time, so I guess in practice this does
not happen too often, that we have the same time, unless they are
client supplied, say for a row to emulate transactions.

Should I open a ticket?

Cheers,
Lars

Re: hbase:meta blocks in L1?

[Lars George]

Ah wait, this was fixed in
https://issues.apache.org/jira/browse/HBASE-15467, but is not right
(if I read this proper) in the 1.x line. It could mean that hbase:meta
is cached in L2 only.

On Thu, Jul 14, 2016 at 7:37 PM, Ted Yu <yuzhih...@gmail.com> wrote:
> Looking at FSTableDescriptors#createMetaTableDescriptor() , both column
> families have the following:
>
> .setCacheDataInL1(true),
>
> Is there something else you were looking for ?
>
> Cheers
>
> On Thu, Jul 14, 2016 at 6:59 AM, Lars George <lars.geo...@gmail.com> wrote:
>
>> Hey,
>>
>> HBASE-11364 did add `.setCacheDataInL1(true)`, but the later change to
>> use `HTableDescriptor.metaTableDescriptor()` changes the layout of the
>> `hbase:meta` to omit the L1 cache setting. Is this on purpose or an
>> oversight?
>>
>> In other words, are the meta blocks currently never cached in L1?
>>
>> Lars
>>

hbase:meta blocks in L1?

[Lars George]

Hey,

HBASE-11364 did add `.setCacheDataInL1(true)`, but the later change to
use `HTableDescriptor.metaTableDescriptor()` changes the layout of the
`hbase:meta` to omit the L1 cache setting. Is this on purpose or an
oversight?

In other words, are the meta blocks currently never cached in L1?

Lars

Cache features in shell missing?

[Lars George]

Hi,

Looking at `CacheConfig` and noticed that
`CACHE_DATA_BLOCKS_COMPRESSED_KEY` and
`DROP_BEHIND_CACHE_COMPACTION_KEY` both have no equivalent in
`HColumnDescriptor` and hence cannot be set from the Shell or admin
API. Is that not an oversight?

Should we add this?

Lars

[jira] [Resolved] (HBASE-13928) Correct doc bug introduced in HBASE-11735

[Lars George (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-13928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lars George resolved HBASE-13928.
-
Resolution: Duplicate

Was fixed in HBASE-15528.

> Correct doc bug introduced in HBASE-11735
> -
>
> Key: HBASE-13928
> URL: https://issues.apache.org/jira/browse/HBASE-13928
> Project: HBase
>  Issue Type: Task
>  Components: documentation
>Affects Versions: 0.99.0, 0.98.4, 0.98.5
>Reporter: Misty Stanley-Jones
>Assignee: Misty Stanley-Jones
> Fix For: 2.0.0, 0.98.6, 0.99.0
>
>
> {quote}Biju Nair added a comment - 09/Jun/15 04:53
> I think the parameter hbase.bucketcache.sizes is used in the document patch 
> instead of hbase.bucketcache.bucket.sizes to configure bucket sizes.
> {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Reopened] (HBASE-13928) Correct doc bug introduced in HBASE-11735

[Lars George (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-13928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lars George reopened HBASE-13928:
-

[~gsbiju] is right, the added {{hbase-default.xml}} key is wrong, and never 
used. It is missing the {{.bucket.}} as mentioned and should be reading 
{{hbase.bucketcache.bucket.sizes}}. Only then an operator can set the sizes. 
The wrong property is misleading at most, but needs fixing anyways.

> Correct doc bug introduced in HBASE-11735
> -
>
> Key: HBASE-13928
> URL: https://issues.apache.org/jira/browse/HBASE-13928
> Project: HBase
>  Issue Type: Task
>  Components: documentation
>Affects Versions: 0.99.0, 0.98.4, 0.98.5
>Reporter: Misty Stanley-Jones
>Assignee: Misty Stanley-Jones
> Fix For: 0.99.0, 2.0.0, 0.98.6
>
>
> {quote}Biju Nair added a comment - 09/Jun/15 04:53
> I think the parameter hbase.bucketcache.sizes is used in the document patch 
> instead of hbase.bucketcache.bucket.sizes to configure bucket sizes.
> {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-16188) Add EventCounter information to log4j properties file

[Lars George (JIRA)]

Lars George created HBASE-16188:
---

 Summary: Add EventCounter information to log4j properties file
 Key: HBASE-16188
 URL: https://issues.apache.org/jira/browse/HBASE-16188
 Project: HBase
  Issue Type: Improvement
Affects Versions: 1.2.1
Reporter: Lars George
Priority: Minor


Hadoop's {{JvmMetrics}}, which HBase also is using in Metrics2 and provides it 
as an MBean, has the ability to count log4j log calls. This is tracked by a 
special {{Appender}} class, also provided by Hadoop, called {{EventCounter}}. 

We should add some info how to enable this (or maybe even enable it by 
default?).

The appender needs to be added in two places, shown here:

{noformat}
hbase.root.logger=INFO,console
...
# Define the root logger to the system property "hbase.root.logger".
log4j.rootLogger=${hbase.root.logger}, EventCounter

log4j.appender.EventCounter=org.apache.hadoop.log.metrics.EventCounter
{noformat}

We could simply add this commented out akin to the {{hbase-env.sh}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-16187) Fix typo in blog post for metrics2

[Lars George (JIRA)]

Lars George created HBASE-16187:
---

 Summary: Fix typo in blog post for metrics2
 Key: HBASE-16187
 URL: https://issues.apache.org/jira/browse/HBASE-16187
 Project: HBase
  Issue Type: Bug
  Components: website
Reporter: Lars George
Assignee: Sean Busbey


See https://blogs.apache.org/hbase/entry/migration_to_the_new_metrics

s/sudo/pseudo





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-16186) Fix AssignmentManager MBean name

[Lars George (JIRA)]

Lars George created HBASE-16186:
---

 Summary: Fix AssignmentManager MBean name
 Key: HBASE-16186
 URL: https://issues.apache.org/jira/browse/HBASE-16186
 Project: HBase
  Issue Type: Bug
  Components: master
Affects Versions: 1.2.1
Reporter: Lars George
 Fix For: 2.0.0


The MBean has a spelling error, listed as "AssignmentManger" (note the missing 
"a"). This is a publicly available name that tools might already use to filter 
metrics etc. We should change this across major versions only?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: MutableQuantiles

[Lars George]

Hi Elliott,

I was referring to MutableQuantiles vs MetricsMutableQuantiles. The
Histogram ones are cleaned up as you say and I saw. But what about the
former pair?

Lars

On Mon, Jun 20, 2016 at 7:13 PM, Elliott Clark <ecl...@apache.org> wrote:
> MetricMutableHistogram and all ( The classes which were ports of hadoop's
> classes) have been removed. They are no longer used.  However the ones
> hadoop supplies were very slow so instead we use MutableHistogram.
>
> See:  HBASE-15222
>
> On Mon, Jun 20, 2016 at 5:19 AM, Lars George <lars.geo...@gmail.com> wrote:
>
>> BTW, I am looking at 1.2 branch, though here the Hadoop one does
>> exactly the same as what the HBase one does. Where do I see the
>> difference? Master looks the same too. Are you referring to the
>> histogram classes?
>>
>> On Mon, Jun 20, 2016 at 1:36 PM, Lars George <lars.geo...@gmail.com>
>> wrote:
>> > Ah thanks Andy. It seemed mostly a copy (with some internal
>> > modification). Now, where is that used at all?
>> >
>> > On Sun, Jun 19, 2016 at 7:06 PM, Andrew Purtell
>> > <andrew.purt...@gmail.com> wrote:
>> >> We have additional functionality that the Hadoop supplied one does not,
>> importantly the ability to dump counts by latency bucket rather than
>> percentile measures at the moment. The former can be used to calculate
>> mathematically meaningful percentile measures over the whole fleet and over
>> longer timescales, the latter cannot.
>> >>
>> >>
>> >>> On Jun 19, 2016, at 9:36 AM, Lars George <lars.geo...@gmail.com>
>> wrote:
>> >>>
>> >>> Hi,
>> >>>
>> >>> As per https://issues.apache.org/jira/browse/HBASE-6409 we rolled our
>> >>> own class. Is that still needed? Since 2012 lot's has changed and we
>> >>> should have all in place to use the Hadoop supplied one?
>> >>>
>> >>> Just curious.
>> >>>
>> >>> Cheers,
>> >>> Lars
>>

Re: MutableQuantiles

[Lars George]

BTW, I am looking at 1.2 branch, though here the Hadoop one does
exactly the same as what the HBase one does. Where do I see the
difference? Master looks the same too. Are you referring to the
histogram classes?

On Mon, Jun 20, 2016 at 1:36 PM, Lars George <lars.geo...@gmail.com> wrote:
> Ah thanks Andy. It seemed mostly a copy (with some internal
> modification). Now, where is that used at all?
>
> On Sun, Jun 19, 2016 at 7:06 PM, Andrew Purtell
> <andrew.purt...@gmail.com> wrote:
>> We have additional functionality that the Hadoop supplied one does not, 
>> importantly the ability to dump counts by latency bucket rather than 
>> percentile measures at the moment. The former can be used to calculate 
>> mathematically meaningful percentile measures over the whole fleet and over 
>> longer timescales, the latter cannot.
>>
>>
>>> On Jun 19, 2016, at 9:36 AM, Lars George <lars.geo...@gmail.com> wrote:
>>>
>>> Hi,
>>>
>>> As per https://issues.apache.org/jira/browse/HBASE-6409 we rolled our
>>> own class. Is that still needed? Since 2012 lot's has changed and we
>>> should have all in place to use the Hadoop supplied one?
>>>
>>> Just curious.
>>>
>>> Cheers,
>>> Lars

Re: MutableQuantiles

[Lars George]

Ah thanks Andy. It seemed mostly a copy (with some internal
modification). Now, where is that used at all?

On Sun, Jun 19, 2016 at 7:06 PM, Andrew Purtell
<andrew.purt...@gmail.com> wrote:
> We have additional functionality that the Hadoop supplied one does not, 
> importantly the ability to dump counts by latency bucket rather than 
> percentile measures at the moment. The former can be used to calculate 
> mathematically meaningful percentile measures over the whole fleet and over 
> longer timescales, the latter cannot.
>
>
>> On Jun 19, 2016, at 9:36 AM, Lars George <lars.geo...@gmail.com> wrote:
>>
>> Hi,
>>
>> As per https://issues.apache.org/jira/browse/HBASE-6409 we rolled our
>> own class. Is that still needed? Since 2012 lot's has changed and we
>> should have all in place to use the Hadoop supplied one?
>>
>> Just curious.
>>
>> Cheers,
>> Lars

MutableQuantiles

[Lars George]

Hi,

As per https://issues.apache.org/jira/browse/HBASE-6409 we rolled our
own class. Is that still needed? Since 2012 lot's has changed and we
should have all in place to use the Hadoop supplied one?

Just curious.

Cheers,
Lars

[jira] [Created] (HBASE-15445) Add support for ACLs for web based UIs

[Lars George (JIRA)]

Lars George created HBASE-15445:
---

 Summary: Add support for ACLs for web based UIs
 Key: HBASE-15445
 URL: https://issues.apache.org/jira/browse/HBASE-15445
 Project: HBase
  Issue Type: Bug
  Components: master, regionserver, REST, Thrift
Affects Versions: 1.1.3, 1.0.3, 1.2.0
Reporter: Lars George


Since 0.99 and HBASE-10336 we have our own HttpServer class that (like the 
counterpart in Hadoop) supports setting an ACL to allow only named users to 
access the web based UIs of the server processes. In secure mode we should 
support this as it works hand-in-hand with Kerberos authorization and the UGI 
class. It seems all we have to do is add a property allowing to set the ACL 
property as a list of users and/or groups that have access to the UIs if needed.

As an add-on, we could combine this with the {{read-only}} flag, so that some 
users can only access the UIs with any option to trigger, for example, splits. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Where to set UI ACLs?

[Lars George]

Indeed, that was my thinking. There is already provisioning for it in
the code, so we just need to wire it in. f no objections I'll create a
JIRA and give it a go.

On Wed, Mar 9, 2016 at 9:25 PM, Andrew Purtell <apurt...@apache.org> wrote:
> I think we need an JIRA. We haven't considered access control for the UIs
> before. IMHO, they are inherently unsafe except for operator use ("no user
> serviceable parts inside") so random folks should not be given network
> paths to them.
>
> On Wed, Mar 9, 2016 at 5:31 AM, Lars George <lars.geo...@gmail.com> wrote:
>
>> Hi,
>>
>> Reading the whole HttpServer code base, and while this is a copy it
>> seems from HttpServer2, including the ability to set ACLs with users
>> who are allowed to access (admins), I cannot see this ever being set.
>> Am I missing something, or is there a JIRA documenting that this needs
>> adding?
>>
>> Thanks,
>> Lars
>>
>
>
>
> --
> Best regards,
>
>- Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)

Where to set UI ACLs?

[Lars George]

Hi,

Reading the whole HttpServer code base, and while this is a copy it
seems from HttpServer2, including the ability to set ACLs with users
who are allowed to access (admins), I cannot see this ever being set.
Am I missing something, or is there a JIRA documenting that this needs
adding?

Thanks,
Lars

Normalizer in 1.2+?

[Lars George]

Hi,

Just looking through the new properties and seeing this

  
hbase.normalizer.enabled
false
If set to true, Master will try to keep region size
  within each table approximately the same.
  

Searching both branches 1.2 and master reveals it is only used in a test to
enable it, but never anywhere else to check it is disabled. What am I
missing? Is the normalizer always on no matter what you set (with the
default "false" not working then)?

Lars

[jira] [Created] (HBASE-15098) Normalizer switch in configuration is not used

[Lars George (JIRA)]

Lars George created HBASE-15098:
---

 Summary: Normalizer switch in configuration is not used
 Key: HBASE-15098
 URL: https://issues.apache.org/jira/browse/HBASE-15098
 Project: HBase
  Issue Type: Bug
  Components: master
Affects Versions: 1.2.0
Reporter: Lars George
 Fix For: 2.0.0, 1.2.0, 1.2.1


The newly added global switch to enable the new normalizer functionality is 
never used apparently, meaning it is always on. The {{hbase-default.xml}} has 
this:

{noformat}
  
hbase.normalizer.enabled
false
If set to true, Master will try to keep region size
  within each table approximately the same.
  
{noformat}

But only a test class uses it to set the switch to "true". We should implement 
a proper {{if}} statement that checks this value and properly disables the 
feature cluster wide if not wanted.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Normalizer in 1.2+?

[Lars George]

Hi Ted,

Wait a minute... this means it is always on by default. Is this wanted?
This is a new feature and should be tested in the field first?

But you agree this is not right, so I will create a JIRA to track this.

Cheers,
Lars

On Wed, Jan 13, 2016 at 11:13 AM, Ted Yu <yuzhih...@gmail.com> wrote:

> There is RegionNormalizerTracker which tracks region normalizer state up in
> ZK.
>
> When user toggles normalizer switch through shell, the following method
> in MasterRpcServices is called:
>
>   public boolean normalizerSwitch(boolean on) {
>
> boolean oldValue =
> master.getRegionNormalizerTracker().isNormalizerOn();
>
> ...
>
> master.getRegionNormalizerTracker().setNormalizerOn(newValue);
>
> In RegionNormalizerTracker :
>
>   public boolean isNormalizerOn() {
>
> byte [] upData = super.getData(false);
>
> try {
>
>   // if data in ZK is null, use default of on.
>
>   return upData == null || parseFrom(upData).getNormalizerOn();
>
> So I guess the config parameter hbase.normalizer.enabled can be dropped.
>
>
> Cheers
>
> On Wed, Jan 13, 2016 at 3:01 AM, Lars George <lars.geo...@gmail.com>
> wrote:
>
> > Hi,
> >
> > Just looking through the new properties and seeing this
> >
> >   
> > hbase.normalizer.enabled
> > false
> > If set to true, Master will try to keep region size
> >   within each table approximately the same.
> >   
> >
> > Searching both branches 1.2 and master reveals it is only used in a test
> to
> > enable it, but never anywhere else to check it is disabled. What am I
> > missing? Is the normalizer always on no matter what you set (with the
> > default "false" not working then)?
> >
> > Lars
> >
>

[jira] [Created] (HBASE-14864) Add support for bucketing of keys into client library

[Lars George (JIRA)]

Lars George created HBASE-14864:
---

 Summary: Add support for bucketing of keys into client library
 Key: HBASE-14864
 URL: https://issues.apache.org/jira/browse/HBASE-14864
 Project: HBase
  Issue Type: New Feature
  Components: Client
Reporter: Lars George


This has been discussed and taught so many times, I believe it is time to 
support it properly. The idea is to be able to assign an optional _bucketing_ 
strategy to a table, which translates the user given row keys into a bucketed 
version. This is done by either simple count, or by parts of the key. Possibly 
some simple functionality should help _compute_ bucket keys. 

For example, given a key {{---...}} you could imagine 
that a rule can be defined that takes the _epoch_ part and chunks it into, for 
example, 5 minute buckets. This allows to store small time series together and 
make reading (especially over many servers) much more efficient.

The client also supports the proper scan logic to fan a scan over the buckets 
as needed. There may be an executor service (implicitly or explicitly provided) 
that is used to fetch the original data with user visible ordering from the 
distributed buckets. 

Note that this has been attempted a few times to various extends out in the 
field, but then withered away. This is an essential feature that when present 
in the API will make users consider this earlier, instead of when it is too 
late (when hot spotting occurs for example).

The selected bucketing strategy and settings could be stored in the table 
descriptor key/value pairs. This will allow any client to observe the strategy 
transparently. If not set the behaviour is the same as today, so the new 
feature is not touching any critical path in terms of code, and is fully client 
side. (But could be considered for say UI support as well - if needed).

The strategies are pluggable using classes, but a few default implementations 
are supplied.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-14556) Make prefetchOnOpen configurable for index and/or data blocks

[Lars George (JIRA)]

Lars George created HBASE-14556:
---

 Summary: Make prefetchOnOpen configurable for index and/or data 
blocks
 Key: HBASE-14556
 URL: https://issues.apache.org/jira/browse/HBASE-14556
 Project: HBase
  Issue Type: Bug
  Components: BlockCache, regionserver
Reporter: Lars George


This came up in user discussions. It would be great to add an extra option to 
the {{CacheConfig}} that allows to specify what blocks are cached during 
region/file opening. This should allows to set {{BlockIndexOnly}}, 
{{BloomFilterOnly}}, {{AllIndexesOnly}}, {{DataOnly}}, and {{AllBlocks}}. 

For large datasets it is not viable to load all blocks into memory, but to 
speed up access it still makes sense to prefetch the index blocks (being block 
index and Bloom filter blocks).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-14367) Add normalization support to shell

[Lars George (JIRA)]

Lars George created HBASE-14367:
---

 Summary: Add normalization support to shell
 Key: HBASE-14367
 URL: https://issues.apache.org/jira/browse/HBASE-14367
 Project: HBase
  Issue Type: Bug
  Components: shell
Affects Versions: 1.1.2
Reporter: Lars George
 Fix For: 2.0.0, 1.2.0, 1.3.0


https://issues.apache.org/jira/browse/HBASE-13103 adds support for setting a 
normalization flag per {{HTableDescriptor}}, along with the server side chore 
to do the work.

What is lacking is to easily set this from the shell, right now you need to use 
the Java API to modify the descriptor. This issue is to add the flag as a known 
attribute key and/or other means to toggle this per table.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-14342) Recursive call in RegionMergeTransactionImpl.getJournal()

[Lars George (JIRA)]

Lars George created HBASE-14342:
---

 Summary: Recursive call in RegionMergeTransactionImpl.getJournal()
 Key: HBASE-14342
 URL: https://issues.apache.org/jira/browse/HBASE-14342
 Project: HBase
  Issue Type: Bug
  Components: regionserver
Affects Versions: 1.1.1
Reporter: Lars George
 Fix For: 1.2.0, 1.1.2


HBASE-12975 in its branch-1 patch 
(https://issues.apache.org/jira/secure/attachment/12708578/HBASE-12975-branch-1.patch)
 introduced a recursive call for {{getJournal()}}. Needs to return just the 
{{journal}} variable like master patch does.





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-14343) Fix debug message in SimpleRegionNormalizer for small regions

[Lars George (JIRA)]

Lars George created HBASE-14343:
---

 Summary: Fix debug message in SimpleRegionNormalizer for small 
regions
 Key: HBASE-14343
 URL: https://issues.apache.org/jira/browse/HBASE-14343
 Project: HBase
  Issue Type: Bug
  Components: regionserver
Affects Versions: 1.1.1
Reporter: Lars George
Priority: Trivial


The {{SimpleRegionNormalizer}} has this:

{code}
 if ((smallestRegion.getSecond() + 
smallestNeighborOfSmallestRegion.getSecond()
  < avgRegionSize)) {
LOG.debug("Table " + table + ", smallest region size: " + 
smallestRegion.getSecond()
  + " and its smallest neighbor size: " + 
smallestNeighborOfSmallestRegion.getSecond()
  + ", less than half the avg size, merging them");
{code}

It does *not* check for "less than half the avg size" but only "less than the 
avg size", that is, drop the "half". Fix message.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-14287) Bootstrapping a cluster leaves temporary WAL directory laying around

[Lars George (JIRA)]

Lars George created HBASE-14287:
---

 Summary: Bootstrapping a cluster leaves temporary WAL directory 
laying around
 Key: HBASE-14287
 URL: https://issues.apache.org/jira/browse/HBASE-14287
 Project: HBase
  Issue Type: Bug
  Components: master, regionserver
Affects Versions: 1.1.1, 1.0.2
Reporter: Lars George
Priority: Minor


When a new cluster is started, it creates a temporary WAL as {{hbase:meta}} is 
created during bootstrapping the system. Then this log is closed before 
properly opened on a region server. The temp WAL file is scheduled for removal, 
moved to oldWALs and eventually claimed. Issue is that the WAL directory with 
the temp region is not removed. For example:

{noformat}
drwxr-xr-x   - hadoop hadoop  0 2015-05-28 10:21 
/hbase/WALs/hregion-65589555
{noformat}

The directory is empty and does not harm, but on the other hand it is not 
needed anymore and should be removed. Cosmetic and good housekeeping.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-14256) Flush task message may be confusing when region is recovered

[Lars George (JIRA)]

Lars George created HBASE-14256:
---

 Summary: Flush task message may be confusing when region is 
recovered
 Key: HBASE-14256
 URL: https://issues.apache.org/jira/browse/HBASE-14256
 Project: HBase
  Issue Type: Bug
  Components: regionserver
Affects Versions: 2.0.0, 1.2.0
Reporter: Lars George


In {{HRegion.setRecovering()}} we have this code:

{code}
// force a flush only if region replication is set up for this region. 
Otherwise no need.
  boolean forceFlush = getTableDesc().getRegionReplication()  1;

  // force a flush first
  MonitoredTask status = TaskMonitor.get().createStatus(
Flushing region  + this +  because recovery is finished);
  try {
if (forceFlush) {
  internalFlushcache(status);
}
{code}

So we only optionally force flush after a recovery of a region, but the message 
always is set to Flushing..., which might be confusing. We should change the 
message based on {{forceFlush}}.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-14255) Simplify Cell creation post 1.0

[Lars George (JIRA)]

Lars George created HBASE-14255:
---

 Summary: Simplify Cell creation post 1.0
 Key: HBASE-14255
 URL: https://issues.apache.org/jira/browse/HBASE-14255
 Project: HBase
  Issue Type: Improvement
  Components: Client
Affects Versions: 1.0.0, 2.0.0
Reporter: Lars George


After the switch to the new Cell based client API, and making KeyValue private 
(but especially as soon as DBB backed Cells land) it is rather difficult to 
create a {{Cell}} instance. I am using this now:

{code}
 @Override
  public void postGetOp(ObserverContextRegionCoprocessorEnvironment e,
Get get, ListCell results) throws IOException {
Put put = new Put(get.getRow());
put.addColumn(get.getRow(), FIXED_COLUMN, Bytes.toBytes(counter.get()));
CellScanner scanner = put.cellScanner();
scanner.advance();
Cell cell = scanner.current();
LOG.debug(Adding fake cell:  + cell);
results.add(cell);
  }
{code}

That is, I have to create a {{Put}} instance to add a cell and then retrieve 
its instance. The {{KeyValue}} methods are private now and should not be used. 
Create a CellBuilder helper?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-14257) Periodic flusher only handles hbase:meta, not other system tables

[Lars George (JIRA)]

Lars George created HBASE-14257:
---

 Summary: Periodic flusher only handles hbase:meta, not other 
system tables
 Key: HBASE-14257
 URL: https://issues.apache.org/jira/browse/HBASE-14257
 Project: HBase
  Issue Type: Bug
  Components: regionserver
Affects Versions: 2.0.0, 1.2.0
Reporter: Lars George


In {{HRegion.shouldFlush}} we have

{code}
long modifiedFlushCheckInterval = flushCheckInterval;
if (getRegionInfo().isMetaRegion() 
getRegionInfo().getReplicaId() == HRegionInfo.DEFAULT_REPLICA_ID) {
  modifiedFlushCheckInterval = META_CACHE_FLUSH_INTERVAL;
}
{code}

That method is called by the {{PeriodicMemstoreFlusher}} thread, and prefers 
the {{hbase:meta}} only for faster flushing. It should be doing the same for 
other system tables. I suggest to use {{HRI.isSystemTable()}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-14224) Fix coprocessor handling of duplicate classes

[Lars George (JIRA)]

Lars George created HBASE-14224:
---

 Summary: Fix coprocessor handling of duplicate classes
 Key: HBASE-14224
 URL: https://issues.apache.org/jira/browse/HBASE-14224
 Project: HBase
  Issue Type: Bug
  Components: Coprocessors
Affects Versions: 1.1.1, 1.0.1, 2.0.0, 1.2.0
Reporter: Lars George


While discussing with [~misty] over on HBASE-13907 we noticed some 
inconsistency when copros are loaded. Sometimes you can load them more than 
once, sometimes you can not. Need to consolidate.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)