[jira] [Created] (HBASE-18523) FilterList should break if the filter succeeds (row passes) in case of MUST_PASS_ONE and if the filter fails (row is skipped) for MUST_PASS_ALL

[Akshita Malhotra (JIRA)]

Akshita Malhotra created HBASE-18523:


 Summary: FilterList should break if the filter succeeds (row 
passes) in case of MUST_PASS_ONE and if the filter fails (row is skipped) for 
MUST_PASS_ALL
 Key: HBASE-18523
 URL: https://issues.apache.org/jira/browse/HBASE-18523
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.98.24
Reporter: Akshita Malhotra


FilterList should break if the filter succeeds (row passes) in case of 
MUST_PASS_ONE and if the filter fails (row is skipped) for MUST_PASS_ALL



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HBASE-18522) Add RowMutations support to Batch

[Jerry He (JIRA)]

Jerry He created HBASE-18522:


 Summary: Add RowMutations support to Batch
 Key: HBASE-18522
 URL: https://issues.apache.org/jira/browse/HBASE-18522
 Project: HBase
  Issue Type: Improvement
Affects Versions: 1.2.6
Reporter: Jerry He
Assignee: Jerry He
 Fix For: 2.0.0, 3.0.0, 1.4.0, 1.5.0


RowMutations is multiple Puts and/or Deletes atomically on a single row. 
Current Batch call does not support RowMutations as part of the batch.
We should add this missing part. We should be able to batch RowMutations.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Bug in FIFOCompactionPolicy pre-checks?

[Vladimir Rodionov]

Yes, file a JIRA, Lars

I will take a look

-Vlad


On Thu, Aug 3, 2017 at 11:41 PM, Lars George  wrote:

> Hi,
>
> See https://issues.apache.org/jira/browse/HBASE-14468
>
> It adds this check to {{HMaster.checkCompactionPolicy()}}:
>
> {code}
> // 1. Check TTL
> if (hcd.getTimeToLive() == HColumnDescriptor.DEFAULT_TTL) {
>   message = "Default TTL is not supported for FIFO compaction";
>   throw new IOException(message);
> }
>
> // 2. Check min versions
> if (hcd.getMinVersions() > 0) {
>   message = "MIN_VERSION > 0 is not supported for FIFO compaction";
>   throw new IOException(message);
> }
>
> // 3. blocking file count
> String sbfc = htd.getConfigurationValue(HStore.BLOCKING_STOREFILES_KEY);
> if (sbfc != null) {
>   blockingFileCount = Integer.parseInt(sbfc);
> }
> if (blockingFileCount < 1000) {
>   message =
>   "blocking file count '" + HStore.BLOCKING_STOREFILES_KEY + "' "
> + blockingFileCount
>   + " is below recommended minimum of 1000";
>   throw new IOException(message);
> }
> {code}
>
> Why does it only check the blocking file count on the HTD level, while
> others are check on the HCD level? Doing this for example fails
> because of it:
>
> {noformat}
> hbase(main):008:0> create 'ttltable', { NAME => 'cf1', TTL => 300,
> CONFIGURATION => { 'hbase.hstore.defaultengine.compactionpolicy.class'
> => 'org.apache.hadoop.hbase.regionserver.compactions.
> FIFOCompactionPolicy',
> 'hbase.hstore.blockingStoreFiles' => 2000 } }
>
> ERROR: org.apache.hadoop.hbase.DoNotRetryIOException: blocking file
> count 'hbase.hstore.blockingStoreFiles' 10 is below recommended
> minimum of 1000 Set hbase.table.sanity.checks to false at conf or
> table descriptor if you want to bypass sanity checks
> at org.apache.hadoop.hbase.master.HMaster.warnOrThrowExceptionForFailure
> (HMaster.java:1782)
> at org.apache.hadoop.hbase.master.HMaster.sanityCheckTableDescriptor(
> HMaster.java:1663)
> at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1545)
> at org.apache.hadoop.hbase.master.MasterRpcServices.
> createTable(MasterRpcServices.java:469)
> at org.apache.hadoop.hbase.protobuf.generated.
> MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:58549)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2339)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(
> RpcExecutor.java:188)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(
> RpcExecutor.java:168)
> Caused by: java.io.IOException: blocking file count
> 'hbase.hstore.blockingStoreFiles' 10 is below recommended minimum of
> 1000
> at org.apache.hadoop.hbase.master.HMaster.checkCompactionPolicy(HMaster.
> java:1773)
> at org.apache.hadoop.hbase.master.HMaster.sanityCheckTableDescriptor(
> HMaster.java:1661)
> ... 7 more
> {noformat}
>
> That should work on the column family level, right? Shall I file a JIRA?
>
> Cheers,
> Lars
>

[jira] [Created] (HBASE-18521) Nightly job should check branches for API breakage

[Sean Busbey (JIRA)]

Sean Busbey created HBASE-18521:
---

 Summary: Nightly job should check branches for API breakage
 Key: HBASE-18521
 URL: https://issues.apache.org/jira/browse/HBASE-18521
 Project: HBase
  Issue Type: Improvement
  Components: API, community
Reporter: Sean Busbey


Now that HBASE-18020 has landed, we should expand the nightly per-branch jobs 
to run it.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (HBASE-18020) Update API Compliance Checker to Incorporate Improvements Done in Hadoop

[Sean Busbey (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-18020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Busbey resolved HBASE-18020.
-
   Resolution: Fixed
Fix Version/s: 1.1.12
   1.2.7
   1.5.0
   1.3.2
   1.4.0
   3.0.0

Pushed to all branches. Thanks for working through this [~awleblang]. Thanks 
for the thorough review [~dimaspivak]!

> Update API Compliance Checker to Incorporate Improvements Done in Hadoop
> 
>
> Key: HBASE-18020
> URL: https://issues.apache.org/jira/browse/HBASE-18020
> Project: HBase
>  Issue Type: Improvement
>  Components: API, community
>Reporter: Alex Leblang
>Assignee: Alex Leblang
> Fix For: 2.0.0, 3.0.0, 1.4.0, 1.3.2, 1.5.0, 1.2.7, 1.1.12
>
> Attachments: HBASE-18020.0.patch, HBASE-18020.branch-1.2.001.patch, 
> HBASE-18020.branch-1.2.002.patch, HBASE-18020.branch-1.2.003.patch, 
> HBASE-18020.branch-1.2.004.patch
>
>
> Recently the Hadoop community has made a number of improvements in their api 
> compliance checker based on feedback from the hbase and kudu community. We 
> should adopt these changes ourselves.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: [DISCUSS] hbase-2.0.0 compatibility expectations

[Andrew Purtell]

This is a really good question. I think many operators will give a lot of 
leeway to data format changes as long as data can be copied from A to B 
(perhaps with batch rewrite to upgrade (ideally, not required)) and replication 
can be enabled to sync up to the current moment for cut over. 


> On Aug 4, 2017, at 10:00 AM, Esteban Gutierrez  wrote:
> 
> Should we add additional details around replication as well? for instance,
> shall we consider a hbase-1.x cluster as a client for a hbase-2.x cluster?
> 
> Thanks for starting this discussion Stack,
> 
> esteban.
> 
> --
> Cloudera, Inc.
> 
> 
>> On Fri, Aug 4, 2017 at 1:05 AM, stack  wrote:
>> 
>> Thanks Zach for clarification. Let me work up a list and then come back to
>> this thread.  Jira needs an edit pass to.
>> 
>> S
>> 
>> On Aug 3, 2017 23:54, "Zach York"  wrote:
>> 
>> This kinda helps, but these seem more like expectations. I was going more
>> for things like HFile format changed, meta table structure changed,
>> coprocessor implementations changed (these are just examples, I don't know
>> if any of these actually changed).
>> 
>> More technical differences between branch-1 and branch-2 which then can
>> help us get the right expectations for compatibility.
>> 
>>> On Wed, Aug 2, 2017 at 6:34 PM, Stack  wrote:
>>> 
>>> On Wed, Aug 2, 2017 at 5:25 PM, Zach York 
>>> wrote:
>>> 
 Do we know what the major pain points for migration are? Can we discuss
 that/get a list going?
 
 
>>> Here's a few in outline:
>>> 
>>> + There is issue of formats, of hbase-2.x being able to read hbase-1.x
>> data
>>> whether from HDFS or ZooKeeper or off the wire.
>>> + An hbase-1.x client should be able to Get/Put and Scan an hbase-2.x
>>> cluster; no holes in the API or unintelligible serializations.
>>> + There is then the little dance that has us rolling restart from an
>>> hbase-1.x cluster to hbase-2.x; i.e. upgrade master first and then it
>> will
>>> assign regions to the new hbase-2.x regionservers as they come on line.
>>> TBD.
>>> 
>>> Is this what you mean sir?
>>> 
>>> S
>>> 
>>> 
 I think without that knowledge it is hard (for me at least :) ) to
 determine where we should set our sights in terms of migration.
 
 Thanks,
 Zach
 
> On Wed, Aug 2, 2017 at 4:38 PM, Stack  wrote:
> 
> What are our expectations regards compatibility between hbase1 and
 hbase2?
> 
> Lets have a chat about it. Here are some goal posts.
> 
> + You have to upgrade to hbase-1.x before you can migrate to hbase-2.
>>> No
> migration from < hbase-1 (Is this too onerous? Should we support 0.98
>>> =>
> 2.0?).
> + You do NOT have to upgrade to the latest release of hbase1 to
>> migrate
 to
> hbase2; being up on hbase-1.0.0+ will be sufficient.
> + You'll have to update your hbase1 coprocessors to deploy them on
 hbase2.
> A bunch of CP API has/will change by the time hbase2 comes out; e.g.
> watching for region split on RegionServer no longer makes sense given
> Master runs all splits now.
> + An hbase1 client can run against an hbase2 cluster but it will only
>>> be
> able to do DML (Get/Put/Scan, etc.). We do not allow being able to do
 admin
> ops using an hbase1 Admin client against an hbase2 cluster. We have
>>> some
> egregious API violations in branch-1; e.g. we have protobuf in our
>> API
 (See
> HBASE-15607). The notion is that we can't afford a deprecation cycle
> purging this stuff from our Admin API.
> 
> What you all think?
> 
> St.Ack
> 
 
>>> 
>> 

[jira] [Created] (HBASE-18520) Add jmx value to determine true Master Start time

[Zach York (JIRA)]

Zach York created HBASE-18520:
-

 Summary: Add jmx value to determine true Master Start time
 Key: HBASE-18520
 URL: https://issues.apache.org/jira/browse/HBASE-18520
 Project: HBase
  Issue Type: Bug
  Components: metrics
Reporter: Zach York
Assignee: Zach York
Priority: Minor


The masterActiveTime is being set before regions are assigned. This patch adds 
a new jmx metric to expose the final time when the master has become the active 
master (All regions are assigned, etc.).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: [DISCUSS] hbase-2.0.0 compatibility expectations

[Esteban Gutierrez]

Should we add additional details around replication as well? for instance,
shall we consider a hbase-1.x cluster as a client for a hbase-2.x cluster?

Thanks for starting this discussion Stack,

esteban.

--
Cloudera, Inc.


On Fri, Aug 4, 2017 at 1:05 AM, stack  wrote:

> Thanks Zach for clarification. Let me work up a list and then come back to
> this thread.  Jira needs an edit pass to.
>
> S
>
> On Aug 3, 2017 23:54, "Zach York"  wrote:
>
> This kinda helps, but these seem more like expectations. I was going more
> for things like HFile format changed, meta table structure changed,
> coprocessor implementations changed (these are just examples, I don't know
> if any of these actually changed).
>
> More technical differences between branch-1 and branch-2 which then can
> help us get the right expectations for compatibility.
>
> On Wed, Aug 2, 2017 at 6:34 PM, Stack  wrote:
>
> > On Wed, Aug 2, 2017 at 5:25 PM, Zach York 
> > wrote:
> >
> > > Do we know what the major pain points for migration are? Can we discuss
> > > that/get a list going?
> > >
> > >
> > Here's a few in outline:
> >
> > + There is issue of formats, of hbase-2.x being able to read hbase-1.x
> data
> > whether from HDFS or ZooKeeper or off the wire.
> > + An hbase-1.x client should be able to Get/Put and Scan an hbase-2.x
> > cluster; no holes in the API or unintelligible serializations.
> > + There is then the little dance that has us rolling restart from an
> > hbase-1.x cluster to hbase-2.x; i.e. upgrade master first and then it
> will
> > assign regions to the new hbase-2.x regionservers as they come on line.
> > TBD.
> >
> > Is this what you mean sir?
> >
> > S
> >
> >
> > > I think without that knowledge it is hard (for me at least :) ) to
> > > determine where we should set our sights in terms of migration.
> > >
> > > Thanks,
> > > Zach
> > >
> > > On Wed, Aug 2, 2017 at 4:38 PM, Stack  wrote:
> > >
> > > > What are our expectations regards compatibility between hbase1 and
> > > hbase2?
> > > >
> > > > Lets have a chat about it. Here are some goal posts.
> > > >
> > > > + You have to upgrade to hbase-1.x before you can migrate to hbase-2.
> > No
> > > > migration from < hbase-1 (Is this too onerous? Should we support 0.98
> > =>
> > > > 2.0?).
> > > > + You do NOT have to upgrade to the latest release of hbase1 to
> migrate
> > > to
> > > > hbase2; being up on hbase-1.0.0+ will be sufficient.
> > > > + You'll have to update your hbase1 coprocessors to deploy them on
> > > hbase2.
> > > > A bunch of CP API has/will change by the time hbase2 comes out; e.g.
> > > > watching for region split on RegionServer no longer makes sense given
> > > > Master runs all splits now.
> > > > + An hbase1 client can run against an hbase2 cluster but it will only
> > be
> > > > able to do DML (Get/Put/Scan, etc.). We do not allow being able to do
> > > admin
> > > > ops using an hbase1 Admin client against an hbase2 cluster. We have
> > some
> > > > egregious API violations in branch-1; e.g. we have protobuf in our
> API
> > > (See
> > > > HBASE-15607). The notion is that we can't afford a deprecation cycle
> > > > purging this stuff from our Admin API.
> > > >
> > > > What you all think?
> > > >
> > > > St.Ack
> > > >
> > >
> >
>

[jira] [Created] (HBASE-18519) Use zero-copy strategy to optimize CellUtil.createCell

[Chia-Ping Tsai (JIRA)]

Chia-Ping Tsai created HBASE-18519:
--

 Summary: Use zero-copy strategy to optimize CellUtil.createCell
 Key: HBASE-18519
 URL: https://issues.apache.org/jira/browse/HBASE-18519
 Project: HBase
  Issue Type: Improvement
Reporter: Chia-Ping Tsai
Assignee: Chia-Ping Tsai


The IndividualBytesFieldCell, which is introduced by HBASE-14882, carries the 
input arguments without copying.  We can substitute IndividualBytesFieldCell 
for KeyValue used by CellUtil.createCell.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: [ANNOUNCE] Asynchronous client is available in HBase 2.0

[Guanghao Zhang]

@stack, sorry for reply late, sir. As HBASE-16388
 said, the client
threads may be blocked by only one slow region server. But a better
solution for this problem is to use async hbase client. And there are many
reasons to make a region server slow, slow read from hdfs, gc stw, etc. So
this should be a advantage of the async client. PS: my colleague Zheng Hu
talked about this problem on today's HBaseCon Asia :-). About the
performance of async client, there are two important metrics: latency and
QPS. Our PE tool can be used to test latency and it will be addressed by
HBASE-17994 . And YCSB
maybe better to test QPS. I will publish the test result when we get more
test data. Thanks.


2017-07-22 0:35 GMT+08:00 Andrew Purtell :

> This is exciting!
> Thanks so much for this hard work.
>
> On Fri, Jul 21, 2017 at 2:21 AM, Phil Yang  wrote:
>
> > I filed an issue HBASE-18334
> >  to remove the old
> sync
> > implementation. I think we can do it in 3.0 and keep sync logic in
> > branch-2? Of course we can also remove it in 2.1 or higher 2.x branch but
> > if we remove it since 2.0 it may be a little radical  :)
> >
> > Thanks,
> > Phil
> >
> >
> > 2017-07-21 6:15 GMT+08:00 Stack :
> >
> > > Very sweet Guanghao. Thanks for all the hard work (you, Duo and all who
> > > contributed to the async work).
> > >
> > > When you think we should move the sync client to be on top of the async
> > > chassis?
> > >
> > > Thanks,
> > > St.Ack
> > >
> > > P.S. It looks like you updated its status in this doc,
> > > https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9iEu_
> > > ktczrlKHK8N4SZzs/edit#,
> > > thanks.
> > >
> > > On Thu, Jul 20, 2017 at 4:51 AM, Guanghao Zhang 
> > > wrote:
> > >
> > > > Dear all,
> > > >
> > > > Asynchronous client aims to provide the ability to access HBase
> > > > asynchronously. You can obtain an AsyncConnection from
> > ConnectionFactory,
> > > > and then get an asynchronous table instance (for DML operations) or
> an
> > > > asynchronous admin instance (for DDL operations) from it to access
> > HBase.
> > > > For the asynchronous table or admin, most methods have the same
> meaning
> > > > with the old Table or Admin interface, expect that the return value
> is
> > > > wrapped with a CompletableFuture (java 8 or higher is required)
> > usually.
> > > > You can get more details from the documentation[1][2].
> > > >
> > > > There are two examples about the asynchronous client.
> > > AsyncClientExample[3]
> > > > is a simple example to show you how to use AsyncTable.
> > > HttpProxyExample[4]
> > > > is an example for advance user to show you how to use RawAsyncTable
> to
> > > > write a fully asynchronous HTTP proxy server. Welcome to try the
> > > > asynchronous client in your project.
> > > >
> > > > We tested the performance of asynchronous client by
> > > > the PerformanceEvaluation tool. And it showed that async client has
> > > almost
> > > > same latency with sync client[5]. Notice: when we did performance
> test
> > > for
> > > > asynchronous client. We found some performance issue when use jdk
> > > 1.8_111.
> > > > And it was fixed when use jdk 1.8_131 [6]. So suggest use a bigger
> > > version
> > > > than jdk8_131 when you use the asynchronous client.
> > > >
> > > > The major umbrella issues include:
> > > > HBASE-16432 Revisit the asynchronous ipc implementation[7]
> > > > HBASE-16833 Implement asynchronous hbase client based on
> HBASE-15921[8]
> > > > HBASE-17359 Implement async admin[9]
> > > >
> > > > There is also an issue HBASE-17856 to track the further improvements
> > for
> > > > asynchronous hbase client[10].
> > > >
> > > > Finally, thanks Duo Zhang and Zheng Hu who worked with me together to
> > > > implement the asynchronous client.
> > > >
> > > > Best Regards,
> > > > Guanghao
> > > >
> > > > [1] https://hbase.apache.org/book.html#async.client
> > > > [2] https://hbase.apache.org/book.html#async.admin
> > > > [3]
> > > > https://github.com/apache/hbase/blob/master/hbase-
> > > > examples/src/main/java/org/apache/hadoop/hbase/client/
> > > > example/AsyncClientExample.java
> > > > [4]
> > > > https://github.com/apache/hbase/blob/master/hbase-
> > > > examples/src/main/java/org/apache/hadoop/hbase/client/
> > > > example/HttpProxyExample.java
> > > > [5] https://www.slideshare.net/HBaseCon/hbase-at-xiaomi
> > > > [6]
> > > > http://hg.openjdk.java.net/jdk8/jdk8/jdk/file/
> 687fd7c7986d/src/share/
> > > > classes/java/util/concurrent/CompletableFuture.java
> > > > [7] https://issues.apache.org/jira/browse/HBASE-16432
> > > > [8] https://issues.apache.org/jira/browse/HBASE-16833
> > > > [9] https://issues.apache.org/jira/browse/HBASE-17359
> > > > [10] https://issues.apache.org/jira/browse/HBASE-17856
> > > >
> > >
> >
>
>
>
> --
> Best 

[jira] [Created] (HBASE-18518) Remove jersey1* dependencies form project and jersey1* jars from lib dir

[Samir Ahmic (JIRA)]

Samir Ahmic created HBASE-18518:
---

 Summary: Remove jersey1* dependencies form project and jersey1* 
jars from lib dir
 Key: HBASE-18518
 URL: https://issues.apache.org/jira/browse/HBASE-18518
 Project: HBase
  Issue Type: Task
  Components: dependencies, pom, REST
Affects Versions: 2.0.0-alpha-1, 3.0.0
Reporter: Samir Ahmic
Assignee: Samir Ahmic


Recently i have opened https://issues.apache.org/jira/browse/HBASE-18506 and it 
is clear that is caused by mixing jersey1 and jersey2 jars in classpath. With 
https://issues.apache.org/jira/browse/HBASE-12894 we have introduced jersey2 to 
project,  and we also  have bunch of transitive dependencies (mainly from 
hadoop) on jersey1 which is not happiest situation since jersey1 and jersey2 
under same classpath can case runtime issues as it was case with rest.
This task will have following steps
* Clean code and replace jersey1 constructs with jersey2 versions(there should 
not be much of this)
* Add exclusions for transitive jersey1 dependencies in pom.xml
* Add exclusions  in hadoop-two-compat.xml to prevent jersey1 jars in lib dir



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: [DISCUSS] Removing jersey1* dependencies from project and jersey1*jars form lib dir

[Samir Ahmic]

Ted, stack thanks,

Then let me turn on my cleaning equipment...:). I will make appropriate
JIRAs.

Cheers
Samir

On Fri, Aug 4, 2017 at 8:08 AM, stack  wrote:

> This sort of cleanup would be.much appreciated.
> Thanks,
> S
>
> On Aug 4, 2017 05:04, "Samir Ahmic"  wrote:
>
> > Hi all,
> >
> > Recently i have opened https://issues.apache.org/jira/browse/HBASE-18506
> > and it is clear that is caused by mixing jersey1 and jersey2 jars in
> > classpath. With https://issues.apache.org/jira/browse/HBASE-12894 we
> have
> > introduced jersey2 to project,  and we also  have bunch of transitive
> > dependencies (mainly from hadoop) on jersey1 which is not happiest
> > situation since jersey1 and jersey2 under same classpath can case runtime
> > issues as it was case with rest.
> > So if nobody objects i would like to remove hbase dependency on jersey1
> in
> > following steps:
> >
> >1. Clean code and replace jersey1 constructs with jersey2
> versions(there
> >should not be much of this)
> >2. Add exclusions for transitive jersey1 dependencies in pom.xml
> >3. Add exclusions  in hadoop-two-compat.xml to prevent jersey1 jars in
> >lib dir
> >
> > Is there any reason why we can not remove jersey1 from project ? Any
> > additional concerns ?
> >
> > Best Regards
> > Samir
> >
>

Bug in FIFOCompactionPolicy pre-checks?

[Lars George]

Hi,

See https://issues.apache.org/jira/browse/HBASE-14468

It adds this check to {{HMaster.checkCompactionPolicy()}}:

{code}
// 1. Check TTL
if (hcd.getTimeToLive() == HColumnDescriptor.DEFAULT_TTL) {
  message = "Default TTL is not supported for FIFO compaction";
  throw new IOException(message);
}

// 2. Check min versions
if (hcd.getMinVersions() > 0) {
  message = "MIN_VERSION > 0 is not supported for FIFO compaction";
  throw new IOException(message);
}

// 3. blocking file count
String sbfc = htd.getConfigurationValue(HStore.BLOCKING_STOREFILES_KEY);
if (sbfc != null) {
  blockingFileCount = Integer.parseInt(sbfc);
}
if (blockingFileCount < 1000) {
  message =
  "blocking file count '" + HStore.BLOCKING_STOREFILES_KEY + "' "
+ blockingFileCount
  + " is below recommended minimum of 1000";
  throw new IOException(message);
}
{code}

Why does it only check the blocking file count on the HTD level, while
others are check on the HCD level? Doing this for example fails
because of it:

{noformat}
hbase(main):008:0> create 'ttltable', { NAME => 'cf1', TTL => 300,
CONFIGURATION => { 'hbase.hstore.defaultengine.compactionpolicy.class'
=> 'org.apache.hadoop.hbase.regionserver.compactions.FIFOCompactionPolicy',
'hbase.hstore.blockingStoreFiles' => 2000 } }

ERROR: org.apache.hadoop.hbase.DoNotRetryIOException: blocking file
count 'hbase.hstore.blockingStoreFiles' 10 is below recommended
minimum of 1000 Set hbase.table.sanity.checks to false at conf or
table descriptor if you want to bypass sanity checks
at 
org.apache.hadoop.hbase.master.HMaster.warnOrThrowExceptionForFailure(HMaster.java:1782)
at 
org.apache.hadoop.hbase.master.HMaster.sanityCheckTableDescriptor(HMaster.java:1663)
at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1545)
at 
org.apache.hadoop.hbase.master.MasterRpcServices.createTable(MasterRpcServices.java:469)
at 
org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:58549)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2339)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168)
Caused by: java.io.IOException: blocking file count
'hbase.hstore.blockingStoreFiles' 10 is below recommended minimum of
1000
at 
org.apache.hadoop.hbase.master.HMaster.checkCompactionPolicy(HMaster.java:1773)
at 
org.apache.hadoop.hbase.master.HMaster.sanityCheckTableDescriptor(HMaster.java:1661)
... 7 more
{noformat}

That should work on the column family level, right? Shall I file a JIRA?

Cheers,
Lars

Re: [DISCUSS] Removing jersey1* dependencies from project and jersey1*jars form lib dir

[stack]

This sort of cleanup would be.much appreciated.
Thanks,
S

On Aug 4, 2017 05:04, "Samir Ahmic"  wrote:

> Hi all,
>
> Recently i have opened https://issues.apache.org/jira/browse/HBASE-18506
> and it is clear that is caused by mixing jersey1 and jersey2 jars in
> classpath. With https://issues.apache.org/jira/browse/HBASE-12894 we have
> introduced jersey2 to project,  and we also  have bunch of transitive
> dependencies (mainly from hadoop) on jersey1 which is not happiest
> situation since jersey1 and jersey2 under same classpath can case runtime
> issues as it was case with rest.
> So if nobody objects i would like to remove hbase dependency on jersey1 in
> following steps:
>
>1. Clean code and replace jersey1 constructs with jersey2 versions(there
>should not be much of this)
>2. Add exclusions for transitive jersey1 dependencies in pom.xml
>3. Add exclusions  in hadoop-two-compat.xml to prevent jersey1 jars in
>lib dir
>
> Is there any reason why we can not remove jersey1 from project ? Any
> additional concerns ?
>
> Best Regards
> Samir
>

Re: [DISCUSS] hbase-2.0.0 compatibility expectations

[stack]

Thanks Zach for clarification. Let me work up a list and then come back to
this thread.  Jira needs an edit pass to.

S

On Aug 3, 2017 23:54, "Zach York"  wrote:

This kinda helps, but these seem more like expectations. I was going more
for things like HFile format changed, meta table structure changed,
coprocessor implementations changed (these are just examples, I don't know
if any of these actually changed).

More technical differences between branch-1 and branch-2 which then can
help us get the right expectations for compatibility.

On Wed, Aug 2, 2017 at 6:34 PM, Stack  wrote:

> On Wed, Aug 2, 2017 at 5:25 PM, Zach York 
> wrote:
>
> > Do we know what the major pain points for migration are? Can we discuss
> > that/get a list going?
> >
> >
> Here's a few in outline:
>
> + There is issue of formats, of hbase-2.x being able to read hbase-1.x
data
> whether from HDFS or ZooKeeper or off the wire.
> + An hbase-1.x client should be able to Get/Put and Scan an hbase-2.x
> cluster; no holes in the API or unintelligible serializations.
> + There is then the little dance that has us rolling restart from an
> hbase-1.x cluster to hbase-2.x; i.e. upgrade master first and then it will
> assign regions to the new hbase-2.x regionservers as they come on line.
> TBD.
>
> Is this what you mean sir?
>
> S
>
>
> > I think without that knowledge it is hard (for me at least :) ) to
> > determine where we should set our sights in terms of migration.
> >
> > Thanks,
> > Zach
> >
> > On Wed, Aug 2, 2017 at 4:38 PM, Stack  wrote:
> >
> > > What are our expectations regards compatibility between hbase1 and
> > hbase2?
> > >
> > > Lets have a chat about it. Here are some goal posts.
> > >
> > > + You have to upgrade to hbase-1.x before you can migrate to hbase-2.
> No
> > > migration from < hbase-1 (Is this too onerous? Should we support 0.98
> =>
> > > 2.0?).
> > > + You do NOT have to upgrade to the latest release of hbase1 to
migrate
> > to
> > > hbase2; being up on hbase-1.0.0+ will be sufficient.
> > > + You'll have to update your hbase1 coprocessors to deploy them on
> > hbase2.
> > > A bunch of CP API has/will change by the time hbase2 comes out; e.g.
> > > watching for region split on RegionServer no longer makes sense given
> > > Master runs all splits now.
> > > + An hbase1 client can run against an hbase2 cluster but it will only
> be
> > > able to do DML (Get/Put/Scan, etc.). We do not allow being able to do
> > admin
> > > ops using an hbase1 Admin client against an hbase2 cluster. We have
> some
> > > egregious API violations in branch-1; e.g. we have protobuf in our API
> > (See
> > > HBASE-15607). The notion is that we can't afford a deprecation cycle
> > > purging this stuff from our Admin API.
> > >
> > > What you all think?
> > >
> > > St.Ack
> > >
> >
>

Re: [DISCUSS] Planning changes on RegionServer totalRequestCount metrics

[stack]

+1

We need a fat release note on this change so operators can quickly learn
why traffic went down on upgrade.

S

On Aug 3, 2017 14:49, "Yu Li"  wrote:

> Dear all,
>
> Recently in HBASE-18469  >
> we found some inconsistency on regionserver request related metrics,
> including:
> 1. totalRequestCount could be less than readRequestCount+writeRequestCount
> 2. For multi request, we count action count into totalRequestCount, while
> for scan with caching we count only one.
>
> To fix the inconsistency, we plan to make below changes:
> 1. Make totalRequestCount only counts rpc request, thus multi request will
> only count as one for totalRequestCount
> 2. Introduce a new metrics in name of "totalRowsRequestCount", which will
> count the DML workloads on RS by row-level action, and for this metrics we
> will count how many rows included for multi and scan-with-caching request.
>
> After the change, there won't be any compatibility issue -- existing
> monitoring system could still work -- only that totalRequestCount will be
> less than previous. And it's recommended to use totalRowsRequestCount to
> check the RS DML workload.
>
> Please kindly let us know if you have any different idea or suggestion
> (operators' opinion is especially welcomed).
>
> Let's make this discussion open for 72 hours and will make the change if no
> objections.
>
> Thanks!
>
> Best Regards,
> Yu
>