Re: [ANNOUNCE] New HBase committer Jingyun Tian

[Ted Yu]

Congratulations, Jingyun!
 Original message From: Srinivas Reddy 
 Date: 11/13/18  12:46 AM  (GMT-08:00) To: 
d...@hbase.apache.org Cc: Hbase-User  Subject: Re: 
[ANNOUNCE] New HBase committer Jingyun Tian Congratulations Jingyun-Srinivas- 
Typed on tiny keys. pls ignore typos.{mobile app}On Tue 13 Nov, 2018, 15:54 
张铎(Duo Zhang)  On behalf of the Apache HBase PMC, 
I am pleased to announce that Jingyun> Tian has accepted the PMC's invitation 
to become a committer on the> project. We appreciate all of Jingyun's generous 
contributions thus far and> look forward to his continued involvement.>> 
Congratulations and welcome, Jingyun!>

Re: [ANNOUNCE] New Committer: Balazs Meszaros

[Ted Yu]

Congratulations, Balazs.

On Thu, Oct 11, 2018 at 12:50 PM Mike Drob  wrote:

> Welcome, Balazs!
>
> On Thu, Oct 11, 2018 at 2:49 PM Sean Busbey  wrote:
>
> > On behalf of the HBase PMC, I'm pleased to announce that Balazs
> > Meszaros has accepted our invitation to become an HBase committer.
> >
> > Thanks for all your hard work Balazs; we look forward to more
> > contributions!
> >
> > Please join me in extending congratulations to Balazs!
> >
>

Re: Can MOB be enabled for existing table?

[Ted Yu]

bq. region server crashes after ttl archiver runs

This is the first time I hear of such problem. Can you give us more
information (stack trace, server log snippet for that period) ?

Thanks

On Thu, Sep 20, 2018 at 11:14 PM meirav.malka 
wrote:

> Hi,Since you probably don't have enough space for backup for such a large
> table, test it on a small one and rowcount before and after.We work
> extensively with mob and there are several issues with it that we had to
> work around.1.Mob and ttl - region server crashes after ttl archiver
> runs.2.Mob and snapshot - didn't work in cdh 5.7.13.Mob compaction-
> currently performed only by the master.if your flushes aren't big enough ,
> compaction will never end.4.Deletes - major mob compaction is the only one
> doing mob delete,it is also done by the master, on the entire data( it is
> not scheduled )If you have deletes, the will never actually be deleted.
> Regards,Meirav
> Sent from my Samsung Galaxy smartphone.
>  Original message From: Jean-Marc Spaggiari <
> jean-m...@spaggiari.org> Date: 9/21/18  01:09  (GMT+02:00) To: user <
> user@hbase.apache.org> Subject: Re: Can MOB be enabled for existing
> table?
> Hi Andrea,
>
> I will say that the easiest way to know is to try it... I'm guessing that
> it should work. When HBase will compact the file, it will figure that some
> fields are bigger than the configures MOB threshold and will move them
> under the MOB umbrella. But this has to be tested. Let us know.
>
> Thanks,
>
> JMS
>
> Le jeu. 20 sept. 2018 à 08:26, Andrea Fonti  a
> écrit :
>
> > Hi,
> >
> > I’m currently running HBASE on a Cloudera CDH 5.8.4 distribution
> > (hbase-1.2.0+cdh5.8.5+263), is it possible to enable MOB for a column of
> an
> > existing multi terabyte table?
> >
> > If the answer is affirmative what will hbase do with the existing data?
> > Will it rewrite with MOB files the next major compaction?
> >
> > Thank you
> >
> > Andrea
>

Re: Stripe Compactions Stability

[Ted Yu]

Hi,
To my knowledge, stripe compaction has not seen patches for a few years.

Have you looked at :
http://hbase.apache.org/book.html#ops.date.tiered

If the above doesn't suit your needs, can you tell us more about your use
case ?

Thanks

On Mon, Sep 17, 2018 at 11:39 AM Austin Heyne  wrote:

> The HBase cluster we're running has well over 100TB of data spread
> across two tables and as you'd guess we're suffering from compaction
> times taking way to long (and we need to double or triple that volume).
> I've found information on Stripe Compactions [1] which it seems like we
> could benefit a lot from, however, 'experimental' is a scary word for
> productions systems so I just wanted to get a general sentiment around
> the stability of the feature. Any experience or input would be very
> helpful.
>
> Thanks again,
> Austin
>
> [1] https://hbase.apache.org/book.html#ops.stripe
>
>

Re: Extremely high CPU usage after upgrading to Hbase 1.4.4

[Ted Yu]

In the previous stack trace you sent, shortCompactions and longCompactions
threads were not active.

Was the stack trace captured during period when the number of client
operations was low ?

If not, can you capture stack trace during off peak hours ?

Cheers

On Mon, Sep 10, 2018 at 12:08 PM Srinidhi Muppalla 
wrote:

> Hi Ted,
>
> The highest number of filters used is 10, but the average is generally
> close to 1. Is it possible the CPU usage spike has to do with Hbase
> internal maintenance operations? It looks like post-upgrade the spike isn’t
> correlated with the frequency of reads/writes we are making, because the
> high CPU usage persisted when the number of operations went down.
>
> Thank you,
> Srinidhi
>
> On 9/8/18, 9:44 AM, "Ted Yu"  wrote:
>
> Srinidhi :
> Do you know the average / highest number of ColumnPrefixFilter's in the
> FilterList ?
>
> Thanks
>
> On Fri, Sep 7, 2018 at 10:00 PM Ted Yu  wrote:
>
> > Thanks for detailed background information.
> >
> > I assume your code has done de-dup for the filters contained in
> > FilterListWithOR.
> >
> > I took a look at JIRAs which
> > touched hbase-client/src/main/java/org/apache/hadoop/hbase/filter in
> > branch-1.4
> > There were a few patches (some were very big) since the release of
> 1.3.0
> > So it is not obvious at first glance which one(s) might be related.
> >
> > I noticed ColumnPrefixFilter.getNextCellHint (and
> > KeyValueUtil.createFirstOnRow) appearing many times in the stack
> trace.
> >
> > I plan to dig more in this area.
> >
> > Cheers
> >
> > On Fri, Sep 7, 2018 at 11:30 AM Srinidhi Muppalla <
> srinid...@trulia.com>
> > wrote:
> >
> >> Sure thing. For our table schema, each row represents one user and
> the
> >> row key is that user’s unique id in our system. We currently only
> use one
> >> column family in the table. The column qualifiers represent an item
> that
> >> has been surfaced to that user as well as additional information to
> >> differentiate the way the item has been surfaced to the user.
> Without
> >> getting into too many specifics, the qualifier follows the rough
> format of:
> >>
> >> “Channel-itemId-distinguisher”.
> >>
> >> The channel here is the channel through the item was previously
> surfaced
> >> to the user. The itemid is the unique id of the item that has been
> surfaced
> >> to the user. A distinguisher is some attribute about how that item
> was
> >> surfaced to the user.
> >>
> >> When we run a scan, we currently only ever run it on one row at a
> time.
> >> It was chosen over ‘get’ because (from our understanding) the
> performance
> >> difference is negligible, and down the road using scan would allow
> us some
> >> more flexibility.
> >>
> >> The filter list that is constructed with scan works by using a
> >> ColumnPrefixFilter as you mentioned. When a user is being
> communicated to
> >> on a particular channel, we have a list of items that we want to
> >> potentially surface for that user. So, we construct a prefix list
> with the
> >> channel and each of the item ids in the form of: “channel-itemId”.
> Then we
> >> run a scan on that row with that filter list using “WithOr” to get
> all of
> >> the matching channel-itemId combinations currently in that
> row/column
> >> family in the table. This way we can then know which of the items
> we want
> >> to surface to that user on that channel have already been surfaced
> on that
> >> channel. The reason we query using a prefix filter is so that we
> don’t need
> >> to know the ‘distinguisher’ part of the record when writing the
> actual
> >> query, because the distinguisher is only relevant in certain
> circumstances.
> >>
> >> Let me know if this is the information about our query pattern that
> you
> >> were looking for and if there is anything I can clarify or add.
> >>
> >> Thanks,
> >> Srinidhi
> >>
> >> On 9/6/18, 12:24 PM, "Ted Yu"  wrote:
> >>
> >> From the stack trace, ColumnPrefixFilter is used during scan.
> >>
> >> Can you illustrate how various filters are formed thru

Re: Improving on MTTR of cluster [Hbase - 1.1.13]

[Ted Yu]

For the second config you mentioned, hbase.master.distributed.log.replay,
see http://hbase.apache.org/book.html#upgrade2.0.distributed.log.replay

FYI

On Mon, Sep 10, 2018 at 8:52 AM sahil aggarwal 
wrote:

> Hi,
>
> My cluster has around 50k regions and 130 RS. In case of unclean shutdown,
> the cluster take around 40 50 mins to come up(mostly slow on region
> assignment from observation). Trying to optimize it found following
> possible configs:
>
> *hbase.assignment.usezk:* which will co-host meta table and Hmaster and
> avoid zk interaction for region assignment.
> *hbase.master.distributed.log.replay:* to replay the edit logs in
> distributed manner.
>
>
> Testing *hbase.assignment.usezk* alone on small cluster(2200 regions, 4 RS)
> gave following results:
>
> hbase.assignment.usezk=true -> 12 mins
> hbase.assignment.usezk=false -> 9 mins
>
>
> From this blog
> , i
> was expecting better results so probably I am missing something. Will
> appreciate any pointers.
>
> Thanks,
> Sahil
>

Re: Extremely high CPU usage after upgrading to Hbase 1.4.4

[Ted Yu]

Srinidhi :
Do you know the average / highest number of ColumnPrefixFilter's in the
FilterList ?

Thanks

On Fri, Sep 7, 2018 at 10:00 PM Ted Yu  wrote:

> Thanks for detailed background information.
>
> I assume your code has done de-dup for the filters contained in
> FilterListWithOR.
>
> I took a look at JIRAs which
> touched hbase-client/src/main/java/org/apache/hadoop/hbase/filter in
> branch-1.4
> There were a few patches (some were very big) since the release of 1.3.0
> So it is not obvious at first glance which one(s) might be related.
>
> I noticed ColumnPrefixFilter.getNextCellHint (and
> KeyValueUtil.createFirstOnRow) appearing many times in the stack trace.
>
> I plan to dig more in this area.
>
> Cheers
>
> On Fri, Sep 7, 2018 at 11:30 AM Srinidhi Muppalla 
> wrote:
>
>> Sure thing. For our table schema, each row represents one user and the
>> row key is that user’s unique id in our system. We currently only use one
>> column family in the table. The column qualifiers represent an item that
>> has been surfaced to that user as well as additional information to
>> differentiate the way the item has been surfaced to the user. Without
>> getting into too many specifics, the qualifier follows the rough format of:
>>
>> “Channel-itemId-distinguisher”.
>>
>> The channel here is the channel through the item was previously surfaced
>> to the user. The itemid is the unique id of the item that has been surfaced
>> to the user. A distinguisher is some attribute about how that item was
>> surfaced to the user.
>>
>> When we run a scan, we currently only ever run it on one row at a time.
>> It was chosen over ‘get’ because (from our understanding) the performance
>> difference is negligible, and down the road using scan would allow us some
>> more flexibility.
>>
>> The filter list that is constructed with scan works by using a
>> ColumnPrefixFilter as you mentioned. When a user is being communicated to
>> on a particular channel, we have a list of items that we want to
>> potentially surface for that user. So, we construct a prefix list with the
>> channel and each of the item ids in the form of: “channel-itemId”. Then we
>> run a scan on that row with that filter list using “WithOr” to get all of
>> the matching channel-itemId combinations currently in that row/column
>> family in the table. This way we can then know which of the items we want
>> to surface to that user on that channel have already been surfaced on that
>> channel. The reason we query using a prefix filter is so that we don’t need
>> to know the ‘distinguisher’ part of the record when writing the actual
>> query, because the distinguisher is only relevant in certain circumstances.
>>
>> Let me know if this is the information about our query pattern that you
>> were looking for and if there is anything I can clarify or add.
>>
>> Thanks,
>> Srinidhi
>>
>> On 9/6/18, 12:24 PM, "Ted Yu"  wrote:
>>
>> From the stack trace, ColumnPrefixFilter is used during scan.
>>
>> Can you illustrate how various filters are formed thru
>> FilterListWithOR ?
>> It would be easier for other people to reproduce the problem given
>> your
>> query pattern.
>>
>> Cheers
>>
>> On Thu, Sep 6, 2018 at 11:43 AM Srinidhi Muppalla <
>> srinid...@trulia.com>
>> wrote:
>>
>> > Hi Vlad,
>> >
>> > Thank you for the suggestion. I recreated the issue and attached
>> the stack
>> > traces I took. Let me know if there’s any other info I can provide.
>> We
>> > narrowed the issue down to occurring when upgrading from 1.3.0 to
>> any 1.4.x
>> > version.
>> >
>> > Thanks,
>> > Srinidhi
>> >
>> > On 9/4/18, 8:19 PM, "Vladimir Rodionov" 
>> wrote:
>> >
>> > Hi, Srinidhi
>> >
>> > Next time you will see this issue, take jstack of a RS several
>> times
>> > in a
>> > row. W/o stack traces it is hard
>> > to tell what was going on with your cluster after upgrade.
>> >
>> > -Vlad
>> >
>> >
>> >
>> > On Tue, Sep 4, 2018 at 3:50 PM Srinidhi Muppalla <
>> srinid...@trulia.com
>> > >
>> > wrote:
>> >
>> > > Hello all,
>> > >
>> > > We are currently running Hbase 1.3.0

Re: Extremely high CPU usage after upgrading to Hbase 1.4.4

[Ted Yu]

The createFirstOnRow() is used by ColumnXXFilter's getNextCellHint() method.
I am thinking about adding a variant to getNextCellHint() which returns a
tuple, representing first on row, consisting of:
  Cell - the passed in Cell instance
  byte[] - qualifier array
  int - qualifier offset
  int - qualifier length
This variant doesn't allocate (new) Cell / KeyValue.

This way, FilterListWithOR#shouldPassCurrentCellToFilter can use the
returned tuple for comparison.

FYI

On Fri, Sep 7, 2018 at 10:00 PM Ted Yu  wrote:

> Thanks for detailed background information.
>
> I assume your code has done de-dup for the filters contained in
> FilterListWithOR.
>
> I took a look at JIRAs which
> touched hbase-client/src/main/java/org/apache/hadoop/hbase/filter in
> branch-1.4
> There were a few patches (some were very big) since the release of 1.3.0
> So it is not obvious at first glance which one(s) might be related.
>
> I noticed ColumnPrefixFilter.getNextCellHint (and
> KeyValueUtil.createFirstOnRow) appearing many times in the stack trace.
>
> I plan to dig more in this area.
>
> Cheers
>
> On Fri, Sep 7, 2018 at 11:30 AM Srinidhi Muppalla 
> wrote:
>
>> Sure thing. For our table schema, each row represents one user and the
>> row key is that user’s unique id in our system. We currently only use one
>> column family in the table. The column qualifiers represent an item that
>> has been surfaced to that user as well as additional information to
>> differentiate the way the item has been surfaced to the user. Without
>> getting into too many specifics, the qualifier follows the rough format of:
>>
>> “Channel-itemId-distinguisher”.
>>
>> The channel here is the channel through the item was previously surfaced
>> to the user. The itemid is the unique id of the item that has been surfaced
>> to the user. A distinguisher is some attribute about how that item was
>> surfaced to the user.
>>
>> When we run a scan, we currently only ever run it on one row at a time.
>> It was chosen over ‘get’ because (from our understanding) the performance
>> difference is negligible, and down the road using scan would allow us some
>> more flexibility.
>>
>> The filter list that is constructed with scan works by using a
>> ColumnPrefixFilter as you mentioned. When a user is being communicated to
>> on a particular channel, we have a list of items that we want to
>> potentially surface for that user. So, we construct a prefix list with the
>> channel and each of the item ids in the form of: “channel-itemId”. Then we
>> run a scan on that row with that filter list using “WithOr” to get all of
>> the matching channel-itemId combinations currently in that row/column
>> family in the table. This way we can then know which of the items we want
>> to surface to that user on that channel have already been surfaced on that
>> channel. The reason we query using a prefix filter is so that we don’t need
>> to know the ‘distinguisher’ part of the record when writing the actual
>> query, because the distinguisher is only relevant in certain circumstances.
>>
>> Let me know if this is the information about our query pattern that you
>> were looking for and if there is anything I can clarify or add.
>>
>> Thanks,
>> Srinidhi
>>
>> On 9/6/18, 12:24 PM, "Ted Yu"  wrote:
>>
>> From the stack trace, ColumnPrefixFilter is used during scan.
>>
>> Can you illustrate how various filters are formed thru
>> FilterListWithOR ?
>> It would be easier for other people to reproduce the problem given
>> your
>> query pattern.
>>
>> Cheers
>>
>> On Thu, Sep 6, 2018 at 11:43 AM Srinidhi Muppalla <
>> srinid...@trulia.com>
>> wrote:
>>
>> > Hi Vlad,
>> >
>> > Thank you for the suggestion. I recreated the issue and attached
>> the stack
>> > traces I took. Let me know if there’s any other info I can provide.
>> We
>> > narrowed the issue down to occurring when upgrading from 1.3.0 to
>> any 1.4.x
>> > version.
>> >
>> > Thanks,
>> > Srinidhi
>> >
>> > On 9/4/18, 8:19 PM, "Vladimir Rodionov" 
>> wrote:
>> >
>> > Hi, Srinidhi
>> >
>> > Next time you will see this issue, take jstack of a RS several
>> times
>> > in a
>> > row. W/o stack traces it is hard
>> > to tell what was going on with your cluster after upgrade.
>> >
>> > -Vlad

Re: Extremely high CPU usage after upgrading to Hbase 1.4.4

[Ted Yu]

Thanks for detailed background information.

I assume your code has done de-dup for the filters contained in
FilterListWithOR.

I took a look at JIRAs which
touched hbase-client/src/main/java/org/apache/hadoop/hbase/filter in
branch-1.4
There were a few patches (some were very big) since the release of 1.3.0
So it is not obvious at first glance which one(s) might be related.

I noticed ColumnPrefixFilter.getNextCellHint (and
KeyValueUtil.createFirstOnRow) appearing many times in the stack trace.

I plan to dig more in this area.

Cheers

On Fri, Sep 7, 2018 at 11:30 AM Srinidhi Muppalla 
wrote:

> Sure thing. For our table schema, each row represents one user and the row
> key is that user’s unique id in our system. We currently only use one
> column family in the table. The column qualifiers represent an item that
> has been surfaced to that user as well as additional information to
> differentiate the way the item has been surfaced to the user. Without
> getting into too many specifics, the qualifier follows the rough format of:
>
> “Channel-itemId-distinguisher”.
>
> The channel here is the channel through the item was previously surfaced
> to the user. The itemid is the unique id of the item that has been surfaced
> to the user. A distinguisher is some attribute about how that item was
> surfaced to the user.
>
> When we run a scan, we currently only ever run it on one row at a time. It
> was chosen over ‘get’ because (from our understanding) the performance
> difference is negligible, and down the road using scan would allow us some
> more flexibility.
>
> The filter list that is constructed with scan works by using a
> ColumnPrefixFilter as you mentioned. When a user is being communicated to
> on a particular channel, we have a list of items that we want to
> potentially surface for that user. So, we construct a prefix list with the
> channel and each of the item ids in the form of: “channel-itemId”. Then we
> run a scan on that row with that filter list using “WithOr” to get all of
> the matching channel-itemId combinations currently in that row/column
> family in the table. This way we can then know which of the items we want
> to surface to that user on that channel have already been surfaced on that
> channel. The reason we query using a prefix filter is so that we don’t need
> to know the ‘distinguisher’ part of the record when writing the actual
> query, because the distinguisher is only relevant in certain circumstances.
>
> Let me know if this is the information about our query pattern that you
> were looking for and if there is anything I can clarify or add.
>
> Thanks,
> Srinidhi
>
> On 9/6/18, 12:24 PM, "Ted Yu"  wrote:
>
> From the stack trace, ColumnPrefixFilter is used during scan.
>
> Can you illustrate how various filters are formed thru
> FilterListWithOR ?
> It would be easier for other people to reproduce the problem given your
> query pattern.
>
> Cheers
>
> On Thu, Sep 6, 2018 at 11:43 AM Srinidhi Muppalla <
> srinid...@trulia.com>
> wrote:
>
> > Hi Vlad,
> >
> > Thank you for the suggestion. I recreated the issue and attached the
> stack
> > traces I took. Let me know if there’s any other info I can provide.
> We
> > narrowed the issue down to occurring when upgrading from 1.3.0 to
> any 1.4.x
> > version.
> >
> > Thanks,
> > Srinidhi
> >
> > On 9/4/18, 8:19 PM, "Vladimir Rodionov" 
> wrote:
> >
> > Hi, Srinidhi
> >
> > Next time you will see this issue, take jstack of a RS several
> times
> > in a
> > row. W/o stack traces it is hard
> > to tell what was going on with your cluster after upgrade.
> >
> > -Vlad
> >
> >
> >
> > On Tue, Sep 4, 2018 at 3:50 PM Srinidhi Muppalla <
> srinid...@trulia.com
> > >
> > wrote:
> >
> > > Hello all,
> > >
> > > We are currently running Hbase 1.3.0 on an EMR cluster running
> EMR
> > 5.5.0.
> > > Recently, we attempted to upgrade our cluster to using Hbase
> 1.4.4
> > (along
> > > with upgrading our EMR cluster to 5.16). After upgrading, the
> CPU
> > usage for
> > > all of our region servers spiked up to 90%. The load_one for
> all of
> > our
> > > servers spiked from roughly 1-2 to 10 threads. After
> upgrading, the
> > number
> > > of operations to the cluster hasn’t increased. After giving the
> > cluster a
> > > few hours, we had to revert the upgrade. From the logs, we are
> > unable to
> > > tell what is occupying the CPU resources. Is this a known
> issue with
> > 1.4.4?
> > > Any guidance or ideas for debugging the cause would be greatly
> > > appreciated.  What are the best steps for debugging CPU usage?
> > >
> > > Thank you,
> > > Srinidhi
> > >
> >
> >
> >
>
>
>

Re: Extremely high CPU usage after upgrading to Hbase 1.4.4

[Ted Yu]

>From the stack trace, ColumnPrefixFilter is used during scan.

Can you illustrate how various filters are formed thru FilterListWithOR ?
It would be easier for other people to reproduce the problem given your
query pattern.

Cheers

On Thu, Sep 6, 2018 at 11:43 AM Srinidhi Muppalla 
wrote:

> Hi Vlad,
>
> Thank you for the suggestion. I recreated the issue and attached the stack
> traces I took. Let me know if there’s any other info I can provide. We
> narrowed the issue down to occurring when upgrading from 1.3.0 to any 1.4.x
> version.
>
> Thanks,
> Srinidhi
>
> On 9/4/18, 8:19 PM, "Vladimir Rodionov"  wrote:
>
> Hi, Srinidhi
>
> Next time you will see this issue, take jstack of a RS several times
> in a
> row. W/o stack traces it is hard
> to tell what was going on with your cluster after upgrade.
>
> -Vlad
>
>
>
> On Tue, Sep 4, 2018 at 3:50 PM Srinidhi Muppalla  >
> wrote:
>
> > Hello all,
> >
> > We are currently running Hbase 1.3.0 on an EMR cluster running EMR
> 5.5.0.
> > Recently, we attempted to upgrade our cluster to using Hbase 1.4.4
> (along
> > with upgrading our EMR cluster to 5.16). After upgrading, the CPU
> usage for
> > all of our region servers spiked up to 90%. The load_one for all of
> our
> > servers spiked from roughly 1-2 to 10 threads. After upgrading, the
> number
> > of operations to the cluster hasn’t increased. After giving the
> cluster a
> > few hours, we had to revert the upgrade. From the logs, we are
> unable to
> > tell what is occupying the CPU resources. Is this a known issue with
> 1.4.4?
> > Any guidance or ideas for debugging the cause would be greatly
> > appreciated.  What are the best steps for debugging CPU usage?
> >
> > Thank you,
> > Srinidhi
> >
>
>
>

Re: a table is neither disable or enable

[Ted Yu]

I doubt the procedure would finish, considering it has run for so long.

You can check the tail of master log to see if it is stuck.
If it is stuck, see if you can use abort_procedure.rb to stop.

After the procedure is stopped, see if running hbck can fix the issue (I
haven't worked with 1.3 release in production).
When running hbck, run without -fix parameter first to see what
inconsistencies hbck reports.

Cheers

On Wed, Aug 29, 2018 at 3:42 PM Antonio Si  wrote:

> Forgot to mention that all regions of the table is offline now. Wondering
> if the table will eventually got disable as it has been running for almost
> 24 hrs now.
>
> Thanks.
>
> Antonio.
>
> On Wed, Aug 29, 2018 at 3:40 PM Antonio Si  wrote:
>
> > Thanks Ted.
> > Now that the table is in neither disable or enable state, will the table
> > eventually got disable completely?
> > From the "Procedure" tab of the hbase ui, I see the "disable" is still
> > running.
> >
> > Thanks.
> >
> > Antonio.
> >
> > On Wed, Aug 29, 2018 at 3:31 PM Ted Yu  wrote:
> >
> >> The 'missing table descriptor' error should have been fixed by running
> >> hbck
> >> (with selected parameters).
> >>
> >> FYI
> >>
> >> On Wed, Aug 29, 2018 at 2:46 PM Antonio Si 
> wrote:
> >>
> >> > Thanks Ted.
> >> >
> >> > The log says "java.io.IOException: missing table descriptor for
> >> > ba912582f295f7ac0b83e7e419351602
> >> >
> >> > [AM.ZK.Worker-pool2-t6552] master.RegionStates: Failed to open/close
> >> > ba912582f295f7ac0b83e7e419351602  set to FAILED_OPEN"
> >> >
> >> >
> >> > The version of hbase is 1.3.1
> >> >
> >> >
> >> > Thanks.
> >> >
> >> >
> >> > Antonio.
> >> >
> >> > On Wed, Aug 29, 2018 at 2:28 PM Ted Yu  wrote:
> >> >
> >> > > Do you have access to master / region logs for when FAILED_OPEN
> state
> >> was
> >> > > noticed ?
> >> > >
> >> > > There should be some hint there as to why some region couldn't open.
> >> > >
> >> > > The length of table DDL is related to number of regions the table
> has.
> >> > But
> >> > > the length should be less related to data amount.
> >> > >
> >> > > Which version of hbase are you using ?
> >> > >
> >> > > Thanks
> >> > >
> >> > > On Wed, Aug 29, 2018 at 2:22 PM Antonio Si 
> >> wrote:
> >> > >
> >> > > > Hi,
> >> > > >
> >> > > > We have a table which is stuck in FAILED_OPEN state. So, we
> planned
> >> to
> >> > > drop
> >> > > > the table and re-clone the table from an old snapshot. We disabled
> >> the
> >> > > > table, but the disable procedure has been running for more than 20
> >> hrs.
> >> > > >
> >> > > > I went to hbase shell and found out "is_disabled" and "is_enabled"
> >> both
> >> > > > return false. Is that a normal behavior since the table is in the
> >> > middle
> >> > > of
> >> > > > being disabled?
> >> > > >
> >> > > > Is it normal that the disable took that many hours even though the
> >> > table
> >> > > is
> >> > > > large in size (about 33TB)?
> >> > > >
> >> > > > Thanks.
> >> > > >
> >> > > > Antonio.
> >> > > >
> >> > >
> >> >
> >>
> >
>

Re: a table is neither disable or enable

[Ted Yu]

The 'missing table descriptor' error should have been fixed by running hbck
(with selected parameters).

FYI

On Wed, Aug 29, 2018 at 2:46 PM Antonio Si  wrote:

> Thanks Ted.
>
> The log says "java.io.IOException: missing table descriptor for
> ba912582f295f7ac0b83e7e419351602
>
> [AM.ZK.Worker-pool2-t6552] master.RegionStates: Failed to open/close
> ba912582f295f7ac0b83e7e419351602  set to FAILED_OPEN"
>
>
> The version of hbase is 1.3.1
>
>
> Thanks.
>
>
> Antonio.
>
> On Wed, Aug 29, 2018 at 2:28 PM Ted Yu  wrote:
>
> > Do you have access to master / region logs for when FAILED_OPEN state was
> > noticed ?
> >
> > There should be some hint there as to why some region couldn't open.
> >
> > The length of table DDL is related to number of regions the table has.
> But
> > the length should be less related to data amount.
> >
> > Which version of hbase are you using ?
> >
> > Thanks
> >
> > On Wed, Aug 29, 2018 at 2:22 PM Antonio Si  wrote:
> >
> > > Hi,
> > >
> > > We have a table which is stuck in FAILED_OPEN state. So, we planned to
> > drop
> > > the table and re-clone the table from an old snapshot. We disabled the
> > > table, but the disable procedure has been running for more than 20 hrs.
> > >
> > > I went to hbase shell and found out "is_disabled" and "is_enabled" both
> > > return false. Is that a normal behavior since the table is in the
> middle
> > of
> > > being disabled?
> > >
> > > Is it normal that the disable took that many hours even though the
> table
> > is
> > > large in size (about 33TB)?
> > >
> > > Thanks.
> > >
> > > Antonio.
> > >
> >
>

Re: a table is neither disable or enable

[Ted Yu]

Do you have access to master / region logs for when FAILED_OPEN state was
noticed ?

There should be some hint there as to why some region couldn't open.

The length of table DDL is related to number of regions the table has. But
the length should be less related to data amount.

Which version of hbase are you using ?

Thanks

On Wed, Aug 29, 2018 at 2:22 PM Antonio Si  wrote:

> Hi,
>
> We have a table which is stuck in FAILED_OPEN state. So, we planned to drop
> the table and re-clone the table from an old snapshot. We disabled the
> table, but the disable procedure has been running for more than 20 hrs.
>
> I went to hbase shell and found out "is_disabled" and "is_enabled" both
> return false. Is that a normal behavior since the table is in the middle of
> being disabled?
>
> Is it normal that the disable took that many hours even though the table is
> large in size (about 33TB)?
>
> Thanks.
>
> Antonio.
>

Re: question on reducing number of versions

[Ted Yu]

This depends on how far down you revise the max versions for table t2.
If your data normally only reaches 15000 versions and you lower max
versions to ~15000, there wouldn't be much saving.

FYI

On Sun, Aug 26, 2018 at 3:52 PM Antonio Si  wrote:

> Thanks Anil.
>
> We are using hbase on s3. Yes, I understand 18000 is very high. We are in
> the process of reducing it.
>
> If I have a snapshot and I restore the table from this snapshot. Let's call
> this table t1.
> I then clone another table from the same snapshot, call it t2.
>
> If I reduce the max versions of t2 and run a major compaction on t2, will I
> see the decrease in table size for t2? If I compare the size of t2 and t1,
> I should see a smaller size for t2?
>
> Thanks.
>
> Antonio.
>
> On Sun, Aug 26, 2018 at 3:33 PM Anil Gupta  wrote:
>
> > You will need to do major compaction on table for the table to
> > clean/delete up extra version.
> > Btw, 18000 max version is a unusually high value.
> >
> > Are you using hbase on s3 or hbase on hdfs?
> >
> > Sent from my iPhone
> >
> > > On Aug 26, 2018, at 2:34 PM, Antonio Si  wrote:
> > >
> > > Hello,
> > >
> > > I have a hbase table whose definition has a max number of versions set
> to
> > > 36000.
> > > I have verified that there are rows which have more than 2 versions
> > > saved.
> > >
> > > Now, I change the definition of the table and reduce the max number of
> > > versions to 18000. Will I see the size of the table being reduced as I
> am
> > > not seeing that?
> > >
> > > Also, after I reduce the max number of versions, I try to create a
> > > snapshot, but I am getting a
> > > com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.services.s3.mo
> > >
> > > del.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404;
> > > Error Code: 404 Not Found;
> > >
> > >
> > > What may be the cause of that?
> > >
> > > I am using s3 as my storage.
> > >
> > >
> > > Thanks in advance for your suggestions.
> > >
> > >
> > > Antonio.
> >
>

Re: time out when running CellCounter

[Ted Yu]

Please also take a look at:
hbase-examples/src/main//java/org/apache/hadoop/hbase/coprocessor/example/RowCountEndpoint.java

You can reuse some code from CellCounter to enhance the above example
endpoint so that counting versions is done on server side instead of
through mapreduce.

FYI

On Sat, Aug 25, 2018 at 2:49 PM Antonio Si  wrote:

> Thanks Ted.
>
> I try passing "-Dhbase.client.scanner.timeout.period=180" when I invoke
> CellCounter, but it is still saying timeout after 600 sec.
>
> Thanks.
>
> Antonio.
>
> On Sat, Aug 25, 2018 at 2:09 PM Ted Yu  wrote:
>
> > It seems CellCounter doesn't have such (commandline) option.
> >
> > You can specify, e.g. scan timerange, scan max versions, start row, stop
> > row, etc. so that individual run has shorter runtime.
> >
> > Cheers
> >
> > On Sat, Aug 25, 2018 at 9:35 AM Antonio Si  wrote:
> >
> > > Hi,
> > >
> > > When I run  org.apache.hadoop.hbase.mapreduce.*CellCounter*, I am
> getting
> > > Timed
> > > out after 600 secs. Is there a way to override the timeout value rather
> > > than changing it in hbase-site.xml and restart hbase?
> > >
> > > Any suggestions would be helpful.
> > >
> > > Thank you.
> > >
> > > Antonio.
> > >
> >
>

Re: time out when running CellCounter

[Ted Yu]

It seems CellCounter doesn't have such (commandline) option.

You can specify, e.g. scan timerange, scan max versions, start row, stop
row, etc. so that individual run has shorter runtime.

Cheers

On Sat, Aug 25, 2018 at 9:35 AM Antonio Si  wrote:

> Hi,
>
> When I run  org.apache.hadoop.hbase.mapreduce.*CellCounter*, I am getting
> Timed
> out after 600 secs. Is there a way to override the timeout value rather
> than changing it in hbase-site.xml and restart hbase?
>
> Any suggestions would be helpful.
>
> Thank you.
>
> Antonio.
>

Re: how to get rowkey with largest number of versions

[Ted Yu]

Antonio:
Please take a look at CellCounter under hbase-mapreduce module which may be
of use to you:

 * 6. Total number of versions of each qualifier.


Please note that the max versions may fluctuate depending on when major
compaction kicks in.


FYI

On Wed, Aug 22, 2018 at 11:53 AM Ankit Singhal 
wrote:

> I don't think so if there is any direct way.
> You may need to do a raw scan of a full table and count the number of
> versions of a column returned for each row to calculate the max. (you can
> optimize this with custom coprocessor by returning a single row key having
> the largest versions of a column through each regionserver and at client
> select max out of all results)
>
> On Wed, Aug 22, 2018 at 11:28 AM Antonio Si  wrote:
>
> > Hi,
> >
> > I am new to hbase. I am wondering how I could find out which rowkey has
> the
> > largest number of versions in a column family.
> >
> > Any pointer would be very helpful.
> >
> > Thanks.
> >
> > Antonio.
> >
>

Re: build hbase2.1 without htrace-core-3.1.0-incubating.jar

[Ted Yu]

The previous snippet was generated on hbase-2.0 branch.

Regenerating the dependency tree out with hbase-2.1 branch, I don't see
3.2.0 of htrace at all.

FYI

On Mon, Aug 20, 2018 at 6:32 PM Sean Busbey  wrote:

> That does not look like an Apache HBase version number on
> hbase-server.  陈叶超  even said "HBase 2.1" in the original email.
>
>
>
>
> On Mon, Aug 20, 2018 at 2:17 PM, Ted Yu  wrote:
> > Looking at the dependency tree output, I see the following:
> >
> > [INFO] org.apache.hbase:hbase-server:jar:2.0.0.3.0.0.0-SNAPSHOT
> > ...
> > [INFO] +- org.apache.htrace:htrace-core:jar:3.2.0-incubating:compile
> >
> > FYI
> >
> > On Mon, Aug 20, 2018 at 8:10 AM Sean Busbey  wrote:
> >
> >> neither Hadoop 3.1 nor HBase 2.1 use that version of HTrace. what are
> >> you trying to do?
> >>
> >> On Sun, Aug 19, 2018 at 11:24 PM, 陈叶超 
> >> wrote:
> >> > hi:
> >> >
> >> >
> >> > I build hbase 2.1 with hadoop 3.1,and will lost
> >> htrace-core-3.1.0-incubating.jar in \lib\client-facing-thirdparty\
> >> >
> >> > but i found in the apache site download tar.gz ,
> >> >
> >> > my build command is : "mvn  -P build-with-jdk8,hadoop-3.0
> >> -DskipTests=true clean package install  assembly:single"
> >> >
> >> > what am i missing ?
> >> >
> >> >
> >> >
> >> >
> >> > 陈叶超 Yechao Chen
> >> > 中移（苏州）软件技术有限公司|中国移动苏州研发中心 | 大数据产品部
> >> > China Mobile (Suzhou) Software Technology Co., Ltd.
> >> > Mobile: (+86) 18896724791
> >> > Email: chenyec...@cmss.chinamobile.com
> >>
>

Re: build hbase2.1 without htrace-core-3.1.0-incubating.jar

[Ted Yu]

Looking at the dependency tree output, I see the following:

[INFO] org.apache.hbase:hbase-server:jar:2.0.0.3.0.0.0-SNAPSHOT
...
[INFO] +- org.apache.htrace:htrace-core:jar:3.2.0-incubating:compile

FYI

On Mon, Aug 20, 2018 at 8:10 AM Sean Busbey  wrote:

> neither Hadoop 3.1 nor HBase 2.1 use that version of HTrace. what are
> you trying to do?
>
> On Sun, Aug 19, 2018 at 11:24 PM, 陈叶超 
> wrote:
> > hi:
> >
> >
> > I build hbase 2.1 with hadoop 3.1,and will lost
> htrace-core-3.1.0-incubating.jar in \lib\client-facing-thirdparty\
> >
> > but i found in the apache site download tar.gz ,
> >
> > my build command is : "mvn  -P build-with-jdk8,hadoop-3.0
> -DskipTests=true clean package install  assembly:single"
> >
> > what am i missing ?
> >
> >
> >
> >
> > 陈叶超 Yechao Chen
> > 中移（苏州）软件技术有限公司|中国移动苏州研发中心 | 大数据产品部
> > China Mobile (Suzhou) Software Technology Co., Ltd.
> > Mobile: (+86) 18896724791
> > Email: chenyec...@cmss.chinamobile.com
>

Re: difference - filterlist with rowfilters vs multiget

[Ted Yu]

For the first code snippet, it seems start row /  stop row can be specified
to narrow the range of scan.

For your last question:

bq. all the filters in the filterList are RowFilters ...

The filter list currently is not doing optimization even if all filters in
the list are RowFilters.

Cheers

On Wed, Aug 15, 2018 at 2:07 AM Biplob Biswas 
wrote:

> Hi,
>
> During our implementation for fetching multiple records from an HBase
> table, we came across a discussion regarding the best way to get records
> out.
>
> The first implementation is something like:
>
>   FilterList filterList = new FilterList(Operator.MUST_PASS_ONE);
> >   for (String rowKey : rowKeys) {
> > filterList.addFilter(new RowFilter(CompareOp.EQUAL,new
> > BinaryComparator(Bytes.toBytes(rowKey;
> >   }
> >
> >   Scan scan = new Scan();
> >   scan.setFilter(filterList);
> >   ResultScanner resultScanner = table.getScanner(scan);
>
>
> and the second implementation is somethign like this:
>
>  List listGet = rowKeys.stream()
> >   .map(entry -> {
> > Get get = new Get(Bytes.toBytes(entry));
> > return get;
> >   })
> >   .collect(Collectors.toList());
> >   Result[] results = table.get(listGet)
>
>
> The only difference I see directly is that filterList would do a full table
> scan whereas multiget wouldn't do anything as such.
>
> But what other benefits one has over the other? Also, when HBase finds out
> that all the filters in the filterList are RowFilters, would it perform
> some kind of optimization and perform multiget rather than doing a full
> table scan?
>
>
> Thanks & Regards
> Biplob Biswas
>

Re: HBase 2.1.0 - RUNNABLE:SERVER_CRASH_START; ServerCrashProcedure

[Ted Yu]

jcl:
I can see that DEBUG log wasn't turned on.

Can you set log4j to DEBUG level and see if there is more information ?

Cheers

On Tue, Aug 14, 2018 at 6:56 AM Allan Yang  wrote:

> Those log is not enough to locate the problem.
> Best Regards
> Allan Yang
>
>
> jcl <515951...@163.com> 于2018年8月14日周二 下午9:18写道：
>
> > Power off and restart(Hadoop and HBase), Master is initializing - Hbase
> > ServerManager: but crash processing already in progress
> >
> > command jps, HMaster and HRegionServer is live
> >
> >
> > WARN  [Thread-14] master.ServerManager: Expiration called on
> > hbase-115,16020,1534248994825 but crash processing already in progress
> > WARN  [Thread-14] master.ServerManager: Expiration called on
> > hbase-116,16020,1534248590107 but crash processing already in progress
> > WARN  [Thread-14] master.ServerManager: Expiration called on
> > hbase-115,16020,1534249077856 but crash processing already in progress
> > WARN  [Thread-14] master.ServerManager: Expiration called on
> > hbase-116,16020,1534248994045 but crash processing already in progress
> > WARN  [Thread-14] master.ServerManager: Expiration called on
> > hbase-115,16020,1534248708149 but crash processing already in progress
> > WARN  [Thread-14] master.ServerManager: Expiration called on
> > hbase-116,16020,1534248707381 but crash processing already in progress
> >
> >
> >
> > LOG:
> >
> > core file size (blocks, -c) 0
> > data seg size (kbytes, -d) unlimited
> > scheduling priority (-e) 0
> > file size (blocks, -f) unlimited
> > pending signals (-i) 64091
> > max locked memory (kbytes, -l) 64
> > max memory size (kbytes, -m) unlimited
> > open files (-n) 1024
> > pipe size (512 bytes, -p) 8
> > POSIX message queues (bytes, -q) 819200
> > real-time priority (-r) 0
> > stack size (kbytes, -s) 8192
> > cpu time (seconds, -t) unlimited
> > max user processes (-u) 64091
> > virtual memory (kbytes, -v) unlimited
> > file locks (-x) unlimited
> > 2018-08-14 17:25:00,173 INFO [main] master.HMaster: STARTING service
> > HMaster
> > 2018-08-14 17:25:00,174 INFO [main] util.VersionInfo: HBase 2.1.0
> > 2018-08-14 17:25:00,174 INFO [main] util.VersionInfo: Source code
> > repository revision=4531d1c947a25b28a9a994b60c791a112c12a2b4
> > 2018-08-14 17:25:00,174 INFO [main] util.VersionInfo: Compiled by hbase
> on
> > Wed Aug 1 11:25:59 2018
> > 2018-08-14 17:25:00,174 INFO [main] util.VersionInfo: From source with
> > checksum fc32566f7e030ff71458fbf6dc77bce9
> > 2018-08-14 17:25:00,516 INFO [main] util.ServerCommandLine:
> hbase.tmp.dir:
> > /tmp/hbase-root
> > 2018-08-14 17:25:00,516 INFO [main] util.ServerCommandLine:
> hbase.rootdir:
> > hdfs://192.168.101.114:9000/hbase
> > 2018-08-14  17:25:00,516
> > INFO [main] util.ServerCommandLine: hbase.cluster.distributed: true
> > 2018-08-14 17:25:00,516 INFO [main] util.ServerCommandLine:
> > hbase.zookeeper.quorum: 192.168.101.114:2181
> > 2018-08-14 17:25:00,517 INFO [main] util.ServerCommandLine:
> >
> env:PATH=/opt/apache-phoenix-5.0.0-HBase-2.0-bin/bin:/opt/hbase-2.1.0/bin:/opt/hadoop-2.8.4/bin:/opt/jdk1.8.0_172/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin
> > 2018-08-14 17:25:00,517 INFO [main] util.ServerCommandLine:
> > env:HADOOP_CONF_DIR=/opt/hadoop-2.8.4/etc/hadoop
> > 2018-08-14 17:25:00,517 INFO [main] util.ServerCommandLine:
> > env:HISTCONTROL=ignoredups
> > 2018-08-14 17:25:00,517 INFO [main] util.ServerCommandLine:
> >
> env:JAVA_LIBRARY_PATH=/opt/hadoop-2.8.4/lib/native::/opt/hadoop-2.8.4/lib/native:
> > 2018-08-14 17:25:00,517 INFO [main] util.ServerCommandLine:
> > env:HBASE_REGIONSERVER_OPTS= -Xdebug -Xnoagent -Djava.compiler=NONE
> > -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8071
> > 2018-08-14 17:25:00,517 INFO [main] util.ServerCommandLine:
> > env:HBASE_CONF_DIR=/opt/hbase-2.1.0/conf
> > 2018-08-14 17:25:00,517 INFO [main] util.ServerCommandLine:
> > env:HDFS_DATANODE_SECURE_USER=root
> > 2018-08-14 17:25:00,517 INFO [main] util.ServerCommandLine:
> > env:MAIL=/var/spool/mail/root
> > 2018-08-14 17:25:00,517 INFO [main] util.ServerCommandLine:
> > env:PHOENIX_HOME=/opt/apache-phoenix-5.0.0-HBase-2.0-bin
> > 2018-08-14 17:25:00,517 INFO [main] util.ServerCommandLine:
> >
> env:LD_LIBRARY_PATH=:/opt/hadoop-2.8.4/lib/native::/opt/hadoop-2.8.4/lib/native:
> > 2018-08-14 17:25:00,517 INFO [main] util.ServerCommandLine:
> > env:LOGNAME=root
> > 2018-08-14 17:25:00,518 INFO [main] util.ServerCommandLine:
> > env:HBASE_REST_OPTS=
> > 2018-08-14 17:25:00,518 INFO [main] util.ServerCommandLine:
> > env:PWD=/opt/hbase-2.1.0/bin
> > 2018-08-14 17:25:00,518 INFO [main] util.ServerCommandLine:
> > env:HADOOP_PREFIX=/opt/hadoop-2.8.4
> > 2018-08-14 17:25:00,518 INFO [main] util.ServerCommandLine:
> > env:HADOOP_INSTALL=/opt/hadoop-2.8.4
> > 2018-08-14 17:25:00,518 INFO [main] util.ServerCommandLine:
> > env:HBASE_ROOT_LOGGER=INFO,RFA
> > 2018-08-14 17:25:00,518 INFO [main] util.ServerCommandLine:
> > 

Re: HDP3.0: failed to map segment from shared object: Operation not permitted

[Ted Yu]

Since you are using a vendor's distro, can you post on their user group ?
Cheers
 Original message From: Lian Jiang  
Date: 8/13/18  4:03 PM  (GMT-08:00) To: user@hbase.apache.org Subject: HDP3.0: 
failed to map segment from shared object: Operation not permitted 
Hi,

I installed hadoop cluster using HDP3.0 but hbase does not work due to this
error:

Caused by: java.lang.UnsatisfiedLinkError: failed to load the required
native library

    at
org.apache.hbase.thirdparty.io.netty.channel.epoll.Epoll.ensureAvailability(Epoll.java:81)

    at
org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.(EpollEventLoop.java:55)

    at
org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoopGroup.newChild(EpollEventLoopGroup.java:134)

    at
org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoopGroup.newChild(EpollEventLoopGroup.java:35)

    at
org.apache.hbase.thirdparty.io.netty.util.concurrent.MultithreadEventExecutorGroup.(MultithreadEventExecutorGroup.java:84)

    at
org.apache.hbase.thirdparty.io.netty.util.concurrent.MultithreadEventExecutorGroup.(MultithreadEventExecutorGroup.java:58)

    at
org.apache.hbase.thirdparty.io.netty.util.concurrent.MultithreadEventExecutorGroup.(MultithreadEventExecutorGroup.java:47)

    at
org.apache.hbase.thirdparty.io.netty.channel.MultithreadEventLoopGroup.(MultithreadEventLoopGroup.java:59)

    at
org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoopGroup.(EpollEventLoopGroup.java:104)

    at
org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoopGroup.(EpollEventLoopGroup.java:91)

    at
org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoopGroup.(EpollEventLoopGroup.java:68)

    at
org.apache.hadoop.hbase.util.NettyEventLoopGroupConfig.(NettyEventLoopGroupConfig.java:61)

    at
org.apache.hadoop.hbase.regionserver.HRegionServer.setupNetty(HRegionServer.java:673)

    at
org.apache.hadoop.hbase.regionserver.HRegionServer.(HRegionServer.java:532)

    at org.apache.hadoop.hbase.master.HMaster.(HMaster.java:472)

    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)

    at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)

    at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)

    at
org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:2923)

    ... 5 more

Caused by: java.lang.UnsatisfiedLinkError:
/tmp/liborg_apache_hbase_thirdparty_netty_transport_native_epoll_x86_644257058602762792223.so:
/tmp/liborg_apache_hbase_thirdparty_netty_transport_native_ep

oll_x86_644257058602762792223.so: failed to map segment from shared object:
Operation not permitted

    at java.lang.ClassLoader$NativeLibrary.load(Native Method)

    at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1941)

    at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1824)

    at java.lang.Runtime.load0(Runtime.java:809)

    at java.lang.System.load(System.java:1086)

    at
org.apache.hbase.thirdparty.io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:36)

    at
org.apache.hbase.thirdparty.io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:243)

    at
org.apache.hbase.thirdparty.io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:187)

    at
org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.loadNativeLibrary(Native.java:207)

    at
org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.(Native.java:65)

    at
org.apache.hbase.thirdparty.io.netty.channel.epoll.Epoll.(Epoll.java:33)

    ... 24 more


Looks it is caused by nonexec permission of /tmp. To make hbase not use
/tmp, I added

{
  "hbase-site": {
    "properties": {
  "hbase.tmp.dir": "/u01/tmp"
    }
  }
    },

But this does not work and hbase still uses /tmp. Any idea? Appreciate any
help.

Re: Adding same row filter multiple times

[Ted Yu]

I looked at Filter one more time this morning - it doesn't have method
which returns the row in which the Filter is interested in.

Since each of your RowFilter instances may be constructed from unique
objects in memory (though they can be considered equal after byte wise
comparison), we need more formal way of defining equality - which is
required for implementing Set.

In this regard, we should revisit HBASE-19008
<https://issues.apache.org/jira/browse/HBASE-19008> first.

FYI

On Mon, Jul 23, 2018 at 7:10 AM Biplob Biswas 
wrote:

> Thanks for the response, I indeed have MUST_PASS_ONE operator defined for
> the filterlist, and my list passes and returns result as well. I just
> wanted to know how duplicate filters are handled within filterlist, which
> you already clarified by saying that it would bloat up the list.
>
> In this case, won't a filterset make more sense? Do you think creating a
> bug(or a feature request) for this, makes sense?
>
> Thanks & Regards
> Biplob Biswas
>
>
> On Fri, Jul 20, 2018 at 6:31 PM Ted Yu  wrote:
>
> > Assuming your filter list uses MUST_PASS_ONE operator, the recurring key
> > would pass one of these RowFilter's. So the filter list would pass.
> >
> > However, depending on the actual number of repetitions for a given key,
> > your filter list would be bloated.
> > You can use ByteArrayComparable#getValue() to determine if the key you
> want
> > to check is already specified by any of the existing filters in the
> filter
> > list.
> >
> > Cheers
> >
> > On Fri, Jul 20, 2018 at 8:46 AM Biplob Biswas 
> > wrote:
> >
> > > For example,
> > >
> > > I have an iterative process which generates the key and inserts the key
> > as
> > > a RowFilter in a  filter list.
> > >
> > > For example:
> > >
> > > for(String k:keys){
> > >  filterlist.addfilter(new RowFilter(EQ, BinaryComparator(k)))
> > > }
> > >
> > > Now if the list "keys" has same keys multiple times, we are adding the
> > same
> > > RowFilter to the FilterList multiple times. What is HBase doing in that
> > > circumstance
> > >
> > > Thanks & Regards
> > > Biplob Biswas
> > >
> > >
> > > On Fri, Jul 20, 2018 at 4:12 PM Ted Yu  wrote:
> > >
> > > > Did you mean chaining the same row filter 'n' times using FilterList
> ?
> > > > Is the row filter from hbase (RowFilter) ?
> > > > What operator do you use (MUST_PASS_ALL or MUST_PASS_ONE) ?
> > > >
> > > > For second question, I wonder how filter set would handle the
> > constituent
> > > > filters differently from how FilterList handles them.
> > > > Maybe by 'Set' you mean when any one of the constituent filters
> passes,
> > > the
> > > > row is returned ?
> > > >
> > > > Cheers
> > > >
> > > > On Fri, Jul 20, 2018 at 4:45 AM Biplob Biswas <
> > revolutioni...@gmail.com>
> > > > wrote:
> > > >
> > > > > What happens if I insert the same row filter 'n' times? Would the
> > table
> > > > be
> > > > > queried n times with the same row key?
> > > > >
> > > > > Why is it that there is a filterList and not a filterSet?
> > > > >
> > > > > Thanks & Regards
> > > > > Biplob Biswas
> > > > >
> > > >
> > >
> >
>

Re: Compactions after bulk load

[Ted Yu]

Have you checked the output from bulk load and see if there were lines in
the following form (from LoadIncrementalHFiles#splitStoreFile) ?

LOG.info("HFile at " + hfilePath + " no longer fits inside a
single " + "region.
Splitting...");

In the server log, you should see log in the following form:

  if (LOG.isDebugEnabled()) {
LOG.debug("Compacting " + file +
  ", keycount=" + keyCount +
  ", bloomtype=" + r.getBloomFilterType().toString() +
  ", size=" + TraditionalBinaryPrefix.long2String(r.length(), "",
1) +
  ", encoding=" + r.getHFileReader().getDataBlockEncoding() +
  ", seqNum=" + seqNum +
  (allFiles ? ", earliestPutTs=" + earliestPutTs: ""));
  }

where allFiles being true indicates major compaction.

The above should give you some idea of the cause for the compaction
activity.

Thanks

On Tue, Jul 17, 2018 at 11:12 AM Austin Heyne  wrote:

> Hi all,
>
> I'm trying to bulk load a large amount of data into HBase. The bulk load
> succeeds but then HBase starts running compactions. My input files are
> typically ~5-6GB and there are over 3k files. I've used the same table
> splits for the bulk ingest and the bulk load so there should be no
> reason for hbase to run any compactions. However, I'm seeing it first
> start compacting the hfiles into 25+GB files and then into 200+GB files
> but didn't let it run any longer. Additionally, I've talked with another
> coworker who's tried this process in the past and he's experience the
> same thing, eventually giving up on the feature. My attempts have been
> on HBase 1.4.2. Does anyone have information on why HBase is insisting
> on running these compactions or how I can stop them? They are
> essentially breaking the feature for us.
>
> Thanks,
>
> --
> Austin L. Heyne
>
>

Re: Adding same row filter multiple times

[Ted Yu]

Assuming your filter list uses MUST_PASS_ONE operator, the recurring key
would pass one of these RowFilter's. So the filter list would pass.

However, depending on the actual number of repetitions for a given key,
your filter list would be bloated.
You can use ByteArrayComparable#getValue() to determine if the key you want
to check is already specified by any of the existing filters in the filter
list.

Cheers

On Fri, Jul 20, 2018 at 8:46 AM Biplob Biswas 
wrote:

> For example,
>
> I have an iterative process which generates the key and inserts the key as
> a RowFilter in a  filter list.
>
> For example:
>
> for(String k:keys){
>  filterlist.addfilter(new RowFilter(EQ, BinaryComparator(k)))
> }
>
> Now if the list "keys" has same keys multiple times, we are adding the same
> RowFilter to the FilterList multiple times. What is HBase doing in that
> circumstance
>
> Thanks & Regards
> Biplob Biswas
>
>
> On Fri, Jul 20, 2018 at 4:12 PM Ted Yu  wrote:
>
> > Did you mean chaining the same row filter 'n' times using FilterList ?
> > Is the row filter from hbase (RowFilter) ?
> > What operator do you use (MUST_PASS_ALL or MUST_PASS_ONE) ?
> >
> > For second question, I wonder how filter set would handle the constituent
> > filters differently from how FilterList handles them.
> > Maybe by 'Set' you mean when any one of the constituent filters passes,
> the
> > row is returned ?
> >
> > Cheers
> >
> > On Fri, Jul 20, 2018 at 4:45 AM Biplob Biswas 
> > wrote:
> >
> > > What happens if I insert the same row filter 'n' times? Would the table
> > be
> > > queried n times with the same row key?
> > >
> > > Why is it that there is a filterList and not a filterSet?
> > >
> > > Thanks & Regards
> > > Biplob Biswas
> > >
> >
>

Re: Adding same row filter multiple times

[Ted Yu]

Did you mean chaining the same row filter 'n' times using FilterList ?
Is the row filter from hbase (RowFilter) ?
What operator do you use (MUST_PASS_ALL or MUST_PASS_ONE) ?

For second question, I wonder how filter set would handle the constituent
filters differently from how FilterList handles them.
Maybe by 'Set' you mean when any one of the constituent filters passes, the
row is returned ?

Cheers

On Fri, Jul 20, 2018 at 4:45 AM Biplob Biswas 
wrote:

> What happens if I insert the same row filter 'n' times? Would the table be
> queried n times with the same row key?
>
> Why is it that there is a filterList and not a filterSet?
>
> Thanks & Regards
> Biplob Biswas
>

Re: issue while reading data from hbase

[Ted Yu]

Putting dev@ to bcc.

Which hbase-spark connector are you using ?
What's the hbase release in your deployment ?

bq. some of the columns in dataframe becomes null

Is it possible to characterize what type of columns become null ? Earlier
you said one column has xml data. Did you mean this column from some rows
returned null ?

Have you checked region server logs where the corresponding regions reside ?

Thanks

On Fri, Jul 13, 2018 at 4:08 AM hnk45  wrote:

> I am reading data from hbase using spark sql. one column has xml data. when
> xml size is small , I am able to read correct data. but as soon as size
> increases too much, some of the columns in dataframe becomes null. xml is
> still coming correctly.
> while reading data from sql to hbase I have used this constraint:
> hbase.client.keyvalue.maxsize=0 in my sqoop.
>
>
>
> --
> Sent from:
> http://apache-hbase.679495.n3.nabble.com/HBase-Developer-f679493.html
>

Re: I am a subscribe please add me thanks

[Ted Yu]

Please see for subscription information:

http://hbase.apache.org/mail-lists.html

On Wed, Jul 11, 2018 at 4:19 AM bill.zhou  wrote:

> I am a subscriber please add me thanks
>
>

Re: HBase scan with setBatch for more than 1 column family

[Ted Yu]

Please see the following two constants defined in TableInputFormat :

  /** Column Family to Scan */

  public static final String SCAN_COLUMN_FAMILY =
"hbase.mapreduce.scan.column.family";

  /** Space delimited list of columns and column families to scan. */

  public static final String SCAN_COLUMNS = "hbase.mapreduce.scan.columns";

CellCounter accepts these parameters. You can play with CellCounter to see
how they work.


FYI

On Mon, Jul 2, 2018 at 4:01 AM, revolutionisme 
wrote:

> Hi,
>
> I am using HBase with Spark and as I have wide columns (> 1) I wanted
> to
> use the "setbatch(num)" option to not read all the columns for a row but in
> batches.
>
> I can create a scan and set the batch size I want with
> TableInputFormat.SCAN_BATCHSIZE, but I am a bit confused how this would
> work
> with more than 1 column family.
>
> Any help is appreciated.
>
> PS: Also any documentation or inputs on newAPIHadoopRDD would be really
> appreciated as well.
>
> Thanks & Regards,
> Biplob
>
>
>
> --
> Sent from: http://apache-hbase.679495.n3.nabble.com/HBase-User-
> f4020416.html
>

Re: Hbase hbase-2.0.1, region server does not start on Hadoop 3.1

[Ted Yu]

Have you tried setting the value for the config to filesystem ?

Cheers

On Sat, Jun 30, 2018 at 4:07 PM, Mich Talebzadeh 
wrote:

> One way would be to set WAL outside of Hadoop environment. Will that work?
>
> The following did not work
>
> 
>   hbase.wal.provider
>   multiwal
> 
>
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * https://www.linkedin.com/profile/view?id=
> AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> <https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCd
> OABUrV8Pw>*
>
>
>
> http://talebzadehmich.wordpress.com
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
>
> On Sat, 30 Jun 2018 at 23:36, Ted Yu  wrote:
>
> > Please read :
> >
> > http://hbase.apache.org/book.html#wal.providers
> >
> > On Sat, Jun 30, 2018 at 3:31 PM, Mich Talebzadeh <
> > mich.talebza...@gmail.com>
> > wrote:
> >
> > > Thanks
> > >
> > > In your point below
> > >
> > > …. or you can change default WAL to FSHLog.
> > >
> > > is there any configuration parameter to allow me to do so in
> > > hbase-site.xml?
> > >
> > > Dr Mich Talebzadeh
> > >
> > >
> > >
> > > LinkedIn * https://www.linkedin.com/profile/view?id=
> > > AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> > > <https://www.linkedin.com/profile/view?id=
> AAEWh2gBxianrbJd6zP6AcPCCd
> > > OABUrV8Pw>*
> > >
> > >
> > >
> > > http://talebzadehmich.wordpress.com
> > >
> > >
> > > *Disclaimer:* Use it at your own risk. Any and all responsibility for
> any
> > > loss, damage or destruction of data or any other property which may
> arise
> > > from relying on this email's technical content is explicitly
> disclaimed.
> > > The author will in no case be liable for any monetary damages arising
> > from
> > > such loss, damage or destruction.
> > >
> > >
> > >
> > >
> > > On Sat, 30 Jun 2018 at 23:25, Ted Yu  wrote:
> > >
> > > > Do you plan to deploy onto hadoop 3.1.x ?
> > > >
> > > > If so, you'd better build against hadoop 3.1.x yourself.
> > > > You can either patch in HBASE-20244 and use asyncfswal.
> > > > Or you can change default WAL to FSHLog.
> > > >
> > > > If you don't have to deploy onto hadoop 3.1.x, you can use hbase
> 2.0.1
> > > >
> > > > FYI
> > > >
> > > > On Sat, Jun 30, 2018 at 3:21 PM, Mich Talebzadeh <
> > > > mich.talebza...@gmail.com>
> > > > wrote:
> > > >
> > > > > so what options do I have her?. Is there any conf parameter I can
> set
> > > in
> > > > > hbase-site,xml to make this work? or shall I go back to a more
> stable
> > > > > version of Hbase?
> > > > >
> > > > > cheers
> > > > >
> > > > > Dr Mich Talebzadeh
> > > > >
> > > > >
> > > > >
> > > > > LinkedIn * https://www.linkedin.com/profile/view?id=
> > > > > AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> > > > > <https://www.linkedin.com/profile/view?id=
> > > AAEWh2gBxianrbJd6zP6AcPCCd
> > > > > OABUrV8Pw>*
> > > > >
> > > > >
> > > > >
> > > > > http://talebzadehmich.wordpress.com
> > > > >
> > > > >
> > > > > *Disclaimer:* Use it at your own risk. Any and all responsibility
> for
> > > any
> > > > > loss, damage or destruction of data or any other property which may
> > > arise
> > > > > from relying on this email's technical content is explicitly
> > > disclaimed.
> > > > > The author will in no case be liable for any monetary damages
> arising
> > > > from
> > > > > such loss, damage or destruction.
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Sat, 30 Jun 2018 at 23:15, Ted Yu  wrote:
> > > > >
> > > > > > trunk version would correspond to hbase 3.0 which 

Re: Hbase hbase-2.0.1, region server does not start on Hadoop 3.1

[Ted Yu]

Please read :

http://hbase.apache.org/book.html#wal.providers

On Sat, Jun 30, 2018 at 3:31 PM, Mich Talebzadeh 
wrote:

> Thanks
>
> In your point below
>
> …. or you can change default WAL to FSHLog.
>
> is there any configuration parameter to allow me to do so in
> hbase-site.xml?
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * https://www.linkedin.com/profile/view?id=
> AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> <https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCd
> OABUrV8Pw>*
>
>
>
> http://talebzadehmich.wordpress.com
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
>
> On Sat, 30 Jun 2018 at 23:25, Ted Yu  wrote:
>
> > Do you plan to deploy onto hadoop 3.1.x ?
> >
> > If so, you'd better build against hadoop 3.1.x yourself.
> > You can either patch in HBASE-20244 and use asyncfswal.
> > Or you can change default WAL to FSHLog.
> >
> > If you don't have to deploy onto hadoop 3.1.x, you can use hbase 2.0.1
> >
> > FYI
> >
> > On Sat, Jun 30, 2018 at 3:21 PM, Mich Talebzadeh <
> > mich.talebza...@gmail.com>
> > wrote:
> >
> > > so what options do I have her?. Is there any conf parameter I can set
> in
> > > hbase-site,xml to make this work? or shall I go back to a more stable
> > > version of Hbase?
> > >
> > > cheers
> > >
> > > Dr Mich Talebzadeh
> > >
> > >
> > >
> > > LinkedIn * https://www.linkedin.com/profile/view?id=
> > > AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> > > <https://www.linkedin.com/profile/view?id=
> AAEWh2gBxianrbJd6zP6AcPCCd
> > > OABUrV8Pw>*
> > >
> > >
> > >
> > > http://talebzadehmich.wordpress.com
> > >
> > >
> > > *Disclaimer:* Use it at your own risk. Any and all responsibility for
> any
> > > loss, damage or destruction of data or any other property which may
> arise
> > > from relying on this email's technical content is explicitly
> disclaimed.
> > > The author will in no case be liable for any monetary damages arising
> > from
> > > such loss, damage or destruction.
> > >
> > >
> > >
> > >
> > > On Sat, 30 Jun 2018 at 23:15, Ted Yu  wrote:
> > >
> > > > trunk version would correspond to hbase 3.0 which has lot more
> changes
> > > > compared to hbase 2.
> > > > The trunk build wouldn't serve you if your goal is to run hbase on
> > hadoop
> > > > 3.1 (see HBASE-20244)
> > > >
> > > > FYI
> > > >
> > > > On Sat, Jun 30, 2018 at 3:11 PM, Mich Talebzadeh <
> > > > mich.talebza...@gmail.com>
> > > > wrote:
> > > >
> > > > > Thanks Ted.
> > > > >
> > > > > I downloaded the latest Hbase binary which is 2.0.1 2018/06/19
> > > > >
> > > > > Is there any trunc version build for Hadoop 3.1 please and if so
> > where
> > > > can
> > > > > I download it?
> > > > >
> > > > > Regards,
> > > > >
> > > > > Dr Mich Talebzadeh
> > > > >
> > > > >
> > > > >
> > > > > LinkedIn * https://www.linkedin.com/profile/view?id=
> > > > > AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> > > > > <https://www.linkedin.com/profile/view?id=
> > > AAEWh2gBxianrbJd6zP6AcPCCd
> > > > > OABUrV8Pw>*
> > > > >
> > > > >
> > > > >
> > > > > http://talebzadehmich.wordpress.com
> > > > >
> > > > >
> > > > > *Disclaimer:* Use it at your own risk. Any and all responsibility
> for
> > > any
> > > > > loss, damage or destruction of data or any other property which may
> > > arise
> > > > > from relying on this email's technical content is explicitly
> > > disclaimed.
> > > > > The author will in no case be liable for any monetary damages
> arising
> > > > from
> > > > > such loss, damage or destruction.
> > > > >
> > > >

Re: Hbase hbase-2.0.1, region server does not start on Hadoop 3.1

[Ted Yu]

Do you plan to deploy onto hadoop 3.1.x ?

If so, you'd better build against hadoop 3.1.x yourself.
You can either patch in HBASE-20244 and use asyncfswal.
Or you can change default WAL to FSHLog.

If you don't have to deploy onto hadoop 3.1.x, you can use hbase 2.0.1

FYI

On Sat, Jun 30, 2018 at 3:21 PM, Mich Talebzadeh 
wrote:

> so what options do I have her?. Is there any conf parameter I can set in
> hbase-site,xml to make this work? or shall I go back to a more stable
> version of Hbase?
>
> cheers
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * https://www.linkedin.com/profile/view?id=
> AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> <https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCd
> OABUrV8Pw>*
>
>
>
> http://talebzadehmich.wordpress.com
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
>
> On Sat, 30 Jun 2018 at 23:15, Ted Yu  wrote:
>
> > trunk version would correspond to hbase 3.0 which has lot more changes
> > compared to hbase 2.
> > The trunk build wouldn't serve you if your goal is to run hbase on hadoop
> > 3.1 (see HBASE-20244)
> >
> > FYI
> >
> > On Sat, Jun 30, 2018 at 3:11 PM, Mich Talebzadeh <
> > mich.talebza...@gmail.com>
> > wrote:
> >
> > > Thanks Ted.
> > >
> > > I downloaded the latest Hbase binary which is 2.0.1 2018/06/19
> > >
> > > Is there any trunc version build for Hadoop 3.1 please and if so where
> > can
> > > I download it?
> > >
> > > Regards,
> > >
> > > Dr Mich Talebzadeh
> > >
> > >
> > >
> > > LinkedIn * https://www.linkedin.com/profile/view?id=
> > > AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> > > <https://www.linkedin.com/profile/view?id=
> AAEWh2gBxianrbJd6zP6AcPCCd
> > > OABUrV8Pw>*
> > >
> > >
> > >
> > > http://talebzadehmich.wordpress.com
> > >
> > >
> > > *Disclaimer:* Use it at your own risk. Any and all responsibility for
> any
> > > loss, damage or destruction of data or any other property which may
> arise
> > > from relying on this email's technical content is explicitly
> disclaimed.
> > > The author will in no case be liable for any monetary damages arising
> > from
> > > such loss, damage or destruction.
> > >
> > >
> > >
> > >
> > > On Sat, 30 Jun 2018 at 22:52, Ted Yu  wrote:
> > >
> > > > Which hadoop release was the 2.0.1 built against ?
> > > >
> > > > In order to build hbase 2 against hadoop 3.0.1+ / 3.1.0+, you will
> need
> > > > HBASE-20244.
> > > >
> > > > FYI
> > > >
> > > > On Sat, Jun 30, 2018 at 2:34 PM, Mich Talebzadeh <
> > > > mich.talebza...@gmail.com>
> > > > wrote:
> > > >
> > > > > I am using the following hbase-site.xml
> > > > >
> > > > > 
> > > > >   
> > > > > hbase.rootdir
> > > > > hdfs://rhes75:9000/hbase
> > > > >   
> > > > >   
> > > > > hbase.zookeeper.property.dataDir
> > > > > /home/hduser/zookeeper-3.4.6
> > > > >   
> > > > > 
> > > > > hbase.master
> > > > > localhost:6
> > > > > 
> > > > > 
> > > > >   hbase.zookeeper.property.clientPort
> > > > >   2181
> > > > >
> > > > >   
> > > > > hbase.cluster.distributed
> > > > > true
> > > > >   
> > > > > 
> > > > >  hbase.defaults.for.version.skip
> > > > >  true
> > > > > 
> > > > > 
> > > > >  phoenix.query.dateFormatTimeZone
> > > > >  Europe/London
> > > > > 
> > > > > 
> > > > > hbase.procedure.store.wal.use.hsync
> > > > > false
> > > > > `
> > > > > 
> > > > >   hbase.unsafe.stream.capability.enforce
> > > > >   false
> > > > > 
> > > > > 
> > > 

Re: Hbase hbase-2.0.1, region server does not start on Hadoop 3.1

[Ted Yu]

trunk version would correspond to hbase 3.0 which has lot more changes
compared to hbase 2.
The trunk build wouldn't serve you if your goal is to run hbase on hadoop
3.1 (see HBASE-20244)

FYI

On Sat, Jun 30, 2018 at 3:11 PM, Mich Talebzadeh 
wrote:

> Thanks Ted.
>
> I downloaded the latest Hbase binary which is 2.0.1 2018/06/19
>
> Is there any trunc version build for Hadoop 3.1 please and if so where can
> I download it?
>
> Regards,
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * https://www.linkedin.com/profile/view?id=
> AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> <https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCd
> OABUrV8Pw>*
>
>
>
> http://talebzadehmich.wordpress.com
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
>
> On Sat, 30 Jun 2018 at 22:52, Ted Yu  wrote:
>
> > Which hadoop release was the 2.0.1 built against ?
> >
> > In order to build hbase 2 against hadoop 3.0.1+ / 3.1.0+, you will need
> > HBASE-20244.
> >
> > FYI
> >
> > On Sat, Jun 30, 2018 at 2:34 PM, Mich Talebzadeh <
> > mich.talebza...@gmail.com>
> > wrote:
> >
> > > I am using the following hbase-site.xml
> > >
> > > 
> > >   
> > > hbase.rootdir
> > > hdfs://rhes75:9000/hbase
> > >   
> > >   
> > > hbase.zookeeper.property.dataDir
> > > /home/hduser/zookeeper-3.4.6
> > >   
> > > 
> > > hbase.master
> > > localhost:6
> > > 
> > > 
> > >   hbase.zookeeper.property.clientPort
> > >   2181
> > >
> > >   
> > > hbase.cluster.distributed
> > > true
> > >   
> > > 
> > >  hbase.defaults.for.version.skip
> > >  true
> > > 
> > > 
> > >  phoenix.query.dateFormatTimeZone
> > >  Europe/London
> > > 
> > > 
> > > hbase.procedure.store.wal.use.hsync
> > > false
> > > `
> > > 
> > >   hbase.unsafe.stream.capability.enforce
> > >   false
> > > 
> > > 
> > >
> > > master starts OK but region server throws some errors
> > >
> > > 2018-06-30 22:23:56,607 INFO  [regionserver/rhes75:16020]
> > > wal.AbstractFSWAL: WAL configuration: blocksize=256 MB, rollsize=128
> MB,
> > > prefix=rhes75%2C16020%2C1530393832024, suffix=,
> > > logDir=hdfs://rhes75:9000/hbase/WALs/rhes75,16020,153
> > > 0393832024, archiveDir=hdfs://rhes75:9000/hbase/oldWALs
> > > 2018-06-30 22:23:56,629 ERROR [regionserver/rhes75:16020]
> > > regionserver.HRegionServer: ason:
> > > Type 'org/apache/hadoop/fs/ContentSummary' (current frame,
> stack[1])
> > > is
> > > not assignable to 'org/apache/hadoop/fs/QuotaUsage'
> > >   Current Frame:
> > > bci: @105
> > > flags: { }
> > > locals: { 'org/apache/hadoop/fs/ContentSummary',
> > > 'org/apache/hadoop/hdfs/protocol/proto/HdfsProtos$
> > > ContentSummaryProto$Builder'
> > > }
> > > stack: {
> > > 'org/apache/hadoop/hdfs/protocol/proto/HdfsProtos$
> > > ContentSummaryProto$Builder',
> > > 'org/apache/hadoop/fs/ContentSummary' }
> > >   Bytecode:
> > > 0x000: 2ac7 0005 01b0 b805 984c 2b2a b605 99b6
> > > 0x010: 059a 2ab6 059b b605 9c2a b605 9db6 059e
> > > 0x020: 2ab6 059f b605 a02a b605 a1b6 05a2 2ab6
> > > 0x030: 05a3 b605 a42a b605 a5b6 05a6 2ab6 05a7
> > > 0x040: b605 a82a b605 a9b6 05aa 2ab6 05ab b605
> > > 0x050: ac2a b605 adb6 05ae 572a b605 af9a 000a
> > > 0x060: 2ab6 05b0 9900 0c2b 2ab8 0410 b605 b157
> > > 0x070: 2bb6 05b2 b0
> > >   Stackmap Table:
> > > same_frame(@6)
> > > append_frame(@103,Object[#2940])
> > > same_frame(@112)
> > >  *
> > > java.lang.VerifyError: Bad type on operand stack
> > > Exception Details:
> > >   Location:
> > >
> > > org/apache/hadoop/hdfs/protocolPB/PBHelperClient.
> > > convert(Lorg/apache/hadoop/fs/ContentSummary;)Lorg/apache/
> > > hadoop/hdfs/protocol/proto/HdfsProtos$Con

Re: Hbase hbase-2.0.1, region server does not start on Hadoop 3.1

[Ted Yu]

Which hadoop release was the 2.0.1 built against ?

In order to build hbase 2 against hadoop 3.0.1+ / 3.1.0+, you will need
HBASE-20244.

FYI

On Sat, Jun 30, 2018 at 2:34 PM, Mich Talebzadeh 
wrote:

> I am using the following hbase-site.xml
>
> 
>   
> hbase.rootdir
> hdfs://rhes75:9000/hbase
>   
>   
> hbase.zookeeper.property.dataDir
> /home/hduser/zookeeper-3.4.6
>   
> 
> hbase.master
> localhost:6
> 
> 
>   hbase.zookeeper.property.clientPort
>   2181
>
>   
> hbase.cluster.distributed
> true
>   
> 
>  hbase.defaults.for.version.skip
>  true
> 
> 
>  phoenix.query.dateFormatTimeZone
>  Europe/London
> 
> 
> hbase.procedure.store.wal.use.hsync
> false
> `
> 
>   hbase.unsafe.stream.capability.enforce
>   false
> 
> 
>
> master starts OK but region server throws some errors
>
> 2018-06-30 22:23:56,607 INFO  [regionserver/rhes75:16020]
> wal.AbstractFSWAL: WAL configuration: blocksize=256 MB, rollsize=128 MB,
> prefix=rhes75%2C16020%2C1530393832024, suffix=,
> logDir=hdfs://rhes75:9000/hbase/WALs/rhes75,16020,153
> 0393832024, archiveDir=hdfs://rhes75:9000/hbase/oldWALs
> 2018-06-30 22:23:56,629 ERROR [regionserver/rhes75:16020]
> regionserver.HRegionServer: ason:
> Type 'org/apache/hadoop/fs/ContentSummary' (current frame, stack[1])
> is
> not assignable to 'org/apache/hadoop/fs/QuotaUsage'
>   Current Frame:
> bci: @105
> flags: { }
> locals: { 'org/apache/hadoop/fs/ContentSummary',
> 'org/apache/hadoop/hdfs/protocol/proto/HdfsProtos$
> ContentSummaryProto$Builder'
> }
> stack: {
> 'org/apache/hadoop/hdfs/protocol/proto/HdfsProtos$
> ContentSummaryProto$Builder',
> 'org/apache/hadoop/fs/ContentSummary' }
>   Bytecode:
> 0x000: 2ac7 0005 01b0 b805 984c 2b2a b605 99b6
> 0x010: 059a 2ab6 059b b605 9c2a b605 9db6 059e
> 0x020: 2ab6 059f b605 a02a b605 a1b6 05a2 2ab6
> 0x030: 05a3 b605 a42a b605 a5b6 05a6 2ab6 05a7
> 0x040: b605 a82a b605 a9b6 05aa 2ab6 05ab b605
> 0x050: ac2a b605 adb6 05ae 572a b605 af9a 000a
> 0x060: 2ab6 05b0 9900 0c2b 2ab8 0410 b605 b157
> 0x070: 2bb6 05b2 b0
>   Stackmap Table:
> same_frame(@6)
> append_frame(@103,Object[#2940])
> same_frame(@112)
>  *
> java.lang.VerifyError: Bad type on operand stack
> Exception Details:
>   Location:
>
> org/apache/hadoop/hdfs/protocolPB/PBHelperClient.
> convert(Lorg/apache/hadoop/fs/ContentSummary;)Lorg/apache/
> hadoop/hdfs/protocol/proto/HdfsProtos$ContentSummaryProto;
> @105: invokestatic
>   Reason:
> Type 'org/apache/hadoop/fs/ContentSummary' (current frame, stack[1])
> is
> not assignable to 'org/apache/hadoop/fs/QuotaUsage'
>   Current Frame:
> bci: @105
> flags: { }
> locals: { 'org/apache/hadoop/fs/ContentSummary',
> 'org/apache/hadoop/hdfs/protocol/proto/HdfsProtos$
> ContentSummaryProto$Builder'
> }
> stack: {
> 'org/apache/hadoop/hdfs/protocol/proto/HdfsProtos$
> ContentSummaryProto$Builder',
> 'org/apache/hadoop/fs/ContentSummary' }
>   Bytecode:
> 0x000: 2ac7 0005 01b0 b805 984c 2b2a b605 99b6
> 0x010: 059a 2ab6 059b b605 9c2a b605 9db6 059e
> 0x020: 2ab6 059f b605 a02a b605 a1b6 05a2 2ab6
> 0x030: 05a3 b605 a42a b605 a5b6 05a6 2ab6 05a7
> 0x040: b605 a82a b605 a9b6 05aa 2ab6 05ab b605
> 0x050: ac2a b605 adb6 05ae 572a b605 af9a 000a
> 0x060: 2ab6 05b0 9900 0c2b 2ab8 0410 b605 b157
> 0x070: 2bb6 05b2 b0
>   Stackmap Table:
> same_frame(@6)
> append_frame(@103,Object[#2940])
> same_frame(@112)
>
> any ideas?
>
> thanks
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * https://www.linkedin.com/profile/view?id=
> AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>  OABUrV8Pw>*
>
>
>
> http://talebzadehmich.wordpress.com
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>

Re: Eclipse and hbase-server version 2.0.1- bad imports

[Ted Yu]

If you want to run against hadoop 3.1.x, you would need HBASE-20244.

You can wait for hadoop 3.1.1 to come out which is more stable than 3.1.0

You should add dependency on hadoop-mapreduce-client-jobcli
ent-3.1.0-tests.jar

bq. Similarly "test" for other I can't found classes?

I already explained which jars contain the other two classes.
Better approach is to let 'mvn eclipse:eclipse' generate the dependencies
for you.

bq. I don't have in .m2 directory

Have you looked under ~/.m2/repository ?

Cheers

On Fri, Jun 22, 2018 at 10:27 AM, Andrzej  wrote:

> W dniu 22.06.2018 o 18:47, Ted Yu pisze:
>
>> $ jar tvf
>> ~/.m2/repository/org/apache/hadoop/hadoop-mapreduce-client-
>> jobclient/3.0.0/hadoop-mapreduce-client-jobclient-3.0.0-tests.jar
>> | grep MiniMRCluster
>>
>
> andrzej@andrzej-VirtualBox ~/jars $ jar tvf 
> hadoop-mapreduce-client-jobclient-3.1.0.jar
> | grep MiniMRCluster
>
> <--nothing
>
> andrzej@andrzej-VirtualBox ~/jars $ jar tvf 
> hadoop-mapreduce-client-jobclient-3.1.0-tests.jar
> | grep MiniMRCluster
>   1863 Mon Apr 02 21:08:24 CEST 2018 org/apache/hadoop/mapred/Clust
> erMapReduceTestCase$ConfigurableMiniMRCluster.class
>573 Mon Apr 02 21:08:24 CEST 2018 org/apache/hadoop/mapred/MiniM
> RCluster$JobTrackerRunner.class
>576 Mon Apr 02 21:08:24 CEST 2018 org/apache/hadoop/mapred/MiniM
> RCluster$TaskTrackerRunner.class
>   9937 Mon Apr 02 21:08:24 CEST 2018 org/apache/hadoop/mapred/MiniM
> RCluster.class
>
>
> *Solution*: I must hadoop-mapreduce-client-jobclient-3.1.0-tests.jar
> instead of hadoop-mapreduce-client-jobclient-3.1.0.jar ?
> Similarly "test" for other I can't found classes?
>
> Better is 3.1.0 than 3.0.0 for v2.0.1?
> I don't have in .m2 directory, I must run mvn on hbase-server folder?
>

Re: Eclipse and hbase-server version 2.0.1- bad imports

[Ted Yu]

$ jar tvf
~/.m2/repository/org/apache/hadoop/hadoop-mapreduce-client-jobclient/3.0.0/hadoop-mapreduce-client-jobclient-3.0.0-tests.jar
| grep MiniMRCluster
  1863 Fri Dec 08 11:31:44 PST 2017
org/apache/hadoop/mapred/ClusterMapReduceTestCase$ConfigurableMiniMRCluster.class
  9947 Fri Dec 08 11:31:44 PST 2017
org/apache/hadoop/mapred/MiniMRCluster.class
   576 Fri Dec 08 11:31:44 PST 2017
org/apache/hadoop/mapred/MiniMRCluster$TaskTrackerRunner.class
   573 Fri Dec 08 11:31:44 PST 2017
org/apache/hadoop/mapred/MiniMRCluster$JobTrackerRunner.class

KeyStoreTestUtil is in hbase-http module.
ProcedureTestingUtility is in hbase-procedure module.

You can run the following command prior to importing projects into Eclipse:

mvn clean package -DskipTests eclipse:eclipse

On Fri, Jun 22, 2018 at 8:13 AM, Andrzej  wrote:

> W dniu 22.06.2018 o 17:02, Andrzej pisze:
>
>>  hadoop-mapreduce-client-jobclient-3.1.0.jar
>> import org.apache.hadoop.mapred.MiniMRCluster;
>>
>
> for example :
> fc:org.apache.hadoop.mapred.MiniMRCluster
> give me hadoop-mapreduce-client-jobclient but this jar has package
> org.apache.hadoop.mapred but not have class MiniMRCluster inside this
> package.
>
>

Re: Bulk Load is pulling files locally with same FS and validation disabled.

[Ted Yu]

Since S3FileSystem is not taken into account in FSHDFSUtils#isSameHdfs, we
need to add more code to avoid the overhead.

Can you log a JIRA with what you discovered ?

Thanks

On Thu, Jun 21, 2018 at 2:08 PM, Austin Heyne  wrote:

> Hi again,
>
> I've been doing more digging into this and I've found that with the way
> the code it written it's actually impossible. In FSHDFSUtils [1] HBase
> attempts to get the Canonical Service Name from Hadoop. Since we're running
> on EMR our filesystem is the S3NativeFileSystem (com.amazon) which extends,
> I believe, the NativeS3FileSystem (org.apache). Since the
> NativeS3FileSystem [2] and S3FileSystem [2] both always return null for
> getCannonicalServiceName and from testing it appears the S3NativeFileSystem
> does the same it looks like there is no way to get past the check in
> FSHDFSUtils.isSameHdfs when running on S3 of any kind.
>
> Does anyone know of a workaround for this issue?
>
> Thanks,
> Austin Heyne
>
> [1] https://github.com/apache/hbase/blob/rel/1.4.2/hbase-server/
> src/main/java/org/apache/hadoop/hbase/util/FSHDFSUtils.java#L120-L123
> [2] https://github.com/apache/hadoop/blob/release-2.8.3-RC0/hado
> op-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/
> s3native/NativeS3FileSystem.java#L788-L791
> [3] https://github.com/apache/hadoop/blob/release-2.8.3-RC0/hado
> op-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3/
> S3FileSystem.java#L403-L406
>
>
> On 06/20/2018 07:07 PM, Austin Heyne wrote:
>
>> Hi everyone,
>>
>> I'm trying to run a bulk load of about 15TB of data sitting in s3 that
>> I've bulk ingested. When I initiate the load I'm seeing the data get copied
>> down to the workers and then back up to s3 even though the HBase root is
>> that same bucket.
>>
>> Data to be bulk loaded is at s3://bucket_name/data/bulkload/z3/d/ and
>> the HBase root is s3://bucket_name/. I have validation disabled through the
>> "hbase.loadincremental.validate.hfile" = "false" config set in code
>> before I call LoadIncrementalHFiles.doBulkLoad. (Code is available at
>> [1]) The splits have already been generated with the same config that was
>> used during the ingest so they'll line up. I'm currently running HBase
>> 1.4.2 on AWS EMR. The logs of interest were pulled from a worker on the
>> cluster:
>>
>> """
>> 2018-06-20 22:42:15,888 INFO 
>> [RpcServer.default.FPBQ.Fifo.handler=29,queue=2,port=16020]
>> s3n.S3NativeFileSystem: Opening 's3n://bucket_name/data/bulklo
>> ad/z3/d/b9371885084e4060ac157799e5c89b59' for reading
>> 2018-06-20 22:42:16,026 INFO 
>> [RpcServer.default.FPBQ.Fifo.handler=29,queue=2,port=16020]
>> compress.CodecPool: Got brand-new decompressor [.snappy]
>> 2018-06-20 22:42:16,056 INFO 
>> [RpcServer.default.FPBQ.Fifo.handler=29,queue=2,port=16020]
>> regionserver.HRegionFileSystem: Bulk-load file
>> s3n://bucket_name/data/bulkload/z3/d/b9371885084e4060ac157799e5c89b59 is
>> on different filesystem than the destination store. Copying file over to
>> destination filesystem.
>> 2018-06-20 22:42:16,109 INFO 
>> [RpcServer.default.FPBQ.Fifo.handler=29,queue=2,port=16020]
>> s3n.S3NativeFileSystem: Opening 's3n://bucket_name/data/bulklo
>> ad/z3/d/b9371885084e4060ac157799e5c89b59' for reading
>> 2018-06-20 22:42:17,910 INFO 
>> [RpcServer.default.FPBQ.Fifo.handler=29,queue=2,port=16020]
>> s3n.MultipartUploadOutputStream: close closed:false
>> s3://bucket_name/data/default/z3_v2/c34a364639dab40857a8d188
>> 71a3f705/.tmp/3d69598daa9841f986da2341a5901444
>> 2018-06-20 22:42:17,927 INFO 
>> [RpcServer.default.FPBQ.Fifo.handler=29,queue=2,port=16020]
>> regionserver.HRegionFileSystem: Copied s3n://bucket_name/data/bulkloa
>> d/z3/d/b9371885084e4060ac157799e5c89b59 to temporary path on destination
>> filesystem: s3://bucket_name/data/default/z3_v2/c34a364639dab40857a8d188
>> 71a3f705/.tmp/3d69598daa9841f986da2341a5901444
>> """
>>
>> I'm seeing the behavior using 's3://' or 's3n://'. Has anyone experienced
>> this or have advice?
>>
>> Thanks,
>> Austin Heyne
>>
>> [1] https://github.com/aheyne/geomesa/blob/b36507f4e999b295ebab8
>> b1fb47a38f53e2a0e93/geomesa-hbase/geomesa-hbase-tools/src/
>> main/scala/org/locationtech/geomesa/hbase/tools/ingest/HBase
>> BulkLoadCommand.scala#L47
>>
>
> --
> Austin L. Heyne
>
>

Re: I am searching Java examples for HBase 2.0.0

[Ted Yu]

See code HBaseTestingUtility :

  public Connection getConnection() throws IOException {

if (this.connection == null) {

  this.connection = ConnectionFactory.createConnection(this.conf);

Once you have the connection, you can call:

  this.hbaseAdmin = (HBaseAdmin) getConnection().getAdmin();

Code under hbase-server/src/test would be a good place to start searching.

FYI

On Thu, Jun 21, 2018 at 1:52 PM, Andrzej  wrote:

> Hi, I am trying use Java with installed HBase 2.0.0.
> I see, Java for HBase has much more possibilities,like creating tables
> (must not using shell)
> I can find many examples, but all are old (for example using HBaseAdmin)
>
> import java.io.IOException;
> import org.apache.hadoop.conf.Configuration;
> import org.apache.hadoop.hbase.HBaseConfiguration;
> import org.apache.hadoop.hbase.HColumnDescriptor;
> import org.apache.hadoop.hbase.HTableDescriptor;
> import org.apache.hadoop.hbase.TableName;
> import org.apache.hadoop.hbase.client.ConnectionFactory;
> import org.apache.hadoop.hbase.client.ConnectionUtils;
> import org.apache.hadoop.hbase.client.HBaseAdmin;
>
> public class Example {
> public static void mk() throws IOException {
> Configuration configuration = HBaseConfiguration.create();
> configuration.set("hbase.zookeeper.quorum", "127.0.0.1");
> configuration.set("hbase.zookeeper.property.clientPort",
> "2181");
> final TableName tableName = TableName.valueOf("testEclipse
> ");
> final HTableDescriptor desc = new
> HTableDescriptor(tableName);
> desc.addFamily(new HColumnDescriptor("Id"));
> desc.addFamily(new HColumnDescriptor("Name"));
> System.out.println( "Connecting..." );
> HBaseAdmin hbase_admin = new HBaseAdmin( configuration );
> System.out.println( "Creating Table..." );
> hbase_admin.createTable( desc );
> System.out.println("Done!");
> }
>
>public static void main(String[] args) throws IOException {
>   mk();
>}
> }
>
> HTableDescriptor and HColumnDescriptor are deprecated,
> but more important, that HBaseAdmin not works with configuration
> parameter, but with interface ClusterConnection.
> One implementation of ClusterConnection is ConnectionImplementation,
> but I can't write
> HBaseAdmin hbase_admin = new HBaseAdmin(new ConnectionImplementation());
> nor
> org.apache.hadoop.hbase.client.ConnectionImplementation
>
> Where can I find newer examples for version 2.0.0?
> In
> https://github.com/apache/hbase/tree/master/hbase-client/
> src/test/java/org/apache/hadoop/hbase
> in master branch?
> or version is differ than 2.0? Maybe on other branch will for version 2.0?
> I see there is Admin admin = new HBaseAdmin(mockConnection)
> Admin, not HBaseAdmin.
>
> Thanks in advance!
>
> @Previous post: I am going "mvn package"
>

Re: Alternative to native client c++

[Ted Yu]

I executed the following commands:

mvn clean
mvn compile

There was no error.

Andrzej:
Can you tell us which mvn / Java versions you use ?

I use the following:

Java HotSpot(TM) 64-Bit Server VM warning: ignoring option
MaxPermSize=812M; support was removed in 8.0
Apache Maven 3.5.2 (138edd61fd100ec658bfa2d307c43b76940a5d7d;
2017-10-18T00:58:13-07:00)
Maven home: /Users/tyu/apache-maven-3.5.2
Java version: 1.8.0_151, vendor: Oracle Corporation
Java home:
/Library/Java/JavaVirtualMachines/jdk1.8.0_151.jdk/Contents/Home/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "mac os x", version: "10.11.3", arch: "x86_64", family: "mac"

Cheers

On Thu, Jun 21, 2018 at 1:42 PM, Josh Elser  wrote:

> Use `mvn package`, not `compile`.
>
> On 6/21/18 10:41 AM, Andrzej wrote:
>
>> W dniu 21.06.2018 o 19:01, Andrzej pisze:
>>
>>> Is any alternative to fast control HBase from C++ sources?
>>> Or is Java client?
>>> Native Client C++ (HBASE-14850) sources are old and mismatch to folly
>>> library (Futures.h)
>>>
>>>
>> Now I first tried compiling hbase-client in master branch with Maven...
>> next, I try compile whole repository with Maven:
>> command "mvn compile":
>>
>> [INFO] 
>> 
>> [INFO] Reactor Summary:
>> [INFO]
>> [INFO] Apache HBase ... SUCCESS
>> [01:41 min]
>> [INFO] Apache HBase - Checkstyle .. SUCCESS [
>> 0.753 s]
>> [INFO] Apache HBase - Build Support ... SUCCESS [
>> 0.233 s]
>> [INFO] Apache HBase - Error Prone Rules ... SUCCESS [
>> 11.211 s]
>> [INFO] Apache HBase - Annotations . SUCCESS [
>> 0.327 s]
>> [INFO] Apache HBase - Build Configuration . SUCCESS [
>> 0.419 s]
>> [INFO] Apache HBase - Shaded Protocol . SUCCESS [
>> 53.398 s]
>> [INFO] Apache HBase - Common .. SUCCESS [
>> 25.850 s]
>> [INFO] Apache HBase - Metrics API . SUCCESS [
>> 2.835 s]
>> [INFO] Apache HBase - Hadoop Compatibility  SUCCESS [
>> 3.432 s]
>> [INFO] Apache HBase - Metrics Implementation .. SUCCESS [
>> 7.296 s]
>> [INFO] Apache HBase - Hadoop Two Compatibility  SUCCESS [
>> 12.773 s]
>> [INFO] Apache HBase - Protocol  SUCCESS [
>> 18.311 s]
>> [INFO] Apache HBase - Client .. SUCCESS [
>> 10.863 s]
>> [INFO] Apache HBase - Zookeeper ... SUCCESS [
>> 4.104 s]
>> [INFO] Apache HBase - Replication . FAILURE [
>> 2.351 s]
>> [INFO] Apache HBase - Resource Bundle . SKIPPED
>> [INFO] Apache HBase - HTTP  SKIPPED
>> [INFO] Apache HBase - Procedure ... SKIPPED
>> [INFO] Apache HBase - Server .. SKIPPED
>> [INFO] Apache HBase - MapReduce ... SKIPPED
>> [INFO] Apache HBase - Testing Util  SKIPPED
>> [INFO] Apache HBase - Thrift .. SKIPPED
>> [INFO] Apache HBase - RSGroup . SKIPPED
>> [INFO] Apache HBase - Shell ... SKIPPED
>> [INFO] Apache HBase - Coprocessor Endpoint  SKIPPED
>> [INFO] Apache HBase - Backup .. SKIPPED
>> [INFO] Apache HBase - Integration Tests ... SKIPPED
>> [INFO] Apache HBase - Rest  SKIPPED
>> [INFO] Apache HBase - Examples  SKIPPED
>> [INFO] Apache HBase - Shaded .. SKIPPED
>> [INFO] Apache HBase - Shaded - Client (with Hadoop bundled) SKIPPED
>> [INFO] Apache HBase - Shaded - Client . SKIPPED
>> [INFO] Apache HBase - Shaded - MapReduce .. SKIPPED
>> [INFO] Apache HBase - External Block Cache  SKIPPED
>> [INFO] Apache HBase - Spark ... SKIPPED
>> [INFO] Apache HBase - Spark Integration Tests . SKIPPED
>> [INFO] Apache HBase - Assembly  SKIPPED
>> [INFO] Apache HBase Shaded Packaging Invariants ... SKIPPED
>> [INFO] Apache HBase Shaded Packaging Invariants (with Hadoop bundled)
>> SKIPPED
>> [INFO] Apache HBase - Archetypes .. SKIPPED
>> [INFO] Apache HBase - Exemplar for hbase-client archetype . SKIPPED
>> [INFO] Apache HBase - Exemplar for hbase-shaded-client archetype SKIPPED
>> [INFO] Apache HBase - Archetype builder ... SKIPPED
>> [INFO] 
>> 
>> [INFO] BUILD FAILURE
>> [INFO] 
>> 
>> [INFO] Total time: 04:19 min
>> [INFO] Finished at: 2018-06-21T19:37:37+02:00
>> [INFO] Final Memory: 100M/458M
>> 

Re: Native client c++

[Ted Yu]

Please use HBASE-14850 branch which works with hbase 2.0

Cheers

On Tue, Jun 19, 2018 at 7:30 AM, Andrzej  wrote:

> I have written previous my library based on native_client sources in any
> branch. Those worked with HBase 1.3 but now not works with HBase 2.0
> I again want compile my library with sources of native_client.
> I see it in master branch, but its are very old (5 years)
> This sources will works with HBase 2.0 or is new version in any branch?
>
> Andrzej
>

Re: Odd cell result

[Ted Yu]

bq. is it available in version 1.2.6?

If you were talking
about 
hbase-spark/src/main/scala/org/apache/hadoop/hbase/spark/HBaseContext.scala
, it is not in 1.2.6 since hbase-spark module was not part of 1.2.6 release.

FYI

On Mon, Jun 11, 2018 at 2:08 AM, Kang Minwoo 
wrote:

> Thank you for giving me a good way.
> But HBaseContext does not seem to exist in version 1.2.6
> is it available in version 1.2.6?
>
> And the problem that I can not use HBaseContext is that I am using
> CustomTableInputFormat which extends TableInputFormat.
>
> Best regards,
> Minwoo Kang
>
> 
> 보낸 사람: Juan Jose Escobar 
> 보낸 날짜: 2018년 6월 9일 토요일 18:36
> 받는 사람: user@hbase.apache.org
> 제목: Re: Odd cell result
>
> Hello,
>
> Are you trying to read exported files or similar? Otherwise I think you
> need to indicate the format of the data you are reading. I think what you
> want to do is easier like this:
>
> val sparkConf = new SparkConf()
> 
> val sc = new SparkContext(sparkConf)
>
> val conf = HBaseConfiguration.create()
> val hbaseContext = new HBaseContext(sc, conf)
> val scan = new Scan()
> // ... scan config
> val rdd = hbaseContext.hbaseRDD(TableName.valueOf(tableName), scan)
> rdd.count()
>
> or use a Spark-HBase connector which encapsulates the details
>
> Regards
>
>
> On Sat, Jun 9, 2018 at 8:48 AM, Kang Minwoo 
> wrote:
>
> > 1) I am using just InputFormat. (I do not know it is the right answer to
> > the question.)
> >
> > 2) code snippet
> >
> > ```
> > val rdd = sc.newAPIHadoopFile(...)
> > rdd.count()
> > ```
> >
> > 3) hbase version 1.2.6
> >
> > Best regards,
> > Minwoo Kang
> >
> > 
> > 보낸 사람: Ted Yu 
> > 보낸 날짜: 2018년 6월 8일 금요일 20:01
> > 받는 사람: hbase-user
> > 제목: Re: Odd cell result
> >
> > Which connector do you use for Spark 2.1.2 ?
> >
> > Is there any code snippet which may reproduce what you experienced ?
> >
> > Which hbase release are you using ?
> >
> > Thanks
> >
> > On Fri, Jun 8, 2018 at 1:50 AM, Kang Minwoo 
> > wrote:
> >
> > > Hello, Users
> > >
> > > I recently met an unusual situation.
> > > That is the cell result does not contain column family.
> > >
> > > I thought the cell is the smallest unit where data could be transferred
> > in
> > > HBase.
> > > But cell does not contain column family means the cell is not the
> > smallest
> > > unit.
> > > I'm wrong?
> > >
> > > It occurred in Spark 2.1.2 and did not occur in MR.
> > > And now it is not reappearance.
> > >
> > > Best regards,
> > > Minwoo Kang
> > >
> >
>

Re: Odd cell result

[Ted Yu]

Which connector do you use for Spark 2.1.2 ?

Is there any code snippet which may reproduce what you experienced ?

Which hbase release are you using ?

Thanks

On Fri, Jun 8, 2018 at 1:50 AM, Kang Minwoo  wrote:

> Hello, Users
>
> I recently met an unusual situation.
> That is the cell result does not contain column family.
>
> I thought the cell is the smallest unit where data could be transferred in
> HBase.
> But cell does not contain column family means the cell is not the smallest
> unit.
> I'm wrong?
>
> It occurred in Spark 2.1.2 and did not occur in MR.
> And now it is not reappearance.
>
> Best regards,
> Minwoo Kang
>

Re: [ANNOUNCE] New HBase committer Guangxu Cheng

[Ted Yu]

Congratulations, Guangxu!
 Original message From: "张铎(Duo Zhang)"  
Date: 6/4/18  12:00 AM  (GMT-08:00) To: HBase Dev List , 
hbase-user  Subject: [ANNOUNCE] New HBase committer 
Guangxu Cheng 
On behalf of the Apache HBase PMC, I am pleased to announce that Guangxu
Cheng has accepted the PMC's invitation to become a committer on the
project. We appreciate all of Guangxu's generous contributions thus far and
look forward to his continued involvement.

Congratulations and welcome, Guangxu!

Re: Incremental snapshot export

[Ted Yu]

See previous reply.

HBASE-14123 has been integrated to master branch which corresponds to hbase
3.0

FYI

On Tue, May 29, 2018 at 9:02 PM, Manjeet Singh 
wrote:

> Hi All
>
> Does incremental snapshot export from source cluster to destination cluster
> is available..?
> Can anyone suggest me the best way to export/import entire table from
> source cluster to destination cluster (its live system)
>
> Thanks
> Manjeet singh
>
> On Thu, 19 Jan 2017, 06:31 Neelesh,  wrote:
>
> > Thanks Ted!
> >
> > On Wed, Jan 18, 2017 at 9:11 AM, Ted Yu  wrote:
> >
> > > Currently ExportSnapshot utility doesn't support incremental export.
> > > Here is the help message for overwrite:
> > >
> > > static final Option OVERWRITE = new Option(null, "overwrite",
> false,
> > >
> > > "Rewrite the snapshot manifest if already exists.");
> > >
> > > Managing dependencies across snapshots may not be trivial (considering
> > > region split / merge in between snapshots).
> > >
> > > If you are interested, you can watch HBASE-14123 where incremental
> > backup /
> > > restore has solved this problem.
> > >
> > > Cheers
> > >
> > > On Wed, Jan 18, 2017 at 9:04 AM, Neelesh  wrote:
> > >
> > > > Hi,
> > > >   Does the ExportSnapshot utility incrementally export HFiles ? From
> > the
> > > > code, it looks like I can specify an overwrite flag to delete and
> > > recreate
> > > > the output dir, but there is no control on individual HFiles.  This
> is
> > on
> > > > HBase 1.1.2.
> > > >
> > > > I was wondering how difficult would it be to extend ExportSnapshot to
> > > > optionally skip copying HFiles if it already exists, given that
> HFiles
> > in
> > > > the context of snapshots are immutable.
> > > >
> > > >
> > > > Thanks!
> > > > -neelesh
> > > >
> > >
> >
>

Re: MOB integration

[Ted Yu]

regionserver logs on your server would tell you what happened during the
ingestion.

BTW Mob feature is not in 1.1.x releases. You're likely using a vendor's
backport.

There have been continuous improvements to Mob feature since its initial
checkin.
So there may be some difference in the details between the release you use
and hbase 2.0 (which I used to generate the logs I quoted).

On Sat, May 19, 2018 at 11:29 AM, Nicolas Paris <nipari...@gmail.com> wrote:

> 2018-05-19 20:08 GMT+02:00 Ted Yu <yuzhih...@gmail.com>:
>
> > Mob store file is renamed from /apps/hbase/data/mobdir to the final
> > location under region directory for the table.
> >
> > This explains why you don't see much data under mobdir since data
> ingestion
> > has finished.
> >
>
>
> ​Well, I monitored the mobdir folder during ingestion. Nothing happened in
> it.​
> Data were directly going under the table in the region.
>
> BTW, thats why I was thinking the pdf were treated as regular binary files.
> Certainly reading the regionserver logs will help.
>
> Finallly, if hive is able to load mob columns, that's a good news for me.
>
>
>
>
> >
> > Cheers
> >
> > On Sat, May 19, 2018 at 9:58 AM, Nicolas Paris <nipari...@gmail.com>
> > wrote:
> >
> > > Not having access cluster for few days, but I will be looking
> > > to logs.
> > >
> > > However, when looking at your logs, it seems that I mispell
> > > my mlob dir in the first post. It was "mobdir".
> > > The /apps/hbase/data/mobdir/ is nearly empty, sizing 4 or 10 kb
> > >
> > > Would this confirm the mob flushing process wouldn't be activated ?
> > >
> > >
> > >
> > > 2018-05-19 18:38 GMT+02:00 Ted Yu <yuzhih...@gmail.com>:
> > >
> > > > If you have a chance to look at region server log, you would see some
> > > line
> > > > such as the following:
> > > >
> > > > 2018-05-19 16:31:23,548 INFO  [MemStoreFlusher.0]
> > regionserver.HMobStore:
> > > > Renaming flushed file from
> > > > hdfs://mycluster/apps/hbase/data/mobdir/.tmp/
> > > > 28e252d7f013973174750d483d358fa020180519dd8e7c3d67814eb0b5fb
> > 06fb9e800377
> > > > to
> > > > hdfs://mycluster/apps/hbase/data/mobdir/data/default/
> > > > IntegrationTestIngestWithMOB/e9b5d936e7f55a4f1c3246a8d5ce53
> c2/test_cf/
> > > > 28e252d7f013973174750d483d358fa020180519dd8e7c3d67814eb0b5fb
> > 06fb9e800377
> > > >
> > > > Meaning Mob store file is first saved under
> > > /apps/hbase/data/mobdir/.tmp/ ,
> > > > then renamed to under the usual location under region directory for
> the
> > > > table.
> > > >
> > > > From high level, as long as you can query what you ingested, you can
> be
> > > > assured that Mob data is persisted.
> > > >
> > > > Cheers
> > > >
> > > > On Sat, May 19, 2018 at 8:43 AM, Nicolas Paris <nipari...@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi
> > > > >
> > > > > ​Yes the data comes back as expected.
> > > > > My table is not called "mlob" however since I found such folder
> > > > > I thought it was storing mob objects.
> > > > >
> > > > > I do have 500 folder hashed as you mentionned. They contains the
> > > > > whole dataset (2TO)
> > > > > However, how beeing sure the data is actually stored as MOB (and
> not
> > > > > as traditional binary)
> > > > >
> > > > > Thanks
> > > > >
> > > > >
> > > > > 2018-05-19 15:59 GMT+02:00 Ted Yu <yuzhih...@gmail.com>:
> > > > >
> > > > > > bq. look into hdfs hbase/data/mlob
> > > > > >
> > > > > > Is 'mlob' name of your table ?
> > > > > >
> > > > > > bq. nearly empty folder
> > > > > >
> > > > > > Here is listing under a one region table:
> > > > > >
> > > > > > drwxr-xr-x   - hbase hdfs  0 2018-05-16 23:51
> > > > > > /apps/hbase/data/data/default/atlas_janus/.tabledesc
> > > > > > drwxr-xr-x   - hbase hdfs  0 2018-05-16 23:51
> > > > > > /apps/hbase/data/data/default/atlas_janus/.tmp
> > > > > > drwxr-xr-x   - hbase hdfs  0 2018-05-17 00:55

Re: MOB integration

[Ted Yu]

Mob store file is renamed from /apps/hbase/data/mobdir to the final
location under region directory for the table.

This explains why you don't see much data under mobdir since data ingestion
has finished.

Cheers

On Sat, May 19, 2018 at 9:58 AM, Nicolas Paris <nipari...@gmail.com> wrote:

> Not having access cluster for few days, but I will be looking
> to logs.
>
> However, when looking at your logs, it seems that I mispell
> my mlob dir in the first post. It was "mobdir".
> The /apps/hbase/data/mobdir/ is nearly empty, sizing 4 or 10 kb
>
> Would this confirm the mob flushing process wouldn't be activated ?
>
>
>
> 2018-05-19 18:38 GMT+02:00 Ted Yu <yuzhih...@gmail.com>:
>
> > If you have a chance to look at region server log, you would see some
> line
> > such as the following:
> >
> > 2018-05-19 16:31:23,548 INFO  [MemStoreFlusher.0] regionserver.HMobStore:
> > Renaming flushed file from
> > hdfs://mycluster/apps/hbase/data/mobdir/.tmp/
> > 28e252d7f013973174750d483d358fa020180519dd8e7c3d67814eb0b5fb06fb9e800377
> > to
> > hdfs://mycluster/apps/hbase/data/mobdir/data/default/
> > IntegrationTestIngestWithMOB/e9b5d936e7f55a4f1c3246a8d5ce53c2/test_cf/
> > 28e252d7f013973174750d483d358fa020180519dd8e7c3d67814eb0b5fb06fb9e800377
> >
> > Meaning Mob store file is first saved under
> /apps/hbase/data/mobdir/.tmp/ ,
> > then renamed to under the usual location under region directory for the
> > table.
> >
> > From high level, as long as you can query what you ingested, you can be
> > assured that Mob data is persisted.
> >
> > Cheers
> >
> > On Sat, May 19, 2018 at 8:43 AM, Nicolas Paris <nipari...@gmail.com>
> > wrote:
> >
> > > Hi
> > >
> > > ​Yes the data comes back as expected.
> > > My table is not called "mlob" however since I found such folder
> > > I thought it was storing mob objects.
> > >
> > > I do have 500 folder hashed as you mentionned. They contains the
> > > whole dataset (2TO)
> > > However, how beeing sure the data is actually stored as MOB (and not
> > > as traditional binary)
> > >
> > > Thanks
> > >
> > >
> > > 2018-05-19 15:59 GMT+02:00 Ted Yu <yuzhih...@gmail.com>:
> > >
> > > > bq. look into hdfs hbase/data/mlob
> > > >
> > > > Is 'mlob' name of your table ?
> > > >
> > > > bq. nearly empty folder
> > > >
> > > > Here is listing under a one region table:
> > > >
> > > > drwxr-xr-x   - hbase hdfs  0 2018-05-16 23:51
> > > > /apps/hbase/data/data/default/atlas_janus/.tabledesc
> > > > drwxr-xr-x   - hbase hdfs  0 2018-05-16 23:51
> > > > /apps/hbase/data/data/default/atlas_janus/.tmp
> > > > drwxr-xr-x   - hbase hdfs  0 2018-05-17 00:55
> > > > /apps/hbase/data/data/default/atlas_janus/
> > 8033ea259cb7272d43bc137ca0ab29
> > > 06
> > > >
> > > > Not sure if the above matches your description of being nearly empty.
> > > > Here data is stored under 8033ea259cb7272d43bc137ca0ab2906
> > > >
> > > > If you query the table, does the data come back as expected ?
> > > >
> > > > Thanks
> > > >
> > > > On Sat, May 19, 2018 at 12:23 AM, Nicolas Paris <nipari...@gmail.com
> >
> > > > wrote:
> > > >
> > > > > Hi
> > > > >
> > > > >
> > > > > I am using hbase 1.1 and hive 1.2
> > > > >
> > > > > I created an hbase table with a mob column with the default
> > > > > threshold (100K)
> > > > > I mapped the table into hive with a binary format, and loaded
> > > > > 20M of pdf of size between 50k and 20mb
> > > > >
> > > > > Apparently the mob is not populated because when I look into
> > > > > hdfs hbase/data/mlob, it is a nearly empty folder.
> > > > >
> > > > > Does it mean hive cannot populate hbase mob columns  ?
> > > > >
> > > > > Thanks
> > > > >
> > > >
> > >
> >
>

Re: MOB integration

[Ted Yu]

If you have a chance to look at region server log, you would see some line
such as the following:

2018-05-19 16:31:23,548 INFO  [MemStoreFlusher.0] regionserver.HMobStore:
Renaming flushed file from
hdfs://mycluster/apps/hbase/data/mobdir/.tmp/28e252d7f013973174750d483d358fa020180519dd8e7c3d67814eb0b5fb06fb9e800377
to
hdfs://mycluster/apps/hbase/data/mobdir/data/default/IntegrationTestIngestWithMOB/e9b5d936e7f55a4f1c3246a8d5ce53c2/test_cf/28e252d7f013973174750d483d358fa020180519dd8e7c3d67814eb0b5fb06fb9e800377

Meaning Mob store file is first saved under /apps/hbase/data/mobdir/.tmp/ ,
then renamed to under the usual location under region directory for the
table.

>From high level, as long as you can query what you ingested, you can be
assured that Mob data is persisted.

Cheers

On Sat, May 19, 2018 at 8:43 AM, Nicolas Paris <nipari...@gmail.com> wrote:

> Hi
>
> ​Yes the data comes back as expected.
> My table is not called "mlob" however since I found such folder
> I thought it was storing mob objects.
>
> I do have 500 folder hashed as you mentionned. They contains the
> whole dataset (2TO)
> However, how beeing sure the data is actually stored as MOB (and not
> as traditional binary)
>
> Thanks
>
>
> 2018-05-19 15:59 GMT+02:00 Ted Yu <yuzhih...@gmail.com>:
>
> > bq. look into hdfs hbase/data/mlob
> >
> > Is 'mlob' name of your table ?
> >
> > bq. nearly empty folder
> >
> > Here is listing under a one region table:
> >
> > drwxr-xr-x   - hbase hdfs  0 2018-05-16 23:51
> > /apps/hbase/data/data/default/atlas_janus/.tabledesc
> > drwxr-xr-x   - hbase hdfs  0 2018-05-16 23:51
> > /apps/hbase/data/data/default/atlas_janus/.tmp
> > drwxr-xr-x   - hbase hdfs  0 2018-05-17 00:55
> > /apps/hbase/data/data/default/atlas_janus/8033ea259cb7272d43bc137ca0ab29
> 06
> >
> > Not sure if the above matches your description of being nearly empty.
> > Here data is stored under 8033ea259cb7272d43bc137ca0ab2906
> >
> > If you query the table, does the data come back as expected ?
> >
> > Thanks
> >
> > On Sat, May 19, 2018 at 12:23 AM, Nicolas Paris <nipari...@gmail.com>
> > wrote:
> >
> > > Hi
> > >
> > >
> > > I am using hbase 1.1 and hive 1.2
> > >
> > > I created an hbase table with a mob column with the default
> > > threshold (100K)
> > > I mapped the table into hive with a binary format, and loaded
> > > 20M of pdf of size between 50k and 20mb
> > >
> > > Apparently the mob is not populated because when I look into
> > > hdfs hbase/data/mlob, it is a nearly empty folder.
> > >
> > > Does it mean hive cannot populate hbase mob columns  ?
> > >
> > > Thanks
> > >
> >
>

Re: MOB integration

[Ted Yu]

bq. look into hdfs hbase/data/mlob

Is 'mlob' name of your table ?

bq. nearly empty folder

Here is listing under a one region table:

drwxr-xr-x   - hbase hdfs  0 2018-05-16 23:51
/apps/hbase/data/data/default/atlas_janus/.tabledesc
drwxr-xr-x   - hbase hdfs  0 2018-05-16 23:51
/apps/hbase/data/data/default/atlas_janus/.tmp
drwxr-xr-x   - hbase hdfs  0 2018-05-17 00:55
/apps/hbase/data/data/default/atlas_janus/8033ea259cb7272d43bc137ca0ab2906

Not sure if the above matches your description of being nearly empty.
Here data is stored under 8033ea259cb7272d43bc137ca0ab2906

If you query the table, does the data come back as expected ?

Thanks

On Sat, May 19, 2018 at 12:23 AM, Nicolas Paris  wrote:

> Hi
>
>
> I am using hbase 1.1 and hive 1.2
>
> I created an hbase table with a mob column with the default
> threshold (100K)
> I mapped the table into hive with a binary format, and loaded
> 20M of pdf of size between 50k and 20mb
>
> Apparently the mob is not populated because when I look into
> hdfs hbase/data/mlob, it is a nearly empty folder.
>
> Does it mean hive cannot populate hbase mob columns  ?
>
> Thanks
>

Re: How to improve HBase read performance.

[Ted Yu]

bq. store a lot of logs in HBase

Kang:
Can you tell us a bit more about how you store (and access) the log(s) -
size of each log, whether log is encoded in hbase ?

ORC is columnar format while hbase uses different format.

Thanks

On Wed, May 16, 2018 at 6:41 AM, Marcell Ortutay 
wrote:

> This ticket: https://issues.apache.org/jira/browse/HBASE-20459 was fixed
> in
> the latest version of HBase, upgrading to latest may help with performance
>
> On Wed, May 16, 2018 at 3:55 AM, Kang Minwoo 
> wrote:
>
> > Hi, Users.
> >
> > I store a lot of logs in HBase.
> > However, the reading speed of the log is too slow compared to the Hive
> ORC
> > file.
> > I know that HBase is slow compared to the Hive ORC file.
> > The problem is that it is too slow.
> > HBase is about 6 times slower.
> >
> > Is there a good way to speed up HBase's reading speed?
> > Should I put a lot of servers?
> >
> > I am using HBase 1.2.6.
> >
> > Best regards,
> > Minwoo Kang
> >
>

Re: Visibility auths only visible to hbase user

[Ted Yu]

For the open source download, can you tell us which release you downloaded ?

Did you install it on the docker image ?

Please share hbase-site.xml (thru pastebin) if possible.

Thanks

On Sun, May 6, 2018 at 8:04 AM, Mike Thomsen <mikerthom...@gmail.com> wrote:

> Ted,
>
> As I mentioned, I tried this with HDP and a copy downloaded from
> hbase.apache.org which is why I'm scratching my head on this...
>
> Thanks,
>
> Mike
>
> On Sun, May 6, 2018 at 10:58 AM Ted Yu <yuzhih...@gmail.com> wrote:
>
> > Please use vendor forum for vendor specific question(s).
> >
> > To my knowledge, this feature works in Apache hbase releases.
> >
> > Cheers
> >
> > On Sun, May 6, 2018 at 7:55 AM, Mike Thomsen <mikerthom...@gmail.com>
> > wrote:
> >
> > > I've tried this in the HDP docker sandbox and outside that with a basic
> > > installation of HBase.
> > >
> > > su - hbase
> > > hbase shell
> > > set_auths 'test_user', ['OPEN', 'BILLING', 'PII']
> > > get_auths 'test_user'
> > > OPEN
> > > BILLING
> > > PII
> > >
> > > So far so good.
> > >
> > > su - test_user
> > > hbase shell
> > > get_auths 'test_user'
> > > 0 returned
> > >
> > > If su  back to hbase and check, that user reports OPEN, BILLING and PII
> > are
> > > set for test_user.
> > >
> > > Weird thing is I had it working under a previous iteration of running
> the
> > > HDP sandbox, but it won't work in either environment now.
> > >
> > > Any ideas?
> > >
> > > Thanks,
> > >
> > > Mike
> > >
> >
>

Re: Visibility auths only visible to hbase user

[Ted Yu]

Please use vendor forum for vendor specific question(s).

To my knowledge, this feature works in Apache hbase releases.

Cheers

On Sun, May 6, 2018 at 7:55 AM, Mike Thomsen  wrote:

> I've tried this in the HDP docker sandbox and outside that with a basic
> installation of HBase.
>
> su - hbase
> hbase shell
> set_auths 'test_user', ['OPEN', 'BILLING', 'PII']
> get_auths 'test_user'
> OPEN
> BILLING
> PII
>
> So far so good.
>
> su - test_user
> hbase shell
> get_auths 'test_user'
> 0 returned
>
> If su  back to hbase and check, that user reports OPEN, BILLING and PII are
> set for test_user.
>
> Weird thing is I had it working under a previous iteration of running the
> HDP sandbox, but it won't work in either environment now.
>
> Any ideas?
>
> Thanks,
>
> Mike
>

Re: [HELP] on AWS EC2, responseTooSlow and rollback when dropping a table with a lot of regions

[Ted Yu]

For #2, the reason was:

state=ROLLEDBACK exec-time=1mins,59.108sec
exception=org.apache.hadoop.hbase.TableNotFoundException:
mytable

For #3, 179108 corresponded with '1mins,59.108sec' shown above which was
the processing time (> 10,000ms)

I think you only posted part of the master log, right ?
>From 12:39:58 to 12:42:57, did the master log contain more lines ?

Please turn DEBUG log on if you haven't.

On Fri, May 4, 2018 at 6:34 AM, 李响  wrote:

> Dear community,
>
> Really appreciate it if someone could provide some guides on how to debug
> this.
>
> We are using HBase 1.1.9 ( + rsgroup patch) on AWS EC2 instances, and I
> found it is quite slow to drop a table with a lot of regions (300+).
>
> The Master's log shows:
>
> 2018-04-27 12:39:58,483 INFO
> [B.defaultRpcServer.handler=11,queue=2,port=16000] master.HMaster:
> Client=ec2-user/null delete mytable
> 2018-04-27 12:39:58,737 INFO  [ProcedureExecutorThread-0]
> hbase.MetaTableAccessor: Deleted [{ENCODED =>
> 0b3a3c3e991b38dd0921905f7200f094, NAME =>
> 'mytable,,1524805200089.0b3a3c3e991b38dd0921905f7200f094.', STARTKEY =>
> '',
> ENDKEY => '01-000-##'}, {ENCODED =>
> 5fba610c04629121e53fec53af869fe3, NAME =>
> 'mytable,01-000-##,1524805200089.
> 5fba610c04629121e53fec53af869fe3.',
> STARTKEY => '01-000-##', ENDKEY =>
> '02-000-##'}, {ENCODED =>
> 653dbf4a1aba1fedb56fea9522d7c96a, NAME =>
> 'mytable,02-000-##,1524805200089.
> 653dbf4a1aba1fedb56fea9522d7c96a.',
> STARTKEY => '02-000-##', ENDKEY =>
> '03-000-##'}, {ENCODED =>
> 7ba6bce2e7cc53a9bcc234244c835ac8, NAME =>
> 'mytable,03-000-##,1524805200089.
> 7ba6bce2e7cc53a9bcc234244c835ac8.',
> STARTKEY => '03-000-##', ENDKEY =>
> '04-000-##'}, {ENCODED =>
> 5106e0ca76d2c026d2d8ff5ce59316cc,...
> 2018-04-27 12:40:58,687 INFO
> [B.defaultRpcServer.handler=20,queue=2,port=16000] master.HMaster:
> Client=ec2-user/null delete mytable
> 2018-04-27 12:42:57,592 WARN
> [B.defaultRpcServer.handler=11,queue=2,port=16000] ipc.RpcServer:
> (responseTooSlow):
> {"call":"DeleteTable(org.apache.hadoop.hbase.protobuf.
> generated.MasterProtos$DeleteTableRequest)","starttimems":1524832798483,"
> responsesize":4,"method":"DeleteTable","param":"TODO:
> class
> org.apache.hadoop.hbase.protobuf.generated.MasterProtos$
> DeleteTableRequest","processingtimems":179108,"client":"
> 10.22.22.236:45640","queuetimems":0,"class":"HMaster"}
> 2018-04-27 12:42:57,924 INFO [ProcedureExecutorThread-3]
> procedure2.ProcedureExecutor: Rolledback procedure DeleteTableProcedure
> (table=mytable) id=16515 owner=ec2-user state=ROLLEDBACK exec-time=1mins,
> 59.108sec exception=org.apache.hadoop.hbase.TableNotFoundException:
> mytable
>
>
> The stack trace is like
> Caused by:
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hbase.
> TableNotFoundException):
> mytable
> at
> org.apache.hadoop.hbase.master.HMaster.checkTableModifiable(HMaster.
> java:2216)
> at
> org.apache.hadoop.hbase.master.procedure.DeleteTableProcedure.
> prepareDelete(DeleteTableProcedure.java:255)
> at
> org.apache.hadoop.hbase.master.procedure.DeleteTableProcedure.
> executeFromState(DeleteTableProcedure.java:101)
> ...
>
>
> It is quite weird for me that:
> 1. Why is the following message logged twice (with different handlers: 20
> and 11) ?  Does it means MasterProcedureUtil.NonceProcedureRunnable#run()
> is called twice and handled by different handlers?
>INFO [B.defaultRpcServer.handler=20(or 11),queue=2,port=16000]
> master.HMaster: Client=ec2-user/null delete mytable"
>
> 2. The rollback is triggered by the warning of "responseTooSlow" ?
>
> 3. I set hbase.ipc.warn.response.time to take the default value (10,000
> ms), why the responseTooSlow warning is reported with
> "processingtimems":179108
> ?
>
> 4. How could I debug further?
> --
>
>李响 Xiang Li
>
> 邮件 e-mail  ：wate...@gmail.com
>

Re: hbase scan with column family filter - limit results per column prefix

[Ted Yu]

>From your description, you can combine ColumnPrefixFilter with PageFilter
(thru FilterList).

FYI


On Tue, May 1, 2018 at 6:06 AM, mrmiroslav  wrote:

> I'd like to perform Get / Scan with java client.
>
> In a given column family I'd like to limit the number of results per given
> column qualifier prefix.
>
> Is there a way of achieving this without redesigning schema and moving
> Column Qualifiers into Column Families?
>
> Column Qualifiers are dynamic, however they can be grouped by prefix. I'd
> like to limit the number of cells returned from prefix / group?
>
>
>
> --
> Sent from: http://apache-hbase.679495.n3.nabble.com/HBase-User-
> f4020416.html
>

Re: Should Taking A Snapshot Work Even If Balancer Is Moving A Few Regions Around?

[Ted Yu]

Looking at
hbase-client/src/main/java/org/apache/hadoop/hbase/client/Admin.java in
branch-1.4 :

  boolean[] setSplitOrMergeEnabled(final boolean enabled, final boolean
synchronous,
   final MasterSwitchType... switchTypes)
throws IOException;

  boolean isSplitOrMergeEnabled(final MasterSwitchType switchType) throws
IOException;

Please also see the following script:

hbase-shell/src/main/ruby/shell/commands/splitormerge_switch.rb

FYI

On Wed, Mar 21, 2018 at 11:33 AM, Vladimir Rodionov 
wrote:

> >>So my question is whether taking a snapshot is supposed to work even with
> >>regions being moved around. In our case it is usually only a couple here
> >>and there.
>
> No, if region was moved, split or merged during snapshot operation -
> snapshot will fail.
> This is why taking snapshots on a large table is a 50/50 game.
>
> Disabling balancer,region merging and split before snapshot should help.
> This works in 2.0
>
> Not sure if merge/split switch is available in 1.4
>
> -Vlad
>
> On Tue, Mar 20, 2018 at 8:00 PM, Saad Mufti  wrote:
>
> > Hi,
> >
> > We are using HBase 1.4.0 on AWS EMR based Hbase. Since snapshots are in
> S3,
> > they take much longer than when using local disk. We have a cron script
> to
> > take regular snapshots as backup, and they fail quite often on our
> largest
> > table which takes close to an hour to complete the snapshot.
> >
> > The only thing I have noticed in the errors usually is a message about
> the
> > region moving or closing.
> >
> > So my question is whether taking a snapshot is supposed to work even with
> > regions being moved around. In our case it is usually only a couple here
> > and there.
> >
> > Thanks.
> >
> > 
> > Saad
> >
>

Re: Balance Regions Faster

[Ted Yu]

Saad:
You didn't mention the version of hbase you are using.
Please check the version to see if the following were included:

HBASE-18164 Fast locality computation in balancer
HBASE-16570 Compute region locality in parallel at startup
HBASE-15515 Improve LocalityBasedCandidateGenerator in Balancer

Cheers

On Tue, Mar 20, 2018 at 6:11 PM, Ted Yu <yuzhih...@gmail.com> wrote:

> Please consider tuning the following parameters of stochastic load
> balancer :
>
> "hbase.master.balancer.stochastic.maxRunningTime"
>
> default value is 30 seconds. It controls the duration of runtime for each 
> balanceCluster()
> call.
>
> "hbase.balancer.period"
>
> default is 300 seconds. It controls the maximum time master runs balancer
> for.
>
> You can turn on DEBUG logging and observe the following output in master
> log:
>
> balancer.StochasticLoadBalancer: Finished computing new load balance plan.  
> Computation took 1200227ms to try 2254 different iterations.  Found a 
> solution that moves 550 regions; Going from a computed cost of 
> 77.52829271038965 to a new cost of 74.32764924425548
>
> If you have a dev cluster, you can try different combinations of the above
> two parameters and get best performance by checking the above log.
>
> Cheers
>
> On Tue, Mar 20, 2018 at 5:18 PM, Saad Mufti <saad.mu...@gmail.com> wrote:
>
>> Hi,
>>
>> We are using the stochastic load balancer, and have tuned it to do a
>> maximum of 1% of regions in any calculation. But it is way too
>> conservative
>> after that, it moves one region at a time. Is there a way to tell it to go
>> faster with whatever number of regions it decided to do? I have been
>> looking at the settings in the code but so far a little confused about
>> which exact setting will achieve this. Is it one of the steps settings?
>>
>> Also our cluster was in bad shape for a while as a bunch of region servers
>> aborted for different reasons. When we stopped everything and brought it
>> back up and enabled the tables, most of the region servers were assigned
>> no
>> regions. Is it for locality reasons that HBase is trying to assign regions
>> where they were assigned before? Is there a way to tell HBase to ignore
>> that on startup?
>>
>> Thanks.
>>
>> 
>> Saad
>>
>
>

Re: Balance Regions Faster

[Ted Yu]

Please consider tuning the following parameters of stochastic load balancer
:

"hbase.master.balancer.stochastic.maxRunningTime"

default value is 30 seconds. It controls the duration of runtime for
each balanceCluster()
call.

"hbase.balancer.period"

default is 300 seconds. It controls the maximum time master runs balancer
for.

You can turn on DEBUG logging and observe the following output in master
log:

balancer.StochasticLoadBalancer: Finished computing new load balance
plan.  Computation took 1200227ms to try 2254 different iterations.
Found a solution that moves 550 regions; Going from a computed cost of
77.52829271038965 to a new cost of 74.32764924425548

If you have a dev cluster, you can try different combinations of the above
two parameters and get best performance by checking the above log.

Cheers

On Tue, Mar 20, 2018 at 5:18 PM, Saad Mufti  wrote:

> Hi,
>
> We are using the stochastic load balancer, and have tuned it to do a
> maximum of 1% of regions in any calculation. But it is way too conservative
> after that, it moves one region at a time. Is there a way to tell it to go
> faster with whatever number of regions it decided to do? I have been
> looking at the settings in the code but so far a little confused about
> which exact setting will achieve this. Is it one of the steps settings?
>
> Also our cluster was in bad shape for a while as a bunch of region servers
> aborted for different reasons. When we stopped everything and brought it
> back up and enabled the tables, most of the region servers were assigned no
> regions. Is it for locality reasons that HBase is trying to assign regions
> where they were assigned before? Is there a way to tell HBase to ignore
> that on startup?
>
> Thanks.
>
> 
> Saad
>

Re: TableSnapshotInputFormat Behavior In HBase 1.4.0

[Ted Yu]

Saad:
I encourage you to open an HBase JIRA outlining your use case and the
config knobs you added through a patch.

We can see the details for each config and make recommendation accordingly.

Thanks

On Mon, Mar 12, 2018 at 8:43 AM, Saad Mufti  wrote:

> I have create a company specific branch and added 4 new flags to control
> this behavior, these gave us a huge performance boost when running Spark
> jobs on snapshots of very large tables in S3. I tried to do everything
> cleanly but
>
> a) not being familiar with the whole test strategies I haven't had time to
> add any useful tests, though of course I left the default behavior the
> same, and a lot of the behavior I control wit these flags only affect
> performance, not the final result, so I would need some pointers on how to
> add useful tests
> b) I added a new flag to be an overall override for prefetch behavior that
> overrides any setting even in the column family descriptor, not sure if
> what I did was entirely in the spirit of what HBase does
>
> Again these if used properly would only impact jobs using
> TableSnapshotInputFormat in their Spark or M-R jobs. Would someone from the
> core team be willing to look at my patch? I have never done this before, so
> would appreciate a quick pointer on how to send a patch and get some quick
> feedback.
>
> Cheers.
>
> 
> Saad
>
>
>
> On Sat, Mar 10, 2018 at 9:56 PM, Saad Mufti  wrote:
>
> > The question remain though of why it is even accessing a column family's
> > files that should be excluded based on the Scan. And that column family
> > does NOT specify prefetch on open in its schema. Only the one we want to
> > read specifies prefetch on open, which we want to override if possible
> for
> > the Spark job.
> >
> > 
> > Saad
> >
> >
> > On Sat, Mar 10, 2018 at 9:51 PM, Saad Mufti 
> wrote:
> >
> >> See below more I found on item 3.
> >>
> >> Cheers.
> >>
> >> 
> >> Saad
> >>
> >> On Sat, Mar 10, 2018 at 7:17 PM, Saad Mufti 
> wrote:
> >>
> >>> Hi,
> >>>
> >>> I am running a Spark job (Spark 2.2.1) on an EMR cluster in AWS. There
> >>> is no Hbase installed on the cluster, only HBase libs linked to my
> Spark
> >>> app. We are reading the snapshot info from a HBase folder in S3 using
> >>> TableSnapshotInputFormat class from HBase 1.4.0 to have the Spark job
> read
> >>> snapshot info directly from the S3 based filesystem instead of going
> >>> through any region server.
> >>>
> >>> I have observed a few behaviors while debugging performance that are
> >>> concerning, some we could mitigate and other I am looking for clarity
> on:
> >>>
> >>> 1)  the TableSnapshotInputFormatImpl code is trying to get locality
> >>> information for the region splits, for a snapshots with a large number
> of
> >>> files (over 35 in our case) this causing single threaded scan of
> all
> >>> the file listings in a single thread in the driver. And it was useless
> >>> because there is really no useful locality information to glean since
> all
> >>> the files are in S3 and not HDFS. So I was forced to make a copy of
> >>> TableSnapshotInputFormatImpl.java in our code and control this with a
> >>> config setting I made up. That got rid of the hours long scan, so I am
> good
> >>> with this part for now.
> >>>
> >>> 2) I have set a single column family in the Scan that I set on the
> hbase
> >>> configuration via
> >>>
> >>> scan.addFamily(str.getBytes()))
> >>>
> >>> hBaseConf.set(TableInputFormat.SCAN, convertScanToString(scan))
> >>>
> >>>
> >>> But when this code is executing under Spark and I observe the threads
> >>> and logs on Spark executors, I it is reading from S3 files for a column
> >>> family that was not included in the scan. This column family was
> >>> intentionally excluded because it is much larger than the others and
> so we
> >>> wanted to avoid the cost.
> >>>
> >>> Any advice on what I am doing wrong would be appreciated.
> >>>
> >>> 3) We also explicitly set caching of blocks to false on the scan,
> >>> although I see that in TableSnapshotInputFormatImpl.java it is again
> >>> set to false internally also. But when running the Spark job, some
> >>> executors were taking much longer than others, and when I observe their
> >>> threads, I see periodic messages about a few hundred megs of RAM used
> by
> >>> the block cache, and the thread is sitting there reading data from S3,
> and
> >>> is occasionally blocked a couple of other threads that have the
> >>> "hfile-prefetcher" name in them. Going back to 2) above, they seem to
> be
> >>> reading the wrong column family, but in this item I am more concerned
> about
> >>> why they appear to be prefetching blocks and caching them, when the
> Scan
> >>> object has a setting to not cache blocks at all?
> >>>
> >>
> >> I think I figured out item 3, the column family descriptor for the table
> >> in question has prefetch on open set in its schema. Now for 

Re: MultipleColumnPrefixFilter issues with SingleColumnValueFilter

[Ted Yu]

I looked at git log for MultipleColumnPrefixFilter - there has been no more
fix since 1.2.0

Can you reproduce what you observed using a unit test ?

It would be easier to understand the scenario through unit test.

Cheers

On Thu, Mar 1, 2018 at 9:57 PM, Vikash Agarwal 
wrote:

> Hi Team,
>
>
> Currently I am trying to use MultipleColumnPrefixFilter along with
> SingleColumnValueFilter and the results  that I am getting seems to be
> problematic.
>
> I have an hbase table "test_messages" with column family  "main" and Column
> Qualifiers as : "pa", "cs" ,"eps","cst","ver","nm","parl".
> Assume all the row Keys of the table  has all the above columns.
>
>
> Now When I am creating Filters as below:
>
> FilterList temp=new FilterList(FilterList.Operator.MUST_PASS_ALL);
>
> SingleColumnValueFilter filter1= new SingleColumnValueFilter("main"
> ,"pa",CompareFilter.CompareOp.LESS_OR_EQUAL, Bytes.toBytes("SomeValue"));
> filter1 .setFilterIfMissing(true);
>
> SingleColumnValueFilter filter2= new SingleColumnValueFilter("main"
> ,"cs",CompareFilter.CompareOp.EQUAL, Bytes.toBytes("SomeValue"));
> filter2.setFilterIfMissing(true);
>
>
> SingleColumnValueFilter filter3= new SingleColumnValueFilter("main"
> ,eps",CompareFilter.CompareOp.EQUAL, Bytes.toBytes("SomeValue"));
> filter3.setFilterIfMissing(true);
>
> temp.addFilter(filter1);
>
> temp.addFilter(filter2);
>
> temp.addFilter(filter3);
>
> Case 1:
>
> MultipleColumnPrefixFilter cpfxf=new MultipleColumnPrefixFilter(new
> byte[][]{ Bytes.toBytes("pa"),Bytes.toBytes("cs"),Bytes.toBytes("eps")});
> temp.addFilter(cpfxf); The above filters results in proper extratcion of
> data from Hbase. Now if I create a MultipleColumnPrefixFilter as below:
> Case 2:
> Consider we have filter1,2,3 and below filter
>
> MultipleColumnPrefixFilter cpfxf=new MultipleColumnPrefixFilter(new
> byte[][]{ Bytes.toBytes("pa"),Bytes.toBytes("eps")}); //Note "cs" is
> removed temp.addFilter(cpfxf); I don't get any data However if I create a
> filter as Case 3:
>
> Consider we have filter1,2,3 and below filter
>
> MultipleColumnPrefixFilter cpfxf=new MultipleColumnPrefixFilter(new
> byte[][]{ Bytes.toBytes("pa"),Bytes.toBytes("cs")}); // Note "eps" is
> removed and "cs" is added back. temp.addFilter(cpfxf); I get lot of data.
> EIther both Case 2 and Case 3 should give data or none of them. Can someone
> Explain me the reason for this to happen ? Or is there a bug in
> MultipleColumnPrefixFilter .
>
> Hbase Version: 1.2.0
>
>
>
> --
> Vikash Agarwal
>

Re: How to get bulk delete [Hbase1.2.0]

[Ted Yu]

For #2, BulkDeleteEndpoint still exists - in hbase-examples branch.

./hbase-examples/src/main/java/org/apache/hadoop/hbase/coprocessor/example/BulkDeleteEndpoint.java

On Thu, Mar 1, 2018 at 10:02 PM, Vikash Agarwal 
wrote:

> Hi Team,
>
> I am looking for  a bulk delete in Hbase.
>
> I am looking here for two things:
>
> 1. In case I provide a Row key Prefix , Is it possible to delete all the
> rows with that prefix.
>
> 2. In case Where we can scan and delete and the server End. Something like
> BulkDeleteEndpoint which uses to exist in old hbase version 0.94.
>
>
>
> --
> Vikash Agarwal
>

Re: Bucket Cache Failure In HBase 1.3.1

[Ted Yu]

Did the vendor say whether the patch is for hbase or some other component ?

Thanks

On Wed, Feb 28, 2018 at 6:33 PM, Saad Mufti <saad.mu...@gmail.com> wrote:

> Thanks for the feedback, so you guys are right the bucket cache is getting
> disabled due to too many I/O errors from the underlying files making up the
> bucket cache. Still do not know the exact underlying cause, but we are
> working with our vendor to test a patch they provided that seems to have
> resolved the issue for now. They say if it works out well they will
> eventually try to promote the patch to the open source versions.
>
> Cheers.
>
> 
> Saad
>
>
> On Sun, Feb 25, 2018 at 11:10 AM, Ted Yu <yuzhih...@gmail.com> wrote:
>
> > Here is related code for disabling bucket cache:
> >
> > if (this.ioErrorStartTime > 0) {
> >
> >   if (cacheEnabled && (now - ioErrorStartTime) > this.
> > ioErrorsTolerationDuration) {
> >
> > LOG.error("IO errors duration time has exceeded " +
> > ioErrorsTolerationDuration +
> >
> >   "ms, disabling cache, please check your IOEngine");
> >
> > disableCache();
> >
> > Can you search in the region server log to see if the above occurred ?
> >
> > Was this server the only one with disabled cache ?
> >
> > Cheers
> >
> > On Sun, Feb 25, 2018 at 6:20 AM, Saad Mufti <saad.mu...@oath.com.invalid
> >
> > wrote:
> >
> > > HI,
> > >
> > > I am running an HBase 1.3.1 cluster on AWS EMR. The bucket cache is
> > > configured to use two attached EBS disks of 50 GB each and I
> provisioned
> > > the bucket cache to be a bit less than the total, at a total of 98 GB
> per
> > > instance to be on the safe side. My tables have column families set to
> > > prefetch on open.
> > >
> > > On some instances during cluster startup, the bucket cache starts
> > throwing
> > > errors, and eventually the bucket cache gets completely disabled on
> this
> > > instance. The instance still stays up as a valid region server and the
> > only
> > > clue in the region server UI is that the bucket cache tab reports a
> count
> > > of 0, and size of 0 bytes.
> > >
> > > I have already opened a ticket with AWS to see if there are problems
> with
> > > the EBS volumes, but wanted to tap the open source community's
> hive-mind
> > to
> > > see what kind of problem would cause the bucket cache to get disabled.
> If
> > > the application depends on the bucket cache for performance, wouldn't
> it
> > be
> > > better to just remove that region server from the pool if its bucket
> > cache
> > > cannot be recovered/enabled?
> > >
> > > The error look like the following. Would appreciate any insight, thank:
> > >
> > > 2018-02-25 01:12:47,780 ERROR [hfile-prefetch-1519513834057]
> > > bucket.BucketCache: Failed reading block
> > > 332b0634287f4c42851bc1a55ffe4042_1348128 from bucket cache
> > > java.nio.channels.ClosedByInterruptException
> > > at
> > > java.nio.channels.spi.AbstractInterruptibleChannel.end(
> > > AbstractInterruptibleChannel.java:202)
> > > at sun.nio.ch.FileChannelImpl.readInternal(FileChannelImpl.
> > > java:746)
> > > at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:727)
> > > at
> > > org.apache.hadoop.hbase.io.hfile.bucket.FileIOEngine$
> > > FileReadAccessor.access(FileIOEngine.java:219)
> > > at
> > > org.apache.hadoop.hbase.io.hfile.bucket.FileIOEngine.
> > > accessFile(FileIOEngine.java:170)
> > > at
> > > org.apache.hadoop.hbase.io.hfile.bucket.FileIOEngine.
> > > read(FileIOEngine.java:105)
> > > at
> > > org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.
> > > getBlock(BucketCache.java:492)
> > > at
> > > org.apache.hadoop.hbase.io.hfile.CombinedBlockCache.
> > > getBlock(CombinedBlockCache.java:84)
> > > at
> > > org.apache.hadoop.hbase.io.hfile.HFileReaderV2.
> > > getCachedBlock(HFileReaderV2.java:279)
> > > at
> > > org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(
> > > HFileReaderV2.java:420)
> > > at
> > > org.apache.hadoop.hbase.io.hfile.HFileReaderV2$1.run(
> > > HFileReaderV2.java:209)
> > > at
> > > java.util.concurrent.Executors$RunnableAdapter.
> call(Execu

Re: How Long Will HBase Hold A Row Write Lock?

[Ted Yu]

bq. timing out trying to obtain write locks on rows in that region.

Can you confirm that the region under contention was the one being major
compacted ?

Can you pastebin thread dump so that we can have better idea of the
scenario ?

For the region being compacted, how long would the compaction take (just
want to see if there was correlation between this duration and timeout) ?

Cheers

On Wed, Feb 28, 2018 at 6:31 PM, Saad Mufti  wrote:

> Hi,
>
> We are running on Amazon EMR based HBase 1.4.0 . We are currently seeing a
> situation where sometimes a particular region gets into a situation where a
> lot of write requests to any row in that region timeout saying they failed
> to obtain a lock on a row in a region and eventually they experience an IPC
> timeout. This causes the IPC queue to blow up in size as requests get
> backed up, and that region server experiences a much higher than normal
> timeout rate for all requests, not just those timing out for failing to
> obtain the row lock.
>
> The strange thing is the rows are always different but the region is always
> the same. So the question is, is there a region component to how long a row
> write lock would be held? I looked at the debug dump and the RowLocks
> section shows a long list of write row locks held, all of them are from the
> same region but different rows.
>
> Will trying to obtain a write row lock experience delays if no one else
> holds a lock on the same row but the region itself is experiencing read
> delays? We do have an incremental compaction tool running that major
> compacts one region per region server at a time, so that will drive out
> pages from the bucket cache. But for most regions the impact is
> transitional until the bucket cache gets populated by pages from the new
> HFile. But for this one region we start timing out trying to obtain write
> locks on rows in that region.
>
> Any insight anyone can provide would be most welcome.
>
> Cheers.
>
> 
> Saad
>

Re: Incrementally load HFiles outside of MR/Spark

[Ted Yu]

You can refer to HFilePerformanceEvaluation where creation of Writer is
demonstrated:

  writer = HFile.getWriterFactoryNoCache(conf)

  .withPath(fs, mf)

  .withFileContext(hFileContext)

  .withComparator(CellComparator.getInstance())

  .create();

Cheers

On Sun, Feb 25, 2018 at 12:42 PM, Mike Thomsen 
wrote:

> I'm looking into creating HFiles directly from NiFi using the HBase API. It
> seems pretty straight forward:
>
> 1. Open a HFile.Writer pointing to a file path in HDFS.
> 2. Write the cells with the HFile API.
> 3. Call the incremental loader API to have it tell HBase to load the
> generated segments.
>
> Is that right? If so, are there any gotchas that I should be aware of?
>
> Thanks,
>
> Mike
>

Re: Bucket Cache Failure In HBase 1.3.1

[Ted Yu]

Here is related code for disabling bucket cache:

if (this.ioErrorStartTime > 0) {

  if (cacheEnabled && (now - ioErrorStartTime) > this.
ioErrorsTolerationDuration) {

LOG.error("IO errors duration time has exceeded " +
ioErrorsTolerationDuration +

  "ms, disabling cache, please check your IOEngine");

disableCache();

Can you search in the region server log to see if the above occurred ?

Was this server the only one with disabled cache ?

Cheers

On Sun, Feb 25, 2018 at 6:20 AM, Saad Mufti 
wrote:

> HI,
>
> I am running an HBase 1.3.1 cluster on AWS EMR. The bucket cache is
> configured to use two attached EBS disks of 50 GB each and I provisioned
> the bucket cache to be a bit less than the total, at a total of 98 GB per
> instance to be on the safe side. My tables have column families set to
> prefetch on open.
>
> On some instances during cluster startup, the bucket cache starts throwing
> errors, and eventually the bucket cache gets completely disabled on this
> instance. The instance still stays up as a valid region server and the only
> clue in the region server UI is that the bucket cache tab reports a count
> of 0, and size of 0 bytes.
>
> I have already opened a ticket with AWS to see if there are problems with
> the EBS volumes, but wanted to tap the open source community's hive-mind to
> see what kind of problem would cause the bucket cache to get disabled. If
> the application depends on the bucket cache for performance, wouldn't it be
> better to just remove that region server from the pool if its bucket cache
> cannot be recovered/enabled?
>
> The error look like the following. Would appreciate any insight, thank:
>
> 2018-02-25 01:12:47,780 ERROR [hfile-prefetch-1519513834057]
> bucket.BucketCache: Failed reading block
> 332b0634287f4c42851bc1a55ffe4042_1348128 from bucket cache
> java.nio.channels.ClosedByInterruptException
> at
> java.nio.channels.spi.AbstractInterruptibleChannel.end(
> AbstractInterruptibleChannel.java:202)
> at sun.nio.ch.FileChannelImpl.readInternal(FileChannelImpl.
> java:746)
> at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:727)
> at
> org.apache.hadoop.hbase.io.hfile.bucket.FileIOEngine$
> FileReadAccessor.access(FileIOEngine.java:219)
> at
> org.apache.hadoop.hbase.io.hfile.bucket.FileIOEngine.
> accessFile(FileIOEngine.java:170)
> at
> org.apache.hadoop.hbase.io.hfile.bucket.FileIOEngine.
> read(FileIOEngine.java:105)
> at
> org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.
> getBlock(BucketCache.java:492)
> at
> org.apache.hadoop.hbase.io.hfile.CombinedBlockCache.
> getBlock(CombinedBlockCache.java:84)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2.
> getCachedBlock(HFileReaderV2.java:279)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(
> HFileReaderV2.java:420)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$1.run(
> HFileReaderV2.java:209)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$
> ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(
> ScheduledThreadPoolExecutor.java:293)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
>
> and
>
> 2018-02-25 01:12:52,432 ERROR [regionserver/
> ip-xx-xx-xx-xx.xx-xx-xx.us-east-1.ec2.xx.net/xx.xx.xx.xx:
> 16020-BucketCacheWriter-7]
> bucket.BucketCache: Failed writing to bucket cache
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.FileChannelImpl.ensureOpen(FileChannelImpl.java:110)
> at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:758)
> at
> org.apache.hadoop.hbase.io.hfile.bucket.FileIOEngine$
> FileWriteAccessor.access(FileIOEngine.java:227)
> at
> org.apache.hadoop.hbase.io.hfile.bucket.FileIOEngine.
> accessFile(FileIOEngine.java:170)
> at
> org.apache.hadoop.hbase.io.hfile.bucket.FileIOEngine.
> write(FileIOEngine.java:116)
> at
> org.apache.hadoop.hbase.io.hfile.bucket.BucketCache$
> RAMQueueEntry.writeToCache(BucketCache.java:1357)
> at
> org.apache.hadoop.hbase.io.hfile.bucket.BucketCache$WriterThread.doDrain(
> BucketCache.java:883)
> at
> org.apache.hadoop.hbase.io.hfile.bucket.BucketCache$
> WriterThread.run(BucketCache.java:838)
> at java.lang.Thread.run(Thread.java:748)
>
> and later
> 2018-02-25 01:13:47,783 INFO  [regionserver/
> ip-10-194-246-70.aolp-ds-dev.us-east-1.ec2.aolcloud.net/10.
> 194.246.70:16020-BucketCacheWriter-4]
> bucket.BucketCach
> e: 

Re: New installation of HBase ignoring cell visibility (even though it's active)

[Ted Yu]

bq. a warning message in the shell should be displayed if simple auth and
cell visibility are in use together.

Makes sense.

Please log a JIRA.

On Sat, Feb 24, 2018 at 9:06 AM, Mike Thomsen <mikerthom...@gmail.com>
wrote:

> Ted/Anoop,
>
> I realized what the problem was. When I installed HBase previously (and had
> this working) it was on a Linux machine with a package that created the
> hbase superuser and loaded the services with that. I was starting HBase on
> the Mac my company just gave me and was using my user account. Ergo, I was
> scanning as the superuser...
>
> Sorry about that. I'd suggest for newbs like me that a warning message in
> the shell should be displayed if simple auth and cell visibility are in use
> together.
>
> BTW, the reason I have been trying to get this work is that I'm working on
> a patch for NiFi to integrate visibility label support.
>
> Thanks,
>
> Mike
>
> On Sat, Feb 24, 2018 at 10:51 AM, Ted Yu <yuzhih...@gmail.com> wrote:
>
> > I noted that SIMPLE_AUTHENTICATION was returned.
> > Here is related code for getSecurityCapabilities():
> >
> >   if (User.isHBaseSecurityEnabled(master.getConfiguration())) {
> >
> > capabilities.add(SecurityCapabilitiesResponse.Capability.
> > SECURE_AUTHENTICATION);
> >
> >   } else {
> >
> > capabilities.add(SecurityCapabilitiesResponse.Capability.
> > SIMPLE_AUTHENTICATION);
> >
> >   }
> > Did "hbase.security.authentication" have value of "kerberos" ?
> >
> > If it does, please pastebin your hbase-site.xml
> >
> > Thanks
> >
> > On Sat, Feb 24, 2018 at 4:59 AM, Mike Thomsen <mikerthom...@gmail.com>
> > wrote:
> >
> > > (hbase-site.xml is attached)
> > >
> > > I reinstalled HBase on my development machine and the console shows
> that
> > > it's just ignoring the cell visibility settings even though it shows
> > > they're active:
> > >
> > > hbase(main):001:0> list
> > > TABLE
> > >
> > >
> > >
> > > 0 row(s) in 0.1630 seconds
> > >
> > > => []
> > > hbase(main):002:0> get_auths "michaelthomsen"
> > > 0 row(s) in 10.1940 seconds
> > >
> > > hbase(main):003:0> add_labels ["U", "PII", "PHI"]
> > > 0 row(s) in 0.1900 seconds
> > >
> > > hbase(main):004:0> set_auths "michaelthomsen",  ["U", "PII", "PHI"]
> > > 0 row(s) in 0.0500 seconds
> > >
> > > hbase(main):005:0> get_auths "michaelthomsen"
> > > U
> > >
> > >
> > >
> > > PII
> > >
> > >
> > >
> > > PHI
> > >
> > >
> > >
> > > 0 row(s) in 0.0470 seconds
> > >
> > > hbase(main):006:0> create 'test', {NAME => 'prop' }
> > > 0 row(s) in 1.2780 seconds
> > >
> > > => Hbase::Table - test
> > > hbase(main):007:0> put "test", "user1", "prop:name", "John Smith"
> > > 0 row(s) in 0.0500 seconds
> > >
> > > hbase(main):008:0> scan "test"
> > > ROW
> > > COLUMN+CELL
> > >
> > >
> > >  user1 column=prop:name,
> > > timestamp=1519476818510, value=John Smith
> > >
> > > 1 row(s) in 0.0260 seconds
> > >
> > > hbase(main):009:0> set_visibility "test", "PII", { COLUMNS => "prop" }
> > > 1 row(s) in 0.0130 seconds
> > >
> > > hbase(main):010:0> scan "test", { AUTHORIZATIONS => ["PHI", "U"] }
> > > ROW
> > > COLUMN+CELL
> > >
> > >
> > >  user1 column=prop:name,
> > > timestamp=1519476818510, value=John Smith
> > >
> > > 1 row(s) in 0.0180 seconds
> > >
> > > hbase(main):011:0> list_security_capabilities
> > > SIMPLE_AUTHENTICATION
> > > CELL_VISIBILITY
> > >
> > > => ["SIMPLE_AUTHENTICATION", "CELL_VISIBILITY"]
> > > hbase(main):012:0> scan "test", { AUTHORIZATIONS => [] }
> > > ROW
> > > COLUMN+CELL
> > >
> > >
> > >  user1 column=prop:name,
> > > timestamp=1519476818510, value=John Smith
> > >
> > > 1 row(s) in 0.0060 seconds
> > >
> > >
> > > I'm running this on a Mac w/out HDFS. It's HBase 1.3.1. This makes no
> > > sense because it's allowing me to assign authorizations to my simple
> auth
> > > user, set_visibility runs successfully and it's reporting that cell
> > > visibility is enabled.
> > >
> > >
> >
>

Re: New installation of HBase ignoring cell visibility (even though it's active)

[Ted Yu]

I noted that SIMPLE_AUTHENTICATION was returned.
Here is related code for getSecurityCapabilities():

  if (User.isHBaseSecurityEnabled(master.getConfiguration())) {

capabilities.add(SecurityCapabilitiesResponse.Capability.
SECURE_AUTHENTICATION);

  } else {

capabilities.add(SecurityCapabilitiesResponse.Capability.
SIMPLE_AUTHENTICATION);

  }
Did "hbase.security.authentication" have value of "kerberos" ?

If it does, please pastebin your hbase-site.xml

Thanks

On Sat, Feb 24, 2018 at 4:59 AM, Mike Thomsen 
wrote:

> (hbase-site.xml is attached)
>
> I reinstalled HBase on my development machine and the console shows that
> it's just ignoring the cell visibility settings even though it shows
> they're active:
>
> hbase(main):001:0> list
> TABLE
>
>
>
> 0 row(s) in 0.1630 seconds
>
> => []
> hbase(main):002:0> get_auths "michaelthomsen"
> 0 row(s) in 10.1940 seconds
>
> hbase(main):003:0> add_labels ["U", "PII", "PHI"]
> 0 row(s) in 0.1900 seconds
>
> hbase(main):004:0> set_auths "michaelthomsen",  ["U", "PII", "PHI"]
> 0 row(s) in 0.0500 seconds
>
> hbase(main):005:0> get_auths "michaelthomsen"
> U
>
>
>
> PII
>
>
>
> PHI
>
>
>
> 0 row(s) in 0.0470 seconds
>
> hbase(main):006:0> create 'test', {NAME => 'prop' }
> 0 row(s) in 1.2780 seconds
>
> => Hbase::Table - test
> hbase(main):007:0> put "test", "user1", "prop:name", "John Smith"
> 0 row(s) in 0.0500 seconds
>
> hbase(main):008:0> scan "test"
> ROW
> COLUMN+CELL
>
>
>  user1 column=prop:name,
> timestamp=1519476818510, value=John Smith
>
> 1 row(s) in 0.0260 seconds
>
> hbase(main):009:0> set_visibility "test", "PII", { COLUMNS => "prop" }
> 1 row(s) in 0.0130 seconds
>
> hbase(main):010:0> scan "test", { AUTHORIZATIONS => ["PHI", "U"] }
> ROW
> COLUMN+CELL
>
>
>  user1 column=prop:name,
> timestamp=1519476818510, value=John Smith
>
> 1 row(s) in 0.0180 seconds
>
> hbase(main):011:0> list_security_capabilities
> SIMPLE_AUTHENTICATION
> CELL_VISIBILITY
>
> => ["SIMPLE_AUTHENTICATION", "CELL_VISIBILITY"]
> hbase(main):012:0> scan "test", { AUTHORIZATIONS => [] }
> ROW
> COLUMN+CELL
>
>
>  user1 column=prop:name,
> timestamp=1519476818510, value=John Smith
>
> 1 row(s) in 0.0060 seconds
>
>
> I'm running this on a Mac w/out HDFS. It's HBase 1.3.1. This makes no
> sense because it's allowing me to assign authorizations to my simple auth
> user, set_visibility runs successfully and it's reporting that cell
> visibility is enabled.
>
>

Re: HBaseTestingUtility with visibility labels enabled

[Ted Yu]

labels table is created by VisibilityController#postStartMaster().

You can add the following call in the @BeforeClass method:


TEST_UTIL.waitTableEnabled(LABELS_TABLE_NAME.getName(), 5);


See TestVisibilityLabelsWithACL for complete example.



On Thu, Feb 22, 2018 at 12:07 PM, Mike Thomsen 
wrote:

> I'm trying to spin up a mini cluster for integration testing. Can someone
> give me an idea of what I'm doing wrong?
>
> public static void main(String[] args) throws Throwable {
>
> Configuration conf =
> org.apache.hadoop.hbase.HBaseConfiguration.create();
> conf.set("hbase.coprocessor.region.classes",
> "org.apache.hadoop.hbase.security.visibility.VisibilityController");
> conf.set("hbase.coprocessor.master.classes",
> "org.apache.hadoop.hbase.security.visibility.VisibilityController");
>
> utility = new HBaseTestingUtility(conf);
>
> utility.startMiniCluster();
>
> VisibilityClient.addLabels(utility.getConnection(), new String[]{
> "X", "Y", "Z" });
> }
>
> That results in this:
>
> org.apache.hadoop.hbase.TableNotFoundException: hbase:labels
>
> Thanks,
>
> Mike
>

Re: How to recover a table

[Ted Yu]

It seems there were 3 files on s3 (they're all on the same line).

If possible, can you pastebin parts of master log which were related to the
table ?

That may give us more clue.

On Thu, Feb 22, 2018 at 10:01 AM, Vikas Kanth <
kanth_vi...@yahoo.co.in.invalid> wrote:

>  Hi Ted,
> Thanks for replying.I can see the descriptors under the table:
> s3://mybucket/hbasedir/data/db1/mytable/.tabledesc/.
> tableinfo.01s3://mybucket/hbasedir/data/db1/
> mytable/.tabledescs3://mybucket/hbasedir/data/db1/mytable/.tmp
>
> This is what I see in the HBM logs:
> 2018-02-21 09:02:29,918 WARN  [x,16000,1519199162102_ChoreService_3]
> master.CatalogJanitor: CatalogJanitor disabled! Not running scan.
> $ hbase versionHBase 1.3.1
> Thanks

Re: Region not initializing in 2.0.0-beta-1

[Ted Yu]

Can you show more of the region server log ?
Was the cluster started clean (without any data) ?

There have been a lot of changes since 2.0.0-beta-1 was released (both in
terms of correctness and performance).

If possible, please deploy 2.0 SNAPSHOT for further testing.

Cheers

On Thu, Feb 22, 2018 at 9:39 AM, sahil aggarwal 
wrote:

> Hi,
>
> I am trying to get 2.0.0-beta-1 cluster up to do some perf test but not
> able to get region servers up. Its stuck in initializing state. Looks like
> it is stuck in getting the hbaseId from zk:
>
> jstack says:
>
> "regionserver/perf-rs-1/10.32.73.176:16020" #24 prio=5 os_prio=0
> tid=0x7f19e45cc000 nid=0x1b13 waiting on condition [0x7f19de6a7000]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0001846d8b98> (a
> java.util.concurrent.CompletableFuture$WaitNode)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.
> java:175)
> at
> java.util.concurrent.CompletableFuture$WaitNode.
> block(CompletableFuture.java:271)
> at
> java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3226)
> at
> java.util.concurrent.CompletableFuture.waitingGet(
> CompletableFuture.java:319)
> at
> java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2224)
> at
> org.apache.hadoop.hbase.client.ConnectionImplementation.retrieveClusterId(
> ConnectionImplementation.java:526)
> at
> org.apache.hadoop.hbase.client.ConnectionImplementation.(
> ConnectionImplementation.java:286)
> at
> org.apache.hadoop.hbase.client.ConnectionUtils$
> ShortCircuitingClusterConnection.(ConnectionUtils.java:141)
> at
> org.apache.hadoop.hbase.client.ConnectionUtils$
> ShortCircuitingClusterConnection.(ConnectionUtils.java:132)
> at
> org.apache.hadoop.hbase.client.ConnectionUtils.
> createShortCircuitConnection(ConnectionUtils.java:185)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.
> createClusterConnection(HRegionServer.java:770)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.setupClusterConnection(
> HRegionServer.java:801)
> - locked <0x00019ccd1bc8> (a
> org.apache.hadoop.hbase.regionserver.HRegionServer)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.
> preRegistrationInitialization(HRegionServer.java:816)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.
> run(HRegionServer.java:925)
> at java.lang.Thread.run(Thread.java:745)
>
>
> Even though it seem to got the response from zk:
>
> 2018-02-22 22:17:22,959 DEBUG [ReadOnlyZKClient-SendThread(
> perf-zk-1:2181)]
> zookeeper.ClientCnxn: Reading reply sessionid:0x26146d5de5c7181, packet::
> clientPath:/hbase-unsecure2/hbaseid serverPath:/hbase-unsecure2/hbaseid
> finished:false header:: 1,4  replyHeader:: 1,111751356003,0  request::
> '/hbase-unsecure2/hbaseid,F  response::
> #000146d61737465723a3630303030395e49ffefff98f
> fd0262150425546a2430653037386566362d363931362d343665332d
> 386335652d653237666264303135326337,s{111750564985,
> 111750564985,1519309853186,1519309853186,0,0,0,0,67,0,111750564985}
>
> Any pointers?
>
>
> Thanks,
> Sahil
>

Re: How to recover a table

[Ted Yu]

For a user table, you should see the following in the table dir:

drwxr-xr-x   - hbase hdfs  0 2018-02-16 22:20
/apps/hbase/data/data/default/t1/.tabledesc
drwxr-xr-x   - hbase hdfs  0 2018-02-16 22:20
/apps/hbase/data/data/default/t1/.tmp

Is the table descriptor under mytable ?

At the time of 'Unknown table mytable!' error, what do you see in master
log ?

Which hbase release are you using ?

Cheers

On Wed, Feb 21, 2018 at 12:44 PM, Vikas Kanth <
kanth_vi...@yahoo.co.in.invalid> wrote:

> I can see the regions of my table at s3://mybucket/hbasedir/data/
> db1/mytable/77b98d085239ed8668596ea659a7ad7dHowever, Hbase fails to
> identify the table:
>
> hbase(main):002:0> describe 'mytable'ERROR: Unknown table mytable!
> Commands like scan and count are failing as well.I have tried running the
> repair and fix commands (-fixMeta, -fixAssignment and others) without any
> success. These commands show 0 inconsistency.Other tables look healthy.
> Please let me know how to recover the table. I can't afford to drop the
> table

Re: Hbase integration testing

[Ted Yu]

It seems you're using sbt.

Can you run this command and pastebin the output:

sbt "inspect tree clean"



On Wed, Feb 21, 2018 at 8:21 AM, Gauthier Feuillen 
wrote:

> Yeah I already tested that (thanks for the help btw)
>
> Here are my dependencies:
>
> 
> lazy val hbaseTesting = "org.apache.hbase" % "hbase-testing-util" %
> version % Test
> lazy val hbaseServerTest = "org.apache.hbase" % "hbase-server" % version %
> Test
>
> lazy val hbaseTestDependencies = Seq(hbaseTesting, hbaseServerTest)
> 
>
> See anything unusual ?
>
>
> > On 21 Feb 2018, at 17:11, Debraj Manna  wrote:
> >
> > Use hbase-testing-util
> >
> > https://mvnrepository.com/artifact/org.apache.hbase/hbase-testing-util
> >
> > On 21-Feb-2018 9:40 PM, "Gauthier Feuillen" 
> wrote:
> >
> >> I did that but still the class is not available. Strange.
> >>
> >>> On 21 Feb 2018, at 17:08, Debraj Manna 
> wrote:
> >>>
> >>> You have to use hbase-testing-util dependency. Check the below
> discussion
> >>> in stackoverflow
> >>>
> >>> https://stackoverflow.com/q/48114833/785523
> >>>
> >>> On 21-Feb-2018 8:37 PM, "Gauthier Feuillen" 
> >> wrote:
> >>>
> >>> Hi,
> >>>
> >>> I’d like to be able to unit test my HBase application. I see a lot of
> >> posts
> >>> using the HBaseTestingUtility. I can’t get it into my project. Which
> >>> dependency(ies) should I use ?
> >>>
> >>> Best,
> >>> Gauthier.
> >>
> >>
>
>

Re: Hbase integration testing

[Ted Yu]

HBaseTestingUtility is in hbase-server module.

You can add hbase-server module with test scope.

On Wed, Feb 21, 2018 at 7:07 AM, Gauthier Feuillen 
wrote:

> Hi,
>
> I’d like to be able to unit test my HBase application. I see a lot of
> posts using the HBaseTestingUtility. I can’t get it into my project. Which
> dependency(ies) should I use ?
>
> Best,
> Gauthier.

Re: Inconsistent rows exported/counted when looking at a set, unchanged past time frame.

[Ted Yu]

If you look at
https://www.cloudera.com/documentation/enterprise/release-notes/topics/cdh_rn_fixed_in_58.html#fixed_issues585
, you would see the following:

HBASE-15378 - Scanner cannot handle heartbeat message with no results

which fixed what you observed in previous release.

FYI

On Tue, Feb 20, 2018 at 9:07 PM, Andrew Kettmann <
andrew.kettm...@evolve24.com> wrote:

> Josh,
>
> We upgraded from CDH 5.8.0 -> 5.8.5 seems to have fixed the issue. 3
> Rowcounts in a row that were not consistent before on a static table are
> now consistent. We are doing some further testing but it looks like you
> called it with:
>
> 'scans on RegionServers stop prematurely before all of the data is read'
>
> Thanks for the pointer in that direction, I was bashing my face against
> this for two weeks trying to figure out this inconsistency. I appreciate
> the clue!
>
> Andrew Kettmann
> Consultant, Platform Services Group
>
> -Original Message-
> From: Josh Elser [mailto:els...@apache.org]
> Sent: Monday, February 12, 2018 11:59 AM
> To: user@hbase.apache.org
> Subject: Re: Inconsistent rows exported/counted when looking at a set,
> unchanged past time frame.
>
> Hi Andrew,
>
> Yes. The answer is, of course, that you should see consistent results from
> HBase if there are no mutations in flight to that table. Whether you're
> reading "current" or "back-in-time", as long as you're not dealing with raw
> scans (where compactions may persist delete tombstones), this should hold
> just the same.
>
> Are you modifying older cells with newer data when you insert data?
> Remember that MAX_VERSIONS for a table defaults to 1. Consider the
> following:
>
> * Timestamps are of the form "tX", and t1 < t2 < t3 < ..
> * You are querying from the time range: [t1, t5].
> * You have a cell for "row1" with at t3 with value "foo".
> * RowCounter over [t1, t5] would return "1"
> * Your ingest writes a new cell for "row1" of "bar" at t6.
> * RowCounter over [t1, t5] would return "0" normally, or "1" is you use
> RAW scans ***
> * A compaction would run over the region containing "row1"
> * RowCounter over [t1, t5] would return "0" (RAW or normal)
>
> It's also possible that you're hitting some sort of bug around missing
> records at query time. I'm not sure what the CDH versions you're using line
> up to, but there have certainly been issues in the past around query-time
> data loss (e.g. scans on RegionServers stop prematurely before all of the
> data is read).
>
> Good luck!
>
> *** Going off of memory here. I think this is how it works, but you should
> be able to test easily ;)
>
> On 2/9/18 5:30 PM, Andrew Kettmann wrote:
> > A simpler question would be this:
> >
> > Given:
> >
> >
> >*   a set timeframe in the past (2-3 days roughly a year ago)
> >*   we are NOT removing records from the table at all
> >*   We ARE inserting into this table actively
> >
> > Should I expect two consecutive runs of the rowcounter mapreduce job to
> return an identical number?
> >
> >
> > Andrew Kettmann
> > Consultant, Platform Services Group
> >
> > From: Andrew Kettmann
> > Sent: Thursday, February 08, 2018 11:35 AM
> > To: user@hbase.apache.org
> > Subject: Inconsistent rows exported/counted when looking at a set,
> unchanged past time frame.
> >
> > First the version details:
> >
> > Running HBASE/Yarn/HDFS using Cloudera manager 5.12.1.
> > Hbase: Version 1.2.0-cdh5.8.0
> > HDFS/YARN: Hadoop 2.6.0-cdh5.8.0
> > Hbck and hdfs fsck return healthy
> >
> > 15 nodes, sized down recently from 30 (other service requirements
> > reduced. Solr, etc)
> >
> >
> > The simplest example of the inconsistency is using rowcounter. If I run
> the same mapreduce job twice in a row, I get different counts:
> >
> > hbase org.apache.hadoop.hbase.mapreduce.Driver rowcounter
> > -Dmapreduce.map.speculative=false TABLENAME --starttime=148590720
> > --endtime=148605840
> >
> > Looking at org.​apache.​hadoop.​hbase.​mapreduce.​RowCounter​$
> RowCounterMapper​$Counters:
> > Run 1: 4876683
> > Run 2: 4866351
> >
> > Similarly with exports of the same date/time. Consecutive runs of the
> export get different results:
> > hbase org.apache.hadoop.hbase.mapreduce.Export \
> > -Dmapred.map.tasks.speculative.execution=false \
> > -Dmapred.reduce.tasks.speculative.execution=false \ TABLENAME \
> > HDFSPATH 1 148590720 148605840
> >
> >  From Map Input/output records:
> > Run 1: 4296778
> > Run 2: 4297307
> >
> > None of the results show anything for spilled records, no failed maps.
> Sometimes the row count increases, sometimes it decreases. We aren’t using
> any row filter queries, we just want to export chunks of the data for a
> specific time range. This table is actively being read/written to, but I am
> asking about a date range in early 2017 in this case, so that should have
> no impact I would have thought. Another point is that the rowcount job and
> the export return ridiculously different numbers. There should be no older
> versions of rows 

Re: Query data like opentsdb

[Ted Yu]

Have you looked at FuzzyRowFilter ?

Cheers

On Mon, Feb 19, 2018 at 8:00 AM, kitex101  wrote:

> I have key design like:byte[] rowKey =
> =Bytes.add(Bytes.toBytes("3"),Bytes.toBytes(customer_id),
> Bytes.toBytes(timestamp));
> customer_id and timestamp are long type. As opentsdb uses:[…]I would like
> to
> filter my key by customer_id and timestamp. How do I do that?I have tried
> using prefixfilter. e.g.byte[] prefix = Bytes.add(Bytes.toBytes("3"),
> Bytes.toBytes(customer_id),Bytes.toBytes(cal.getTime().getTime()));
> PrefixFilter prefixFilter = new PrefixFilter(prefix); Scan scan = new Scan(
> prefix);
>
>
>
> --
> Sent from: http://apache-hbase.679495.n3.nabble.com/HBase-User-
> f4020416.html

Re: How to get string from hbase composite key?

[Ted Yu]

I mentioned the method accepting int because of the example in your first
email.
I verified in Eclipse that Bytes.toBytes(98) would end up in that
method.

I understand in your application data is typed as long. However that is not
shown in your previous example.

On Sat, Feb 17, 2018 at 9:00 PM, Ted Yu <yuzhih...@gmail.com> wrote:

> It seems there are 3 components in the row key.
> Assuming the 2nd and 3rd are integers, you can take a look at the
> following method of Bytes:
>
>   public static byte[] toBytes(int val) {
>
> which returns 4 byte long byte array.
> You can use this knowledge to decode each component of the row key.
>
> FYI
>
> On Sat, Feb 17, 2018 at 8:44 PM, kitex101 <sugandha.ama...@gmail.com>
> wrote:
>
>> I have key in following format:
>>
>> byte[] rowKey =
>> =Bytes.add(Bytes.toBytes("3"),Bytes.toBytes(98),Bytes.toBytes(1211));
>>
>> It is stored as byte array.
>>
>> How to decode it when using java?
>>
>> Bytes.toString(CellUtil.cloneRow(cell))
>>
>> results in a�~��
>>
>>
>>
>>
>> --
>> Sent from: http://apache-hbase.679495.n3.nabble.com/HBase-User-f4020416
>> .html
>>
>
>

Re: How to get string from hbase composite key?

[Ted Yu]

It seems there are 3 components in the row key.
Assuming the 2nd and 3rd are integers, you can take a look at the following
method of Bytes:

  public static byte[] toBytes(int val) {

which returns 4 byte long byte array.
You can use this knowledge to decode each component of the row key.

FYI

On Sat, Feb 17, 2018 at 8:44 PM, kitex101  wrote:

> I have key in following format:
>
> byte[] rowKey =
> =Bytes.add(Bytes.toBytes("3"),Bytes.toBytes(98),Bytes.toBytes(1211));
>
> It is stored as byte array.
>
> How to decode it when using java?
>
> Bytes.toString(CellUtil.cloneRow(cell))
>
> results in a�~��
>
>
>
>
> --
> Sent from: http://apache-hbase.679495.n3.nabble.com/HBase-User-
> f4020416.html
>

Re: MultipleColumnPrefixFilter issues with SingleColumnValueFilter

[Ted Yu]

I don't see filter[1-3] being used in the cases.
Was any of them in the FilterList ?

Which release of hbase are you using ?

Cheers

On Fri, Feb 16, 2018 at 12:36 AM, Vikash Agarwal 
wrote:

> Hi Team,
>
>
> Currently I am trying to use MultipleColumnPrefixFilter along with
> SingleColumnValueFilter and the results  that I am getting seems to be
> problematic.
>
> I have an hbase table "test_messages" with column family  "main" and Column
> Qualifiers as : "pa", "cs" ,"eps","cst","ver","nm","parl".
> Assume all the row Keys of the table  has all the above columns.
>
>
> Now When I am creating Filters as below:
>
>
> FilterList temp=new FilterList(FilterList.Operator.MUST_PASS_ALL);
>
> SingleColumnValueFilter filter1= new
> SingleColumnValueFilter("main","pa",CompareFilter.CompareOp.LESS_OR_EQUAL,
> Bytes.toBytes("SomeValue"));
> filter1 .setFilterIfMissing(true);
>
> SingleColumnValueFilter filter2= new
> SingleColumnValueFilter("main","cs",CompareFilter.CompareOp.EQUAL,
> Bytes.toBytes("SomeValue"));
> filter2.setFilterIfMissing(true);
>
>
> SingleColumnValueFilter filter3= new
> SingleColumnValueFilter("main",eps",CompareFilter.CompareOp.EQUAL,
> Bytes.toBytes("SomeValue"));
> filter3.setFilterIfMissing(true);
>
> Case 1:
>
> MultipleColumnPrefixFilter cpfxf=new MultipleColumnPrefixFilter(new
> byte[][]{ Bytes.toBytes("pa"),Bytes.toBytes("cs"),Bytes.toBytes("eps")});
> temp.addFilter(cpfxf);
>
> The above filters results in proper extratcion of data from Hbase.
>
>
> Now if I create a MultipleColumnPrefixFilter  as below:
>
> Case 2:
> MultipleColumnPrefixFilter cpfxf=new MultipleColumnPrefixFilter(new
> byte[][]{ Bytes.toBytes("pa"),Bytes.toBytes("eps")});//Note "cs"
> is
> removed
> temp.addFilter(cpfxf);
>
> I don't get any data
>
> However if I create a filter as
>
> Case 3:
> MultipleColumnPrefixFilter cpfxf=new MultipleColumnPrefixFilter(new
> byte[][]{ Bytes.toBytes("pa"),Bytes.toBytes("cs")});  // Note
> "eps" is removed and "cs"  is added back.
> temp.addFilter(cpfxf);
>
> I get lot of data.
>
>
> EIther both Case 2 and Case 3 should give data or none of them.
> Can someone Explain me the reason for this to happen ? Or is there a bug in
> MultipleColumnPrefixFilter .
>
>
>
>
>
> --
> Vikash Agarwal
>

Re: [Production Impacted] Any workaround for https://issues.apache.org/jira/browse/HBASE-16464?

[Ted Yu]

bq. Apache projects are supposed to encourage collaboration

I totally agree with you.

Cheers

On Sat, Feb 10, 2018 at 10:32 AM, anil gupta <anilgupt...@gmail.com> wrote:

> Thanks Ted. Will try to do the clean-up. Unfortunately, we ran out of
> support for this cluster since its nearing End-of-life. For our new
> clusters we are in process of getting support.
>
> PS: IMO, I agree that i should use vendor forum/list for any vendor
> specific stuff but i think its appropriate to use this mailing Apache HBase
> questions/issues related to HBase. As per my understanding, Apache projects
> are supposed to encourage collaboration rather building boundaries around
> vendors.("collaboration and openness" is one of the reason i like Apache
> Projects)
>
> On Sat, Feb 10, 2018 at 10:11 AM, Ted Yu <yuzhih...@gmail.com> wrote:
>
> > You can cleanup oldwal directory beginning with oldest data.
> >
> > Please open support case with the vendor.
> >
> > On Sat, Feb 10, 2018 at 10:02 AM, anil gupta <anilgupt...@gmail.com>
> > wrote:
> >
> > > Hi Ted,
> > >
> > > We cleaned up all the snaphsots around Feb 7-8th. You were right that i
> > > dont see the CorruptedSnapshotException since then. Nice observation!
> > > So, i am again back to square one. Not really, sure why oldwals and
> > > recovered.edits are not getting cleaned up. I have already removed all
> > the
> > > replication peer and deleted all the snapshots.
> > > Is it ok if i just ahead and cleanup oldwal directory manually? Can i
> > also
> > > clean up recovered.edits?
> > >
> > > Thanks,
> > > Anil
> > >
> > > On Sat, Feb 10, 2018 at 9:37 AM, Ted Yu <yuzhih...@gmail.com> wrote:
> > >
> > > > Can you clarify whether /apps/hbase/data/.hbase-snapshot/.tmp/
> became
> > > > empty
> > > > after 2018-02-07 09:10:08 ?
> > > >
> > > > Do you see CorruptedSnapshotException for file outside of
> > > > /apps/hbase/data/.hbase-snapshot/.tmp/ ?
> > > >
> > > > Cheers
> > > >
> > >
> > >
> > >
> > > --
> > > Thanks & Regards,
> > > Anil Gupta
> > >
> >
>
>
>
> --
> Thanks & Regards,
> Anil Gupta
>

Re: [Production Impacted] Any workaround for https://issues.apache.org/jira/browse/HBASE-16464?

[Ted Yu]

You can cleanup oldwal directory beginning with oldest data.

Please open support case with the vendor.

On Sat, Feb 10, 2018 at 10:02 AM, anil gupta <anilgupt...@gmail.com> wrote:

> Hi Ted,
>
> We cleaned up all the snaphsots around Feb 7-8th. You were right that i
> dont see the CorruptedSnapshotException since then. Nice observation!
> So, i am again back to square one. Not really, sure why oldwals and
> recovered.edits are not getting cleaned up. I have already removed all the
> replication peer and deleted all the snapshots.
> Is it ok if i just ahead and cleanup oldwal directory manually? Can i also
> clean up recovered.edits?
>
> Thanks,
> Anil
>
> On Sat, Feb 10, 2018 at 9:37 AM, Ted Yu <yuzhih...@gmail.com> wrote:
>
> > Can you clarify whether /apps/hbase/data/.hbase-snapshot/.tmp/ became
> > empty
> > after 2018-02-07 09:10:08 ?
> >
> > Do you see CorruptedSnapshotException for file outside of
> > /apps/hbase/data/.hbase-snapshot/.tmp/ ?
> >
> > Cheers
> >
>
>
>
> --
> Thanks & Regards,
> Anil Gupta
>

Re: [Production Impacted] Any workaround for https://issues.apache.org/jira/browse/HBASE-16464?

[Ted Yu]

Can you clarify whether /apps/hbase/data/.hbase-snapshot/.tmp/ became empty
after 2018-02-07 09:10:08 ?

Do you see CorruptedSnapshotException for file outside of
/apps/hbase/data/.hbase-snapshot/.tmp/ ?

Cheers

Re: [Production Impacted] Any workaround for https://issues.apache.org/jira/browse/HBASE-16464?

[Ted Yu]

Please the first few review comments of HBASE-16464.

You can sideline the corrupt snapshots (according to master log).

You can also contact the vendor for a HOTFIX.

Cheers

On Sat, Feb 10, 2018 at 8:13 AM, anil gupta  wrote:

> Hi Folks,
>
> We are running HBase1.1.2. It seems like we are hitting
> https://issues.apache.org/jira/browse/HBASE-16464 in our Production
> cluster. Our oldwals folder has grown to 9.5Tb. I am aware that this is
> fixed in releases after 2016 but unfortunately we need to operate this
> production cluster for few more months. (We are already migrating to a
> newer version of HBase).
>
> I have verified that we dont have any snapshots in this cluster. Also, we
> removed all the replication_peers from that cluster. We have already
> restarted HBase master a few days ago but it didnt help.  We have TB's of
> oldwal and tens of thousand of recovered edit files.(assuming recovered
> edits files are cleaned up by chore cleaner). Seems like the problem
> started happening around mid december but at that time we didnt do any
> major thing on this cluster.
>
> I would like to see if there is a workaround for HBASE-16464? Is there any
> references left to those deleted snapshots in hdfs or zk? If yes, how can i
> clean up?
>
> I keep on seeing this in HMaster logs:
> 2018-02-07 09:10:08,514 ERROR
> [hdpmaster6.bigdataprod1.wh.truecarcorp.com,6,
> 1517601353645_ChoreService_3]
> snapshot.SnapshotHFileCleaner: Exception while checking if files were
> valid, keeping them just in case.
> org.apache.hadoop.hbase.snapshot.CorruptedSnapshotException: Couldn't read
> snapshot info
> from:hdfs://PRODNN/apps/hbase/data/.hbase-snapshot/.tmp/
> LEAD_SALES-1517979610/.snapshotinfo
> at
> org.apache.hadoop.hbase.snapshot.SnapshotDescriptionUtils.
> readSnapshotInfo(SnapshotDescriptionUtils.java:313)
> at
> org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.getHFileNames(
> SnapshotReferenceUtil.java:328)
> at
> org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner$1.
> filesUnderSnapshot(SnapshotHFileCleaner.java:85)
> at
> org.apache.hadoop.hbase.master.snapshot.SnapshotFileCache.
> getSnapshotsInProgress(SnapshotFileCache.java:303)
> at
> org.apache.hadoop.hbase.master.snapshot.SnapshotFileCache.
> getUnreferencedFiles(SnapshotFileCache.java:194)
> at
> org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner.
> getDeletableFiles(SnapshotHFileCleaner.java:62)
> at
> org.apache.hadoop.hbase.master.cleaner.CleanerChore.checkAndDeleteFiles(
> CleanerChore.java:233)
> at
> org.apache.hadoop.hbase.master.cleaner.CleanerChore.checkAndDeleteEntries(
> CleanerChore.java:157)
> at
> org.apache.hadoop.hbase.master.cleaner.CleanerChore.
> checkAndDeleteDirectory(CleanerChore.java:180)
> at
> org.apache.hadoop.hbase.master.cleaner.CleanerChore.checkAndDeleteEntries(
> CleanerChore.java:149)
> at
> org.apache.hadoop.hbase.master.cleaner.CleanerChore.
> checkAndDeleteDirectory(CleanerChore.java:180)
> at
> org.apache.hadoop.hbase.master.cleaner.CleanerChore.checkAndDeleteEntries(
> CleanerChore.java:149)
> at
> org.apache.hadoop.hbase.master.cleaner.CleanerChore.
> checkAndDeleteDirectory(CleanerChore.java:180)
> at
> org.apache.hadoop.hbase.master.cleaner.CleanerChore.checkAndDeleteEntries(
> CleanerChore.java:149)
> at
> org.apache.hadoop.hbase.master.cleaner.CleanerChore.
> checkAndDeleteDirectory(CleanerChore.java:180)
> at
> org.apache.hadoop.hbase.master.cleaner.CleanerChore.checkAndDeleteEntries(
> CleanerChore.java:149)
> at
> org.apache.hadoop.hbase.master.cleaner.CleanerChore.
> checkAndDeleteDirectory(CleanerChore.java:180)
> at
> org.apache.hadoop.hbase.master.cleaner.CleanerChore.checkAndDeleteEntries(
> CleanerChore.java:149)
> at
> org.apache.hadoop.hbase.master.cleaner.CleanerChore.
> chore(CleanerChore.java:124)
> at org.apache.hadoop.hbase.ScheduledChore.run(ScheduledChore.java:185)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$
> ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(
> ScheduledThreadPoolExecutor.java:294)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.FileNotFoundException: File does not exist:
> /apps/hbase/data/.hbase-snapshot/.tmp/LEAD_SALES-1517979610/.snapshotinfo
> at
> org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(
> INodeFile.java:71)
> at
> org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(
> INodeFile.java:61)
> at
> 

Re: Frequent Region Server Failures with namenode.LeaseExpiredException

[Ted Yu]

Do you use Phoenix functionality ?

If not, you can try disabling the Phoenix side altogether (removing Phoenix
coprocessors).

2.3.4 is really old - please upgrade to 2.6.3

You should consider asking on the vendor's community forum.

Cheers

On Thu, Feb 8, 2018 at 3:06 PM, anil gupta  wrote:

> Hi Folks,
>
> We are running a 60 Node MapReduce/HBase HDP cluster. HBase 1.1.2 , HDP:
> 2.3.4.0-3485. Phoenix is enabled on this cluster.
> Each slave has ~120gb ram. RS has 20 Gb heap, 12 disk of 2Tb each and 24
> cores.  This cluster has been running OK for last 2 years but recently with
> few disk failures(we unmounted those disks) it hasnt been running fine. I
> have checked hbck and hdfs fsck. Both of them report no inconsistency.
>
> Some our RegionServers keeps on aborting with following error:
> 1 ==>
> org.apache.hadoop.ipc.RemoteException(org.apache.
> hadoop.hdfs.server.namenode.LeaseExpiredException):
> No lease on
> /apps/hbase/data/data/default/DE.TABLE_NAME/35aa0de96715c33e1f0664aa4d9292
> ba/recovered.edits/03948161445.temp
> (inode 420864666): File does not exist. [Lease.  Holder:
> DFSClient_NONMAPREDUCE_-64710857_1, pendingcreates: 1]
>
> 2 ==> 2018-02-08 03:09:51,653 ERROR [regionserver/
> hdpslave26.bigdataprod1.com/1.16.6.56:16020] regionserver.HRegionServer:
> Shutdown / close of WAL failed:
> org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease on
> /apps/hbase/data/oldWALs/hdpslave26.bigdataprod1.com%
> 2C16020%2C1518027416930.default.1518085177903
> (inode 420996935): File is not open for writing. Holder
> DFSClient_NONMAPREDUCE_649736540_1 does not have any open files.
>
> All the LeaseExpiredException are happening for recovered.edits and
> oldWALs.
>
> HDFS is around 48% full. Most of the DN's have 30-40% space left on them.
> NN heap is at 60% use. I have tried googling around but cant find anything
> concrete to fix this problem. Currently, 15/60 nodes are already down in
> last 2 days.
> Can someone please point out what might be causing these RegionServer
> failures?
>
>
> --
> Thanks & Regards,
> Anil Gupta
>

Re: Hbase disaster recovery scenario

[Ted Yu]

The built-in hbase client doesn't support failing over automatically to DR
cluster.

Switching "zookeeper.quorum" should be done on client side for failover.

Cheers

On Wed, Feb 7, 2018 at 3:16 PM, Daniel Połaczański 
wrote:

> Hi,
> I want to configure HBase in DR scenario. I create two separete clusters
> and configure master-master replication. At the begining I use only PROD
> cluster and in case of disaster of data center I want to switch client to
> DR cluster.
>
> Is client able to switch automatically from PROD to DR cluster? Is there
> any mechanism which supports this kind of switching?
>
> Or maybe client has to explicitly and manualy change "zookeeper.quorum"
> from Prod to DR?
> How do you solve this kind of problems?
>
> Regards
>

Re: WrongRegionException

[Ted Yu]

w.r.t. region split, do you verify that the new rowkey is in the same
region as the rowkey from incoming Put ?

If not, there is a chance that the new rowkey is in different region which
is going thru split.

FYI

On Mon, Jan 29, 2018 at 6:40 AM, Yang Zhang  wrote:

> Both are the same question.
> I want to prevent the incoming puts and then copy it as one or more new
> puts with different rowkey.
> So when there is only one region, my rowkey will belong to it. But when
> region splited, some rowkeys may not belong to the new region.
> I used to thought HBase will stop new coming puts, finish all of the puts
> in the batch, and then try to split.
> But this maybe not right according to the exception that I got.
>
> BTY , It seems that I can't add put
> to MiniBatchOperationInProgress miniBatchOp. There are only some
> functions for get.
>
> Thank you very much for your help
>
> 2018-01-29 18:46 GMT+08:00 Anoop John :
>
> > Another related Q was also there..  Can you tell the actual
> > requirement?  So the incoming puts you want to change the RKs of that?
> > Or you want to insert those as well as some new cells with a changed
> > RK?
> >
> > -Anoop-
> >
> > On Mon, Jan 29, 2018 at 3:49 PM, Yang Zhang 
> > wrote:
> > > Hello Everyone
> > >
> > > I am using coprocesser to prevent the normal put and replace it
> > > with another rowkey, The method is HRegion.put(). It works fine, but
> when
> > > the region splited, There will be an WrongRegionException.
> > >
> > > 2018-01-28 09:32:51,528 WARN
> > > [B.DefaultRpcServer.handler=21,queue=3,port=60020]
> regionserver.HRegion:
> > > Failed getting lock in batch put,
> > > row=\xF0\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x0F\x10\xC5r
> > > org.apache.hadoop.hbase.regionserver.WrongRegionException: Requested
> row
> > > out of range for row lock on HRegion
> > > GISdoop_GeoKey,,1517085124215.341534e84727245f1c67f345c3e467ac.,
> > > startKey='', getEndKey()='\xE6G8\x00\x00\x00\x00\x00\x00\x00\x00\x00',
> > > row='\xF0\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x0F\x10\xC5r'
> > > at org.apache.hadoop.hbase.regionserver.HRegion.checkRow(
> > HRegion.java:4677)
> > > at
> > > org.apache.hadoop.hbase.regionserver.HRegion.
> > getRowLock(HRegion.java:4695)
> > > at
> > > org.apache.hadoop.hbase.regionserver.HRegion.
> > doMiniBatchMutation(HRegion.java:2786)
> > > at
> > > org.apache.hadoop.hbase.regionserver.HRegion.
> > batchMutate(HRegion.java:2653)
> > > at
> > > org.apache.hadoop.hbase.regionserver.HRegion.
> > batchMutate(HRegion.java:2589)
> > > at
> > > org.apache.hadoop.hbase.regionserver.HRegion.
> doBatchMutate(HRegion.java:
> > 3192)
> > > at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:2459)
> > > at site.luoyu.Core.Index.JavaTreeMap.insertPoint(JavaTreeMap.java:287)
> > > at site.luoyu.Core.Index.JavaTreeMap.insertRecord(
> JavaTreeMap.java:256)
> > > at site.luoyu.Core.Observer.IndexCopressor.prePut(
> > IndexCopressor.java:130)
> > > at
> > > org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.prePut(
> > RegionCoprocessorHost.java:1122)
> > > at
> > > org.apache.hadoop.hbase.regionserver.HRegion.
> doPreMutationHook(HRegion.
> > java:2674)
> > > at
> > > org.apache.hadoop.hbase.regionserver.HRegion.
> > batchMutate(HRegion.java:2649)
> > > at
> > > org.apache.hadoop.hbase.regionserver.HRegion.
> > batchMutate(HRegion.java:2589)
> > > at
> > > org.apache.hadoop.hbase.regionserver.HRegion.
> > batchMutate(HRegion.java:2593)
> > > at
> > > org.apache.hadoop.hbase.regionserver.HRegionServer.
> > doBatchOp(HRegionServer.java:4402)
> > > at
> > > org.apache.hadoop.hbase.regionserver.HRegionServer.
> > doNonAtomicRegionMutation(HRegionServer.java:3584)
> > > at
> > > org.apache.hadoop.hbase.regionserver.HRegionServer.
> > multi(HRegionServer.java:3474)
> > > at
> > > org.apache.hadoop.hbase.protobuf.generated.
> ClientProtos$ClientService$2.
> > callBlockingMethod(ClientProtos.java:3)
> > > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2078)
> > > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:108)
> > > at
> > > org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(
> > RpcExecutor.java:114)
> > > at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94)
> > > at java.lang.Thread.run(Thread.java:745)
> > >
> > > It is said rowkey is out of region's bound.  This Exception is logged
> in
> > > regionserver's log as an warning , I can't catch  and handle it.
> > >
> > > According the source code,
> > > RowLock rowLock = null;
> > > try {
> > >   rowLock = getRowLock(mutation.getRow(), shouldBlock);
> > > } catch (IOException ioe) {
> > >   LOG.warn("Failed getting lock in batch put, row="
> > > + Bytes.toStringBinary(mutation.getRow()), ioe);
> > > }
> > >
> > > HBase just cache and log this exception , I guess it even didn't remove
> > it
> > > from the batch. So I got so many Exception log  and can't 

Re: Hbase and Database distribution

[Ted Yu]

Have you looked at http://hbase.apache.org/book.html#rsgroup ?

It is in 1.4.x release.

FYI

On Fri, Jan 26, 2018 at 6:06 AM, Oussema BEN LAMINE 
wrote:

> Hello,
> i am using hbase 1.1.2, i have 5 region servers already working on my
> cluster.
> i want to create a hbase table that will be on just two of those five
> region servers.
> in other words, i want to save new table data on just two servers, not all.
> any help will be appreciated,
> thanks
>

Re: Coprocessor prePut

[Ted Yu]

bq. can call context.complete() to skip other coprocessors.

FYI in hbase-2, complete() has been folded into bypass().
See javadoc of bypass() for details.

On Mon, Jan 22, 2018 at 9:34 PM, Yang Zhang <zhang.yang...@gmail.com> wrote:

> In fact, I prevent normal writes, and put the same data with a different
> row key(Except row key, they have the same columns and same data).
> I have already solved the infinite call by add and check some flag
> attribute to Put. If the flag exist then skip my code.
>
> This will prevent the infinite call of prePut(). But I want to know If
> there are any method support by coprocessor context to do that.
> Just like that you can call context.complete() to skip other coprocessors.
>
> Thanks for your advice
>
> 2018-01-23 13:01 GMT+08:00 Ted Yu <yuzhih...@gmail.com>:
>
> > Your prePut would write to a different column in the table, right ?
> > Otherwise do you prevent normal writes from getting into the same
> column(s)
> > ?
> >
> > If your prePut writes to dedicated column(s), you can check the presence
> of
> > such column(s) so that the call is not infinite.
> >
> > Cheers
> >
> > On Mon, Jan 22, 2018 at 8:56 PM, Yang Zhang <zhang.yang...@gmail.com>
> > wrote:
> >
> > > Yes, It is the same table.
> > >
> > > 2018-01-23 1:46 GMT+08:00 Ted Yu <yuzhih...@gmail.com>:
> > >
> > > > Can you clarify your use case ?
> > > >
> > > > bq. put a data into table
> > > >
> > > > Does your coprocessor write to the same table which receives user
> data
> > ?
> > > >
> > > > Cheers
> > > >
> > > > On Mon, Jan 22, 2018 at 4:24 AM, Yang Zhang <zhang.yang...@gmail.com
> >
> > > > wrote:
> > > >
> > > > > Hello Everyone
> > > > >
> > > > > I am using the coprocessor and want to put another data
> when
> > > > > someone put a data into table. So I try prePut(), but when you call
> > > > > HRegion.put(), this will call my coprocessor's prePut function
> again,
> > > and
> > > > > it will cause an dead loop.
> > > > >
> > > > > My code looks like below, If anyone know how to make HBase
> > > call
> > > > my
> > > > > coprocessor just for the first time?
> > > > >
> > > > > @Override
> > > > > public void prePut(ObserverContext<
> RegionCoprocessorEnvironment>
> > > > > context, Put put, WALEdit edit,
> > > > >Durability durability) throws IOException {
> > > > > logger.info(" Coprocessor shouldComplete :
> > > > > "+context.shouldComplete()+"");
> > > > > context.bypass();
> > > > >
> > > > > Put myPut = new Put(new RowKey(1).toBytes());
> > > > >
> > > > > //this call will caused an dead loop
> > > > > context.getEnvironment().getRegion().put(put);
> > > > >
> > > > > }
> > > > >
> > > > > Thanks anyway
> > > > >
> > > >
> > >
> >
>

Re: Coprocessor prePut

[Ted Yu]

Your prePut would write to a different column in the table, right ?
Otherwise do you prevent normal writes from getting into the same column(s)
?

If your prePut writes to dedicated column(s), you can check the presence of
such column(s) so that the call is not infinite.

Cheers

On Mon, Jan 22, 2018 at 8:56 PM, Yang Zhang <zhang.yang...@gmail.com> wrote:

> Yes, It is the same table.
>
> 2018-01-23 1:46 GMT+08:00 Ted Yu <yuzhih...@gmail.com>:
>
> > Can you clarify your use case ?
> >
> > bq. put a data into table
> >
> > Does your coprocessor write to the same table which receives user data ?
> >
> > Cheers
> >
> > On Mon, Jan 22, 2018 at 4:24 AM, Yang Zhang <zhang.yang...@gmail.com>
> > wrote:
> >
> > > Hello Everyone
> > >
> > > I am using the coprocessor and want to put another data when
> > > someone put a data into table. So I try prePut(), but when you call
> > > HRegion.put(), this will call my coprocessor's prePut function again,
> and
> > > it will cause an dead loop.
> > >
> > > My code looks like below, If anyone know how to make HBase
> call
> > my
> > > coprocessor just for the first time?
> > >
> > > @Override
> > > public void prePut(ObserverContext
> > > context, Put put, WALEdit edit,
> > >Durability durability) throws IOException {
> > > logger.info(" Coprocessor shouldComplete :
> > > "+context.shouldComplete()+"");
> > > context.bypass();
> > >
> > > Put myPut = new Put(new RowKey(1).toBytes());
> > >
> > > //this call will caused an dead loop
> > > context.getEnvironment().getRegion().put(put);
> > >
> > > }
> > >
> > > Thanks anyway
> > >
> >
>

Re: Coprocessor prePut

[Ted Yu]

Can you clarify your use case ?

bq. put a data into table

Does your coprocessor write to the same table which receives user data ?

Cheers

On Mon, Jan 22, 2018 at 4:24 AM, Yang Zhang  wrote:

> Hello Everyone
>
> I am using the coprocessor and want to put another data when
> someone put a data into table. So I try prePut(), but when you call
> HRegion.put(), this will call my coprocessor's prePut function again, and
> it will cause an dead loop.
>
> My code looks like below, If anyone know how to make HBase  call my
> coprocessor just for the first time?
>
> @Override
> public void prePut(ObserverContext
> context, Put put, WALEdit edit,
>Durability durability) throws IOException {
> logger.info(" Coprocessor shouldComplete :
> "+context.shouldComplete()+"");
> context.bypass();
>
> Put myPut = new Put(new RowKey(1).toBytes());
>
> //this call will caused an dead loop
> context.getEnvironment().getRegion().put(put);
>
> }
>
> Thanks anyway
>

Re: How spark writes to HBASE

[Ted Yu]

I did a search in SHC for saveAsNewHadoop (case insensitive) - there was no
match.

I suggest you use SHC forum for related questions.

On Mon, Jan 22, 2018 at 9:07 AM, vignesh <vignesh...@gmail.com> wrote:

> It would be similar to case 2 right. Say for example in spark I read a file
> of size 512mb which would span 4 cores(if block size is 128). Executor will
> be spanned based on data locality and if the executor is launched
> machine1,2,3,4. May be if block3 region is handled by machine6, then when I
> bulk load via spark hbase connector (which uses saveAsNewHadoopApidataset)
> then in this case the HFILE write of block3 would be to any of those 4
> machines and not to machine6. Is that right? Or i misunderstood?
>
> On Jan 22, 2018 22:27, "Ted Yu" <yuzhih...@gmail.com> wrote:
>
> > For case 1, HFile would be loaded into the region (via staging
> directory).
> >
> > Please see:
> > http://hbase.apache.org/book.html#arch.bulk.load
> >
> > On Mon, Jan 22, 2018 at 8:52 AM, vignesh <vignesh...@gmail.com> wrote:
> >
> > > If it is a bulk load I use spark hbase connector provided by
> hortonworks.
> > > For time series writes I use normal hbase client API's.
> > >
> > > So does that mean in case 2(client API write)  the write to memstore
> will
> > > happen via network? In case 1(bulk load)the HFile will be moved to the
> > > region which is supposed to hold or it will write to local and keep
> that
> > as
> > > a copy and the second replication would go to that particular region?
> > >
> > > On Jan 22, 2018 22:16, "Ted Yu" <yuzhih...@gmail.com> wrote:
> > >
> > > Which connector do you use to perform the write ?
> > >
> > > bq. Or spark will wisely launch an executor on that machine
> > >
> > > I don't think that is the case. Multiple writes may be performed which
> > > would end up on different region servers. Spark won't provide the
> > affinity
> > > described above.
> > >
> > > On Mon, Jan 22, 2018 at 7:19 AM, vignesh <vignesh...@gmail.com> wrote:
> > >
> > > > Hi,
> > > >
> > > > I have a Spark job which reads some timeseries data and pushes that
> to
> > > > HBASE using HBASE client API. I am executing this Spark job on a 10
> > > > node cluster. Say at first when spark kicks off it picks
> > > > machine1,machine2,machine3 as its executors. Now when the job inserts
> > > > a row to HBASE. Below is what my undersatnding on what it does.
> > > >
> > > > Based on the row key a particular region(from the META) would be
> > > > chosen and that row will be pushed to that RegionServer's memstore
> and
> > > > WAL and once the memestore is full it would be flushed to the
> disk.Now
> > > > if assume a particular row is being processed by a executor on
> > > > machine2 and the regionserver which handles that region to which the
> > > > put is to be made is on machine6. Will the data be transferred from
> > > > machine2 to machine6 over network and then the data will be stored in
> > > > memstore of machine6. Or spark will wisely launch an executor on that
> > > > machine during write(if the dynamic allocation is turned on) and
> > > > pushes to it?
> > > >
> > > >
> > > > --
> > > > I.VIGNESH
> > > >
> > >
> >
>

Re: How spark writes to HBASE

[Ted Yu]

For case 1, HFile would be loaded into the region (via staging directory).

Please see:
http://hbase.apache.org/book.html#arch.bulk.load

On Mon, Jan 22, 2018 at 8:52 AM, vignesh <vignesh...@gmail.com> wrote:

> If it is a bulk load I use spark hbase connector provided by hortonworks.
> For time series writes I use normal hbase client API's.
>
> So does that mean in case 2(client API write)  the write to memstore will
> happen via network? In case 1(bulk load)the HFile will be moved to the
> region which is supposed to hold or it will write to local and keep that as
> a copy and the second replication would go to that particular region?
>
> On Jan 22, 2018 22:16, "Ted Yu" <yuzhih...@gmail.com> wrote:
>
> Which connector do you use to perform the write ?
>
> bq. Or spark will wisely launch an executor on that machine
>
> I don't think that is the case. Multiple writes may be performed which
> would end up on different region servers. Spark won't provide the affinity
> described above.
>
> On Mon, Jan 22, 2018 at 7:19 AM, vignesh <vignesh...@gmail.com> wrote:
>
> > Hi,
> >
> > I have a Spark job which reads some timeseries data and pushes that to
> > HBASE using HBASE client API. I am executing this Spark job on a 10
> > node cluster. Say at first when spark kicks off it picks
> > machine1,machine2,machine3 as its executors. Now when the job inserts
> > a row to HBASE. Below is what my undersatnding on what it does.
> >
> > Based on the row key a particular region(from the META) would be
> > chosen and that row will be pushed to that RegionServer's memstore and
> > WAL and once the memestore is full it would be flushed to the disk.Now
> > if assume a particular row is being processed by a executor on
> > machine2 and the regionserver which handles that region to which the
> > put is to be made is on machine6. Will the data be transferred from
> > machine2 to machine6 over network and then the data will be stored in
> > memstore of machine6. Or spark will wisely launch an executor on that
> > machine during write(if the dynamic allocation is turned on) and
> > pushes to it?
> >
> >
> > --
> > I.VIGNESH
> >
>

Re: How spark writes to HBASE

[Ted Yu]

Which connector do you use to perform the write ?

bq. Or spark will wisely launch an executor on that machine

I don't think that is the case. Multiple writes may be performed which
would end up on different region servers. Spark won't provide the affinity
described above.

On Mon, Jan 22, 2018 at 7:19 AM, vignesh  wrote:

> Hi,
>
> I have a Spark job which reads some timeseries data and pushes that to
> HBASE using HBASE client API. I am executing this Spark job on a 10
> node cluster. Say at first when spark kicks off it picks
> machine1,machine2,machine3 as its executors. Now when the job inserts
> a row to HBASE. Below is what my undersatnding on what it does.
>
> Based on the row key a particular region(from the META) would be
> chosen and that row will be pushed to that RegionServer's memstore and
> WAL and once the memestore is full it would be flushed to the disk.Now
> if assume a particular row is being processed by a executor on
> machine2 and the regionserver which handles that region to which the
> put is to be made is on machine6. Will the data be transferred from
> machine2 to machine6 over network and then the data will be stored in
> memstore of machine6. Or spark will wisely launch an executor on that
> machine during write(if the dynamic allocation is turned on) and
> pushes to it?
>
>
> --
> I.VIGNESH
>

Re: Uninitialized Message Exception thrown while getting values.

[Ted Yu]

>From the exception message, it seems some GetRequest was empty.

How often did this happen ?
If you can describe characteristics of the get request, that may give some
clue.

Can you come up with a unit test that reproduces the issue ?

On Thu, Jan 18, 2018 at 11:40 PM, Karthick Ram 
wrote:

> -- Forwarded message --
> From: Karthick Ram 
> Date: Thu, Jan 18, 2018 at 12:56 PM
> Subject: Re: Uninitialized Message Exception thrown while getting values.
> To: ramkrishna vasudevan 
>
>
> Hbase version 1.2.5 and both client and server have pb version 2.5.0 only.
>
> On Thu, Jan 18, 2018 at 12:46 PM, ramkrishna vasudevan <
> ramkrishna.s.vasude...@gmail.com> wrote:
>
> > Hi
> >
> > Which version of HBase you get this problem? Do you have any pb classpath
> > issues?
> >
> > Regards
> > Ram
> >
> > On Thu, Jan 18, 2018 at 12:40 PM, Karthick Ram 
> > wrote:
> >
> >> "UninitializedMessageException : Message missing required fields :
> >> region, get", is thrown while performing Get. Due to this all the Get
> >> requests to the same Region Server are getting stalled.
> >>
> >> com.google.protobuf.UninitializedMessageException: Message missing
> >> required fields : region, get
> >> at com.google.protobuf.AbstractMessage$Build.newUninitializedMe
> >> ssageException(AbstractMessage.java:770)
> >> at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$GetR
> >> equest$Builder.build(ClientProtos.java:6377)
> >> at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$GetR
> >> equest$Builder.build(ClientProtos.java:6309)
> >> at org.apache.hadoop.hbase.ipc.RpcServer$Connection.processRequ
> >> est(RpcServer.java:1840)
> >> at org.apache.hadoop.hbase.ipc.RpcServer$Connection.processOneR
> >> pc(RpcServer.java:1775)
> >> at org.apache.hadoop.hbase.ipc.RpcServer$Connection.process(RPc
> >> Server.java:1623)
> >> at org.apache.hadoop.hbase.ipc.RpcServer$Connection.readAndProc
> >> ess(RpcServer.java:1603)
> >> at org.apache.hadoop.hbase.ipc.RpcServer$Listener.doRead(RpcSer
> >> ver.java:861)
> >> at org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunL
> >> oop(RpcServer.java:643)
> >> at org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(Rp
> >> cServer.java:619)
> >> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
> >> Executor.java:1145)
> >> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
> >> lExecutor.java:615)
> >> at java.lang.Thread.run(Thread.java:745)
> >>
> >
> >
>

Re: Scan Sequence

[Ted Yu]

bq. to get an suggest range search to client, and do this suggested range
search

Since you're using coprocessor, the coprocessor can act (range search)
using the suggestion and return the result to the client.
There is no need to return intermediate result to client.

BTW using zookeeper for locking is tricky - how do you keep up with region
split / merge ?

FYI

On Mon, Jan 15, 2018 at 7:24 PM, Yang Zhang <zhang.yang...@gmail.com> wrote:

> In fact , I am using coprocessor to Implement geography index. Each
> region will only have part of the Index to index the data that it contained
> . When I try to implement KNN search, only one region will contain the
> index according to the KNN center point. So this region will give an
> suggest of range search to other region（Because other region may also
> contain part of the data in the range search）.
>
> I am using zookeeper to implement the lock, other region will watch
> until they get the suggest. So if region1 is locked and wait for suggest
> ,and by the time region2 will never get the sacn request  and never give an
> suggest, this caused an dead lock.
>
> In other way, I can search region2 first (by limit the start and
> end)  to get an suggest range search to client, and do this suggested range
> search at the second time. This will work, but It will cost more time.
>
> Any suggest?
>
> Thanks
>
> 2018-01-15 23:49 GMT+08:00 Ted Yu <yuzhih...@gmail.com>:
>
> > bq. need region2 to search fist, because it has my own index.
> >
> > Looks like only some of the regions have your index. Can you tell us more
> > about the characteristics of the region(s) where index would be present.
> >
> > bq. My scan will be blocked for my lock on region1,
> >
> > By 'my' I assume the lock is placed by your (application) code. Since
> hbase
> > wouldn't know that a subsequent region has index (or suggestion) on what
> to
> > scan.
> > Can you give us more detail ?
> >
> > Thanks
> >
> > On Mon, Jan 15, 2018 at 7:42 AM, Yang Zhang <zhang.yang...@gmail.com>
> > wrote:
> >
> > > Hello Everyone
> > >
> > > I want to know If the Scan is send to HBase by sequence? For
> > > example, If I have a table that has two region（region1,region2）, and I
> > sent
> > > a Scan() to this Table. This Scan will be send to region1 first and
> until
> > > region1 returned, region2 can get this Scan? So If There are some
> methods
> > > than I can control the sequence?
> > >
> > >   some time I need region2 to search fist, because it has my own
> > index.
> > > Then It will give an suggest to other region about where to search.. I
> am
> > > using zookeeper to do this, But I found that My scan will be blocked
> for
> > my
> > > lock on region1, and region2 who has the suggest, even't can't receive
> > this
> > > scan.
> > >
> > >   Anyone have suggest?
> > >
> > >
> > > Thanks very much
> > >
> >
>

Re: Scan Sequence

[Ted Yu]

bq. need region2 to search fist, because it has my own index.

Looks like only some of the regions have your index. Can you tell us more
about the characteristics of the region(s) where index would be present.

bq. My scan will be blocked for my lock on region1,

By 'my' I assume the lock is placed by your (application) code. Since hbase
wouldn't know that a subsequent region has index (or suggestion) on what to
scan.
Can you give us more detail ?

Thanks

On Mon, Jan 15, 2018 at 7:42 AM, Yang Zhang  wrote:

> Hello Everyone
>
> I want to know If the Scan is send to HBase by sequence? For
> example, If I have a table that has two region（region1,region2）, and I sent
> a Scan() to this Table. This Scan will be send to region1 first and until
> region1 returned, region2 can get this Scan? So If There are some methods
> than I can control the sequence?
>
>   some time I need region2 to search fist, because it has my own index.
> Then It will give an suggest to other region about where to search.. I am
> using zookeeper to do this, But I found that My scan will be blocked for my
> lock on region1, and region2 who has the suggest, even't can't receive this
> scan.
>
>   Anyone have suggest?
>
>
> Thanks very much
>

Re: Avoiding duplicate writes

[Ted Yu]

Peter:
Normally java.lang.System.nanoTime() is used for measuring duration of time.

See also
https://www.javacodegeeks.com/2012/02/what-is-behind-systemnanotime.html

bq. the prePut co-processor is executed inside a record lock

The prePut hook is called with read lock on the underlying region.


Have you heard of HLC ? See HBASE-14070

The work hasn't been active recently.

FYI

On Thu, Jan 11, 2018 at 2:16 AM, Peter Marron 
wrote:

> Hi,
>
> We have a problem when we are writing lots of records to HBase.
> We are not specifying timestamps explicitly and so the situation arises
> where multiple records are being written in the same millisecond.
> Unfortunately when the records are written and the timestamps are the same
> then later writes are treated as updates of the previous records and not
> separate records, which is what we want.
> So we want to be able to guarantee that records are not treated as
> overwrites (unless we explicitly make them so).
>
> As I understand it there are (at least) two different ways to proceed.
>
> The first approach is to increase the resolution of the timestamp.
> So we could use something like java.lang.System.nanoTime()
> However although this seems to ameliorate the problem it seems to
> introduce other problems.
> Also ideally we would like something that guarantees that we don't lose
> writes rather than making them more unlikely.
>
> The second approach is to write a prePut co-processor.
> In the prePut I can do a read using the same rowkey, column family and
> column qualifier and omit the timestamp.
> As I understand it this will return me the latest timestamp.
> Then I can update the timestamp that I am going to write, if necessary, to
> make sure that the timestamp is always unique.
> In this way I can guarantee that none of my writes are accidentally turned
> into updates.
>
> However this approach seems to be expensive.
> I have to do a read before each write, and although (I believe) it will be
> on the same region server, it's still going to slow things down a lot.
> Also I am assuming that the prePut co-processor is executed inside a
> record lock so that I don't have to worry about synchronization.
> Is this true?
>
> Is there a better way?
>
> Maybe there is some implementation of this already that I can pick up?
>
> Maybe there is some way that I can implement this more efficiently?
>
> It seems to me that this might be better handled at compaction.
> Shouldn't there be some way that I can mark writes with some sort of
> special value of timestamp that means that this write should never be
> considered as an update but always as a separate write?
>
> Any advice gratefully received.
>
> Peter Marron
>

Re: Hbase hbck not working properly.

[Ted Yu]

Was hbase-site.xml on the classpath when you executed the command ?
 Original message From: Ravi Hemnani  
Date: 1/9/18  3:11 AM  (GMT-08:00) To: user@hbase.apache.org Subject: Re: Hbase 
hbck not working properly. 
@Ted,

I can the following command,
'sudo -u hbase hbase hbck', now the rootdir being taken is
/tmp/hbase-hbase/hbase instead of /hbase and still reports error for
inconsistencies.

I apologize for the delay in reply/update.

On Fri, Jan 5, 2018 at 6:05 PM, Ravi Hemnani 
wrote:

> Hello all,
>
> I am in a fix right now where hbase hbck command is not running properly.
> It is picking up incorrect rootdir in order to search for regions info in
> hdfs and hence ultimately throwing 'No HDFS dir found' for all the regions
> in the cluster. It is searching in hdfs://nameservice1/tmp/hbase-{user
> running hbck command}/hbase as show below. Also along with this, none of
> the hbase hbck options are working, repair etc and exiting with status code
> 255. I am using hbase version 1.2. If anyone has faced something like this
> please help.
>
>
> """
>
>
>    1. 2018-01-05 11:28:44,051 INFO  [main] util.HBaseFsck: Loading region 
>directories from HDFS
>    2. ERROR: Version file does not exist in root dir 
>hdfs://nameservice1/tmp/hbase-ravi/hbase
>    3.
>    4. 2018-01-05 11:28:44,162 INFO  [main] util.HBaseFsck: Loading region 
>information from HDFS
>
> """
>
> --
> Thank you and Regards,
>
> Ravi Hemnani.
>



-- 
Thank you and Regards,

Ravi Hemnani.

Re: StackOverflowError while compaction?

[Ted Yu]

The fix would be backported in HBASE-19730 and appear in 1.2.7 release.

Cheers

On Sun, Jan 7, 2018 at 8:41 PM, Kang Minwoo <minwoo.k...@outlook.com> wrote:

> Sorry, I have a mistake.
> StackOverflowError was due to the reverse scan.
>
> I dug system. While I had StackOverflowError, client executed the reverse
> scan.
> The reverse scan occurs StackOverFlowError.
>
> Related issue is https://issues.apache.org/jira/browse/HBASE-14497
>
> I wonder why that patch did not apply 1.2.6.
>
> Best regards,
> Minwoo Kang
>
>
> ____
> 보낸 사람: Ted Yu <yuzhih...@gmail.com>
> 보낸 날짜: 2018년 1월 7일 일요일 오전 9:50
> 받는 사람: user@hbase.apache.org
> 제목: Re: StackOverflowError while compaction?
>
> Can you provide a bit more information ?
>
> data block encoding for the column family where this error occurred
>
> pastebin of more of the region server prior to the StackOverflowError
> (after redaction)
>
> release of hadoop for the hdfs cluster
>
> non-default config which may be related
>
> Thanks
>
>
> On Sat, Jan 6, 2018 at 4:36 PM, Kang Minwoo <minwoo.k...@outlook.com>
> wrote:
>
> > Hello,
> >
> > I have met StackOverflowError in region server.
> > Detail Error log here...
> >
> > HBase version is 1.2.6
> >
> > DAYS:36,787 DEBUG [regionserver/longCompactions] regionserver.
> CompactSplitThread:
> > Not compacting xxx. because compaction request was cancelled
> > DAYS:36,787 DEBUG [regionserver/shortCompactions] compactions.
> ExploringCompactionPolicy:
> > Exploring compaction algori
> > thm has selected 0 files of size 0 starting at candidate #-1 after
> > considering 3 permutations with 0 in ratio
> > DAYS:36,787 DEBUG [regionserver/shortCompactions] compactions.
> RatioBasedCompactionPolicy:
> > Not compacting files becau
> > se we only have 0 files ready for compaction. Need 3 to initiate.
> > DAYS:36,787 DEBUG [regionserver/shortCompactions] regionserver.
> CompactSplitThread:
> > Not compacting xxx. because compaction request was cancelled
> > DAYS:38,028 ERROR [B.defaultRpcServer.handler=x,queue=x,port=x]
> > ipc.RpcServer: Unexpected throwable object
> > java.lang.StackOverflowError
> > at org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache.unref(
> > ShortCircuitCache.java:525)
> > at org.apache.hadoop.hdfs.shortcircuit.
> ShortCircuitReplica.unref(
> > ShortCircuitReplica.java:141)
> > at org.apache.hadoop.hdfs.BlockReaderLocal.close(
> > BlockReaderLocal.java:644)
> > at org.apache.hadoop.hdfs.DFSInputStream.
> closeCurrentBlockReader(
> > DFSInputStream.java:1682)
> > at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(
> > DFSInputStream.java:616)
> > at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(
> > DFSInputStream.java:882)
> > at org.apache.hadoop.hdfs.DFSInputStream.read(
> > DFSInputStream.java:934)
> > at java.io.DataInputStream.read(DataInputStream.java:149)
> > at org.apache.hadoop.hbase.io.hfile.HFileBlock.
> > readWithExtra(HFileBlock.java:709)
> > at org.apache.hadoop.hbase.io.hfile.HFileBlock$
> > AbstractFSReader.readAtOffset(HFileBlock.java:1440)
> > at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.
> > readBlockDataInternal(HFileBlock.java:1648)
> > at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.
> > readBlockData(HFileBlock.java:1532)
> > at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(
> > HFileReaderV2.java:452)
> > at org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$
> > BlockIndexReader.loadDataBlockWithScanInfo(HFileBlockIndex.java:271)
> > at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$
> > AbstractScannerV2.seekTo(HFileReaderV2.java:649)
> > at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$
> > AbstractScannerV2.seekTo(HFileReaderV2.java:599)
> > at org.apache.hadoop.hbase.regionserver.StoreFileScanner.
> > seekAtOrAfter(StoreFileScanner.java:268)
> > at org.apache.hadoop.hbase.regionserver.StoreFileScanner.
> > seekToPreviousRow(StoreFileScanner.java:461)
> > at org.apache.hadoop.hbase.regionserver.StoreFileScanner.
> > seekToPreviousRow(StoreFileScanner.java:476)
> > at org.apache.hadoop.hbase.regionserver.StoreFileScanner.
> > seekToPreviousRow(StoreFileScanner.java:476)
> > at org.apache.hadoop.hbase.regionserver.StoreFileScanner.
> > seekToPreviousRow(StoreFileScanner.java:476)
> > at o

Re: StackOverflowError while compaction?

[Ted Yu]

Can you provide a bit more information ?

data block encoding for the column family where this error occurred

pastebin of more of the region server prior to the StackOverflowError
(after redaction)

release of hadoop for the hdfs cluster

non-default config which may be related

Thanks


On Sat, Jan 6, 2018 at 4:36 PM, Kang Minwoo  wrote:

> Hello,
>
> I have met StackOverflowError in region server.
> Detail Error log here...
>
> HBase version is 1.2.6
>
> DAYS:36,787 DEBUG [regionserver/longCompactions] 
> regionserver.CompactSplitThread:
> Not compacting xxx. because compaction request was cancelled
> DAYS:36,787 DEBUG [regionserver/shortCompactions] 
> compactions.ExploringCompactionPolicy:
> Exploring compaction algori
> thm has selected 0 files of size 0 starting at candidate #-1 after
> considering 3 permutations with 0 in ratio
> DAYS:36,787 DEBUG [regionserver/shortCompactions] 
> compactions.RatioBasedCompactionPolicy:
> Not compacting files becau
> se we only have 0 files ready for compaction. Need 3 to initiate.
> DAYS:36,787 DEBUG [regionserver/shortCompactions] 
> regionserver.CompactSplitThread:
> Not compacting xxx. because compaction request was cancelled
> DAYS:38,028 ERROR [B.defaultRpcServer.handler=x,queue=x,port=x]
> ipc.RpcServer: Unexpected throwable object
> java.lang.StackOverflowError
> at org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache.unref(
> ShortCircuitCache.java:525)
> at org.apache.hadoop.hdfs.shortcircuit.ShortCircuitReplica.unref(
> ShortCircuitReplica.java:141)
> at org.apache.hadoop.hdfs.BlockReaderLocal.close(
> BlockReaderLocal.java:644)
> at org.apache.hadoop.hdfs.DFSInputStream.closeCurrentBlockReader(
> DFSInputStream.java:1682)
> at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(
> DFSInputStream.java:616)
> at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(
> DFSInputStream.java:882)
> at org.apache.hadoop.hdfs.DFSInputStream.read(
> DFSInputStream.java:934)
> at java.io.DataInputStream.read(DataInputStream.java:149)
> at org.apache.hadoop.hbase.io.hfile.HFileBlock.
> readWithExtra(HFileBlock.java:709)
> at org.apache.hadoop.hbase.io.hfile.HFileBlock$
> AbstractFSReader.readAtOffset(HFileBlock.java:1440)
> at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.
> readBlockDataInternal(HFileBlock.java:1648)
> at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.
> readBlockData(HFileBlock.java:1532)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(
> HFileReaderV2.java:452)
> at org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$
> BlockIndexReader.loadDataBlockWithScanInfo(HFileBlockIndex.java:271)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$
> AbstractScannerV2.seekTo(HFileReaderV2.java:649)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$
> AbstractScannerV2.seekTo(HFileReaderV2.java:599)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.
> seekAtOrAfter(StoreFileScanner.java:268)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.
> seekToPreviousRow(StoreFileScanner.java:461)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.
> seekToPreviousRow(StoreFileScanner.java:476)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.
> seekToPreviousRow(StoreFileScanner.java:476)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.
> seekToPreviousRow(StoreFileScanner.java:476)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.
> seekToPreviousRow(StoreFileScanner.java:476)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.
> seekToPreviousRow(StoreFileScanner.java:476)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.
> seekToPreviousRow(StoreFileScanner.java:476)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.
> seekToPreviousRow(StoreFileScanner.java:476)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.
> seekToPreviousRow(StoreFileScanner.java:476)
>
> Has anyone experienced similar problems?
>
> Best regards,
> Minwoo Kang
>

Re: Hbase hbck not working properly.

[Ted Yu]

Please run hbck as hbase superuser, normally user hbase.

Cheers


On Fri, Jan 5, 2018 at 4:35 AM, Ravi Hemnani 
wrote:

> Hello all,
>
> I am in a fix right now where hbase hbck command is not running properly.
> It is picking up incorrect rootdir in order to search for regions info in
> hdfs and hence ultimately throwing 'No HDFS dir found' for all the regions
> in the cluster. It is searching in hdfs://nameservice1/tmp/hbase-{user
> running hbck command}/hbase as show below. Also along with this, none of
> the hbase hbck options are working, repair etc and exiting with status code
> 255. I am using hbase version 1.2. If anyone has faced something like this
> please help.
>
>
> """
>
>
>1. 2018-01-05 11:28:44,051 INFO  [main] util.HBaseFsck: Loading
> region directories from HDFS
>2. ERROR: Version file does not exist in root dir
> hdfs://nameservice1/tmp/hbase-ravi/hbase
>3.
>4. 2018-01-05 11:28:44,162 INFO  [main] util.HBaseFsck: Loading
> region information from HDFS
>
> """
>
> --
> Thank you and Regards,
>
> Ravi Hemnani.
>

Re: assert in hbase 1.4

[Ted Yu]

Thanks, Clément

On Wed, Jan 3, 2018 at 2:54 PM, Clément Guillaume <clem...@guillaume.bzh>
wrote:

> Done https://issues.apache.org/jira/browse/HBASE-19700
>
> 2017-12-31 8:16 GMT-08:00 Ted Yu <yuzhih...@gmail.com>:
>
>> Clément:
>> Since you have reproduced the assertion error, can you log a JIRA ?
>>
>> Thanks
>>
>> On Sun, Dec 31, 2017 at 7:46 AM, Ted Yu <yuzhih...@gmail.com> wrote:
>>
>>> I modified the test so that the unit test runs against branch-1.4 :
>>>
>>> +  Configuration hbaseConfig = TEST_UTIL.getConfiguration();
>>> +  //hbaseConfig.set(HConstants.ZOOKEEPER_QUORUM,
>>> "hbase123.docker");
>>>
>>> However the test runs to completion without assertion - meaning the
>>> assertion wouldn't be triggered in homogeneous cluster.
>>>
>>> Here were the last few lines in the test output:
>>>
>>> 2017-12-31 07:33:36,865 INFO  [Time-limited test]
>>> client.HBaseAdmin$CreateTableFuture(793): Created AssertionTest
>>> table created
>>> 2017-12-31 07:33:36,871 DEBUG [Time-limited test]
>>> client.ClientScanner(248): Advancing internal scanner to startKey at
>>> '\x01\x02\x03\x04\x05\x06\x07', inclusive
>>> 2017-12-31 07:33:36,873 DEBUG [hconnection-0x6e0982ab-shared--pool24-t3]
>>> ipc.RpcConnection(126): Use SIMPLE authentication for service
>>> ClientService, sasl=false
>>> 2017-12-31 07:33:36,873 DEBUG [hconnection-0x6e0982ab-shared--pool24-t3]
>>> ipc.BlockingRpcConnection(431): Connecting to
>>> tyus-mbp.attlocal.net/192.168.1.64:61585
>>> 2017-12-31 07:33:36,875 DEBUG [RpcServer.listener,port=61585]
>>> ipc.RpcServer$Listener(920): RpcServer.listener,port=61585: connection from
>>> 192.168.1.64:61633; # active  connections: 3
>>> 2017-12-31 07:33:36,875 INFO  [RpcServer.reader=0,bindAddress=
>>> tyus-mbp.attlocal.net,port=61585] ipc.RpcServer$Connection(1809): Auth
>>> successful for tyu (auth:SIMPLE)
>>> 2017-12-31 07:33:36,875 INFO  [RpcServer.reader=0,bindAddress=
>>> tyus-mbp.attlocal.net,port=61585] ipc.RpcServer$Connection(1839):
>>> Connection from 192.168.1.64 port: 61633 with   version info: version:
>>> "1.4.1-SNAPSHOT" url: "git://TYus-MBP.attlocal.net/Users/tyu/1.4-hbase"
>>> revision: "a7383851b9e62f23eb6f06eb23f09967f636c985" user: "tyu" date:
>>> "Sun Dec  31 07:32:58 PST 2017" src_checksum: 
>>> "39bdbbe40374e2094f53ce77af539671"
>>> version_major: 1 version_minor: 4
>>> 2017-12-31 07:33:36,917 INFO  [main] hbase.ResourceChecker(171): after:
>>> client.TestHCM#test Thread=325 (was 305)
>>>
>>> I assume you pointed the test at a 1.2.3 cluster which triggered the
>>> assertion.
>>>
>>> It seems the patch which introduced the assertion didn't take into
>>> account client running against older release cluster deployment.
>>>
>>> FYI
>>>
>>> On Sun, Dec 31, 2017 at 6:09 AM, Clément Guillaume <
>>> clem...@guillaume.bzh> wrote:
>>>
>>>> I have consistent failure with this code (a table with pre-splits):
>>>>
>>>> @Test
>>>>> public void test() throws IOException{
>>>>> Configuration hbaseConfig = HBaseConfiguration.create();
>>>>> hbaseConfig.set(HConstants.ZOOKEEPER_QUORUM, "hbase123.docker");
>>>>> Connection connection = ConnectionFactory.createConnec
>>>>> tion(hbaseConfig);
>>>>>
>>>>
>>>>
>>>> TableName tableName = TableName.valueOf("AssertionTest");
>>>>> Admin admin = connection.getAdmin();
>>>>>
>>>>> if(!admin.tableExists(tableName)){
>>>>> HTableDescriptor htable = new HTableDescriptor(tableName);
>>>>> htable.addFamily(new HColumnDescriptor(new byte[]{(byte)'a'}));
>>>>> byte[][] splitPoints = {{1, 2, 3, 4, 5, 6, 7}};
>>>>> admin.createTable(htable, splitPoints);
>>>>> System.out.println("table created");
>>>>> }
>>>>>
>>>>> Table table = connection.getTable(tableName);
>>>>> ResultScanner scanner = table.getScanner(new Scan());
>>>>> scanner.iterator().hasNext(); // Exception thrown here
>>>>> }
>>>>
>>>>
>>>> 2017-12-30 5:04 GMT-08:00 Ted Yu <yuzhih...@gmail.com>:
>>>>
>>>>> bq. a table has been creat

Re: Delete a CF paramter?

[Ted Yu]

Currently there is no support for family_att_unset operation.

You can log a JIRA for adding such support.

Cheers

On Mon, Jan 1, 2018 at 8:55 AM, Jean-Marc Spaggiari <jean-m...@spaggiari.org
> wrote:

> At the table create I used BLOOMFILTER => 'NONE'. Which is an overwrite of
> the default value. What I'm looking for is something like:
> alter 'test', { NAME => 'cf', METHOD => 'family_att_unset', FNAME =>
> 'BLOOMFILTER' }
>
> To restore what default value is.
>
> I agree that your option works, but what if you don't know the default
> value? What I want is just remove what has been setup.
>
> Same here:
> {NAME => '@', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false',
> NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'false',
> CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL =>
> 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER =>
> 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false',
> CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false',
> COMPRESSION => 'SNAPPY', BLOCKCACHE => 'true', BLOCKSIZE => '65536',
> *METADATA
> => {'ENCODE_ON_DISK' => 'true'}*}
>
> If I want to remove the METADATA parameter for the '@' Column Familiy...
> How do I do?
>
> 2018-01-01 11:49 GMT-05:00 Ted Yu <yuzhih...@gmail.com>:
>
> > Not sure what you meant by deleting bloom filter type.
> > Bloom filter type is used when constructing store file. See the following
> > in HStore:
> >
> >   private HStoreFile createStoreFileAndReader(StoreFileInfo info) throws
> > IOException {
> >
> > info.setRegionCoprocessorHost(this.region.getCoprocessorHost());
> >
> > HStoreFile storeFile = new HStoreFile(this.getFileSystem(), info,
> > this.
> > conf, this.cacheConf,
> >
> > this.family.getBloomFilterType(), isPrimaryReplicaStore());
> >
> >
> > If you want to use a different bloom filter, you can issue this command:
> >
> >
> > alter 'test', { NAME => 'cf', BLOOMFILTER => 'ROW' }
> >
> >
> > Cheers
> >
> > On Mon, Jan 1, 2018 at 6:22 AM, Jean-Marc Spaggiari <
> > jean-m...@spaggiari.org
> > > wrote:
> >
> > > Hi,
> > >
> > > What is the magic to delete a CF parameter?
> > >
> > > Like in this example:
> > > hbase(main):033:0> desc 'table'
> > > Table dns is ENABLED
> > >
> > >
> > >
> > >
> > > table
> > >
> > >
> > >
> > >
> > > COLUMN FAMILIES DESCRIPTION
> > >
> > >
> > >
> > >
> > > {NAME => '@', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false',
> > > NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'false',
> > > CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL =>
> > > 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER
> =>
> > > 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEM
> > > ORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false',
> PREFETCH_BLOCKS_ON_OPEN
> > > => 'false', COMPRESSION => 'SNAPPY', BLOCKCACHE => 'true', BLOCKSIZE =>
> > > '65536', METADATA => {'ENCODE_ON_DISK' => 'true'}}
> > >
> > >
> > > 1 row(s)
> > > Took 0.0368 seconds
> > >
> > >
> > >
> > >
> > > I want to remove the BLOOMFILTER => 'NONE',  parameter that I have
> set. I
> > > found how to remove the CF, table parameters and options, etc. But I
> have
> > > not found how to delete the CF parameter. IIRC there was something with
> > > :NILL or similar, but I'm not 100% sure.
> > >
> > > In any case, I feel this should be better documented :(
> > >
> > > JMS
> > >
> >
>