Re: Upgrading cdh5.16.2 to apache hbase 2.4 using replication

[Anoop John]

Yes the command enable_table_replication will check whether a table exists
in  peer cluster and if so compare the CFs.  Ya you correctly said. The
difference in the table description results in failure of this command. You
can enable replication at src using the alter table command.  We can fix
this issue. WDYT @Wellington ?

Anoop

On Thu, May 20, 2021 at 5:07 PM bq zhao  wrote:

> We tested replication between Apache HBase1.4 and Apache HBase2.2. We found
> that if you use 'enable_table_replication' command to enable replication,
> it will compares table schemas before start. But HBase2 has more default
> parameters than HBase1, which leads to schema comparison failed. However,
> if you use alter command like “alter 'test:test_replication',{NAME =>
> 'f',REPLICATION_SCOPE => '1'}”, the replication can start successfully !
>
> There is also a related jira tracking this issue:
> https://issues.apache.org/jira/browse/HBASE-24353
>
> Wellington Chevreuil  于2021年5月20日周四
> 下午5:07写道：
>
> > Yes, replication interfaces are compatible between these two major
> > versions.
> >
> > So I created two clusters in AWS and tried enable replication between
> HBase
> > > 1.4.13 and 2.2.5. But have got error "table exists but descriptors are
> > not
> > > the same" (I will put screenshot in the attachment but not sure it will
> > > work here).
> > >
> > Can you describe in detail which steps (commands/configurations) have you
> > executed to get the mentioned error? After which exact action have you
> seen
> > this message? And are the tables schema identical in both clusters?
> >
> > Em qua., 19 de mai. de 2021 às 21:00, Sergey Semenoff <
> > box4semen...@gmail.com> escreveu:
> >
> > > We are thinking about simulator issue. Our clusters much less - 4 by
> 100
> > > RS however we need process data continuously too. So I created two
> > clusters
> > > in AWS and tried enable replication between HBase 1.4.13 and 2.2.5. But
> > > have got error "table exists but descriptors are not the same" (I will
> > put
> > > screenshot in the attachment but not sure it will work here).
> > >
> > > I have some ideas how to make upgrade by another way and would glad to
> > > discuss it with you. So you could write me at box4semen...@gmail.com
> to
> > > dig to details.
> > >
> > > ср, 19 мая 2021 г., 15:50 Bryan Beaudreault
> > > :
> > >
> > >> We are running about 40 HBase clusters, with over 5000 regionservers
> > >> total.
> > >> These are all running cdh5.16.2. We also have thousands of clients
> (from
> > >> APIs to kafka workers to hadoop jobs, etc) hitting these various
> > clusters,
> > >> also running cdh5.16.2.
> > >>
> > >> We are starting to plan an upgrade to hbase 2.x and hadoop 3.x. I've
> > read
> > >> through the docs on https://hbase.apache.org/book.html#_upgrade_paths
> ,
> > >> and
> > >> am starting to plan our approach. More than a few seconds of downtime
> is
> > >> not an option, but rolling upgrade also seems risky (if not impossible
> > for
> > >> our version).
> > >>
> > >> One thought I had is whether replication is compatible between these
> two
> > >> versions. If so, we probably would consider swapping onto upgraded
> > >> clusters
> > >> using backup/restore + replication. If we were to go this route we'd
> > >> probably want to consider bi-directional replication so that we can
> roll
> > >> back to the old cluster if there's a regression.
> > >>
> > >> Does anyone have any experience with this approach? Is replication
> > >> protocol
> > >> compatible across the seversions? Any concerns, tips or other
> > >> considerations to keep in mind? We do the backup/restore + replication
> > >> approach pretty regularly to move tables between clusters.
> > >>
> > >> Thanks!
> > >>
> > >
> >
>

Re: JMX Issues

[Anoop John]

> ERROR [1955596624@qtp-58762-113898] mortbay.log:
/jmxjava.lang.IllegalStateException: Committed
This second exception trace is because of code level bug in
JMXJsonServlet.   That wont come if the jmx url is corrected anyways.  I
will file a jira to fix this code level issue

Anoop

On Tue, Jun 30, 2020 at 12:00 AM Stack  wrote:

> On Mon, Jun 29, 2020 at 11:07 AM Govindhan S 
> wrote:
>
> >  I got this figured out and solved.
> > Issue with the AMSà Advanced AMS-hbase-env-template à optional_cp (given
> > below)
> > Had extra : present with wrong Jar file, hence the issue. Correcting the
> > same cleared me.
> > Thanks everyone.
> > ~ Govins
> >
>
> Glad you figured it.
> S
>
>
>
> > On Friday, 26 June, 2020, 10:00:56 pm IST, Govindhan S <
> > govin...@yahoo.com> wrote:
> >
> >  Hello Team,
> > I am getting the below error flooding on my hmaster logs.. Any
> suggestions
> > or way forwards might help me out.
> > Version -> 1.1.2
> > ERROR [1955596624@qtp-58762-113898] jmx.JMXJsonServlet: Caught an
> > exception while processing JMX
> > requestjavax.management.MalformedObjectNameException: Invalid character
> ':'
> > in key part of property
> >  WARN  [1955596624@qtp-58762-113898] mortbay.log: Committed before
> > 400 null
> >
> > ERROR [1955596624@qtp-58762-113898] mortbay.log:
> > /jmxjava.lang.IllegalStateException: Committed
> >
> >
> > ~ Govins.
>

Re: HBase 2 slower than HBase 1?

[Anoop John]

In another mail thread Zheng Hu brought up an important Jra fix
https://issues.apache.org/jira/browse/HBASE-21657
Can u pls check with this once?

Anoop


On Tue, Jun 9, 2020 at 8:08 PM Jan Van Besien  wrote:

> On Sun, Jun 7, 2020 at 7:49 AM Anoop John  wrote:
> > As per the above configs, it looks like Bucket Cache is not being used.
> > Only on heap LRU cache in use.
>
> True (but it is large enough to hold everything, so I don't think it
> matters).
>
> > @Jan - Is it possible for you to test with off heap Bucket Cache?
>  Config
> > bucket cache off heap mode with size ~7.5 GB
>
> I did a quick test but it seems not to make a measurable difference.
> If anything, it is actually slightly slower even. I see 100% hit ratio
> in the L1
> LruBlockCache and effectively also 100% in the L2 BucketCache (hit
> ratio is not yet at 100% but hits increase with every test and misses
> do not).
>
> Given that the LruBlockCache was already large enough to cache all the
> data anyway, I did not expect this to help either, to be honest.
>
> > Do you have any DataBlockEncoding enabled on the CF?
>
> Yes, FAST_DIFF. But this is of course true in both the tests with
> hbase2 and hbase1.
>
> Jan
>

Re: HBase 2 slower than HBase 1?

[Anoop John]

As per the above configs, it looks like Bucket Cache is not being used.
Only on heap LRU cache in use.
@Jan - Is it possible for you to test with off heap Bucket Cache?   Config
bucket cache off heap mode with size ~7.5 GB
Do you have any DataBlockEncoding enabled on the CF?

Anoop

On Fri, Jun 5, 2020 at 8:32 PM 张铎(Duo Zhang)  wrote:

> IIRC on branch-2.x, there are some changes on the bucket cache
> implementation? Off-heap by default? Not sure if this is the problem.
>
>
> Jan Van Besien  于2020年6月5日周五 下午9:17写道：
>
> > On Fri, Jun 5, 2020 at 2:54 PM 张铎(Duo Zhang) 
> > wrote:
> > > So the result is for all data in block cache? What is the block cache
> > > settings for these two clusters?
> >
> > yes.
> >
> > Configuration:
> > - hfile.block.cache.size = 0.4
> > - heap size 18.63 GiB (don't ask me why it was not 20 ;-))
> >
> > some relevant statistics from one of the region servers:
> > - Cache Size Limit 7.5 GB
> > - Size of Blocks 1.6 GB
> >
> > When doing one of the scan tests, the hits increase, the misses stay
> > stable, and consequently the hit ratio increases as well. So
> > essentially it is a 100% hit ratio for the scope of the tests.
> >
> > Jan - NGDATA
> >
>

Re: [ANNOUNCE] New HBase committer Viraj Jasani

[Anoop John]

Congrats Viraj !!!

Anoop

On Tue, Dec 31, 2019 at 9:26 AM Sukumar Maddineni
 wrote:

> Wow congrats Viraj and Keep up the good work.
>
> --
> Sukumar
>
> On Mon, Dec 30, 2019 at 5:45 PM 宾莉金（binlijin）  wrote:
>
> > Welcome and Congratulations, Viraj!
> >
> > 张铎(Duo Zhang)  于2019年12月30日周一 下午1:18写道：
> >
> > > Congratulations!
> > >
> > > 李响  于2019年12月30日周一 上午10:12写道：
> > >
> > > >- Congratulations and warmly welcome \^o^/
> > > >
> > > >
> > > > On Sun, Dec 29, 2019 at 2:14 AM Jan Hentschel <
> > > > jan.hentsc...@ultratendency.com> wrote:
> > > >
> > > > > Congrats and welcome, Viraj! Well deserved.
> > > > >
> > > > > From: Peter Somogyi 
> > > > > Reply-To: "d...@hbase.apache.org" 
> > > > > Date: Friday, December 27, 2019 at 2:02 PM
> > > > > To: HBase Dev List , hbase-user <
> > > > > user@hbase.apache.org>
> > > > > Subject: [ANNOUNCE] New HBase committer Viraj Jasani
> > > > >
> > > > > On behalf of the Apache HBase PMC I am pleased to announce that
> > > > > Viraj Jasani has accepted the PMC's invitation to become a
> > > > > commiter on the project.
> > > > >
> > > > > Thanks so much for the work you've been contributing. We look
> forward
> > > > > to your continued involvement.
> > > > >
> > > > > Congratulations and welcome!
> > > > >
> > > > >
> > > >
> > > > --
> > > >
> > > >李响 Xiang Li
> > > >
> > > > 手机 cellphone ：+86-136-8113-8972
> > > > 邮件 e-mail  ：wate...@gmail.com
> > > >
> > >
> >
> >
> > --
> > *Best Regards,*
> >  lijin bin
> >
>
>
> --
>
> 
>

Re: HBase Scan consumes high cpu

[Anoop John]

Hi
When you did a put with a lower qualifier int  (put 'mytable',
'MY_ROW', "pcf:\x0A", "\x00") the system flow is getting a valid cell at
1st step itself and that getting passed to the Filter.  The Filter is doing
a seek which just avoids all the in between deletes and puts processing..
In 1st case the Filter wont get into action at all unless the scan flow
sees a valid cell.  The delete processing happens as 1st step before the
filter processinf step happening.

In this case I am wondering why you can not add the specific 1st qualifier
in the get part itself along with the column range filter.   I mean

get 'mytable', 'MY_ROW', {COLUMN=>['pcf: *1499000 * '],
FILTER=>ColumnRangeFilter.new(Bytes.toBytes(1499000.to_java(:int)),
true, Bytes.toBytes(1499010.to_java(:int)), false)}

Pardon the syntax it might not be proper for the shell..  Can this be done?
This will make the scan to make a seek to the given qualifier at 1st step
itself.

Anoop

On Thu, Sep 12, 2019 at 10:18 PM Udai Bhan Kashyap (BLOOMBERG/ PRINCETON) <
ukashy...@bloomberg.net> wrote:

> Are you keeping the deleted cells? Check 'VERSIONS' for the column family
> and set it to 1 if you don't want to keep the deleted cells.
>
> From: user@hbase.apache.org At: 09/12/19 12:40:01To:
> user@hbase.apache.org
> Subject: Re: HBase Scan consumes high cpu
>
> Hi,
>
> As said earlier, we have populated the rowkey "MY_ROW" with integers
> from 0 to 150 as column qualifiers. Then we have deleted the
> qualifiers from 0 to 1499000.
>
> We executed the following query. It took 15.3750 seconds to execute.
>
> hbase(main):057:0> get 'mytable', 'MY_ROW', {COLUMN=>['pcf'],
> FILTER=>ColumnRangeFilter.new(Bytes.toBytes(1499000.to_java(:int)),
> true, Bytes.toBytes(1499010.to_java(:int)), false)}
> COLUMNCELL
>   pcf:\x00\x16\xDFxtimestamp=1568123881899,
> value=\x00\x16\xDFx
>   pcf:\x00\x16\xDFytimestamp=1568123881899,
> value=\x00\x16\xDFy
>   pcf:\x00\x16\xDFztimestamp=1568123881899,
> value=\x00\x16\xDFz
>   pcf:\x00\x16\xDF{timestamp=1568123881899,
> value=\x00\x16\xDF{
>   pcf:\x00\x16\xDF|timestamp=1568123881899,
> value=\x00\x16\xDF|
>   pcf:\x00\x16\xDF}timestamp=1568123881899,
> value=\x00\x16\xDF}
>   pcf:\x00\x16\xDF~timestamp=1568123881899,
> value=\x00\x16\xDF~
>   pcf:\x00\x16\xDF\x7F timestamp=1568123881899,
> value=\x00\x16\xDF\x7F
>   pcf:\x00\x16\xDF\x80 timestamp=1568123881899,
> value=\x00\x16\xDF\x80
>   pcf:\x00\x16\xDF\x81 timestamp=1568123881899,
> value=\x00\x16\xDF\x81
> 1 row(s) in 15.3750 seconds
>
>
> Now we inserted a new column with qualifier 10 (\x0A), such that it
> comes earlier in lexicographical order. Now we executed the same query.
> It only took 0.0240 seconds.
>
> hbase(main):058:0> put 'mytable', 'MY_ROW', "pcf:\x0A", "\x00"
> 0 row(s) in 0.0150 seconds
> hbase(main):059:0> get 'mytable', 'MY_ROW', {COLUMN=>['pcf'],
> FILTER=>ColumnRangeFilter.new(Bytes.toBytes(1499000.to_java(:int)),
> true, Bytes.toBytes(1499010.to_java(:int)), false)}
> COLUMNCELL
>   pcf:\x00\x16\xDFxtimestamp=1568123881899,
> value=\x00\x16\xDFx
>   pcf:\x00\x16\xDFytimestamp=1568123881899,
> value=\x00\x16\xDFy
>   pcf:\x00\x16\xDFztimestamp=1568123881899,
> value=\x00\x16\xDFz
>   pcf:\x00\x16\xDF{timestamp=1568123881899,
> value=\x00\x16\xDF{
>   pcf:\x00\x16\xDF|timestamp=1568123881899,
> value=\x00\x16\xDF|
>   pcf:\x00\x16\xDF}timestamp=1568123881899,
> value=\x00\x16\xDF}
>   pcf:\x00\x16\xDF~timestamp=1568123881899,
> value=\x00\x16\xDF~
>   pcf:\x00\x16\xDF\x7F timestamp=1568123881899,
> value=\x00\x16\xDF\x7F
>   pcf:\x00\x16\xDF\x80 timestamp=1568123881899,
> value=\x00\x16\xDF\x80
>   pcf:\x00\x16\xDF\x81 timestamp=1568123881899,
> value=\x00\x16\xDF\x81
> 1 row(s) in 0.0240 seconds
> hbase(main):060:0>
>
>
> We were able to reproduce the result consistently same, the pattern
> being bulk insert followed by bulk delete of most of the earlier columns.
>
>
> We observed the following behaviour while debugging the StoreScanner
> (regionserver).
>
> Case 1:
>
> 1. When StoreScanner.next() is called, it starts to iterate over the
> cells from the start of the rowkey.
>
> 2. As all the cells are deleted (from 0 to 1499000), we could see
> alternate delete and put type cells. Now, the
> NormalUserScanQueryMatcher.match() returns
> ScanQueryMatcher.MatchCode.SKIP and
> ScanQueryMatcher.MatchCode.SEEK_NEXT_COL for Delete and Put type cell
> respectively. This iteration happens throughout the range of 0 to 1499000.
>
> 3. This happens until a valid Put type cell is encountered, where the
> matcher applies the ColumnRangeFilter to the cell, which in turm returns
> 

Re: [ANNOUNCE] New HBase committer Tak-Lon (Stephen) Wu

[Anoop John]

Congratulations Stephen.

-Anoop-


On Tue, Aug 6, 2019 at 11:19 AM Pankaj kr  wrote:

> Congratulations Stephen..!!
>
> Regards,
> Pankaj
>
> -Original Message-
> From: Sean Busbey [mailto:bus...@apache.org]
> Sent: 06 August 2019 00:27
> To: dev ; user@hbase.apache.org
> Subject: [ANNOUNCE] New HBase committer Tak-Lon (Stephen) Wu
>
> On behalf of the Apache HBase PMC I am super pleased to announce that
> Tak-Lon (Stephen) Wu has accepted the PMC's invitation to become a commiter
> on the project.
>
> Thanks so much for the work you've been contributing. We look forward to
> your continued involvement.
>
> Congratulations and welcome!
>
> -busbey
>

Re: Re: [ANNOUNCE] Please welcome Zheng Hu to the HBase PMC

[Anoop John]

Congrats Zheng.

Anoop

On Tue, Aug 6, 2019 at 8:52 AM OpenInx  wrote:

> I'm so glad to join the PMC, Apache HBase is a great open source project
> and the
> community is also very nice and friendly.  In the comming days,   will do
> more to make
> the project & community forward, also we need more work to attract people
> to enhance
> and expand the community: blog/post/book/talk to share hbase experience or
> ecosystem,
> answer the question from our user list in time, help the beginner to
> contribute the code, quick
> feedback to our release vote / patch reviewing, etc.  all those things will
> be helpful I think and
> I will try to do more things like the above :-)
>
> Thanks all.
>
> On Tue, Aug 6, 2019 at 10:27 AM Chunhui Shen  wrote:
>
> > Congratulations, Zheng Hu!
> >
> >
> > 在 2019-08-06 00:54:51，"Xu Cang"  写道：
> > >Congratulations, Zheng Hu!
> > >
> > >On Mon, Aug 5, 2019 at 6:07 AM Guangxu Cheng 
> > wrote:
> > >
> > >> Congratulations, Zheng!
> > >>
> > >> Allan Yang  于2019年8月5日周一 下午5:13写道：
> > >>
> > >> > Congratulations, Hu！
> > >> > Best Regards
> > >> > Allan Yang
> > >> >
> > >> >
> > >> > Peter Somogyi  于2019年8月5日周一 下午4:47写道：
> > >> >
> > >> > > Congratulations!
> > >> > >
> > >> > > On Mon, Aug 5, 2019 at 8:57 AM Pankaj kr 
> > wrote:
> > >> > >
> > >> > > > Congratulations Zheng..!!
> > >> > > >
> > >> > > > Regards,
> > >> > > > Pankaj
> > >> > > >
> > >> > > > -Original Message-
> > >> > > > From: Duo Zhang [mailto:zhang...@apache.org]
> > >> > > > Sent: 05 August 2019 07:38
> > >> > > > To: HBase Dev List ; hbase-user <
> > >> > > > user@hbase.apache.org>
> > >> > > > Subject: [ANNOUNCE] Please welcome Zheng Hu to the HBase PMC
> > >> > > >
> > >> > > > On behalf of the Apache HBase PMC I am pleased to announce that
> > Zheng
> > >> > Hu
> > >> > > > has accepted our invitation to become a PMC member on the Apache
> > >> HBase
> > >> > > > project. We appreciate Zheng Hu stepping up to take more
> > >> responsibility
> > >> > > in
> > >> > > > the HBase project.
> > >> > > >
> > >> > > > Please join me in welcoming Zheng Hu to the HBase PMC!
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
>

Re: Atomic read-modify-write: constructing an interesting TimeRange for CheckAndMutateBuilder.timeRange

[Anoop John]

We might need a static creator method which takes start and end time in TR?

Anoop

On Tue, Aug 6, 2019 at 9:11 AM OpenInx  wrote:

> Hi
> I've checked the code, I think you can use the deprecated TimeRange(ts+1)
> first,  also the methods defined in TimeRange
> is not good enough now, we may need to create an issue to address this
> thing.
>
> Thanks.
>
> On Tue, Aug 6, 2019 at 10:15 AM Wilson, Huon (Data61, Eveleigh) <
> huon.wil...@data61.csiro.au> wrote:
>
> > Hi,
> >
> > I’m trying to do an atomic read-modify-write where a cell is read and
> then
> > rewritten with an updated value (updated in a complicated way, more than
> > just incrementing), but only if there’s been no writes since it was read,
> > otherwise try again from the read.
> >
> > My plan is to use something like
> >
> > table.checkAndMutate(row, family).qualifier(qualifier).timeRange(new
> > TimeRange(ts + 1)).ifNotExists().put(put)
> >
> > to insert if the cell row/family/qualifier doesn't exist in the range
> `[ts
> > + 1, infinity)`, which I think means no writes after time `ts`. However,
> > that TimeRange constructor is @Deprecated and @Private, and the only
> > remaining ways to publicly construct a TimeRange is
> `TimeRange.allTime()`
> > ([0, infinity)) and `TimeRange.at(long ts)` ([ts, ts + 1)), neither of
> > which are right.
> >
> > Is there a way to create this time range, or another way to achieve this
> > RMW operation?
> >
> > An alternative is to do a value comparison with `.ifEquals(...)` but my
> > values could be quite large (kilobytes), so I was thinking this range
> > comparison would be better.
> >
> > ---
> > Huon Wilson
> > CSIRO | Data61
> > https://www.data61.csiro.au
> >
> >
> >
> >
> >
>

Re: [ANNOUNCE] new HBase committer Sakthi

[Anoop John]

Congratulations Sakthi 

-Anoop-

On Sat, Aug 3, 2019 at 2:11 AM Stack  wrote:

> Hurray!
> S
>
> On Wed, Jul 31, 2019 at 5:04 PM Sean Busbey  wrote:
>
> > On behalf of the HBase PMC, I'm pleased to announce that Sakthi has
> > accepted our invitation to become an HBase committer.
> >
> > We'd like to thank Sakthi for all of his diligent contributions to the
> > project thus far. We look forward to his continued participation in our
> > community.
> >
> > Congrats and welcome Sakthi!
> >
>

Re: Re: question on hfile size upper limit

[Anoop John]

Based on what u pasted as the config
"
hbase.hregion.max.filesize
10737418240

Maximum HStoreFile size. If any one of a column families' HStoreFiles
has
grown to exceed this value, the hosting HRegion is split in
two.
  "

I can say the issue is the version of HBase.

Older HBase versions had this behave what u said.  When a file under a
region's CF grow above the max limit, the region will split.The reason
why the check was like that is we any way try to major compact files under
a CF into one large file.  So the check based on larger file was ok/

This way is changed later and we start checking the sum of all files under
a region:cf.  Am not sure which version introduced this.   This became a
need when we supported feature like Date Tiered Compaction/ Stripe
Compaction.

So for you to have the required behave, try upgrade to a newer version.

Anoop


On Thu, Jun 20, 2019 at 9:55 PM Jean-Marc Spaggiari 
wrote:

> Hi,
>
> Just updating what I said (Thanks Anoop for the warning). I took the
> assumption that you have a single CF... The maxfilesize is per CF, not per
> region. If you have a single CF, then it become the same as per region, but
> a region will split whenever one of the CFs reaches the limit.
>
> HBase will not split a single row. So if you have a single row that grows
> bigger than the maxfilesize, the region will keep growing. You need to
> assess this risk when you do your table design and avoid it. It will not
> split even if there is millions of column qualifiers. A region is defines
> by a start row and a stop row. Therefore a single row can belong only to a
> single region.
>
> JMS
>
> Le jeu. 20 juin 2019 à 05:00, Roshan  a écrit :
>
> > Hi,
> >
> > If the single rowkey in the table exceeds the size of defined
> > hbase.hregion.max.filesize, whether the region will split or not. In this
> > case, what are the performance issues we face in the Cluster?
> >
> > If the rowkey (belongs to single columnfamily) has different Column
> > qualifier also, the Hfile will not split?
> >
> >
> >
> > On Thu, 20 Jun 2019 at 11:38, wangyongqiang0...@163.com <
> > wangyongqiang0...@163.com> wrote:
> >
> > > this conf:
> > >   
> > > hbase.hregion.max.filesize
> > > 10737418240
> > > 
> > > Maximum HStoreFile size. If any one of a column families'
> HStoreFiles
> > > has
> > > grown to exceed this value, the hosting HRegion is split in
> > > two.
> > >   
> > >
> > >
> > >
> > >
> > >
> > > wangyongqiang0...@163.com
> > >
> > > From: Jean-Marc Spaggiari
> > > Date: 2019-06-19 06:52
> > > To: user
> > > Subject: Re: question on hfile size upper limit
> > > Hi,
> > >
> > > Can you please confirm which parameter you are talking about? The
> default
> > > HBase setting is to limit the size per region (10GB by default), and
> not
> > by
> > > HFiles. This can be configured at the HBase lever, or at the table
> level.
> > >
> > > HTH,
> > >
> > > JMS
> > >
> > > Le mar. 18 juin 2019 à 11:32, wangyongqiang0...@163.com <
> > > wangyongqiang0...@163.com> a écrit :
> > >
> > > > we set size upper limit for hfile, but not region
> > > > so region has different actural size, leading to some analysis task
> has
> > > > different input size
> > > >
> > > > can we set size limit on region
> > > >
> > > >
> > > >
> > > >
> > > > wangyongqiang0...@163.com
> > > >
> > >
> >
>

Re: Debugging High I/O Wait

[Anoop John]

Hi Srinidhi
You have File mode bucket cache.  What is the size of the
cache? You configure single file path for cache or 1+ paths?   If former,
splitting the cache into multiple files (paths can be given as comma
separated in the config) will help?

Anoop

On Fri, Apr 5, 2019 at 2:58 AM Srinidhi Muppalla 
wrote:

> After some more digging, I discovered that during the time that the RS is
> stuck the kernel message buffer outputted only this message
>
> "[1031214.108110] XFS: java(6522) possible memory allocation deadlock size
> 32944 in kmem_alloc (mode:0x2400240)"
>
> From my reading online, the cause of this error appears to generally be
> excessive memory and file fragmentation. We haven't changed the mslab
> config and we are running HBase 1.3.0 so it should be running by default.
> The issue tends to arise consistently and regularly (every 10 or so days)
> and once one node is affected other nodes start to follow after a few
> hours. What could be causing this to happen and is there any way to prevent
> or minimize fragmentation?
>
> Best,
> Srinidhi
>
> On 3/29/19, 11:02 AM, "Srinidhi Muppalla"  wrote:
>
> Stack and Ram,
>
> Attached the thread dumps. 'Jstack normal' is the normal node. 'Jstack
> problematic' was taken when the node was stuck.
>
> We don't have full I/O stats for the problematic node. Unfortunately,
> it was impacting production so we had to recreate the cluster as soon as
> possible and couldn't get full data. I attached the dashboards with the
> wait I/O and other CPU stats. Thanks for helping look into the issue!
>
> Best,
> Srinidhi
>
>
>
> On 3/28/19, 2:41 PM, "Stack"  wrote:
>
> Mind putting up a thread dump?
>
> How many spindles?
>
> If you compare the i/o stats between a good RS and a stuck one,
> how do they
> compare?
>
> Thanks,
> S
>
>
> On Wed, Mar 27, 2019 at 11:57 AM Srinidhi Muppalla <
> srinid...@trulia.com>
> wrote:
>
> > Hello,
> >
> > We've noticed an issue in our HBase cluster where one of the
> > region-servers has a spike in I/O wait associated with a spike
> in Load for
> > that node. As a result, our request times to the cluster increase
> > dramatically. Initially, we suspected that we were experiencing
> > hotspotting, but even after temporarily blocking requests to the
> highest
> > volume regions on that region-servers the issue persisted.
> Moreover, when
> > looking at request counts to the regions on the region-server
> from the
> > HBase UI, they were not particularly high and our own
> application level
> > metrics on the requests we were making were not very high
> either. From
> > looking at a thread dump of the region-server, it appears that
> our get and
> > scan requests are getting stuck when trying to read from the
> blocks in our
> > bucket cache leaving the threads in a 'runnable' state. For
> context, we are
> > running HBase 1.30 on a cluster backed by S3 running on EMR and
> our bucket
> > cache is running in File mode. Our region-servers all have SSDs.
> We have a
> > combined cache with the L1 standard LRU cache and the L2 file
> mode bucket
> > cache. Our Bucket Cache utilization is less than 50% of the
> allocated space.
> >
> > We suspect that part of the issue is our disk space utilization
> on the
> > region-server as our max disk space utilization also increased
> as this
> > happened. What things can we do to minimize disk space
> utilization? The
> > actual HFiles are on S3 -- only the cache, application logs, and
> write
> > ahead logs are on the region-servers. Other than the disk space
> > utilization, what factors could cause high I/O wait in HBase and
> is there
> > anything we can do to minimize it?
> >
> > Right now, the only thing that works is terminating and
> recreating the
> > cluster (which we can do safely because it's S3 backed).
> >
> > Thanks!
> > Srinidhi
> >
>
>
>
>
>

Re: how do disable encryption on a CF

[Anoop John]

And once you successfully reset the encryption for ur table:cf,  do run a
major compaction for the table:cf so that encrypted data will get rewritten
and this with no encryption.  Pls remember that major compaction will have
lots of IO and putting load on your RS too..  It depends on total  data
size getting compacted.. Pls keep that in mind and schedule

Anoop

On Thu, Jan 24, 2019 at 9:49 PM marjana  wrote:

> Thanks for the info!
>
>
>
> --
> Sent from:
> http://apache-hbase.679495.n3.nabble.com/HBase-User-f4020416.html
>

Re: [ANNOUNCE] New HBase committer Jingyun Tian

[Anoop John]

Congrats Jingyun !!!

-Anoop-

On Wed, Nov 14, 2018 at 8:18 AM Jingyun Tian  wrote:

> Thank you all!
>
> Sincerely,
> Jingyun Tian
>
> On Wed, Nov 14, 2018 at 8:59 AM stack  wrote:
>
> > Welcome jingyun.
> > S
> >
> > On Mon, Nov 12, 2018, 11:54 PM 张铎(Duo Zhang)  wrote:
> >
> > > On behalf of the Apache HBase PMC, I am pleased to announce that
> Jingyun
> > > Tian has accepted the PMC's invitation to become a committer on the
> > > project. We appreciate all of Jingyun's generous contributions thus far
> > and
> > > look forward to his continued involvement.
> > >
> > > Congratulations and welcome, Jingyun!
> > >
> >
>

Re: can we set a table to use a HDFS specific HSM Storage policy?

[Anoop John]

Oh ya seems yes..  I was under the impression that it is there in 1.x.
Some HBase users use it. Might be, they back ported it.

-Anoop-

On Tue, Apr 17, 2018 at 5:38 PM, Ming <ovis_p...@sina.com> wrote:
>
> Hi, Anoop,
>
> In which release this API is supported? From the JIRA 
> https://issues.apache.org/jira/browse/HBASE-14061, it seems this is only 
> available  in HBase 2.0?
>
> Thanks,
> Ming
>
> -----Original Message-
> From: Anoop John <anoop.hb...@gmail.com>
> Sent: Tuesday, April 17, 2018 1:42 PM
> To: user@hbase.apache.org
> Subject: Re: can we set a table to use a HDFS specific HSM Storage policy?
>
> When creating table, user can set storage policy on column family.
> Each of the CF can have different policy.  Pls see
> setStoragePolicy(String) API in HColumnDescriptor.
>
> -Anoop-
>
> On Tue, Apr 17, 2018 at 7:16 AM, Ming <ovis_p...@sina.com> wrote:
>> Hi, all,
>>
>>
>>
>> HDFS support HSM, one can set a file or dir storage policy to use different
>> hardware disks. I am wondering is there a way in HBase, when create a table,
>> specify the policy that that table uses?
>>
>> So that table can be put into SSD, or HDD defined by the storage policy.
>>
>> I read the HBase manual, but not find related topics, so asked help here.
>>
>>
>>
>> Thanks,
>>
>> Ming
>>
>>
>>
>
>

Re: can we set a table to use a HDFS specific HSM Storage policy?

[Anoop John]

When creating table, user can set storage policy on column family.
Each of the CF can have different policy.  Pls see
setStoragePolicy(String) API in HColumnDescriptor.

-Anoop-

On Tue, Apr 17, 2018 at 7:16 AM, Ming  wrote:
> Hi, all,
>
>
>
> HDFS support HSM, one can set a file or dir storage policy to use different
> hardware disks. I am wondering is there a way in HBase, when create a table,
> specify the policy that that table uses?
>
> So that table can be put into SSD, or HDD defined by the storage policy.
>
> I read the HBase manual, but not find related topics, so asked help here.
>
>
>
> Thanks,
>
> Ming
>
>
>

Re: Cell-level security with Coprocessors

[Anoop John]

Hi Jim
Just taking your eg: Use cell level labels along with
AggregationClient - Tt will NOT work.  The reason is the Aggregation
impl for the server side will create scanner directly over the Region.
As you know, the cell level security features work with the help of a
co processor and it expects the CP to be informed while opening of
scanner.   But the CP contact while opening the scanner is one layer
above the Region level (ie at RSRpcServices). So when the
Aggregation service impl is opening the scanner, the visibility
related CP will not get into pic.If that was not the case, the
rest of the things would have been just fine for ur need.  Mind
opening an issue and we discuss under that for the possible fixes?

-Anoop-

On Fri, Mar 30, 2018 at 8:57 PM, Jim Hughes  wrote:
> Hi all,
>
> I'm interested in combining an HBase Coprocessor with cell level security.
> As a quick example, suppose one wanted to use the ColumnAggregation
> Processor with cells with different authorizations.  Different users should
> be able to scan and see the sum of the entries that they'd normally be able
> to scan.
>
> From the first few hours of looking, it appears that Coprocessors call out
> to the RegionServer's getScanner method which does not apply any security by
> default.
>
> Have others had success combining custom coprocessors with cell-level
> security?  That is, are there any pointers for getting started?
>
> Cheers,
>
> Jim
>

Re: ASYNC_WAL and visibility

[Anoop John]

>can it affect visibility in any way? Does it
still follow the principle "When a client receives a "success" response for
any mutation, that mutation is immediately visible to both that client and
any client with whom it later communicates through side channels" ?

No it wont affect the mentioned way of visibility to readers

>is data loss possible in any situation other than region
failure?

There MIGHT be data loss when the server crashes (Region closed in a
non graceful way)


-Anoop-


On Fri, Mar 23, 2018 at 2:31 PM, Viacheslav Krot
 wrote:
> Hi all,
> I cannot find any good information about ASYNC_WAL semantics.
> As much as I understand the worst I can get - data loss in case of region
> failure.
> But the question I have - can it affect visibility in any way? Does it
> still follow the principle "When a client receives a "success" response for
> any mutation, that mutation is immediately visible to both that client and
> any client with whom it later communicates through side channels" ?
> Another question - is data loss possible in any situation other than region
> failure?

Re: How Long Will HBase Hold A Row Write Lock?

[Anoop John]

Hi Saad
   In your initial mail you mentioned that there are lots
of checkAndPut ops but on different rows. The failure in obtaining
locks (write lock as it is checkAndPut) means there is contention on
the same row key.  If that is the case , ya that is the 1st step
before BC reads and it make sense.

On the Q on why not caching the compacted file content, yes it is this
way. Even if cache on write is true. This is because some times the
compacted result file could be so large (what is major compaction) and
that will exhaust the BC if written. Also it might contain some data
which are very old.  There is a jira recently raised jira which
discuss abt this.  Pls see HBASE-20045


-Anoop-

On Sun, Mar 11, 2018 at 7:57 AM, Saad Mufti <saad.mu...@gmail.com> wrote:
> Although now that I think about this a bit more, all the failures we saw
> were failure to obtain a row lock, and in the thread stack traces we always
> saw it somewhere inside getRowLockInternal and similar. Never saw any
> contention on bucket cache lock that I could see.
>
> Cheers.
>
> 
> Saad
>
>
> On Sat, Mar 10, 2018 at 8:04 PM, Saad Mufti <saad.mu...@gmail.com> wrote:
>
>> Also, for now we have mitigated this problem by using the new setting in
>> HBase 1.4.0 that prevents one slow region server from blocking all client
>> requests. Of course it causes some timeouts but our overall ecosystem
>> contains Kafka queues for retries, so we can live with that. From what I
>> can see, it looks like this setting also has the good effect of preventing
>> clients from hammering a region server that is slow because its IPC queues
>> are backed up, allowing it to recover faster.
>>
>> Does that make sense?
>>
>> Cheers.
>>
>> 
>> Saad
>>
>>
>> On Sat, Mar 10, 2018 at 7:04 PM, Saad Mufti <saad.mu...@gmail.com> wrote:
>>
>>> So if I understand correctly, we would mitigate the problem by not
>>> evicting blocks for archived files immediately? Wouldn't this potentially
>>> lead to problems later if the LRU algo chooses to evict blocks for active
>>> files and leave blocks for archived files in there?
>>>
>>> I would definitely love to test this!!! Unfortunately we are running on
>>> EMR and the details of how to patch HBase under EMR are not clear to me :-(
>>>
>>> What we would really love would be a setting for actually immediately
>>> caching blocks for a new compacted file. I have seen in the code that even
>>> is we have the cache on write setting set to true, it will refuse to cache
>>> blocks for a file that is a newly compacted one. In our case we have sized
>>> the bucket cache to be big enough to hold all our data, and really want to
>>> avoid having to go to S3 until the last possible moment. A config setting
>>> to test this would be great.
>>>
>>> But thanks everyone for your feedback. Any more would also be welcome on
>>> the idea to let a user cache all newly compacted files.
>>>
>>> 
>>> Saad
>>>
>>>
>>> On Wed, Mar 7, 2018 at 12:00 AM, Anoop John <anoop.hb...@gmail.com>
>>> wrote:
>>>
>>>> >>a) it was indeed one of the regions that was being compacted, major
>>>> compaction in one case, minor compaction in another, the issue started
>>>> just
>>>> after compaction completed blowing away bucket cached blocks for the
>>>> older
>>>> HFile's
>>>>
>>>> About this part.Ya after the compaction, there is a step where the
>>>> compacted away HFile's blocks getting removed from cache. This op takes a
>>>> write lock for this region (In Bucket Cache layer)..  Every read op which
>>>> is part of checkAndPut will try read from BC and that in turn need a read
>>>> lock for this region.  So there is chances that the read locks starve
>>>> because of so many frequent write locks .  Each block evict will attain
>>>> the
>>>> write lock one after other.  Will it be possible for you to patch this
>>>> evict and test once? We can avoid the immediate evict from BC after
>>>> compaction. I can help you with a patch if you wish
>>>>
>>>> Anoop
>>>>
>>>>
>>>>
>>>> On Mon, Mar 5, 2018 at 11:07 AM, ramkrishna vasudevan <
>>>> ramkrishna.s.vasude...@gmail.com> wrote:
>>>> > Hi Saad
>>>> >
>>>> > Your argument here
>>>> >>> The
>>>> >>>theory is that since pref

How Long Will HBase Hold A Row Write Lock?

[Anoop John]

>>a) it was indeed one of the regions that was being compacted, major
compaction in one case, minor compaction in another, the issue started just
after compaction completed blowing away bucket cached blocks for the older
HFile's

About this part.Ya after the compaction, there is a step where the
compacted away HFile's blocks getting removed from cache. This op takes a
write lock for this region (In Bucket Cache layer)..  Every read op which
is part of checkAndPut will try read from BC and that in turn need a read
lock for this region.  So there is chances that the read locks starve
because of so many frequent write locks .  Each block evict will attain the
write lock one after other.  Will it be possible for you to patch this
evict and test once? We can avoid the immediate evict from BC after
compaction. I can help you with a patch if you wish

Anoop



On Mon, Mar 5, 2018 at 11:07 AM, ramkrishna vasudevan <
ramkrishna.s.vasude...@gmail.com> wrote:
> Hi Saad
>
> Your argument here
>>> The
>>>theory is that since prefetch is an async operation, a lot of the reads
in
>>>the checkAndPut for the region in question start reading from S3 which is
>>>slow. So the write lock obtained for the checkAndPut is held for a longer
>>>duration than normal. This has cascading upstream effects. Does that
sound
>>>plausible?
>
> Seems very much plausible. So before even the prefetch happens say for
> 'block 1' - and you have already issues N checkAndPut calls for the rows
in
> that 'block 1' -  all those checkAndPut will have to read that block from
> S3 to perform the get() and then apply the mutation.
>
> This may happen for multiple threads at the same time because we are not
> sure when the prefetch would have actually been completed. I don know what
> are the general read characteristics when a read happens from S3 but you
> could try to see how things work when a read happens from S3 and after the
> prefetch completes ensure the same checkandPut() is done (from cache this
> time) to really know the difference what S3 does there.
>
> Regards
> Ram
>
> On Fri, Mar 2, 2018 at 2:57 AM, Saad Mufti  wrote:
>
>> So after much investigation I can confirm:
>>
>> a) it was indeed one of the regions that was being compacted, major
>> compaction in one case, minor compaction in another, the issue started
just
>> after compaction completed blowing away bucket cached blocks for the
older
>> HFile's
>> b) in another case there was no compaction just a newly opened region in
a
>> region server that hadn't finished perfetching its pages from S3
>>
>> We have prefetch on open set to true. Our load is heavy on checkAndPut
.The
>> theory is that since prefetch is an async operation, a lot of the reads
in
>> the checkAndPut for the region in question start reading from S3 which is
>> slow. So the write lock obtained for the checkAndPut is held for a longer
>> duration than normal. This has cascading upstream effects. Does that
sound
>> plausible?
>>
>> The part I don't understand still is all the locks held are for the same
>> region but are all for different rows. So once the prefetch is completed,
>> shouldn't the problem clear up quickly? Or does the slow region slow down
>> anyone trying to do checkAndPut on any row in the same region even after
>> the prefetch has completed. That is, do the long held row locks prevent
>> others from getting a row lock on a different row in the same region?
>>
>> In any case, we trying to use
>> https://issues.apache.org/jira/browse/HBASE-16388 support in HBase 1.4.0
>> to
>> both insulate the app a bit from this situation and hoping that it will
>> reduce pressure on the region server in question, allowing it to recover
>> faster. I haven't quite tested that yet, any advice in the meantime would
>> be appreciated.
>>
>> Cheers.
>>
>> 
>> Saad
>>
>>
>>
>> On Thu, Mar 1, 2018 at 9:21 AM, Saad Mufti  wrote:
>>
>> > Actually it happened again while some minior compactions were running,
so
>> > don't think it related to our major compaction tool, which isn't even
>> > running right now. I will try to capture a debug dump of threads and
>> > everything while the event is ongoing. Seems to last at least half an
>> hour
>> > or so and sometimes longer.
>> >
>> > 
>> > Saad
>> >
>> >
>> > On Thu, Mar 1, 2018 at 7:54 AM, Saad Mufti 
wrote:
>> >
>> >> Unfortunately I lost the stack trace overnight. But it does seem
related
>> >> to compaction, because now that the compaction tool is done, I don't
see
>> >> the issue anymore. I will run our incremental major compaction tool
>> again
>> >> and see if I can reproduce the issue.
>> >>
>> >> On the plus side the system stayed stable and eventually recovered,
>> >> although it did suffer all those timeouts.
>> >>
>> >> 
>> >> Saad
>> >>
>> >>
>> >> On Wed, Feb 28, 2018 at 10:18 PM, Saad Mufti 
>> >> wrote:
>> >>
>> >>> I'll paste a thread dump later, writing this 

Re: How to get bulk delete [Hbase1.2.0]

[Anoop John]

BulkDeleteEP allows to pass a Scan .  What all rows this Scan returns
all can be deleted.  So you can create an appropriate Scan to fetch
all rows with a common prefix.   Pls try once.

-Anoop-

On Sat, Mar 3, 2018 at 2:49 AM, Ted Yu  wrote:
> For #2, BulkDeleteEndpoint still exists - in hbase-examples branch.
>
> ./hbase-examples/src/main/java/org/apache/hadoop/hbase/coprocessor/example/BulkDeleteEndpoint.java
>
> On Thu, Mar 1, 2018 at 10:02 PM, Vikash Agarwal 
> wrote:
>
>> Hi Team,
>>
>> I am looking for  a bulk delete in Hbase.
>>
>> I am looking here for two things:
>>
>> 1. In case I provide a Row key Prefix , Is it possible to delete all the
>> rows with that prefix.
>>
>> 2. In case Where we can scan and delete and the server End. Something like
>> BulkDeleteEndpoint which uses to exist in old hbase version 0.94.
>>
>>
>>
>> --
>> Vikash Agarwal
>>

Re: New installation of HBase ignoring cell visibility (even though it's active)

[Anoop John]

As which user you are running the scan shell command? It has to be as
michaelthomasen as u have set auths to this uname. Looks like the user
running the command is the hbase super user who started the RS process.
Then all cells will be returned irrespective of its visibility and scan
auths.

Anoop

On Saturday, February 24, 2018, Mike Thomsen  wrote:

> (hbase-site.xml is attached)
>
> I reinstalled HBase on my development machine and the console shows that
> it's just ignoring the cell visibility settings even though it shows
> they're active:
>
> hbase(main):001:0> list
> TABLE
>
>
>
> 0 row(s) in 0.1630 seconds
>
> => []
> hbase(main):002:0> get_auths "michaelthomsen"
> 0 row(s) in 10.1940 seconds
>
> hbase(main):003:0> add_labels ["U", "PII", "PHI"]
> 0 row(s) in 0.1900 seconds
>
> hbase(main):004:0> set_auths "michaelthomsen",  ["U", "PII", "PHI"]
> 0 row(s) in 0.0500 seconds
>
> hbase(main):005:0> get_auths "michaelthomsen"
> U
>
>
>
> PII
>
>
>
> PHI
>
>
>
> 0 row(s) in 0.0470 seconds
>
> hbase(main):006:0> create 'test', {NAME => 'prop' }
> 0 row(s) in 1.2780 seconds
>
> => Hbase::Table - test
> hbase(main):007:0> put "test", "user1", "prop:name", "John Smith"
> 0 row(s) in 0.0500 seconds
>
> hbase(main):008:0> scan "test"
> ROW
> COLUMN+CELL
>
>
>  user1 column=prop:name,
> timestamp=1519476818510, value=John Smith
>
> 1 row(s) in 0.0260 seconds
>
> hbase(main):009:0> set_visibility "test", "PII", { COLUMNS => "prop" }
> 1 row(s) in 0.0130 seconds
>
> hbase(main):010:0> scan "test", { AUTHORIZATIONS => ["PHI", "U"] }
> ROW
> COLUMN+CELL
>
>
>  user1 column=prop:name,
> timestamp=1519476818510, value=John Smith
>
> 1 row(s) in 0.0180 seconds
>
> hbase(main):011:0> list_security_capabilities
> SIMPLE_AUTHENTICATION
> CELL_VISIBILITY
>
> => ["SIMPLE_AUTHENTICATION", "CELL_VISIBILITY"]
> hbase(main):012:0> scan "test", { AUTHORIZATIONS => [] }
> ROW
> COLUMN+CELL
>
>
>  user1 column=prop:name,
> timestamp=1519476818510, value=John Smith
>
> 1 row(s) in 0.0060 seconds
>
>
> I'm running this on a Mac w/out HDFS. It's HBase 1.3.1. This makes no
> sense because it's allowing me to assign authorizations to my simple auth
> user, set_visibility runs successfully and it's reporting that cell
> visibility is enabled.
>
>

Re: [ANNOUNCE] New HBase committer Peter Somogyi

[Anoop John]

Congrats Peter..

Anoop


On Friday, February 23, 2018, ramkrishna vasudevan <
ramkrishna.s.vasude...@gmail.com> wrote:

> Congratulations Peter !!!
>
> On Fri, Feb 23, 2018 at 3:40 PM, Peter Somogyi 
> wrote:
>
> > Thank you very much everyone!
> >
> > On Thu, Feb 22, 2018 at 8:08 PM, Sean Busbey  wrote:
> >
> > > On behalf of the Apache HBase PMC, I am pleased to announce that Peter
> > > Somogyi has accepted the PMC's invitation to become a committer on the
> > > project.
> > >
> > > We appreciate all of Peter's great work thus far and look forward to
> > > continued involvement.
> > >
> > > Please join me in congratulating Peter!
> > >
> >
>

Re: Trying To Understand BucketCache Evictions In HBase 1.3.1

[Anoop John]

Hi
  Seems you have write ops happening as you mentioned abt
minor compactions.  When the compaction happens, the compacted file's
blocks will get evicted.  Whatever be the value of
'hbase.rs.evictblocksonclose'.  This config comes to play when the
Store is closed. Means the region movement is happening or split and
so a close on stores. Also the table might get disabled or deleted.
All such store close cases this config comes to picture.  But minor
compactions means there will be evictions.  These are not via the
eviction threads which monitor the less spaces and select LRU blocks
for eviction.  This is done by the compaction threads. That is why you
can see the evict ops (done by Eviction thread) is zero but the
#evicted blocks are there.  Those might be the blocks of the compacted
away files.  Hope this helps you to understand what is going on.

-Anoop-


On Mon, Feb 19, 2018 at 5:25 AM, Saad Mufti  wrote:
> Sorry I meant BLOCKCACHE => 'false' on the one column family we don't want
> getting cached.
>
> Cheers.
>
> 
> Saad
>
>
> On Sun, Feb 18, 2018 at 6:51 PM, Saad Mufti  wrote:
>
>> Hi,
>>
>> We have an HBase system running HBase 1.3.1 on an AWS EMR service. Our
>> BucketCache is configured for 400 GB on a set of attached EBS disk volumes,
>> with all column families marked for in-memory in their column family
>> schemas using INMEMORY => 'true' (except for one column family we only ever
>> write to, so we set BUCKETCACHE => 'false' on that one).
>>
>> Even though all column families are marked INMEMORY, we have the following
>> ratios set:
>>
>> "hbase.bucketcache.memory.factor":"0.8",
>>
>> "hbase.bucketcache.single.factor":"0.1",
>>
>>
>> "hbase.bucketcache.multi.factor":"0.1",
>>
>> Currently the bucket cache shows evictions even though it has tons of free
>> space. I am trying to understand why we get any evictions at all? We do
>> have minor compactions going on, but we have not set 
>> hbase.rs.evictblocksonclose
>> to any value and from looking at the code, it defaults to false. The total
>> bucket cache size is nowhere near any of the above limits, in fact on some
>> long running servers where we stopped traffic, the cache size went down to
>> 0. Which makes me think something is evicting blocks from the bucket cache
>> in the background.
>>
>> You can see a screenshot from one of the regionserver L2 stats UI pages at
>> https://imgur.com/a/2ZUSv . Another interesting thing to me on this page
>> is that it has non-zero evicted blocks but says Evictions: 0
>>
>> Any help understanding this would be appreciated.
>>
>> 
>> Saad
>>
>>

Re: WrongRegionException

[Anoop John]

Or else you should do below in prePut
Create the new Put and get the COnnection from the CP environment and
do a put op on the connection. (Not directly on Region)  And then call
bypass() to bypass this Put op with old RK.  Just suggesting. But I
some how feel that why not a middle client layer can do this convert

-Anoop-

On Thu, Feb 1, 2018 at 1:07 PM, Anoop John <anoop.hb...@gmail.com> wrote:
> Theoretically I dont think what you are trying to do is correct. I
> mean the incoming keys are fully ignored and new one is been made at
> CP layers. When CP is been contacted already the mutations has reached
> upto that Region.  The new RK might be outside the boundary of this
> region. (As the split happens).   Dont know your use case. But I feel
> like you should handle this conversion of RKs and create new Put in a
> client tier not at CP.
>
> -Anoop-
>
> On Mon, Jan 29, 2018 at 9:44 PM, Ted Yu <yuzhih...@gmail.com> wrote:
>> w.r.t. region split, do you verify that the new rowkey is in the same
>> region as the rowkey from incoming Put ?
>>
>> If not, there is a chance that the new rowkey is in different region which
>> is going thru split.
>>
>> FYI
>>
>> On Mon, Jan 29, 2018 at 6:40 AM, Yang Zhang <zhang.yang...@gmail.com> wrote:
>>
>>> Both are the same question.
>>> I want to prevent the incoming puts and then copy it as one or more new
>>> puts with different rowkey.
>>> So when there is only one region, my rowkey will belong to it. But when
>>> region splited, some rowkeys may not belong to the new region.
>>> I used to thought HBase will stop new coming puts, finish all of the puts
>>> in the batch, and then try to split.
>>> But this maybe not right according to the exception that I got.
>>>
>>> BTY , It seems that I can't add put
>>> to MiniBatchOperationInProgress miniBatchOp. There are only some
>>> functions for get.
>>>
>>> Thank you very much for your help
>>>
>>> 2018-01-29 18:46 GMT+08:00 Anoop John <anoop.hb...@gmail.com>:
>>>
>>> > Another related Q was also there..  Can you tell the actual
>>> > requirement?  So the incoming puts you want to change the RKs of that?
>>> > Or you want to insert those as well as some new cells with a changed
>>> > RK?
>>> >
>>> > -Anoop-
>>> >
>>> > On Mon, Jan 29, 2018 at 3:49 PM, Yang Zhang <zhang.yang...@gmail.com>
>>> > wrote:
>>> > > Hello Everyone
>>> > >
>>> > > I am using coprocesser to prevent the normal put and replace it
>>> > > with another rowkey, The method is HRegion.put(). It works fine, but
>>> when
>>> > > the region splited, There will be an WrongRegionException.
>>> > >
>>> > > 2018-01-28 09:32:51,528 WARN
>>> > > [B.DefaultRpcServer.handler=21,queue=3,port=60020]
>>> regionserver.HRegion:
>>> > > Failed getting lock in batch put,
>>> > > row=\xF0\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x0F\x10\xC5r
>>> > > org.apache.hadoop.hbase.regionserver.WrongRegionException: Requested
>>> row
>>> > > out of range for row lock on HRegion
>>> > > GISdoop_GeoKey,,1517085124215.341534e84727245f1c67f345c3e467ac.,
>>> > > startKey='', getEndKey()='\xE6G8\x00\x00\x00\x00\x00\x00\x00\x00\x00',
>>> > > row='\xF0\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x0F\x10\xC5r'
>>> > > at org.apache.hadoop.hbase.regionserver.HRegion.checkRow(
>>> > HRegion.java:4677)
>>> > > at
>>> > > org.apache.hadoop.hbase.regionserver.HRegion.
>>> > getRowLock(HRegion.java:4695)
>>> > > at
>>> > > org.apache.hadoop.hbase.regionserver.HRegion.
>>> > doMiniBatchMutation(HRegion.java:2786)
>>> > > at
>>> > > org.apache.hadoop.hbase.regionserver.HRegion.
>>> > batchMutate(HRegion.java:2653)
>>> > > at
>>> > > org.apache.hadoop.hbase.regionserver.HRegion.
>>> > batchMutate(HRegion.java:2589)
>>> > > at
>>> > > org.apache.hadoop.hbase.regionserver.HRegion.
>>> doBatchMutate(HRegion.java:
>>> > 3192)
>>> > > at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:2459)
>>> > > at site.luoyu.Core.Index.JavaTreeMap.insertPoint(JavaTreeMap.java:287)
>>> > > at site.luoyu.Core.Index.JavaTreeMap.insertRecord(
>>> JavaTreeMap.java:256)
>>

Re: WrongRegionException

[Anoop John]

Theoretically I dont think what you are trying to do is correct. I
mean the incoming keys are fully ignored and new one is been made at
CP layers. When CP is been contacted already the mutations has reached
upto that Region.  The new RK might be outside the boundary of this
region. (As the split happens).   Dont know your use case. But I feel
like you should handle this conversion of RKs and create new Put in a
client tier not at CP.

-Anoop-

On Mon, Jan 29, 2018 at 9:44 PM, Ted Yu <yuzhih...@gmail.com> wrote:
> w.r.t. region split, do you verify that the new rowkey is in the same
> region as the rowkey from incoming Put ?
>
> If not, there is a chance that the new rowkey is in different region which
> is going thru split.
>
> FYI
>
> On Mon, Jan 29, 2018 at 6:40 AM, Yang Zhang <zhang.yang...@gmail.com> wrote:
>
>> Both are the same question.
>> I want to prevent the incoming puts and then copy it as one or more new
>> puts with different rowkey.
>> So when there is only one region, my rowkey will belong to it. But when
>> region splited, some rowkeys may not belong to the new region.
>> I used to thought HBase will stop new coming puts, finish all of the puts
>> in the batch, and then try to split.
>> But this maybe not right according to the exception that I got.
>>
>> BTY , It seems that I can't add put
>> to MiniBatchOperationInProgress miniBatchOp. There are only some
>> functions for get.
>>
>> Thank you very much for your help
>>
>> 2018-01-29 18:46 GMT+08:00 Anoop John <anoop.hb...@gmail.com>:
>>
>> > Another related Q was also there..  Can you tell the actual
>> > requirement?  So the incoming puts you want to change the RKs of that?
>> > Or you want to insert those as well as some new cells with a changed
>> > RK?
>> >
>> > -Anoop-
>> >
>> > On Mon, Jan 29, 2018 at 3:49 PM, Yang Zhang <zhang.yang...@gmail.com>
>> > wrote:
>> > > Hello Everyone
>> > >
>> > > I am using coprocesser to prevent the normal put and replace it
>> > > with another rowkey, The method is HRegion.put(). It works fine, but
>> when
>> > > the region splited, There will be an WrongRegionException.
>> > >
>> > > 2018-01-28 09:32:51,528 WARN
>> > > [B.DefaultRpcServer.handler=21,queue=3,port=60020]
>> regionserver.HRegion:
>> > > Failed getting lock in batch put,
>> > > row=\xF0\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x0F\x10\xC5r
>> > > org.apache.hadoop.hbase.regionserver.WrongRegionException: Requested
>> row
>> > > out of range for row lock on HRegion
>> > > GISdoop_GeoKey,,1517085124215.341534e84727245f1c67f345c3e467ac.,
>> > > startKey='', getEndKey()='\xE6G8\x00\x00\x00\x00\x00\x00\x00\x00\x00',
>> > > row='\xF0\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x0F\x10\xC5r'
>> > > at org.apache.hadoop.hbase.regionserver.HRegion.checkRow(
>> > HRegion.java:4677)
>> > > at
>> > > org.apache.hadoop.hbase.regionserver.HRegion.
>> > getRowLock(HRegion.java:4695)
>> > > at
>> > > org.apache.hadoop.hbase.regionserver.HRegion.
>> > doMiniBatchMutation(HRegion.java:2786)
>> > > at
>> > > org.apache.hadoop.hbase.regionserver.HRegion.
>> > batchMutate(HRegion.java:2653)
>> > > at
>> > > org.apache.hadoop.hbase.regionserver.HRegion.
>> > batchMutate(HRegion.java:2589)
>> > > at
>> > > org.apache.hadoop.hbase.regionserver.HRegion.
>> doBatchMutate(HRegion.java:
>> > 3192)
>> > > at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:2459)
>> > > at site.luoyu.Core.Index.JavaTreeMap.insertPoint(JavaTreeMap.java:287)
>> > > at site.luoyu.Core.Index.JavaTreeMap.insertRecord(
>> JavaTreeMap.java:256)
>> > > at site.luoyu.Core.Observer.IndexCopressor.prePut(
>> > IndexCopressor.java:130)
>> > > at
>> > > org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.prePut(
>> > RegionCoprocessorHost.java:1122)
>> > > at
>> > > org.apache.hadoop.hbase.regionserver.HRegion.
>> doPreMutationHook(HRegion.
>> > java:2674)
>> > > at
>> > > org.apache.hadoop.hbase.regionserver.HRegion.
>> > batchMutate(HRegion.java:2649)
>> > > at
>> > > org.apache.hadoop.hbase.regionserver.HRegion.
>> > batchMutate(HRegion.java:2589)
>> > > at
>> > > org.apache.hadoop.hbase.regionser

Re: WrongRegionException

[Anoop John]

Another related Q was also there..  Can you tell the actual
requirement?  So the incoming puts you want to change the RKs of that?
Or you want to insert those as well as some new cells with a changed
RK?

-Anoop-

On Mon, Jan 29, 2018 at 3:49 PM, Yang Zhang  wrote:
> Hello Everyone
>
> I am using coprocesser to prevent the normal put and replace it
> with another rowkey, The method is HRegion.put(). It works fine, but when
> the region splited, There will be an WrongRegionException.
>
> 2018-01-28 09:32:51,528 WARN
> [B.DefaultRpcServer.handler=21,queue=3,port=60020] regionserver.HRegion:
> Failed getting lock in batch put,
> row=\xF0\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x0F\x10\xC5r
> org.apache.hadoop.hbase.regionserver.WrongRegionException: Requested row
> out of range for row lock on HRegion
> GISdoop_GeoKey,,1517085124215.341534e84727245f1c67f345c3e467ac.,
> startKey='', getEndKey()='\xE6G8\x00\x00\x00\x00\x00\x00\x00\x00\x00',
> row='\xF0\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x0F\x10\xC5r'
> at org.apache.hadoop.hbase.regionserver.HRegion.checkRow(HRegion.java:4677)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getRowLock(HRegion.java:4695)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:2786)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2653)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2589)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.doBatchMutate(HRegion.java:3192)
> at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:2459)
> at site.luoyu.Core.Index.JavaTreeMap.insertPoint(JavaTreeMap.java:287)
> at site.luoyu.Core.Index.JavaTreeMap.insertRecord(JavaTreeMap.java:256)
> at site.luoyu.Core.Observer.IndexCopressor.prePut(IndexCopressor.java:130)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.prePut(RegionCoprocessorHost.java:1122)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.doPreMutationHook(HRegion.java:2674)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2649)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2589)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2593)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.doBatchOp(HRegionServer.java:4402)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.doNonAtomicRegionMutation(HRegionServer.java:3584)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3474)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:3)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2078)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:108)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94)
> at java.lang.Thread.run(Thread.java:745)
>
> It is said rowkey is out of region's bound.  This Exception is logged in
> regionserver's log as an warning , I can't catch  and handle it.
>
> According the source code,
> RowLock rowLock = null;
> try {
>   rowLock = getRowLock(mutation.getRow(), shouldBlock);
> } catch (IOException ioe) {
>   LOG.warn("Failed getting lock in batch put, row="
> + Bytes.toStringBinary(mutation.getRow()), ioe);
> }
>
> HBase just cache and log this exception , I guess it even didn't remove it
> from the batch. So I got so many Exception log  and can't put data anymore.
>
> Why HBase handle this WrongRegionException like this? Anyone can help?
> Thanks verymuch.

Re: Coprocessor prePut

[Anoop John]

What fix u tried now will help you.  You can not avoid the loop by
using the complete or bypass way.. Because that is on the present
context.  Another put on region will create a new context and so that
continues.   One more thing/suggestion would be to see the hook
preBatchMutate.  You will receive the Mutations being processed in a
MiniBatchOperationInProgress object to which u can add more Mutations
(Puts) as CP added.  These also will get executed.  As you are writing
to the same Region, this will work.

-Anoop-

On Tue, Jan 23, 2018 at 11:25 AM, Ted Yu  wrote:
> bq. can call context.complete() to skip other coprocessors.
>
> FYI in hbase-2, complete() has been folded into bypass().
> See javadoc of bypass() for details.
>
> On Mon, Jan 22, 2018 at 9:34 PM, Yang Zhang  wrote:
>
>> In fact, I prevent normal writes, and put the same data with a different
>> row key(Except row key, they have the same columns and same data).
>> I have already solved the infinite call by add and check some flag
>> attribute to Put. If the flag exist then skip my code.
>>
>> This will prevent the infinite call of prePut(). But I want to know If
>> there are any method support by coprocessor context to do that.
>> Just like that you can call context.complete() to skip other coprocessors.
>>
>> Thanks for your advice
>>
>> 2018-01-23 13:01 GMT+08:00 Ted Yu :
>>
>> > Your prePut would write to a different column in the table, right ?
>> > Otherwise do you prevent normal writes from getting into the same
>> column(s)
>> > ?
>> >
>> > If your prePut writes to dedicated column(s), you can check the presence
>> of
>> > such column(s) so that the call is not infinite.
>> >
>> > Cheers
>> >
>> > On Mon, Jan 22, 2018 at 8:56 PM, Yang Zhang 
>> > wrote:
>> >
>> > > Yes, It is the same table.
>> > >
>> > > 2018-01-23 1:46 GMT+08:00 Ted Yu :
>> > >
>> > > > Can you clarify your use case ?
>> > > >
>> > > > bq. put a data into table
>> > > >
>> > > > Does your coprocessor write to the same table which receives user
>> data
>> > ?
>> > > >
>> > > > Cheers
>> > > >
>> > > > On Mon, Jan 22, 2018 at 4:24 AM, Yang Zhang > >
>> > > > wrote:
>> > > >
>> > > > > Hello Everyone
>> > > > >
>> > > > > I am using the coprocessor and want to put another data
>> when
>> > > > > someone put a data into table. So I try prePut(), but when you call
>> > > > > HRegion.put(), this will call my coprocessor's prePut function
>> again,
>> > > and
>> > > > > it will cause an dead loop.
>> > > > >
>> > > > > My code looks like below, If anyone know how to make HBase
>> > > call
>> > > > my
>> > > > > coprocessor just for the first time?
>> > > > >
>> > > > > @Override
>> > > > > public void prePut(ObserverContext<
>> RegionCoprocessorEnvironment>
>> > > > > context, Put put, WALEdit edit,
>> > > > >Durability durability) throws IOException {
>> > > > > logger.info(" Coprocessor shouldComplete :
>> > > > > "+context.shouldComplete()+"");
>> > > > > context.bypass();
>> > > > >
>> > > > > Put myPut = new Put(new RowKey(1).toBytes());
>> > > > >
>> > > > > //this call will caused an dead loop
>> > > > > context.getEnvironment().getRegion().put(put);
>> > > > >
>> > > > > }
>> > > > >
>> > > > > Thanks anyway
>> > > > >
>> > > >
>> > >
>> >
>>

Re: Uninitialized Message Exception thrown while getting values.

[Anoop John]

Karthick
What is ur server and client versions?  U using Java
client or some thing else?

-Anoop-

On Thu, Jan 18, 2018 at 12:46 PM, ramkrishna vasudevan
 wrote:
> Hi
>
> Which version of HBase you get this problem? Do you have any pb classpath
> issues?
>
> Regards
> Ram
>
> On Thu, Jan 18, 2018 at 12:40 PM, Karthick Ram 
> wrote:
>
>> "UninitializedMessageException : Message missing required fields : region,
>> get", is thrown while performing Get. Due to this all the Get requests to
>> the same Region Server are getting stalled.
>>
>> com.google.protobuf.UninitializedMessageException: Message missing
>> required fields : region, get
>> at com.google.protobuf.AbstractMessage$Build.
>> newUninitializedMessageException(AbstractMessage.java:770)
>> at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$GetRequest$
>> Builder.build(ClientProtos.java:6377)
>> at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$GetRequest$
>> Builder.build(ClientProtos.java:6309)
>> at org.apache.hadoop.hbase.ipc.RpcServer$Connection.
>> processRequest(RpcServer.java:1840)
>> at org.apache.hadoop.hbase.ipc.RpcServer$Connection.
>> processOneRpc(RpcServer.java:1775)
>> at org.apache.hadoop.hbase.ipc.RpcServer$Connection.process(
>> RPcServer.java:1623)
>> at org.apache.hadoop.hbase.ipc.RpcServer$Connection.
>> readAndProcess(RpcServer.java:1603)
>> at org.apache.hadoop.hbase.ipc.RpcServer$Listener.doRead(
>> RpcServer.java:861)
>> at org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.
>> doRunLoop(RpcServer.java:643)
>> at org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(
>> RpcServer.java:619)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(
>> ThreadPoolExecutor.java:1145)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
>> ThreadPoolExecutor.java:615)
>> at java.lang.Thread.run(Thread.java:745)
>>

Re: hbase.bucketcache.bucket.sizes had set multiple of 1024 but still got "Invalid HFile block magic"

[Anoop John]

FYI

It has to be multiples of 256.


hbase.bucketcache.bucket.sizes

A comma-separated list of sizes for buckets for the
bucketcache.
Can be multiple sizes. List block sizes in order from smallest to largest.
The sizes you use will depend on your data access patterns.
Must be a multiple of 256 else you will run into
'java.io.IOException: Invalid HFile block magic' when you go to
read from cache.
If you specify no values here, then you pick up the default bucketsizes set
in code (See BucketAllocator#DEFAULT_BUCKET_SIZES).
  
  

-Anoop-

On Sat, Nov 25, 2017 at 7:53 PM, 苏国东  wrote:
> Hi：
>
> Class BucketEntry calculate offset like as :
> long offset()
> { // Java has no unsigned numbers long o = ((long) offsetBase) & 0x; 
> o += (((long) (offset1)) & 0xFF) << 32; return o << 8; }
> private void setOffset(long value)
> { assert (value & 0xFF) == 0; value >>= 8; offsetBase = (int) value; offset1 
> = (byte) (value >> 32); }
> If offset not multiple of 1024 , the method offset() will return wrong value. 
> if bucketsizes must be multiple of 1024, probably waste memory .
>
> 198211 is not multiple of 1024
>
>
> At 2017-11-25 20:52:10, "Weizhan Zeng"  wrote:
>>Hi , guys
>>   In https://issues.apache.org/jira/browse/HBASE-16993 
>>  , I found that
>>
>>hbase.bucketcache.bucket.sizes must set multiple of 1024, But when I set
>>
>>  
>>hbase.bucketcache.bucket.sizes
>>
>> 6144,9216,41984,50176,58368,66560,99328,132096,198211,263168,394240,525312,1049600,2099200
>>  
>>
>>And I still got  error :
>>
>>
>>2017-11-25 20:37:37,222 ERROR 
>>[B.defaultRpcServer.handler=20,queue=1,port=60020] bucket.BucketCache: Failed 
>>reading block d444ab4b244140c199f23a3870f59136_250591965 from bucket cache
>>java.io.IOException: Invalid HFile block magic: 
>>\x00\x00\x00\x00\x00\x00\x00\x00
>>   at org.apache.hadoop.hbase.io.hfile.BlockType.parse(BlockType.java:155)
>>   at org.apache.hadoop.hbase.io.hfile.BlockType.read(BlockType.java:167)
>>   at 
>> org.apache.hadoop.hbase.io.hfile.HFileBlock.(HFileBlock.java:275)
>>   at 
>> org.apache.hadoop.hbase.io.hfile.HFileBlock$1.deserialize(HFileBlock.java:136)
>>   at 
>> org.apache.hadoop.hbase.io.hfile.HFileBlock$1.deserialize(HFileBlock.java:123)
>>   at 
>> org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.getBlock(BucketCache.java:428)
>>   at 
>> org.apache.hadoop.hbase.io.hfile.CombinedBlockCache.getBlock(CombinedBlockCache.java:85)
>>   at 
>> org.apache.hadoop.hbase.io.hfile.HFileReaderV2.getCachedBlock(HFileReaderV2.java:278)
>>   at 
>> org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:418)
>>   at 
>> org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.loadDataBlockWithScanInfo(HFileBlockIndex.java:271)
>>   at 
>> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.seekTo(HFileReaderV2.java:649)
>>   at 
>> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.seekTo(HFileReaderV2.java:599)
>>   at 
>> org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:268)
>>   at 
>> org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:173)
>>   at 
>> org.apache.hadoop.hbase.regionserver.StoreScanner.seekScanners(StoreScanner.java:350)
>>   at 
>> org.apache.hadoop.hbase.regionserver.StoreScanner.(StoreScanner.java:199)
>>   at 
>> org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:2077)
>>   at 
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.(HRegion.java:5556)
>>   at 
>> org.apache.hadoop.hbase.regionserver.HRegion.instantiateRegionScanner(HRegion.java:2574)
>>   at 
>> org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2560)
>>   at 
>> org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2541)
>>   at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:6830)
>>   at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:6809)
>>   at 
>> org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2049)
>>   at 
>> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:33644)
>>   at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2196)
>>   at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)
>>   at 
>> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
>>   at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
>>   at java.lang.Thread.run(Thread.java:748)
>>
>>
>>Is there anything I missed ?

Re: [ANNOUNCE] New HBase committer Zheng Hu

[Anoop John]

Congrats and welcome Zheng...

-Anoop-

On Mon, Oct 23, 2017 at 11:51 AM, ramkrishna vasudevan
 wrote:
> Welcome Zheng and congratulations !!!
>
> On Mon, Oct 23, 2017 at 11:48 AM, Duo Zhang  wrote:
>
>> On behalf of the Apache HBase PMC, I am pleased to announce that Zheng Hu
>> has accepted the PMC's invitation to become a committer on the project. We
>> appreciate all of Zheng's generous contributions thus far and look forward
>> to his continued involvement.
>>
>> Congratulations and welcome, Zheng!
>>

Re: Welcome Chia-Ping Tsai to the HBase PMC

[Anoop John]

Congrats Chia-Ping  Welcome..

-Anoop-

On Sat, Sep 30, 2017 at 12:28 PM, Ashish Singhi  wrote:
> Congratulations, Chia-Ping.
>
> On Sat, Sep 30, 2017 at 3:49 AM, Misty Stanley-Jones 
> wrote:
>
>> The HBase PMC is delighted to announce that Chia-Ping Tsai has agreed to
>> join
>> the HBase PMC, and help to make the project run smoothly. Chia-Ping became
>> an
>> HBase committer over 6 months ago, based on long-running participate in the
>> HBase project, a consistent record of resolving HBase issues, and
>> contributions
>> to testing and performance.
>>
>> Thank you for stepping up to serve, Chia-Ping!
>>
>> As a reminder, if anyone would like to nominate another person as a
>> committer or PMC member, even if you are not currently a committer or PMC
>> member, you can always drop a note to priv...@hbase.apache.org to let us
>> know!
>>
>> Thanks,
>> Misty (on behalf of the HBase PMC)
>>

Re: Offheap config question for Hbase 1.1.2

[Anoop John]

+1 on what Ram said abt why HM also need this setting in ur version..
HM starts a RS internally..You can do one thing.. Use different
hbase config xml for HM process other than what used for RS.   May be
we need some fix also as Ram said..  Specially when no regions are
allowed in HM side.

In 0.98 also, u set this MaxDirectMemory config specifically?  Or
there this setting was not present at all in hbase-env.sh?  Because by
def there is no cap for the Java process for off heap memory usage.
You have to give some value bigger than the bucket cache size for this
MaxDirectMemory..Some other areas of code also uses the off heap
memory.. Like the RPC uses java nio socket communication which would
internally need some off heap memory to be allocated.. Just saying for
ur ref.

-Anoop-

On Tue, Sep 12, 2017 at 10:41 PM, ramkrishna vasudevan
 wrote:
> Why Master also accepts MaxDirectMemory config is probably because in 1.1
> onwards we treat HMaster as HregionServer but it does the region managment
> also. Am not very sure in 1.1.2 is HMaster allowed to host regions ? If so
> you need to configure MaxDirectMemory, if not probably we can see how we
> can avoid it. We need to raise a JIRA for that.
>
> Coming to the size of MaxDirectMemory less than bucket cache - I am
> wondering was there a bug previously? Because assume you need 25G offheap
> bucket cache then atleast 25G MaxDirectMemory is a must. Ideally you may
> need some delta more than 25G.
>
> 0.98 is obsolete now so its better we go with how 1.1.2 works. But if you
> feel there is a documentation that could help I think it is better we
> provide one so that users like you are not affected.
>
> Regards
> Ram
>
>
>
>
> On Tue, Sep 12, 2017 at 10:22 PM, Arul Ramachandran 
> wrote:
>
>> Thank you, Ram.
>>
>> >> So are you trying to use bucket cache feature in offheap mode with
>> 1.1.2?
>>
>> Yes.
>>
>> >> So even in 0.98 you were using bucket cache in offheap mode?
>>
>> Yes, but it is a different hbase cluster and it run 0.98. The one I am
>> trying to setup offheap cache is hbase 1.1.2
>>
>>
>> -arul
>>
>>
>> On Tue, Sep 12, 2017 at 9:34 AM, ramkrishna vasudevan <
>> ramkrishna.s.vasude...@gmail.com> wrote:
>>
>> > Hi
>> >
>> > So are you trying to use bucket cache feature in offheap mode with 1.1.2?
>> > If so then it is needed that the MaxDirectMemory is greater than the
>> > offheap bucket cache size.
>> >
>> > If you are not using in offheap mode then probably there is no need for
>> > MaxDirectMemory to be greater than bucket cache size.
>> >
>> > >>in Hbase 0.98, I had to set -XX:MaxDirectMemorySize less than
>> > hbase.bucket.cache.size
>> > So even in 0.98 you were using bucket cache in offheap mode?
>> >
>> > Regards
>> > Ram
>> >
>> > On Tue, Sep 12, 2017 at 9:40 PM, Ted Yu  wrote:
>> >
>> > > Looks like the config you meant should be hbase.bucketcache.size
>> > >
>> > > As the refguide says:
>> > >
>> > > A float that EITHER represents a percentage of total heap memory size
>> to
>> > > give to the cache (if < 1.0) OR, it is the total capacity in megabytes
>> of
>> > > BucketCache. Default: 0.0
>> > >
>> > > If you specify the size as capacity, -XX:MaxDirectMemorySize should be
>> > > bigger than the capacity.
>> > >
>> > > For #2, did you encounter some error ?
>> > >
>> > > Cheers
>> > >
>> > > On Tue, Sep 12, 2017 at 8:52 AM, Arul Ramachandran > >
>> > > wrote:
>> > >
>> > > > In HBase 1.1.2, I am setting up bucket cache. I set MaxDirectMemory
>> > size
>> > > > greater than hbase.bucket.cache.size  - only then it would work.
>> > > >
>> > > > 1) Does HBASE_REGIONSERVER_OPTS -XX:MaxDirectMemorySize needs to be
>> > > greater
>> > > > than hbase.bucket.cache.size?
>> > > > 2) It seems with hbase 1.1.2, HBASE_MASTER_OPTS also needs the
>> > > >  -XX:MaxDirectMemorySize setting?
>> > > >
>> > > > IIRC, in Hbase 0.98, I had to set -XX:MaxDirectMemorySize less than
>> > > > hbase.bucket.cache.size --and-- I did not have to set
>> > > >  -XX:MaxDirectMemorySize for HBASE_MASTER_OPTS.
>> > > >
>> > > >
>> > > > Thanks,
>> > > > Arul
>> > > >
>> > >
>> >
>>

Re: Co-processors: Know if row is being deleted by TTL

[Anoop John]

You cannot rely on postDelete for cells removed because of TTL..
Because it is not a user started delete op and there wont be any
postDelete happening for this cell..The cell is there in some
HFile.  Even after the TTL time elapsed also..  When a read happens,
the read flow will filter out such cells from reaching client..  Also
if there been a compaction happened involving this file,  the cell
would have got removed.So you can not really guarantee.

-Anoop-

On Wed, Aug 30, 2017 at 3:15 AM, Ted Yu  wrote:
> I wonder if you can make use of per Cell TTL ?
>
> See third paragraph under http://hbase.apache.org/book.html#ttl
>
> Basically there would be two flags (same row): one for marking whether user
> has submitted the comment, the other is for the TTL you described.
> When postDelete() is called on the TTL'ed cell, you can retrieve the first
> flag and make decision.
>
> FYI
>
> On Mon, Aug 28, 2017 at 10:31 PM, Cheyenne Forbes <
> cheyenne.osanu.for...@gmail.com> wrote:
>
>> I'm sorry, "*postDelete*" wasnt supposed to be a link, I copied the text
>> and gmail sent it as a link.
>>
>> *Use case:*
>>
>> When a user is commenting with a photo it is uploaded to be able to be
>> previewed and the id of the photo is added to a hbase table "tempFiles"
>> with a 5 hour TTL for each row, if the user submits the comment, the row is
>> deleted manually and *postDelete* would "ignore" it but if it isnt
>> submitted and the TTL reaches, *postDelete* would run a piece of code which
>> would delete the file from my file server
>>
>>
>> On Mon, Aug 28, 2017 at 5:33 PM, Ted Yu  wrote:
>>
>> > There is no such indication from the book you mentioned.
>> > Can you tell us your use case where such distinction is needed ?
>> >  Original message From: Cheyenne Forbes <
>> > cheyenne.osanu.for...@gmail.com> Date: 8/28/17  2:51 PM  (GMT-08:00) To:
>> > user@hbase.apache.org Subject: Co-processors: Know if row is being
>> > deleted by TTL
>> > Is is possible to know if a row is being deleted manually or by the
>> > reaching of the TTL on "*postDelete
>> > > > hbase/coprocessor/BaseRegionObserver.html#postDelete(org.apache.hadoop.
>> > hbase.coprocessor.ObserverContext,%20org.apache.
>> > hadoop.hbase.client.Delete,%20org.apache.hadoop.hbase.
>> > regionserver.wal.WALEdit,%20boolean)>*
>> > "?
>> >
>> > Regards,
>> >
>> > Cheyenne O. Forbes
>> >
>>

Re: why use at least one rowlock to update

[Anoop John]

> since  if i have 10 rows to update,then it means that 9 rows have no 
> rowlocks.so is 'at least one rowlock' more symbolic than practical 
> significane?

Pls note that it is doMiniBatchMutation.  10 rows there in ur batch.
The batch might get complete in many mini batches.  When one mini
batch is mutated, all row locks in that mini batch will be obtained.
So there is no mutate of a row with out getting its lock

-Anoop-

On Mon, Aug 28, 2017 at 9:53 PM, Josh Elser  wrote:
> Please be patient when waiting for a response. Apache HBase is made up of
> community volunteers. Your expectation should be on the order of many hours
> (although, we strive to be faster).
>
> I'd recommend that you begin by looking at a newer release than 0.94.2 as
> there have likely been over 50 (guessing) newer releases since this release
> was made.
>
> In general, rowlocks must be acquired to ensure that multiple updates to the
> same row are applied in full. Consider two transactions against the same
> row:
>
> tx1: row=r1 set a=1, b=1
> tx2: row=r1 set a=2, b=2
>
> After tx1 and tx2 are both applied, r1 must either have the columns a=1,b=1
> or a=2,b=2. It must not have a=1,b=2 or a=2,b=1. This is why some form of
> mutual exclusion must be applied inside of the RegionServer.
>
>
> On 8/28/17 10:06 AM, JH Lin wrote:
>>
>> ping...
>>
>>
>>
>>
>>
>>
>> 在2017年08月28 17时55分, "JH Lin"写道:
>>
>> hi all,i dived into hbase source these days.when i read at the
>> HRegion#doMiniBatchMutation(...) block then one question flashed in my
>> mind:why did writer use at least one rowlock to implement mutation?(ie.Puts
>> or Deletes etc)
>>   since  if i have 10 rows to update,then it means that 9 rows have no
>> rowlocks.so is 'at least one rowlock' more symbolic than practical
>> significane?
>>   AFAIK now,i think it's used to verify hbase' basic functions eg. rowlock
>> mechanism.
>> ---
>> release:hbase-0.94.2
>> pardon me if it's a bit old,but i just wonder to know its principle inside
>> it.
>> one more thing, my english is poor,so sorry for any typo.
>> many thanks~
>>
>>
>>
>

Re: [Help] minor compact is continuously consuming the disk space until run out of space?

[Anoop John]

>We think minor compact is just combining files , say, if we have 10 hfiles 
>using 100G disk space, after minor compact, it still should be 100G, if not 
>less.

That is partially true.  The thing is when the new compacted files is
created, temp there will be duplicate data. Means unless the
compaction process is over and system takes new files, even the older
files can not get deleted.  The issue seems like the storage space is
less for u.  You r saying that when compaction is starting already 80%
space is occupied.The remaining is not just enough to accommodate
this temp increase in space need.

-Anoop-

On Sat, Aug 26, 2017 at 7:47 PM, Ted Yu  wrote:
> Even if you disable minor compaction during bulk load, wouldn't subsequent
> compaction(s) run into the same problem ?
>
> Please take a look at the 3rd paragraph under
> http://hbase.apache.org/book.html#compaction .
>
> You can also read
> http://hbase.apache.org/book.html#compaction.file.selection.old to see how
> different parameters are used for file selection.
>
> By "controling the number of hfiles" I mean reducing the amount of data for
> each bulk load.
>
> If the regions for this table are not evenly distributed, some region
> server(s) may receive more data than the other servers.
>
> Cheers
>
> On Sat, Aug 26, 2017 at 7:03 AM, Liu, Ming (Ming)  wrote:
>
>> Thanks Ted,
>>
>> I don't know how to control the number of hfiles, need to check the
>> importtsv tool. But is there anyway we can disable 'minor compaction' now?
>> And why 'minor compaction' will increase the disk usage. The system is
>> idle, there are no other workload, just after load data, and HBase start to
>> do minor compact and we see disk space are smaller and smaller until
>> running out.
>> We think minor compact is just combining files , say, if we have 10 hfiles
>> using 100G disk space, after minor compact, it still should be 100G, if not
>> less. It is called compaction, isn't it? So we don't understand why it is
>> using so many extra disk space? Anything wrong in our system?
>>
>> thanks,
>> Ming
>>
>> -Original Message-
>> From: Ted Yu [mailto:yuzhih...@gmail.com]
>> Sent: Saturday, August 26, 2017 9:54 PM
>> To: user@hbase.apache.org
>> Subject: Re: [Help] minor compact is continuously consuming the disk space
>> until run out of space?
>>
>> bq. on each Region Server there are about 800 hfiles
>>
>> Is it possible to control the number of hfiles during each bulk load ?
>>
>> For this big table, are the regions evenly spread across the servers ? If
>> so, consider increasing the capacity of your cluster.
>>
>> From the doc for hbase.hstore.compactionThreshold :
>>
>> Larger values delay compaction, but when compaction does occur, it takes
>> longer to complete.
>>
>>
>> On Sat, Aug 26, 2017 at 6:48 AM, Liu, Ming (Ming) 
>> wrote:
>>
>> > hi, all,
>> >
>> > We have a system with 17 nodes, with a big table about 28T in size. We
>> use
>> > native hbase bulkloader (importtsv) to load data, and it generated a lot
>> of
>> > hfiles, on each Region Server there are about 800 hfiles.  We turned off
>> > Major Compact, but the Minor compaction is running due to so many hfile.
>> > The problem is, after the initial loading, there are about 80% disk space
>> > used, when minor compaction is going on, we notice the disk space is
>> > reducing rapidly until all disk spaces are used and hbase went down.
>> >
>> > We try to change the hbase.hstore.compactionThreshold to 2000, but the
>> > minor compaction is still triggered.
>> >
>> > The system is CDH 5.7, HBase is 1.2.
>> >
>> > Could anyone help to give us some suggestions? We are really stuck.
>> Thanks
>> > in advance.
>> >
>> > Thanks,
>> > Ming
>> >
>>

Re: Multiple column families - scan performance

[Anoop John]

So on the 2nd table, even if there are 4 CFs , while scanning you need
only data from single CF.  And this under test CF is similar to what u
have in the 1st table?  I mean same encoding and compression schema
and data size?   While creating scan for 2nd table how u make?  I hope
u do
Scan s = new Scan();
s.setStartRow
s.setStopRow
s.addFamily(cf)

Correct?

-Anoop-

On Thu, Aug 17, 2017 at 4:42 PM, Partha  wrote:
> I have 2 HBase tables - one with a single column family, and other has 4
> column families. Both tables are keyed by same rowkey, and the column
> families all have a single column qualifier each, with a json string as
> value (each json payload is about 10-20K in size). All column families use
> fast-diff encoding and gzip compression.
>
> After loading about 60MM rows to each table, a scan test on (any) single
> column family in the 2nd table takes 4x the time to scan the single column
> family from the 1st table. In both cases, the scanner is bounded by a start
> and stop key to scan 1MM rows. Performance did not change much even after
> running a major compaction on both tables.
>
> Though HBase doc and other tech forums recommend not using more than 1
> column family per table, nothing I have read so far suggests scan
> performance will linearly degrade based on number of column families. Has
> anyone else experienced this, and is there a simple explanation for this?
>
> To note, the reason second table has 4 column families is even though I
> only scan one column family at a time now, there are requirements to scan
> multiple column families from that table given a set of rowkeys.
>
> Thanks for any insight into the performance question.

Re: [DISCUSS] Planning changes on RegionServer totalRequestCount metrics

[Anoop John]

I see..  Good...  Ya +1

On Tue, Aug 8, 2017 at 8:46 AM, Yu Li <car...@gmail.com> wrote:
> Thanks for chiming in @Anoop. Jerry raised the same question in JIRA and
> the patch is already updated there, will rename the metrics to
> "totalRowActionRequestCount". Will add release note to make it clear for
> user what the final changes are
>
> Best Regards,
> Yu
>
> On 7 August 2017 at 15:19, Anoop John <anoop.hb...@gmail.com> wrote:
>
>> Sorry for being late here Yu Li.
>> Regarding counting the rows (for the new metric) in multi..  There
>> might be 2 Actions in multi request for the same row. This is possible
>> some time.  I dont think we should check that and try to make it
>> perfect. That will have perf penalty also.  So just saying that we
>> will have some possible inconsistency even after.  May be we can say
>> how many actions in multi not rows affected!  any better name ?
>>
>> On Mon, Aug 7, 2017 at 8:07 AM, Yu Li <car...@gmail.com> wrote:
>> > Thanks for chiming in @stack and @Jerry, will try to add a good release
>> > note when the work done.
>> >
>> > Since already more than 72 hours passed and no objections, I'd like to
>> call
>> > this discussion closed and apply the change in HBASE-18469. Thanks.
>> >
>> > Best Regards,
>> > Yu
>> >
>> > On 4 August 2017 at 13:59, stack <saint@gmail.com> wrote:
>> >
>> >> +1
>> >>
>> >> We need a fat release note on this change so operators can quickly learn
>> >> why traffic went down on upgrade.
>> >>
>> >> S
>> >>
>> >> On Aug 3, 2017 14:49, "Yu Li" <car...@gmail.com> wrote:
>> >>
>> >> > Dear all,
>> >> >
>> >> > Recently in HBASE-18469 <https://issues.apache.org/
>> >> jira/browse/HBASE-18469
>> >> > >
>> >> > we found some inconsistency on regionserver request related metrics,
>> >> > including:
>> >> > 1. totalRequestCount could be less than readRequestCount+
>> >> writeRequestCount
>> >> > 2. For multi request, we count action count into totalRequestCount,
>> while
>> >> > for scan with caching we count only one.
>> >> >
>> >> > To fix the inconsistency, we plan to make below changes:
>> >> > 1. Make totalRequestCount only counts rpc request, thus multi request
>> >> will
>> >> > only count as one for totalRequestCount
>> >> > 2. Introduce a new metrics in name of "totalRowsRequestCount", which
>> will
>> >> > count the DML workloads on RS by row-level action, and for this
>> metrics
>> >> we
>> >> > will count how many rows included for multi and scan-with-caching
>> >> request.
>> >> >
>> >> > After the change, there won't be any compatibility issue -- existing
>> >> > monitoring system could still work -- only that totalRequestCount
>> will be
>> >> > less than previous. And it's recommended to use totalRowsRequestCount
>> to
>> >> > check the RS DML workload.
>> >> >
>> >> > Please kindly let us know if you have any different idea or suggestion
>> >> > (operators' opinion is especially welcomed).
>> >> >
>> >> > Let's make this discussion open for 72 hours and will make the change
>> if
>> >> no
>> >> > objections.
>> >> >
>> >> > Thanks!
>> >> >
>> >> > Best Regards,
>> >> > Yu
>> >> >
>> >>
>>

Re: hbaseconasia2017 slides and a few phtotos

[Anoop John]

Thanks for the write up Stack.

Thanks Huawei and all sponsors and specially to Jieshan.  Few months
back, such a conference was just a dream and it is mainly because of
him this could happen.  Hope we can continue this HBaseconAsia.

-Anoop-

On Tue, Aug 8, 2017 at 7:02 AM, Bijieshan  wrote:
> Thanks for uploading all the slides and the good write-up, stackHope more 
> people will join us next year:)
>
> Jieshan.
> -Original Message-
> From: saint@gmail.com [mailto:saint@gmail.com] On Behalf Of Stack
> Sent: 2017年8月7日 23:12
> To: HBase Dev List ; Hbase-User 
> Subject: hbaseconasia2017 slides and a few phtotos
>
> We had a nice day out in Shenzhen at HBaseCon Asia last friday [0] (August 
> 4th). There were a bunch of great talks [1] by the likes of jd.com, huawei, 
> xiaomi, alibaba and others. Those given in Chinese had best attendance. A 
> good few folks graciously had slides in English for the language-dumb (like
> myself) while their talk native.
>
> A couple of hbase-as-a-service in the cloud are coming down the pipe, a few 
> presenters talked about being at current scale limits, China Life keeps all 
> data as JSON in hbase instances, and there was an interesting talk on upping 
> utilization by deploying hbase with kubernetes (>1 container per node).
>
> Best quote: "HBase is just a kid--it is only ten years old" from our Yu Li 
> talking about interesting write speedups coming from Alibaba arguing there 
> are many more speedups to be had.
>
> I attached a few pictures. I'll put up more after I fixup the hbasecon home 
> page and redirect later.
>
> The day after, the 5th, there was a dev meetup at the Huawei office; notes to 
> follow.
>
> Thanks again to the program committee, the sponsors, and to our gracious host 
> Huawei. Jieshan in particular did an amazing job running the show taking care 
> of speakers.
>
> St.Ack
> 0.  Click on 'view details' on the right hand side of this page for agenda 
> (eventbrite won't show the event page at the end of the link
> post-the-event): https://www.eventbrite.com/e/hbasecon-asia-2017-
> tickets-34935546159#
> 1. Slides up on slideshare: https://www.slideshare.net/ 
> search/slideshow?searchfrom=header=hbaseconasia2017=
> any=all=en=
> 
>
>  _YP99027.JPG
> 
>
> The program committee taking questions at the end of the day
>
>  _YP99181.JPG
> 
>
> This one is of all the speakers
>
>  _YP99193.JPG
> 
>

Re: [DISCUSS] Planning changes on RegionServer totalRequestCount metrics

[Anoop John]

Sorry for being late here Yu Li.
Regarding counting the rows (for the new metric) in multi..  There
might be 2 Actions in multi request for the same row. This is possible
some time.  I dont think we should check that and try to make it
perfect. That will have perf penalty also.  So just saying that we
will have some possible inconsistency even after.  May be we can say
how many actions in multi not rows affected!  any better name ?

On Mon, Aug 7, 2017 at 8:07 AM, Yu Li  wrote:
> Thanks for chiming in @stack and @Jerry, will try to add a good release
> note when the work done.
>
> Since already more than 72 hours passed and no objections, I'd like to call
> this discussion closed and apply the change in HBASE-18469. Thanks.
>
> Best Regards,
> Yu
>
> On 4 August 2017 at 13:59, stack  wrote:
>
>> +1
>>
>> We need a fat release note on this change so operators can quickly learn
>> why traffic went down on upgrade.
>>
>> S
>>
>> On Aug 3, 2017 14:49, "Yu Li"  wrote:
>>
>> > Dear all,
>> >
>> > Recently in HBASE-18469 > jira/browse/HBASE-18469
>> > >
>> > we found some inconsistency on regionserver request related metrics,
>> > including:
>> > 1. totalRequestCount could be less than readRequestCount+
>> writeRequestCount
>> > 2. For multi request, we count action count into totalRequestCount, while
>> > for scan with caching we count only one.
>> >
>> > To fix the inconsistency, we plan to make below changes:
>> > 1. Make totalRequestCount only counts rpc request, thus multi request
>> will
>> > only count as one for totalRequestCount
>> > 2. Introduce a new metrics in name of "totalRowsRequestCount", which will
>> > count the DML workloads on RS by row-level action, and for this metrics
>> we
>> > will count how many rows included for multi and scan-with-caching
>> request.
>> >
>> > After the change, there won't be any compatibility issue -- existing
>> > monitoring system could still work -- only that totalRequestCount will be
>> > less than previous. And it's recommended to use totalRowsRequestCount to
>> > check the RS DML workload.
>> >
>> > Please kindly let us know if you have any different idea or suggestion
>> > (operators' opinion is especially welcomed).
>> >
>> > Let's make this discussion open for 72 hours and will make the change if
>> no
>> > objections.
>> >
>> > Thanks!
>> >
>> > Best Regards,
>> > Yu
>> >
>>

Re: Awesome HBase - a curated list

[Anoop John]

>What's about the HBase to SOLR Lily indexer? Works like a charm!
Already listed under " Secondary Indices"

-Anoop-

On Wed, Aug 2, 2017 at 5:25 PM, Jean-Marc Spaggiari
<jean-m...@spaggiari.org> wrote:
> What's about the HBase to SOLR Lily indexer? Works like a charm!
>
> Le 2 août 2017 7:50 AM, "Sanel Zukan" <san...@gmail.com> a écrit :
>
>> Few more integrations:
>>
>> * Apache Drill (https://drill.apache.org/docs/querying-hbase/)
>> * Apache Impala (https://impala.apache.org/index.html and
>> https://www.cloudera.com/documentation/enterprise/
>> latest/topics/impala_hbase.html)
>> * ElasticSearch HBase River (https://github.com/mallocator/Elasticsearch-
>> HBase-River)
>> * hbase-cascalog (https://github.com/sorenmacbeth/hbase-cascalog) -
>>   Cascalog queries on top of HBase
>>
>> GUI clients:
>>
>> * hrider (https://github.com/NiceSystems/hrider)
>> * DBSchema (http://www.dbschema.com/index.html)
>> * SquirrelSQL and DBeaver - they are generic SQL gui clients, but can be
>>   used with HBase in combination with dedicated JDBC driver or
>> Hive/Phoenix.
>>
>> Commercial JDBC driver (haven't tried it):
>>
>> * https://www.cdata.com/drivers/hbase/download/jdbc/
>>
>> Best,
>> Sanel
>>
>> Robert Yokota <rayok...@gmail.com> writes:
>> > Thanks, I added EsgynDB.
>> >
>> > On Tue, Aug 1, 2017 at 11:27 AM, Dave Birdsall <dave.birds...@esgyn.com>
>> > wrote:
>> >
>> >> Hi,
>> >>
>> >> Another SQL engine you may wish to add to your list is EsgynDB (
>> >> https://esgyn.com/). This engine is a commercial SQL engine providing
>> >> ACID transactions and BI analytics on top of Hadoop. It is based on
>> Apache
>> >> Trafodion (incubating).
>> >>
>> >> Regards,
>> >>
>> >> Dave
>> >>
>> >> -Original Message-
>> >> From: ramkrishna vasudevan [mailto:ramkrishna.s.vasude...@gmail.com]
>> >> Sent: Tuesday, August 1, 2017 11:16 AM
>> >> To: user@hbase.apache.org
>> >> Subject: Re: Awesome HBase - a curated list
>> >>
>> >> I was going thro those links yesterday and wanted to ask you is these
>> >> projects http://www.geomesa.org/ still active?
>> >>
>> >> BTW thanks for letting me know  the ecosystem that are available on
>> HBase.
>> >>
>> >> Regards
>> >> Ram
>> >>
>> >> On Tue, Aug 1, 2017 at 10:59 PM, Anoop John <anoop.hb...@gmail.com>
>> wrote:
>> >>
>> >> > This is great and very useful.. Tks.
>> >> >
>> >> > On Mon, Jul 31, 2017 at 8:34 PM, Robert Yokota <rayok...@gmail.com>
>> >> wrote:
>> >> > > To help me keep track of all the awesome stuff in the HBase
>> >> > > ecosystem, I started a list.  Let me know if I missed anything
>> awesome.
>> >> > >
>> >> > > https://github.com/rayokota/awesome-hbase
>> >> >
>> >>
>>

Re: Awesome HBase - a curated list

[Anoop John]

This is great and very useful.. Tks.

On Mon, Jul 31, 2017 at 8:34 PM, Robert Yokota  wrote:
> To help me keep track of all the awesome stuff in the HBase ecosystem, I
> started a list.  Let me know if I missed anything awesome.
>
> https://github.com/rayokota/awesome-hbase

Re: [ANNOUNCE] New HBase committer Abhishek Singh Chouhan

[Anoop John]

Congrats Abhishek

On Mon, Jul 31, 2017 at 1:48 PM, Yu Li  wrote:
> Congratulations, Abhishek!
>
> Best Regards,
> Yu
>
> On 31 July 2017 at 15:40, Jingcheng Du  wrote:
>
>> Congratulations!
>>
>> Regards,
>> Jingcheng
>>
>> 2017-07-31 12:49 GMT+08:00 ramkrishna vasudevan <
>> ramkrishna.s.vasude...@gmail.com>:
>>
>> > Congratulations Abhishek !!!
>> >
>> > Regards
>> > Ram
>> >
>> > On Mon, Jul 31, 2017 at 10:16 AM, Pankaj kr 
>> wrote:
>> >
>> > > Congratulations Abhishek..!!
>> > >
>> > > Thanks & Regards,
>> > > Pankaj
>> > >
>> > > HUAWEI TECHNOLOGIES CO.LTD.
>> > > Huawei Tecnologies India Pvt. Ltd.
>> > > Near EPIP Industrial Area, Kundalahalli Village
>> > > Whitefield, Bangalore-560066
>> > > www.huawei.com
>> > > 
>> > > -
>> > > This e-mail and its attachments contain confidential information from
>> > > HUAWEI, which
>> > > is intended only for the person or entity whose address is listed
>> above.
>> > > Any use of the
>> > > information contained herein in any way (including, but not limited to,
>> > > total or partial
>> > > disclosure, reproduction, or dissemination) by persons other than the
>> > > intended
>> > > recipient(s) is prohibited. If you receive this e-mail in error, please
>> > > notify the sender by
>> > > phone or email immediately and delete it!
>> > >
>> > > -Original Message-
>> > > From: Andrew Purtell [mailto:apurt...@apache.org]
>> > > Sent: Saturday, July 29, 2017 6:32 AM
>> > > To: d...@hbase.apache.org; user@hbase.apache.org
>> > > Subject: [ANNOUNCE] New HBase committer Abhishek Singh Chouhan
>> > >
>> > > On behalf of the Apache HBase PMC, I am pleased to announce that
>> Abhishek
>> > > Singh Chouhan has accepted the PMC's invitation to become a committer
>> on
>> > > the project.
>> > >
>> > > We appreciate all of Abhishek's great work thus far and look forward to
>> > > continued involvement.
>> > >
>> > > Please join me in congratulating Abhishek!
>> > >
>> > > --
>> > > Best regards,
>> > > Andrew
>> > >
>> >
>>

Re: EndpointCoprocessors with multi-region access

[Anoop John]

You got it correct.
Within your EP (this handles one region), you can get that Region
memstore size and add them all at client side.

-Anoop-

On Tue, Jul 25, 2017 at 12:10 AM, Dave Birdsall  wrote:
> Hi,
>
> I think I understand the answer.
>
> My question was based on incorrect premises. (You can tell I am new to this.)
>
> The CoprocesserService() method will send requests to all region servers 
> serving regions within a given key range. So each coprocessor instance is 
> handling just one region. I suppose one could write badly behaved code in a 
> coprocessor instance that does cross servers, but the natural architecture of 
> an EndPoint coprocessor is to work on one region locally.
>
> The client code that calls CoprocessorService is responsible for processing 
> the set of responses from each region server that was called.
>
> So in my example, some client side code has to loop through these, adding 
> together the results from each response.
>
> Thanks,
>
> Dave
>
> From: Dave Birdsall
> Sent: Monday, July 24, 2017 9:30 AM
> To: user@hbase.apache.org
> Subject: EndpointCoprocessors with multi-region access
>
> Hi,
>
> I have a basic question about Endpoint coprocessors.
>
> Suppose I want to write a coprocessor that returns the total number of 
> memstore bytes used by a table.
>
> I can write code that loops through all the regions, asking their region 
> servers to tell me the memstore bytes for each given region, and then add 
> them all up.
>
> Such code, of course, will have a RegionServer talking to other RegionServers 
> in the cluster.
>
> Is there any problem with this? For example, when a RegionServer does an RPC 
> to another RegionServer, does that tie up a thread in the calling 
> RegionServer? And if so, and if my coprocessor is popular, might I get 
> deadlocks or thread exhaustion errors if multiple RegionServers run my 
> coprocessor?
>
> The more general architectural question is, should an EndPoint coprocessor 
> limit itself to the regions that are on its own RegionServer? Or does HBase 
> possess appropriate layers to robustly manage prolific cross-server traffic?
>
> Thanks,
>
> Dave

Re: HBase GET operation max row size - partial results

[Anoop John]

You mean within your RegionObserver you are doing the get?  Within
which hook?   What is the way you are doing the get?   Can u paste
that sample code.

-Anoop-

On Wed, Jul 26, 2017 at 8:02 PM, Veerraju Tadimeti  wrote:
> Hi,
>
> If i use GET operation, is there any chance of getting partial result? If
> Yes, under what circumstances.  Is there any way to reproduce it?
>
> I am using GET operation in my coProcessor ( to the same region), adding
> the resut to the List .  I am afraid that any chance  of partial
> result when using GET operation, since GET uses SCAN operation internally.
>
> Thanks,
> Raju,
> (972)273-0155.

Re: scope of RegionCoprocessorEnvironment sharedData

[Anoop John]

No co processors can be configured either globally at cluster level
(RS configuration) or at table level.

You dont need preScannerOpen.  I checked the code and we are passing
the actual scanner been created to postScannerOpen() method.  This
being null in ur case been strange!  Pls make sure u r implementing
post hook not pre.

-Anoop-

On Wed, Jul 12, 2017 at 3:01 AM, Veerraju Tadimeti <tvvr...@gmail.com> wrote:
> Can I load coprocessor dynamically for a scan operation, it should not be 
> loaded for another scan operation if not intended
>  btw I invoke scan from hive
>
> Sent from my iPhone
>
>> On Jul 11, 2017, at 4:15 PM, Veerraju Tadimeti <tvvr...@gmail.com> wrote:
>>
>> hi,
>>
>> Hi John,
>>
>> Thanks for the reply.
>>
>> I implemented #2 (another way) in ur above post:
>>
>>
>>
>> i debug the logs  : in PostScannerOpen() , regionScanner method parameter 
>> object is null
>>
>> Also, in preScannerOpen() , i returned return super.preScannerOpen(e, scan, 
>> new DelegateRegionScanner(s));
>> in postScannerNext() , internalScanner object is 
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl
>>
>> #1 way (Put scanner in local map) - may not be possible, cos if two 
>> different scan operation with and without attribute hits at the same time, 
>> how can we differentiate in postScannerNext.
>>
>>
>> Thanks,
>> Raju,
>> (972)273-0155.
>>
>>> On Tue, Jul 11, 2017 at 8:05 AM, Anoop John <anoop.hb...@gmail.com> wrote:
>>> Ya. It is the same RegionScanner impl in use only being passed.  Ya
>>> the param type should have been RegionScanner  I guess. We made that
>>> mistake!
>>> -Anoop-
>>>
>>> On Mon, Jul 10, 2017 at 8:37 PM, Ted Yu <yuzhih...@gmail.com> wrote:
>>> > The tricky part is that postScannerNext() passes InternalScanner parameter
>>> > instead of RegionScanner.
>>> >
>>> > FYI
>>> >
>>> > On Sun, Jul 9, 2017 at 10:57 PM, Anoop John <anoop.hb...@gmail.com> wrote:
>>> >
>>> >> Ya as Ted said, u are not getting Scan object in the postScannerNext
>>> >> and so can not make use of the attribute in Scan within this hook.
>>> >> Just setting the sharedData variable will cause issue with concurrent
>>> >> scans. (As u imagine)
>>> >>
>>> >> So I can think of solving this in 2 possible ways. (May be more ways
>>> >> possible)
>>> >>
>>> >> 1.  U keep a Map within ur CP impl.  You implement postScannerOpen
>>> >> where u will get the ref to Scanner been created as well as the Scan.
>>> >> If the Scan is having attribute, keep that scanner within ur Map.
>>> >> During postScannerNext  check if the coming in scanner is there in ur
>>> >> Map. If so that means this is the one where u can do the action.
>>> >> Also dont forget to implement postScannerClose and remove that scanner
>>> >> from the Map.   Here u might have some perf penalty as u have to add
>>> >> and get from Map which has to be a concurrent map too.
>>> >>
>>> >> Another way
>>> >>
>>> >> 2. Create a custom scanner implementing RegionScanner.   The new one
>>> >> has to take an original Region Scanner and just delegate the calls. On
>>> >> postScannerOpen, u will get the original scanner been created and u
>>> >> can just wrap it with ur new scanner object. ( If the Scan object is
>>> >> having required attribute)..  In postScannerNext() u can check for ur
>>> >> own RegionScanner type and if so u can do action.
>>> >>
>>> >>
>>> >> -Anoop-
>>> >>
>>> >>
>>> >> On Sat, Jul 8, 2017 at 9:13 PM, Ted Yu <yuzhih...@gmail.com> wrote:
>>> >> > if (canUseGetOperation(e)) {
>>> >> >//logic goes here
>>> >> >
>>> >> > Does your Get target the same region being scanned ?
>>> >> > If not, issuing the Get is not advised since the other region may be
>>> >> hosted
>>> >> > on different region server.
>>> >> >
>>> >> > Cheers
>>> >> >
>>> >> > On Thu, Jul 6, 2017 at 7:14 AM, Veerraju Tadimeti <tvvr...@gmail.com>
>>> >> wrote:
>>

Re: scope of RegionCoprocessorEnvironment sharedData

[Anoop John]

Ya. It is the same RegionScanner impl in use only being passed.  Ya
the param type should have been RegionScanner  I guess. We made that
mistake!
-Anoop-

On Mon, Jul 10, 2017 at 8:37 PM, Ted Yu <yuzhih...@gmail.com> wrote:
> The tricky part is that postScannerNext() passes InternalScanner parameter
> instead of RegionScanner.
>
> FYI
>
> On Sun, Jul 9, 2017 at 10:57 PM, Anoop John <anoop.hb...@gmail.com> wrote:
>
>> Ya as Ted said, u are not getting Scan object in the postScannerNext
>> and so can not make use of the attribute in Scan within this hook.
>> Just setting the sharedData variable will cause issue with concurrent
>> scans. (As u imagine)
>>
>> So I can think of solving this in 2 possible ways. (May be more ways
>> possible)
>>
>> 1.  U keep a Map within ur CP impl.  You implement postScannerOpen
>> where u will get the ref to Scanner been created as well as the Scan.
>> If the Scan is having attribute, keep that scanner within ur Map.
>> During postScannerNext  check if the coming in scanner is there in ur
>> Map. If so that means this is the one where u can do the action.
>> Also dont forget to implement postScannerClose and remove that scanner
>> from the Map.   Here u might have some perf penalty as u have to add
>> and get from Map which has to be a concurrent map too.
>>
>> Another way
>>
>> 2. Create a custom scanner implementing RegionScanner.   The new one
>> has to take an original Region Scanner and just delegate the calls. On
>> postScannerOpen, u will get the original scanner been created and u
>> can just wrap it with ur new scanner object. ( If the Scan object is
>> having required attribute)..  In postScannerNext() u can check for ur
>> own RegionScanner type and if so u can do action.
>>
>>
>> -Anoop-
>>
>>
>> On Sat, Jul 8, 2017 at 9:13 PM, Ted Yu <yuzhih...@gmail.com> wrote:
>> > if (canUseGetOperation(e)) {
>> >//logic goes here
>> >
>> > Does your Get target the same region being scanned ?
>> > If not, issuing the Get is not advised since the other region may be
>> hosted
>> > on different region server.
>> >
>> > Cheers
>> >
>> > On Thu, Jul 6, 2017 at 7:14 AM, Veerraju Tadimeti <tvvr...@gmail.com>
>> wrote:
>> >
>> >> hi,
>> >>
>> >> I have few questions regarding scope of *RegionCoprocessorEnvironment*
>> >>  sharedData.
>> >>
>> >>
>> >>
>> >>- *Is sharedData map is shared accross all instances simultaneously
>> ?*
>> >>   -  I am putting a variable in sharedData in preScannerOpen()
>> based on
>> >>   scan attribute,
>> >>   - check that variable exists in postScannerNext() then apply
>> logic,
>> >>   - remove the variable postScannerClose().
>> >>   - If data is in multiple regions, when one coprocessor removes
>> >>   variable in postScannerClose(), will the variable is NULL for
>> another
>> >>   region coprocessor in postScannerNext() ?
>> >>
>> >>
>> >>- *Is sharedData map is shared across all the client request
>> >>operations ?*
>> >>
>> >> If a variable is set in sharedData for one client operation(say SCAN),
>> will
>> >> the variable is available for another client operation(new SCAN) ?
>> >>
>> >>
>> >>-  *Will the variables be garbage collected even if we dont implement
>> >>(removed variables in sharedData) postScannerClose() method*
>> >>
>> >>
>> >> Please find below the logic that I am using currently
>> >> *CODE: *
>> >>
>> >> public RegionScanner
>> >> *preScannerOpen*(ObserverContext
>> >> e, Scan scan, RegionScanner s) throws IOException {
>> >> byte[] useGetInPostScannerNext = scan.getAttribute(USE_GET_
>> >> OPERATION_IN_POST_SCANNER_NEXT);
>> >> String useGetInPostScannerNextStr = Bytes.toString(
>> >> useGetInPostScannerNext);
>> >> if (Boolean.parseBoolean(useGetInPostScannerNextStr)) {
>> >> e.getEnvironment().getSharedData().put(USE_GET_
>> >> OPERATION_IN_POST_SCANNER_NEXT, useGetInPostScannerNextStr);
>> >> }
>> >> return super.preScannerOpen(e, scan, s);
>> >> }
>> >>
>> >> @Override
&g

Re: scope of RegionCoprocessorEnvironment sharedData

[Anoop John]

Ya as Ted said, u are not getting Scan object in the postScannerNext
and so can not make use of the attribute in Scan within this hook.
Just setting the sharedData variable will cause issue with concurrent
scans. (As u imagine)

So I can think of solving this in 2 possible ways. (May be more ways possible)

1.  U keep a Map within ur CP impl.  You implement postScannerOpen
where u will get the ref to Scanner been created as well as the Scan.
If the Scan is having attribute, keep that scanner within ur Map.
During postScannerNext  check if the coming in scanner is there in ur
Map. If so that means this is the one where u can do the action.
Also dont forget to implement postScannerClose and remove that scanner
from the Map.   Here u might have some perf penalty as u have to add
and get from Map which has to be a concurrent map too.

Another way

2. Create a custom scanner implementing RegionScanner.   The new one
has to take an original Region Scanner and just delegate the calls. On
postScannerOpen, u will get the original scanner been created and u
can just wrap it with ur new scanner object. ( If the Scan object is
having required attribute)..  In postScannerNext() u can check for ur
own RegionScanner type and if so u can do action.


-Anoop-


On Sat, Jul 8, 2017 at 9:13 PM, Ted Yu  wrote:
> if (canUseGetOperation(e)) {
>//logic goes here
>
> Does your Get target the same region being scanned ?
> If not, issuing the Get is not advised since the other region may be hosted
> on different region server.
>
> Cheers
>
> On Thu, Jul 6, 2017 at 7:14 AM, Veerraju Tadimeti  wrote:
>
>> hi,
>>
>> I have few questions regarding scope of *RegionCoprocessorEnvironment*
>>  sharedData.
>>
>>
>>
>>- *Is sharedData map is shared accross all instances simultaneously ?*
>>   -  I am putting a variable in sharedData in preScannerOpen() based on
>>   scan attribute,
>>   - check that variable exists in postScannerNext() then apply logic,
>>   - remove the variable postScannerClose().
>>   - If data is in multiple regions, when one coprocessor removes
>>   variable in postScannerClose(), will the variable is NULL for another
>>   region coprocessor in postScannerNext() ?
>>
>>
>>- *Is sharedData map is shared across all the client request
>>operations ?*
>>
>> If a variable is set in sharedData for one client operation(say SCAN), will
>> the variable is available for another client operation(new SCAN) ?
>>
>>
>>-  *Will the variables be garbage collected even if we dont implement
>>(removed variables in sharedData) postScannerClose() method*
>>
>>
>> Please find below the logic that I am using currently
>> *CODE: *
>>
>> public RegionScanner
>> *preScannerOpen*(ObserverContext
>> e, Scan scan, RegionScanner s) throws IOException {
>> byte[] useGetInPostScannerNext = scan.getAttribute(USE_GET_
>> OPERATION_IN_POST_SCANNER_NEXT);
>> String useGetInPostScannerNextStr = Bytes.toString(
>> useGetInPostScannerNext);
>> if (Boolean.parseBoolean(useGetInPostScannerNextStr)) {
>> e.getEnvironment().getSharedData().put(USE_GET_
>> OPERATION_IN_POST_SCANNER_NEXT, useGetInPostScannerNextStr);
>> }
>> return super.preScannerOpen(e, scan, s);
>> }
>>
>> @Override
>> public boolean *postScannerNext*(final
>> ObserverContext
>> e,
>> final InternalScanner s, final List results, final int
>> limit,
>> final boolean hasMore) throws IOException {
>> try {
>>
>> if (canUseGetOperation(e)) {
>>
>>//logic goes here
>> }
>> } catch (Exception ex) {
>> logger.error("Exception in postScannerNext ", ex);
>> throw new IOException(ex);
>> }
>> return hasMore;
>> }
>>
>> @Override
>> public void
>> *postScannerClose*(ObserverContext
>> e, InternalScanner s) throws IOException {
>> if (canUseGetOperation(e)) {
>> e.getEnvironment().getSharedData().remove(USE_
>> GET_OPERATION_IN_POST_SCANNER_NEXT);
>> }
>> super.postScannerClose(e, s);
>> }
>>
>> private boolean *canUseGetOperation*(final
>> ObserverContext
>> e) {
>> String useGetOperationInPostScannerNext = (String)
>> e.getEnvironment().getSharedData().get(USE_GET_OPERATION_IN_POST_SCANNER_
>> NEXT);
>> return Boolean.parseBoolean(useGetOperationInPostScannerNext);
>> }
>>
>> Thanks,
>> Raju,
>> (972)273-0155 <(972)%20273-0155>.
>>

Re: [ANNOUNCE] Chunhui Shen joins the Apache HBase PMC

[Anoop John]

Congrats Chunhui..

On Wed, Jul 5, 2017 at 6:55 AM, Pankaj kr  wrote:
> Congratulations Chunhui..!!
>
> Regards,
> Pankaj
>
>
> -Original Message-
> From: Yu Li [mailto:car...@gmail.com]
> Sent: Tuesday, July 04, 2017 1:24 PM
> To: d...@hbase.apache.org; Hbase-User
> Subject: [ANNOUNCE] Chunhui Shen joins the Apache HBase PMC
>
> On behalf of the Apache HBase PMC I am pleased to announce that Chunhui Shen 
> has accepted our invitation to become a PMC member on the Apache HBase 
> project. He has been an active contributor to HBase for past many years. 
> Looking forward for many more contributions from him.
>
> Please join me in welcoming Chunhui to the HBase PMC!
>
> Best Regards,
> Yu

Re: Setting TTL at the row level

[Anoop John]

Yes we have per cell level TTL setting from 0.98.9  release onwards.
That should be the best way for ur usecase.
See Mutation#setTTL(long)

-Anoop-

On Fri, Jun 23, 2017 at 2:17 PM, yonghu  wrote:
> I did not quite understand what you mean by "row timestamp"? As far as I
> know, timestamp is associated to each data version (cell). Will you store
> multiple data versions in a single column?
>
>
>
> On Thu, Jun 22, 2017 at 4:35 AM, Jean-Marc Spaggiari <
> jean-m...@spaggiari.org> wrote:
>
>> Why not using the cell level ttl?
>>
>> Le 2017-06-21 2:35 PM, "Vladimir Rodionov"  a
>> écrit :
>>
>> > Should work
>> >
>> > On Wed, Jun 21, 2017 at 11:31 AM,  wrote:
>> >
>> > > Hi all,
>> > >
>> > > I know it is possible to set TTL in HBase at the column family level -
>> > > which makes HBase delete rows in the column family when they reach a
>> > > certain age.
>> > >
>> > > Rather than expire a row after it's reached a certain age, I would like
>> > to
>> > > expire each specific row at a specific time in the future (I.e. set
>> > expiry
>> > > at the row level, rather than at the column family level). To achieve
>> > this,
>> > > I am planning on setting the column family TTL to something very short
>> > > (e.g. 1 minute) and then when I write my rows, I will set the row
>> > timestamp
>> > > to [current datetime + time until I want row to expire]. Since HBase
>> uses
>> > > row timestamp for TTL, this should let me effectively set TTL on the
>> row
>> > > level.
>> > >
>> > > Will this work? Is there any reason not to do this?
>> > >
>> > > Thanks!
>> > > Josh
>> > >
>> > >
>> > >
>> >
>>

Re: [ANNOUNCE] New Apache HBase committer Ashu Pachauri

[Anoop John]

Congrats and welcome...

-Anoop-

On Mon, Jun 19, 2017 at 11:44 AM, Huaxiang Sun  wrote:
> Congratulations!
>
>
> On Sun, Jun 18, 2017 at 9:43 PM, ramkrishna vasudevan <
> ramkrishna.s.vasude...@gmail.com> wrote:
>
>> Congratulations !!!
>>
>> On Mon, Jun 19, 2017 at 9:11 AM, Jingcheng Du  wrote:
>>
>> > Congratulations!
>> >
>> > 2017-06-19 11:26 GMT+08:00 宾莉金（binlijin） :
>> >
>> > > Congratulations, Ashu !
>> > >
>> > > 2017-06-17 7:27 GMT+08:00 Gary Helmling :
>> > >
>> > > > On behalf of the Apache HBase PMC, I am pleased to announce that Ashu
>> > > > Pachauri has accepted the PMC's invitation to become a committer on
>> the
>> > > > project.  We appreciate all of Ashu's generous contributions thus far
>> > and
>> > > > look forward to his continued involvement.
>> > > >
>> > > > Congratulations and welcome, Ashu!
>> > > >
>> > >
>> > >
>> > >
>> > > --
>> > > *Best Regards,*
>> > >  lijin bin
>> > >
>> >
>>
>
>
>
> --
> Huaxiang Sun

Re: [ANNOUNCE] New HBase committer Allan Yang

[Anoop John]

Congrats Allan..   Welcome !!

-Anoop-

On Fri, Jun 9, 2017 at 9:27 AM, 张铎(Duo Zhang)  wrote:
> Congratulations!
>
> 2017-06-09 11:55 GMT+08:00 Ted Yu :
>
>> Congratulations, Allan !
>>
>> On Thu, Jun 8, 2017 at 8:49 PM, Yu Li  wrote:
>>
>> > On behalf of the Apache HBase PMC, I am pleased to announce that Allan
>> Yang
>> > has accepted the PMC's invitation to become a committer on the
>> > project. We appreciate all of Allan's generous contributions thus far and
>> > look forward to his continued involvement.
>> >
>> > Congratulations and welcome, Allan!
>> >
>>

Re: Any Repercussions of using Multiwal

[Anoop John]

You can config this max WALs. (As said by Yu , hbase.regionserver.maxlogs)
When the total un archived WAL files count exceeds this, we will do
force flushes and so release some of the WALs.   As Yu mentioned, when
we use multi WAL and say we have 2 WAL groups, this WAL count
effectively will be  32 * 2 = 64.   But u can config it to a lower
value than the def 32.

-Anoop-

On Tue, Jun 6, 2017 at 6:12 PM, Sachin Jain  wrote:
> Thanks Yu!
> It was certainly helpful.
>
>> Regarding the issue you met, what's the setting of
> hbase.regionserver.maxlogs in your env? By default it's 32 which means for
> each RS the un-archived wal number shouldn't exceed 32. However, when
> multiwal enabled, it allows 32 logs for each group, thus becoming 64 wals
> allowed for a single RS.
>
> I used default configuration for this. By multiWal, I understand there is
> different wal per region. Can you please explain how did you get 64 wals
> for a Region Server.
>
>> when multiwal enabled, it allows 32 logs for each group, thus becoming 64
> wals allowed for a single RS.
>
> I thought one of the side effects of having multiwal enabled is that there
> will be *large amount of data waiting in unarchived wals.*
> So if a region server fails, it would take more time to playback the wal
> files and hence it could *compromise Availability.*
>
> Wdyt ?
>
> Thanks
> -Sachin
>
>
> On Tue, Jun 6, 2017 at 2:04 PM, Yu Li  wrote:
>
>> Hi Sachin,
>>
>> We have been using multiwal in production here in Alibaba for over 2 years
>> and see no problem. Facebook is also running multiwal online. Please refer
>> to HBASE-14457  for
>> more
>> details.
>>
>> There's also a JIRA HBASE-15131
>>  proposing to turn on
>> multiwal by default but still under discussion, please feel free to leave
>> your voice there.
>>
>> Regarding the issue you met, what's the setting of
>> hbase.regionserver.maxlogs in your env? By default it's 32 which means for
>> each RS the un-archived wal number shouldn't exceed 32. However, when
>> multiwal enabled, it allows 32 logs for each group, thus becoming 64 wals
>> allowed for a single RS.
>>
>> Let me further explain how it leads to RegionTooBusyException:
>> 1. if the number of un-archived wal exceeds the setting, it will check the
>> oldest WAL and flush all regions involved in it
>> 2. if the data ingestion speed is high and wal keeps rolling, there'll be
>> many small hfiles flushed out, that compaction speed cannot catch up
>> 3. when hfile number of one store exceeds the setting of
>> hbase.hstore.blockingStoreFiles (10 by default), it will delay the flush
>> for hbase.hstore.blockingWaitTime (90s by default)
>> 4. when data ingestion continues but flush delayed, the memstore size might
>> exceed the upper limit thus throw RegionTooBusyException
>>
>> Hope these information helps.
>>
>> Best Regards,
>> Yu
>>
>> On 6 June 2017 at 13:39, Sachin Jain  wrote:
>>
>> > Hi,
>> >
>> > I was in the middle of a situation where I was getting
>> > *RegionTooBusyException* with log something like:
>> >
>> > *Above Memstore limit, regionName = X ... memstore size = Y and
>> > blockingMemstoreSize = Z*
>> >
>> > This potentially hinted me towards *hotspotting* of a particular region.
>> So
>> > I fixed my keyspace partitioning to have more uniform distribution per
>> > region. It did not completely fix the problem but definitely delayed it a
>> > bit.
>> >
>> > Next thing, I enabled *multiWal*. As I remember there is a configuration
>> > which leads to flushing of memstores when the threshold of wal is
>> reached.
>> > Upon doing this, problem seems to go away.
>> >
>> > But, this raises couple of questions
>> >
>> > 1. Are there any reprecussions of using *multiWal* in production
>> > environment ?
>> > 2. If there are no repercussions and only benefits of using *multiWal*,
>> why
>> > is this not turned on by default. Let other consumers turn it off in
>> > certain (whatever) scenarios.
>> >
>> > PS: *Hbase Configuration*
>> > Single Node (Local Setup) v1.3.1 Ubuntu 16 Core machine.
>> >
>> > Thanks
>> > -Sachin
>> >
>>

Re: Call function from a coprocessor which extends BaseRegionObserver from the client side

[Anoop John]

Have a look at classes VisibilityController or AccessController .
These are observers as well as Endpoint which exposes some APIs for
client

-Anoop-

On Mon, May 15, 2017 at 4:34 PM, Rajeshkumar J
 wrote:
> Hi,
>
>Check this example in the below link
>
>   https://www.3pillarglobal.com/insights/hbase-coprocessors
>
> Thanks
>
> On Mon, May 15, 2017 at 4:00 PM, Cheyenne Forbes <
> cheyenne.osanu.for...@gmail.com> wrote:
>
>> I have a *Coprocessor* which overrides *postPut* and I also want to call a
>> custom function in this Coprocessor from the hbase client. In hbase 0.94 I
>> would use "*public class MyCoprocessor extends BaseRegionObserver
>> implements MyCoprocessorProtocol*" but *CoprocessorProtocol *no longer
>> exists in hbase 1.2.5. I've done some goggling and found
>> *CoprocessorService* but I cant find any way to use *CoprocessorService *
>> with* BaseRegionObserver*
>>
>> Regards,
>>
>> Cheyenne O. Forbes
>>

Re: HBase 1.0 Per Put TTL Not Being Obeyed On Replication

[Anoop John]

At server side (RS) as well as at client side, put the config
"hbase.client.rpc.codec" with a value
org.apache.hadoop.hbase.codec.KeyValueCodecWithTags. Then u will
be able to retrieve the tags back to client side and check

-Anoop-

On Mon, May 1, 2017 at 2:59 AM, Saad Mufti <saad.mu...@gmail.com> wrote:
> Is there any facility to check what tags are on a Cell from a client side
> program? I started writing some Java code to look at the tags on a Cell
> retrieved via a simple Get, but then started reading around and it seems
> tags are not even returned (not returned at all or only in certain cases,
> I'm not clear) to client side code. So how can I verify that a Cell in one
> cluster has the TTL tag whereas the same replicated C3ell in the next
> cluster does or doesn't?
>
> Thanks.
>
> 
> Saad
>
>
> On Fri, Apr 28, 2017 at 1:06 PM, Saad Mufti <saad.mu...@gmail.com> wrote:
>
>> Thanks for the feedback, I have confirmed that in both the main and
>> replica cluster, hbase.replication.rpc.codec is set to:
>>
>> org.apache.hadoop.hbase.codec.KeyValueCodecWithTags
>>
>> I have also run a couple of tests and it looks like the TTL is not being
>> obeyed on the replica for any entry. Almost as if the TTL cell tags are not
>> being replicated. I couldn't find any significant clock skew. If it
>> matters, the HBase version on both sides is 1.0.0-cdh5.5.2
>>
>> Any ideas?
>>
>> Thanks.
>>
>> 
>> Saad
>>
>>
>> On Thu, Apr 27, 2017 at 3:24 AM, Anoop John <anoop.hb...@gmail.com> wrote:
>>
>>> Ya can u check whether the replica cluster is NOT removing ANY of the
>>> TTL expired cells (as per ur expectation from master cluster) or some.
>>> Is there too much clock time skew for the source RS and peer cluster
>>> RS? Just check.
>>>
>>> BTW can u see what is the hbase.replication.rpc.codec configuration
>>> value in both clusters?
>>>
>>> -Anoop-
>>>
>>> On Thu, Apr 27, 2017 at 2:08 AM, Saad Mufti <saad.mu...@gmail.com> wrote:
>>> > Hi,
>>> >
>>> > I have a main HBase 1.x cluster and some of the tables are being
>>> replicated
>>> > to a separate HBase cluster of the same version, and the table schemas
>>> are
>>> > identical. The column family being used has TTL set to "FOREVER", but
>>> we do
>>> > a per put TTL in every Put we issue on the main cluster.
>>> >
>>> > Data is being replicated but we recently caught a number of data items
>>> that
>>> > have disappeared in the main cluster as expected based on their TTL but
>>> not
>>> > in the replica. Both HBase clusters have hfile.format.version set to 3
>>> so
>>> > TTL tags should be obeyed. I haven't checked yet whether it is a case of
>>> > the replica not obeying ANY TTL's or just some.
>>> >
>>> > Before we dig deeper, I was hoping someone in the community would point
>>> it
>>> > out if we have missed any obvious gotchas.
>>> >
>>> > Thanks.
>>> >
>>> > ---
>>> > Saad
>>>
>>
>>

Re: HBase 1.0 Per Put TTL Not Being Obeyed On Replication

[Anoop John]

Ya can u check whether the replica cluster is NOT removing ANY of the
TTL expired cells (as per ur expectation from master cluster) or some.
Is there too much clock time skew for the source RS and peer cluster
RS? Just check.

BTW can u see what is the hbase.replication.rpc.codec configuration
value in both clusters?

-Anoop-

On Thu, Apr 27, 2017 at 2:08 AM, Saad Mufti  wrote:
> Hi,
>
> I have a main HBase 1.x cluster and some of the tables are being replicated
> to a separate HBase cluster of the same version, and the table schemas are
> identical. The column family being used has TTL set to "FOREVER", but we do
> a per put TTL in every Put we issue on the main cluster.
>
> Data is being replicated but we recently caught a number of data items that
> have disappeared in the main cluster as expected based on their TTL but not
> in the replica. Both HBase clusters have hfile.format.version set to 3 so
> TTL tags should be obeyed. I haven't checked yet whether it is a case of
> the replica not obeying ANY TTL's or just some.
>
> Before we dig deeper, I was hoping someone in the community would point it
> out if we have missed any obvious gotchas.
>
> Thanks.
>
> ---
> Saad

Re: Baffling situation with tableExists and createTable

[Anoop John]

Ur earlier attempt to create this table would have failed in btw..  So
the status of the table in zk and in master may be diff.. Table exist
might be checking one and the next steps of crate table another..
Sorry forgot that area of code.. But have seen this kind of situation.
  Not sure whether in some latest versions, these kind of probs are
solved or not.

-Anoop-

On Wed, Apr 26, 2017 at 6:12 AM, Ted Yu  wrote:
> Which hbase release are you using ?
>
> Can you check master log to see if there is some clue w.r.t. LoadTest ?
>
> Using "hbase zkcli", you can inspect the znode status. Below is a sample:
>
> [zk: cn011.x.com:2181,cn013.x.com:2181,cn012.x.com:2181(CONNECTED) 2] ls
> /hbase-unsecure/table
> [hbase:meta, hbase:namespace, IntegrationTestBigLinkedList, datatsv,
> usertable, hbase:backup, TestTable, t2]
> [zk: cn011.x.com:2181,cn013.x.com:2181,cn012.x.com:2181(CONNECTED) 3] ls
> /hbase-unsecure/table/2
> Node does not exist: /hbase-unsecure/table/2
> [zk: cn011.x.com:2181,cn013.x.com:2181,cn012.x.com:2181(CONNECTED) 4] ls
> /hbase-unsecure/table/t2
> []
> [zk: cn011.x.com:2181,cn013.x.com:2181,cn012.x.com:2181(CONNECTED) 5] get
> /hbase-unsecure/table/t2
> �master:16000K��W�,�PBUF
> cZxid = 0x1000a7f01
> ctime = Mon Mar 27 16:50:52 UTC 2017
> mZxid = 0x1000a7f17
> mtime = Mon Mar 27 16:50:52 UTC 2017
> pZxid = 0x1000a7f01
> cversion = 0
> dataVersion = 2
>
> On Tue, Apr 25, 2017 at 4:09 PM, jeff saremi  wrote:
>
>> BTW on the page
>> http://localhost:16010/master-status#userTables
>> there is no sign of the supposedly existing table either
>>
>> 
>> From: jeff saremi 
>> Sent: Tuesday, April 25, 2017 4:05:56 PM
>> To: user@hbase.apache.org
>> Subject: Baffling situation with tableExists and createTable
>>
>> I have a super simple piece of code which tries to create a test table if
>> it does not exist
>>
>> calling admin.tableExists(TableName.valueOf(table)) returns false causing
>> the control to be passed to the line that creates it 
>> admin.createTable(tableDescriptor).
>> Then i get an exception that the table exists!
>>
>> Exception in thread "main" org.apache.hadoop.hbase.TableExistsException:
>> LoadTest
>>
>>
>> String table = config.tableName;
>> ...
>> Connection conn = ConnectionFactory.createConnection(hbaseconf);
>> Admin admin = conn.getAdmin();
>> if(!admin.tableExists(TableName.valueOf(table))) {
>> Log.info("table " + table + " does not exist. Creating it...");
>> HTableDescriptor tableDescriptor = new HTableDescriptor(TableName.
>> valueOf(table));
>> tableDescriptor.addFamily(new HColumnDescriptor(config.FAMILY));
>> admin.createTable(tableDescriptor);
>> }
>>
>> Jeff
>>

Re: Unlimited number of versions for a column family ?

[Anoop John]

This will result in a big sized region..  A row can not be split
across 2 regions.  So that region can not get split

-Anoop-

On Tue, Apr 25, 2017 at 2:58 PM, schausson  wrote:
> OK, so this is not supported for now...
>
> Additional questions related to region size :
>
> If I set "versions" to high value (several millions for instance), the size
> of a row may exceed the region size...What will happen in such case ?
>
> And if I consider another design, like saving versions in additional
> colmumns (column id could be concatenated with history timestamp), the same
> question raises up : if we save millions of columns, what is the behavior
> when row size exceeds the region size ?
>
> Thanks again for your help
>
>
>
>
>
> --
> View this message in context: 
> http://apache-hbase.679495.n3.nabble.com/Unlimited-number-of-versions-for-a-column-family-tp4087673p4087677.html
> Sent from the HBase User mailing list archive at Nabble.com.

Re: ValueFilter returning earlier values

[Anoop John]

This is because of the way with which filters and versions are checked
in RS.  We first do filter op and then apply version. The value filter
might have filtered out latest cell but it is applied and succeed on
older version and then only the version count begin..  There is an
open issue in HBase jira which aims at solving this unexpected
behavior.

-Anoop-

On Thu, Apr 20, 2017 at 1:04 PM, Sreeram  wrote:
> Hi,
>
>  When I scan with ValueFilter on a column, I see that it returns older
> versions too if they happen to match the value in the ValueFilter.
>
> The table column family has the property VERSIONS set to 1. I had set
> setMaxVersions to 1 in the scan object.
>
> I was expecting the value filter to return only the latest values for
> the column, provided they match the filter.
>
> Is this the expected behaviour of ValueFilter? Any suggestions if I
> must be setting any options to skip the older values from coming in
> the result?
>
> Thank you
>
> Regards,
> Sreeram

ANNOUNCE: Yu Li joins the Apache HBase PMC

[Anoop John]

On behalf of the Apache HBase PMC I"m pleased to announce that Yu Li
has accepted our invitation to become a PMC member on the Apache HBase
project. He has been an active contributor to HBase for past many
years. Looking forward for
many more contributions from him.

Welcome to the PMC, Yu Li...


-Anoop-

Re: All memstores flushed to be quite small files

[Anoop John]

You have to keep in mind one thing.  The memstore size 128 MB said  is
the memstore's heap size.  Not the sum of all cell's key+ value size.
 When a Cell is added into memstore there will be big overhead also.
(~100 bytes per cell java overhead).  Ur cell size is so small that
more than half of the total heap size per cell comes from overhead.
There itself ur effective data size flushed getting half.  And then on
top of that u have compression.   Does this sounds reasonable answer?
 (And u said u have only 1 CF so there is no split of 128 MB across
diff CFs)

-Anoop-

On Tue, Mar 28, 2017 at 7:37 AM, Hef  wrote:
> We have only one column family.
> Table properties:
>  {NAME => 't', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'ROW',
> REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'SNAPPY',
> MIN_VERSIONS => '0',KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '6216900',
> IN_MEMORY => 'false', BLOCKCACHE => 'true',TTL=> 5184000}
>
> Each cell is around 90bytes, with rowkey taking 60bytes, column qualifier
> taking 2 bytes, and value 4bytes. There may be 3600 columns at max per row,
> but generally most rows have no more than 1000 columns.
>
> Though SNAPPY is used, I don't think the compression ratio can be so high
> to be 18x.
>
> HBase Release:
> CDH5.9 HBase1.2
>
> Here are more logs:
>
> 2017-03-28 01:53:14,868 INFO org.apache.hadoop.hbase.regionserver.HRegion:
> Flushing 1/1 column families, memstore=128.02 MB
> 2017-03-28 01:53:15,024 INFO org.apache.hadoop.hbase.regionserver.HRegion:
> writing data to region
> mytable,6A1066441DF045C6960D836EA5B82B16\x00\x00\x00\x00\x00\x00\x00\x19\x00\x06P\xC6,1490340809164.e9c7ee2a526aa7b696545dccb3f3feda.
> with WAL disabled. Data may be lost in the event of a crash.
> 2017-03-28 01:53:16,002 INFO
> org.apache.hadoop.hbase.regionserver.wal.FSHLog: Rolled WAL
> /hbase/WALs/ip-10-1-1-107,60020,1490588175688/ip-10-1-1-107%2C60020%2C1490588175688.null1.1490665962353
> with entries=107241, filesize=122.08 MB; new WAL
> /hbase/WALs/ip-10-1-1-107,60020,1490588175688/ip-10-1-1-107%2C60020%2C1490588175688.null1.1490665995822
> 2017-03-28 01:53:16,026 INFO
> org.apache.hadoop.hbase.regionserver.DefaultStoreFlusher: Flushed,
> sequenceid=14159831, memsize=128.0 M, hasBloomFilter=true, into tmp file
> hdfs://nameservice1/hbase/data/default/mytable/e9c7ee2a526aa7b696545dccb3f3feda/.tmp/225732a8f58842ddb10008e1dc6a872c
> 2017-03-28 01:53:16,288 INFO org.apache.hadoop.hbase.regionserver.HStore:
> Added
> hdfs://nameservice1/hbase/data/default/mytable/e9c7ee2a526aa7b696545dccb3f3feda/t/225732a8f58842ddb10008e1dc6a872c,
> entries=546485, sequenceid=14159831, *filesize=7.5 M*
> 2017-03-28 01:53:16,290 INFO org.apache.hadoop.hbase.regionserver.HRegion:
> Finished memstore flush of ~128.02 MB/134235854, currentsize=428.55
> KB/438840 for region
> mytable,6A1066441DF045C6960D836EA5B82B16\x00\x00\x00\x00\x00\x00\x00\x19\x00\x06P\xC6,1490340809164.e9c7ee2a526aa7b696545dccb3f3feda.
> in 1422ms, sequenceid=14159831, compaction requested=true
> 2017-03-28 01:53:16,334 INFO
> org.apache.hadoop.hbase.regionserver.wal.FSHLog: Slow sync cost: 283 ms,
> current pipeline:
> [DatanodeInfoWithStorage[10.1.1.107:50010,DS-07fc4de8-62bd-4aa3-8e18-c2ce56381113,DISK],
> DatanodeInfoWithStorage[10.1.1.104:50010,DS-c9749aaf-7213-41af-87ec-2214cf166830,DISK],
> DatanodeInfoWithStorage[10.1.1.102:50010
> ,DS-59b137d0-30cb-404f-8b7f-40ee8c18f72b,DISK]]
> 2017-03-28 01:53:16,386 INFO
> org.apache.hadoop.hbase.regionserver.wal.FSHLog: Rolled WAL
> /hbase/WALs/ip-10-1-1-107,60020,1490588175688/ip-10-1-1-107%2C60020%2C1490588175688.null0.1490665962438
> with entries=75434, filesize=90.34 MB; new WAL
> /hbase/WALs/ip-10-1-1-107,60020,1490588175688/ip-10-1-1-107%2C60020%2C1490588175688.null0.1490665996002
> 2017-03-28 01:53:16,387 INFO
> org.apache.hadoop.hbase.regionserver.wal.FSHLog: Archiving
> hdfs://nameservice1/hbase/WALs/ip-10-1-1-107,60020,1490588175688/ip-10-1-1-107%2C60020%2C1490588175688.null0.1490664817220
> to
> hdfs://nameservice1/hbase/oldWALs/ip-10-1-1-107%2C60020%2C1490588175688.null0.1490664817220
> 2017-03-28 01:53:16,435 INFO
> org.apache.hadoop.hbase.regionserver.wal.FSHLog: Archiving
> hdfs://nameservice1/hbase/WALs/ip-10-1-1-107,60020,1490588175688/ip-10-1-1-107%2C60020%2C1490588175688.null0.1490664854680
> to
> hdfs://nameservice1/hbase/oldWALs/ip-10-1-1-107%2C60020%2C1490588175688.null0.1490664854680
> 2017-03-28 01:53:16,485 INFO
> org.apache.hadoop.hbase.regionserver.wal.FSHLog: Archiving
> hdfs://nameservice1/hbase/WALs/ip-10-1-1-107,60020,1490588175688/ip-10-1-1-107%2C60020%2C1490588175688.null0.1490664886354
> to
> hdfs://nameservice1/hbase/oldWALs/ip-10-1-1-107%2C60020%2C1490588175688.null0.1490664886354
> 2017-03-28 01:53:16,584 INFO org.apache.hadoop.hbase.regionserver.HStore:
> Completed compaction of 10 file(s) in t of
> 

Re: [ANNOUNCE] - Welcome our new HBase committer Anastasia Braginsky

[Anoop John]

Congrats and Welcome Anastasia !

-Anoop-

On Mon, Mar 27, 2017 at 6:07 PM, ramkrishna vasudevan
 wrote:
> Hi All
>
> Welcome Anastasia Braginsky, one more female committer to HBase. She has
> been active now for a while with her Compacting memstore feature and she
> along with Eshcar have done lot of talks in various meetups and HBaseCon on
> their feature.
>
> Welcome onboard and looking forward to work with you Anastasia !!!
>
> Regards
> Ram

Re: Optimizations for a Read-only database

[Anoop John]

>From HBase server perspective we need restrict memstore size + block
cache size to be max 80%.  And memstore size alone can go down to 5%
if am not wrong.

We need to be careful when using G1 and giving this 80%.  The cache
will be mostly full as u said it will be read workload. Making ur
working size large.  The default value for Initial Heap Occupancy
Percentage (IHOP)  in G1 is 45%.   You can up this.  But having this
>80% am not sure whether really advisable.


-Anoop-

On Fri, Mar 17, 2017 at 11:50 PM, jeff saremi  wrote:
> I'll go through these recommendations, Kevin. Thanks a lot
>
> 
> From: Kevin O'Dell 
> Sent: Friday, March 17, 2017 10:55:49 AM
> To: user@hbase.apache.org
> Subject: Re: Optimizations for a Read-only database
>
> Hi Jeff,
>
>   You can definitely lower the memstore, the last time I looked there it
> had to be set to .1 at lowest it could go. I would not recommend disabling
> compactions ever, bad things will occur and it can end up impacting your
> read performance greatly.  I would recommend looking at the Intel G1GC
> 
> blog series to leverage really large chunks of block cache, and then using
> the remaining memory for off heap caching. You should make sure to turn on
> things like Snappy compression, FAST_DIFF for data block encoding, and with
> all the extra memory you will have available it might be worth using the
> ROW+COL bloom filters, though you should have very few underlying HFiles
> depending on how often you bulk load. I think short-circuit reads are on by
> default these days, but it will greatly speed up read performance if not
> already turned on. From an upfront design make sure you pre-split your
> tables so your first few bulk loads don't cause split and compaction
> pains.  Hope this helps!
>
> On Fri, Mar 17, 2017 at 1:32 PM, jeff saremi  wrote:
>
>> We're creating a readonly database and would like to know the recommended
>> optimizations we could do. We'd be loading data via direct write to HFiles.
>>
>> One thing i could immediately think of is to eliminate the memory for
>> Memstore. What is the minimum that we could get away with?
>>
>> How about disabling some regular operations to save CPU time. I think
>> Compaction is one of those we'd like to stop.
>>
>> thanks
>>
>> Jeff
>>
>
>
>
> --
> Kevin O'Dell
> Field Engineer
> 850-496-1298 | ke...@rocana.com
> @kevinrodell
> 

Re: hbase CMS gc pause serious program

[Anoop John]

96 GB of heap size? !!

What kind of work load is this?

-Anoop-

On Fri, Mar 10, 2017 at 10:54 AM, gehaijiang  wrote:
> CMS GC  program:
>
> 2017-03-10T10:15:25.741+0800: 4555916.378: [GC2017-03-10T10:15:25.741+0800:
> 4555916.378: [ParNew: 3067136K->340736K(3067136K), 2.0813220 secs]
> 79945091K->77675170K(100322560K), 2.0816590 secs] [Times: user=4.07
> sys=0.35, real=2.09 secs]
> 2017-03-10T10:15:29.524+0800: 4555920.160: [GC2017-03-10T10:15:29.524+0800:
> 4555920.160: [ParNew: 3067133K->340736K(3067136K), 2.0586980 secs]
> 80328431K->78058138K(100322560K), 2.0590280 secs] [Times: user=3.94
> sys=0.34, real=2.05 secs]
> 2017-03-10T10:15:32.911+0800: 4555923.547: [CMS-concurrent-sweep:
> 1441.773/1618.869 secs] [Times: user=2518.60 sys=59.25, real=1618.62 secs]
> 2017-03-10T10:15:32.911+0800: 4555923.547: [CMS-concurrent-reset-start]
>
> 2017-03-10T10:15:33.126+0800: 4555923.762: [CMS-concurrent-reset:
> 0.215/0.215 secs] [Times: user=1.23 sys=0.08, real=0.22 secs]
> 2017-03-10T10:15:33.236+0800: 4555923.873: [GC2017-03-10T10:15:33.237+0800:
> 4555923.873: [ParNew: 3067011K->340736K(3067136K), 2.4140270 secs]
> 80615855K->78315999K(100322560K), 2.4144230 secs] [Times: user=4.63
> sys=0.36, real=2.41 secs]
> 2017-03-10T10:15:35.655+0800: 4555926.292: [GC [1 CMS-initial-mark:
> 77975263K(97255424K)] 78316286K(100322560K), 0.0149650 secs] [Times:
> user=0.01 sys=0.00, real=0.01 secs]
> 2017-03-10T10:15:35.671+0800: 4555926.307: [CMS-concurrent-mark-start]
> 2017-03-10T10:15:36.098+0800: 4555926.734: [CMS-concurrent-mark: 0.427/0.427
> secs] [Times: user=5.72 sys=0.05, real=0.43 secs]
> 2017-03-10T10:15:36.098+0800: 4555926.734: [CMS-concurrent-preclean-start]
> 2017-03-10T10:15:36.291+0800: 4555926.928: [CMS-concurrent-preclean:
> 0.192/0.193 secs] [Times: user=0.80 sys=0.03, real=0.19 secs]
> 2017-03-10T10:15:36.291+0800: 4555926.928:
> [CMS-concurrent-abortable-preclean-start]
> 2017-03-10T10:15:37.378+0800: 4555928.014: [GC2017-03-10T10:15:37.378+0800:
> 4555928.014: [ParNew: 3067083K->340736K(3067136K), 2.6221190 secs]
> 81042347K->78771078K(100322560K), 2.6224970 secs] [Times: user=4.79
> sys=0.48, real=2.62 secs]
> 2017-03-10T10:15:41.012+0800: 4555931.648:
> [CMS-concurrent-abortable-preclean: 2.083/4.721 secs] [Times: user=13.51
> sys=0.87, real=4.72 secs]
> 2017-03-10T10:15:41.015+0800: 4555931.652: [GC[YG occupancy: 2011637 K
> (3067136 K)]2017-03-10T10:15:41.016+0800: 4555931.652:
> [GC2017-03-10T10:15:41.016+0800: 4555931.652: [ParNew:
> 2011637K->340736K(3067136K), 2.0773980 secs]
> 80441979K->79117650K(100322560K), 2.0777380 secs] [Times: user=4.09
> sys=0.38, real=2.07 secs]
>
>
> regionserver JVM config:
>
> export HBASE_REGIONSERVER_OPTS="$HBASE_REGIONSERVER_OPTS -XX:PermSize=256m
> -XX:MaxPermSize=256m -Xms96G -Xmx96G"
> export HBASE_OPTS="$HBASE_OPTS -Djava.net.preferIPv4Stack=true
> -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=60
> -XX:+CMSParallelRemarkEnabled -XX:+CMSConcurrentMTEnabled
> -XX:ParallelGCThreads=40 -XX:+DisableExplicitGC -XX:+PrintGCDetails
> -XX:+PrintGCDateStamps -verbose:gc
> -XX:+UseCMSCompactAtFullCollection -XX:CMSFullGCsBeforeCompaction=1
> -XX:+CMSScavengeBeforeRemark -XX:+HeapDumpOnOutOfMemoryError
>
>
> attachment：hdfs-site.xml

Re: Custom Retention of data based on Rowkey

[Anoop John]

>From 0.98.9 version onward, there a per cell TTL feature available.
(See HBASE-10560)..  TTL can be controlled even per cell level.

It might eat up more space as the TTL is stored with every cell.  What
u want is same TTL for one group and another for another.  The major
concern is u need "dynamically updatable" TTL..  Per cell TTL can not
do this.

So u might need a custom compaction policy plugged in.. Might be like
u need to decorate the HBase compaction policy using CPs.  Also to
remember that u will need some CP work in read (Scan/Get) to make sure
TTL expired data is not retrieved back (Compaction might be not
happened yet but data is TTL expired. Actually at HBase level u dont
give any TTL)

-Anoop-

On Thu, Mar 9, 2017 at 4:34 PM, Gaurav Agarwal  wrote:
> Hi,
>
> Looking deeper, I found that RegionObserver interface provides general hooks 
> to intercept the pre-compaction scanner. That should suffice for our purpose!
>
> In any case, if there are any suggestions/guidelines, it will be much 
> appreciated.
>
>
>
> From: Gaurav Agarwal 
> Date: Thursday, 9 March 2017 at 2:08 PM
> To: 
> Cc: Kshitij Gupta , Mukul Gupta 
> Subject: Custom Retention of data based on Rowkey
>
>
>
> Hi All,
>
>
>
> We have an application that stores information on multiple 
> users/customers/tenants in a common table. Each tenant has a unique id which 
> we encode in the row key of the records that are stored in the table.
>
>
>
> We want to apply custom (and dynamically updatable) data retention policies 
> for each tenant.  What would be a reasonable way to achieve that?
>
>
>
> Searching through forums, I came across this link that suggests to either 
> write an external process to retrieve and delete cells based on the retention 
> policy or write a custom compaction policy:
>
> https://community.hortonworks.com/questions/14883/best-way-to-achieve-custom-retention-of-some-rows.html
>
>
>
> We felt that writing an external scanner for managing retention would be 
> simpler but very inefficient as it would require getting the entire data set 
> out of the hbase server and then issuing delete calls back to it.
>
>
>
> Does any one know if there has been any recent progress on this aspect of 
> data retention in hbase?
>
>
>
> Additionally, if I go the route of writing my own custom compaction policy, 
> what would be the best place to start? Maybe I could copy/extend the 
> “default” hbase compaction policy and enhance it to look at rowkey inside 
> every Cell to make a call if the cell needs to be deleted?
>
>
>
> --
>
> cheers,
>
> gaurav
>
>
>
>
>

Re: Hbase Locality/table balancing

[Anoop John]

There is one config  "hbase.master.loadbalance.bytable"  using which
user can say whether the balancing has to be done per table or
globally balance.  I believe u r using StochasticLoadBalancer only
which is the default.

The move calls are affecting ur locality?

-Anoop-

On Wed, Mar 8, 2017 at 1:05 AM, Pradheep Shanmugam
 wrote:
> Hi,
>
> I have a Hbase(1.1.2) cluster with 8 region servers in AWS..I bring down the 
> cluster everyday and bring it up again next day.
>
> Every day the locality keeps decreasing(95, 80, 70) and does not go up even 
> after doing major compaction of all the tables. Why is it so?
> Also the regions of a particular table are not gettting equally distrubuted 
> after the cluster is up..I have to run the hbase move commands to move the 
> table regions around..but the locality does not go up even after this..
> But the overall regions are distributed equally across the region servers. I 
> guess this is done by the balancer.
>
> Thanks,
> Pradheep

Re: On HBase Read Replicas

[Anoop John]

Thanks Enis.. I was not knowing the way of setting replica id
specifically..  So what will happen if that said replica is down at
the read time?  Will that go to another replica?

-Anoop-

On Sat, Feb 18, 2017 at 3:34 AM, Enis Söztutar <enis@gmail.com> wrote:
> You can do gets using two different "modes":
>  - Do a read with backup RPCs. In case, the algorithm that I have above
> will be used. 1 RPC to primary, and 2 more RPCs after primary timeouts.
>  - Do a read to a single replica. In this case, there is only 1 RPC that
> will happen to that given replica.
>
> Enis
>
> On Fri, Feb 17, 2017 at 12:03 PM, jeff saremi <jeffsar...@hotmail.com>
> wrote:
>
>> Enis
>>
>> Thanks for taking the time to reply
>>
>> So i thought that a read request is sent to all Replicas regardless. If we
>> have the option of Sending to one, analyzing response, and then sending to
>> another, this bodes well with our scenarios.
>>
>> Please confirm
>>
>> thanks
>>
>> 
>> From: Enis Söztutar <enis@gmail.com>
>> Sent: Friday, February 17, 2017 11:38:42 AM
>> To: hbase-user
>> Subject: Re: On HBase Read Replicas
>>
>> You can use read-replicas to distribute the read-load if you are fine with
>> stale reads. The read replicas normally have a "backup rpc" path, which
>> implements a logic like this:
>>  - Send the RPC to the primary replica
>>  - if no response for 100ms (or configured timeout), send RPCs to the other
>> replicas
>>  - return the first non-exception response.
>>
>> However, there is also another feature for read replicas, where you can
>> indicate which exact replica_id you want to read from when you are doing a
>> get. If you do this:
>> Get get = new Get(row);
>> get.setReplicaId(2);
>>
>> the Get RPC will only go to the replica_id=2. Note that if you have region
>> replication = 3, then you will have regions with replica ids: {0, 1, 2}
>> where replica_id=0 is the primary.
>>
>> So you can do load-balancing with a get.setReplicaId(random() %
>> num_replicas) kind of pattern.
>>
>> Enis
>>
>>
>>
>> On Thu, Feb 16, 2017 at 9:41 AM, Anoop John <anoop.hb...@gmail.com> wrote:
>>
>> > Never saw this kind of discussion.
>> >
>> > -Anoop-
>> >
>> > On Thu, Feb 16, 2017 at 10:13 PM, jeff saremi <jeffsar...@hotmail.com>
>> > wrote:
>> > > Thanks Anoop.
>> > >
>> > > Understood.
>> > >
>> > > Have there been enhancement requests or discussions on load balancing
>> by
>> > providing additional replicas in the past? Has anyone else come up with
>> > anything on this?
>> > > thanks
>> > >
>> > > 
>> > > From: Anoop John <anoop.hb...@gmail.com>
>> > > Sent: Thursday, February 16, 2017 2:35:48 AM
>> > > To: user@hbase.apache.org
>> > > Subject: Re: On HBase Read Replicas
>> > >
>> > > The region replica feature came in so as to reduce the MTTR and so
>> > > increase the data availability.  When the master region containing RS
>> > > dies, the clients can read from the secondary regions.  But to keep
>> > > one thing in mind that this data from secondary regions will be bit
>> > > out of sync as the replica is eventual consistent.   Because of this
>> > > said reason,  change client so as to share the load across diff RSs
>> > > might be tough.
>> > >
>> > > -Anoop-
>> > >
>> > > On Sun, Feb 12, 2017 at 8:13 AM, jeff saremi <jeffsar...@hotmail.com>
>> > wrote:
>> > >> Yes indeed. thank you very much Ted
>> > >>
>> > >> 
>> > >> From: Ted Yu <yuzhih...@gmail.com>
>> > >> Sent: Saturday, February 11, 2017 3:40:50 PM
>> > >> To: user@hbase.apache.org
>> > >> Subject: Re: On HBase Read Replicas
>> > >>
>> > >> Please take a look at the design doc attached to
>> > >> https://issues.apache.org/jira/browse/HBASE-10070.
>> > >>
>> > >> Your first question would be answered by that document.
>> > >>
>> > >> Cheers
>> > >>
>> > >> On Sat, Feb 11, 2017 at 2:06 PM, jeff saremi <jeffsar...@hotmail.com>
>> > wrote:
>> > >>
>> > >>> The first time I heard replicas in HBase the following thought
>> > immediately
>> > >>> came to my mind:
>> > >>> To alleviate the load in read-heavy clusters, one could assign Region
>> > >>> servers to be replicas of others so that the load is distributed and
>> > there
>> > >>> is less pressure on the main RS.
>> > >>>
>> > >>> Just 2 days ago a colleague quoted a paragraph from HBase manual that
>> > >>> contradicted this completely. Apparently, the replicas do not help
>> > with the
>> > >>> load but they actually contribute to more traffic on the network and
>> > on the
>> > >>> underlying file system
>> > >>>
>> > >>> Would someone be able to give us some insight on why anyone would
>> want
>> > >>> replicas?
>> > >>>
>> > >>> And also could one easily change this behavior in the HBase native
>> Java
>> > >>> client to support what I had been imagining as the concept for
>> > replicas?
>> > >>>
>> > >>>
>> > >>> thanks
>> > >>>
>> >
>>

Re: On HBase Read Replicas

[Anoop John]

Never saw this kind of discussion.

-Anoop-

On Thu, Feb 16, 2017 at 10:13 PM, jeff saremi <jeffsar...@hotmail.com> wrote:
> Thanks Anoop.
>
> Understood.
>
> Have there been enhancement requests or discussions on load balancing by 
> providing additional replicas in the past? Has anyone else come up with 
> anything on this?
> thanks
>
> ____
> From: Anoop John <anoop.hb...@gmail.com>
> Sent: Thursday, February 16, 2017 2:35:48 AM
> To: user@hbase.apache.org
> Subject: Re: On HBase Read Replicas
>
> The region replica feature came in so as to reduce the MTTR and so
> increase the data availability.  When the master region containing RS
> dies, the clients can read from the secondary regions.  But to keep
> one thing in mind that this data from secondary regions will be bit
> out of sync as the replica is eventual consistent.   Because of this
> said reason,  change client so as to share the load across diff RSs
> might be tough.
>
> -Anoop-
>
> On Sun, Feb 12, 2017 at 8:13 AM, jeff saremi <jeffsar...@hotmail.com> wrote:
>> Yes indeed. thank you very much Ted
>>
>> 
>> From: Ted Yu <yuzhih...@gmail.com>
>> Sent: Saturday, February 11, 2017 3:40:50 PM
>> To: user@hbase.apache.org
>> Subject: Re: On HBase Read Replicas
>>
>> Please take a look at the design doc attached to
>> https://issues.apache.org/jira/browse/HBASE-10070.
>>
>> Your first question would be answered by that document.
>>
>> Cheers
>>
>> On Sat, Feb 11, 2017 at 2:06 PM, jeff saremi <jeffsar...@hotmail.com> wrote:
>>
>>> The first time I heard replicas in HBase the following thought immediately
>>> came to my mind:
>>> To alleviate the load in read-heavy clusters, one could assign Region
>>> servers to be replicas of others so that the load is distributed and there
>>> is less pressure on the main RS.
>>>
>>> Just 2 days ago a colleague quoted a paragraph from HBase manual that
>>> contradicted this completely. Apparently, the replicas do not help with the
>>> load but they actually contribute to more traffic on the network and on the
>>> underlying file system
>>>
>>> Would someone be able to give us some insight on why anyone would want
>>> replicas?
>>>
>>> And also could one easily change this behavior in the HBase native Java
>>> client to support what I had been imagining as the concept for replicas?
>>>
>>>
>>> thanks
>>>

Re: How to avoid update blocking

[Anoop John]

The blocking happens in write path only.  While reading there is no
blocking as such.  Are you speaking abt throttling?

In write path the blocking comes in because of 2 reasons

1.  Per region u have 128 MB of flush size. Means when this region
reaches this size, a flush is initiated. But that happens in
background. Still cells getting written to this region. But there is a
blocking multiplier which is 2 in ur case.  Means before actual flush
completion, if the region size reaches  2 * 128 MB,  that region will
not allow to take any writes.  But here we will throw Exception back
to client so as to make it retry.
2. Global memstore size reaches the max.   U have this as 40%.  Means
40% of ur xmx for RS process.  How much is this?  And how many regions
u have?  U allow every region to be 256 MB max.  256 * No of regions
is what u ideally try to allow.  But ur global memstore size may be
lesser than this.  So once this happens, the write are blocked.  Not
throwing back any exceptions but the writer handlers at RS side are
blocked.. U can see these in logs also.. U referring to this  only?

The IO on ur cluster decides the flush speed.  Also there is config to
specify the number of flusher threads.  Same way u can control the
number of threads for compactions (Minor).   When compaction process i
lagging so behind, there might reach a situation that there are so
many HFiles under a store and in such cases, the flushers has to wait
for the compactions to get completed..  Pls see all these thread
counts and notice how big ur flush Q and compaction Q grows..

-Anoop-



On Sun, Feb 12, 2017 at 2:38 AM, Ted Yu  wrote:
> Which hbase release are you using ?
>
> You have given estimate on write load. How about read load ?
> Do you observe blocking / slow down on write path ?
>
> Consider increasing memstore, decreasing block cache if read load is light.
> I assume the 4 nodes with 125 GB Ram have HDD (instead of SSD). If you have
> SSD, you can use SSD for WAL - see HBASE-12848
>
> bq. its on weekend when load is low.
>
> This should have low impact on write load and is good practice.
>
> Cheers
>
> On Mon, Feb 6, 2017 at 9:15 PM, Manjeet Singh 
> wrote:
>
>> How to avoid read/write update blocking in hbase and on what condition
>> Hbase block read/write?
>>
>>
>>
>> what should the frequency of Major compaction ( I have daily data insert on
>> average is 18 GB and 48 GB max, its with SNAPPY and FAST_DIFF)
>>
>>  a. should it be on daily basis?
>>
>>  b. should it be in weekends?
>>
>>  c. does Major compaction on daily basis will avoid blocking?
>>
>>
>> What is the calculation? Right now its on weekend when load is low.
>>
>> we have :
>> total 7 (3+4) data nodes
>> 3 80GB Ram 1.8TB HDD
>> 4 125 GB Ram HDD 6 TB
>>
>> below are few important configurations:
>>
>> hbase.regionserver.global.memstore.lowerLimit = .38
>> hbase.regionserver.global.memstore.upperLimit = .4
>> hbase.hregion.memstore.block.multiplier=2
>> hbase.hregion.majorcompaction=0 ( scheduled on weekend)
>> hbase.hregion.memstore.flush.size 128 MB
>> hbase.hregion.preclose.flush.size 5 MB
>> hbase.hstore.compactionThreshold 3
>>
>> --
>> luv all
>>

Re: On HBase Read Replicas

[Anoop John]

The region replica feature came in so as to reduce the MTTR and so
increase the data availability.  When the master region containing RS
dies, the clients can read from the secondary regions.  But to keep
one thing in mind that this data from secondary regions will be bit
out of sync as the replica is eventual consistent.   Because of this
said reason,  change client so as to share the load across diff RSs
might be tough.

-Anoop-

On Sun, Feb 12, 2017 at 8:13 AM, jeff saremi  wrote:
> Yes indeed. thank you very much Ted
>
> 
> From: Ted Yu 
> Sent: Saturday, February 11, 2017 3:40:50 PM
> To: user@hbase.apache.org
> Subject: Re: On HBase Read Replicas
>
> Please take a look at the design doc attached to
> https://issues.apache.org/jira/browse/HBASE-10070.
>
> Your first question would be answered by that document.
>
> Cheers
>
> On Sat, Feb 11, 2017 at 2:06 PM, jeff saremi  wrote:
>
>> The first time I heard replicas in HBase the following thought immediately
>> came to my mind:
>> To alleviate the load in read-heavy clusters, one could assign Region
>> servers to be replicas of others so that the load is distributed and there
>> is less pressure on the main RS.
>>
>> Just 2 days ago a colleague quoted a paragraph from HBase manual that
>> contradicted this completely. Apparently, the replicas do not help with the
>> load but they actually contribute to more traffic on the network and on the
>> underlying file system
>>
>> Would someone be able to give us some insight on why anyone would want
>> replicas?
>>
>> And also could one easily change this behavior in the HBase native Java
>> client to support what I had been imagining as the concept for replicas?
>>
>>
>> thanks
>>

Re: Question on WALEdit

[Anoop John]

Attributes added to Mutation (via setAttribute) will NOT get
persistent in cells..  As Ted suggested, tags is the way..   There is
no way to pass Tags from client as of now.. I believe u have eventId
thing in server side only.. So  u can add Tags to cells at server side
only  (via some CP hooks).  Tags getting persisted in HFiles, WAL and
getting replicated also.

-Anoop-

On Mon, Jan 30, 2017 at 8:24 AM, Sreeram  wrote:
> Thank you very much Ted.  I understand fetching the tags will fetch the
> associated attributes for a mutation. Will try out the same.
>
> Regards,
> Sreeram
>
> On 29 Jan 2017 00:37, "Ted Yu"  wrote:
>
> In CellUtil, there is the following method:
>
>   public static Tag getTag(Cell cell, byte type){
>
>
> In MobUtils, you can find sample usage:
>
>   public static Tag getTableNameTag(Cell cell) {
>
> if (cell.getTagsLength() > 0) {
>
>   return CellUtil.getTag(cell, TagType.MOB_TABLE_NAME_TAG_TYPE);
>
> FYI
>
> On Sat, Jan 28, 2017 at 8:29 AM, Ted Yu  wrote:
>
>> I haven't found the API you were looking for.
>>
>> Which release of hbase are you using ?
>> I assume it supports tags.
>>
>> If you use tag to pass event-id, you can retrieve thru this method of
>> WALEdit:
>>
>>   public ArrayList getCells() {
>>
>> From Cell, there're 3 methods for retrieving tag starting with:
>>
>>   byte[] getTagsArray();
>>
>> Cheers
>>
>> On Sat, Jan 28, 2017 at 4:23 AM, Sreeram  wrote:
>>
>>> Hi,
>>>
>>> TL;DR:  In my use case I am setting attributes for Puts and Deletes using
>>> setAttribute(). I would like to know if it is possible to get the
>>> attributes that I had set from the WALEdit ?
>>>
>>> Here is my use case in detail: I have a replicated cluster A which gets
>>> replicated to cluster B. From cluster B, I would like to track the events
>>> as they get written to B. I set the event-id as an attribute to the
>>> mutation in cluster A.
>>>
>>> I will have a coProcessor in cluster B that gets invoked on postWALWrite,
>>> If I can retrieve the event-id from the WALEdit, I would be able to track
>>> those events that got replicated successfully in cluster B.
>>>
>>> I went through the WALEdit API and it is not obvious to me if it is
>>> possible to retrieve the attributes set on the row mutation.
>>>
>>> Kindly let me know your suggestions.
>>>
>>> Regards,
>>> Sreeram
>>>
>>
>>

Re: [ANNOUNCE] New HBase committer Guanghao Zhang

[Anoop John]

Congrats Guanghao..

-Anoop-


On Tue, Dec 20, 2016 at 4:09 PM, ramkrishna vasudevan
 wrote:
> Congratulations and welcome Guanghao.
>
> Regards
> Ram
>
> On Tue, Dec 20, 2016 at 4:07 PM, Guanghao Zhang  wrote:
>
>> Thanks all. Looking forward to work with you guys and keep contributing for
>> HBase. Thanks.
>>
>> 2016-12-20 16:48 GMT+08:00 Yu Li :
>>
>> > Congratulations and welcome Guanghao!
>> >
>> > Best Regards,
>> > Yu
>> >
>> > On 20 December 2016 at 12:59, 宾莉金 or binlijin 
>> wrote:
>> >
>> > > Congratulations and welcome!
>> > >
>> > > 2016-12-20 12:54 GMT+08:00 Nick Dimiduk :
>> > >
>> > > > Congratulations Guanghao and thank you for all your contributions!
>> > > >
>> > > > On Mon, Dec 19, 2016 at 5:37 PM Duo Zhang 
>> wrote:
>> > > >
>> > > > > On behalf of the Apache HBase PMC, I am pleased to announce that
>> > > Guanghao
>> > > > >
>> > > > > Zhang has accepted the PMC's invitation to become a committer on
>> the
>> > > > >
>> > > > > project. We appreciate all of Guanghao's generous contributions
>> thus
>> > > far
>> > > > >
>> > > > > and look forward to his continued involvement.
>> > > > >
>> > > > >
>> > > > >
>> > > > > Congratulations and welcome, Guanghao!
>> > > > >
>> > > > >
>> > > >
>> > >
>> > >
>> > >
>> > > --
>> > > *Best Regards,*
>> > >  lijin bin
>> > >
>> >
>>

Re: [ANNOUNCE] New HBase committer Lijin Bin

[Anoop John]

Congrats and welcome Binlijin.

-Anoop-

On Tue, Nov 29, 2016 at 3:18 PM, Duo Zhang  wrote:
> On behalf of the Apache HBase PMC, I am pleased to announce that Lijin
> Bin(binlijin) has accepted the PMC's invitation to become a committer on
> the project. We appreciate all of Lijin's generous contributions thus far
> and look forward to his continued involvement.
>
> Congratulations and welcome, Lijin!

Re: Use experience and performance data of offheap from Alibaba online cluster

[Anoop John]

Because of some compatibility issues, we decide that this will be done
in 2.0 only..  Ya as Andy said, it would be great to share the 1.x
backported patches.  Is it a mega patch at ur end?  Or issue by issue
patches?  Latter would be best.  Pls share patches in some place and a
list of issues backported. I can help with verifying the issues once
so as to make sure we dont miss any...

-Anoop-

On Sat, Nov 19, 2016 at 12:32 AM, Enis Söztutar <enis@gmail.com> wrote:
> Thanks for sharing this. Great work.
>
> I don't see any reason why we cannot backport to branch-1.
>
> Enis
>
> On Fri, Nov 18, 2016 at 9:37 AM, Andrew Purtell <andrew.purt...@gmail.com>
> wrote:
>
>> Yes, please, the patches will be useful to the community even if we decide
>> not to backport into an official 1.x release.
>>
>>
>> > On Nov 18, 2016, at 12:25 PM, Bryan Beaudreault <
>> bbeaudrea...@hubspot.com> wrote:
>> >
>> > Is the backported patch available anywhere? Not seeing it on the
>> referenced
>> > JIRA. If it ends up not getting officially backported to branch-1 due to
>> > 2.0 around the corner, some of us who build our own deploy may want to
>> > integrate into our builds. Thanks! These numbers look great
>> >
>> >> On Fri, Nov 18, 2016 at 12:20 PM Anoop John <anoop.hb...@gmail.com>
>> wrote:
>> >>
>> >> Hi Yu Li
>> >>   Good to see that the off heap work help you..  The perf
>> >> numbers looks great.  So this is a compare of on heap L1 cache vs off
>> heap
>> >> L2 cache(HBASE-11425 enabled).   So for 2.0 we should make L2 off heap
>> >> cache ON by default I believe.  Will raise a jira for that we can
>> discuss
>> >> under that.   Seems like L2 off heap cache for data blocks and L1 cache
>> for
>> >> index blocks seems a right choice.
>> >>
>> >> Thanks for the backport and the help in testing the feature..  You were
>> >> able to find some corner case bugs and helped community to fix them..
>> >> Thanks goes to ur whole team.
>> >>
>> >> -Anoop-
>> >>
>> >>
>> >>> On Fri, Nov 18, 2016 at 10:14 PM, Yu Li <car...@gmail.com> wrote:
>> >>>
>> >>> Sorry guys, let me retry the inline images:
>> >>>
>> >>> Performance w/o offheap:
>> >>>
>> >>>
>> >>> Performance w/ offheap:
>> >>>
>> >>>
>> >>> Peak Get QPS of one single RS during Singles' Day (11/11):
>> >>>
>> >>>
>> >>>
>> >>> And attach the files in case inline still not working:
>> >>>
>> >>> Performance_without_offheap.png
>> >>> <
>> >> https://drive.google.com/file/d/0B017Q40_F5uwbWEzUGktYVIya3JkcXVjRkFvVG
>> NtM0VxWC1n/view?usp=drive_web
>> >>>
>> >>>
>> >>> Performance_with_offheap.png
>> >>> <
>> >> https://drive.google.com/file/d/0B017Q40_F5uweGR2cnJEU0M1MWwtRFJ5YkxUeF
>> VrcUdPc2ww/view?usp=drive_web
>> >>>
>> >>>
>> >>> Peak_Get_QPS_of_Single_RS.png
>> >>> <
>> >> https://drive.google.com/file/d/0B017Q40_F5uwQ2FkR2k0ZmEtRVNGSFp5RUxHM3
>> F6bHpNYnJz/view?usp=drive_web
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> Best Regards,
>> >>> Yu
>> >>>
>> >>>> On 18 November 2016 at 19:29, Ted Yu <yuzhih...@gmail.com> wrote:
>> >>>>
>> >>>> Yu:
>> >>>> With positive results, more hbase users would be asking for the
>> backport
>> >>>> of offheap read path patches.
>> >>>>
>> >>>> Do you think you or your coworker has the bandwidth to publish
>> backport
>> >>>> for branch-1 ?
>> >>>>
>> >>>> Thanks
>> >>>>
>> >>>>> On Nov 18, 2016, at 12:11 AM, Yu Li <car...@gmail.com> wrote:
>> >>>>>
>> >>>>> Dear all,
>> >>>>>
>> >>>>> We have backported read path offheap (HBASE-11425) to our customized
>> >>>> hbase-1.1.2 (thanks @Anoop for the help/support) and run it online for
>> >> more
>> >>>> than a month, and would like to share our experience, for what it's
>> >> worth
&

Re: Use experience and performance data of offheap from Alibaba online cluster

[Anoop John]

Hi Yu Li
   Good to see that the off heap work help you..  The perf
numbers looks great.  So this is a compare of on heap L1 cache vs off heap
L2 cache(HBASE-11425 enabled).   So for 2.0 we should make L2 off heap
cache ON by default I believe.  Will raise a jira for that we can discuss
under that.   Seems like L2 off heap cache for data blocks and L1 cache for
index blocks seems a right choice.

Thanks for the backport and the help in testing the feature..  You were
able to find some corner case bugs and helped community to fix them..
Thanks goes to ur whole team.

-Anoop-


On Fri, Nov 18, 2016 at 10:14 PM, Yu Li  wrote:

> Sorry guys, let me retry the inline images:
>
> Performance w/o offheap:
>
> ​
> Performance w/ offheap:
>
> ​
> Peak Get QPS of one single RS during Singles' Day (11/11):
>
> ​
>
> And attach the files in case inline still not working:
> ​​​
>  Performance_without_offheap.png
> 
> ​​
>  Performance_with_offheap.png
> 
> ​​
>  Peak_Get_QPS_of_Single_RS.png
> 
> ​
>
>
> Best Regards,
> Yu
>
> On 18 November 2016 at 19:29, Ted Yu  wrote:
>
>> Yu:
>> With positive results, more hbase users would be asking for the backport
>> of offheap read path patches.
>>
>> Do you think you or your coworker has the bandwidth to publish backport
>> for branch-1 ?
>>
>> Thanks
>>
>> > On Nov 18, 2016, at 12:11 AM, Yu Li  wrote:
>> >
>> > Dear all,
>> >
>> > We have backported read path offheap (HBASE-11425) to our customized
>> hbase-1.1.2 (thanks @Anoop for the help/support) and run it online for more
>> than a month, and would like to share our experience, for what it's worth
>> (smile).
>> >
>> > Generally speaking, we gained a better and more stable
>> throughput/performance with offheap, and below are some details:
>> > 1. QPS become more stable with offheap
>> >
>> > Performance w/o offheap:
>> >
>> >
>> >
>> > Performance w/ offheap:
>> >
>> >
>> >
>> > These data come from our online A/B test cluster (with 450 physical
>> machines, and each with 256G memory + 64 core) with real world workloads,
>> it shows using offheap we could gain a more stable throughput as well as
>> better performance
>> >
>> > Not showing fully online data here because for online we published the
>> version with both offheap and NettyRpcServer together, so no standalone
>> comparison data for offheap
>> >
>> > 2. Full GC frequency and cost
>> >
>> > Average Full GC STW time reduce from 11s to 7s with offheap.
>> >
>> > 3. Young GC frequency and cost
>> >
>> > No performance degradation observed with offheap.
>> >
>> > 4. Peak throughput of one single RS
>> >
>> > On Singles Day (11/11), peak throughput of one single RS reached 100K,
>> among which 90K from Get. Plus internet in/out data we could know the
>> average result size of get request is ~1KB
>> >
>> >
>> >
>> > Offheap are used on all online machines (more than 1600 nodes) instead
>> of LruCache, so the above QPS is gained from offheap bucketcache, along
>> with NettyRpcServer(HBASE-15756).
>> >
>> > Just let us know if any comments. Thanks.
>> >
>> > Best Regards,
>> > Yu
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>>
>
>

Re: Retrive HBase row based on dynamic qualifier filters

[Anoop John]

Hi
   Ya the QualifierFilter is returning only the matched cells and skip
the remaining cells in the row.   I am not able to see any ready made
FIlter for ur usecase. U may have to write a custom one.

-Anoop-

On Tue, Nov 8, 2016 at 2:50 AM, Mukesh Jha  wrote:
> Hello HBase users,
>
> I have an hbase tables with timestamp based dynamic column qualifiers,
> where it can look like below* [1]*.
>
> I want to use a qualifier filter which basically checks for the presences
> of a timestamp say ts1, and if it's present it should return the whole
> hbase row.
>
> I looked into using QualifierFilter *[2]* but it returns only the matched
> rows and not the entire HBaseRow.
>
> What is the best way to deal with this scenario?
>
> Thanks for the help, you guys are super helpful.
>
> *[1] HbaseRow*
> rowKey1 => {
>   cf1 => {
>  ts1 => val1,
>  ts2 => val2,
>  ts6 => val6,
>  foo => Array[Byte],
>  bar => Array[Byte]
>   }
> }
>
> *[2]*
> https://hbase.apache.org/apidocs/org/apache/hadoop/hbase/filter/QualifierFilter.html
>
> --
> Thanks & Regards,
>
> *Mukesh Jha *

Re: How can I achieve HBase row level atomicity?

[Anoop John]

You can use increment API in Table then?  Within RS it will do a get
and then do + and put the new Cell.  All under rowlock so that
atomicity is guaranteed.

And when u call the API for the 1st time (No existing row) the
increment value as such (y) will get added.  I believe this will make
things easy for u. Pls try. Let us know any further doubts/issues.

-Anoop-

On Tue, Nov 8, 2016 at 5:55 AM, Manjeet Singh
<manjeet.chand...@gmail.com> wrote:
> Increment by y x+y
>
> On 7 Nov 2016 23:52, "Anoop John" <anoop.hb...@gmail.com> wrote:
>
>> So what is the update op doing?  Say the cur available value in table
>> is 'x' and incoming data from ETL is 'y'  what is the updated value u
>> try put?  Is it an increment by 'y"? Or some complex logic behind?
>>
>> -Anoop-
>>
>> On Mon, Nov 7, 2016 at 11:09 PM, Manjeet Singh
>> <manjeet.chand...@gmail.com> wrote:
>> > Yes Anoop you are right
>> > my input source is kafka pipeline we have 7 spark ETL jobs which are
>> > responsible for aggregation, and to get last value from Hbase and put the
>> > updated one.
>> > we set zero version so we always have only one copy in Hbase.
>> > Problem is if my 2 different ETL job having the same rowkey and one ETL
>> > process get the value and at same time second also get that value and
>> > update it now first ETL job will replace that updated value.
>> >
>> > it can happen in same ETL job too.
>> >
>> > Thanks
>> > Manjeet
>> >
>> > On Mon, Nov 7, 2016 at 7:07 PM, Anoop John <anoop.hb...@gmail.com>
>> wrote:
>> >
>> >> Seems u want to get an already present row and do some op and put the
>> >> updated value.  What is that op?  If u can explain those we can try
>> >> help u with ways (if available)  As such, the above kind of API do not
>> >> guarantee u any atomicity.
>> >>
>> >> -Anoop-
>> >>
>> >> On Fri, Nov 4, 2016 at 4:12 AM, Ted Yu <yuzhih...@gmail.com> wrote:
>> >> > bq. from some api its going for update (means get is performed)
>> >> >
>> >> > Update on hbase would correlate with Put or Delete (not sure what
>> 'get'
>> >> > above means).
>> >> >
>> >> > Looks like your concern is that two concurrent updates may overwrite
>> the
>> >> > data for same rowkey.
>> >> >
>> >> > Have you considered using:
>> >> >
>> >> >   public boolean checkAndPut(final byte [] row,
>> >> >
>> >> >   final byte [] family, final byte [] qualifier, final byte []
>> value,
>> >> >
>> >> >   final Put put)
>> >> >
>> >> > Cheers
>> >> >
>> >> > On Thu, Nov 3, 2016 at 10:57 AM, Manjeet Singh <
>> >> manjeet.chand...@gmail.com>
>> >> > wrote:
>> >> >
>> >> >> Hi Ted,
>> >> >>
>> >> >> code not required for this case and how MVCC can help?
>> >> >>
>> >> >> question is if I have record which come in hbase and from some api
>> its
>> >> >> going for update (means get is performed)
>> >> >> now other operation also performed get operation both update on same
>> >> rwokey
>> >> >> at end we will not have correct value.
>> >> >>
>> >> >> in seprate mail thread i asked same type of question regarding rowkey
>> >> lock,
>> >> >> but still i did't get correct anser
>> >> >>
>> >> >> Thanks
>> >> >> Manjeet
>> >> >>
>> >> >> On Wed, Nov 2, 2016 at 11:36 PM, Ted Yu <yuzhih...@gmail.com> wrote:
>> >> >>
>> >> >> > Were you including code in the image (which didn't come through) ?
>> >> >> >
>> >> >> > MultiVersionConcurrencyControl is involved in answering your
>> question.
>> >> >> >
>> >> >> > See http://hbase.apache.org/book.html#hregion.scans
>> >> >> >
>> >> >> > Cheers
>> >> >> >
>> >> >> > On Wed, Nov 2, 2016 at 10:57 AM, Manjeet Singh <
>> >> >> manjeet.chand...@gmail.com
>> >> >> > >
>> >> >> > wrote:
>> >> >> >
>> >> >> > > Hi All
>> >> >> > >
>> >> >> > > I have ETL process for inserting data into hbase for this I have
>> >> spark
>> >> >> > > jobs which are responsible for reading data from kafka topics.
>> so my
>> >> >> > > question is if I have some rowkey already exist in hbase and I
>> have
>> >> 3
>> >> >> > spark
>> >> >> > > job running and they all try to update on same rowkey how hbase
>> deal
>> >> >> for
>> >> >> > > atomicity?
>> >> >> > >
>> >> >> > > for more understanding i have 3 rowkey coming from 3 seprate
>> spark
>> >> job
>> >> >> > and
>> >> >> > > all trying to update same rowkey which is already exist in hbase
>> >> table.
>> >> >> > >
>> >> >> > > [image: Inline image 1]
>> >> >> > >
>> >> >> > > --
>> >> >> > > luv all
>> >> >> > >
>> >> >> >
>> >> >>
>> >> >>
>> >> >>
>> >> >> --
>> >> >> luv all
>> >> >>
>> >>
>> >
>> >
>> >
>> > --
>> > luv all
>>

Re: How can I achieve HBase row level atomicity?

[Anoop John]

So what is the update op doing?  Say the cur available value in table
is 'x' and incoming data from ETL is 'y'  what is the updated value u
try put?  Is it an increment by 'y"? Or some complex logic behind?

-Anoop-

On Mon, Nov 7, 2016 at 11:09 PM, Manjeet Singh
<manjeet.chand...@gmail.com> wrote:
> Yes Anoop you are right
> my input source is kafka pipeline we have 7 spark ETL jobs which are
> responsible for aggregation, and to get last value from Hbase and put the
> updated one.
> we set zero version so we always have only one copy in Hbase.
> Problem is if my 2 different ETL job having the same rowkey and one ETL
> process get the value and at same time second also get that value and
> update it now first ETL job will replace that updated value.
>
> it can happen in same ETL job too.
>
> Thanks
> Manjeet
>
> On Mon, Nov 7, 2016 at 7:07 PM, Anoop John <anoop.hb...@gmail.com> wrote:
>
>> Seems u want to get an already present row and do some op and put the
>> updated value.  What is that op?  If u can explain those we can try
>> help u with ways (if available)  As such, the above kind of API do not
>> guarantee u any atomicity.
>>
>> -Anoop-
>>
>> On Fri, Nov 4, 2016 at 4:12 AM, Ted Yu <yuzhih...@gmail.com> wrote:
>> > bq. from some api its going for update (means get is performed)
>> >
>> > Update on hbase would correlate with Put or Delete (not sure what 'get'
>> > above means).
>> >
>> > Looks like your concern is that two concurrent updates may overwrite the
>> > data for same rowkey.
>> >
>> > Have you considered using:
>> >
>> >   public boolean checkAndPut(final byte [] row,
>> >
>> >   final byte [] family, final byte [] qualifier, final byte [] value,
>> >
>> >   final Put put)
>> >
>> > Cheers
>> >
>> > On Thu, Nov 3, 2016 at 10:57 AM, Manjeet Singh <
>> manjeet.chand...@gmail.com>
>> > wrote:
>> >
>> >> Hi Ted,
>> >>
>> >> code not required for this case and how MVCC can help?
>> >>
>> >> question is if I have record which come in hbase and from some api its
>> >> going for update (means get is performed)
>> >> now other operation also performed get operation both update on same
>> rwokey
>> >> at end we will not have correct value.
>> >>
>> >> in seprate mail thread i asked same type of question regarding rowkey
>> lock,
>> >> but still i did't get correct anser
>> >>
>> >> Thanks
>> >> Manjeet
>> >>
>> >> On Wed, Nov 2, 2016 at 11:36 PM, Ted Yu <yuzhih...@gmail.com> wrote:
>> >>
>> >> > Were you including code in the image (which didn't come through) ?
>> >> >
>> >> > MultiVersionConcurrencyControl is involved in answering your question.
>> >> >
>> >> > See http://hbase.apache.org/book.html#hregion.scans
>> >> >
>> >> > Cheers
>> >> >
>> >> > On Wed, Nov 2, 2016 at 10:57 AM, Manjeet Singh <
>> >> manjeet.chand...@gmail.com
>> >> > >
>> >> > wrote:
>> >> >
>> >> > > Hi All
>> >> > >
>> >> > > I have ETL process for inserting data into hbase for this I have
>> spark
>> >> > > jobs which are responsible for reading data from kafka topics. so my
>> >> > > question is if I have some rowkey already exist in hbase and I have
>> 3
>> >> > spark
>> >> > > job running and they all try to update on same rowkey how hbase deal
>> >> for
>> >> > > atomicity?
>> >> > >
>> >> > > for more understanding i have 3 rowkey coming from 3 seprate spark
>> job
>> >> > and
>> >> > > all trying to update same rowkey which is already exist in hbase
>> table.
>> >> > >
>> >> > > [image: Inline image 1]
>> >> > >
>> >> > > --
>> >> > > luv all
>> >> > >
>> >> >
>> >>
>> >>
>> >>
>> >> --
>> >> luv all
>> >>
>>
>
>
>
> --
> luv all

Re: How can I achieve HBase row level atomicity?

[Anoop John]

Seems u want to get an already present row and do some op and put the
updated value.  What is that op?  If u can explain those we can try
help u with ways (if available)  As such, the above kind of API do not
guarantee u any atomicity.

-Anoop-

On Fri, Nov 4, 2016 at 4:12 AM, Ted Yu  wrote:
> bq. from some api its going for update (means get is performed)
>
> Update on hbase would correlate with Put or Delete (not sure what 'get'
> above means).
>
> Looks like your concern is that two concurrent updates may overwrite the
> data for same rowkey.
>
> Have you considered using:
>
>   public boolean checkAndPut(final byte [] row,
>
>   final byte [] family, final byte [] qualifier, final byte [] value,
>
>   final Put put)
>
> Cheers
>
> On Thu, Nov 3, 2016 at 10:57 AM, Manjeet Singh 
> wrote:
>
>> Hi Ted,
>>
>> code not required for this case and how MVCC can help?
>>
>> question is if I have record which come in hbase and from some api its
>> going for update (means get is performed)
>> now other operation also performed get operation both update on same rwokey
>> at end we will not have correct value.
>>
>> in seprate mail thread i asked same type of question regarding rowkey lock,
>> but still i did't get correct anser
>>
>> Thanks
>> Manjeet
>>
>> On Wed, Nov 2, 2016 at 11:36 PM, Ted Yu  wrote:
>>
>> > Were you including code in the image (which didn't come through) ?
>> >
>> > MultiVersionConcurrencyControl is involved in answering your question.
>> >
>> > See http://hbase.apache.org/book.html#hregion.scans
>> >
>> > Cheers
>> >
>> > On Wed, Nov 2, 2016 at 10:57 AM, Manjeet Singh <
>> manjeet.chand...@gmail.com
>> > >
>> > wrote:
>> >
>> > > Hi All
>> > >
>> > > I have ETL process for inserting data into hbase for this I have spark
>> > > jobs which are responsible for reading data from kafka topics. so my
>> > > question is if I have some rowkey already exist in hbase and I have 3
>> > spark
>> > > job running and they all try to update on same rowkey how hbase deal
>> for
>> > > atomicity?
>> > >
>> > > for more understanding i have 3 rowkey coming from 3 seprate spark job
>> > and
>> > > all trying to update same rowkey which is already exist in hbase table.
>> > >
>> > > [image: Inline image 1]
>> > >
>> > > --
>> > > luv all
>> > >
>> >
>>
>>
>>
>> --
>> luv all
>>

Re: Re: What way to improve MTTR other than DLR(distributed log replay)

[Anoop John]

Agree with ur observation.. But DLR feature we wanted to get removed..
Because it is known to have issues..  Or else we need major work to
correct all these issues.

-Anoop-

On Tue, Oct 18, 2016 at 7:41 AM, Ted Yu  wrote:
> If you have a cluster, I suggest you turn on DLR and observe the effect
> where fewer than half the region servers are up after the crash.
> You would have first hand experience that way.
>
> On Mon, Oct 17, 2016 at 6:33 PM, allanwin  wrote:
>
>>
>>
>>
>> Yes, region replica is a good way to improve MTTR. Specially if one or two
>> servers are down, region replica can improve data availability. But for big
>> disaster like 1/3 or 1/2 region servers shutdown, I think DLR still useful
>> to bring regions online more quickly and with less IO usage.
>>
>>
>> Regards
>> Allan Yang
>>
>>
>>
>>
>>
>>
>> At 2016-10-17 21:01:16, "Ted Yu"  wrote:
>> >Here was the thread discussing DLR:
>> >
>> >http://search-hadoop.com/m/YGbbOxBK2n4ES12=Re+
>> DISCUSS+retiring+current+DLR+code
>> >
>> >> On Oct 17, 2016, at 4:15 AM, allanwin  wrote:
>> >>
>> >> Hi, All
>> >>  DLR can improve MTTR dramatically, but since it have many bugs like
>> HBASE-13567, HBASE-12743, HBASE-13535, HBASE-14729(any more I'don't know?),
>> it was proved unreliable, and has been deprecated almost in all branches
>> now.
>> >>
>> >>
>> >> My question is, is there any other way other than DLR to improve MTTR?
>> 'Cause If a big cluster crashes, It takes a long time to bring regions
>> online, not to mention it will create huge pressure on the IOs.
>> >>
>> >>
>> >> To tell the truth, I still want DLR back, if the community don't have
>> any plan to bring back DLR, I may want to figure out the problems in DLR
>> and make it working and reliable, Any suggests for that?
>> >>
>> >>
>> >> sincerely
>> >> Allan Yang
>>

Re: [ANNOUNCE] Stephen Yuan Jiang joins Apache HBase PMC

[Anoop John]

Congrats Stephen...

On Tue, Oct 18, 2016 at 10:20 AM, ramkrishna vasudevan
 wrote:
> Congrats Stephen!!
>
> On Tue, Oct 18, 2016 at 2:37 AM, Stack  wrote:
>
>> Wahoo!
>>
>> On Fri, Oct 14, 2016 at 11:27 AM, Enis Söztutar  wrote:
>>
>> > On behalf of the Apache HBase PMC, I am happy to announce that Stephen
>> has
>> > accepted our invitation to become a PMC member of the Apache HBase
>> project.
>> >
>> > Stephen has been working on HBase for a couple of years, and is already a
>> > committer for more than a year. Apart from his contributions in proc v2,
>> > hbck and other areas, he is also helping for the 2.0 release which is the
>> > most important milestone for the project this year.
>> >
>> > Welcome to the PMC Stephen,
>> > Enis
>> >
>>

Re: Modifying the existing column family and value in a Put

[Anoop John]

Say us more?  What u would like to do?  How is ur data and the table
structure.  And u try doing some aggregation or so? If u can say us
more abt what u r really trying to do, we can try to point you to the
best possible option wrt CPs.

-Anoop-

On Thu, Oct 13, 2016 at 8:39 PM, Nkechi Achara  wrote:
> Thanks!
>
> I'll extract and make another object
>
> On 13 Oct 2016 4:54 p.m., "Ted Yu"  wrote:
>
>> Looking at prePut / postPut methods, I think the answer is no.
>>
>> On Thu, Oct 13, 2016 at 7:46 AM, Nkechi Achara 
>> wrote:
>>
>> > Hi All,
>> >
>> > I am writing a transformation observer co-processor, and I would like to
>> > know if there is a way to modify the family, and the value without having
>> > to reconstruct the Put object again?
>> >
>> > Thanks,
>> >
>> > K
>> >
>>

Re: Hbase Coprocessor postPut not triggered for

[Anoop John]

Ya what API used for this update op u mention?

-Anoop-

On Mon, Oct 17, 2016 at 9:25 AM, Ted Yu  wrote:
> Which release of hbase are you using ?
>
> How did you determine that coprocessor's postPut method is not triggered
> for the update ? By additional logging ?
>
> Can you give code snippet around the update ?
>
> Thanks
>
> On Sun, Oct 16, 2016 at 8:48 PM, Begar, Veena  wrote:
>
>> Hello,
>>
>> We are using coprocessor feature by extending BaseRegionServer class.
>>
>> In postPut method, we send a message to kafka. Soon after creating a row,
>> we update it. We see it triggering postPut for creating row, but for the
>> subsequent update, it doesn't trigger postPut method:
>> void createAndUpdate(.. )
>> {
>> //create new row - Hbase put method -- coprocessor's postPut is triggered
>> //update the row - Hbase put method -- coprocessor's postPut method is not
>> triggered
>> .. . ...
>> }
>>
>> Please let me know why postPut method is not triggered for the 2nd put
>> call.
>>
>> Thanks,
>> Veena.
>>

Re: get first row of every region

[Anoop John]

You will have to set the stop row also (As Phil Yang suggested) .
Otherwise the scan might return row from next region (if this region
is empty)

-Anoop-

On Wed, Aug 3, 2016 at 12:23 PM, jinhong lu  wrote:
> Thanks a lot. I make a mistake in my code, it should be:
> sc.setStartRow(startRowkey),
> not:
>sc.setRowPrefixFilter(startRowkey);
> After fix that, it works.
>
>
> Thanks,
> lujinhong
>
>> 在 2016年8月2日，11:12，Phil Yang  写道：
>>
>> If you pre-split this table with some split points when you create the
>> table, the start key may not be the prefix of the first row key. So you
>> should use setStartRow(regionInfo.getStartKey())
>> and setStopRow(nextRegionInfo.getStartKey()) and setBatch(1), if the result
>> is still null, this region is indeed empty.
>>
>> Thanks,
>> Phil
>>
>>
>> 2016-08-02 10:21 GMT+08:00 jinhong lu :
>>
>>> Thanks. Here is my code, but in most case, r is null? why this happened?
>>>
>>>byte[] startRowkey =
>>> regionInfo.getStartKey();
>>>Scan sc = new Scan();
>>>sc.setBatch(1);
>>>sc.setRowPrefixFilter(startRowkey);
>>>try {
>>>scanner = table.getScanner(sc);
>>>r = scanner.next();
>>>scanner.close();
>>>}
>>> Thanks,
>>> lujinhong
>>>
 在 2016年8月1日，18:49，Ted Yu > yuzhih...@gmail.com>> 写道：

 .
>>>
>>>
>

Re: Hbase regionserver.MultiVersionConcurrencyControl Warning

[Anoop John]

Ya it comes with write workload.  Not like with concurrent reads.
Once the write is done (memstore write and WAL write), we mark that
MVCC operation corresponding to this as complete and wait for a global
read point to advance to atleast this point. (Every write op will have
a number corresponding to its mvcc op)
As the mvcc op begin, the write entries (which is having this number)
added to a Q.  Once write is over and we complete the mvcc op, we will
mark the entry corresponding to it (It is in Q now) as completed/done.
  So there are other entries also in the Q corresponding to other
parallel writes.   Now we will try to move the global read point at
least upto this mark.  Means we will have to make sue all the write
ops with lesser number than this to get completed.  Until then , there
is a wait.   So what is happening here is a later write op is
completed and that is still waiting for an older write to get
completed.   What sort of write ops u have?   Batch put and/or
increment/append?

-Anoop-


On Tue, Aug 9, 2016 at 6:45 PM, Sterfield  wrote:
> 2016-08-05 17:52 GMT+02:00 Bryan Beaudreault :
>
>> I'm also interested in an answer here. We see this from time to time in our
>> production HBase clusters (non-opentsdb). It seems to be related to
>> contention under heavy reads or heavy writes. But it's not clear what the
>> impact is here.
>>
>> On Fri, Aug 5, 2016 at 5:14 AM Sterfield  wrote:
>>
>> > Hi,
>> >
>> > I'm currently testing Hbase 1.2.1 + OpenTSDB. For that, I'm generating a
>> > high load of HTTP PUT on OpenTSDB, which then writes in Hbase. Currently,
>> > I'm able to feed 300k data point per seconds, and I'm trying to achieve
>> > higher speed.
>> >
>> > I have also activate JMX on both Master and Region servers in order to
>> grab
>> > a bunch of metrics from Hbase.
>> >
>> > At some point, I'm noticing a bunch of warnings in a region server log
>> file
>> > :
>> >
>> > 2016-08-04 05:12:06,537 WARN
>> >  [B.defaultRpcServer.handler=21,queue=3,port=16020]
>> > regionserver.MultiVersionConcurrencyControl: STUCK:
>> > MultiVersionConcurrencyControl{readPoint=7714719, writePoint=7714778}
>> > 2016-08-04 05:12:06,538 WARN
>> >  [B.defaultRpcServer.handler=19,queue=1,port=16020]
>> > regionserver.MultiVersionConcurrencyControl: STUCK:
>> > MultiVersionConcurrencyControl{readPoint=7714719, writePoint=7714778}
>> > 2016-08-04 05:12:06,538 WARN
>> >  [B.defaultRpcServer.handler=40,queue=4,port=16020]
>> > regionserver.MultiVersionConcurrencyControl: STUCK:
>> > MultiVersionConcurrencyControl{readPoint=7714719, writePoint=7714778}
>> > 2016-08-04 05:12:06,548 WARN
>> >  [B.defaultRpcServer.handler=0,queue=0,port=16020]
>> > regionserver.MultiVersionConcurrencyControl: STUCK:
>> > MultiVersionConcurrencyControl{readPoint=7714719, writePoint=7714778}
>> > 2016-08-04 05:12:06,549 WARN
>> >  [B.defaultRpcServer.handler=35,queue=5,port=16020]
>> > regionserver.MultiVersionConcurrencyControl: STUCK:
>> > MultiVersionConcurrencyControl{readPoint=7714719, writePoint=7714778}
>> > 2016-08-04 05:12:06,556 WARN
>> >  [B.defaultRpcServer.handler=7,queue=1,port=16020]
>> > regionserver.MultiVersionConcurrencyControl: STUCK:
>> > MultiVersionConcurrencyControl{readPoint=7714719, writePoint=7714779}
>> > 2016-08-04 05:12:06,558 WARN
>> >  [B.defaultRpcServer.handler=20,queue=2,port=16020]
>> > regionserver.MultiVersionConcurrencyControl: STUCK:
>> > MultiVersionConcurrencyControl{readPoint=7714719, writePoint=7714779}
>> > 2016-08-04 05:12:06,559 WARN
>> >  [B.defaultRpcServer.handler=14,queue=2,port=16020]
>> > regionserver.MultiVersionConcurrencyControl: STUCK:
>> > MultiVersionConcurrencyControl{readPoint=7714719, writePoint=7714779}
>> > 2016-08-04 05:12:06,609 WARN
>> >  [B.defaultRpcServer.handler=42,queue=0,port=16020]
>> > regionserver.MultiVersionConcurrencyControl: STUCK:
>> > MultiVersionConcurrencyControl{readPoint=7714719, writePoint=7714779}
>> > 2016-08-04 05:12:06,609 WARN
>> >  [B.defaultRpcServer.handler=32,queue=2,port=16020]
>> > regionserver.MultiVersionConcurrencyControl: STUCK:
>> > MultiVersionConcurrencyControl{readPoint=7714719, writePoint=7714779}
>> > 2016-08-04 05:12:06,619 WARN
>> >  [B.defaultRpcServer.handler=2,queue=2,port=16020]
>> > regionserver.MultiVersionConcurrencyControl: STUCK:
>> > MultiVersionConcurrencyControl{readPoint=7714719, writePoint=7714779}
>> > 2016-08-04 05:12:06,619 WARN
>> >  [B.defaultRpcServer.handler=48,queue=0,port=16020]
>> > regionserver.MultiVersionConcurrencyControl: STUCK:
>> > MultiVersionConcurrencyControl{readPoint=7714719, writePoint=7714779}
>> >
>> > Can someone shed some lights on this ? I've googling + searching the
>> > documentation, but found no clue on this subject.
>> >
>> > Is it a problem ? If so, how can I fix it ?
>> >
>> > Thanks a lot for your help,
>> >
>> > Guillaume
>> >
>>
>
> Hello,
>
> Sorry to bump this thread, but no one knows anything about this issue ?
>

Re: Writing visibility labels with HFileOutputFormat2

[Anoop John]

ibilityController will check this.
>> > >
>> > > We could I guess create multiple puts for cells in the same row with
>> > > different labels and use the setCellVisibility on each individual
>> > > put/cell, but will this create additional overhead?
>> > >
>> > > Cheers,
>> > >
>> > > Tom Ellis
>> > > Consultant Developer – Excelian
>> > > Data Lake | Financial Markets IT
>> > > LLOYDS BANK COMMERCIAL BANKING
>> > >
>> > >
>> > > E: tom.el...@lloydsbanking.com
>> > > Website: www.lloydsbankcommercial.com , , , Reduce printing. Lloyds
>> > > Banking Group is helping to build the low carbon economy.
>> > > Corporate Responsibility Report:
>> > > www.lloydsbankinggroup-cr.com/downloads
>> > >
>> > >
>> > > -Original Message-
>> > > From: ramkrishna vasudevan [mailto:ramkrishna.s.vasude...@gmail.com]
>> > > Sent: 15 June 2016 11:24
>> > > To: user@hbase.apache.org
>> > > Subject: Re: Writing visibility labels with HFileOutputFormat2
>> > >
>> > > -- This email has reached the Bank via an external source --
>> > >
>> > >
>> > > The visibility expression resolver tries to scan the labels table
>> > > and the user using the resolver should have the SYSTEM privileges.
>> > > Since the information that is getting accessed is sensitive
>> information.
>> > >
>> > > Suppose in your above case you have the client user added as a an
>> > > admin then when you scan the label table you should be able to  scan
>> it.
>> > >
>> > > Regards
>> > > Ram
>> > >
>> > > On Wed, Jun 15, 2016 at 3:09 PM, Ellis, Tom (Financial Markets IT) <
>> > > tom.el...@lloydsbanking.com.invalid> wrote:
>> > >
>> > > > Yeah, thanks for this Ram. Although in my testing I have found
>> > > > that a client user attempting to use the visibility expression
>> > > > resolver doesn't seem to have the ability to scan the hbase:labels
>> > > > table for the full list of labels and thus can't get the
>> > > > ordinals/tags to add to the cell. Does the client user attempting
>> > > > to use the VisibilityExpressionResolver have to have some special
>> permissions?
>> > > >
>> > > > Scan of hbase:labels by client user:
>> > > >
>> > > > hbase(main):003:0> scan 'hbase:labels'
>> > > > ROW COLUMN+CELL
>> > > >  \x00\x00\x00\x01   column=f:\x00,
>> > > > timestamp=1465216652662, value=system
>> > > > 1 row(s) in 0.0650 seconds
>> > > >
>> > > > Scan of hbase:labels by hbase user:
>> > > >
>> > > > hbase(main):001:0> scan 'hbase:labels'
>> > > > ROW COLUMN+CELL
>> > > >  \x00\x00\x00\x01   column=f:\x00,
>> > > > timestamp=1465216652662, value=system
>> > > >  \x00\x00\x00\x02   column=f:\x00,
>> > > > timestamp=1465216944935, value=protected
>> > > >  \x00\x00\x00\x02   column=f:hbase,
>> > > > timestamp=1465547138533, value=
>> > > >  \x00\x00\x00\x02   column=f:tom,
>> > > > timestamp=1465980236882, value=
>> > > >  \x00\x00\x00\x03   column=f:\x00,
>> > > > timestamp=1465500156667, value=testtesttest
>> > > >  \x00\x00\x00\x03   column=f:@hadoop,
>> > > > timestamp=1465980236967, value=
>> > > >  \x00\x00\x00\x03   column=f:hadoop,
>> > > > timestamp=1465547304610, value=
>> > > >  \x00\x00\x00\x03   column=f:hive,
>> > > > timestamp=1465501322616, value=
>> > > >  \x00\x00\x00\x04   column=f:\x00,
>> > > > timestamp=1465570719901, value=confidential
>> > > >  \x00\x00\x00\x05   column=f:\x00,
>> > > > timestamp=1465835047835, value=branch
>> > > >  \x00\x00\x00\x05   column=f:hdfs,
>> > > > timestamp=1465980237060, value=
>> > > >  \x00\x0

Re: Combining PageFilter gives inconsistent number of rows

[Anoop John]

Can u paste code how u make the FilterList?
I think you want to use MUST_PASS_ALL op in FL.

Ya the order of passing filters in FL can matter.

In your query case, you have to pass filter as  SCVF, KeyOnlyF, PageFilter.
The PageFilter must come at the end.

Any filter working scope is within one scanner at server end mean
within one Region scope.  So the Paging number guarantee is within one
region.  When u have many regions there is chance that from diff
regions u get diff numbers and so at client end.



-Anoop-

On Thu, Jun 9, 2016 at 7:36 PM, Ted Yu  wrote:
> Which version of hbase are you using ?
>
> Is it possible to come up with unit test that shows what you observed ?
>
> There is already coverage in existing unit tests, e.g. TestFilterList which
> you can use as template.
>
> Thanks
>
> On Thu, Jun 9, 2016 at 3:41 AM, Eko Susilo 
> wrote:
>
>> Hi All,
>>
>>
>>
>> I have problems when combining PageFilter with other filters. I am
>> combining SingleColumnValueFilter, KeyOnlyFilter, and PageFilter.
>>
>> Scan returns different total number of row when pageFilter is present; the
>> number of row without page filter is greater. Furthermore, the order I
>> specify these filter in filter list also changes the number of returned
>> rows.
>>
>>
>>
>> Anyone have had similar experience?
>>
>>
>>
>> Thanks
>>
>> --
>> Best Regards,
>> Eko Susilo
>>

Re: Writing visibility labels with HFileOutputFormat2

[Anoop John]

Thanks Ram.. Ya that seems the best way as CellCreator is public
exposed class. May be we should explain abt this in hbase book under
the Visibility labels area.  Good to know you have Visibility labels
based usecase. Let us know in case of any trouble.  Thanks.

-Anoop-

On Wed, Jun 8, 2016 at 1:43 PM, ramkrishna vasudevan
 wrote:
> Hi
>
> It can be done. See the class CellCreator which is Public facing interface.
> When you create your spark job to create the hadoop files that produces the
> HFileOutputformat2 data. While creating the KeyValues you can use the
> CellCreator to create your KeyValues and use the
> CellCreator.getVisibilityExpressionResolver() to map your String Visibility
> tags with the system generated ordinals.
>
> For eg, you can see how TextSortReducer works.  I think this should help
> you solve your problem. Let us know if you need further information.
>
> Regards
> Ram
>
> On Tue, Jun 7, 2016 at 3:58 PM, Ellis, Tom (Financial Markets IT) <
> tom.el...@lloydsbanking.com.invalid> wrote:
>
>> Hi Ram,
>>
>> We're attempting to do it programmatically so:
>>
>> The HFile is created by a Spark job using saveAsNewAPIHadoopFile, and
>> using ImmutableBytesWritable as the key (rowkey) with KeyValue as the
>> value, and using the HFilOutputFormat2 format.
>> This HFile is then loaded using HBase client's
>> LoadIncrementalHFiles.doBulkLoad
>>
>> Is there a way to do this programmatically without using the ImportTsv
>> tool? I was taking a look at VisibilityUtils.createVisibilityExpTags and
>> maybe being able to just create the Tags myself that way (although it's
>> obviously @InterfaceAudience.Private) but it seems to be able to use that
>> I'd need to know Label ordinality client side..
>>
>> Thanks for your help,
>>
>> Tom
>>
>> -Original Message-
>> From: ramkrishna vasudevan [mailto:ramkrishna.s.vasude...@gmail.com]
>> Sent: 07 June 2016 11:19
>> To: user@hbase.apache.org
>> Subject: Re: Writing visibility labels with HFileOutputFormat2
>>
>> -- This email has reached the Bank via an external source --
>>
>>
>> Hi Ellis
>>
>> How is the HFileOutputFormat2 files created?  Are you using the ImportTsv
>> tool?  If you are using the ImportTsv tool then yes there is a way to
>> specify visibility tags while loading from the ImportTsv tool and those
>> visibility tags are also bulk loaded as HFile.
>>
>> There is an attribute CELL_VISIBILITY_COLUMN_SPEC that can be used to
>> indicate that the data will have Visibility Tags and the tool will
>> automatically parse the specified field as Visibility Tag.
>>
>> In case you have access to the code you can see the test case
>> TestImportTSVWithVisibilityLabels to get an initial idea of how it is being
>> done. If not get back to us, happy to help .
>>
>> Regards
>> Ram
>>
>>
>>
>> On Tue, Jun 7, 2016 at 3:36 PM, Ellis, Tom (Financial Markets IT) <
>> tom.el...@lloydsbanking.com.invalid> wrote:
>>
>> > Hi,
>> >
>> > I was wondering if it's possible/how to write Visibility Labels to an
>> > HFileOutputFormat2? I believe Visibility Labels are just implemented
>> > as Tags, but with the normal way of writing them with
>> > Mutation#setCellVisibility these are formally written as Tags to the
>> > cells during the VisibilityController coprocessor as we need to assert
>> > the expression is valid for the labels configured.
>> >
>> > How can we add visibility labels to cells if we have a job that
>> > creates an HFile with HFileOutputFormat2 which is then subsequently
>> > loaded using LoadIncrementalHFiles?
>> >
>> > Cheers,
>> >
>> > Tom Ellis
>> > Consultant Developer - Excelian
>> > Data Lake | Financial Markets IT
>> > LLOYDS BANK COMMERCIAL BANKING
>> > 
>> >
>> > E: tom.el...@lloydsbanking.com
>> > Website:
>> > www.lloydsbankcommercial.com> > >
>> > , , ,
>> > Reduce printing. Lloyds Banking Group is helping to build the low
>> > carbon economy.
>> > Corporate Responsibility Report:
>> > www.lloydsbankinggroup-cr.com/downloads<
>> > http://www.lloydsbankinggroup-cr.com/downloads>
>> >
>> >
>> >
>> > Lloyds Banking Group plc. Registered Office: The Mound, Edinburgh EH1
>> 1YZ.
>> > Registered in Scotland no. SC95000. Telephone: 0131 225 4555. Lloyds
>> > Bank plc. Registered Office: 25 Gresham Street, London EC2V 7HN.
>> > Registered in England and Wales no. 2065. Telephone 0207626 1500. Bank
>> of Scotland plc.
>> > Registered Office: The Mound, Edinburgh EH1 1YZ. Registered in Scotland
>> no.
>> > SC327000. Telephone: 03457 801 801. Cheltenham & Gloucester plc.
>> > Registered
>> > Office: Barnett Way, Gloucester GL4 3RL. Registered in England and
>> > Wales 2299428. Telephone: 0345 603 1637
>> >
>> > Lloyds Bank plc, Bank of Scotland plc are authorised by the Prudential
>> > Regulation Authority and regulated by the Financial Conduct Authority
>> > and Prudential Regulation Authority.
>> >
>> > 

Re: IndexOutOfBoundsException during retrieving region split point

[Anoop John]

In ur test env also u have numDataIndexLevels=2?  Or it is 1 only?

-Anoop-

On Mon, Jun 6, 2016 at 1:12 PM, Pankaj kr  wrote:
> Thanks Ted for replying.
> Yeah, We have a plan to upgrade. But currently I want to know the reason 
> behind this. I tried to reproduce this in our test environment but didn’t 
> happen.
>
> in HFilePrettyPrinter output "numDataIndexLevels=2", so there were multilevel 
> data index. Is which circumstances this problem can happen?
>
> Regards,
> Pankaj
>
> -Original Message-
> From: Ted Yu [mailto:yuzhih...@gmail.com]
> Sent: Saturday, June 04, 2016 12:16 AM
> To: user@hbase.apache.org
> Cc: bhupendra jain; Sharanabasappa G Keriwaddi
> Subject: Re: IndexOutOfBoundsException during retrieving region split point
>
> 1.0.0 is quite old.
>
> Is it possible to upgrade to 1.1 or 1.2 release ?
>
> Thanks
>
> On Fri, Jun 3, 2016 at 8:12 AM, Pankaj kr  wrote:
>
>> Hi,
>>
>> We met a weird scenario in our production environment.
>> IndexOutOfBoundsException is thrown while retrieving mid key of the
>> storefile after region compaction.
>>
>> Log Snippet :
>> -
>> 2016-05-30 01:41:58,484 | INFO  |
>> regionserver/RS-HOSTNAME/RS-IP:21302-longCompactions-1464247799749 |
>> Completed compaction of 1 (all) file(s) in CF of
>> User_Namespace:User_Table,100050007010803_20140126_308010717550001_756781_99_36_0_01,1464543296529.676ee7e9902c066b0e8c15745463d3c5.
>> into eee1f433635d478197b212e2e378fce8(size=22.0 G), total size for
>> store is
>> 22.0 G. This selection was in queue for 0sec, and took 6mins, 25sec to
>> execute. |
>> org.apache.hadoop.hbase.regionserver.HStore.logCompactionEndMessage(HS
>> tore.java:1356)
>> 2016-05-30 01:41:58,485 | INFO  |
>> regionserver/RS-HOSTNAME/RS-IP:21302-longCompactions-1464247799749 |
>> Completed compaction: Request =
>> regionName=User_Namespace:User_Table,100050007010803_20140126_30801071
>> 7550001_756781_99_36_0_01,1464543296529.676ee7e9902c066b0e8c15745463d3
>> c5., storeName=CF, fileCount=1, fileSize=44.0 G, priority=6,
>> time=295643974900644; duration=6mins, 25sec |
>> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunn
>> er.run(CompactSplitThread.java:544)
>> 2016-05-30 01:41:58,529 | ERROR |
>> regionserver/RS-HOSTNAME/RS-IP:21302-longCompactions-1464247799749 |
>> Compaction failed Request =
>> regionName=User_Namespace:User_Table,100050007010803_20140126_30801071
>> 7550001_756781_99_36_0_01,1464543296529.676ee7e9902c066b0e8c15745463d3
>> c5., storeName=CF, fileCount=1, fileSize=44.0 G, priority=6,
>> time=295643974900644 |
>> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunn
>> er.run(CompactSplitThread.java:563)
>> java.lang.IndexOutOfBoundsException
>> at java.nio.Buffer.checkIndex(Buffer.java:540)
>> at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:139)
>> at
>> org.apache.hadoop.hbase.util.ByteBufferUtils.toBytes(ByteBufferUtils.java:490)
>> at
>> org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.midkey(HFileBlockIndex.java:349)
>> at
>> org.apache.hadoop.hbase.io.hfile.HFileReaderV2.midkey(HFileReaderV2.java:512)
>> at
>> org.apache.hadoop.hbase.regionserver.StoreFile$Reader.midkey(StoreFile.java:1480)
>> at
>> org.apache.hadoop.hbase.regionserver.StoreFile.getFileSplitPoint(StoreFile.java:685)
>> at
>> org.apache.hadoop.hbase.regionserver.DefaultStoreFileManager.getSplitPoint(DefaultStoreFileManager.java:126)
>> at
>> org.apache.hadoop.hbase.regionserver.HStore.getSplitPoint(HStore.java:1986)
>> at
>> org.apache.hadoop.hbase.regionserver.RegionSplitPolicy.getSplitPoint(RegionSplitPolicy.java:82)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegion.checkSplit(HRegion.java:7914)
>> at
>> org.apache.hadoop.hbase.regionserver.CompactSplitThread.requestSplit(CompactSplitThread.java:240)
>> at
>> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.run(CompactSplitThread.java:552)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> -
>> Observation:
>> >> HFilePrettyPrinter also print the message "Unable to retrieve the
>> midkey" for the mid key.
>> >> HDFS fsck report the hfile healthy.
>>
>> Though successful region compaction were also there, only few region
>> compaction failed with same error.
>>
>> Have anyone faced this issue? Any help will be appreciated.
>> HBase version is 1.0.0.
>>
>> Regards,
>> Pankaj
>>

Re: Major compaction cannot remove deleted rows until the region is split. Strange!

[Anoop John]

So u feel the call  s.close() would have created some Exception and
the removal of entry from scannerReadPoints would not have happened.
hmm.. sounds possible

public synchronized void close() {
  if (storeHeap != null) {
storeHeap.close();
storeHeap = null;
  }
  if (joinedHeap != null) {
joinedHeap.close();
joinedHeap = null;
  }
  // no need to synchronize here.
  scannerReadPoints.remove(this);
  this.filterClosed = true;
}

We may need to change this close impl and make sure the removal from
scannerReadPoints happen in a finally block.

-Anoop-


On Fri, Jun 3, 2016 at 7:33 AM, hao zhang  wrote:
> Hi, Stack
>
>   We found this in our production cluster, too.  I take a look about the
> code and found one case which will make the major compact not work.
>
>   1. put some rows
>   2. scan
>   3. delete
>   4. scanner didn't close rightly, it will keep the  the read point in
> region's scannerReadPoints.
>   5. major compact. It can't work because we have a scanner which have
> small read point than the delete.
>
>   But if move the region to new rs, the scannerReadPoints will update to
> the biggest memstoreTs which form sotrefile. So major compact will work.
>   I thought the try...catch module in Scanner.leaseExpired() method needs a
> finally module to close the region scanner rightly.
>
> public void leaseExpired() {
>   RegionScannerHolder rsh = scanners.remove(this.scannerName);
>   if (rsh != null) {
> RegionScanner s = rsh.s;
> LOG.info("Scanner " + this.scannerName + " lease expired on region "
>   + s.getRegionInfo().getRegionNameAsString());
> try {
>   Region region =
> regionServer.getRegion(s.getRegionInfo().getRegionName());
>   if (region != null && region.getCoprocessorHost() != null) {
> region.getCoprocessorHost().preScannerClose(s);
>   }
>   s.close();
>   if (region != null && region.getCoprocessorHost() != null) {
> region.getCoprocessorHost().postScannerClose(s);
>   }
> } catch (IOException e) {
>   LOG.error("Closing scanner for "
> + s.getRegionInfo().getRegionNameAsString(), e);
> }
>   } else {
> LOG.warn("Scanner " + this.scannerName + " lease expired, but no
> related" +
>   " scanner found, hence no chance to close that related scanner!");
>   }
> }
>
>
> 2016-06-02 2:50 GMT+08:00 Stack :
>
>> On Wed, Jun 1, 2016 at 10:56 AM, Tianying Chang  wrote:
>>
>> > Hi, Stack
>> >
>> > After moving the region and issue a major compact on that region, its
>> size
>> > shrink from 99G down to 24G. So it looks like the region is in a bad
>> state
>> > that cannot recover, close/open it fixed the issue. And from the region
>> > size metric graph, we can see major compaction stop working  since March
>> > 31, so some bug that caused region enter into bad state... Unfortunately,
>> > we don't have DEBUG enabled and that is the last region that has the
>> issue,
>> > it is hard to figure out what is the bug that caused the bad state...
>> >
>> >
>> Interesting. So moving it to another RS make it major-compactable? That
>> would seem to indicate some state kept in the RS memory is preventing the
>> major compaction running. Is moving the region a workaround for you until
>> we figure what it is Tian-Ying?
>>
>> St.
>>
>>
>>
>> > Thanks
>> > Tian-Ying
>> >
>> > On Tue, May 31, 2016 at 3:43 PM, Tianying Chang 
>> wrote:
>> >
>> > > Hi, Stack
>> > >
>> > > Based on the log, the major compaction was run, and it took 5+ hours.
>> > And
>> > > I also manually run major_compact from hbase shell explicitly to
>> verify.
>> > >
>> > > I just moved the region to a different RS and issued a major_compact on
>> > > that region again, let me see if the major compaction can succeed and
>> > will
>> > > report back.
>> > >
>> > > Thanks
>> > > Tian-Ying
>> > >
>> > > On Sun, May 29, 2016 at 4:35 PM, Stack  wrote:
>> > >
>> > >> On Fri, May 27, 2016 at 3:17 PM, Tianying Chang 
>> > >> wrote:
>> > >>
>> > >> > Yes, it is 94.26.  By a quick glance, I didn't  see any put that is
>> > >> older
>> > >> > than the delete marker's TS, which could go as far as about couple
>> > weeks
>> > >> > ago since major compaction on it for long time seems.
>> > >> >
>> > >> Also it is really strange that if the region is split, then seems
>> > >> > everything is working as expected. Also we noticed, the same region
>> > >> > replicated at the slave side is totally normal, i.e. at 20+G
>> > >> >
>> > >> >
>> > >> If you move the region to another server, does that work?
>> > >>
>> > >> Looking in 0.94 codebase, I see this in Compactor#compact
>> > >>
>> > >>
>> > >>   // For major compactions calculate the earliest put timestamp
>> > >>
>> > >>   // of all 

Re: Cell Level TTL And hfile.format.version

[Anoop John]

Ya as said, there is a warn message while doing the flush.  To check
with every mutation whether it having some tags and hfile version is
<3 would be bit costlier.  Suggest you change the version and things
will work as expected for you.

-Anoop-

On Fri, May 6, 2016 at 10:09 PM, Huaxiang Sun  wrote:
> Hi Saad,
>
>Please see HBASE-14730 
> . With 
> hfile.format.version 2, it will work partially. As long as the cells are in 
> the memstore, the TTL tag will be honored.
>   Once they are flushed into hfiles, ttl tag is lost which means cells will 
> be there forever.
>
>Your db will have cells which are supposed to be cleaned up with ttl. 
> Suggest to change the hfile format to version 3.
>
>Thanks,
>Huaxiang
>
>
>> On May 6, 2016, at 6:43 AM, Saad Mufti  wrote:
>>
>> HI,
>>
>> We're running a  CDH 5.5.2 HBase cluster (HBase Version 1.0.0-cdh5.5.2,
>> revision=Unknown). We are using the per-cell TTL feature (Mutation.setTTL)
>>
>>
>> As I learn more about and read up on HBase, I realized that in our HBase
>> config hfile.format.version was set to 2 (the default, we haven't touched
>> this config yet), and from what I read that version of the HFile format
>> does NOT support cell tags which are needed for cell TTL which use tags.
>>
>> Of course I am in the process of writing a test to check whether our
>> production db is indeed getting filled with cells that should in actuality
>> be expired given their TTL value.
>>
>> We haven't seen any errors at runtime, does this mean our efforts to set a
>> TTL are being silently ignored? Isn't this bad behavior? Even if the
>> hfile.format.version is set to the wrong version, wouldn't it be better to
>> throw an error instead of just silently dropping any tags that are set?
>>
>> Thanks.
>>
>> -
>> Saad
>

Re: Hbase ACL

[Anoop John]

>>One thing I don't understand - if I don't do the initial RW grant then
userX will never be allowed to write to the table,
Valid Q.. Came to my mind as well.. So the ans is the 1t write of the
cell to be done with a user who is having permission. (super user?)
While doing that write he has to give R and/or W permission to the
specific user..  Then onwards that user can read and/or update that
cell.  So u can consider a per cell W permission as an update
permission. Because one write is needed any way to give this cell
level permission.

As per the above Qs anw, we can help you how you can solve ur business use case.

-Anoop-

On Fri, May 6, 2016 at 12:20 PM, ramkrishna vasudevan
<ramkrishna.s.vasude...@gmail.com> wrote:
> Thanks Anoop for pointing to the JIRA.
>>>One thing I don't understand - if I don't do the initial RW grant then
> userX will never be allowed to write to the table,
>
> Even if you don't set a grant permisison to a table for userX - you will be
> able to write and read to the table provided the cell to which you are
> writing / reading is having the specific permission for that userX.
>
> I think in your case it is better that userX is given access to specific
> Qualifier and specific family only if you know the list of qualifiers
> before hand rather than giving permission for the entire table. This way
> you can ensure that your cell level permission takes effect.  I verified
> this with a test case too.
>
> Regards
> Ram
>
> On Fri, May 6, 2016 at 12:14 PM, Anoop John <anoop.hb...@gmail.com> wrote:
>
>> >>I am working on Hbase ACLs in order to lock a particular cell value for
>> writes by a user for an indefinite amount of time. This same user will be
>> writing to Hbase during normal program execution, and he needs to be able
>> to continue to write to other cells during the single cell lock period.
>> I’ve been experimenting with simple authentication (i.e. No Kerberos), and
>> the plan is to extend to a Kerberized cluster once I get this working.
>>
>> So you want one user to be not allowed to write to a single cell or
>> one column (cf:qual)? Do u know all possible column names for this
>> table?
>>
>> -Anoop-
>>
>> On Fri, May 6, 2016 at 11:59 AM, Anoop John <anoop.hb...@gmail.com> wrote:
>> > HBASE-11432 removed the cell first strategy
>> >
>> > -Anoop-
>> >
>> > On Thu, May 5, 2016 at 4:49 PM, Tokayer, Jason M.
>> > <jason.toka...@capitalone.com> wrote:
>> >> Hi Ram,
>> >>
>> >> I very much appreciate the guidance. I should be able to run through
>> the gambit of tests via code this afternoon and will report back when I do.
>> >>
>> >> One thing I don't understand - if I don't do the initial RW grant then
>> userX will never be allowed to write to the table, so I don't see how I can
>> use that approach.
>> >>
>> >> Thanks,
>> >> Jason
>> >>
>> >>
>> >>
>> >> Sent with Good (www.good.com)
>> >> 
>> >> From: ramkrishna vasudevan <ramkrishna.s.vasude...@gmail.com>
>> >> Sent: Thursday, May 5, 2016 4:03:48 AM
>> >> To: user@hbase.apache.org
>> >> Subject: Re: Hbase ACL
>> >>
>> >> I verified the above behaviour using test case as the cluster was busy
>> with
>> >> other activities.
>> >> So in the above example that you mentioned, you had already issued RW
>> >> access to user-X on the table. Then a specific cell is over written
>> with R
>> >> permission using the special 'grant' command.
>> >>
>> >> Now as per the code since you already have a Write permission granted
>> the
>> >> cell level access does not work. Instead if you don't grant the RW
>> >> permission to the user-X and try your steps it should work fine.
>> >>
>> >> So when a user with no permission on a table tries to do some mutations
>> >> then if there is already a cell level permission granted by the super
>> user
>> >> then the cell level permission takes precedence.
>> >>
>> >> One thing to note is that the special 'grant' command is only for
>> testing
>> >> and not for production use case. You should always go with storing the
>> ACL
>> >> per mutation.
>> >>
>> >> Also see this section
>> >> ACL Granularity and Evaluation Order
>> >>
>> >> ACLs are evaluated from least granular to most granular, an

Re: Hbase ACL

[Anoop John]

>>I am working on Hbase ACLs in order to lock a particular cell value for 
>>writes by a user for an indefinite amount of time. This same user will be 
>>writing to Hbase during normal program execution, and he needs to be able to 
>>continue to write to other cells during the single cell lock period. I’ve 
>>been experimenting with simple authentication (i.e. No Kerberos), and the 
>>plan is to extend to a Kerberized cluster once I get this working.

So you want one user to be not allowed to write to a single cell or
one column (cf:qual)? Do u know all possible column names for this
table?

-Anoop-

On Fri, May 6, 2016 at 11:59 AM, Anoop John <anoop.hb...@gmail.com> wrote:
> HBASE-11432 removed the cell first strategy
>
> -Anoop-
>
> On Thu, May 5, 2016 at 4:49 PM, Tokayer, Jason M.
> <jason.toka...@capitalone.com> wrote:
>> Hi Ram,
>>
>> I very much appreciate the guidance. I should be able to run through the 
>> gambit of tests via code this afternoon and will report back when I do.
>>
>> One thing I don't understand - if I don't do the initial RW grant then userX 
>> will never be allowed to write to the table, so I don't see how I can use 
>> that approach.
>>
>> Thanks,
>> Jason
>>
>>
>>
>> Sent with Good (www.good.com)
>> 
>> From: ramkrishna vasudevan <ramkrishna.s.vasude...@gmail.com>
>> Sent: Thursday, May 5, 2016 4:03:48 AM
>> To: user@hbase.apache.org
>> Subject: Re: Hbase ACL
>>
>> I verified the above behaviour using test case as the cluster was busy with
>> other activities.
>> So in the above example that you mentioned, you had already issued RW
>> access to user-X on the table. Then a specific cell is over written with R
>> permission using the special 'grant' command.
>>
>> Now as per the code since you already have a Write permission granted the
>> cell level access does not work. Instead if you don't grant the RW
>> permission to the user-X and try your steps it should work fine.
>>
>> So when a user with no permission on a table tries to do some mutations
>> then if there is already a cell level permission granted by the super user
>> then the cell level permission takes precedence.
>>
>> One thing to note is that the special 'grant' command is only for testing
>> and not for production use case. You should always go with storing the ACL
>> per mutation.
>>
>> Also see this section
>> ACL Granularity and Evaluation Order
>>
>> ACLs are evaluated from least granular to most granular, and when an ACL is
>> reached that grants permission, evaluation stops. This means that cell ACLs
>> do not override ACLs at less granularity.
>> I will raise a bug to remove the OP_ATTRIBUTE_ACL_STRATEGY from
>> AccessControlConstants if it is misleading to users.
>>
>> Feel free to write back incase am missing something or need further inputs.
>> Happy to help.
>>
>> Regards
>> Ram
>>
>>
>>
>>
>>
>> On Wed, May 4, 2016 at 11:38 PM, ramkrishna vasudevan <
>> ramkrishna.s.vasude...@gmail.com> wrote:
>>
>>> I tried out with the examples already available in the code base. Will try
>>> it out on a cluster which I did not have access to today. Will probably
>>> have access tomorrow.
>>>
>>> I was not aware of that 'grant' feature which allows to set permission on
>>> all the cells with a specific prefix and on a specific qualifier. I will
>>> check and get back to you on that.
>>>
>>> Regards
>>> Ram
>>>
>>> On Wed, May 4, 2016 at 10:25 PM, Tokayer, Jason M. <
>>> jason.toka...@capitalone.com> wrote:
>>>
>>>> Hi Ram,
>>>>
>>>> Thanks for the reply. I can take a look at that Mutation documentation.
>>>> But I wanted to first confirm that this works at all, which is why I
>>>> started in the shell. The docs I’ve been using are here:
>>>>
>>>> https://github.com/apache/hbase/blob/master/src/main/asciidoc/_chapters/sec
>>>> urity.adoc. If you search for 'The syntax for granting cell ACLs uses the
>>>> following syntax:’ you'll find the example I’ve been following for cell
>>>> ACLs. According to the docs, "The shell will run a scanner with the given
>>>> criteria, rewrite the found cells with new ACLs, and store them back to
>>>> their exact coordinates.”. So I was under the impression that this would
>>>> lock ALL cells that meet the criteria,

Re: Hbase ACL

[Anoop John]

HBASE-11432 removed the cell first strategy

-Anoop-

On Thu, May 5, 2016 at 4:49 PM, Tokayer, Jason M.
 wrote:
> Hi Ram,
>
> I very much appreciate the guidance. I should be able to run through the 
> gambit of tests via code this afternoon and will report back when I do.
>
> One thing I don't understand - if I don't do the initial RW grant then userX 
> will never be allowed to write to the table, so I don't see how I can use 
> that approach.
>
> Thanks,
> Jason
>
>
>
> Sent with Good (www.good.com)
> 
> From: ramkrishna vasudevan 
> Sent: Thursday, May 5, 2016 4:03:48 AM
> To: user@hbase.apache.org
> Subject: Re: Hbase ACL
>
> I verified the above behaviour using test case as the cluster was busy with
> other activities.
> So in the above example that you mentioned, you had already issued RW
> access to user-X on the table. Then a specific cell is over written with R
> permission using the special 'grant' command.
>
> Now as per the code since you already have a Write permission granted the
> cell level access does not work. Instead if you don't grant the RW
> permission to the user-X and try your steps it should work fine.
>
> So when a user with no permission on a table tries to do some mutations
> then if there is already a cell level permission granted by the super user
> then the cell level permission takes precedence.
>
> One thing to note is that the special 'grant' command is only for testing
> and not for production use case. You should always go with storing the ACL
> per mutation.
>
> Also see this section
> ACL Granularity and Evaluation Order
>
> ACLs are evaluated from least granular to most granular, and when an ACL is
> reached that grants permission, evaluation stops. This means that cell ACLs
> do not override ACLs at less granularity.
> I will raise a bug to remove the OP_ATTRIBUTE_ACL_STRATEGY from
> AccessControlConstants if it is misleading to users.
>
> Feel free to write back incase am missing something or need further inputs.
> Happy to help.
>
> Regards
> Ram
>
>
>
>
>
> On Wed, May 4, 2016 at 11:38 PM, ramkrishna vasudevan <
> ramkrishna.s.vasude...@gmail.com> wrote:
>
>> I tried out with the examples already available in the code base. Will try
>> it out on a cluster which I did not have access to today. Will probably
>> have access tomorrow.
>>
>> I was not aware of that 'grant' feature which allows to set permission on
>> all the cells with a specific prefix and on a specific qualifier. I will
>> check and get back to you on that.
>>
>> Regards
>> Ram
>>
>> On Wed, May 4, 2016 at 10:25 PM, Tokayer, Jason M. <
>> jason.toka...@capitalone.com> wrote:
>>
>>> Hi Ram,
>>>
>>> Thanks for the reply. I can take a look at that Mutation documentation.
>>> But I wanted to first confirm that this works at all, which is why I
>>> started in the shell. The docs I’ve been using are here:
>>>
>>> https://github.com/apache/hbase/blob/master/src/main/asciidoc/_chapters/sec
>>> urity.adoc. If you search for 'The syntax for granting cell ACLs uses the
>>> following syntax:’ you'll find the example I’ve been following for cell
>>> ACLs. According to the docs, "The shell will run a scanner with the given
>>> criteria, rewrite the found cells with new ACLs, and store them back to
>>> their exact coordinates.”. So I was under the impression that this would
>>> lock ALL cells that meet the criteria, and if I wanted to lock a specific
>>> cell I could add some more filters. Might I be reading that wrong?
>>>
>>> I can access the examples and will take a look. Were you able to confirm
>>> proper functioning for table overrides on existing cells?
>>>
>>> --
>>> Warmest Regards,
>>> Jason Tokayer, PhD
>>>
>>>
>>>
>>>
>>> On 5/4/16, 12:30 PM, "ramkrishna vasudevan"
>>>  wrote:
>>>
>>> >Superuser:
>>> >grant 'ns1:t1', {'userX' => 'R' }, { COLUMNS => 'cf1', FILTER =>
>>> >"(PrefixFilter ('r2'))" }
>>> >
>>> >So you are trying to grant R permission to user-X for a given qualifier.
>>> >Please not that this is NOT for a given Cell.
>>> >
>>> >Reiterating from your first mail
>>> >>>What I need to be able to do next is to set user-X’s permissions on a
>>> >particular cell to read only and have that take precedence over the table
>>> >permissions.
>>> >So where is this  being done in your above example? I may be missing
>>> >something here.
>>> >
>>> >You need to create Put mutation and set READ permission using the
>>> >Mutation.setACL API for User-X for that specific cell.
>>> >
>>> >Can you see an example in TestCellACLs in case you have access to the
>>> >code?
>>> >
>>> >The idea of cell level ACLs is to give cell level access. So in this case
>>> >your super-user can pass a mutation with ACL set on the mutation which
>>> >could say - Grant R permission to user-X.
>>> >
>>> >So only user-X can read the cell but he will not be able to do any
>>> updates
>>> >to that cell.
>>> 

Re: Access cell tags from HBase shell

[Anoop John]

You have to config the Codec KeyValueCodecWithTags at client side.
Server also will use same Codec to talk with this client.  Ya just
check with a java client 1st and then experiment with shell.

-Anoop-

On Tue, May 3, 2016 at 9:43 PM, ramkrishna vasudevan
<ramkrishna.s.vasude...@gmail.com> wrote:
> Hi Benedict
>
> As super user you should be able to get back the tags as ordinals and make
> sure you set the codec KeyValueCodecWithTags.
> But I am not sure if it is possible to do it from the hBase shell. Can you
> try from a java client ?
>
> I did not do the hands on of late on this but I can do it if you face any
> difficulties and revert back if needed.
>
> Regards
> Ram
>
> On Tue, May 3, 2016 at 8:19 PM, <benedict.whittamsm...@thomsonreuters.com>
> wrote:
>
>> Hi Anoop,
>>
>> Can I still get the labels back (as ordinals, as a super user, and using
>> the KeyValueCodecWithTags codec) using the HBase shell?
>>
>> If so, what are the steps I need to take (i.e. doesn't seem to be working
>> for me, but then I've likely made a mistake setting the codec).
>>
>> Thanks,
>>
>> Ben
>>
>> > -Original Message-
>> > From: Anoop John [mailto:anoop.hb...@gmail.com]
>> > Sent: 15 September 2015 14:28
>> > To: user@hbase.apache.org
>> > Subject: Re: Access cell tags from HBase shell
>> >
>> > We are not returning back the cell labels back to client.  So what I
>> > will
>> > recommend you to test is by having a predicate in scan and test you see
>> > only the relevant data back.
>> > But there is way to return cells (all*) with out any vis check and
>> > cells in
>> > client will have the vis label tag also in it. This is by issuing the
>> > scan
>> > as a super user.  And also set the codec as KeyValueCodecWithTags.
>> > But one thing we wont be storing the vis label with Cells as string..
>> > We
>> > will optimize..  We will store them as ordinals and & and | condition
>> > also
>> > we will optimize. So even if you read back the vis label tags back in
>> > client it will be hard to parse it and understand..  Any thing more you
>> > would like to know, pls let me know..  Will be happy to help.
>> >
>> > BTW  once you test and if start to use the feature pls let me know..
>> > Will
>> > be great to hear the usage cases and feedback.
>> >
>> > -Anoop-
>> >
>> >
>> > On Fri, Sep 11, 2015 at 5:35 AM, Suresh Subbiah
>> > <suresh.subbia...@gmail.com>
>> > wrote:
>> >
>> > > Hi Anoop,
>> > >
>> > > Thank you very much for the offer to help.
>> > >
>> > > I have been thinking some more about what it is that we need to do
>> > and have
>> > > realized that we don't need custom cell tags.
>> > > We we will only be using visibility labels. This is basically for
>> > testing
>> > > purpose and to understand exactly how data looks.
>> > >
>> > > How do we see visibility labels that are applied to a particular
>> > cell? For
>> > > ex, if we want to know all the labels that have been applied to
>> > > all cells, how do we do that? Or can that only be done by applying a
>> > > predicate and then check to see if the pred passes?
>> > >
>> > > Is there a way to pass visibility labels to client is a test mode ?
>> > >
>> > > Thanks
>> > > Suresh
>> > >
>> > >
>> > > On Thu, Sep 3, 2015 at 11:07 PM, Anoop John <anoop.hb...@gmail.com>
>> > wrote:
>> > >
>> > > > Hi Suresh
>> > > > You wan to use ur own custom tags with cells?  The
>> > > features
>> > > > like cell level vis labels etc are also implemented by storing them
>> > as
>> > > cell
>> > > > tags.  Yes as others said, the tags is by default a server only
>> > thing.
>> > > > Means you can not pass tags from/to client along with cells.  There
>> > is
>> > > some
>> > > > security reasons why we had opted this path.  And there were no
>> > custom
>> > > tag
>> > > > needs by then. Pls let us know what you want to achieve.   There is
>> > ways
>> > > to
>> > > > pass tags to/from client. I can help you.
>> > > >
>> > > > -Anoop-
>> > > >
>

Re: Question on writing scan coprocessors

[Anoop John]

Different ways there for you to achieve

1. preScannerOpen -> Here u will get Scan object. You can add a new
Filter into the Scan  object and pass the Scan object (or the
attribute you look for) into this Filter.  The later scan op will use
this Filter and within that u can do filter of cells.

2. postScannerOpen -> Use this hook.  Here the RegionScanner is
already been made and u will get it passed to this hook. Also u have
the Scan object.  Create a wrapper for the scanner and return that
wrapper object from this hook.. Within the wrapper u can pass the Scan
attribute.   Use the wrapped scanner instance for the real next()
ops..  On the returned cells do the filtering ops



-Anoop-




On Thu, Apr 28, 2016 at 4:06 AM, James Johansville
 wrote:
> Hello,
>
> I'd like to write a similar coprocessor to the example
> RegionObserverExample at
> http://www.3pillarglobal.com/insights/hbase-coprocessors  : that is, a scan
> coprocessor which intercepts and selectively filters scan results.
>
> My problem is, I need to be able to filter out Results based on a Scan
> attribute. preScannerNext() as used in above example does not allow for
> this, as Scan object is not passed down to the method.
>
> Any guidance on how to accomplish this?
>
> Thanks,
> James

Re: HBase Region Server Memory Configuration

[Anoop John]

So u have given the max heap size for the RS process via Cloudera
manager and u fear that that much is not taken by the process.  U may
need to check the logs in CDH manager and the RS to see whether it is
correctly set

So u r using bucket cache(BC) in offheap mode right.  In 1.0 version
there wont be any direct off heap space usage from HBase code other
than BC code.  (Later versions have it).  But the SocketChannel uses
off heap space to move the data to network.  When written bytes to
socket channel, it first copy it to off heap bytebuffers and then the
OS level calls to do the network transfers.  So ya the process as such
would need more offheap memory than the size of the bucket cache.

At least in 1.0 the perf of off heap Bucket cache is not same as that
of L1 cache.  When both are in place, by default, we will keep the
index blocks etc (non data) in L1 and data blocks in L2.  That would
be an ideal choice IMO.  The separation allows the L1 blocks to stay
there with out getting possibly evicted. These are meta information
for the data blocks.   The data blocks can stay in L2 cache.

-Anoop-


On Wed, Apr 6, 2016 at 3:24 PM, Amit Shah  wrote:
> Hello,
>
> I am trying to understand hbase's memory configuration. I am using hbase
> version 1.0 through cloudera (version 5.5.2). I need to benchmark query
> performance (through phoenix 4.6) on hbase with and without bucket cache.
>
> While reading about hbase memory configuration I have following doubts
>
> 1. I have configured java heap size on the region server as 3.5 GB on the
> cloudera manager console though the region server web ui (server metrics ->
> Memory tab) shows the max heap to be 1.7. hbase.block.cache.size is
> configured to be 0.6, hbase.regionserver.global.memstore.lowerLimit is
> configured to be 0.1 while hbase.regionserver.global.memstore.upperLimit is
> configured to be 0.2 so that the blockcache + memstore combination does not
> exceed 0.8.
>
> What could be the reason behind the difference shown on the region server
> web ui?
>
> 2. While configuring bucket cache, there are mainly two configs to set the
> size of the bucket cache -XX:MaxDirectMemorySize
> and hbase.bucketcache.size. It is recommended that XX:MaxDirectMemorySize
>  should be of a higher value than hbase.bucketcache.size. What are the
> hbase operations that use the offheap memory apart from bucket cache?
>
> 3. Is it worth benchmarking bucketcache in isolation i.e. without
> LruBlockCache - hbase.block.cache.size set to 0?
>
> Appreciate your inputs.
>
> Thanks,
> Amit

Re: Leveraging As Much Memory As Possible

[Anoop John]

Ya having HBase side cache will be a better choice rather than HDFS
cache IMO.   Yes u r correct...  You might not want to give a very
large size for the heap. You can make use of the off heap BucketCache.

-Anoop-

On Thu, Mar 31, 2016 at 4:35 AM, Ted Yu  wrote:
> For #1, please see the top two blogs @ https://blogs.apache.org/hbase/
>
> FYI
>
> On Wed, Mar 30, 2016 at 7:59 AM, Amit Shah  wrote:
>
>> Hi,
>>
>> I am trying to configure my hbase (version 1.0) phoenix (version - 4.6)
>> cluster to utilize as much memory as possible on the server hardware. We
>> have an OLAP workload that allows users to perform interactive analysis
>> over huge sets of data. While reading about hbase configuration I came
>> across two configs
>>
>> 1. Hbase bucket cache
>> <
>> http://blog.asquareb.com/blog/2014/11/24/how-to-leverage-large-physical-memory-to-improve-hbase-read-performance
>> >
>> (off heap) which looks like a good option to bypass garbage collection.
>> 2. Hadoop pinned hdfs blocks
>> 
>> (max locked memory) concept that loads the hdfs blocks in memory, but given
>> that hbase is configured with short circuit reads I assume this config may
>> not be of much help. Instead it would be right to increase hbase region
>> server heap memory. Is my understanding right?
>>
>> We use HBase with Phoenix.
>> Kindly let me know your thoughts or suggestions on any more options that I
>> should explore
>>
>> Thanks,
>> Amit.
>>

Re: Inconsistent scan performance

[Anoop John]

I see you set cacheBlocks to be false on the Scan.   By any chance on
some other RS(s), the data you are looking for is already in cache?
(Any previous scan or  by cache on write)  And there are no concurrent
writes any way right?   This much difference in time !  One
possibility is blocks avail or not avail in cache..

-Anoop-

On Fri, Mar 25, 2016 at 11:04 AM, Stack  wrote:
> On Thu, Mar 24, 2016 at 4:45 PM, James Johansville <
> james.johansvi...@gmail.com> wrote:
>
>> Hello all,
>>
>> So, I wrote a Java application for HBase that does a partitioned full-table
>> scan according to a set number of partitions. For example, if there are 20
>> partitions specified, then 20 separate full scans are launched that cover
>> an equal slice of the row identifier range.
>>
>> The rows are uniformly distributed throughout the RegionServers.
>
>
> How many RegionServers? How many Regions? Are Regions evenly distributed
> across the servers? If you put all partitions on one machine and then run
> your client, do the timings even out?
>
> The disparity seems really wide.
>
> St.Ack
>
>
>
>
>> I
>> confirmed this through the hbase shell. I have only one column family, and
>> each row has the same number of column qualifiers.
>>
>> My problem is that the individual scan performance is wildly inconsistent
>> even though they fetch approximately a similar number of rows. This
>> inconsistency appears to be random with respect to hosts or regionservers
>> or partitions or CPU cores. I am the only user of the fleet and not running
>> any other concurrent HBase operation.
>>
>> I started measuring from the beginning of the scan and stopped measuring
>> after the scan was completed. I am not doing any logic with the results,
>> just scanning them.
>>
>> For ~230K rows fetched per scan, I am getting anywhere from 4 seconds to
>> 100+ seconds. This seems a little too bouncy for me. Does anyone have any
>> insight? By comparison, a similar utility I wrote to upsert to
>> regionservers was very consistent in ops/sec and I had no issues with it.
>>
>> Using 13 partitions on a machine that has 32 CPU cores and 16 GB heap, I
>> see anywhere between 3K ops/sec to 82K ops/sec. Here's an example of log
>> output I saved that used 130 partitions.
>>
>> total # partitions:130; partition id:47; rows:232730 elapsed_sec:6.401
>> ops/sec:36358.38150289017
>> total # partitions:130; partition id:100; rows:206890 elapsed_sec:6.636
>> ops/sec:31176.91380349608
>> total # partitions:130; partition id:63; rows:233437 elapsed_sec:7.586
>> ops/sec:30772.08014764039
>> total # partitions:130; partition id:9; rows:232585 elapsed_sec:32.985
>> ops/sec:7051.235410034865
>> total # partitions:130; partition id:19; rows:234192 elapsed_sec:38.733
>> ops/sec:6046.3170939508955
>> total # partitions:130; partition id:1; rows:232860 elapsed_sec:48.479
>> ops/sec:4803.316900101075
>> total # partitions:130; partition id:125; rows:205334 elapsed_sec:41.911
>> ops/sec:4899.286583474505
>> total # partitions:130; partition id:123; rows:206622 elapsed_sec:42.281
>> ops/sec:4886.875901705258
>> total # partitions:130; partition id:54; rows:232811 elapsed_sec:49.083
>> ops/sec:4743.210480206996
>>
>> I use setCacheBlocks(false), setCaching(5000).  Does anyone have any
>> insight into how I can make the read performance more consistent?
>>
>> Thanks!
>>

Re: Region server getting aborted in every one or two days

[Anoop John]

So seems like the issue also comes out just after a log roll. (?)   So
we no longer have the old WAL file and still that write op try to
write to old file?  From the WAL file path name u can confirm this

-Anoop-

On Wed, Mar 23, 2016 at 6:14 PM, Pankaj kr <pankaj...@huawei.com> wrote:
> Thanks Anoop for replying..
>
> No explicit close op happened on the WAL file (this log was rolled few sec 
> before). As per HDFS log, there is no close call to this WAL file.
>
>
> Same issue happened again on 19th March,
>
> Here WAL was rolled just before the issue happened,
> 2016-03-19 05:38:07,153 | INFO  | 
> regionserver/RS-HOSTNAME/RS-IP:21302.logRoller | Rolled WAL 
> /hbase/WALs/RS-HOSTNAME,21302,1458301420876/RS-HOSTNAME%2C21302%2C1458301420876.default.1458337083824
>  with entries=6508, filesize=61.03 MB; new WAL 
> /hbase/WALs/RS-HOSTNAME,21302,1458301420876/RS-HOSTNAME%2C21302%2C1458301420876.default.1458337087136
>  | 
> org.apache.hadoop.hbase.regionserver.wal.FSHLog.replaceWriter(FSHLog.java:972)
>
> And after some sec during sync op,
> 2016-03-19 05:38:10,075 | ERROR | sync.1 | Error syncing, request close of 
> wal  | 
> org.apache.hadoop.hbase.regionserver.wal.FSHLog$SyncRunner.run(FSHLog.java:1346)
> java.nio.channels.ClosedChannelException
> at 
> org.apache.hadoop.hdfs.DataStreamer$LastExceptionInStreamer.throwException4Close(DataStreamer.java:208)
> at 
> org.apache.hadoop.hdfs.DFSOutputStream.checkClosed(DFSOutputStream.java:142)
> at 
> org.apache.hadoop.hdfs.DFSOutputStream.flushOrSync(DFSOutputStream.java:545)
> at 
> org.apache.hadoop.hdfs.DFSOutputStream.hflush(DFSOutputStream.java:490)
> at 
> org.apache.hadoop.fs.FSDataOutputStream.hflush(FSDataOutputStream.java:130)
> at 
> org.apache.hadoop.hbase.regionserver.wal.ProtobufLogWriter.sync(ProtobufLogWriter.java:190)
> at 
> org.apache.hadoop.hbase.regionserver.wal.FSHLog$SyncRunner.run(FSHLog.java:1342)
> at java.lang.Thread.run(Thread.java:745)
> 2016-03-19 05:38:10,076 | INFO  | 
> regionserver/RS-HOSTNAME/RS-IP:21302.logRoller | Rolled WAL 
> /hbase/WALs/RS-HOSTNAME,21302,1458301420876/RS-HOSTNAME%2C21302%2C1458301420876.default.1458337087136
>  with entries=6383, filesize=61.51 MB; new WAL 
> /hbase/WALs/RS-HOSTNAME,21302,1458301420876/RS-HOSTNAME%2C21302%2C1458301420876.default.1458337090049
>  | 
> org.apache.hadoop.hbase.regionserver.wal.FSHLog.replaceWriter(FSHLog.java:972)
> 2016-03-19 05:38:10,087 | FATAL | 
> regionserver/RS-HOSTNAME/RS-IP:21302.logRoller | ABORTING region server 
> RS-HOSTNAME,21302,1458301420876: IOE in log roller | 
> org.apache.hadoop.hbase.regionserver.HRegionServer.abort(HRegionServer.java:2055)
> java.nio.channels.ClosedChannelException
> at 
> org.apache.hadoop.hdfs.DataStreamer$LastExceptionInStreamer.throwException4Close(DataStreamer.java:208)
> at 
> org.apache.hadoop.hdfs.DFSOutputStream.checkClosed(DFSOutputStream.java:142)
> at 
> org.apache.hadoop.hdfs.DFSOutputStream.flushOrSync(DFSOutputStream.java:545)
> at 
> org.apache.hadoop.hdfs.DFSOutputStream.hflush(DFSOutputStream.java:490)
> at 
> org.apache.hadoop.fs.FSDataOutputStream.hflush(FSDataOutputStream.java:130)
> at 
> org.apache.hadoop.hbase.regionserver.wal.ProtobufLogWriter.sync(ProtobufLogWriter.java:190)
> at 
> org.apache.hadoop.hbase.regionserver.wal.FSHLog$SyncRunner.run(FSHLog.java:1342)
> at java.lang.Thread.run(Thread.java:745)
> 2016-03-19 05:38:10,088 | FATAL | 
> regionserver/RS-HOSTNAME/RS-IP`:21302.logRoller | RegionServer abort: loaded 
> coprocessors are: 
> [org.apache.hadoop.hbase.index.coprocessor.regionserver.IndexRegionObserver, 
> org.apache.hadoop.hbase.JMXListener, 
> org.apache.hadoop.hbase.index.coprocessor.wal.IndexWALObserver] | 
> org.apache.hadoop.hbase.regionserver.HRegionServer.abort(HRegionServer.java:2063)
>
> Here also, no error details in DN/NN log.
>
> I am still checking this, will update if any findings.
>
> Regards,
> Pankaj
>
> -Original Message-
> From: Anoop John [mailto:anoop.hb...@gmail.com]
> Sent: Wednesday, March 23, 2016 3:50 PM
> To: user@hbase.apache.org
> Subject: Re: Region server getting aborted in every one or two days
>
> At the same time, any explicit close op happened on the WAL file?  Any log 
> rolling?  Can u check the logs to know this?  May be check HDFS logs to know 
> abt the close calls to WAL file?
>
> -Anoop-
>
> On Wed, Mar 23, 2016 at 12:10 PM, Pankaj kr <pankaj...@huawei.com> wrote:
>> Hi,
>>
>> In our production environment, RS is getting aborted in every one or two 
>> days with following exception.
>