Re: [ANNOUNCE] New HBase committer Nihal Jain

[ramkrishna vasudevan]

Congratulations !!!

On Thu, May 4, 2023 at 8:39 AM 张铎(Duo Zhang)  wrote:

> Congratulations!
>
> Viraj Jasani  于2023年5月3日周三 23:47写道：
>
> > Congratulations Nihal!! Very well deserved!!
> >
> > On Wed, May 3, 2023 at 5:12 AM Nick Dimiduk  wrote:
> >
> > > Hello!
> > >
> > > On behalf of the Apache HBase PMC, I am pleased to announce that Nihal
> > Jain
> > > has accepted the PMC's invitation to become a committer on the project.
> > We
> > > appreciate all of Nihal's generous contributions thus far and look
> > forward
> > > to his continued involvement.
> > >
> > > Congratulations and welcome, Nihal Jain!
> > >
> > > Thanks,
> > > Nick
> > >
> >
>

Re: [DISCUSS] Remove checkRefCnt for on-heap ByteBuffs?

[ramkrishna vasudevan]

+1 on removing  it from onheap path

On Sat, 11 Mar 2023, 11:06 张铎(Duo Zhang),  wrote:

> +1 if we can get a performance boost, as this is on the critical read/write
> path, performance is important.
>
> Thanks.
>
> Bryan Beaudreault  于2023年3月10日周五 01:36写道：
>
> > Hey all,
> >
> > We have a use-case for HFile.Reader at my company. The use-case in
> question
> > is scanning many hfiles for a small subset of cells, so it mostly is
> > just iterating a large number of HFiles and discarding most of the cells.
> > We recently upgraded that deployable from super old cdh 5.16 (hbase
> > 1.2-ish) to hbase 2.5.3. In doing so, we noticed a performance regression
> > of around 4x. I imagine this regression would also affect
> > ClientSideRegionScanner.
> >
> > We did some profiling and noticed that a large amount of time is spent in
> > SingleByteBuff.checkRefCnt. It seems like every SingleByteBuff method
> calls
> > checkRefCnt and this checks compares a volatile
> > in AtomicIntegerFieldUpdater in the netty code.
> >
> > I believe ref counting is mostly necessary for off-heap buffers, but
> > on-heap buffers are also wrapped in SingleByteBuff and so also go through
> > checkRefCnt. We removed the checkRefCnt call, and the regression
> > disappeared.
> >
> > We created a simple test method which just does HFile.createReader,
> > reader.getScanner(), and then iterates the scanner counting the total
> > cells. The test reads an hfile with 100M cells and takes  over 1 minute
> > with checkRefCnt. Removing checkRefCnt brings the runtime down to 20s.
> >
> > It's worth noting that the regression is most prominent on java17. It's
> > slightly less obvious on java11, with runtime being 40s vs 28s.
> >
> > Thoughts on updating SingleByteBuff to skip the checkRefCnt call for
> > on-heap buffers? We can handle this in the constructor, when wrapping an
> > on-heap buffer here [1].
> >
> > [1]
> >
> >
> https://github.com/apache/hbase/blob/master/hbase-common/src/main/java/org/apache/hadoop/hbase/io/ByteBuffAllocator.java#L300
> >
>

Re: [ANNOUNCE] New HBase Committer Liangjun He

[ramkrishna vasudevan]

Congratulations !!!

On Mon, Dec 5, 2022 at 10:47 PM Viraj Jasani  wrote:

> Congratulations!!
>
> On Sat, Dec 3, 2022 at 5:51 AM Yu Li  wrote:
>
> > Hi All,
> >
> > On behalf of the Apache HBase PMC, I am pleased to announce that Liangjun
> > He (heliangjun) has accepted the PMC's invitation to become a committer
> on
> > the project. We appreciate all of Liangjun's generous contributions thus
> > far and look forward to his continued involvement.
> >
> > Congratulations and welcome, Liangjun!
> >
> > 我很高兴代表 Apache HBase PMC 宣布 Liangjun He (何良均) 已接受我们的邀请，成为 Apache HBase 项目的
> > Committer。感谢何良均一直以来为 HBase 项目做出的贡献，并期待他在未来继续承担更多的责任。
> >
> > 欢迎良均！
> >
> > Best Regards,
> > Yu
> > --
> > Best Regards,
> > Yu
> >
>

Re: Coprocs and Off-Heap Writes

[ramkrishna vasudevan]

Hi
Anoop has clearly answered I believe. the short answer is in your CP it is
better you copy/clone the cells so that there is no reference. I believe
the Index related WAL codec in Phoenix was also trying to do something
similar if I remember correctly. (I may be wrong though).

Regards
Ram

On Thu, Dec 3, 2020 at 9:30 AM Anoop John  wrote:

> Hi Geoffrey,
>
> In case of off heap backed write path (RPC layer itself), the write payload
> is accepted into DBBs that we get from a pool.  And cells will be created
> over this DBB. In case we add Tags in CPs, there will be a new Cell POJO
> created.  But that will anyways refer to old POJO for all parts except
> Tags. See TagRewriteCell for eg:  Anyways, when we add cells to Memsore,
> then only we retrieve it from this RPC side buffer.   In the write path,
> once the call completes and comes back to RPC layer there we will release
> the buffer. So there should not be a worry of a leak.
> The only thing to be careful in CPs, is if you keep reference to Cells.  In
> such cases, it's advised to clone the cell (or parts of it) and keep that
> reference.  When RPC side we used pooled DBB, not doing this correctly can
> cause the Cell being corrupted later. (The buffer would be released once
> RPC call is over and later would be used to read some other write payload)
> Even in case of on heap buffer usage at RPC, keeping such ref without clone
> can cause issues as it will not allow the RPC payload read buffer (much
> larger size than a cell size typically) to get GCed. Anyways I know Phoenix
> Jira's aim is to create Cells with addition of tags , am saying it just as
> a pointer.
>
> Anoop
>
> On Thu, Dec 3, 2020 at 2:41 AM Geoffrey Jacoby  wrote:
>
> > I'm code-reviewing a Phoenix PR [1] right now, which adds Tags to a
> > mutation's Cells in a coproc. A question has come up regarding coprocs
> and
> > the optional off-heaping of the write path in HBase 2.x and up.
> >
> > For what parts of the write path (and hence, which coproc hooks) is it
> safe
> > to change the underlying Cells of a batch mutation without leaking
> off-heap
> > memory?
> >
> > The HBase book entry on off-heap writes [2] just discusses the ability to
> > make the MemStore off-heap, but HBASE-15179 and its design doc[3] say
> that
> > the entire write stack is off-heap.
> >
> > Why this matters is if in a RegionObserver coproc hook (that's before the
> > MemStore commit) the mutation Cells can be assumed to be on-heap, then
> > clearing the internal family map of the mutation and replacing them with
> > new, altered Cells is safe. (Extra GC pressure aside, of course.) If
> not, I
> > presume the coproc would be leaking off-heap memory (unless there's magic
> > cleanup somewhere?)
> >
> > If this is not a safe assumption, what would the recommended way be to
> > alter a Cell's Tags in a coproc, since Tags are explicitly not exposed to
> > the HBase client, Cells are immutable, and hence the only way to do so
> > would be to create new Cells in a coproc? My question's not how to create
> > the new Cells (that's been answered elsewhere) but how to dispose of the
> > old, original ones.
> >
> > Also, if this is not a safe assumption, is there an accepted LP(Coproc)
> or
> > Public API that a coproc can check to see if it's in an "off-heap" mode
> or
> > not so that a leak can be avoided?
> >
> > Thanks,
> >
> > Geoffrey Jacoby
> >
> > References:
> > [1] https://github.com/apache/phoenix/pull/978
> > [2] https://hbase.apache.org/book.html#regionserver.offheap.writepath
> > [3]
> >
> >
> https://docs.google.com/document/d/1fj5P8JeutQ-Uadb29ChDscMuMaJqaMNRI86C4k5S1rQ/edit
> >
>

Re: HBase 2 slower than HBase 1?

[ramkrishna vasudevan]

Hi Bruno

Regarding the perf issue with more CPs you might be interested in seeing
this
https://issues.apache.org/jira/browse/HBASE-25277

Regards
Ram

On Thu, Nov 5, 2020 at 11:25 PM ramkrishna vasudevan <
ramkrishna.s.vasude...@gmail.com> wrote:

> This is great information. Thanks Bruno. Keep us posted if you find
> anything more when you have your isolated tests done.
>
> Regards
> Ram
>
> On Thu, Nov 5, 2020 at 10:36 PM Bruno Dumon  wrote:
>
>> Hi Ram,
>>
>> Sorry I missed your question. Yes, we do use addColumns.
>>
>> Specifically, we also query the Phoenix existence marker, which is a
>> column
>> named "_0", which is typically the last column in the row (we query this
>> through custom scans, not through Phoenix). It does seem that reading the
>> last column in the row does have an extra impact compared to HBase 1.2.
>>
>> I have been trying to further pin down our performance issues in an
>> isolated test but I have not yet arrived at anything conclusive. I'll try
>> to look into it further next week.
>>
>> I did also notice that HBase 2.x spends some more time iterating over the
>> coprocessors, and we quite a few of those (19, some of those are from
>> Phoenix, then some endpoint coprocessors and quite some region observers).
>> We don't do anything in the pre/postScannerNext but still it consumes
>> quite
>> some time during scanning. Replacing all the region observer coprocessors
>> by a single one does have a nice effect, 15-20% improvement, though I
>> still
>> need to test with a wider variety of queries.
>>
>>
>> On Fri, Oct 30, 2020 at 2:46 PM ramkrishna vasudevan <
>> ramkrishna.s.vasude...@gmail.com> wrote:
>>
>> > Hi Bruno/Jan
>> >
>> > Just a query here. I read your emails in this thread. One simple
>> question
>> > just to ensure if your tests were similar to Andrew's test. Did your
>> scan
>> > query have addColumns explicitly added covering all (or most of) the
>> > columns in the rows?
>> >
>> > Regards
>> > Ram
>> >
>> >
>> >
>> >
>> > On Wed, Jul 15, 2020 at 7:02 AM Andrew Purtell 
>> > wrote:
>> >
>> > > This is a good finding, nice work!
>> > > I added a comment on HBASE-24742 that mentions HBASE-24637 on the off
>> > > chance they are related, although I suspect more changes are
>> implicated
>> > by
>> > > the 2.x regression.
>> > >
>> > > On Tue, Jul 14, 2020 at 5:53 PM Bharath Vissapragada <
>> > bhara...@apache.org>
>> > > wrote:
>> > >
>> > > > FYI, we filed this today
>> > > https://issues.apache.org/jira/browse/HBASE-24742
>> > > > .
>> > > > We ran into a similar regression when upgrading from 1.3 based
>> branch
>> > to
>> > > > 1.6 based branch. After some profiling and code analysis we narrowed
>> > down
>> > > > the code paths.
>> > > >
>> > > > On Tue, Jul 14, 2020 at 11:38 AM Josh Elser 
>> wrote:
>> > > >
>> > > > > Wow. Great stuff, Andrew!
>> > > > >
>> > > > > Thank you for compiling and posting it all here. I can only
>> imagine
>> > how
>> > > > > time-consuming this was.
>> > > > >
>> > > > > On 6/26/20 1:57 PM, Andrew Purtell wrote:
>> > > > > > Hey Anoop, I opened
>> > > https://issues.apache.org/jira/browse/HBASE-24637
>> > > > > and
>> > > > > > attached the patches and script used to make the comparison.
>> > > > > >
>> > > > > > On Fri, Jun 26, 2020 at 2:33 AM Anoop John <
>> anoop.hb...@gmail.com>
>> > > > > wrote:
>> > > > > >
>> > > > > >> Great investigation Andy.  Do you know any Jiras which made
>> > changes
>> > > in
>> > > > > SQM?
>> > > > > >> Would be great if you can attach your patch which tracks the
>> scan
>> > > > > flow.  If
>> > > > > >> we have a Jira for this issue, can you pls attach?
>> > > > > >>
>> > > > > >> Anoop
>> > > > > >>
>> > > > > >> On Fri, Jun 26, 2020 at 1:56 AM Andrew Purtell <
>> > > > > andrew.purt...@gmail.com>
>> > >

Re: HBase 2 slower than HBase 1?

[ramkrishna vasudevan]

This is great information. Thanks Bruno. Keep us posted if you find
anything more when you have your isolated tests done.

Regards
Ram

On Thu, Nov 5, 2020 at 10:36 PM Bruno Dumon  wrote:

> Hi Ram,
>
> Sorry I missed your question. Yes, we do use addColumns.
>
> Specifically, we also query the Phoenix existence marker, which is a column
> named "_0", which is typically the last column in the row (we query this
> through custom scans, not through Phoenix). It does seem that reading the
> last column in the row does have an extra impact compared to HBase 1.2.
>
> I have been trying to further pin down our performance issues in an
> isolated test but I have not yet arrived at anything conclusive. I'll try
> to look into it further next week.
>
> I did also notice that HBase 2.x spends some more time iterating over the
> coprocessors, and we quite a few of those (19, some of those are from
> Phoenix, then some endpoint coprocessors and quite some region observers).
> We don't do anything in the pre/postScannerNext but still it consumes quite
> some time during scanning. Replacing all the region observer coprocessors
> by a single one does have a nice effect, 15-20% improvement, though I still
> need to test with a wider variety of queries.
>
>
> On Fri, Oct 30, 2020 at 2:46 PM ramkrishna vasudevan <
> ramkrishna.s.vasude...@gmail.com> wrote:
>
> > Hi Bruno/Jan
> >
> > Just a query here. I read your emails in this thread. One simple question
> > just to ensure if your tests were similar to Andrew's test. Did your scan
> > query have addColumns explicitly added covering all (or most of) the
> > columns in the rows?
> >
> > Regards
> > Ram
> >
> >
> >
> >
> > On Wed, Jul 15, 2020 at 7:02 AM Andrew Purtell 
> > wrote:
> >
> > > This is a good finding, nice work!
> > > I added a comment on HBASE-24742 that mentions HBASE-24637 on the off
> > > chance they are related, although I suspect more changes are implicated
> > by
> > > the 2.x regression.
> > >
> > > On Tue, Jul 14, 2020 at 5:53 PM Bharath Vissapragada <
> > bhara...@apache.org>
> > > wrote:
> > >
> > > > FYI, we filed this today
> > > https://issues.apache.org/jira/browse/HBASE-24742
> > > > .
> > > > We ran into a similar regression when upgrading from 1.3 based branch
> > to
> > > > 1.6 based branch. After some profiling and code analysis we narrowed
> > down
> > > > the code paths.
> > > >
> > > > On Tue, Jul 14, 2020 at 11:38 AM Josh Elser 
> wrote:
> > > >
> > > > > Wow. Great stuff, Andrew!
> > > > >
> > > > > Thank you for compiling and posting it all here. I can only imagine
> > how
> > > > > time-consuming this was.
> > > > >
> > > > > On 6/26/20 1:57 PM, Andrew Purtell wrote:
> > > > > > Hey Anoop, I opened
> > > https://issues.apache.org/jira/browse/HBASE-24637
> > > > > and
> > > > > > attached the patches and script used to make the comparison.
> > > > > >
> > > > > > On Fri, Jun 26, 2020 at 2:33 AM Anoop John <
> anoop.hb...@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > >> Great investigation Andy.  Do you know any Jiras which made
> > changes
> > > in
> > > > > SQM?
> > > > > >> Would be great if you can attach your patch which tracks the
> scan
> > > > > flow.  If
> > > > > >> we have a Jira for this issue, can you pls attach?
> > > > > >>
> > > > > >> Anoop
> > > > > >>
> > > > > >> On Fri, Jun 26, 2020 at 1:56 AM Andrew Purtell <
> > > > > andrew.purt...@gmail.com>
> > > > > >> wrote:
> > > > > >>
> > > > > >>> Related, I think I found a bug in branch-1 where we don’t
> > heartbeat
> > > > in
> > > > > >> the
> > > > > >>> filter all case until we switch store files, so scanning a very
> > > large
> > > > > >> store
> > > > > >>> file might time out with client defaults. Remarking on this
> here
> > > so I
> > > > > >> don’t
> > > > > >>> forget to follow up.
> > > > > >>>
> > > > > >>>> On Jun 25, 2020, at 12

Re: HBase 2 slower than HBase 1?

[ramkrishna vasudevan]

Hi Bruno/Jan

Just a query here. I read your emails in this thread. One simple question
just to ensure if your tests were similar to Andrew's test. Did your scan
query have addColumns explicitly added covering all (or most of) the
columns in the rows?

Regards
Ram




On Wed, Jul 15, 2020 at 7:02 AM Andrew Purtell  wrote:

> This is a good finding, nice work!
> I added a comment on HBASE-24742 that mentions HBASE-24637 on the off
> chance they are related, although I suspect more changes are implicated by
> the 2.x regression.
>
> On Tue, Jul 14, 2020 at 5:53 PM Bharath Vissapragada 
> wrote:
>
> > FYI, we filed this today
> https://issues.apache.org/jira/browse/HBASE-24742
> > .
> > We ran into a similar regression when upgrading from 1.3 based branch to
> > 1.6 based branch. After some profiling and code analysis we narrowed down
> > the code paths.
> >
> > On Tue, Jul 14, 2020 at 11:38 AM Josh Elser  wrote:
> >
> > > Wow. Great stuff, Andrew!
> > >
> > > Thank you for compiling and posting it all here. I can only imagine how
> > > time-consuming this was.
> > >
> > > On 6/26/20 1:57 PM, Andrew Purtell wrote:
> > > > Hey Anoop, I opened
> https://issues.apache.org/jira/browse/HBASE-24637
> > > and
> > > > attached the patches and script used to make the comparison.
> > > >
> > > > On Fri, Jun 26, 2020 at 2:33 AM Anoop John 
> > > wrote:
> > > >
> > > >> Great investigation Andy.  Do you know any Jiras which made changes
> in
> > > SQM?
> > > >> Would be great if you can attach your patch which tracks the scan
> > > flow.  If
> > > >> we have a Jira for this issue, can you pls attach?
> > > >>
> > > >> Anoop
> > > >>
> > > >> On Fri, Jun 26, 2020 at 1:56 AM Andrew Purtell <
> > > andrew.purt...@gmail.com>
> > > >> wrote:
> > > >>
> > > >>> Related, I think I found a bug in branch-1 where we don’t heartbeat
> > in
> > > >> the
> > > >>> filter all case until we switch store files, so scanning a very
> large
> > > >> store
> > > >>> file might time out with client defaults. Remarking on this here
> so I
> > > >> don’t
> > > >>> forget to follow up.
> > > >>>
> > >  On Jun 25, 2020, at 12:27 PM, Andrew Purtell  >
> > > >>> wrote:
> > > 
> > >  
> > >  I repeated this test with pe --filterAll and the results were
> > > >> revealing,
> > > >>> at least for this case. I also patched in thread local hash map for
> > > >> atomic
> > > >>> counters that I could update from code paths in SQM, StoreScanner,
> > > >>> HFileReader*, and HFileBlock. Because a RPC is processed by a
> single
> > > >>> handler thread I could update counters and accumulate micro-timings
> > via
> > > >>> System#nanoTime() per RPC and dump them out of CallRunner in some
> new
> > > >> trace
> > > >>> logging. I spent a couple of days making sure the instrumentation
> was
> > > >>> placed equivalently in both 1.6 and 2.2 code bases and was
> producing
> > > >>> consistent results. I can provide these patches upon request.
> > > 
> > >  Again, test tables with one family and 1, 5, 10, 20, 50, and 100
> > > >>> distinct column-qualifiers per row. After loading the table I made
> a
> > > >>> snapshot and cloned the snapshot for testing, for both 1.6 and 2.2,
> > so
> > > >> both
> > > >>> versions were tested using the exact same data files on HDFS. I
> also
> > > used
> > > >>> the 1.6 version of PE for both, so the only change is on the server
> > > (1.6
> > > >> vs
> > > >>> 2.2 masters and regionservers).
> > > 
> > >  It appears a refactor to ScanQueryMatcher and friends has disabled
> > the
> > > >>> ability of filters to provide SKIP hints, which prevents us from
> > > >> bypassing
> > > >>> version checking (so some extra cost in SQM), and appears to
> disable
> > an
> > > >>> optimization that avoids reseeking, leading to a serious and
> > > proportional
> > > >>> regression in reseek activity and time spent in that code path. So
> > for
> > > >>> queries that use filters, there can be a substantial regression.
> > > 
> > >  Other test cases that did not use filters did not show a
> regression.
> > > 
> > >  A test case where I used ROW_INDEX_V1 encoding showed an expected
> > > >> modest
> > > >>> proportional regression in seeking time, due to the fact it is
> > > optimized
> > > >>> for point queries and not optimized for the full table scan case.
> > > 
> > >  I will come back here when I understand this better.
> > > 
> > >  Here are the results for the pe --filterAll case:
> > > 
> > > 
> > >  1.6.0 c1  2.2.5 c1
> > >  1.6.0 c5  2.2.5 c5
> > >  1.6.0 c10 2.2.5 c10
> > >  1.6.0 c20 2.2.5 c20
> > >  1.6.0 c50 2.2.5 c50
> > >  1.6.0 c1002.2.5 c100
> > >  Counts
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > >  

Re: Time for 2.3.3

[ramkrishna vasudevan]

Hi Nick,
Apologies for the delayed reply.
If you see the current patch we are just ensuring that there is no
behaviour change by moving the archival to an async thread. So I don't
expect any behaviour change there. The existing behaviour was to abort the
server on an archival failure and now we are still doing it except the fact
that if needed user can configure a retry there. By default there is no
retry.
Yes the test related failures and in turn the changes required in the
postLogArchival has been taken care by Duo.
If you still feel it is too invasive then I will refrain from committing
the patch there. Thanks Nick.

Regards
Ram

On Wed, Oct 21, 2020 at 11:14 PM Nick Dimiduk  wrote:

> Hi Ram,
>
> Per my response on jira, I think this looks too invasive for a patch
> release, but I'm willing to be convinced otherwise. Looks like you and Duo
> were able to sort out the associated issue in HBASE-25186. Do you have a
> plan to test that extra WALs don't accumulate inadvertently?
>
> Thanks,
> Nick
>
> On Wed, Oct 21, 2020 at 9:57 AM ramkrishna vasudevan <
> ramkrishna.s.vasude...@gmail.com> wrote:
>
> > Hi Nick
> >
> > Can I port https://issues.apache.org/jira/browse/HBASE-25065 to
> branch-2.2
> > - Note that it is an improvement?
> >
> > Regards
> > Ram
> >
> > On Wed, Oct 21, 2020 at 10:05 PM Nick Dimiduk 
> wrote:
> >
> > > Hello team,
> > >
> > > It's come around to that time again. We have about 30 improvements on
> the
> > > branch, which is my litmus test for another patch release. Please try
> to
> > > land any patches you have in flight, and plan for an RC on or around
> > > Tuesday, 27 Oct.
> > >
> > > Thanks,
> > > Nick
> > >
> >
>

Re: Time for 2.3.3

[ramkrishna vasudevan]

Hi Nick

Can I port https://issues.apache.org/jira/browse/HBASE-25065 to branch-2.2
- Note that it is an improvement?

Regards
Ram

On Wed, Oct 21, 2020 at 10:05 PM Nick Dimiduk  wrote:

> Hello team,
>
> It's come around to that time again. We have about 30 improvements on the
> branch, which is my litmus test for another patch release. Please try to
> land any patches you have in flight, and plan for an RC on or around
> Tuesday, 27 Oct.
>
> Thanks,
> Nick
>

Re: Recommended way of using HBase Cell Tags.

[ramkrishna vasudevan]

Hi Rushabh

If I remember correctly, the decision was not to expose tags for clients
directly. All the tags were used as internal to the cell formation at the
server side (for eg ACL and Visibility labels).

For your case, is there a possibility to have yournew feature as a first
class feature using Tags? Just asking?

Regards
Ram

On Wed, Oct 14, 2020 at 8:17 PM Rushabh Shah
 wrote:

> Hi Everyone,
> I want to understand how to use the Hbase Cell Tags feature. We have a use
> case to identify the source of deletes (not the same as authenticated
> kerberos user). I have added more details about my use case in HBASE-25118
> . At my day job we use
> Phoenix to interact with hbase and we are passing this information via
> Phoenix ConnectionProperties. We are exploring the Cell Tags feature to add
> this metadata to Hbase Cells (only to Delete Markers as of now).
>
> Via HBASE-18995 , we
> have moved all the createCell methods which use Tag(s) as an argument to
> PrivateCellUtil class and made the InterfaceAudience of that class Private.
> I saw some discussion on that jira
> <
> https://issues.apache.org/jira/browse/HBASE-18995?focusedCommentId=16219960=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16219960
> >]
> to expose some methods as LimitedPrivate accessible to CP but was decided
> to do it later. We only expose CellBuilderFactory
> <
> https://github.com/apache/hbase/blob/master/hbase-common/src/main/java/org/apache/hadoop/hbase/CellBuilderFactory.java
> >
> which returns which returns an instance of CellBuilder
> <
> https://github.com/apache/hbase/blob/master/hbase-common/src/main/java/org/apache/hadoop/hbase/CellBuilder.java
> >
> which doesn't have a setTags method. Also the code is vastly different in
> branch-1.
>
> Could someone please educate me on how to populate tags from the client
> side (i.e Phoenix) while creating a Delete object ?
> Thank you !
>

Re: [DISCUSS]HBase2.1.0 is slower than HBase1.2.0

[ramkrishna vasudevan]

Oh great. Thanks for pointing that out. I think that is what is the exact
place that the perf bottleneck was found.

Regards
Ram


On Thu, Jun 11, 2020 at 4:29 PM 张铎(Duo Zhang)  wrote:

> Oh, good. I recall that there is a related issue but I just forget the
> title so I can not find it...
>
> Thanks for chimming in.
>
> OpenInx  于2020年6月11日周四 下午6:39写道：
>
> > Hi Zheng wang.
> >
> > Hope this issue will be helpful for you.
> > https://issues.apache.org/jira/browse/HBASE-21657
> > Thanks.
> >
> > On Tue, Jun 9, 2020 at 5:53 PM Anoop John  wrote:
> >
> > > Thanks for the detailed analysis and update zheng wang.
> > > >The code line below in StoreScanner.next() cost about 100ms in v2.1,
> and
> > > it added from v2.0, see HBASE-17647.
> > > So still there is some additional cost in 2.1 right? Do u have any
> other
> > > observation?  Are we doing more cell compares in 2.x?
> > >
> > > Anoop
> > >
> > >
> > > On Mon, Jun 8, 2020 at 1:50 AM zheng wang <18031...@qq.com> wrote:
> > >
> > > > Hi guys:
> > > >
> > > >
> > > > I did some test on my pc to find the reason as Jan Van Besien
> mentioned
> > > in
> > > > user channel.
> > > >
> > > >
> > > > #test env
> > > > OS : win10
> > > > JDK: 1.8
> > > > MEM: 8GB
> > > >
> > > >
> > > > #test data:
> > > > 1 million rows with only one columnfamily and one qualifier.
> > > >
> > > >
> > > > rowkey: rowkey-#index#
> > > > value: value-#index#
> > > >
> > > >
> > > > #test method:
> > > > just use client api to scan with default config several times, no pe,
> > no
> > > > ycsb
> > > >
> > > >
> > > > #test result(avg):
> > > > v1.2.0: 800ms
> > > > v2.1.0: 1050ms
> > > >
> > > >
> > > > So, it is sure that v2.1 is slower than v1.2, after this, i did some
> > > > statistics on regionserver.
> > > > Then i find the partly reason is related to the size estimated.
> > > >
> > > >
> > > > The code line below in StoreScanner.next() cost about 100ms in v2.1,
> > and
> > > > it added from v2.0, see HBASE-17647.
> > > > "int cellSize = PrivateCellUtil.estimatedSerializedSizeOf(cell);"
> > > >
> > > >
> > > > Should we support to disable the MaxResultSize limit(2MB by default
> > now)
> > > > to get more efficient if user exactly knows their data and could
> limit
> > > > results only by setBatch and setLimit?
> > >
> >
>

Re: [ANNOUNCE] Please welcome Lijin Bin to the HBase PMC

[ramkrishna vasudevan]

Congratulations Lijin Bin

On Tue, May 26, 2020, 6:11 PM Viraj Jasani  wrote:

> Congratulations, Lijin Bin.
>
> On 2020/05/25 14:22:12, Guanghao Zhang  wrote:
> > On behalf of the Apache HBase PMC I am pleased to announce that Lijin Bin
> > has accepted our invitation to become a PMC member on the Apache HBase
> > project. We appreciate Lijin Bin stepping up to take more responsibility
> in
> > the HBase project.
> >
> > Please join me in welcoming Lijin Bin to the HBase PMC!
> >
>

Re: Nightly builds are not running on branch-2.1

[ramkrishna vasudevan]

Oh my bad. I did not realize that. Thanks for letting me know.

On Wed, May 20, 2020, 1:54 PM Peter Somogyi  wrote:

> The branch-2.1 reached End of Life so the builds got disabled. The last
> release from branch-2.1 is 2.1.10.
>
> On Wed, May 20, 2020 at 9:48 AM ramkrishna vasudevan <
> ramkrishna.s.vasude...@gmail.com> wrote:
>
> > Hi All
> >
> > After we did some recent checkins for branch-2.1 and was waiting for the
> > nightly builds observed that the builds have not been running for last 10
> > days.
> > https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/lastBuild/
> >
> > Recently there was a compilation issue caused by  HBASE-24186 which I
> > reverted couple of days ago.
> >
> > Is there a way we can trigger the nightly build once again? Or it is some
> > other known issue as in branch-2.2 (where a recent discussion had
> > happened).
> >
> > Regards
> > Ram
> >
>

Nightly builds are not running on branch-2.1

[ramkrishna vasudevan]

Hi All

After we did some recent checkins for branch-2.1 and was waiting for the
nightly builds observed that the builds have not been running for last 10
days.
https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/lastBuild/

Recently there was a compilation issue caused by  HBASE-24186 which I
reverted couple of days ago.

Is there a way we can trigger the nightly build once again? Or it is some
other known issue as in branch-2.2 (where a recent discussion had happened).

Regards
Ram

Re: [ANNOUNCE] New HBase committer Wei-Chiu Chuang

[ramkrishna vasudevan]

Congratulations Wei-Chiu !!!

Regards
Ram

On Thu, May 14, 2020 at 10:55 AM Viraj Jasani  wrote:

> Congratulations Wei-Chiu !!
>
> On 2020/05/13 19:12:38, Sean Busbey  wrote:
> > Folks,
> >
> > On behalf of the Apache HBase PMC I am pleased to announce that Wei-Chiu
> > Chuang has accepted the PMC's invitation to become a committer on the
> > project.
> >
> > We appreciate all of the great contributions Wei-Chiu has made to the
> > community thus far and we look forward to his continued involvement.
> >
> > Allow me to be the first to congratulate Wei-Chiu on his new role!
> >
> > thanks,
> > busbey
> >
>

Re: [ANNOUNCE] New HBase committer Viraj Jasani

[ramkrishna vasudevan]

Congratulations viraj.

On Tue, Dec 31, 2019 at 10:59 AM Pankaj kr  wrote:

> Congratulations Viraj...!!
>
> Regards,
> Pankaj
>
> From:Anoop John 
> To:dev 
> Cc:hbase-user 
> Date:2019-12-31 10:15:07
> Subject:Re: [ANNOUNCE] New HBase committer Viraj Jasani
>
> Congrats Viraj !!!
>
> Anoop
>
> On Tue, Dec 31, 2019 at 9:26 AM Sukumar Maddineni
>  wrote:
>
> > Wow congrats Viraj and Keep up the good work.
> >
> > --
> > Sukumar
> >
> > On Mon, Dec 30, 2019 at 5:45 PM 宾莉金（binlijin） 
> wrote:
> >
> > > Welcome and Congratulations, Viraj!
> > >
> > > 张铎(Duo Zhang)  于2019年12月30日周一 下午1:18写道：
> > >
> > > > Congratulations!
> > > >
> > > > 李响  于2019年12月30日周一 上午10:12写道：
> > > >
> > > > >- Congratulations and warmly welcome \^o^/
> > > > >
> > > > >
> > > > > On Sun, Dec 29, 2019 at 2:14 AM Jan Hentschel <
> > > > > jan.hentsc...@ultratendency.com> wrote:
> > > > >
> > > > > > Congrats and welcome, Viraj! Well deserved.
> > > > > >
> > > > > > From: Peter Somogyi 
> > > > > > Reply-To: "dev@hbase.apache.org" 
> > > > > > Date: Friday, December 27, 2019 at 2:02 PM
> > > > > > To: HBase Dev List , hbase-user <
> > > > > > u...@hbase.apache.org>
> > > > > > Subject: [ANNOUNCE] New HBase committer Viraj Jasani
> > > > > >
> > > > > > On behalf of the Apache HBase PMC I am pleased to announce that
> > > > > > Viraj Jasani has accepted the PMC's invitation to become a
> > > > > > commiter on the project.
> > > > > >
> > > > > > Thanks so much for the work you've been contributing. We look
> > forward
> > > > > > to your continued involvement.
> > > > > >
> > > > > > Congratulations and welcome!
> > > > > >
> > > > > >
> > > > >
> > > > > --
> > > > >
> > > > >李响 Xiang Li
> > > > >
> > > > > 手机 cellphone ：+86-136-8113-8972
> > > > > 邮件 e-mail  ：wate...@gmail.com
> > > > >
> > > >
> > >
> > >
> > > --
> > > *Best Regards,*
> > >  lijin bin
> > >
> >
> >
> > --
> >
> > 
> >
>

Re: [ANNOUNCE] Please welcome Guangxu Cheng the HBase PMC

[ramkrishna vasudevan]

Welcome and congratulations  Guangxu!!

On Tue, Dec 10, 2019 at 2:48 AM Andrew Purtell  wrote:

> Congratulations and welcome Guangxu!
>
> On Mon, Dec 9, 2019 at 1:47 AM Duo Zhang  wrote:
>
> > On behalf of the Apache HBase PMC I am pleased to announce that Guangxu
> > Cheng has accepted our invitation to become a PMC member on the Apache
> > HBase project. We appreciate Guangxu Cheng stepping up to take more
> > responsibility in the HBase project.
> >
> > Please join me in welcoming Guangxu Cheng to the HBase PMC!
> >
>
>
> --
> Best regards,
> Andrew
>
> Words like orphans lost among the crosstalk, meaning torn from truth's
> decrepit hands
>- A23, Crosstalk
>

Re: [ANNOUNCE] new HBase committer Sakthi

[ramkrishna vasudevan]

Congratulations Sakthi !!!

On Thu, Aug 1, 2019 at 3:34 PM 张铎(Duo Zhang)  wrote:

> Congratulations!
>
> Pankaj kr  于2019年8月1日周四 下午5:56写道：
>
> > Congratulation Sakthi..!!
> >
> > Regards,
> > Pankaj
> >
> > -Original Message-
> > From: Sean Busbey [mailto:bus...@apache.org]
> > Sent: 01 August 2019 05:35
> > To: u...@hbase.apache.org; dev 
> > Subject: [ANNOUNCE] new HBase committer Sakthi
> >
> > On behalf of the HBase PMC, I'm pleased to announce that Sakthi has
> > accepted our invitation to become an HBase committer.
> >
> > We'd like to thank Sakthi for all of his diligent contributions to the
> > project thus far. We look forward to his continued participation in our
> > community.
> >
> > Congrats and welcome Sakthi!
> >
>

Re: Re: [Announce] 张铎 (Duo Zhang) is Apache HBase PMC chair

[ramkrishna vasudevan]

Congratulations Duo and thank you Misty.

Regards
Ram

On Tue, Jul 23, 2019 at 7:36 PM Toshihiro Suzuki 
wrote:

> Congratulations Duo! And Thank you Misty!
>
> On Tue, Jul 23, 2019 at 5:27 PM Chunhui Shen  wrote:
>
> > Thanks Misty  and Congratulations Duo!
> > Best Regards,
> > Chunhui
> >
> >
> >
> >
> >
> >
> >
> > 在 2019-07-23 10:19:58，"Francis Christopher Liu" 
> 写道：
> > >Thanks a lot Misty!
> > >
> > >And congratulations Duo!
> > >
> > >Francis
> > >
> > >On Sun, Jul 21, 2019 at 8:27 PM Yu Li  wrote:
> > >
> > >> Congratulations Duo! And thank you Misty!
> > >>
> > >> Best Regards,
> > >> Yu
> > >>
> > >>
> > >> On Sun, 21 Jul 2019 at 11:21, Allan Yang  wrote:
> > >>
> > >> > Congratulations Duo!
> > >> > Best Regards
> > >> > Allan Yang
> > >> >
> > >> >
> > >> > 宾莉金（binlijin）  于2019年7月21日周日 上午10:05写道：
> > >> >
> > >> > > Congratulations Duo!  and thanks Misty.
> > >> > >
> > >> > > Anoop Sam John  于2019年7月21日周日 上午9:26写道：
> > >> > >
> > >> > > > Congrats Duo.
> > >> > > >
> > >> > > > Thanks Misty for your great work as the PMC chair.
> > >> > > >
> > >> > > > Anoop
> > >> > > >
> > >> > > > On Sat, Jul 20, 2019 at 12:07 AM Xu Cang 
> > wrote:
> > >> > > >
> > >> > > > > Thank you Misty!
> > >> > > > > Congratulations Duo, thanks for taking extra work!
> > >> > > > >
> > >> > > > > On Fri, Jul 19, 2019 at 11:23 AM Zach York <
> > >> > > zyork.contribut...@gmail.com
> > >> > > > >
> > >> > > > > wrote:
> > >> > > > >
> > >> > > > > > Congratulations Duo! Thanks for offering to take on the
> > >> additional
> > >> > > > work!
> > >> > > > > >
> > >> > > > > > On Fri, Jul 19, 2019 at 10:34 AM Stack 
> > wrote:
> > >> > > > > >
> > >> > > > > > > Thank you Misty for your years of service (FYI, for
> > non-PMCers,
> > >> > the
> > >> > > > > > reports
> > >> > > > > > > Misty wrote to the Apache Board on our behalf were
> > repeatedly
> > >> > > called
> > >> > > > > out
> > >> > > > > > > for their quality and thoughtfulness).
> > >> > > > > > >
> > >> > > > > > > Duo Zhang, thank you for taking on the mantle.
> > >> > > > > > >
> > >> > > > > > > S
> > >> > > > > > >
> > >> > > > > > > On Thu, Jul 18, 2019 at 10:46 AM Misty Linville <
> > >> > mi...@apache.org>
> > >> > > > > > wrote:
> > >> > > > > > >
> > >> > > > > > > > Each Apache project has a project management committee
> > (PMC)
> > >> > that
> > >> > > > > > > oversees
> > >> > > > > > > > governance of the project, votes on new committers and
> PMC
> > >> > > members,
> > >> > > > > and
> > >> > > > > > > > ensures that the software we produce adheres to the
> > standards
> > >> > of
> > >> > > > the
> > >> > > > > > > > Foundation. One of the roles on the PMC is the PMC
> chair.
> > The
> > >> > PMC
> > >> > > > > chair
> > >> > > > > > > > represents the project as a Vice President of the
> > Foundation
> > >> > and
> > >> > > > > > > > communicates to the board about the project's health,
> once
> > >> per
> > >> > > > > quarter
> > >> > > > > > > and
> > >> > > > > > > > at other times as needed.
> > >> > > > > > > >
> > >> > > > > > > > It's been my honor to serve as your PMC chair since
> 2017,
> > >> when
> > >> > I
> > >> > > > took
> > >> > > > > > > over
> > >> > > > > > > > from Andrew Purtell. I've decided to step back from my
> > >> > volunteer
> > >> > > > ASF
> > >> > > > > > > > activities to leave room in my life for other things.
> The
> > >> HBase
> > >> > > PMC
> > >> > > > > > > > nominated Duo for this role, and Duo has kindly agreed!
> > The
> > >> > board
> > >> > > > > > passed
> > >> > > > > > > > this resolution in its meeting yesterday[1] and it is
> > already
> > >> > > > > > > official[2].
> > >> > > > > > > > Congratulations, Duo, and thank you for continuing to
> > honor
> > >> the
> > >> > > > > project
> > >> > > > > > > > with your dedication.
> > >> > > > > > > >
> > >> > > > > > > > Misty
> > >> > > > > > > >
> > >> > > > > > > > [1] The minutes have not yet posted at the time of this
> > >> email,
> > >> > > but
> > >> > > > > will
> > >> > > > > > > be
> > >> > > > > > > > available at
> > >> > > > http://www.apache.org/foundation/records/minutes/2019/.
> > >> > > > > > > > [2] https://www.apache.org/foundation/#who-runs-the-asf
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> > >
> > >> > > --
> > >> > > *Best Regards,*
> > >> > >  lijin bin
> > >> > >
> > >> >
> > >>
> >
>

Re: [VOTE] Merge branch HBASE-21879 (Reading HFile's Block to ByteBuffer directly) back to master.

[ramkrishna vasudevan]

+1 to merge to master. Great job Zheng

On Sat, Jun 22, 2019, 8:41 AM Guanghao Zhang  wrote:

> +1 for merge this to master.
>
> OpenInx  于2019年6月21日周五 下午2:56写道：
>
> > Update:
> >
> > The ByteBuffer pread backport is under reviewing now.
> > https://github.com/apache/hadoop/pull/997
> >
> > As Hadoop team said,  the Hadoop 2.8 will be EOL soon, so our HDFS team
> > will backport this patch to
> > branch-2 & branch-2.9,  we may need to upgrade the hadoop dependencies
> from
> > 2.8.5 to 2.9.3 in future.
> >
> > Thanks.
> >
> > On Wed, Jun 19, 2019 at 10:41 PM OpenInx  wrote:
> >
> > > Thanks for your reviewing and flaky test checking, Duo.
> > > Will file a separate issue to address your comment if necessary.
> > >
> > > Thanks.
> > >
> > > On Wed, Jun 19, 2019 at 9:55 PM 张铎(Duo Zhang) 
> > > wrote:
> > >
> > >> +1 from me.
> > >>
> > >> Left a few comments on github PR, not big problems. And the flaky
> > >> dashboard
> > >> is pretty good.
> > >>
> > >>
> > >>
> >
> https://builds.apache.org/job/HBASE-Find-Flaky-Tests/job/HBASE-21879/lastSuccessfulBuild/artifact/dashboard.html
> > >>
> > >>
> > >> The TestConnectionImplementation was also failing on master, and was
> > fixed
> > >> after merging back HBASE-21512.
> > >>
> > >> 张铎(Duo Zhang)  于2019年6月18日周二 下午9:48写道：
> > >>
> > >> > Good. Will take a look soon.
> > >> >
> > >> > OpenInx  于2019年6月18日周二 下午9:41写道：
> > >> >
> > >> >> > Could please open a PR, just like what I have done for
> HBASE-21512,
> > >> so
> > >> >> that others could have a overall view on the modified code?
> > >> >>
> > >> >> OK,  created a PR for this:
> https://github.com/apache/hbase/pull/320
> > >> >> Thanks for your suggestion, Duo.
> > >> >>
> > >> >> On Tue, Jun 18, 2019 at 9:24 PM 张铎(Duo Zhang) <
> palomino...@gmail.com
> > >
> > >> >> wrote:
> > >> >>
> > >> >> > The performance number is great.
> > >> >> >
> > >> >> > Could please open a PR, just like what I have done for
> HBASE-21512,
> > >> so
> > >> >> that
> > >> >> > others could have a overall view on the modified code?
> > >> >> >
> > >> >> > Thanks.
> > >> >> >
> > >> >> > OpenInx  于2019年6月18日周二 下午6:58写道：
> > >> >> >
> > >> >> > > BTW,  when testing this branch,  we found some performance
> issues
> > >> >> about
> > >> >> > > HDFS Client:
> > >> >> > > 1.  we reduced the DFS client's heap allocation from 45% to 27%
> > >> >> > > in HDFS-14535 [1];
> > >> >> > > 2.  we also increased get throughput by 17.8% in disabled block
> > >> cache
> > >> >> > case
> > >> >> > > in HDFS-14541[2].
> > >> >> > >  In theory, it should also helps a lot (especially
> p99/p999)
> > >> even
> > >> >> if
> > >> >> > RS
> > >> >> > > has a high cacheHitRatio.
> > >> >> > >
> > >> >> > > I think the next HDFS 2.8 release will include those patches,
> > >> they're
> > >> >> > very
> > >> >> > > good points for our
> > >> >> > > HBase performance.
> > >> >> > >
> > >> >> > > [1]. https://issues.apache.org/jira/browse/HDFS-14535
> > >> >> > > [2].
> > >> >> > >
> > >> >> > >
> > >> >> >
> > >> >>
> > >>
> >
> https://issues.apache.org/jira/browse/HDFS-14541?focusedCommentId=16866472=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16866472
> > >> >> > >
> > >> >> > > Thanks.
> > >> >> > >
> > >> >> > > On Tue, Jun 18, 2019 at 12:05 PM OpenInx 
> > >> wrote:
> > >> >> > >
> > >> >> > > > The HBASE-21879 has lots of changes: 123 files changed, 5833
> > >> >> > > > insertions(+), 3015 deletions(-).
> > >> >> > > > Currently we developed this issue based on master branch, and
> > >> >> expect to
> > >> >> > > > release it in future HBase3.x.
> > >> >> > > > Of course, if branch-2 want this feature we can do the
> > backport,
> > >> >> should
> > >> >> > > > have some conflicts now but I
> > >> >> > > > don't think it would be hard to fix because I believe the
> > >> branch-2
> > >> >> > > > shouldn't have so much diff with master now
> > >> >> > > > (at least in read path).
> > >> >> > > > The first priority thing for now,   I think it would be
> merging
> > >> the
> > >> >> > > > HBASE-21879 branch to master branch
> > >> >> > > > before diverging.  After that, I can do the backport.
> > >> >> > > >
> > >> >> > > > Thanks for your suggestion, Guanghao !
> > >> >> > > >
> > >> >> > > >
> > >> >> > > >
> > >> >> > > > On Tue, Jun 18, 2019 at 11:39 AM Guanghao Zhang <
> > >> zghao...@gmail.com
> > >> >> >
> > >> >> > > > wrote:
> > >> >> > > >
> > >> >> > > >> This is a improvement not a new feature? So backport to
> > >> branch-2,
> > >> >> too?
> > >> >> > > >>
> > >> >> > > >> OpenInx  于2019年6月17日周一 下午2:45写道：
> > >> >> > > >>
> > >> >> > > >> > Dear HBase dev:
> > >> >> > > >> >
> > >> >> > > >> > In HBASE-21879[1], we redesigned the offheap read path:
> read
> > >> the
> > >> >> > > >> HFileBlock
> > >> >> > > >> > from HDFS to pooled offheap
> > >> >> > > >> > ByteBuffers directly, while before HBASE-21879 we just
> read
> > >> the
> > >> >> > > >> HFileBlock
> > >> >> > > >> > to heap which would still lead
> 

Re: Adding a new balancer to HBase

[ramkrishna vasudevan]

Seems Clay's idea is also a very good idea. that would also make impl much
simpler and the focus would only be on cost functions.

Regards
Ram

On Fri, Jun 21, 2019 at 10:21 AM Anoop John  wrote:

> Same Q as Clay asked.  We can see..
>
> Also generically we can not consider like only one table in cluster.  At
> top level we give options like balance per table level or per cluster level
> only.  This also should be considered for the new balancer also IMO.  Ya if
> it can work with cost function change alone, it will be much smaller
> change.  On high level am +1 for such a simple way to handle the
> heterogeneous nodes cluster.
>
> Anoop
>
> On Fri, Jun 21, 2019 at 5:15 AM Clay Baenziger (BLOOMBERG/ 731 LEX) <
> cbaenzi...@bloomberg.net> wrote:
>
> > Could it work to have the stochastic load balancer use pluggable cost
> > functions[1]? Then, could this type of a load balancer be implemented
> > simply as a new cost function which folks could choose to load and mix
> with
> > the others?
> >
> > -Clay
> >
> > [1]: Instead of this static list of cost functions?
> >
> https://github.com/apache/hbase/blob/baf3ae80f5588ee848176adefc9f56818458a387/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java#L198
> >
> > From: dev@hbase.apache.org At: 06/20/19 12:54:23To:
> dev@hbase.apache.org
> > Subject: Re: Adding a new balancer to HBase
> >
> > Bonjour Pierre,
> >
> > Some time ago I build (for my own purposes) something similar that I
> called
> > "LoadBasedLoadBalancer" that moves the regions based on my servers load
> and
> > capacity. The load balancer is querying the region servers to get the
> > number of cores, the allocated heap, the 5 minutes average load, etc. and
> > balanced the regions based on that.
> >
> > I felt that need already years ago. What you are proposing is a
> simplified
> > version that will most probably be more stable and easier to implement. I
> > will be happy to assist you in the process or getting that into HBase.
> >
> > Have you already opened the JIRA to support that?
> >
> > Thanks,
> >
> > JMS
> >
> > Le jeu. 20 juin 2019 à 01:11, ramkrishna vasudevan <
> > ramkrishna.s.vasude...@gmail.com> a écrit :
> >
> > > Seems a very good idea for cloud servers. Pls feel free to raise a JIRA
> > and
> > > contribute your patch.
> > >
> > > Regards
> > > Ram
> > >
> > > On Tue, Jun 18, 2019 at 8:09 AM 刘新星  wrote:
> > >
> > > >
> > > >
> > > > I'm interested on this. It sounds like a weighted load balancer and
> > > > valuable for those users deploy their hbase cluster on cloud servers.
> > > > You can create a jira and make a patch for better discussion.
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > At 2019-06-18 05:00:54, "Pierre Zemb" 
> > > wrote:
> > > > >Hi!
> > > > >
> > > > >My name is Pierre, I'm working at OVH, an European cloud-provider.
> Our
> > > > >team, Observability, is heavily relying on HBase to store telemetry.
> > We
> > > > >would like to open the discussion about adding into 1.4X and 2.X a
> new
> > > > >Balancer.
> > > > ><
> > > >
> > >
> >
> https://gist.github.com/PierreZ/15560e12c147e661e5c1b5f0edeb9282#our-situation
> > > > >Our
> > > > >situation
> > > > >
> > > > >The Observability team in OVH is responsible to handle logs and
> > metrics
> > > > >from all servers/applications/equipments within OVH. HBase is used
> as
> > > the
> > > > >datastore for metrics. We are using an open-source software called
> > > Warp10
> > > > ><https://warp10.io> to handle all the metrics coming from OVH's
> > > > >infrastructure. We are operating three HBase 1.4 clusters, including
> > one
> > > > >with 218 RegionServers which is growing every month.
> > > > >
> > > > >We found out that *in our usecase*(single table, dedicated HBase and
> > > > Hadoop
> > > > >tuned for our usecase, good key distribution)*, the number of
> regions
> > > per
> > > > >RS was the real limit for us*.
> > > > >
> > > > >Over the years, due to historical reasons and also th

Re: Adding a new balancer to HBase

[ramkrishna vasudevan]

Seems a very good idea for cloud servers. Pls feel free to raise a JIRA and
contribute your patch.

Regards
Ram

On Tue, Jun 18, 2019 at 8:09 AM 刘新星  wrote:

>
>
> I'm interested on this. It sounds like a weighted load balancer and
> valuable for those users deploy their hbase cluster on cloud servers.
> You can create a jira and make a patch for better discussion.
>
>
>
>
>
>
>
> At 2019-06-18 05:00:54, "Pierre Zemb"  wrote:
> >Hi!
> >
> >My name is Pierre, I'm working at OVH, an European cloud-provider. Our
> >team, Observability, is heavily relying on HBase to store telemetry. We
> >would like to open the discussion about adding into 1.4X and 2.X a new
> >Balancer.
> ><
> https://gist.github.com/PierreZ/15560e12c147e661e5c1b5f0edeb9282#our-situation
> >Our
> >situation
> >
> >The Observability team in OVH is responsible to handle logs and metrics
> >from all servers/applications/equipments within OVH. HBase is used as the
> >datastore for metrics. We are using an open-source software called Warp10
> > to handle all the metrics coming from OVH's
> >infrastructure. We are operating three HBase 1.4 clusters, including one
> >with 218 RegionServers which is growing every month.
> >
> >We found out that *in our usecase*(single table, dedicated HBase and
> Hadoop
> >tuned for our usecase, good key distribution)*, the number of regions per
> >RS was the real limit for us*.
> >
> >Over the years, due to historical reasons and also the need to benchmark
> >new machines, we ended-up with differents groups of hardware: some servers
> >can handle only 180 regions, whereas the biggest can handle more than 900.
> >Because of such a difference, we had to disable the LoadBalancing to avoid
> >the roundRobinAssigmnent. We developed some internal tooling which are
> >responsible for load balancing regions across RegionServers. That was 1.5
> >year ago.
> >
> >Today, we are thinking about fully integrate it within HBase, using the
> >LoadBalancer interface. We started working on a new Balancer called
> >HeterogeneousBalancer, that will be able to fullfill our need.
> ><
> https://gist.github.com/PierreZ/15560e12c147e661e5c1b5f0edeb9282#how-does-it-works
> >How
> >does it works?
> >
> >A rule file is loaded before balancing. It contains lines of rules. A rule
> >is composed of a regexp for hostname, and a limit. For example, we could
> >have:
> >
> >rs[0-9] 200
> >rs1[0-9] 50
> >
> >RegionServers with hostname matching the first rules will have a limit of
> >200, and the others 50. If there's no match, a default is set.
> >
> >Thanks to the rule, we have two informations: the max number of regions
> for
> >this cluster, and the rules for each servers. HeterogeneousBalancer will
> >try to balance regions according to their capacity.
> >
> >Let's take an example. Let's say that we have 20 RS:
> >
> >   - 10 RS, named through rs0 to rs9 loaded with 60 regions each, and each
> >   can handle 200 regions.
> >   - 10 RS, named through rs10 to rs19 loaded with 60 regions each, and
> >   each can support 50 regions.
> >
> >Based on the following rules:
> >
> >rs[0-9] 200
> >rs1[0-9] 50
> >
> >The second group is overloaded, whereas the first group has plenty of
> space.
> >
> >We know that we can handle at maximum *2500 regions* (200*10 + 50*10) and
> >we have currently *1200 regions* (60*20). HeterogeneousBalancer will
> >understand that the cluster is *full at 48.0%* (1200/2500). Based on this
> >information, we will then *try to put all the RegionServers to ~48% of
> load
> >according to the rules.* In this case, it will move regions from the
> second
> >group to the first.
> >
> >The balancer will:
> >
> >   - compute how many regions needs to be moved. In our example, by moving
> >   36 regions on rs10, we could go from 120.0% to 46.0%
> >   - select regions with lowest data-locality
> >   - try to find an appropriate RS for the region. We will take the lowest
> >   available RS.
> >
> ><
> https://gist.github.com/PierreZ/15560e12c147e661e5c1b5f0edeb9282#current-status
> >Current
> >status
> >
> >We started the implementation, but it is not finished yet. we are planning
> >to deploy it on a cluster with lower impact for testing, and then put it
> on
> >our biggest cluster.
> >
> >We have some basic implementation of all methods, but we need to add more
> >tests and make the code more robust. You can find the proof-of-concept
> here
> ><
> https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java
> >,
> >and some early tests here
> ><
> https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java
> >,
> >here
> ><
> https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerBalance.java
> >,
> >and here
> ><
> 

Re: [ANNOUNCE] New Committer: Wellington Chevreuil

[ramkrishna vasudevan]

Congratulations Wellington !!

Regards
Ram

On Fri, Jun 7, 2019 at 5:28 PM Sean Busbey  wrote:

> congrats Wellington!
>
> On 2019/06/07 10:11:48, Peter Somogyi  wrote:
> > On behalf of the HBase PMC, I'm pleased to announce that Wellington
> > Chevreuil has accepted our invitation to become an HBase committer.
> >
> > Thanks for all your hard work Wellington; we look forward to more
> > contributions!
> >
> > Please join me in extending congratulations to Wellington!
> >
>

Re: [ANNOUNCE] Please welcome Jan Hentschel to the Apache HBase PMC

[ramkrishna vasudevan]

Congratulations Jan !!

On Thu, May 9, 2019 at 9:33 AM Yu Li  wrote:

> Congratulations and welcome, Jan!
>
> Best Regards,
> Yu
>
>
> On Thu, 9 May 2019 at 10:06, OpenInx  wrote:
>
> > Congratulation, Jan! Thanks for your work.
> >
> > On Thu, May 9, 2019 at 6:08 AM Artem Ervits 
> wrote:
> >
> > > Well deserved Jan!
> > >
> > > On Wed, May 8, 2019, 5:37 PM Sean Busbey  wrote:
> > >
> > > > On behalf of the Apache HBase PMC I am pleased to announce that Jan
> > > > Hentschel has accepted our invitation to become a PMC member on the
> > > > HBase project. We appreciate Jan stepping up to take more
> > > > responsibility in the HBase project.
> > > >
> > > > Please join me in welcoming Jan to the HBase PMC!
> > > >
> > > >
> > > >
> > > > As a reminder, if anyone would like to nominate another person as a
> > > > committer or PMC member, even if you are not currently a committer or
> > > > PMC member, you can always drop a note to priv...@hbase.apache.org
> to
> > > > let us know.
> > > >
> > > > -busbey
> > > >
> > >
> >
>

Re: Replacement for CellUtil.setTimestamp()

[ramkrishna vasudevan]

The later JIRAs like https://issues.apache.org/jira/browse/HBASE-19092
exposed a CellBuilder and the CP has access to a the RawCellBuilder which
will allow you to do a setTimeStamp on it.
But note that it allows does a deep copy of the Cell bytes when the new
cell is created out of the builder.

Let us know if this helps or you need some other APIs to make things easier.

Regards
Ram



On Mon, Apr 1, 2019 at 2:33 AM Wellington Chevreuil <
wellington.chevre...@gmail.com> wrote:

> You could risk use PrivateCellUtil.setTimestamp() method, but it may break
> in future releases since PrivateCellUtil is marked
> as @InterfaceAudience.Private. Another option, given both Put and Delete
> map to KeyValue cell type, is to check if the Cell type is KeyValue, then
> do a cast to access setTimestamp method on the KV.
>
> Em sex, 29 de mar de 2019 às 22:34, Thomas D'Silva
>  escreveu:
>
> > In HBase 2.0 since CellUtil.setTimestamp has been deprecated, what is the
> > correct way to set the timestamp of a cell from a coprocessor? Phoenix
> uses
> > this API to set the timestamp of a call in our mutable indexing
> coprocessor
> > (see https://issues.apache.org/jira/browse/PHOENIX-5219).
> >
> > Thanks,
> > Thomas
> >
>

Re: [ANNOUNCE] Please welcome Peter Somogyi to the HBase PMC

[ramkrishna vasudevan]

Congratulations Peter.

On Tue, Jan 22, 2019 at 11:48 AM Tamas Penzes 
wrote:

> Congrats Peter!
>
> On Tue, Jan 22, 2019, 02:36 Duo Zhang 
> > On behalf of the Apache HBase PMC I am pleased to announce that Peter
> > Somogyi
> > has accepted our invitation to become a PMC member on the Apache HBase
> > project.
> > We appreciate Peter stepping up to take more responsibility in the HBase
> > project.
> >
> > Please join me in welcoming Peter to the HBase PMC!
> >
>

Re: [ANNOUNCE] Allan Yang joins the Apache HBase PMC

[ramkrishna vasudevan]

Congratulations Allan !!

On Thu, Nov 29, 2018 at 2:23 PM Balazs Meszaros
 wrote:

> Congratulations, Allan!
>
> On Thu, Nov 29, 2018 at 7:14 AM Pankaj kr  wrote:
>
> > Congratulations Allan..!! :)
> >
> > Regards,
> > Pankaj
> >
> > -Original Message-
> > From: Yu Li [mailto:car...@gmail.com]
> > Sent: 28 November 2018 21:41
> > To: Hbase-User ; dev 
> > Subject: [ANNOUNCE] Allan Yang joins the Apache HBase PMC
> >
> > On behalf of the Apache HBase PMC I am pleased to announce that Allan
> Yang
> > has accepted our invitation to become a PMC member on the Apache HBase
> > project. We appreciate Allan stepping up to take more responsibility in
> the
> > HBase project.
> >
> > Please join me in welcoming Allan to the HBase PMC!
> >
> > Best Regards,
> > Yu
> >
>

Re: [ANNOUNCE] Please welcome Zach York to the HBase PMC

[ramkrishna vasudevan]

Congratulations Zach!!!

On Thu, Nov 15, 2018 at 3:10 AM Misty Linville  wrote:

> Thanks for your continuing and increasing commitment to the project and the
> community!
>
> On Wed, Oct 17, 2018, 11:14 PM OpenInx 
> > Congratulations Zach!
> >
> > On Wed, Oct 17, 2018 at 8:14 PM ramkrishna vasudevan <
> > ramkrishna.s.vasude...@gmail.com> wrote:
> >
> > > Congratulations Zach!!!
> > >
> > > On Wed, Oct 17, 2018 at 11:15 AM Yu Li  wrote:
> > >
> > > > Congratulations and welcome, Zach!
> > > >
> > > > Best Regards,
> > > > Yu
> > > >
> > > >
> > > > On Wed, 17 Oct 2018 at 13:02, Pankaj kr 
> wrote:
> > > >
> > > > > Congratulations Zach..!! :)
> > > > >
> > > > > Regards,
> > > > > Pankaj
> > > > >
> > > > > -Original Message-
> > > > > From: Sean Busbey [mailto:bus...@apache.org]
> > > > > Sent: 12 October 2018 01:32
> > > > > To: dev 
> > > > > Subject: [ANNOUNCE] Please welcome Zach York to the HBase PMC
> > > > >
> > > > > On behalf of the Apache HBase PMC I am pleased to announce that
> Zach
> > > York
> > > > > has accepted our invitation to become a PMC member on the Apache
> > HBase
> > > > > project. We appreciate Zach stepping up to take more responsibility
> > in
> > > > the
> > > > > HBase project.
> > > > >
> > > > > Please join me in welcoming Zach to the HBase PMC!
> > > > >
> > > > > As a reminder, if anyone would like to nominate another person as a
> > > > > committer or PMC member, even if you are not currently a committer
> or
> > > PMC
> > > > > member, you can always drop a note to priv...@hbase.apache.org to
> > let
> > > us
> > > > > know.
> > > > >
> > > >
> > >
> >
>

Re: [ANNOUNCE] New HBase committer Jingyun Tian

[ramkrishna vasudevan]

Congratulations Jingyun !!

On Thu, Nov 15, 2018 at 3:08 AM Misty Linville  wrote:

> Congratulations, and thanks for opting to spend more time making HBase the
> software and the project better!
>
> On Wed, Nov 14, 2018, 3:26 AM Anoop John 
> > Congrats Jingyun !!!
> >
> > -Anoop-
> >
> > On Wed, Nov 14, 2018 at 8:18 AM Jingyun Tian 
> wrote:
> >
> > > Thank you all!
> > >
> > > Sincerely,
> > > Jingyun Tian
> > >
> > > On Wed, Nov 14, 2018 at 8:59 AM stack  wrote:
> > >
> > > > Welcome jingyun.
> > > > S
> > > >
> > > > On Mon, Nov 12, 2018, 11:54 PM 张铎(Duo Zhang)  > > wrote:
> > > >
> > > > > On behalf of the Apache HBase PMC, I am pleased to announce that
> > > Jingyun
> > > > > Tian has accepted the PMC's invitation to become a committer on the
> > > > > project. We appreciate all of Jingyun's generous contributions thus
> > far
> > > > and
> > > > > look forward to his continued involvement.
> > > > >
> > > > > Congratulations and welcome, Jingyun!
> > > > >
> > > >
> > >
> >
>

Re: [ANNOUNCE] Please welcome Zach York to the HBase PMC

[ramkrishna vasudevan]

Congratulations Zach!!!

On Wed, Oct 17, 2018 at 11:15 AM Yu Li  wrote:

> Congratulations and welcome, Zach!
>
> Best Regards,
> Yu
>
>
> On Wed, 17 Oct 2018 at 13:02, Pankaj kr  wrote:
>
> > Congratulations Zach..!! :)
> >
> > Regards,
> > Pankaj
> >
> > -Original Message-
> > From: Sean Busbey [mailto:bus...@apache.org]
> > Sent: 12 October 2018 01:32
> > To: dev 
> > Subject: [ANNOUNCE] Please welcome Zach York to the HBase PMC
> >
> > On behalf of the Apache HBase PMC I am pleased to announce that Zach York
> > has accepted our invitation to become a PMC member on the Apache HBase
> > project. We appreciate Zach stepping up to take more responsibility in
> the
> > HBase project.
> >
> > Please join me in welcoming Zach to the HBase PMC!
> >
> > As a reminder, if anyone would like to nominate another person as a
> > committer or PMC member, even if you are not currently a committer or PMC
> > member, you can always drop a note to priv...@hbase.apache.org to let us
> > know.
> >
>

Re: Making flakey tests run again

[ramkrishna vasudevan]

Thanks for the information Sean. I will then commit the changes and watch
the flaky tests build.

Regards
Ram

On Tue, Sep 18, 2018 at 11:20 AM Sean Busbey  wrote:

> Unfortunately there is not currently a way to have tests that are in
> the flaky list run in the automated precommit tests[1].
>
> I suggest you test the change locally and then commit. When the test
> no longer fails in the flaky test runner and ages out of what the
> flaky detector looks at[2], it'll start showing up in precommit again.
>
> [1]:
>
> https://issues.apache.org/jira/browse/HBASE-19265
>
> [2]:
>
>
> https://builds.apache.org/job/HBase-Find-Flaky-Tests/job/master/lastSuccessfulBuild/artifact/dashboard.html
> On Mon, Sep 17, 2018 at 11:59 PM ramkrishna vasudevan
>  wrote:
> >
> > Hi
> >
> > As part of HBASE-21102 the test case that was added was reported as Flaky
> > test. I had fixed the code such that the test case is not flaky any more.
> >
> > But when we submit the patch for precommit - it still avoids that test
> case
> > since it is marked flaky.
> >
> > I can see in the console output that
> > DOCKER_EXTRAARGS=--env=EXCLUDE_TESTS_URL=
> >
> https://builds.apache.org/job/HBase-Find-Flaky-Tests/job/master/lastSuccessfulBuild/artifact/excludes/
> >
> > The test TestServerCrashProcedureWithReplicas has been marked in the
> > excludes file and hence it is getting skipped. Is there a way to make it
> > run again?
> >
> > I tried asking the question in the JIRA but thought this is the best
> place
> > to ask. I did check the dev-support folder and the hbase-personality.sh
> > file but was not sure if there was a way to include that flaky test back
> > again to ensure it starts running again.
> >
> > Regards
> > Ram
>

Making flakey tests run again

[ramkrishna vasudevan]

Hi

As part of HBASE-21102 the test case that was added was reported as Flaky
test. I had fixed the code such that the test case is not flaky any more.

But when we submit the patch for precommit - it still avoids that test case
since it is marked flaky.

I can see in the console output that
DOCKER_EXTRAARGS=--env=EXCLUDE_TESTS_URL=
https://builds.apache.org/job/HBase-Find-Flaky-Tests/job/master/lastSuccessfulBuild/artifact/excludes/

The test TestServerCrashProcedureWithReplicas has been marked in the
excludes file and hence it is getting skipped. Is there a way to make it
run again?

I tried asking the question in the JIRA but thought this is the best place
to ask. I did check the dev-support folder and the hbase-personality.sh
file but was not sure if there was a way to include that flaky test back
again to ensure it starts running again.

Regards
Ram

Re: Pictures, Videos and Slides for HBaseConAsia2018

[ramkrishna vasudevan]

Thanks to Yu Li and Stack for enabling the videos and slides.

Regards
Ram

On Sat, Sep 1, 2018 at 12:15 PM Yu Li  wrote:

> Thank you for all the helps Stack! It must have cost lots of your time
> downloading the videos then uploading to youtube, uploading slides onto
> slideshare, and put up all together into the blog!
>
> The success of the conference is attributed to all PC members and supports
> from hbase community rather than me alone (smile)
>
> Best Regards,
> Yu
>
>
> On Sat, 1 Sep 2018 at 03:12, Stack  wrote:
>
> > On Fri, Aug 31, 2018 at 10:22 AM Chetan Khatri <
> > chetan.opensou...@gmail.com>
> > wrote:
> >
> > > Thank you Stack for everything.
> > >
> > >
> > Thanks Chetan but our mighty Yu Li did all the work!
> > S
> >
> >
> >
> > > On Fri, Aug 31, 2018 at 8:18 PM Stack  wrote:
> > >
> > > > I blew the cobwebs off our blog and put up a short note on the
> > conference
> > > > by Yu Li and myself. See here: https://blogs.apache.org/hbase/
> > > >
> > > > S
> > > >
> > > > On Wed, Aug 22, 2018 at 3:03 AM Yu Li  wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > HBaseConAsia2018 is successfully held on Aug. 17th in Beijing,
> China
> > > and
> > > > > please following below links for a quick review:
> > > > >
> > > > > Pictures:
> > > > >
> > >
> https://drive.google.com/drive/folders/1eGuNI029a78s_BdH37VsSr4uOalyLi5O
> > > > >
> > > > > Slides and Video recording:
> > > > > https://yq.aliyun.com/articles/626119
> > > > >
> > > > > Enjoy it and let's expect the next year!
> > > > >
> > > > > Yu - on behalf of HBaseConAsia2018 PC
> > > > >
> > > >
> > >
> >
>

Re: [ANNOUNCE] New HBase committer Reid Chan

[ramkrishna vasudevan]

Congratulations !!!

On Tue, Jun 26, 2018 at 3:13 PM Guanghao Zhang  wrote:

> Congratulations！
>
> 2018-06-26 16:50 GMT+08:00 Yu Li :
>
> > Congratulations and welcome, Reid!
> >
> > Best Regards,
> > Yu
> >
> > On 26 June 2018 at 15:14, Nihal Jain  wrote:
> >
> > > Congrats :)
> > >
> > > On Tue 26 Jun, 2018, 12:22 PM OpenInx,  wrote:
> > >
> > > > Congratulations and welcome !
> > > >
> > > > On Tue, Jun 26, 2018 at 2:48 PM, 张铎(Duo Zhang) <
> palomino...@gmail.com>
> > > > wrote:
> > > >
> > > > >  Congratulations!
> > > > >
> > > > > 2018-06-26 14:44 GMT+08:00 Pankaj kr :
> > > > >
> > > > > > Congratulations Reid Chan..!!
> > > > > >
> > > > > > Regards,
> > > > > > Pankaj Kumar
> > > > > > Technical Project Leader
> > > > > > Enterprise Intelligence, IT BU
> > > > > >
> > > > > > Huawei Technologies India Pvt. Ltd.
> > > > > > Survey No. 37, Next to EPIP Area, Kundalahalli, Whitefield
> > > > > > Bengaluru-560066, Karnataka
> > > > > > Tel: + 91-80-49160700 Ext. 71678, Mob: 9535197664,  Email:
> > > > > > pankaj...@huawei.com
> > > > > >
> > > > > > 
> > > > > > __
> > > > > > This e-mail and its attachments contain confidential information
> > from
> > > > > > HUAWEI, which
> > > > > > is intended only for the person or entity whose address is listed
> > > > above.
> > > > > > Any use of the
> > > > > > information contained herein in any way (including, but not
> limited
> > > to,
> > > > > > total or partial
> > > > > > disclosure, reproduction, or dissemination) by persons other than
> > the
> > > > > > intended
> > > > > > recipient(s) is prohibited. If you receive this e-mail in error,
> > > please
> > > > > > notify the sender by
> > > > > > phone or email immediately and delete it!
> > > > > >
> > > > > >
> > > > > > -Original Message-
> > > > > > From: Chia-Ping Tsai [mailto:chia7...@gmail.com]
> > > > > > Sent: Tuesday, June 26, 2018 9:59 AM
> > > > > > To: dev@hbase.apache.org; 陳浩駿 
> > > > > > Subject: [ANNOUNCE] New HBase committer Reid Chan
> > > > > >
> > > > > > On behalf of the Apache HBase PMC, I am pleased to announce that
> > Reid
> > > > > Chan
> > > > > > has accepted the PMC's invitation to become a committer on the
> > > project.
> > > > > We
> > > > > > appreciate all of Reid’s generous contributions thus far and look
> > > > forward
> > > > > > to his continued involvement.
> > > > > >
> > > > > > Congratulations and welcome, Reid!
> > > > > >
> > > > > > --
> > > > > > Chia-Ping
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [ANNOUNCE] New HBase committer Guangxu Cheng

[ramkrishna vasudevan]

Congratulations !

On Tue, Jun 5, 2018 at 8:20 AM, Guanghao Zhang  wrote:

> Congratulations!
>
> 2018-06-05 9:29 GMT+08:00 OpenInx :
>
> > Congratulations!
> >
> > On Mon, Jun 4, 2018 at 6:16 PM, ashish singhi 
> > wrote:
> >
> > > Congrats and Welcome!
> > >
> > > Regards,
> > > Ashish
> > > -Original Message-
> > > From: 张铎(Duo Zhang) [mailto:palomino...@gmail.com]
> > > Sent: Monday, June 04, 2018 12:30 PM
> > > To: HBase Dev List ; hbase-user <
> > > u...@hbase.apache.org>
> > > Subject: [ANNOUNCE] New HBase committer Guangxu Cheng
> > >
> > > On behalf of the Apache HBase PMC, I am pleased to announce that
> Guangxu
> > > Cheng has accepted the PMC's invitation to become a committer on the
> > > project. We appreciate all of Guangxu's generous contributions thus far
> > and
> > > look forward to his continued involvement.
> > >
> > > Congratulations and welcome, Guangxu!
> > >
> >
>

Re: Running branch-2 and master tests from Eclipse IDE

[ramkrishna vasudevan]

Awesome. This works.  I would have never figured it out (smile). Thanks a
ton Duo. It helps a lot. May be a small note in the dev book can help if in
future some one else faces the same issue.

Regards
Ram

On Mon, May 21, 2018 at 11:48 AM, 张铎(Duo Zhang) <palomino...@gmail.com>
wrote:

> Try adding this into your /etc/hosts
>
> 127.0.0.1 java.sun.com
>
> The problem is that we have this in the web.xml for datanode
>
> http://java.sun.com/xml/ns/j2ee;>
> 
>
> When parsing the web.xml we will fetch from this url, and sometimes it will
> be very very slow. I used to think this only happens for us in China so I
> didn't post it out...
>
>
>
> 2018-05-21 13:44 GMT+08:00 ramkrishna vasudevan <
> ramkrishna.s.vasude...@gmail.com>:
>
> > Hi all
> >
> > Trying to debug some minicluster based testcases on master and branch-2
> > with eclipse IDE. But the cluster startup seems to hang in the HttpServer
> > start up as part of HDFS cluster setup.
> >
> > The older 3.0 branches were not having this issue and seems to happen in
> > the later 2.0 and master  branches.
> >
> > Is there anyway to solve this - wrt to POM changes or anyother config
> > updates?
> >
> > The same tests run fine when we run using 'mvn test' command.
> >
> > "Time-limited test" #17 daemon prio=5 os_prio=0 tid=0x7fc6c4f6b000
> > nid=0xb91 runnable [0x7fc6565c4000]
> >java.lang.Thread.State: RUNNABLE
> > at java.net.PlainSocketImpl.socketConnect(Native Method)
> > at
> > java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:
> > 345)
> > - locked <0x00058857eeb0> (a java.net.SocksSocketImpl)
> > at
> > java.net.AbstractPlainSocketImpl.connectToAddress(
> > AbstractPlainSocketImpl.java:206)
> > at
> > java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:
> 188)
> > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
> > at java.net.Socket.connect(Socket.java:589)
> > at java.net.Socket.connect(Socket.java:538)
> > at sun.net.NetworkClient.doConnect(NetworkClient.java:180)
> > at sun.net.www.http.HttpClient.openServer(HttpClient.java:432)
> > at sun.net.www.http.HttpClient.openServer(HttpClient.java:527)
> > - locked <0x00058857ec70> (a sun.net.www.http.HttpClient)
> > at sun.net.www.http.HttpClient.(HttpClient.java:211)
> > at sun.net.www.http.HttpClient.New(HttpClient.java:308)
> > at sun.net.www.http.HttpClient.New(HttpClient.java:326)
> > at
> > sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(
> > HttpURLConnection.java:1168)
> > at
> > sun.net.www.protocol.http.HttpURLConnection.plainConnect0(
> > HttpURLConnection.java:1104)
> > at
> > sun.net.www.protocol.http.HttpURLConnection.plainConnect(
> > HttpURLConnection.java:998)
> > at
> > sun.net.www.protocol.http.HttpURLConnection.connect(
> > HttpURLConnection.java:932)
> > at
> > sun.net.www.protocol.http.HttpURLConnection.getInputStream0(
> > HttpURLConnection.java:1512)
> > - locked <0x00058857d600> (a
> > sun.net.www.protocol.http.HttpURLConnection)
> > at
> > sun.net.www.protocol.http.HttpURLConnection.getInputStream(
> > HttpURLConnection.java:1440)
> >
> >
> > After some time the test case fails with 'Exception' msg.
> >
> > There is no firewall and so firewall is not the reason for this.
> > I tracked the pom changes but not able to make out any obvious
> differences
> > here.
> > Any inputs here would be very helpful.
> >
> > Regards
> > Ram
> >
>

Running branch-2 and master tests from Eclipse IDE

[ramkrishna vasudevan]

Hi all

Trying to debug some minicluster based testcases on master and branch-2
with eclipse IDE. But the cluster startup seems to hang in the HttpServer
start up as part of HDFS cluster setup.

The older 3.0 branches were not having this issue and seems to happen in
the later 2.0 and master  branches.

Is there anyway to solve this - wrt to POM changes or anyother config
updates?

The same tests run fine when we run using 'mvn test' command.

"Time-limited test" #17 daemon prio=5 os_prio=0 tid=0x7fc6c4f6b000
nid=0xb91 runnable [0x7fc6565c4000]
   java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketConnect(Native Method)
at
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:345)
- locked <0x00058857eeb0> (a java.net.SocksSocketImpl)
at
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at java.net.Socket.connect(Socket.java:538)
at sun.net.NetworkClient.doConnect(NetworkClient.java:180)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:432)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:527)
- locked <0x00058857ec70> (a sun.net.www.http.HttpClient)
at sun.net.www.http.HttpClient.(HttpClient.java:211)
at sun.net.www.http.HttpClient.New(HttpClient.java:308)
at sun.net.www.http.HttpClient.New(HttpClient.java:326)
at
sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1168)
at
sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1104)
at
sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:998)
at
sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:932)
at
sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1512)
- locked <0x00058857d600> (a
sun.net.www.protocol.http.HttpURLConnection)
at
sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1440)


After some time the test case fails with 'Exception' msg.

There is no firewall and so firewall is not the reason for this.
I tracked the pom changes but not able to make out any obvious differences
here.
Any inputs here would be very helpful.

Regards
Ram

Re: Fwd:

[ramkrishna vasudevan]

Regarding the read flow this is what happens

1)  Create a region level scanner
2) the region level scanner can comprise of more than one store scanner
(each store scanner works on one column family).
3) Every store scanner wil comprise of memstore scanner and a set of hfile
scanners (based on number of store files).
4) The scan tries to read data in lexographical order.
 For eg, for simplicty take you have row1 to row5 and there is only one
column family 'f1' and one column 'c1'. Assume row1 was already written and
it is flushed to a store file. Row2 to row5 are in the memstore .
When the scanner starts it will form a heap with all these memstore scanner
and store file (hfile) scanners. Internally since row1 is smaller
lexographically the row1 from the store file is retrieved first. This row1
for the first time will be in HDFS (and not in block cache). The remaining
rows are fetched from memstore scanners. there is no block cache concept at
the memstore level. Memstore is just a simple Key value map.

When the same scan is issued the next time we go through the above steps
but to fetch row1, the store file scanner that has row1, fetches the block
cache that has row1 (instead of HDFS) and returns the value from block
cache and the remaining rows are again fetched from memstore scanners from
the underlying memstore.

Hope this helps.

REgards
Ram

On Thu, May 3, 2018 at 9:17 AM, Xi Yang  wrote:

> Hi Tim,
>
> Thanks for confirm the question.  That question confused me for a long
> time. Really appreciate.
>
>
> About another question, I still don't know whether ModelA is correct or
> Model B is correct. Still confused
>
>
> Thanks,
> Alex
>
> 2018-05-02 13:53 GMT-07:00 Tim Robertson :
>
> > Thanks Alex,
> >
> > Yes, looking at that code I believe you are correct - the memStore
> scanner
> > is appended after the block scanners.
> > The block scanners may or may not see hits in the block cache when they
> > read. If they don't get a hit, they'll open the block from the underlying
> > HFile(s).
> >
> >
> >
> > On Wed, May 2, 2018 at 10:41 PM, Xi Yang  wrote:
> >
> > > Hi Tim,
> > >
> > > Thank you for detailed explanation. Yes, that really helps me! I really
> > > appreciate it!
> > >
> > >
> > > But I still confused about the sequence:
> > >
> > > I've read these codes in *HStore.getScanners* :
> > >
> > >
> > > *// TODO this used to get the store files in descending order,*
> > > *// but now we get them in ascending order, which I think is*
> > > *// actually more correct, since memstore get put at the end.*
> > > *List sfScanners =
> > > StoreFileScanner.getScannersForStoreFiles(storeFilesToScan,*
> > > *  cacheBlocks, usePread, isCompaction, false, matcher, readPt);*
> > > *List scanners = new
> ArrayList<>(sfScanners.size() +
> > > 1);*
> > > *scanners.addAll(sfScanners);*
> > > *// Then the memstore scanners*
> > > *scanners.addAll(memStoreScanners);*
> > >
> > >
> > > Is it mean this step:
> > >
> > >
> > > *2) It looks in the memstore to see if there are any writes still in
> > > memoryready to flush down to the HFiles that needs merged with the data
> > > read in 1) *
> > >
> > > is behind the following step?
> > >
> > > *c) the data is read from the opened block *
> > >
> > >
> > >
> > >
> > > Here are explanation of the images I drew before, so that we don't need
> > the
> > > images:
> > >
> > > When a read request come in
> > > Model A
> > >
> > >1. get Scanners (including StoreScanner and MemStoreScanner).
> > >MemStoreScanner is the last one
> > >2. Begin with the first StoreScanner
> > >3. Try to get the block from BlockCache of the StoreScanner
> > >4. Try to get the block from HFile of the StoreScanner
> > >5. Go to the next StoreScanner
> > >6. Loop #2 - #5 until all StoreScanner been used
> > >7. Try to get the block from memStore
> > >
> > >
> > > Model B
> > >
> > >1. Try to get the block from BlockCache, if failed then go to #2
> > >2. get Scanners (including StoreScanner and MemStoreScanner).
> > >MemStoreScanner is the last on
> > >3. Begin with the first StoreScanner
> > >4. Try to get the block from HFile of the StoreScanner
> > >5. Go to the next StoreScanner
> > >6. Loop #4 - #5 until all StoreScanner been used
> > >7. Try to get the block from memStore
> > >
> > >
> > >
> > > Thanks,
> > > Alex
> > >
> > >
> > > 2018-05-02 1:04 GMT-07:00 Tim Robertson :
> > >
> > > > Hi Alex,
> > > >
> > > > I'm not sure I fully follow your question without the images but I'll
> > try
> > > > and help.
> > > >
> > > > When a read request comes in, my understanding of the order of
> > execution
> > > is
> > > > as follows (perhaps someone can verify this):
> > > >
> > > > 1) It looks in the block cache for the cells (this is a read only
> cache
> > > > containing recently read data)
> > > > 2) 

Re: [ANNOUNCE] Please welcome Francis Liu to the HBase PMC

[ramkrishna vasudevan]

Congratulations Francis !!!

On Thu, Apr 12, 2018 at 11:42 AM, Yu Li  wrote:

> Congratulations and welcome, Francis!
>
> Best Regards,
> Yu
>
> On 12 April 2018 at 13:25, Pankaj kr  wrote:
>
> > Congratulations Francis..!!
> >
> >
> > Regards,
> > Pankaj
> >
> > -Original Message-
> > From: Andrew Purtell [mailto:apurt...@apache.org]
> > Sent: Thursday, April 12, 2018 4:04 AM
> > To: dev
> > Subject: [ANNOUNCE] Please welcome Francis Liu to the HBase PMC
> >
> > On behalf of the Apache HBase PMC I am pleased to announce that Francis
> > Liu has accepted our invitation to become a PMC member on the Apache
> HBase
> > project. We appreciate Francis stepping up to take more responsibility in
> > the HBase project. He has been an active contributor to HBase for many
> > years and recently took over responsibilities as branch RM for
> branch-1.3.
> >
> > Please join me in welcoming Francis to the HBase PMC!
> >
> > --
> > Best regards,
> > Andrew
> >
>

Re: Almost ready for HBaseCon+PhoenixCon 2018 SanJose CFP

[ramkrishna vasudevan]

Hi

I think [2] does not work. Seems to be a broken link.
In the CONTACTs section should it be both dev@hbase and dev@phoenix? Rest
looks good to me.

Regards
Ram

On Wed, Mar 21, 2018 at 9:42 AM, Yu Li  wrote:

> Great to know and thanks for the efforts sir.
>
> Minor: in the CfP sector, first line, "The event's call for proposals is
> available *on on* EasyChair", the double "on" should be merged (smile)
>
> Best Regards,
> Yu
>
> On 21 March 2018 at 10:51, Josh Elser  wrote:
>
> > Hi all,
> >
> > I've published a new website for the upcoming event in June in California
> > at [1][2] for the HBase and Phoenix websites, respectively. 1 & 2 are
> > identical.
> >
> > I've not yet updated any links on either website to link to the new page.
> > I'd appreciate if folks can give their feedback on anything outwardly
> > wrong, incorrect, etc. If folks are happy, then I'll work on linking from
> > the main websites, and coordinating an official announcement via mail
> > lists, social media, etc.
> >
> > The website is generated from [3]. If you really want to be my
> > best-friend, let me know about the above things which are wrong via
> > pull-request ;)
> >
> > - Josh
> >
> > [1] https://hbase.apache.org/hbasecon-phoenixcon-2018/
> > [2] https://phoenix.apache.org/hbasecon-phoenixcon-2018/
> > [3] https://github.com/joshelser/hbasecon-jekyll
> >
>

Re: [ANNOUNCE] New HBase committer Peter Somogyi

[ramkrishna vasudevan]

Keep up your good work Peter!!

On Thu, Mar 8, 2018 at 3:45 AM, Mike Drob <md...@apache.org> wrote:

> Welcome, Peter!
>
> On Fri, Feb 23, 2018 at 7:51 AM, Anoop John <anoop.hb...@gmail.com> wrote:
>
> > Congrats Peter..
> >
> > Anoop
> >
> >
> > On Friday, February 23, 2018, ramkrishna vasudevan <
> > ramkrishna.s.vasude...@gmail.com> wrote:
> >
> > > Congratulations Peter !!!
> > >
> > > On Fri, Feb 23, 2018 at 3:40 PM, Peter Somogyi <psomo...@apache.org>
> > > wrote:
> > >
> > > > Thank you very much everyone!
> > > >
> > > > On Thu, Feb 22, 2018 at 8:08 PM, Sean Busbey <bus...@apache.org>
> > wrote:
> > > >
> > > > > On behalf of the Apache HBase PMC, I am pleased to announce that
> > Peter
> > > > > Somogyi has accepted the PMC's invitation to become a committer on
> > the
> > > > > project.
> > > > >
> > > > > We appreciate all of Peter's great work thus far and look forward
> to
> > > > > continued involvement.
> > > > >
> > > > > Please join me in congratulating Peter!
> > > > >
> > > >
> > >
> >
>

Re: [ANNOUNCE] New HBase committer Zach York

[ramkrishna vasudevan]

Congratulations Zach !!!

On Thu, Mar 8, 2018 at 11:03 AM, Yu Li  wrote:

> Congratulations, Zach!
>
> Best Regards,
> Yu
>
> On 8 March 2018 at 06:13, Mike Drob  wrote:
>
> > Congratulations, Zach!
> >
> > On Wed, Mar 7, 2018 at 4:03 PM, Andrew Purtell 
> > wrote:
> >
> > > Congratulations and welcome Zach!
> > >
> > >
> > > On Wed, Mar 7, 2018 at 8:27 AM, Sean Busbey  wrote:
> > >
> > > > On behalf of the Apache HBase PMC, I am pleased to announce that Zach
> > > > York has accepted the PMC's invitation to become a committer on the
> > > > project.
> > > >
> > > > We appreciate all of Zach's great work thus far and look forward to
> > > > continued involvement.
> > > >
> > > > Please join me in congratulating Zach!
> > > >
> > >
> > >
> > >
> > > --
> > > Best regards,
> > > Andrew
> > >
> > > Words like orphans lost among the crosstalk, meaning torn from truth's
> > > decrepit hands
> > >- A23, Crosstalk
> > >
> >
>

Re: [VOTE] The first hbase-2.0.0-beta-2 Release Candidate is available for download

[ramkrishna vasudevan]

Ya looking at the logs it seems not critical. Also the RawbytesComparator
is no longer serialized and we don use any comparator for the
RawBytesComparator like the RowBloom etc.

Regards
Ram

On Tue, Mar 6, 2018 at 8:52 AM, Chia-Ping Tsai  wrote:

> > We have it as an explicit dependency in our top-level pom. Doing
> > dependency:tree, hadoop minicluster needs it? Its need to run tests? You
> > think we should not include in our binary sir?
> IIRC, those test jars are not in binary.tag before. HBASE-19089 add
> hbase-testing-util module to the assembly so the test jars are packaged to
> binary.tar. I'm fine with them.
>
> On 2018/03/05 19:40:30, Stack  wrote:
> > On Mon, Mar 5, 2018 at 6:56 AM, Chia-Ping Tsai 
> wrote:
> >
> > > +1 (binding) with some questions
> > > unit test (oracle jdk-8u161) - all pass
> > > deploy binary (3 nodes) - ok
> > > browse master/regionserver web - LGTM
> > > put/delete/get/scan 500W rows - ok
> > > create/disable/drop/put/scan/count by shell - ok
> > >
> > > Q1:
> > > the 2.0 binary get fatter than 1.4. (384 MB -> 887MB). Seems we package
> > > the test docs and source code to binary tar. Are all of them necessary?
> > >
> > >
> > No. Let me review.
> >
> >
> >
> > > Q2:
> > > Why some test jars of hadoop are included to hbase/lib? such as
> > > hadoop-common-2.7.4-tests and hadoop-hdfs-2.7.4-tests
> > >
> > >
> > We have it as an explicit dependency in our top-level pom. Doing
> > dependency:tree, hadoop minicluster needs it? Its need to run tests? You
> > think we should not include in our binary sir?
> >
> > [INFO] +- org.apache.hadoop:hadoop-minicluster:jar:2.7.4:test
> > [INFO] |  +- org.apache.hadoop:hadoop-common:test-jar:tests:2.7.4:test
> >
> >
> > Thanks for trying the RC Chia-Ping,
> > S
> >
> >
> >
> >
> >
> >
> >
> > >
> > > On 2018/03/02 23:40:53, Stack  wrote:
> > > > The first release candidate for HBase 2.0.0-beta-2 is up at
> > > >
> > > >  https://dist.apache.org/repos/dist/dev/hbase/hbase-2.0.0-
> beta-2.RC0/
> > > >
> > > > Maven artifacts are available from a staging directory here:
> > > >
> > > >   https://repository.apache.org/content/repositories/
> orgapachehbase-1199
> > > >
> > > > All was signed with my key at 8ACC93D2 [1]
> > > >
> > > > I tagged the RC as 2.0.0-beta-2RC0.2 at
> > > > 9e9b347d667e1fc6165c9f8ae5ae7052147e8895
> > > >
> > > > hbase-2.0.0-beta-2 is a not-for-production preview of hbase-2.0.0.
> It is
> > > > meant for devs and downstreamers to test drive and flag us if we
> messed
> > > up
> > > > on anything ahead of our rolling
> > > > actual 2.0.0 release candidates ("GAs").
> > > >
> > > > hbase-2.0.0-beta-2 is our second beta release. More than 200 fixes
> have
> > > > gone in since
> > > > beta-1. Unit tests generallly pass when run against hadoop2 and
> > > hadoop3[5].
> > > > It includes
> > > > all that was in previous alphas and beta (new assignment manager,
> offheap
> > > > read/write
> > > > path, in-memory compactions, etc).The list of features addressed in
> 2.0.0
> > > > so far can be
> > > > found here [3]. There are thousands. The list of ~3k+ fixes in 2.0.0
> > > > exclusively can be
> > > > found here [4]. Our overview doc. on the state of 2.0.0 is at [6].
> > > >
> > > > This beta was supposed to have as its focus rolling upgrade from
> > > hbase-1.x
> > > > versions but
> > > > this is work not complete (At this late stage, it is looking like it
> will
> > > > be a post-2.0.0 project).
> > > >
> > > > This is our last hbase-2.0.0 beta release. Next up, we'll be rolling
> an
> > > > actual 2.0.0 release
> > > > candidate. Look for this in a week or two after beta-2 goes out,
> after
> > > > we've done more
> > > > testing and documentation (and we fix issues raised by you all
> against
> > > this
> > > > beta).
> > > >
> > > > One known issue, still unaddressed, is that the User API has not been
> > > > properly filtered
> > > > so it shows more than just InterfaceAudience Public content
> (HBASE-19663,
> > > > to be fixed
> > > > by release).
> > > >
> > > > Please take this beta for a spin. Please vote on whether it ok to
> put out
> > > > this RC as our second
> > > > beta (Note CHANGES has not yet been updated). Let the VOTE be open
> for at
> > > > least 72 hours
> > > > (Lets say Wednesday morning, March 7th).
> > > >
> > > > Thanks,
> > > > Your 2.0.0 Release Manager
> > > >
> > > > 1. http://pgp.mit.edu/pks/lookup?op=get=0x9816C7FC8ACC93D2
> > > > 3. https://goo.gl/scYjJr
> > > > 4. https://goo.gl/dFFT8b
> > > > 5. https://builds.apache.org/job/HBase%20Nightly/job/branch-2/
> > > > 
> > > > 6. https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4
> > > > z9iEu_ktczrlKHK8N4SZzs/
> > > >
> > >
> >
>

Re: Re: [ANNOUNCE] Please welcome Guanghao Zhang to the HBase PMC

[ramkrishna vasudevan]

Congratulations, Guanghao!!

On Thu, Mar 1, 2018 at 9:52 AM, Yu Li  wrote:

> Congratulations, Guanghao!
>
> Best Regards,
> Yu
>
> On 1 March 2018 at 11:59, Jerry He  wrote:
>
> >   Congrats,  Guanghao!
> >
> > On Wed, Feb 28, 2018 at 6:17 PM Allan Yang  wrote:
> >
> > >  Congratulations Guanghao!
> > >
> > > Best Regards
> > > Allan Yang
> > >
> > > 2018-03-01 10:11 GMT+08:00 OpenInx :
> > >
> > > > Congratulations !
> > > >
> > > > On Thu, Mar 1, 2018 at 9:54 AM, Chunhui Shen  wrote:
> > > >
> > > > > Congrats！
> > > > >
> > > > >
> > > > > At 2018-03-01 08:29:22, "Esteban Gutierrez" 
> > > > wrote:
> > > > > >Congrats, Guanghao!
> > > > > >
> > > > > >--
> > > > > >Cloudera, Inc.
> > > > > >
> > > > > >
> > > > > >On Wed, Feb 28, 2018 at 3:56 PM, Andrew Purtell <
> > apurt...@apache.org>
> > > > > wrote:
> > > > > >
> > > > > >> Congratulations!
> > > > > >>
> > > > > >>
> > > > > >> On Wed, Feb 28, 2018 at 7:55 AM, Sean Busbey  >
> > > > wrote:
> > > > > >>
> > > > > >> > On behalf of the Apache HBase PMC I am pleased to announce
> that
> > > > > >> > Guanghao Zhang has accepted our invitation to become a PMC
> > member
> > > on
> > > > > >> > the Apache HBase project. We appreciate Guanghao stepping up
> to
> > > take
> > > > > >> > more responsibility in the HBase project.
> > > > > >> >
> > > > > >> > As a reminder, if anyone would like to nominate another person
> > as
> > > a
> > > > > >> > committer or PMC member, even if you are not currently a
> > committer
> > > > or
> > > > > >> > PMC member, you can always drop a note to
> > > priv...@hbase.apache.org
> > > > to
> > > > > >> > let us know!
> > > > > >> >
> > > > > >> > - busbey
> > > > > >> >
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> --
> > > > > >> Best regards,
> > > > > >> Andrew
> > > > > >>
> > > > > >> Words like orphans lost among the crosstalk, meaning torn from
> > > truth's
> > > > > >> decrepit hands
> > > > > >>- A23, Crosstalk
> > > > > >>
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > ==
> > > > Openinx  blog : http://openinx.github.io
> > > >
> > > > TO BE A GREAT HACKER !
> > > > ==
> > > >
> > >
> >
>

Re: 回复：Re: [ANNOUNCE] Please welcome Ashish Singhi to the HBase PMC

[ramkrishna vasudevan]

Congratulations, Ashish!!

On Thu, Mar 1, 2018 at 9:52 AM, Yu Li  wrote:

> Congratulations, Ashish!
>
> Best Regards,
> Yu
>
> On 1 March 2018 at 11:57, Jerry He  wrote:
>
> > Congrats, Ashish!
> >
> > On Wed, Feb 28, 2018 at 6:18 PM Allan Yang  wrote:
> >
> > >  Congratulations  Ashish!
> > >
> > > Best Regards
> > > Allan Yang
> > >
> > > 2018-03-01 10:11 GMT+08:00 OpenInx :
> > >
> > > > Congratulations !
> > > >
> > > > On Thu, Mar 1, 2018 at 9:55 AM, Chunhui Shen  wrote:
> > > >
> > > > > Congrats！
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > At 2018-03-01 09:24:47, "岭秀"  wrote:
> > > > > >Congratulations, Ashish!!  well-deserved
> > > > > >> > On behalf of the Apache HBase PMC I am pleased to announce
> that
> > > > Ashish
> > > > > >> > Singhi has accepted our invitation to become a PMC member on
> the
> > > > > >> > Apache HBase project. We appreciate Ashish stepping up to take
> > > more
> > > > > >> > responsibility in the HBase project.
> > > > > >> >
> > > > > >> > As a reminder, if anyone would like to nominate another person
> > as
> > > a
> > > > > >> > committer or PMC member, even if you are not currently a
> > committer
> > > > or
> > > > > >> > PMC member, you can always drop a note to
> > > priv...@hbase.apache.org
> > > > to
> > > > > >> > let us know!
> > > > > >> >
> > > > > >> > - busbey
> > > > > >> >
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> --
> > > > > >> Best regards,
> > > > > >> Andrew
> > > > > >>
> > > > > >> Words like orphans lost among the crosstalk, meaning torn from
> > > truth's
> > > > > >> decrepit hands
> > > > > >>- A23, Crosstalk
> > > > > >>
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > ==
> > > > Openinx  blog : http://openinx.github.io
> > > >
> > > > TO BE A GREAT HACKER !
> > > > ==
> > > >
> > >
> >
>

Re: Re: [ANNOUNCE] Please welcome Mike Drob to the HBase PMC

[ramkrishna vasudevan]

Congratulations Mike!!!

On Thu, Mar 1, 2018 at 9:52 AM, Yu Li  wrote:

> Congratulations, Mike!
>
> Best Regards,
> Yu
>
> On 1 March 2018 at 11:56, Jerry He  wrote:
>
> > Congrats,  Mike!
> >
> > On Wed, Feb 28, 2018 at 6:19 PM Allan Yang  wrote:
> >
> > >  Congratulations Mike!
> > >
> > > Best Regards
> > > Allan Yang
> > >
> > > 2018-03-01 10:12 GMT+08:00 OpenInx :
> > >
> > > > Congratulations !
> > > >
> > > > On Thu, Mar 1, 2018 at 9:54 AM, Chunhui Shen  wrote:
> > > >
> > > > > Congrats！
> > > > > At 2018-03-01 07:56:11, "Andrew Purtell" 
> > wrote:
> > > > > >Congrats Mike!
> > > > > >
> > > > > >On Wed, Feb 28, 2018 at 7:54 AM, Sean Busbey 
> > > wrote:
> > > > > >
> > > > > >> On behalf of the Apache HBase PMC I am pleased to announce that
> > Mike
> > > > > >> Drob has accepted our invitation to become a PMC member on the
> > > Apache
> > > > > >> HBase project. We appreciate Mike stepping up to take more
> > > > > >> responsibility in the HBase project.
> > > > > >>
> > > > > >>
> > > > > >> As a reminder, if anyone would like to nominate another person
> as
> > a
> > > > > >> committer or PMC member, even if you are not currently a
> committer
> > > or
> > > > > >> PMC member, you can always drop a note to
> > priv...@hbase.apache.org
> > > to
> > > > > >> let us know!
> > > > > >>
> > > > > >> - busbey
> > > > > >>
> > > > > >
> > > > > >
> > > > > >
> > > > > >--
> > > > > >Best regards,
> > > > > >Andrew
> > > > > >
> > > > > >Words like orphans lost among the crosstalk, meaning torn from
> > truth's
> > > > > >decrepit hands
> > > > > >   - A23, Crosstalk
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > ==
> > > > Openinx  blog : http://openinx.github.io
> > > >
> > > > TO BE A GREAT HACKER !
> > > > ==
> > > >
> > >
> >
>

Re: [ANNOUNCE] New HBase committer Peter Somogyi

[ramkrishna vasudevan]

Congratulations Peter !!!

On Fri, Feb 23, 2018 at 3:40 PM, Peter Somogyi  wrote:

> Thank you very much everyone!
>
> On Thu, Feb 22, 2018 at 8:08 PM, Sean Busbey  wrote:
>
> > On behalf of the Apache HBase PMC, I am pleased to announce that Peter
> > Somogyi has accepted the PMC's invitation to become a committer on the
> > project.
> >
> > We appreciate all of Peter's great work thus far and look forward to
> > continued involvement.
> >
> > Please join me in congratulating Peter!
> >
>

Re: [VOTE] The fourth hbase-2.0.0-beta-1 Release Candidate is available for download

[ramkrishna vasudevan]

+1

- checksum ok
- signature ok.
- built from src.
- Ran PE tool. Performed Randomwrites and random reads. No issues observed.

Regards
Ram

On Mon, Jan 15, 2018 at 8:58 PM, Balazs Meszaros <
balazs.mesza...@cloudera.com> wrote:

> +1
>
> - checksums, signatures: OK
> - build from src: OK (8u112)
> - unit tests passes except TestReplicationAdmin. Am I the only one who
> cannot run it? It fails constantly, but it isn't in our flaky list.
> - LTT with 1M rows works.
> - Basic shell functionality works.
> - Web UI for master and RS also seem too be good.
>
> Balazs
>
>
> On Mon, Jan 15, 2018 at 8:36 AM, Yung-An He  wrote:
>
> > +1 non-binding
> >
> > Checked MD5: OK
> > Testing the URL of download is available: OK
> > Starting HBase with cluster mode via docker container: OK
> > Import data with 100 million rows and 10 columns to table via ImportTsv:
> OK
> > Execute basic command via `hbase shell`: OK
> >
> > 2018-01-14 4:23 GMT+08:00 Stack :
> >
> > > Thanks to the votes and testing so far.
> > >
> > > Last night, a nightly build on hadoop2 completed but I notice that it
> > has a
> > > bunch of tests filtered out [1] (I'm trying to get this facility turned
> > > off). There is also a report by Duo that came in last night a test that
> > > passes for me fails for him (HBASE-19791). It happens to be one of
> those
> > > filtered out unfortunately.
> > >
> > > So, running the test suite (JMS!), you might want to apply the same
> > filter
> > > for now. See [1] below. I'll work on the failing tests in meantime
> (Yeah,
> > > they were supposed to be fixed by now but its been taking a while).
> > >
> > > Thanks again for your patience/understanding.
> > >
> > > Yours,
> > > Your RM
> > >
> > > 1.
> > > https://builds.apache.org/job/HBase%20Nightly/job/branch-2/
> > > 183/artifact/output-jdk8-hadoop2/patch-unit-root.txt
> > >
> > >
> > > On Fri, Jan 12, 2018 at 8:48 PM, Stack  wrote:
> > >
> > > > The fourth release candidate for HBase 2.0.0-beta-1 is up at:
> > > >
> > > >   https://dist.apache.org/repos/dist/dev/hbase/hbase-2.0.0-
> beta-1-RC3/
> > > >
> > > > Maven artifacts are available from a staging directory here:
> > > >
> > > >   https://repository.apache.org/content/repositories/
> > orgapachehbase-1193
> > > >
> > > > All was signed with my key at 8ACC93D2 [1].
> > > >
> > > > I tagged the RC as 2.0.0-beta-1-RC1.7 at hash
> 026f535a7747b89003252ded9
> > > > 585e827686aa79f
> > > >
> > > > This RC has some nice bug fixes over RC0-2 including fixing MOST of
> the
> > > > failing and flakey tests (We are still working through them).
> > > >
> > > > hbase-2.0.0-beta-1 is our first beta release. It includes all that
> was
> > in
> > > > previous alphas (new assignment manager, offheap read/write path,
> > > in-memory
> > > > compactions, etc.). The APIs and feature-set are sealed.
> > > >
> > > > hbase-2.0.0-beta-1 is a not-for-production preview of hbase-2.0.0. It
> > is
> > > > meant for devs and downstreamers to test drive and flag us if we
> messed
> > > up
> > > > on anything ahead of our rolling GAs. We are particular interested in
> > > > hearing from Coprocessor developers.
> > > >
> > > > The list of features addressed in 2.0.0 so far can be found here [3].
> > > > There are thousands. The list of ~2k+ fixes in 2.0.0 exclusively can
> be
> > > > found here [4] (My JIRA JQL foo is a bit dodgy -- forgive me if
> > > mistakes).
> > > >
> > > > I've updated our overview doc. on the state of 2.0.0 [6]. We'll do
> one
> > > > more beta before we put up our first 2.0.0 Release Candidate by the
> end
> > > of
> > > > January, 2.0.0-beta-2. Its focus will be making it so users can do a
> > > > rolling upgrade on to hbase-2.x from hbase-1.x (and any bug fixes
> found
> > > > running beta-1). Here is the list of what we have targeted so far for
> > > > beta-2 [5]. Check it out.
> > > >
> > > > One known issue is that the User API has not been properly filtered
> so
> > it
> > > > shows more than just InterfaceAudience Public content (HBASE-19663,
> to
> > be
> > > > fixed by beta-2).
> > > >
> > > > Please take this beta for a spin. Please vote on whether it ok to put
> > out
> > > > this RC as our first beta (Note CHANGES has not yet been updated).
> Let
> > > the
> > > > VOTE be open for 72 hours (Lets say Tuesday morning!)
> > > >
> > > > Thanks,
> > > > Your 2.0.0 Release Manager
> > > >
> > > > 1. http://pgp.mit.edu/pks/lookup?op=get=0x9816C7FC8ACC93D2
> > > > 3. https://goo.gl/scYjJr
> > > > 4. https://goo.gl/dFFT8b
> > > > 5. https://issues.apache.org/jira/projects/HBASE/versions/12340862
> > > > 6. https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4
> > > > z9iEu_ktczrlKHK8N4SZzs/
> > > >
> > >
> >
>

Re: [ANNOUNCE] Please welcome new HBase committer YI Liang

[ramkrishna vasudevan]

Congratulations Yi Liang!!!

On Thu, Dec 21, 2017 at 12:21 PM, Pankaj kr  wrote:

> Congratulations YI Liang..!!
>
> Thanks & Regards,
> Pankaj
>
> HUAWEI TECHNOLOGIES CO.LTD.
> Huawei Tecnologies India Pvt. Ltd.
> Near EPIP Industrial Area, Kundalahalli Village
> Whitefield, Bangalore-560066
> www.huawei.com
> 
> -
> This e-mail and its attachments contain confidential information from
> HUAWEI, which
> is intended only for the person or entity whose address is listed above.
> Any use of the
> information contained herein in any way (including, but not limited to,
> total or partial
> disclosure, reproduction, or dissemination) by persons other than the
> intended
> recipient(s) is prohibited. If you receive this e-mail in error, please
> notify the sender by
> phone or email immediately and delete it!
>
> -Original Message-
> From: Jerry He [mailto:jerry...@gmail.com]
> Sent: Thursday, December 21, 2017 5:37 AM
> To: dev; u...@hbase.apache.org
> Subject: [ANNOUNCE] Please welcome new HBase committer YI Liang
>
> On behalf of the Apache HBase PMC, I am pleased to announce that Yi Liang
> has accepted the PMC's invitation to become a committer on the project.
>
> We appreciate all of Yi's great work thus far and look forward to his
> continued involvement.
>
> Please join me in congratulating Yi!
>
> --
> Thanks,
> Jerry
>

Re: Cleanup and remove the code path where is no hbase.client.rpc.codec

[ramkrishna vasudevan]

So the proposal is to avoid the empty config and have a better defined
config if we need No pb way? Ya I agree to it if this empty way seems odd
to config.
Any non - java client what will be the value for this config?

Regards
Ram

On Wed, Dec 20, 2017 at 8:40 AM, 张铎(Duo Zhang) 
wrote:

> See AbstractTestIPC, there is a testNoCodec. But I agree that we should
> have a default fallback codec always.
>
> 2017-12-20 11:02 GMT+08:00 Jerry He :
>
> > RPC_CODEC_CONF_KEY 'hbase.client.rpc.codec' is a property we use on the
> > client side to determine the RPC codec.
> >
> > It currently has a strange logic. Whereas the default is KeyValueCodec,
> we
> > allow  a user to specify an empty string "" as the a way to indicate
> there
> > is no codec class and we should not use any.
> >
> >   Codec getCodec() {
> > // For NO CODEC, "hbase.client.rpc.codec" must be configured with
> empty
> > string AND
> > // "hbase.client.default.rpc.codec" also -- because default is to do
> > cell block encoding.
> > String className = conf.get(HConstants.RPC_CODEC_CONF_KEY,
> > getDefaultCodec(this.conf));
> > if (className == null || className.length() == 0) {
> >   return null;
> > }
> > try {
> >   return (Codec) Class.forName(className).newInstance();
> > } catch (Exception e) {
> >   throw new RuntimeException("Failed getting codec " + className, e);
> > }
> >   }
> >
> > I don't know the original reason for having this.
> > The consequence of this 'no codec' is that we will pb all RPC payload and
> > not using cell blocks.
> >
> > In the test cases, after these many releases, there is no test that
> > excercises this special case.
> > The code path we test are mostly with a valid or default
> > 'hbase.client.rpc.codec'.
> > The other code path is probably sitting there rotten.
> >
> > For example,
> >
> > In MultiServerCallable:
> >
> >   if (this.cellBlock) {
> > // Build a multi request absent its Cell payload. Send data
> in
> > cellblocks.
> > regionActionBuilder =
> > RequestConverter.buildNoDataRegionAction(regionName,
> > rms, cells,
> >   regionActionBuilder, actionBuilder, mutationBuilder);
> >   } else {
> > regionActionBuilder =
> > RequestConverter.buildRegionAction(regionName,
> > rms);   ==> Will not be exercised in test..
> >   }
> >
> > Proposal:
> >
> > We remove this 'no hbase.rpc.codec' case and all dependent logic. There
> is
> > a default and user can overwrite the default, but have to provide a valid
> > non-empty value.
> > Then we can clean up the code where we choose between pb or no pb.  We
> will
> > always do cell block in these cases.
> >
> > There are cases where we currently only do pb, like some of the
> individual
> > ops (append, increment, mutateRow, etc). We can revisit to see if they
> can
> > be non-pb'ed.
> >
> > The proposed change only cleans up the client side (PRC client).
> > I want to keep the server side handling of pb and no-pb both for now, so
> > that the server can accommodate a 'no hbase.rpc.codec' connection request
> > for now for backward compatibility.
> >
> > Any concerns?
> >
> > Thanks.
> >
> > Jerry
> >
>

Re: Moving 2.0 forward

[ramkrishna vasudevan]

HBASE-18946 Stochastic load balancer assigns replica regions to the same RS

Am on this today and probably will put up another patch.

Regards
Ram

On Fri, Nov 17, 2017 at 5:18 AM, Stack  wrote:

> hbase-2.0.0-beta-1 update (Reminder, beta-1 is where we finish last
> remaining features and apply final polish to API. There will be a beta-2
> but it is about upgrade/rolling-upgrade and bug fixes ONLY).
>
> Myself and Mr. Drob did a pass over the outstanding hbase-2.0.0-beta-1 list
> this morning. See here [1].
>
> There are about ~12 issues in progress with most of these about to land.
> There are 37 TODO. Many of these are tests we need to run, some are related
> to the backup/restore, but a good few are meaty w/o assignees.
>
> The awkward outstanding ones as I see it are the below:
>
> HBASE-18946 Stochastic load balancer assigns replica regions to the same RS
> HBASE-17204 Make L2 off heap cache default ON
> HBASE-19112 Suspect methods on Cell to be deprecated
> HBASE-19147 All branch-2 unit tests pass
>
> We need to make progress on the above or punt on them (can't punt on the
> last one though).
>
> Any ideas on what configs we should update in hbase2? Dump ideas into:
> HBASE-19148 Edit of default configuration
>
> Still hoping for an early December beta-1 RC. beta-2 hopefully will be
> close behind.
>
> Comments? Thoughts?
>
> Thanks all,
> S
>
> 1. https://issues.apache.org/jira/projects/HBASE/versions/12340861
>
> On Tue, Oct 31, 2017 at 2:56 PM, Stack  wrote:
>
> >
> >
> > On Tue, Oct 31, 2017 at 2:31 PM, Mike Drob  wrote:
> >
> >> Hoping to keep momentum going from our Stack working on alpha4, I tried
> to
> >> take a stab at triaging some of the open beta-1 issues.
> >>
> >> I moved some docs stuff out form beta-1 to 2.0-GA, if it gets done
> sooner
> >> then I'm happy to see it pulled back in. Trying to balance optimism with
> >> realism here, and knowing that documentation unfortunately often gets
> >> pushed to the back-burner.
> >>
> >> Also, I poked some folks on unassigned issues that they've filed for
> >> beta-1, especially in the last few days. If issues don't have an owner
> >> they
> >> are unlikely to get worked. I chatted with stack and he agreed to take
> on
> >> some of the tasks, but there's a lot of surface area to cover.
> >>
> >> If you you're working on issues that are critical for beta-1, please
> mark
> >> them as such. Then the rest of the community will know to help
> prioritize
> >> feedback and reviews there.
> >>
> >> Do we have a general theme for the betas like we did with the alphas?
> >>
> >> Beta1 is upgrades work from branch1, beta2 is rolling upgrades work as
> >> well? Continue to work on tests throughout?
> >>
> >>
> > Thanks Mike for helping to kick off the beta-1 train.
> >
> > Regards a theme for beta-1, I'd like it to be 'finish'; beta-1 is what we
> > are going to ship (beta-2 is rolling upgrade and any minor items turned
> up
> > in testing/burn-in).
> >
> > Thanks,
> > S
> >
> >
> >
> >> Mike
> >>
> >> On Tue, Oct 31, 2017 at 10:04 AM, Josh Elser  wrote:
> >>
> >> > +1 go from my POV.
> >> >
> >> >
> >> > On 10/31/17 10:07 AM, Stack wrote:
> >> >
> >> >> I want to push an alpha-4 today. A few items didn't make it
> >> (HBASE-19092).
> >> >> They need more time. We'll pull them in for beta-1. CP API is
> basically
> >> >> done. There may be some changes for beta-1 but hopefully only changes
> >> >> informed by experience trying to port an existing Coprocessor to
> >> hbase2.
> >> >>
> >> >> Shout if there is anything that needs to make alpha-4.
> >> >>
> >> >> Thanks,
> >> >> St.Ack
> >> >>
> >> >>
> >> >> On Sat, Oct 28, 2017 at 2:48 PM, Josh Elser 
> >> wrote:
> >> >>
> >> >> Yup, that was going to be my plan, Mike!
> >> >>>
> >> >>> Making a pass now, and will check back later tonight again. I see
> >> others
> >> >>> have already done some work today on this front.
> >> >>>
> >> >>>
> >> >>> On 10/27/17 11:38 PM, Mike Drob wrote:
> >> >>>
> >> >>> Josh - Do you want to kick off a bunch of QA runs? (Do you know how
> >> to do
> >>  it directly on the jenkins job, so you don't have to bother with
> JIRA
> >>  uploads)
> >> 
> >>  If you're busy, then I can make time tomorrow or Sunday to kick off
> >>  jobs,
> >>  but I want to make sure we're not duplicating effort and jenkins
> >> cycles.
> >> 
> >>  On Fri, Oct 27, 2017 at 7:10 PM, Josh Elser 
> >> wrote:
> >> 
> >>  My turn to bump ;)
> >> 
> >> >
> >> > By my take: HBASE-18770 and HBASE-19092 are the only issues that
> >> remain
> >> > needing some more work. The rest are just awaiting a good QA run.
> >> >
> >> > Unless I hear otherwise, I'll try to keep an eye on things over
> the
> >> > weekend, bump them along as necessary, and get them committed.
> >> Would be
> >> > great to be able get a vote up on Monday.

Re: [VOTE] First hbase-2.0.0-alpha4 Release Candidate is available

[ramkrishna vasudevan]

Done.
https://issues.apache.org/jira/browse/HBASE-19221

Thanks Stack.

Regards
Ram

On Thu, Nov 9, 2017 at 11:36 AM, Stack <st...@duboce.net> wrote:

> Thats a dirty rotten bug I'd say. File blocker on beta-1. Thanks Ram.
> S
>
> On Wed, Nov 8, 2017 at 8:56 PM, ramkrishna vasudevan <
> ramkrishna.s.vasude...@gmail.com> wrote:
>
> > Hi Stack
> > I tried running some IT test cases using the alpha-4 RC. I found this
> issue
> > Exception in thread "main" java.lang.NoClassDefFoundError:
> > org/hamcrest/SelfDescribing
> > at java.lang.ClassLoader.defineClass1(Native Method)
> > at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
> > at
> > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> > at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
> > at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
> > at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
> > at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
> >
> > ...
> >at
> > org.apache.hadoop.hbase.IntegrationTestsDriver.doWork(
> > IntegrationTestsDriver.java:111)
> > at
> > org.apache.hadoop.hbase.util.AbstractHBaseTool.run(
> > AbstractHBaseTool.java:154)
> > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> > at
> > org.apache.hadoop.hbase.IntegrationTestsDriver.main(
> > IntegrationTestsDriver.java:47)
> >
> > The same when run against latest master it runs without any issues. I
> have
> > not dug in here but wanted to just mention it here so that we could fix
> it
> > in beta-1?
> >
> > Regards
> > Ram
> >
> > On Sun, Nov 5, 2017 at 3:03 AM, Jean-Marc Spaggiari <
> > jean-m...@spaggiari.org
> > > wrote:
> >
> > > Thanks for your reply Stack. I will get ready to test the next version.
> > > Excited to see 2.0 coming out
> > >
> > > 2017-11-04 14:44 GMT-04:00 Stack <st...@duboce.net>:
> > >
> > > > On Sat, Nov 4, 2017 at 1:45 AM, Jean-Marc Spaggiari <
> > > > jean-m...@spaggiari.org
> > > > > wrote:
> > > >
> > > > > Folks, are you able to run the tests?
> > > > >
> > > > > I'm trying to run this:
> > > > >
> > > > > mvn test -P runAllTests -Dsurefire.secondPartThreadCount=2
> > > > > -Dtest.build.data.basedirectory=/ram4G
> > > > >
> > > > >
> > > > > But I'm not able to get it running completely. It always fails
> > > somewhere,
> > > > > never in the same place. How are you running the tests, if you are?
> > > > >
> > > > > Else:
> > > > > MD5 and SHA files correct for both Binary and Source
> > > > > *changes.txt file doesn't seems correct. Missing last versions*
> > > > > *checkstyle.html doesn't seems to be accurate. Reports 0 files.*
> > > > > Documetation ok.
> > > > >
> > > > > Will run more tests later when I can get runAllTests pass...
> > > > >
> > > >
> > > > Test fail JMS. We have to fix all for beta-1. I should have said this
> > in
> > > > the announcement. Thanks for giving it a go boss.
> > > > S
> > > >
> > > >
> > > > >
> > > > > 2017-11-03 3:32 GMT-04:00 Yung-An He <mathst...@gmail.com>:
> > > > >
> > > > > > +1 non-binding
> > > > > >
> > > > > > Checked MD5 : OK
> > > > > > Download the bin tarball : OK
> > > > > > Start from bin tar with cluster mode via docker container: OK
> > > > > > Import data with 100 million rows and 10 columns to table via
> > > > ImportTsv:
> > > > > OK
> > > > > > Execute basic command via `hbase shell`: OK
> > > > > >
> > > > > >
> > > > > > ‌
> > > > > >
> > > > > > 2017-11-02 5:02 GMT+08:00 Mike Drob <mad...@cloudera.com>:
> > > > > >
> > > > > > > +1
> > > > > > >
> > > &g

Re: [VOTE] First hbase-2.0.0-alpha4 Release Candidate is available

[ramkrishna vasudevan]

Hi Stack
I tried running some IT test cases using the alpha-4 RC. I found this issue
Exception in thread "main" java.lang.NoClassDefFoundError:
org/hamcrest/SelfDescribing
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
at
java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:361)

...
   at
org.apache.hadoop.hbase.IntegrationTestsDriver.doWork(IntegrationTestsDriver.java:111)
at
org.apache.hadoop.hbase.util.AbstractHBaseTool.run(AbstractHBaseTool.java:154)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at
org.apache.hadoop.hbase.IntegrationTestsDriver.main(IntegrationTestsDriver.java:47)

The same when run against latest master it runs without any issues. I have
not dug in here but wanted to just mention it here so that we could fix it
in beta-1?

Regards
Ram

On Sun, Nov 5, 2017 at 3:03 AM, Jean-Marc Spaggiari  wrote:

> Thanks for your reply Stack. I will get ready to test the next version.
> Excited to see 2.0 coming out
>
> 2017-11-04 14:44 GMT-04:00 Stack :
>
> > On Sat, Nov 4, 2017 at 1:45 AM, Jean-Marc Spaggiari <
> > jean-m...@spaggiari.org
> > > wrote:
> >
> > > Folks, are you able to run the tests?
> > >
> > > I'm trying to run this:
> > >
> > > mvn test -P runAllTests -Dsurefire.secondPartThreadCount=2
> > > -Dtest.build.data.basedirectory=/ram4G
> > >
> > >
> > > But I'm not able to get it running completely. It always fails
> somewhere,
> > > never in the same place. How are you running the tests, if you are?
> > >
> > > Else:
> > > MD5 and SHA files correct for both Binary and Source
> > > *changes.txt file doesn't seems correct. Missing last versions*
> > > *checkstyle.html doesn't seems to be accurate. Reports 0 files.*
> > > Documetation ok.
> > >
> > > Will run more tests later when I can get runAllTests pass...
> > >
> >
> > Test fail JMS. We have to fix all for beta-1. I should have said this in
> > the announcement. Thanks for giving it a go boss.
> > S
> >
> >
> > >
> > > 2017-11-03 3:32 GMT-04:00 Yung-An He :
> > >
> > > > +1 non-binding
> > > >
> > > > Checked MD5 : OK
> > > > Download the bin tarball : OK
> > > > Start from bin tar with cluster mode via docker container: OK
> > > > Import data with 100 million rows and 10 columns to table via
> > ImportTsv:
> > > OK
> > > > Execute basic command via `hbase shell`: OK
> > > >
> > > >
> > > > ‌
> > > >
> > > > 2017-11-02 5:02 GMT+08:00 Mike Drob :
> > > >
> > > > > +1
> > > > >
> > > > > Verified sigs and hashes.
> > > > > Src tar matches git tag.
> > > > > Able to build from src tar w/ hadoop 2 & 3
> > > > > Able to start from bin tar, verified basic functionality.
> > > > >
> > > > > Filed HBASE-19151 for some warning messages on shell startup.
> > > > > Failed to start with Java9, but didn't expect this to work anyway.
> > > > >
> > > > > On Wed, Nov 1, 2017 at 1:38 PM, Artem Ervits <
> artemerv...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > +1
> > > > > >
> > > > > > Checked md5
> > > > > > ran LTT tool with 10k rows
> > > > > > ran PE tool with 6M rows
> > > > > > executed counter in hbase shell
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Wed, Nov 1, 2017 at 1:18 PM, Ted Yu 
> > wrote:
> > > > > >
> > > > > > > +1
> > > > > > >
> > > > > > > Checked signatures
> > > > > > > Built from source
> > > > > > > Ran load test tool with 1 mil rows.
> > > > > > >
> > > > > > > On Wed, Nov 1, 2017 at 7:17 AM, Stack 
> wrote:
> > > > > > >
> > > > > > > > The first release candidate for HBase 2.0.0-alpha4 is up at:
> > > > > > > >
> > > > > > > >   https://dist.apache.org/repos/dist/dev/hbase/hbase-2.0.0-
> > > > > alpha4RC0/
> > > > > > > >
> > > > > > > > Maven artifacts are available from a staging directory here:
> > > > > > > >
> > > > > > > >   https://repository.apache.org/
> content/repositories/orgapach
> > > > > > ehbase-1178
> > > > > > > >
> > > > > > > > All was signed with my key at 8ACC93D2 [1]
> > > > > > > >
> > > > > > > > I tagged the RC as 2.0.0-alpha4RC0
> > > > > > > > (5c4b985f89c99cc8b0f8515a4097c811a0848835)
> > > > > > > >
> > > > > > > > hbase-2.0.0-alpha4 is our fourth alpha release along our
> march
> > > > toward
> > > > > > > > hbase-2.0.0. It includes all that was in previous alphas (new
> > > > > > assignment
> > > > > > > > manager, offheap read/write 

Re: [VOTE] First hbase-2.0.0-alpha4 Release Candidate is available

[ramkrishna vasudevan]

Downloaded src tar and bin tar.
After building the src code tried to start from the src/bin folder. But it
had the cached_classpath issue. I can see that already the issue has been
backported.

Then started master and region server with bin tar. Ran PE tool. All looks
fine.

Infact ran PE tool with AsyncWAL also to verify its performance. Seems it
is faster than the normal WAL. (but we are not looking for this now).
Overall +1.

Regards
Ram

On Thu, Nov 2, 2017 at 10:39 PM, Stack  wrote:

> I backported HBASE-18705.
> St.Ack
>
> On Thu, Nov 2, 2017 at 10:05 AM, Stack  wrote:
>
> > Thank you Guanghao. Let me see about getting the issue backported. Are
> you
> > a +/- 1 sir? Thanks for the digging.
> > S
> >
> > On Thu, Nov 2, 2017 at 1:31 AM, Guanghao Zhang 
> wrote:
> >
> >> There is already a issue HBASE-18705
> >>  about this problem.
> >> But
> >> it was only pushed to master branch.
> >>
> >> 2017-11-02 15:21 GMT+08:00 Guanghao Zhang :
> >>
> >> > Updated f="${HBASE_HOME}/target/cached_classpath.txt" to
> >> > f="${HBASE_HOME}/hbase-build-configuration/target/cached_cla
> >> sspath.txt".
> >> > Then start hbase success. Anything wrong for my environment?
> >> >
> >> > Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5;
> >> > 2015-11-11T00:41:47+08:00)
> >> > Maven home: /opt/soft/apache-maven-3.3.9
> >> > Java version: 1.8.0_131, vendor: Oracle Corporation
> >> > Java home: /opt/soft/jdk1.8.0_131/jre
> >> > Default locale: zh_CN, platform encoding: UTF-8
> >> > OS name: "linux", version: "3.13.0-46-generic", arch: "amd64", family:
> >> > "unix"
> >> >
> >> >
> >> >
> >> > 2017-11-02 15:18 GMT+08:00 Guanghao Zhang :
> >> >
> >> >> Download src and build pass.
> >> >> But when start-hbase.sh, I found the master can't start in standalone
> >> >> mode.
> >> >>
> >> >> ```
> >> >> running master, logging to /home/hao/open_source/hbase_re
> >> >> lease/hbase-2.0.0-alpha4/bin/../logs/hbase-hao-master-Xiaohao.out
> >> >> As this is a development environment, we need
> >> >> /home/hao/open_source/hbase_release/hbase-2.0.0-alpha4/bin/.
> >> ./target/cached_classpath.txt
> >> >> to be generated from maven (command: mvn install -DskipTests)
> >> >> : running regionserver, logging to /home/hao/open_source/hbase_re
> >> >> lease/hbase-2.0.0-alpha4/bin/../logs/hbase-hao-regionserver-
> >> Xiaohao.out
> >> >> : As this is a development environment, we need
> >> >> /home/hao/open_source/hbase_release/hbase-2.0.0-alpha4/bin/.
> >> ./target/cached_classpath.txt
> >> >> to be generated from maven (command: mvn install -DskipTests)
> >> >> ```
> >> >>
> >> >> And I try mvn install -DskipTests, and found the cached_classpath.txt
> >> >> in hbase-build-configuration/target/cached_classpath.txt. The path
> is
> >> >> wrong in bin/hbase. So I update f="${HBASE_HOME}/hbase-
> >> >> build-configuration/target/cached_classpath.txt" in bin/hbase, then
> >> >> start master success. File a jira for this?
> >> >>
> >> >> 2017-11-02 14:02 GMT+08:00 Stack :
> >> >>
> >> >>> On Wed, Nov 1, 2017 at 9:41 PM, Josh Elser 
> wrote:
> >> >>>
> >> >>> > +1 (binding)
> >> >>> >
> >> >>> > * UT's mostly ok. Exceptions being: TestNamespacesInstanceResource
> ,
> >> >>> > TestReplicaWithCluster, TestReplicationAdmin, and
> >> >>> TestRegionServerHostname
> >> >>> > consistently fail for me locally (commented on HBASE-19147)
> >> >>> > * xsum+sigs OK
> >> >>> > * can build from source
> >> >>> > * src archive contents looks OK (exception below)
> >> >>> > * Built, deployed locally, ran PE without issue
> >> >>> >
> >> >>> >
> >> >>> Excellent. Thanks Josh.
> >> >>>
> >> >>>
> >> >>>
> >> >>> > I stumbled across src/main/site/resources/repo/o
> >> >>> > rg/apache/maven/skins/maven-fluido-skin/1.5-HBASE/maven-flui
> >> >>> do-skin-1.5-HBASE.jar
> >> >>> > in the source archive which was introduced via
> >> >>> > https://issues.apache.org/jira/browse/HBASE-14785. Best as I can
> >> tell,
> >> >>> > this was a hack to work around a fluido-skin bug and we should be
> >> able
> >> >>> to
> >> >>> > remove it now. Not sure if we've already discussed the
> >> >>> > binary-file-in-src-release concerns of this previously or not...
> >> >>> >
> >> >>> >
> >> >>> Sean noted this in an alpha-3 VOTE.
> >> >>>
> >> >>> I came across it yesterday.
> >> >>>
> >> >>> What you reckon Misty (if you looking at this?) If you remember,
> what
> >> >>> would
> >> >>> I look for?
> >> >>>
> >> >>> Thanks,
> >> >>> St.Ack
> >> >>>
> >> >>>
> >> >>>
> >> >>>
> >> >>>
> >> >>> > - Josh
> >> >>> >
> >> >>> >
> >> >>> > On 11/1/17 10:17 AM, Stack wrote:
> >> >>> >
> >> >>> >> The first release candidate for HBase 2.0.0-alpha4 is up at:
> >> >>> >>
> >> >>> >>https://dist.apache.org/repos/dist/dev/hbase/hbase-2.0.0-alp
> >> >>> ha4RC0/
> >> >>> >>
> >> >>> >> Maven artifacts are 

Re: Struggles around Cell#getType()

[ramkrishna vasudevan]

Sorry just to clarify I mean deprecating the getType in Cell can we try
doing it in 2.0-alpha 4.

On Fri, Oct 27, 2017 at 9:45 AM, ramkrishna vasudevan <
ramkrishna.s.vasude...@gmail.com> wrote:

> bq.Cell#getType()
> We had this discussion. So getType should only be used for user exposed
> types like Put and Deletes. All others are internal. So having it in public
> interface may not be needed. Shall we do this in 2.0 alpha-4? Am +1 to do
> this.
>
> How ever to solve your problem I think you may need CellUtil#isPut(Cell)
> sort of API in CellUtl like you already have isDelete(Cell).
>
> Regards
> Ram
>
> On Fri, Oct 27, 2017 at 9:08 AM, Ted Yu <yuzhih...@gmail.com> wrote:
>
>> There is also CellBuilder#DataType which is public. However, the ordinals
>> of CellBuilder#DataType are different from KeyValue.Type .
>>
>> What if we align the ordinals of CellBuilder#DataType to be the same as
>> those from KeyValue.Type ?
>>
>> On Thu, Oct 26, 2017 at 4:34 PM, Sergey Soldatov <
>> sergeysolda...@gmail.com>
>> wrote:
>>
>> > DataType class was introduced as part of HBASE-8693 which is more about
>> the
>> > type of data in the cell rather than the type of mutation.
>> >
>> > Thanks,
>> > Sergey
>> >
>> > On Thu, Oct 26, 2017 at 3:40 PM, Josh Elser <els...@apache.org> wrote:
>> >
>> > > Hiya,
>> > >
>> > > (Background: see HBASE-19002)
>> > >
>> > > In trying to write some example Observers, I found myself in a pickle:
>> > how
>> > > do I tell if a Cell is a Put?
>> > >
>> > > * Cell#getType() returns a byte which corresponds to a KeyValue.Type
>> > > * KeyValue.Type has API to convert a byte to Type
>> > > * KeyValue (and thus KeyValue.Type) is IA.Private
>> > > * DataType o.a.h.h.typesDataType _appears to me_ to be the replacement
>> > for
>> > > the KeyValue.Type
>> > >
>> > > Best as I can tell, Cell#getType() should be deprecated and we should
>> > have
>> > > some kind of API (method on Cell or CellUtil) which returns a DataType
>> > > instead of Type. The details of the byte and the KeyValue.Type should
>> be
>> > > hidden inside the implementation.
>> > >
>> > > My hunch is that this is an accidental omission, but Stack recommended
>> > > that I "ask the class" ;). What have I missed? I think this is
>> trivial to
>> > > fix; obviously, I don't want to make a fix if I just didn't look hard
>> > > enough.
>> > >
>> > > Thanks!
>> > >
>> > > - Josh
>> > >
>> >
>>
>
>

Re: Struggles around Cell#getType()

[ramkrishna vasudevan]

bq.Cell#getType()
We had this discussion. So getType should only be used for user exposed
types like Put and Deletes. All others are internal. So having it in public
interface may not be needed. Shall we do this in 2.0 alpha-4? Am +1 to do
this.

How ever to solve your problem I think you may need CellUtil#isPut(Cell)
sort of API in CellUtl like you already have isDelete(Cell).

Regards
Ram

On Fri, Oct 27, 2017 at 9:08 AM, Ted Yu  wrote:

> There is also CellBuilder#DataType which is public. However, the ordinals
> of CellBuilder#DataType are different from KeyValue.Type .
>
> What if we align the ordinals of CellBuilder#DataType to be the same as
> those from KeyValue.Type ?
>
> On Thu, Oct 26, 2017 at 4:34 PM, Sergey Soldatov  >
> wrote:
>
> > DataType class was introduced as part of HBASE-8693 which is more about
> the
> > type of data in the cell rather than the type of mutation.
> >
> > Thanks,
> > Sergey
> >
> > On Thu, Oct 26, 2017 at 3:40 PM, Josh Elser  wrote:
> >
> > > Hiya,
> > >
> > > (Background: see HBASE-19002)
> > >
> > > In trying to write some example Observers, I found myself in a pickle:
> > how
> > > do I tell if a Cell is a Put?
> > >
> > > * Cell#getType() returns a byte which corresponds to a KeyValue.Type
> > > * KeyValue.Type has API to convert a byte to Type
> > > * KeyValue (and thus KeyValue.Type) is IA.Private
> > > * DataType o.a.h.h.typesDataType _appears to me_ to be the replacement
> > for
> > > the KeyValue.Type
> > >
> > > Best as I can tell, Cell#getType() should be deprecated and we should
> > have
> > > some kind of API (method on Cell or CellUtil) which returns a DataType
> > > instead of Type. The details of the byte and the KeyValue.Type should
> be
> > > hidden inside the implementation.
> > >
> > > My hunch is that this is an accidental omission, but Stack recommended
> > > that I "ask the class" ;). What have I missed? I think this is trivial
> to
> > > fix; obviously, I don't want to make a fix if I just didn't look hard
> > > enough.
> > >
> > > Thanks!
> > >
> > > - Josh
> > >
> >
>

Re: [ANNOUNCE] New HBase committer Lars Francke

[ramkrishna vasudevan]

Welcome Lars..

On Thu, Oct 26, 2017 at 9:54 AM, Misty Stanley-Jones 
wrote:

> Welcome Lars!
>
> On Wed, Oct 25, 2017 at 7:24 PM Jingcheng Du  wrote:
>
> > Congratulations!
> >
> > 2017-10-26 9:45 GMT+08:00 OpenInx :
> >
> > > Congratulations.
> > >
> > > On Thu, Oct 26, 2017 at 9:36 AM, 张铎(Duo Zhang) 
> > > wrote:
> > >
> > > > Congratulations!
> > > >
> > > > 2017-10-26 4:35 GMT+08:00 Esteban Gutierrez :
> > > >
> > > > > Congrats Lars F! well deserved!
> > > > >
> > > > > --
> > > > > Cloudera, Inc.
> > > > >
> > > > >
> > > > > On Wed, Oct 25, 2017 at 3:26 PM, Andrew Purtell <
> apurt...@apache.org
> > >
> > > > > wrote:
> > > > >
> > > > > > Congratulations and welcome!
> > > > > >
> > > > > >
> > > > > > On Wed, Oct 25, 2017 at 12:56 PM, Lars George <
> > lars.geo...@gmail.com
> > > >
> > > > > > wrote:
> > > > > >
> > > > > > > On behalf of the Apache HBase PMC, I am pleased to announce
> that
> > > Lars
> > > > > > > Francke has accepted the PMC's invitation to become a committer
> > on
> > > > the
> > > > > > > project.
> > > > > > >
> > > > > > > We appreciate all of Lars' great work thus far and look forward
> > to
> > > > > > > continued involvement.
> > > > > > >
> > > > > > > Please join me in congratulating LarsF! (Opting to use last
> name
> > > > > > > initials as we now have three Lars' as committers)
> > > > > > >
> > > > > > > --
> > > > > > > Best regards,
> > > > > > > LarsG
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Best regards,
> > > > > > Andrew
> > > > > >
> > > > > > Words like orphans lost among the crosstalk, meaning torn from
> > > truth's
> > > > > > decrepit hands
> > > > > >- A23, Crosstalk
> > > > > >
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > ==
> > > Openinx  blog : http://openinx.github.io
> > >
> > > TO BE A GREAT HACKER !
> > > ==
> > >
> >
>

Re: Moving 2.0 forward

[ramkrishna vasudevan]

Thanks Stack. Started working on it. Will post a patch ASAP.

On Tue, Oct 24, 2017 at 11:38 AM, Stack <st...@duboce.net> wrote:

> That'd be a good one to get in Ram.
> S
>
> On Mon, Oct 23, 2017 at 9:42 PM, ramkrishna vasudevan <
> ramkrishna.s.vasude...@gmail.com> wrote:
>
> > Hi Stack
> >
> > Do you want HBASE-18995 before the alpha-4 (REmoving exposed internal
> APIs
> > from CellUtil)? Because you had mentioned no more API changes. If so I
> will
> > start making changes and put up a patch ASAP.
> >
> > Regards
> > Ram
> >
> > On Tue, Oct 24, 2017 at 3:22 AM, Stack <st...@duboce.net> wrote:
> >
> > > On Mon, Oct 23, 2017 at 11:59 AM, Josh Elser <els...@apache.org>
> wrote:
> > >
> > > > +1
> > > >
> > > > I was trying to work on helping out on the outstanding alpha-4 stuff
> > last
> > > > week -- will be continuing to try to do the same this week.
> > > >
> > > > If you need any help, Stack, or if others need reviews where I
> haven't
> > > > noticed on my own: feel free to @mention me.
> > > >
> > > >
> > > Thanks for the offer Josh. All items seem assigned and are being
> actively
> > > worked on. If you get a moment, reviews by you (or anyone else) helps
> > move
> > > the process along.
> > >
> > > We need to merge in HBASE-18410 branch to pick up Filter improvements.
> > Then
> > > HBASE-13346 can go in.
> > >
> > > You are already helping out on HBASE-18906, thanks. Looks like that
> will
> > be
> > > addressed by other alpha-4s about to land.
> > >
> > > St.Ack
> > > TODOs: https://issues.apache.org/jira/projects/HBASE/versions/12341594
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > > On 10/23/17 12:53 PM, Stack wrote:
> > > >
> > > >> (Reviving this thread)
> > > >>
> > > >> Lets push out alpha-4 this week. Alpha-4 is the release that has the
> > > >> refactor of the Coprocessor API shutting down access to internals
> > marked
> > > >> InterfaceAudience.Private.
> > > >>
> > > >> The outstanding list is here:
> > > >> https://issues.apache.org/jira/projects/HBASE/versions/12341594
> > > >>
> > > >> Please push in anything marked alpha-4 that belongs to you.
> > > >>
> > > >> If issue, talk out loud on this thread. If you need a review to land
> > an
> > > >> item, shout on the issue and here; we'll help you out.
> > > >>
> > > >> As is, items are coming along nicely I'd say. We need to merge the
> > > filter
> > > >> branch -- HBASE-18410 -- so APIs are finished for hbase2.
> > > >>
> > > >> Post alpha-4, we'll have to hunt down our downstreamers and help
> them
> > > test
> > > >> on top of alpha-4 so rolling into beta-1, we have confidence our
> > > >> downstreamers know what to expect (or we discover what we missed
> > BEFORE
> > > we
> > > >> beta-1).
> > > >>
> > > >> Thanks for time,
> > > >> S
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >> On Fri, Sep 8, 2017 at 2:04 PM, Stack <st...@duboce.net> wrote:
> > > >>
> > > >> I'll put up an alpha3 RC Monday, probably Monday night. That should
> be
> > > >>> time, if we all sprint, for the public-facing API fixes to be done.
> > > >>>
> > > >>> I had a bunch of Coprocessor refactor and fixup scheduled for
> alpha3
> > > but
> > > >>> it is plain that more time is needed (in spite of valiant effort so
> > far
> > > >>> by
> > > >>> Anoop, Duo, Appy, etc.). Therefore, lets run a 2.0.0-alpha-4 whose
> > > theme
> > > >>> is
> > > >>> "Coprocessor Fixup". Hopefully we can put an alpha-4 up by the
> > > following
> > > >>> week.
> > > >>>
> > > >>> We should then be ready for beta (beta == no new features, no API
> > > >>> changes,
> > > >>> just fixes).
> > > >>>
> > > >>> Thanks,
> > > >>> 

Re: Moving 2.0 forward

[ramkrishna vasudevan]

Hi Stack

Do you want HBASE-18995 before the alpha-4 (REmoving exposed internal APIs
from CellUtil)? Because you had mentioned no more API changes. If so I will
start making changes and put up a patch ASAP.

Regards
Ram

On Tue, Oct 24, 2017 at 3:22 AM, Stack  wrote:

> On Mon, Oct 23, 2017 at 11:59 AM, Josh Elser  wrote:
>
> > +1
> >
> > I was trying to work on helping out on the outstanding alpha-4 stuff last
> > week -- will be continuing to try to do the same this week.
> >
> > If you need any help, Stack, or if others need reviews where I haven't
> > noticed on my own: feel free to @mention me.
> >
> >
> Thanks for the offer Josh. All items seem assigned and are being actively
> worked on. If you get a moment, reviews by you (or anyone else) helps move
> the process along.
>
> We need to merge in HBASE-18410 branch to pick up Filter improvements. Then
> HBASE-13346 can go in.
>
> You are already helping out on HBASE-18906, thanks. Looks like that will be
> addressed by other alpha-4s about to land.
>
> St.Ack
> TODOs: https://issues.apache.org/jira/projects/HBASE/versions/12341594
>
>
>
>
>
>
>
>
>
> > On 10/23/17 12:53 PM, Stack wrote:
> >
> >> (Reviving this thread)
> >>
> >> Lets push out alpha-4 this week. Alpha-4 is the release that has the
> >> refactor of the Coprocessor API shutting down access to internals marked
> >> InterfaceAudience.Private.
> >>
> >> The outstanding list is here:
> >> https://issues.apache.org/jira/projects/HBASE/versions/12341594
> >>
> >> Please push in anything marked alpha-4 that belongs to you.
> >>
> >> If issue, talk out loud on this thread. If you need a review to land an
> >> item, shout on the issue and here; we'll help you out.
> >>
> >> As is, items are coming along nicely I'd say. We need to merge the
> filter
> >> branch -- HBASE-18410 -- so APIs are finished for hbase2.
> >>
> >> Post alpha-4, we'll have to hunt down our downstreamers and help them
> test
> >> on top of alpha-4 so rolling into beta-1, we have confidence our
> >> downstreamers know what to expect (or we discover what we missed BEFORE
> we
> >> beta-1).
> >>
> >> Thanks for time,
> >> S
> >>
> >>
> >>
> >>
> >>
> >> On Fri, Sep 8, 2017 at 2:04 PM, Stack  wrote:
> >>
> >> I'll put up an alpha3 RC Monday, probably Monday night. That should be
> >>> time, if we all sprint, for the public-facing API fixes to be done.
> >>>
> >>> I had a bunch of Coprocessor refactor and fixup scheduled for alpha3
> but
> >>> it is plain that more time is needed (in spite of valiant effort so far
> >>> by
> >>> Anoop, Duo, Appy, etc.). Therefore, lets run a 2.0.0-alpha-4 whose
> theme
> >>> is
> >>> "Coprocessor Fixup". Hopefully we can put an alpha-4 up by the
> following
> >>> week.
> >>>
> >>> We should then be ready for beta (beta == no new features, no API
> >>> changes,
> >>> just fixes).
> >>>
> >>> Thanks,
> >>> St.Ack
> >>>
> >>>
> >>> On Thu, Aug 17, 2017 at 12:35 PM, Stack  wrote:
> >>>
> >>> I put up the hbase-2.0.0-alpha2 release candidate. Please vote on it.
> 
>  For hbase-2.0.0-alpha3, the theme is solidifying API. I hope to get a
>  release out in the next week or so.
> 
>  I did a weeding of 2.0.0 issues over the last day. If folks are
>  interested in helping out, below are the items I think we need done
> for
>  alpha3 (below are at least 'Critical' status, are API possibly
> altering
>  items, and are absent those JIRAs that are making active progress,
> i.e.
>  the
>  HTD/HCD revamp by Chia-Ping Tsai). A project NOT listed that needs
>  doing is
>  what Andrew did comparing 1.3. and 1.4 APIs
> 
>  * HBASE-18622 Mitigate compatibility concerns between branch-1 and
>  branch-2
>  This is to do what Andrew did between 1.3 and 1.4 branches only do it
>  between branch-1 and branch-2.
> 
>  * HBASE-10462 Recategorize some of the client facing Public / Private
>  interfaces
>  This one is almost done. It could do with a finish, attention to the
>  items in last comment, and then our codebase could do with another
> sweep
>  after the spirit of this issue since a bunch has gone in since the
> pass
>  that was the basis of this issue.
> 
>  * HBASE-10504 Define Replication Interface
>  I was going to take a crack at this as part of the revamp forced by
>  'HBASE-15982 Interface ReplicationEndpoint extends Guava's Service'
> but
>  if
>  anyone else is interested, be my guest.
> 
>  * HBASE-14996 Some more API cleanup for 2.0
>  Has a bunch of subtasks, some of which are being worked on. Needs
>  finishing.
> 
>  * HBASE-14998 Unify synchronous and asynchronous methods in Admin and
>  cleanup
>  Needs a pass. Small issue I think. Could also look at new AsyncClient
>  and
>  make sure symmetry.
> 
>  * HBASE-15607 Remove PB references from Admin for 2.0
>  

Re: [ANNOUNCE] New HBase committer Zheng Hu

[ramkrishna vasudevan]

Welcome Zheng and congratulations !!!

On Mon, Oct 23, 2017 at 11:48 AM, Duo Zhang  wrote:

> On behalf of the Apache HBase PMC, I am pleased to announce that Zheng Hu
> has accepted the PMC's invitation to become a committer on the project. We
> appreciate all of Zheng's generous contributions thus far and look forward
> to his continued involvement.
>
> Congratulations and welcome, Zheng!
>

Re: [DISCUSSION] Removing the bypass semantic from the Coprocessor APIs

[ramkrishna vasudevan]

Probably bypass related hooks we should fix by understanding more use
cases. Should we have it only for the mutations and scanner creations? And
if not direct by pass way is there any other way we can provide for the CPs
to create their own behaviour.

Regards
Ram





On Wed, Oct 11, 2017 at 9:07 AM, Stack  wrote:

> On Tue, Oct 10, 2017 at 3:12 PM, Stack  wrote:
>
> > I've been splunking Phoenix code and I see that in SequenceRegionObserver
> > and in Indexer, they do by-pass doing their own version of Increment.
> >
> >
>
> Following in the previous vein of offering alternatives, Phoenix has its
> own Increment because of "...deficiencies in Increment implementation
> (HBASE-10254)":
>
>  1) Lack of recognition and identification of when the key value to
> increment doesn't exist
>  2) Lack of the ability to set the timestamp of the updated key value.
>
>
> Works the same as existing region.increment(), except assumes there is a
> single column to increment and uses Phoenix LONG encoding.
>
>
> I think #1 in above is at least improved and if not good enough, we could
> fix. On #2, could add an ability to search the Increment payload for a
> timestamp to use. Assuming a single column seems like a downgrade. Phoenix
> LONG encoding might be hard to do unless we want to pull-in phoenix libs.
>
>
> St.Ack
>
>
>
>
>
>
>
>
>
>
> > St.Ack
> >
> > On Tue, Oct 10, 2017 at 11:49 AM, Stack  wrote:
> >
> >> On Tue, Oct 10, 2017 at 11:10 AM, Andrew Purtell 
> >> wrote:
> >>
> >>> The coprocessor API provides an environment method, bypass(), that when
> >>> called from a preXXX hook will cause the core code to skip all
> remaining
> >>> processing. This capability was introduced on HBASE-3348. Since this
> >>> time I
> >>> think we are more enlightened about the complications of this feature.
> >>> (Or,
> >>> anyway, speaking for myself:)
> >>>
> >>> Not all hooks provide the bypass semantic. Where this is the case the
> >>> javadoc for the hook says so, but it can be missed. If you call
> bypass()
> >>> in
> >>> a hook where it is not supported it is a no-op. This can lead to a poor
> >>> developer experience.
> >>>
> >>> Where bypass is supported what is being bypassed is all of the core
> code
> >>> implementing the remainder of the operation. In order to understand
> what
> >>> calling bypass() will skip, a coprocessor implementer should read and
> >>> understand all of the remaining code and its nuances. Although I think
> >>> this
> >>> is good practice for coprocessor developers in general, it demands a
> >>> lot. I
> >>> think it would provide a much better developer experience if we didn't
> >>> allow bypass, even though it means - in theory - a coprocessor would
> be a
> >>> lot more limited in some ways than before. What is skipped is extremely
> >>> version dependent. That core code will vary, perhaps significantly,
> even
> >>> between point releases. We do not provide the promise of consistent
> >>> behavior even between point releases for the bypass semantic. To
> achieve
> >>> that we could not change any code between hook points. Therefore the
> >>> coprocessor implementer becomes an HBase core developer in practice as
> >>> soon
> >>> as they rely on bypass(). Every release of HBase may break the
> assumption
> >>> that the replacement for the bypassed code takes care of all necessary
> >>> skipped concerns. Because those concerns can change at any point, such
> an
> >>> assumption is never safe.
> >>>
> >>> I say "in theory" because I would be surprised if anyone is relying on
> >>> the
> >>> bypass for the above reason. I seem to recall that Phoenix might use it
> >>> in
> >>> one place to promote a normal mutation into an atomic operation, by
> >>> substituting one for the other, but if so that objective could be
> >>> reimplemented using their new locking manager.
> >>>
> >>>
> >>
> >> Thanks Andrew for starting the discussion.
> >>
> >> Up in JIRA we've also talked of a 'wonky' case where CPs need
> >> write-access to the internal Metrics system so a CP on by-pass is able
> to
> >> increment standard counters. For example, a CP might bypass a Get
> operation
> >> instead returning a Cell from a CP-managed cache. In this latter case,
> it
> >> might want to increment the Get metrics counters so the by-pass shows in
> >> the general Get counts.
> >>
> >> This is problematic but there is at least an instance happening
> >> downstream. We'd like to keep internal metrics internal. CPs should be
> able
> >> to publish their own metrics.
> >>
> >> Please speak up if you depend on by-pass.
> >>
> >> S
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>> --
> >>> Best regards,
> >>> Andrew
> >>>
> >>
> >>
> >
>

Re: Branch 1.4 update

[ramkrishna vasudevan]

As said in  HBASE-18786  some exception handling for FileNot Found
Exception was the reason for the issues posted by FB. All were actually
related one way or the other. HBASE-18771 has fixed the issue when that
FileNotFound happens.

If facebook can provide feedback post HBASE-18771 I think that can be
considered as a positive outcome to make it as stable pointer?

Regards
Ram

On Thu, Oct 5, 2017 at 5:36 AM, Sean Busbey  wrote:

> Just to be clear my statement was meant to be "I'm looking forward to
> seeing rsgroups in a 1.4 release so we can get more feedback on how useful
> it is." Realized it could have been read as wanting more feedback prior to
> 1.4 inclusion.
>
> Cloudera has been shipping the rsgroups feature backported onto cdh5 (which
> is branch-1-ish) but I haven't gotten any real feedback on it in production
> settings as a result thus far. Also have been personally conservative about
> recommending it as a result.
>
> On Oct 4, 2017 19:00, "Andrew Purtell"  wrote:
>
> > Francis Liu has talked before about how useful this feature has been for
> > them.
> >
> > Unfortunately I won't be able to give you personal feedback until some
> time
> > after it is out in the 1.4 code line. The way the schedule looks a 1.4.0
> > release will precede (and be a prerequisite for) enabling rsgroups in our
> > production.
> >
> >
> > On Wed, Oct 4, 2017 at 4:23 PM, Sean Busbey  wrote:
> >
> > > Sounds great to me. Would love to see more feedback on the utility of
> the
> > > rsgroups feature.
> > >
> > > What ever happened to the issue that was keeping 1.3 from becoming the
> > > stable release line? That still outstanding for 1.4 too?
> > >
> > > On Oct 4, 2017 16:24, "Andrew Purtell"  wrote:
> > >
> > > > I had hoped to make a release candidate of 1.4.0 a long time ago
> when I
> > > > first made the branch, but work/life intervened. Sorry it has been
> slow
> > > > going.
> > > >
> > > > I've been making some test fixes to shore up unit test suite results.
> > > >
> > > > As far as big items, at this point a candidate 1.4.0 is only waiting
> on
> > > > HBASE-15631 (https://reviews.apache.org/r/60672/). I had to go away
> > for
> > > > work for a while but am now back pushing that forward. For me as
> > > branch-1.4
> > > > RM regionserver assignment groups is the must have feature for the
> 1.4
> > > code
> > > > line. If anyone has a strong feeling against this, let's discuss it
> > now.
> > > I
> > > > have asked before and I believe there are no objections to doing the
> > > > backport because the functional changes are entirely contained to the
> > new
> > > > hbase-rsgroups module.
> > > >
> > > > Once HBASE-15631 is resolved I will be able to make a RC. This should
> > be
> > > > soon. I've addressed all of the review feedback up on RB and
> anticipate
> > > > only minor additional changes to fix any issues discovered during
> unit
> > > and
> > > > cluster testing.
> > > >
> > > > --
> > > > Best regards,
> > > > Andrew
> > > >
> > >
> >
> >
> >
> > --
> > Best regards,
> > Andrew
> >
> > Words like orphans lost among the crosstalk, meaning torn from truth's
> > decrepit hands
> >- A23, Crosstalk
> >
>

Re: Error : java.lang.UnsatisfiedLinkError: failed to load the required native library for netty

[ramkrishna vasudevan]

I too tried just now but does not seem to work.

>>Sorry for the irritation.
No problem Stack.

Regards
Ram

On Wed, Oct 4, 2017 at 9:58 PM, Amit Kabra <amitkabrai...@gmail.com> wrote:

> I tried that , didn't work for me.
>
> ramkrishna.s.vasude...@gmail.com - can you try and see if that works for
> you ?
>
>
> Amit.
>
> On Wed, Oct 4, 2017 at 8:34 PM, Stack <st...@duboce.net> wrote:
>
> > On Tue, Oct 3, 2017 at 10:18 PM, ramkrishna vasudevan <
> > ramkrishna.s.vasude...@gmail.com> wrote:
> >
> > > Hi Stack
> > > I just took an update and tried running some test cases with the
> eclipse
> > > IDE working on linux
> > >
> > >
> > If you bring up the test run configuration window in eclipse and add in
> the
> > command-line argument panel
> >
> >
> > -Dorg.apache.hadoop.hbase.shaded.io.netty.packagePrefix=
> > org.apache.hadoop.hbase.shaded.
> >
> > ... does it pass?
> >
> > If so, I'll figure how to get it into eclipse context.
> >
> > Sorry for the irritation.
> >
> > St.Ack
> >
> >
> >
> >
> >
> >
> > > java.io.IOException: Shutting down
> > > at org.apache.hadoop.hbase.MiniHBaseCluster.init(
> > > MiniHBaseCluster.java:232)
> > > at org.apache.hadoop.hbase.MiniHBaseCluster.(
> > > MiniHBaseCluster.java:94)
> > > at
> > > org.apache.hadoop.hbase.HBaseTestingUtility.startMiniHBaseCluster(
> > > HBaseTestingUtility.java:1128)
> > > at
> > > org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(
> > > HBaseTestingUtility.java:1082)
> > > at
> > > org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(
> > > HBaseTestingUtility.java:953)
> > > at
> > > org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(
> > > HBaseTestingUtility.java:935)
> > > at
> > > org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(
> > > HBaseTestingUtility.java:917)
> > > at
> > > org.apache.hadoop.hbase.regionserver.TestRegionReplicasWithModifyTa
> > > ble.before(TestRegionReplicasWithModifyTable.java:65)
> > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > > at
> > > sun.reflect.NativeMethodAccessorImpl.invoke(
> > NativeMethodAccessorImpl.java:
> > > 62)
> > > at
> > > sun.reflect.DelegatingMethodAccessorImpl.invoke(
> > > DelegatingMethodAccessorImpl.java:43)
> > > 
> > >
> > > Caused by: java.lang.UnsatisfiedLinkError: failed to load the required
> > > native library
> > > at
> > > org.apache.hadoop.hbase.shaded.io.netty.channel.epoll.
> > > Epoll.ensureAvailability(Epoll.java:78)
> > > at
> > > org.apache.hadoop.hbase.shaded.io.netty.channel.epoll.
> > > EpollEventLoopGroup.(EpollEventLoopGroup.java:38)
> > > at
> > > org.apache.hadoop.hbase.util.NettyEventLoopGroupConfig.(
> > > NettyEventLoopGroupConfig.java:61)
> > > at
> > > org.apache.hadoop.hbase.regionserver.HRegionServer.<
> > > init>(HRegionServer.java:552)
> > > at org.apache.hadoop.hbase.master.HMaster.(HMaster.java:475)
> > > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> > > at
> > > sun.reflect.NativeConstructorAccessorImpl.newInstance(
> > > NativeConstructorAccessorImpl.java:62)
> > > at
> > > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(
> > > DelegatingConstructorAccessorImpl.java:45)
> > > at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
> > > at
> > > org.apache.hadoop.hbase.util.JVMClusterUtil.createMasterThread(
> > > JVMClusterUtil.java:140)
> > > ... 26 more
> > > Caused by: java.lang.UnsatisfiedLinkError:
> > > org.apache.hadoop.hbase.shaded.io.netty.channel.epoll.
> > > NativeStaticallyReferencedJniMethods.epollin()I
> > > at
> > > org.apache.hadoop.hbase.shaded.io.netty.channel.epoll.
> > > NativeStaticallyReferencedJniMethods.epollin(Native
> > > Method)
> > > at
> > > org.apache.hadoop.hbase.shaded.io.netty.channel.epoll.
> > > Native.(Native.java:66)
> > > at
> > > org.apache.hadoop.hbase.shaded.io.netty.channel.epoll.
> > > Epoll.(Epoll.java:33)
> > > ... 35 more
> > >
> > > This also appears to come out of netty.
> > > When I run the same test case using 'mvn test -Dtest' command then it
> > wo

Re: Error : java.lang.UnsatisfiedLinkError: failed to load the required native library for netty

[ramkrishna vasudevan]

Hi Stack
I just took an update and tried running some test cases with the eclipse
IDE working on linux

java.io.IOException: Shutting down
at org.apache.hadoop.hbase.MiniHBaseCluster.init(MiniHBaseCluster.java:232)
at org.apache.hadoop.hbase.MiniHBaseCluster.(MiniHBaseCluster.java:94)
at
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniHBaseCluster(HBaseTestingUtility.java:1128)
at
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:1082)
at
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:953)
at
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:935)
at
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:917)
at
org.apache.hadoop.hbase.regionserver.TestRegionReplicasWithModifyTable.before(TestRegionReplicasWithModifyTable.java:65)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)


Caused by: java.lang.UnsatisfiedLinkError: failed to load the required
native library
at
org.apache.hadoop.hbase.shaded.io.netty.channel.epoll.Epoll.ensureAvailability(Epoll.java:78)
at
org.apache.hadoop.hbase.shaded.io.netty.channel.epoll.EpollEventLoopGroup.(EpollEventLoopGroup.java:38)
at
org.apache.hadoop.hbase.util.NettyEventLoopGroupConfig.(NettyEventLoopGroupConfig.java:61)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.(HRegionServer.java:552)
at org.apache.hadoop.hbase.master.HMaster.(HMaster.java:475)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
at
org.apache.hadoop.hbase.util.JVMClusterUtil.createMasterThread(JVMClusterUtil.java:140)
... 26 more
Caused by: java.lang.UnsatisfiedLinkError:
org.apache.hadoop.hbase.shaded.io.netty.channel.epoll.NativeStaticallyReferencedJniMethods.epollin()I
at
org.apache.hadoop.hbase.shaded.io.netty.channel.epoll.NativeStaticallyReferencedJniMethods.epollin(Native
Method)
at
org.apache.hadoop.hbase.shaded.io.netty.channel.epoll.Native.(Native.java:66)
at
org.apache.hadoop.hbase.shaded.io.netty.channel.epoll.Epoll.(Epoll.java:33)
... 35 more

This also appears to come out of netty.
When I run the same test case using 'mvn test -Dtest' command then it works
fine.

Regards
Ram

On Wed, Oct 4, 2017 at 12:42 AM, Stack  wrote:

> It just came in... Looks better on jenkins than my linux machine (4m
> total):
> https://issues.apache.org/jira/browse/HBASE-18606?
> focusedCommentId=16190154=com.atlassian.jira.
> plugin.system.issuetabpanels:comment-tabpanel#comment-16190154
>
> Let me try pushing it.
>
> St.Ack
>
>
>
> On Tue, Oct 3, 2017 at 12:00 PM, Stack  wrote:
>
> > Yeah (12x the processors, 8x RAM, 12x the disks...). Lets see how patch
> > does one jenkins... (HBASE-18606).
> > S
> >
> > On Tue, Oct 3, 2017 at 11:48 AM, Sean Busbey  wrote:
> >
> >> the linux box roughly as capable as your laptop?
> >>
> >> On Tue, Oct 3, 2017 at 12:51 PM, Stack  wrote:
> >> > Tests passed eventually for me:
> >> >
> >> > real 47m6.498s
> >> > user 5m29.671s
> >> > sys 0m41.885s
> >> >
> >> > ... which is a big diff from macosx run.
> >> >
> >> > Need to look into this.
> >> >
> >> > St.Ack
> >> >
> >> >
> >> > On Tue, Oct 3, 2017 at 10:03 AM, Stack  wrote:
> >> >
> >> >> The below gets us further but now I see that the spark tests take a
> >> really
> >> >> long time to run on linux but complete promptly on macosx (2m 55s).
> >> >> Looking
> >> >>
> >> >> St.Ack
> >> >>
> >> >> On Tue, Oct 3, 2017 at 9:13 AM, Stack  wrote:
> >> >>
> >> >>> This seems to work for me. Does it work for you?
> >> >>>
> >> >>>
> >> >>> diff --git a/hbase-spark/pom.xml b/hbase-spark/pom.xml
> >> >>> index 594aa2a..6d191e3 100644
> >> >>> --- a/hbase-spark/pom.xml
> >> >>> +++ b/hbase-spark/pom.xml
> >> >>> @@ -568,6 +568,9 @@
> >> >>>.
> >> >>>WDF TestSuite.txt
> >> >>>false
> >> >>> +  
> >> >>> + >> d.io.netty.packagePrefix>org.
> >> >>> apache.hadoop.hbase.shaded. >> >>> shaded.io.netty.packagePrefix>
> >> >>> +  
> >> >>>  
> >> >>>  
> >> >>>
> >> >>>
> >> >>> St.Ack
> >> >>>
> >> >>> On Tue, Oct 3, 2017 at 8:45 AM, Amit Kabra  >
> >> >>> wrote:
> >> >>>
> >>  Thanks Stack / Sean Busbey for replying.
> >> 
> >>  OS : Ubuntu 16.04.2 , 64 bit.
> >>  Eclipse : Version: Neon.3 Release (4.6.3)
> >>  HBase branch : branch-2
> >>  Command line test to 

Re: Error : java.lang.UnsatisfiedLinkError: failed to load the required native library for netty

[ramkrishna vasudevan]

I too got the same error when I tried running one of the test case in the
latest trunk from the eclipse IDE. Thanks Stack for the update. Let me also
try it out.

Regards
Ram

On Tue, Oct 3, 2017 at 9:43 PM, Stack  wrote:

> This seems to work for me. Does it work for you?
>
>
> diff --git a/hbase-spark/pom.xml b/hbase-spark/pom.xml
> index 594aa2a..6d191e3 100644
> --- a/hbase-spark/pom.xml
> +++ b/hbase-spark/pom.xml
> @@ -568,6 +568,9 @@
>.
>WDF TestSuite.txt
>false
> +  
> +
>  
> org.apache.hadoop.hbase.shaded. packagePrefix>
> +  
>  
>  
>
>
> St.Ack
>
> On Tue, Oct 3, 2017 at 8:45 AM, Amit Kabra 
> wrote:
>
> > Thanks Stack / Sean Busbey for replying.
> >
> > OS : Ubuntu 16.04.2 , 64 bit.
> > Eclipse : Version: Neon.3 Release (4.6.3)
> > HBase branch : branch-2
> > Command line test to reproduce : mvn clean package
> > -Dtest=TestIncrementalBackup
> > Reproduce from eclipse , right click on TestIncBackupRestore and run as
> > junit from test class TestIncrementalBackup.
> > No I am not embedding hbase in my application. I have just checked out
> > hbase , switched to branch-2 and run the unit test from command line or
> > from eclipse. Failing with same error in both cases.
> > Yes the trailing period is also present.
> >
> > Thanks,
> > Amit Kabra.
> >
> >
> >
> >
> >
> > On Tue, Oct 3, 2017 at 8:53 PM, Stack  wrote:
> >
> > > Thank you for the detail.
> > >
> > > Pardon the questions below asking for yet more detail. I am unable to
> > > reproduce locally or on another os (though we see this issue up on our
> > > build box).
> > >
> > > What is your OS when you see the below?
> > >
> > > On Tue, Oct 3, 2017 at 2:06 AM, Amit Kabra 
> > > wrote:
> > >
> > > > Hello,
> > > >
> > > > I am using "branch-2" branch of hbase, when I run unit test I get
> > > following
> > > > error for netty "java.lang.UnsatisfiedLinkError: failed to load the
> > > > required native library"
> > > >
> > > >
> > > > This is running a unit test in your eclipse environment?
> > >
> > > You are trying to run an hbase-spark unit test when you see the above?
> > >
> > >
> > >
> > >
> > > > *I already have following set in "maven-surefire-plugin" in pom.xml
> as
> > > > per http://hbase.apache.org/book.html#thirdparty
> > > > *
> > > >
> > > >
> > > >
> > >
> > > Are you embedding hbase into your application?
> > >
> > >
> > >
> > > > 
> > > > 
> > > >   
> > > >shaded.io.netty.packagePrefix>
> > > > org.apache.hadoop.hbase.shaded. > > hbase.shaded.io.netty.
> > > > packagePrefix>
> > > > 
> > > >
> > > >
> > > >
> > > > *And I see in the code as per HBASE-18271, all io.netty is already
> > > replaced
> > > > with org.apache.hadoop.hbase.shaded.io.netty*
> > > >
> > > >
> > > The trailing period is also present?
> > >
> > >
> > >
> > > >
> > > > If I run a test from eclipse , I see the error immediately and my
> test
> > > > doesn't run, but when I run from command line , the test runs but I
> get
> > > the
> > > > error at the end when the mvn command finishes.
> > > >
> > > >
> > > > Is it any eclipse test?
> > >
> > > Thank you. Let me try and fix this this morning.
> > >
> > > S
> > >
> > >
> > >
> > >
> > >
> > > > *Here is the complete error output.*
> > > >
> > > >
> > > >
> > > > [INFO]
> > > > [INFO] --- maven-surefire-plugin:2.19.1:test (default-test) @
> > > hbase-spark
> > > > ---
> > > > [INFO]
> > > > [INFO] --- scalatest-maven-plugin:1.0:test (test) @ hbase-spark ---
> > > > Discovery starting.
> > > > Discovery completed in 1 second, 558 milliseconds.
> > > > Run starting. Expected test count is: 79
> > > > HBaseDStreamFunctionsSuite:
> > > > Formatting using clusterid: testClusterID
> > > > *** RUN ABORTED ***
> > > >   java.io.IOException: Shutting down
> > > >   at org.apache.hadoop.hbase.MiniHBaseCluster.init(
> > > > MiniHBaseCluster.java:232)
> > > >   at org.apache.hadoop.hbase.MiniHBaseCluster.(
> > > > MiniHBaseCluster.java:94)
> > > >   at org.apache.hadoop.hbase.HBaseTestingUtility.
> > startMiniHBaseCluster(
> > > > HBaseTestingUtility.java:1124)
> > > >   at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(
> > > > HBaseTestingUtility.java:1078)
> > > >   at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(
> > > > HBaseTestingUtility.java:949)
> > > >   at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(
> > > > HBaseTestingUtility.java:943)
> > > >   at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(
> > > > HBaseTestingUtility.java:872)
> > > >   at org.apache.hadoop.hbase.spark.HBaseDStreamFunctionsSuite.
> > beforeAll(
> > > > HBaseDStreamFunctionsSuite.scala:41)
> > > >   at org.scalatest.BeforeAndAfterAll$class.
> > beforeAll(BeforeAndAfterAll.
> > > > scala:187)
> > 

Re: Welcome Chia-Ping Tsai to the HBase PMC

[ramkrishna vasudevan]

Congratulations !!!

On Sat, Sep 30, 2017 at 12:19 PM, Anastasia Braginsky <
anas...@oath.com.invalid> wrote:

> Congratulations!
>
> On Saturday, September 30, 2017, 4:52:21 AM GMT+3, Mike Drob <
> md...@apache.org> wrote:
>
>  Well deserved, Chia-Ping!
>
> On Fri, Sep 29, 2017 at 6:04 PM, Esteban Gutierrez 
> wrote:
>
> > Congrats  Chia-Ping! and Welcome!
> >
> > --
> > Cloudera, Inc.
> >
> >
> > On Fri, Sep 29, 2017 at 3:52 PM, Guanghao Zhang 
> > wrote:
> >
> > > Congratulations!
> > >
> > > 2017-09-30 6:38 GMT+08:00 Andrew Purtell :
> > >
> > > > Congratulations, Chia-Ping! Welcome to the PMC.
> > > >
> > > > On Fri, Sep 29, 2017 at 3:19 PM, Misty Stanley-Jones <
> mi...@apache.org
> > >
> > > > wrote:
> > > >
> > > > > The HBase PMC is delighted to announce that Chia-Ping Tsai has
> agreed
> > > to
> > > > > join
> > > > > the HBase PMC, and help to make the project run smoothly. Chia-Ping
> > > > became
> > > > > an
> > > > > HBase committer over 6 months ago, based on long-running
> participate
> > in
> > > > the
> > > > > HBase project, a consistent record of resolving HBase issues, and
> > > > > contributions
> > > > > to testing and performance.
> > > > >
> > > > > Thank you for stepping up to serve, Chia-Ping!
> > > > >
> > > > > As a reminder, if anyone would like to nominate another person as a
> > > > > committer or PMC member, even if you are not currently a committer
> or
> > > PMC
> > > > > member, you can always drop a note to priv...@hbase.apache.org to
> > let
> > > us
> > > > > know!
> > > > >
> > > > > Thanks,
> > > > > Misty (on behalf of the HBase PMC)
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Best regards,
> > > > Andrew
> > > >
> > > > Words like orphans lost among the crosstalk, meaning torn from
> truth's
> > > > decrepit hands
> > > >- A23, Crosstalk
> > > >
> > >
> >
>

Re: [DISCUSS] Move Type out of KeyValue

[ramkrishna vasudevan]

Even if we are trying to move out I think only few of the types are really
user readable. So we should be very careful here. So since we have
CellBuilder way it is better we check what type of cells a user can build.
I think for now the Cellbuilder is not client exposed?
But again moving to Cell means it becomes public which is not right IMO and
I thinks others here also agree to it.

Regards
Ram

On Fri, Sep 29, 2017 at 10:50 AM, Chia-Ping Tsai 
wrote:

> Thanks for all comment.
>
> The problem i want to resolve is the valid code should be exposed as
> IA.Public. Otherwise, end user have to access the IA.Private class to build
> the custom cell.
>
> For example, I have a use case which plays a streaming role in our
> appliaction. It
> applies the CellBuilder(HBASE-18519) to build custom cells. These cells
> have many same fields so they are put in shared-memory for avoiding GC
> pause. Everything is wonderful. However, we have to access the IA.Private
> class - KeyValue#Type - to get the valid code of Put.
>
> I believe there are many use cases of custom cell, and consequently it is
> worth adding a way to get the valid type via IA.Public class. Otherwise, it
> may imply that the custom cell is based on a unstable way, because the
> related code can be changed at any time.
> --
> Chia-Ping
>
> On 2017-09-29 00:49, Andrew Purtell  wrote:
> > I agree with Stack. Was typing up a reply to Anoop but let me move it
> down
> > here.
> >
> > The type code exposes some low level details of how our current stores
> are
> > architected. But what if in the future you could swap out HStore
> implements
> > Store with PStore implements Store, where HStore is backed by HFiles and
> > PStore is backed by Parquet? Just as a hypothetical example. I know there
> > would be larger issues if this were actually attempted. Bear with me. You
> > can imagine some different new Store implementation that has some
> > advantages but is not a design derived from the log structured merge tree
> > if you like. Most values from a new Cell.Type based on KeyValue.Type
> > wouldn't apply to cells from such a thing because they are particular to
> > how LSMs work. I'm sure such a project if attempted would make a number
> of
> > changes requiring a major version increment and low level details could
> be
> > unwound from Cell then, but if we could avoid doing it in the first
> place,
> > I think it would better for maintainability.
> >
> >
> > On Thu, Sep 28, 2017 at 9:39 AM, Stack  wrote:
> >
> > > On Thu, Sep 28, 2017 at 2:25 AM, Chia-Ping Tsai 
> > > wrote:
> > >
> > > > hi folks,
> > > >
> > > > User is allowed to create custom cell but the valid code of type -
> > > > KeyValue#Type - is declared as IA.Private. As i see it, we should
> expose
> > > > KeyValue#Type as Public Client. Three possible ways are shown below:
> > > > 1) Change declaration of KeyValue#Type from IA.Private to IA.Public
> > > > 2) Move KeyValue#Type into Cell.
> > > > 3) Move KeyValue#Type to upper level
> > > >
> > > > Any suggestions?
> > > >
> > > >
> > > What is the problem that we are trying to solve Chia-Ping? You want to
> make
> > > Cells of a new Type?
> > >
> > > My first reaction is that KV#Type is particular to the KV
> implementation.
> > > Any new Cell implementation should not have to adopt the KeyValue
> typing
> > > mechanism.
> > >
> > > S
> > >
> > >
> > >
> > >
> > > > --
> > > > Chia-Ping
> > > >
> > > >
> > >
> >
> >
> >
> > --
> > Best regards,
> > Andrew
> >
> > Words like orphans lost among the crosstalk, meaning torn from truth's
> > decrepit hands
> >- A23, Crosstalk
> >
>

Re: Please congratulate our new PMC Chair Misty Stanley-Jones

[ramkrishna vasudevan]

Congratulations Misty !!!

On Fri, Sep 22, 2017 at 11:06 AM, Huaxiang Sun  wrote:

> Congratulations Misty!
>
> Huaxiang
>
> > On Sep 21, 2017, at 9:51 PM, Yu Li  wrote:
> >
> > Congrats Misty!
> >
> > Best Regards,
> > Yu
> >
> > On 22 September 2017 at 11:26, Guanghao Zhang 
> wrote:
> >
> >> Congratulations!
> >>
> >> 2017-09-22 11:20 GMT+08:00 Chia-Ping Tsai :
> >>
> >>> congrats Misty!!!
> >>>
> >>> On 2017-09-22 03:08, Andrew Purtell  wrote:
>  At today's meeting of the Board, Special Resolution B changing the
> >> HBase
>  project Chair to Misty Stanley-Jones was passed unanimously.
> 
>  Please join me in congratulating Misty on her new role!
> 
>  ​(If you need any help or advice please don't hesitate to ping me,
> >> Misty,
>  but I suspect you'll do just fine and won't need it.)​
> 
> 
>  --
>  Best regards,
>  Andrew
> 
> >>>
> >>
>
>

Re: [DISCUSS] Increase stability on o.a.h.h.Tag?

[ramkrishna vasudevan]

CellUtil  similar type of methods. Coming to Tags yes there are not much
cases where clients can directly set Tags. And I think we don't expose any
APIs which allow you to use mutations with Tags. So probably moving to
LimitedPrivate is better and mark with Evolving if there are some users
depending on the internals of Tags and its impl. But this will be a One of
case.

And also since Tags are internal ideally the CellUtil#getTAgs() should have
been in another Util method that is exposed with LimitedPrivate and also
Tags if tags should be made LimitedPRivate. So this may help in not having
a PRivate interface like Tag in a public CellUtil class.

3.0 is fine but need some clean up in 2.0? Indicating what could happen
going forward from 2.0?

Regards
Ram



On Fri, Sep 22, 2017 at 2:59 AM, Sean Busbey  wrote:

> Yeah. I mean, I think we should improve  the situation. Just think
> it's too much to bite off at this stage of 2.0, we can aim for 3.0 and
> start working in some tooling to help us.
>
> On Thu, Sep 21, 2017 at 3:35 PM, Josh Elser  wrote:
> > That really makes me groan (we have downstream users depending on code
> we've
> > explicitly said "don't use"), but if that's what it is given the current
> > state, so be it. My complaining won't fix it.
> >
> > Thanks.
> >
> >
> > On 9/21/17 4:25 PM, Sean Busbey wrote:
> >>
> >> We have lots of examples of including non-Public stuff in Public APIs.
> >> we have docs that advise folks to be wary on relying on them beyond
> >> opaque symbols.
> >>
> >> ref: http://hbase.apache.org/book.html#hbase.client.api.surface
> >>
> >> On Thu, Sep 21, 2017 at 3:21 PM, Josh Elser  wrote:
> >>>
> >>> I was going to suggest LimitedPrivate in my original, but this doesn't
> >>> make
> >>> sense as we're exposing Public API via CellUtil.
> >>>
> >>> It seems odd to me that we wouldn't treat the cell tags as a supported
> >>> API
> >>> call. However, I'm happy to remain "confused" if the rest of folks
> don't
> >>> consider tags to be intended for users :)
> >>>
> >>>
> >>> On 9/21/17 3:15 PM, Ted Yu wrote:
> 
> 
>  Can we mark Tag LimitedPrivate ?
> 
>  We know how ATS uses Tags so it should be straight forward to keep
> their
>  usage intact.
> 
>  On Thu, Sep 21, 2017 at 12:03 PM, Josh Elser 
> wrote:
> 
> > Hiya,
> >
> > (Background, I'm starting what is likely to be an onerous task of
> > looking
> > through downstream components and seeing what is broken with the
> latest
> > hbase-2.0.0*)
> >
> > Looking at YARN's use of HBase for the Application TimelineServer, I
> > see
> > that they're relying on the Tag interface.
> >
> > Presently, Tag is marked as Private, yet we expose it via the Public
> > CellUtil.
> >
> > My gut reaction is that we should bump Tag up Public since the intent
> > is
> > for downstream users to, ya know, use those Tags. Any objections?
> >
> > If we don't want to expose Tag, we should make a pass over the Public
> > methods and mark them as Private (so not as to provide a Public
> method
> > with
> > Private objects). CellUtil#getTag(Cell, byte) would be one such
> > example.
> >
> > - Josh
> >
> 
> >>>
> >
>

Re: [VOTE] First hbase-2.0.0-alpha-2 Release Candidate is available

[ramkrishna vasudevan]

+1
Untared the tar ball.

Loaded some 100G data with PE .Did full table scan. Then ran randomReads
with 500 threads with and without block cache.
All seems to be fine.

Regards
Ram

On Sat, Aug 19, 2017 at 2:33 AM, Stack  wrote:

> On Fri, Aug 18, 2017 at 1:45 PM, Josh Elser  wrote:
>
> > +1 (binding)
> >
> > * Saw the same unit test failures as Andrew did (obviously)
> > * Ran a brief randomWrite PE locally and didn't have any obvious issues
> >
> > I took a glance at the contents of the lib/ dir for the bin tarball, and
> > noticed that we seem to be duplicating a few dependencies (e.g.
> jetty-util,
> > javax.servlet*, jersey*, aopalliance*). It probably wouldn't be a bad
> idea
> > to proactively look at these and compare to a 1.x release (for licensing
> > purposes)
> >
> >
> Thanks for testing. The REST one is a little involved but will be fixed by
> alpha3 (with some sweet help from Samir).
>
> Good idea on review of lib-dir: HBASE-18630
>
> Thanks Josh,
> St.Ack
>
>
>
> > On 8/16/17 4:43 PM, Stack wrote:
> >
> >> The first release candidate for HBase 2.0.0-alpha-2 is up at:
> >>
> >>https://dist.apache.org/repos/dist/dev/hbase/hbase-2.0.0-alpha-2RC0/
> >>
> >> Maven artifacts are available from a staging directory here:
> >>
> >>https://repository.apache.org/content/repositories/
> orgapachehbase-1173
> >>
> >> All was signed with my key at y 8ACC93D2
> >> 
> >>
> >>
> >> I tagged the RC as 2.0.0-alpha-2RC0
> >> (7149f999786b6fd5a3fc1f7aec1214afb738925e)
> >>
> >> hbase-2.0.0-alpha-2 includes all that was in alpha-1 (new assignment
> >> manager, offheap read/write path, in-memory compactions, etc.) plus a
> >> return to branch-1 deploy mode where master carries no regions (branch-2
> >> and master up to this had master carrying system tables), a purge of all
> >> checked-in generated content, updates to all dependencies including a
> move
> >> to rely on new hbase-thirdparty lib for relocated versions of guava,
> >> netty,
> >> and protobuf, etc.
> >>
> >> This is a rough cut ('alpha'), not-for-production preview of what
> >> hbase-2.0.0 will look like meant for devs and downstreamers to test
> drive
> >> and flag us early if we messed up anything ahead of our rolling GAs.
> >>
> >> The list of features addressed in 2.0.0 so far can be found here [4].
> >> There
> >> are about 2700+. The list of ~500 fixes in 2.0.0 exclusively can be
> found
> >> here [3].
> >>
> >> I've updated our overview doc. on the state of 2.0.0 [1] but JIRA 2.0.0
> >> label [2] has undergone extensive weeding and presents a fairly good
> >> picture on what is yet to do before we 2.0.0. Check it out.
> >>
> >> Please take it for a spin and vote on whether it ok to put out as our
> >> first
> >> alpha (bar is low for an 'alpha'). Let the VOTE be open for 72 hours
> >> (Saturday)
> >>
> >> Thanks,
> >> St.Ack
> >>
> >> 1.
> >> https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9i
> >> Eu_ktczrlKHK8N4SZzs/edit#
> >> 2. https://issues.apache.org/jira/projects/HBASE/versions/12327188
> >> 3.
> >> https://issues.apache.org/jira/browse/HBASE-18611?jql=projec
> >> t%20%3D%20HBASE%20%20and%20(fixVersion%20%3D%202.0.0%20%7C%
> >> 7C%20fixVersion%20%3D%202.0.0.alpha1%20%7C%7C%20fixVersion%
> >> 20%3D%202.0.0-alpha-2)%20and%20fixVersion%20not%20in%20(1.
> >> 0.0%2C%201.0.1%2C%201.0.2%2C%201.0.3%2C%201.0.4%2C%201.0.5%
> >> 2C%201.0.6%2C%201.1.0%2C%201.1.1%2C%201.1.2%2C%201.1.3%2C%
> >> 201.1.4%2C%201.1.5%2C%201.1.6%2C%201.1.7%2C%201.1.8%2C%201.
> >> 1.9%2C%201.1.10%2C%201.2.0%2C%201.2.1%2C%201.2.2%2C%201.2.3%
> >> 2C%201.2.4%2C%201.2.5%2C%201.2.6%2C%201.3.0%2C%201.3.1%2C%
> >> 201.4.0)%20and%20%20(status%20%3D%20Open%20or%20status%20%
> >> 3D%20%22Patch%20Available%22)
> >> 4.
> >> https://issues.apache.org/jira/browse/HBASE-18599?jql=projec
> >> t%20%3D%20HBASE%20%20and%20(%20fixVersion%20%3D%202.0.0%20%7
> >> C%7C%20fixVersion%20%3D%202.0.0-alpha-1%20%7C%7C%20fixVersio
> >> n%20%3D%202.0.0-alpha-2)%20and%20(status%20%3D%20Resolved)
> >>
> >>
>

Re: Notes from dev meetup in Shenzhen, August 5th, 2017

[ramkrishna vasudevan]

Thanks for the write up Stack. I could not make it to Shenzhen. Nice to
know the conference and meet up went great.

Regards
Ram

On Mon, Aug 7, 2017 at 9:36 PM, Stack  wrote:

> At fancy Huawei headquarters, 10:00-12:00AM or so (with nice coffee and
> fancy little cake squares provided about half way through the session).
>
> For list of attendees, see picture at end of this email.
>
> Discussion was mostly in Chinese with about 25% in English plus some
> gracious sideline translation so the below is patchy. Hopefully you get the
> gist.
>
> For client-side scanner going against hfiles directly; is there a means of
> being able to pass the permissions from hbase to hdfs?
>
> Issues w/ the hbase 99th percentile were brought up. "DynamoDB can do
> 10ms". How to do better?
>
> SSD is not enough.
>
> GC messes us up.
>
> Will the Distributed Log Replay come back to help improve MTTR? We could
> redo on new ProcedureV2 basis. ZK timeout is the biggest issue. Do as we
> used to and just rely on the regionserver heartbeating...
>
> Read replica helps w/ MTTR.
>
> Ratis incubator project to do a quorum based hbase?
>
> Digression on licensing issues around fb wangle and folly.
>
> Redo of hbase but quorum based would be another project altogether.
>
> Decided to go around the table to talk about concerns and what people are
> working on.
>
> Jieshan wondered what could be done to improve OLAP over hbase.
>
> Client side scanner was brought up again as means of skipping RS overhead
> and doing better OLAP.
>
> Have HBase compact to parquet files. Query parquet and hbase.
>
> At Huawei, they are using 1.0 hbase. Most problems are assignment. They
> have .5M regions. RIT is a killer. Double assignment issues. And RIT. They
> run their own services. Suggested they upgrade to get fixes at least. Then
> 2.0.
>
> Will HBase federate like HDFS? Can Master handle load at large scale? It
> needs to do federation too?
>
> Anyone using Bulk loaded replication? (Yes, it just works so no one talks
> about it...)
>
> Request that fixes be backported to all active branches, not just most
> current.
>
> Andrew was good at backporting... not all RMs are.
>
> Too many branches. What should we do?
>
> Proliferation of branches makes for too much work.
>
> Need to cleanup bugs in 1.3. Make it stable release now.
>
> Lets do more active EOL'ing of branches. 1.1?.
>
> Hubert asked if we can have clusters where RS are differently capable?
> i.e. several generations of HW all running in the same cluster.
>
> What if fat server goes down.
>
> Balancer could take of it all. RS Capacity. Balancer can take it into
> account.
> Regionserver labels like YARN labels. Characteristics.
>
> Or run it all in docker when heterogeneous cluster. The K8 talk from day
> before was mentioned; we should all look at being able to deploy in k8 and
> docker.
>
> Lets put out kubernetes blog...(Doing).
>
> Alibaba looking at HBase as native YARN app.
>
> i/o is hard even when containers.
>
> Use autoscaler of K8 when heavy user.
>
> Limit i/o use w/ CP. Throttle.
>
> Spark and client-side scanner came up again.
>
> Snapshot input format in spark.
>
> HBase federation came up again. jd.com talking of 3k to 4k nodes in a
> cluster. Millions of regions. Region assignment is messing them up.
>
> Maybe federation is good idea? Argument that it is too much operational
> conplexity. Can we fix master load w/ splittable meta, etc?
>
> Was brought up that even w/ 100s of RS there is scale issue, nvm thousands.
>
> Alibaba talked about disaster recovery. Described issue where HDFS has
> fencing problem during an upgrade. There was no active NN. All RS went down.
> ZK is another POF. If ZK is not available. Operators were being asked how
> much longer the cluster was going to be down but they could not answer the
> question. No indicators from HBase on how much longer it will be down or
> how many WALs its processed and how many more to go. Operator unable to
> tell his org how long it would be before it all came back on line. Should
> say how many regions are online and how many more to do.
>
> Alibaba use SQL to lower cost. HBase API is low-level. Row-key
> construction is tricky. New users make common mistakes. If you don't do
> schema right, high-performance is difficult.
>
> Alibaba are using a subset of Phoenix... simple sql only; throws
> exceptions if user tries to do joins, etc.., anything but basic ops.
>
> HareQL is using hive for meta store.  Don't have data typing in hbase.
>
> HareQL could perhaps contribute some piece... or a module in hbase to
> sql... From phoenix?
>
> Secondary index.
>
> Client is complicated in phoenix. Was suggested thin client just does
> parse... and then offload to server for optimization and execution.
>
> Then secondary index. Need transaction engine. Consistency of secondary
> index.
>
> We adjourned.
>
> Your dodgy secretary,
> St.Ack
> P.S. Please add to this base set of notes if I missed anything.
>
>
>
>

Re: Tags class using wrong length?

[ramkrishna vasudevan]

I think the layout of tags is missing now in the javadoc. May be it got
missed or moved to some other place?
I remember we had a layout explaining the tag structure then this code is
much easier to read this code.

As Chia-Ping said  is the
layout.
So from the KeyValue lay out we extract the tag part which in itself has a
tag length to represent the complete set of tags.

>From the tags offset and tags length from the KV we extract individual tags
in that KV.

For eg
See TagUtil#asList

{code}
 List tags = new ArrayList<>();
int pos = offset;
while (pos < offset + length) {
  int tagLen = Bytes.readAsInt(b, pos, TAG_LENGTH_SIZE);
  tags.add(new ArrayBackedTag(b, pos, tagLen + TAG_LENGTH_SIZE));
  pos += TAG_LENGTH_SIZE + tagLen;
}
return tags;
{code}

Regards
Ram




On Mon, Aug 7, 2017 at 3:25 AM, Ted Yu  wrote:

> The byte following the tag length (a short) is the tag type.
>
> The current code is correct.
>
> On Sun, Aug 6, 2017 at 5:40 AM, Chia-Ping Tsai 
> wrote:
>
> > According to the following code:
> >   public ArrayBackedTag(byte tagType, byte[] tag) {
> > int tagLength = tag.length + TYPE_LENGTH_SIZE;
> > if (tagLength > MAX_TAG_LENGTH) {
> >   throw new IllegalArgumentException(
> >   "Invalid tag data being passed. Its length can not exceed " +
> > MAX_TAG_LENGTH);
> > }
> > length = TAG_LENGTH_SIZE + tagLength;
> > bytes = new byte[length];
> > int pos = Bytes.putAsShort(bytes, 0, tagLength);
> > pos = Bytes.putByte(bytes, pos, tagType);
> > Bytes.putBytes(bytes, pos, tag, 0, tag.length);
> > this.type = tagType;
> >   }
> > The layout of the byte array should be:
> > |tag legnth (2 bytes)|tag type(1 byte)|tag|
> >
> > It seems to me that the "bytes[offset + TYPE_LENGTH_SIZE]" is correct.
> >
> > On 2017-08-06 16:35, Lars George  wrote:
> > > Hi,
> > >
> > > I found this reading through tags in 1.3, but checked in trunk as
> > > well. There is this code:
> > >
> > >   public ArrayBackedTag(byte[] bytes, int offset, int length) {
> > > if (length > MAX_TAG_LENGTH) {
> > >   throw new IllegalArgumentException(
> > >   "Invalid tag data being passed. Its length can not exceed "
> > > + MAX_TAG_LENGTH);
> > > }
> > > this.bytes = bytes;
> > > this.offset = offset;
> > > this.length = length;
> > > this.type = bytes[offset + TAG_LENGTH_SIZE];
> > >   }
> > >
> > > I am concerned about the last line of the code, using the wrong
> constant?
> > >
> > >   public final static int TYPE_LENGTH_SIZE = Bytes.SIZEOF_BYTE;
> > >   public final static int TAG_LENGTH_SIZE = Bytes.SIZEOF_SHORT;
> > >
> > > Should this not read
> > >
> > > this.type = bytes[offset + TYPE_LENGTH_SIZE];
> > >
> > > Would this not read the type from the wrong place in the array?
> > >
> > > Cheers,
> > > Lars
> > >
> >
>

Re: [ANNOUNCE] New HBase committer Mike Drob

[ramkrishna vasudevan]

Welcome Mike !!!

On Tue, Aug 1, 2017 at 10:14 PM, Chia-Ping Tsai  wrote:

> Welcome.  Mike!
>
> On 2017-08-01 23:38, Josh Elser  wrote:
> > On behalf of the Apache HBase PMC, I'm pleased to announce that Mike
> > Drob has accepted the PMC's invitation to become a committer.
> >
> > Mike has been doing some great things lately in the project and this is
> > a simple way that we can express our thanks. As my boss likes to tell
> > me: the reward for a job well-done is more work to do! We're all looking
> > forward to your continued involvement :)
> >
> > Please join me in congratulating Mike!
> >
> > - Josh
> >
>

Re: [DISCUSS] Should flush decisions be made based on data size (key-value only) or based on heap size (including metadata overhead)?

[ramkrishna vasudevan]

>>I'm not sure whether we should rollback, but if stick on current policy
there should be more documents, metrics (monitoring heap/data occupancy
separately) and log message refinements, etc.

Agree to this.

>>Sounds like we should be doing the former, heap occupancy
Stack, so do you mean we need to roll back this new change in trunk? The
background is https://issues.apache.org/jira/browse/HBASE-16747.

Regards
Ram


On Thu, Jul 6, 2017 at 8:40 AM, Yu Li  wrote:

> We've also observed more blocking updates happening with the new policy
> (flush decision made on data size), but could work-around it by reducing
> the hbase.hregion.memstore.flush.size setting. The advantage of current
> policy is we could control the flushed file size more accurately, but
> meanwhile losing some "compatibility" (requires configuration updating
> during rolling upgrade).
>
> I'm not sure whether we should rollback, but if stick on current policy
> there should be more documents, metrics (monitoring heap/data occupancy
> separately) and log message refinements, etc. Attaching some of the logs we
> observed, which is pretty confusing w/o knowing the details of
> implementation:
>
> 2017-07-03 16:11:54,724 INFO
>  [B.defaultRpcServer.handler=182,queue=11,port=16020]
> regionserver.MemStoreFlusher: Blocking updates on
> hadoop0528.et2.tbsite.net,16020,1497336978160:
> global memstore heapsize 7.2 G is >= than blocking 7.2 G size
> 2017-07-03 16:11:54,754 INFO
>  [B.defaultRpcServer.handler=186,queue=15,port=16020]
> regionserver.MemStoreFlusher: Blocking updates on
> hadoop0528.et2.tbsite.net,16020,1497336978160:
> global memstore heapsize 7.2 G is >= than blocking 7.2 G size
> 2017-07-03 16:11:57,571 INFO  [MemStoreFlusher.0]
> regionserver.MemStoreFlusher: Flush of region
> mainv7_main_result_c,1496,1499062935573.02adfa7cbdc606dce5b79a516e16492a.
> due to global heap pressure. Total Memstore size=3.2 G, Region memstore
> size=331.4 M
> 2017-07-03 16:11:57,571 WARN
>  [B.defaultRpcServer.handler=49,queue=11,port=16020]
> regionserver.MemStoreFlusher: Memstore is above high water mark and block
> 2892ms
>
> Best Regards,
> Yu
>
> On 6 July 2017 at 00:56, Stack  wrote:
>
> > On Wed, Jul 5, 2017 at 6:30 AM, Eshcar Hillel
>  > >
> > wrote:
> >
> > > Hi All,
> > > I opened a new Jira https://issues.apache.org/jira/browse/HBASE-18294
> to
> > > discuss this question.
> > > Flush decisions are taken at the region level and also at the region
> > > server level - there is the question of when to trigger a flush and
> then
> > > which region/store to flush.Regions track both their data size
> (key-value
> > > size only) and their total heap occupancy (including index and
> additional
> > > metadata).One option (which was the past policy) is to trigger flushes
> > and
> > > choose flush subjects based on regions heap size - this gives a better
> > > estimation for sysadmin of how many regions can a RS carry.Another
> option
> > > (which is the current policy) is to look at the data size - this gives
> a
> > > better estimation of the size of the files that are created by the
> flush.
> > >
> >
> >
> > Sounds like we should be doing the former, heap occupancy. An
> > OutOfMemoryException puts a nail in any benefit other accountings might
> > have.
> >
> > St.Ack
> >
> >
> >
> > > I see this is as critical to HBase performance and usability, namely
> > > meeting the user expectation from the system, hence I would like to
> hear
> > as
> > > many voices as possible.Please join the discussion in the Jira and let
> us
> > > know what you think.
> > > Thanks,Eshcar
> > >
> > >
> >
>

Re: [ANNOUNCE] Devaraj Das joins the Apache HBase PMC

[ramkrishna vasudevan]

Congratulations, Devaraj !!!

On Wed, Jul 5, 2017 at 10:13 PM, Andrew Purtell  wrote:

> Congratulations, Devaraj!
>
> On Wed, Jul 5, 2017 at 9:27 AM, Josh Elser  wrote:
>
> > I'm pleased to announce yet another PMC addition in the form of Devaraj
> > Das. One of the "old guard" in the broader Hadoop umbrella, he's also a
> > long-standing member in our community. We all look forward to the
> continued
> > contributions and project leadership.
> >
> > Please join me in welcoming Devaraj!
> >
> > - Josh (on behalf of the PMC)
> >
>
>
>
> --
> Best regards,
> Andrew
>
> Words like orphans lost among the crosstalk, meaning torn from truth's
> decrepit hands
>- A23, Crosstalk
>

Re: [ANNOUNCE] Chunhui Shen joins the Apache HBase PMC

[ramkrishna vasudevan]

Congratulations !!!

On Wed, Jul 5, 2017 at 10:33 AM, QI Congyun  wrote:

> Congratulations, Chunhui.
>
> -Original Message-
> From: Jerry He [mailto:jerry...@gmail.com]
> Sent: Wednesday, July 05, 2017 1:02 PM
> To: dev@hbase.apache.org; u...@hbase.apache.org
> Subject: Re: [ANNOUNCE] Chunhui Shen joins the Apache HBase PMC
>
> Congrats,  Chunhui!
>
> Thanks
>
> Jerry
>
> On Tue, Jul 4, 2017 at 8:37 PM Anoop John  wrote:
>
> > Congrats Chunhui..
> >
> > On Wed, Jul 5, 2017 at 6:55 AM, Pankaj kr  wrote:
> > > Congratulations Chunhui..!!
> > >
> > > Regards,
> > > Pankaj
> > >
> > >
> > > -Original Message-
> > > From: Yu Li [mailto:car...@gmail.com]
> > > Sent: Tuesday, July 04, 2017 1:24 PM
> > > To: dev@hbase.apache.org; Hbase-User
> > > Subject: [ANNOUNCE] Chunhui Shen joins the Apache HBase PMC
> > >
> > > On behalf of the Apache HBase PMC I am pleased to announce that
> > > Chunhui
> > Shen has accepted our invitation to become a PMC member on the Apache
> > HBase project. He has been an active contributor to HBase for past many
> years.
> > Looking forward for many more contributions from him.
> > >
> > > Please join me in welcoming Chunhui to the HBase PMC!
> > >
> > > Best Regards,
> > > Yu
> >
>

Re: [ANNOUNCE] Chunhui Shen joins the Apache HBase PMC

[ramkrishna vasudevan]

Congratulations !!

On Tue, Jul 4, 2017 at 11:00 AM, 张铎(Duo Zhang) 
wrote:

> Congratulations!
>
> Yu Li 于2017年7月4日 周二13:24写道：
>
> > On behalf of the Apache HBase PMC I am pleased to announce that Chunhui
> > Shen
> > has accepted our invitation to become a PMC member on the Apache
> > HBase project. He has been an active contributor to HBase for past many
> > years. Looking forward for many more contributions from him.
> >
> > Please join me in welcoming Chunhui to the HBase PMC!
> >
> > Best Regards,
> > Yu
> >
>

Re: [VOTE] hbase-thirdparty project, first release candidate 1.0.0RC0

[ramkrishna vasudevan]

+1.
downloaded the src tar ball.
did 'mvn clean install' - Works fine.

It asks for maven greater than 3.3. But Hbase trunk compiles with maven
3.1.0. So may be unifying both will be better in this case.

Regards
Ram

On Mon, Jul 3, 2017 at 10:03 AM, Stack  wrote:

> On Sun, Jul 2, 2017 at 9:17 PM, Yu Li  wrote:
>
> > +1
> >
> > Checked sums and signatures: ok
> > 'mvn install' succeeds: ok
> > Checked generated jar files: ok
> >
> > Note: it requires maven version > 3.3.3 to compile
> >
> >
> Yeah. I picked an arbitrary version. Older versions would probably work but
> 3.3.3 is old now. Figured it would be ok. I could try w/ a 3.0.x for future
> release if it a pain?
>
>
> > btw, is this only for branch-2 or branch-1 will also follow? Thanks.
> >
> >
> Plan is for hbase2 only currently. No plans to up guava or protobuf
> versions on branch-1. You think we should?
>
> Thanks for voting Yu Li.
>
> St.Ack
>
>
> > Best Regards,
> > Yu
> >
> > On 2 July 2017 at 11:16, 张铎(Duo Zhang)  wrote:
> >
> > > +1 (binding)
> > >
> > > Checked sums and signatures: OK
> > > 'mvn clean install': OK, the size of the generated jar files are
> > reasonable
> > > Unzip the generated jar files: OK, all classes are relocated under
> > > org/apache/hadoop/hbase/shaded
> > >
> > > Only one minor question, seems we do not generate sources jars? There
> is
> > > a hbase-shaded-protobuf-1.0.0-sources.jar after 'mvn install' but the
> > size
> > > is only 22 bytes.
> > >
> > > Thanks.
> > >
> > > 2017-07-02 2:38 GMT+08:00 Stack :
> > >
> > > > +1 from me.
> > > > St.Ack
> > > >
> > > > On Fri, Jun 30, 2017 at 3:07 PM, Stack  wrote:
> > > >
> > > > > This is an interesting vote. The vote is on an RC made out of a new
> > > hbase
> > > > > project, hbase-thirdparty.
> > > > >
> > > > > First the usual stuff.
> > > > >
> > > > > The 1.0.0RC0 is available at:
> > > > >
> > > > >  https://dist.apache.org/repos/dist/dev/hbase/hbase-
> > > thirdparty/1.0.0RC0/
> > > > >
> > > > > Maven artifacts are in the following staging repository:
> > > > >
> > > > >  https://repository.apache.org/content/repositories/
> > > orgapachehbase-1171
> > > > >
> > > > > Artifacts are signed with 8ACC93D2 which is at the tail of our KEYS
> > > file
> > > > > http://www-us.apache.org/dist/hbase/KEYS.
> > > > >
> > > > > I tagged this RC as 1.0.0RC at 8ffaf3dd561052bcc71772148ecd04
> > > cdf9e224f3
> > > > > in the new hbase-thirdparty repository at [1].
> > > > >
> > > > > Now to the unusual.
> > > > >
> > > > > The artifact was made from a new repository, hbase-thirdparty[1].
> > > > > hbase-thirdparty is a project overseen by the hbase PMC. It came of
> > > > > discussion on the dev list in the thread '[DISCUSS] More Shading'
> > [2].
> > > > > Its charge is the hosting of ornery thirdparty libs -- e.g.
> protobuf,
> > > > > guava, netty -- that need patching and/or shading so we are no
> longer
> > > > bound
> > > > > by whatever the version of a lib Hadoop happens to ship (and we can
> > > > change
> > > > > versions w/o damaging out glorious downstreamers). In the past,
> when
> > > > we've
> > > > > been able, we've done this dirty patching and shading work as an
> > > > > unacknowledged 'pre-build' hack step appended to our protobuf file
> > > > > generation. Moving this mess out of our mainline build to live
> > > elsewhere,
> > > > > pulled in as a legit 'dependency', simplifies our build
> particularly
> > > > around
> > > > > the shading of third-part libs; the narrower scope of a dedicated
> > > project
> > > > > that produces transformed jars is easier to reason about, is less
> > > likely
> > > > to
> > > > > clash with existing dependencies whether transitive or otherwise,
> and
> > > it
> > > > > makes for less spaghetti poms.
> > > > >
> > > > > Other (minor) benefits include our being able to generate protobuf
> > > files
> > > > > inline now; no need of the pre-build anymore nor checking in of
> > > generated
> > > > > protobuf files (25MB).  There is a patch to remove them (predicated
> > on
> > > > this
> > > > > hbase-thirdparty release): "HBASE-17056 Remove checked in PB
> > generated
> > > > > files".
> > > > >
> > > > > Downside is having to explicitly reference our version in code as
> in
> > > > > org.apache.hadoop.hbase.shaded.com.google.protobuf.Message rather
> > than
> > > > > just com.google.protobuf.Message and so on for guava references,
> etc.
> > > > > (We've been doing this for a while now so hopefully it not too
> > strange
> > > > > developing)*.
> > > > >
> > > > > There is little to this new project other than a few poms to do
> > > packaging
> > > > > and a couple of patches we apply on build. It works by pulling down
> > > > > dependencies and doing whatever shading, renaming, or patching we
> > need.
> > > > > Currently hbase-thirdparty hosts protobuf (3.1.0 => 3.3.1), guava
> > (12.0
> > > > =>
> > > > > 22.0), gson, 

Fwd: Encryption of exisiting data in Stripe Compaction

[ramkrishna vasudevan]

Hi all

Interesting case with Stripe compactions and Encryptions. Does any one has
any suggestion for Karthick's case? The initial mail was targetted to
issues@ and so forwarding it to dev@ and user@.


Regards
Ram

-- Forwarded message --
From: ramkrishna vasudevan <ramkrishna.s.vasude...@gmail.com>
Date: Tue, Jun 20, 2017 at 4:51 PM
Subject: Re: Encryption of exisiting data in Stripe Compaction
To: Karthick Ram <karthik.28...@gmail.com>


I am not aware of any other mechanism. I just noticed that you had fwded
the message to issues@ and not to dev@ and users@. Let me forward it to
those mailing address. Thanks Karthick.

Regards
Ram

On Mon, Jun 19, 2017 at 1:07 PM, Karthick Ram <karthik.28...@gmail.com>
wrote:

> Hi,
> Yes we are doing exactly the same. We altered the table with
> exploringcompaction and triggered a major compaction. But when it comes to
> key rotation, which we do very often, we have to manually alter the table
> and rollback to previous compaction policy. Currently we have a cron job
> for this. Is there any other way to automate this?
>
> Regards
> Karthick R
>
> On Thu, Jun 15, 2017 at 9:55 AM, ramkrishna vasudevan <
> ramkrishna.s.vasude...@gmail.com> wrote:
>
>> Hi
>> Very interesting case. Ya Stripe compaction does not need to under go a
>> major compaction if it already running under stripe compaction (reading the
>> docs I get this).
>> Since you have enable encryption at a later point of time you face this
>> issue I believe. The naive workaround I can think of is that do a alter
>> table with default compaction and it will do a major compaction and once
>> that is done again move back to Stripe compaction?  Will that work?
>>
>> I would like to hear opinion of others who have experience with Strip
>> compaction.
>>
>> Regards
>> Ram
>>
>> On Wed, Jun 14, 2017 at 10:25 AM, Karthick Ram <karthik.28...@gmail.com>
>> wrote:
>>
>>> We have a table which has time series data with Stripe Compaction
>>> enabled.
>>> After encryption has been enabled for this table the newer entries are
>>> encrypted and inserted. However to encrypt the existing data in the
>>> table,
>>> a major compaction has to run. Since, stripe compaction doesn't allow a
>>> major compaction to run, we are unable to encrypt the previous data.
>>> Please
>>> suggest some ways to rectify this problem.
>>>
>>> Regards,
>>> Karthick R
>>>
>>
>>
>

Re: [ANNOUNCE] New HBase committer Huaxiang Sun

[ramkrishna vasudevan]

Congratulations !!!

On Tue, Jun 20, 2017 at 9:32 AM, Yu Li  wrote:

> Congratulations and welcome, Huaxiang!
>
> Best Regards,
> Yu
>
> On 20 June 2017 at 11:03, Allan Yang  wrote:
>
> > Congratulations and welcome, Huaxiang!
> >
> > Best Regards
> > Allan Yang
> >
> > 2017-06-20 10:32 GMT+08:00 Pankaj kr :
> >
> > > Congratulations Huaxiang..!!
> > >
> > > Thanks & Regards,
> > > Pankaj
> > >
> > > HUAWEI TECHNOLOGIES CO.LTD.
> > > Huawei Tecnologies India Pvt. Ltd.
> > > Near EPIP Industrial Area, Kundalahalli Village
> > > Whitefield, Bangalore-560066
> > > www.huawei.com
> > > 
> > > -
> > > This e-mail and its attachments contain confidential information from
> > > HUAWEI, which
> > > is intended only for the person or entity whose address is listed
> above.
> > > Any use of the
> > > information contained herein in any way (including, but not limited to,
> > > total or partial
> > > disclosure, reproduction, or dissemination) by persons other than the
> > > intended
> > > recipient(s) is prohibited. If you receive this e-mail in error, please
> > > notify the sender by
> > > phone or email immediately and delete it!
> > >
> > >
> > > -Original Message-
> > > From: Sean Busbey [mailto:bus...@apache.org]
> > > Sent: Tuesday, June 20, 2017 3:31 AM
> > > To: dev; u...@hbase.apache.org
> > > Subject: [ANNOUNCE] New HBase committer Huaxiang Sun
> > >
> > > On behalf of the Apache HBase PMC, I am pleased to announce that
> Huaxiang
> > > Sun has accepted the PMC's invitation to become a committer on the
> > project.
> > > We appreciate all of Huaxiang's great work thus far and look forward to
> > > continued involvement.
> > >
> > > Please join me in congratulating Huaxiang!
> > >
> >
>

Re: [ANNOUNCE] New HBase committer Allan Yang

[ramkrishna vasudevan]

Congratulations !!!

On Mon, Jun 19, 2017 at 8:57 AM, 宾莉金（binlijin）  wrote:

> Congratulations, Allan !
>
> 2017-06-09 11:49 GMT+08:00 Yu Li :
>
> > On behalf of the Apache HBase PMC, I am pleased to announce that Allan
> Yang
> > has accepted the PMC's invitation to become a committer on the
> > project. We appreciate all of Allan's generous contributions thus far and
> > look forward to his continued involvement.
> >
> > Congratulations and welcome, Allan!
> >
>
>
>
> --
> *Best Regards,*
>  lijin bin
>

Re: [ANNOUNCE] New HBase committer Allan Yang

[ramkrishna vasudevan]

Congratulations !!!

On Fri, Jun 9, 2017 at 12:19 PM, Chia-Ping Tsai  wrote:

> Welcome !!!
>
> On 2017-06-09 11:49 (+0800), Yu Li  wrote:
> > On behalf of the Apache HBase PMC, I am pleased to announce that Allan
> Yang
> > has accepted the PMC's invitation to become a committer on the
> > project. We appreciate all of Allan's generous contributions thus far and
> > look forward to his continued involvement.
> >
> > Congratulations and welcome, Allan!
> >
>

Re: [VOTE] First release candidate for HBase 1.2.6 is available

[ramkrishna vasudevan]

+1

Downloaded both src and binary artifacts.

Started a single node cluster with the set up created out of binary
artifact. Loaded some data.

Built the package from the src artifact. Killed the Regionserver started
from binary artifact and started a region server created out of src
artifact.

All data and regions were available again.

No obvious issues.

Ran all the tests and there were no test failures too in my linux box.

Regards
Ram

On Tue, May 30, 2017 at 9:49 PM, Misty Stanley-Jones 
wrote:

> +1 binding
>
> Tested on my Macbook Pro:
>
> Source verification:
> - Verified md5sum
> - Extracted binary .tar.gz
> - Scanned CHANGES.txt (nice going fixing a 4-digit JIRA!)
> - Ran mvn install
> - Ran mvn verify, a couple timeouts that ran OK individually
>
> Binary verification:
> - Verified md5sum
> - Extracted .tar.gz
> - Scanned CHANGES.txt, it matches
> - Looked at HTML and PDF docs, version info looks good
> - Ran bin/start-hbase.sh, verified HBase was running
> - Ran bin/stop-hbase.sh, verified HBase stopped
>
>
>
>
>
>
>
> On Mon, May 29, 2017 at 9:31 AM, Sean Busbey  wrote:
>
> > The first release candidate for HBase 1.2.6 is available for download at:
> >
> > https://dist.apache.org/repos/dist/dev/hbase/hbase-1.2.6RC0/
> >
> > Maven artifacts are also available in a staging repository at:
> >
> > https://repository.apache.org/content/repositories/orgapachehbase-1168/
> >
> > Artifacts are signed with my key (0D80DB7C) published in our KEYS
> > file at http://www.apache.org/dist/hbase/KEYS
> >
> > The RC corresponds to the signed tag 1.2.6RC0, which currently points
> > to commit ref
> >
> > 2f9b9e17d0522e36063bf52ecc58576243d20b3f
> >
> > The detailed source and binary compatibility report vs 1.2.5 has been
> > published for your review, at:
> >
> > https://s.apache.org/hbase-1.2.6-rc0-compat-report
> >
> > HBase 1.2.6 is the sixth maintenance release in the HBase 1.2 line,
> > continuing on the theme of bringing a stable, reliable database to
> > the Hadoop and NoSQL communities. This release includes over 25
> > bug fixes since 1.2.5.
> >
> > Critical fixes include:
> >
> > * HBASE-17287 - Master becomes a zombie if filesystem object closes
> > * HBASE-16630 - Fragmentation in long running Bucket Cache
> > * HBASE-17717 - Incorrect ZK ACL set for HBase superuser
> > * HBASE-15941 - HBCK repair should not unsplit healthy splitted region
> >
> > The full list of fixes included in this release is available at:
> >
> > https://s.apache.org/hbase-1.2.6-jira-release-notes
> >
> > and in the CHANGES.txt file included in the distribution.
> >
> > Please try out this candidate and vote +1/-1 on whether we should
> > release these artifacts as HBase 1.2.6.
> >
> > The VOTE will remain open for at least 72 hours. Given sufficient votes
> > I would like to close it on June 5th, 2017.
> >
> > thanks!
> >
>

Re: VOTE: Merge new Assignment Manager AMv2), HBASE-14614

[ramkrishna vasudevan]

I have been following this activity  and I think it is good to be merged.
+1. Any test case failures can be fixed post merge (if any).

Regards
Ram

On Wed, May 31, 2017 at 3:03 AM, Stephen Jiang 
wrote:

> +1 for merge
>
> On Tue, May 30, 2017 at 2:05 PM, Umesh Agashe 
> wrote:
>
> > +1 for merge. I have reviewed most of this work.
> >
> > On Tue, May 30, 2017 at 10:13 AM, Andrew Purtell <
> andrew.purt...@gmail.com
> > >
> > wrote:
> >
> > > +1 for merge after all unit tests are passing. No problem to exclude
> > known
> > > broken tests not working independent of this patch-set.
> > >
> > >
> > > > On May 30, 2017, at 9:09 AM, Stack  wrote:
> > > >
> > > > Moving on from the DISCUSSION thread [4], lets vote on whether to
> merge
> > > the
> > > > new Assignment Manager (AMv2) out on branch HBASE-14614 to master.
> > > >
> > > > Unit tests pass. Tests at scale show that the new AM is at least as
> > good
> > > as
> > > > the old.
> > > >
> > > > AMv2 is a long-standing effort addressing the most common source of
> > > trouble
> > > > operating an hbase cluster at scale, region assignment.
> > > >
> > > > AMv2 is the last hurdle to our being able to branch for hbase2.
> > > >
> > > > Please vote +1 if in favor of merge and -1 if agin.
> > > >
> > > > See below if you need more background on what we're voting on.
> > > >
> > > > St.Ack
> > > >
> > > > The new AssignmentManager on HBASE-14614 branch promises new levels
> of
> > > > scaling, insight, resilience and assignment performance. A skeleton
> dev
> > > > overview is available here [1].
> > > >
> > > > There are outstanding items to complete but they can come in
> > post-merge.
> > > > Critical is reenabling a set of tests disabled because they depend on
> > > > workings no longer present or they depended on a broken behavior in
> > AMv1
> > > > (Disabled tests and the list of TODOs are in [1]).
> > > >
> > > > Currently I am working on polish making it pass the outstanding
> failing
> > > > unit tests, fixing white space, and findbugs but it is ready for
> > commit.
> > > > I've put up an RB patch that is absent protobufs so the bulk is less
> > > > intimidating [3].
> > > >
> > > > This patch is mostly the work of Matteo Bertozzi (inception, bulk of
> > > > implementation). Others have contributed including Stephen Yuan
> Jiang.
> > > > Umesh Agashe and Appy also have contributed fixes that got added to
> the
> > > > HBASE-14614 branch.
> > > >
> > > > 1. https://docs.google.com/document/d/1eVKa7FHdeoJ1-
> > > > 9o8yZcOTAQbv0u0bblBlCCzVSIn69g/edit#
> > > > 2. https://docs.google.com/document/d/1QLXlVERKt5EMbx_
> > > > EL3Y2u0j64FN-_TrVoM5WWxIXh6o/edit#heading=h.df9krsl9k16
> > > > 3. https://reviews.apache.org/r/55117/
> > > > 4.
> > > > http://apache-hbase.679495.n3.nabble.com/DISCUSS-Merge-new-
> > > Assignment-Manager-AMv2-HBASE-14614-td4088089.html
> > >
> >
>

Re: ANNOUNCE: Yu Li joins the Apache HBase PMC

[ramkrishna vasudevan]

Congratulations!!

On Mon, Apr 17, 2017 at 7:05 AM, Guanghao Zhang  wrote:

> Congratulations!
>
> 2017-04-16 21:36 GMT+08:00 Yu Li :
>
> > Thanks all! My honor and will do my best.
> >
> > Best Regards,
> > Yu
> >
> > On 16 April 2017 at 14:20, 张铎(Duo Zhang)  wrote:
> >
> > > Congratulations!
> > >
> > > 2017-04-16 11:24 GMT+08:00 Mikhail Antonov :
> > >
> > >> Congratulations Yu!
> > >>
> > >> -Mikhail
> > >>
> > >> On Sat, Apr 15, 2017 at 12:44 PM, Nick Dimiduk 
> > >> wrote:
> > >>
> > >> > Congratulations Yu and thanks a lot! Keep up the good work!
> > >> >
> > >> > On Fri, Apr 14, 2017 at 7:22 AM Anoop John 
> > >> wrote:
> > >> >
> > >> > > On behalf of the Apache HBase PMC I"m pleased to announce that Yu
> Li
> > >> > > has accepted our invitation to become a PMC member on the Apache
> > HBase
> > >> > > project. He has been an active contributor to HBase for past many
> > >> > > years. Looking forward for
> > >> > > many more contributions from him.
> > >> > >
> > >> > > Welcome to the PMC, Yu Li...
> > >> > >
> > >> > >
> > >> > > -Anoop-
> > >> > >
> > >> >
> > >>
> > >>
> > >>
> > >> --
> > >> Thanks,
> > >> Michael Antonov
> > >>
> > >
> > >
> >
>

[ANNOUNCE] - Welcome our new HBase committer Anastasia Braginsky

[ramkrishna vasudevan]

Hi All

Welcome Anastasia Braginsky, one more female committer to HBase. She has
been active now for a while with her Compacting memstore feature and she
along with Eshcar have done lot of talks in various meetups and HBaseCon on
their feature.

Welcome onboard and looking forward to work with you Anastasia !!!

Regards
Ram

Re: [ANNOUNCE] New HBase committer Chia-Ping Tsai

[ramkrishna vasudevan]

Congratulations !!!

On Mon, Mar 20, 2017 at 10:43 PM, Huaxiang Sun  wrote:

> Congratulations, Chia-Ping!
>
> > On Mar 18, 2017, at 5:38 AM, Ashish Singhi  com> wrote:
> >
> > Many Congratulations !
> >
> > On Sat, Mar 18, 2017 at 4:00 AM, Anoop John 
> wrote:
> >
> >> On behalf of the Apache HBase PMC, I am pleased to announce that
> Chia-Ping
> >> Tsai
> >> has accepted the PMC's invitation to become a committer on the project.
> We
> >> appreciate all of his contributions thus far and look forward
> >> to his continued involvement.
> >>
> >> Congratulation and welcome Chia-Ping Tsai!
> >>
> >> -Anoop-
> >>
>
>

Re: ANNOUNCE: Good news!!!! Two new PMC and a new Committer!

[ramkrishna vasudevan]

Congratulations to all of you!! Welcome on board !!!

Regards
Ram

On Fri, Mar 17, 2017 at 1:00 PM, Yu Li  wrote:

> Congratulations Josh, Jerry and Eshcar! Another female committer, bravo!!
>
> Best Regards,
> Yu
>
> On 17 March 2017 at 15:24, Guanghao Zhang  wrote:
>
> > Congratulations!
> >
> > 2017-03-17 14:49 GMT+08:00 ashish singhi :
> >
> > > Many congratulations, Josh, Jing Chen He and Eshcar !!!
> > >
> > > -Original Message-
> > > From: saint@gmail.com [mailto:saint@gmail.com] On Behalf Of
> > Stack
> > > Sent: 17 March 2017 11:07
> > > To: HBase Dev List 
> > > Subject: ANNOUNCE: Good news Two new PMC and a new Committer!
> > >
> > > Josh Elser and Jing Chen (Jerry) have both been added to the HBase PMC.
> > > These lads have been stellar helping the project along; Jerry for a
> good
> > > few years now and Josh, though less time served, has made up for the
> lack
> > > with style. It makes sense that they be made PMCers!
> > >
> > > Let me take this opportunity while I have your attention to also
> welcome
> > > Eshcar Hillel as a Committer on Apache HBase. We are very glad to have
> > her
> > > on-board. She's a brainbox who has been working on the inmemory
> > compaction
> > > project among other things and is without fear when it comes to taking
> a
> > > deep dive into crud!
> > >
> > > Welcome aboard Josh, Jing Chen He, and Eshcar!
> > >
> > > St.Ack
> > >
> >
>

Re: Moving 2.0 forward

[ramkrishna vasudevan]

Hi All

Thanks Stack. The doc looks great. The offheap write path/read path- I
think from the read path perspective we have some good feedback from
Alibaba folks.
The write path subtasks are all done. We are currently working on some perf
results that would help us to come up with some docs that suggests best
configs and tunings for the offheap write path configurations.


Regards
Ram

On Thu, Jan 19, 2017 at 5:37 AM, Andrew Purtell 
wrote:

> I'm interested in both split meta and rsgroups. Good news. I'd like to
> help test.
>
>
> > On Jan 18, 2017, at 2:53 PM, Stack  wrote:
> >
> >> On Wed, Jan 18, 2017 at 2:26 PM, Francis Liu  wrote:
> >>
> >> Hi Stack,
> >> I'd like to get split meta (HBASE-112288) in 2.x as well. I can have a
> 2.x
> >> draft up next week. Was working on the 1.x version internally.
> >> Also if you'd like I can be the owner for rsgroups as well.
> >> Thanks,Francis
> >>
> >>
> >>
> >> I added splitting meta as a possible and had you and I as owner on
> > rsgroups (I was doing to do a bit of testing and doc for this feature).
> >
> > Would love to see splittable meta show up. Needs to be rolling
> upgradeable
> > though. Lets chat up on the issue.
> > St.Ack
> >
> >
> >
> >
> >>
> >>
> >>
> >>On Wednesday, January 18, 2017 11:29 AM, Stack 
> >> wrote:
> >>
> >>
> >> Done Thiruvel (And thanks Guanghao for adding hbase-replication).
> >> St.Ack
> >>
> >> On Tue, Jan 17, 2017 at 6:11 PM, Thiruvel Thirumoolan <
> >> thiru...@yahoo-inc.com.invalid> wrote:
> >>
> >>> Hi Stack,
> >>> I would like to add Favored Nodes to the ancillary section.
> >>> HBASE-15531: Favored Nodes EnhancementsStatus: Active
> development.Owner:
> >>> Thiruvel Thanks!-Thiruvel
> >>>
> >>>   On Monday, January 16, 2017 2:10 PM, Stack  wrote:
> >>>
> >>>
> >>> On Mon, Jan 16, 2017 at 3:01 AM, Guanghao Zhang 
> >>> wrote:
> >>>
>  For 6. Significant contirbs in master only, there are some issues
> about
>  replication operations routed through master. They are sub-task
>  of HBASE-10504. And there are other umbrella issue for replication,
> >> like
>  HBase-14379 Replication V2 and HBASE-15867 Moving HBase Replication
>  tracking from Zookeeper to HBase. So i thought we can add a new
> section
>  named
>  hbase-replication to possible 2.0.0s. This will help us to track the
> >>> state.
>  Thanks.
> 
> >>>
> >>> Thanks Guanghao Zhang. I agree. I made you an editor. If you want to
> >> have a
> >>> go at a first cut, be my guest. If nothing done in the next day or so,
> >> I'll
> >>> add this section Sir.
> >>> Thanks,
> >>> M
> >>>
> >>>
> >>>
> >>>
> >>
> >>
> >>
> >>
>

Re: Resignation as RM for branch 0.98

[ramkrishna vasudevan]

Thanks for your efforts Andrew. +1 on EOL of 0.98.

On Mon, Jan 16, 2017 at 10:27 PM, Josh Elser  wrote:

> Big +1 to your efforts, Andrew, and an EOL on 0.98
>
>
> Ted Yu wrote:
>
>> Kudos to Andrew.
>>
>> On Mon, Jan 9, 2017 at 11:26 AM, Enis Söztutar  wrote:
>>
>> Thanks Andrew for carrying this.
>>>
>>> +1 on the EOL messaging.
>>>
>>> Enis
>>>
>>> On Mon, Jan 9, 2017 at 10:37 AM, Mikhail Antonov
>>> wrote:
>>>
>>> Yeah - thank you for all hard work and time investment on reviews,
 cherry-picking, backporting and testing Andrew!

 +1 to retire 0.98.

 -Mikhail

 On Mon, Jan 9, 2017 at 10:28 AM, Sean Busbey  wrote:

 Thanks for all the hard work around 0.98 Andrew!
>
> I'm +1 on retiring 0.98, but I think we should solicit feedback from
> user@hbase before making a decision.
>
> On Fri, Jan 6, 2017 at 6:51 PM, Andrew Purtell
> wrote:
>
>> HBasers,
>>
>> Happy new year!
>>
>> The 0.98 branch has had a great run at a fairly constant release
>>
> cadence,

> producing 24 releases in all.
>>
>> I served as the branch RM for 98 for all of that time, and now I feel
>>
> the

> time has come to move on to work with more recent code. Therefore
>>
> please

> consider this my resignation as such (it's an informal position after
>>
> all

> (smile)) from the role of branch 98 RM.
>>
>> It will be up to the PMC to decide what to do about 98 going forward.
>>
> My

> recommendation is to put out the message that 98 branch has been end
>>
> of
>>>
 lifed (EOL). There is no reason someone else couldn't don the RM hat
>>
> and

> put out an additional 98 release, for example, in order to deal with
>>
> a
>>>
 critical fix for something we just can't abide shipping as the last
>>
> release
>
>> of anything. That person might even be me. However, without a
>>
> dedicated
>>>
 shepard the management of the 98 branch is pretty difficult these
>>
> days.
>>>
 It
>
>> requires Java 6 to build, which requires a hack to its security
>>
> classes
>>>
 to
>
>> communicate with modern SSL servers like repository.apache.org, and
>> mandates Hadoop 1 compatibility, which can lead to tricky problems
>>
> found

> only at release time. Backports are rarely clean. Assume 30 minutes
>>
> to
>>>
 2

> hours work for each attempt at a backport, including time for tests
>>
> to
>>>
 complete. Maintaining and releasing 98 branch will be a time
>>
> commitment.

> FWIW. All that said if someone else would like to step up into the 98
>>
> RM

> role, then we don't need to consider calling it EOL. However, as I've
>>
> said,
>
>> that is my humble recommendation to the PMC.
>>
>>
>> --
>> Best regards,
>>
>> - Andy
>>
>> If you are given a choice, you believe you have acted freely. -
>>
> Raymond
>>>
 Teller (via Peter Watts)
>>
>

 --
 Thanks,
 Michael Antonov


>>

Re: [ANNOUNCE] New HBase committer Guanghao Zhang

[ramkrishna vasudevan]

Congratulations and welcome Guanghao.

Regards
Ram

On Tue, Dec 20, 2016 at 4:07 PM, Guanghao Zhang  wrote:

> Thanks all. Looking forward to work with you guys and keep contributing for
> HBase. Thanks.
>
> 2016-12-20 16:48 GMT+08:00 Yu Li :
>
> > Congratulations and welcome Guanghao!
> >
> > Best Regards,
> > Yu
> >
> > On 20 December 2016 at 12:59, 宾莉金 or binlijin 
> wrote:
> >
> > > Congratulations and welcome!
> > >
> > > 2016-12-20 12:54 GMT+08:00 Nick Dimiduk :
> > >
> > > > Congratulations Guanghao and thank you for all your contributions!
> > > >
> > > > On Mon, Dec 19, 2016 at 5:37 PM Duo Zhang 
> wrote:
> > > >
> > > > > On behalf of the Apache HBase PMC, I am pleased to announce that
> > > Guanghao
> > > > >
> > > > > Zhang has accepted the PMC's invitation to become a committer on
> the
> > > > >
> > > > > project. We appreciate all of Guanghao's generous contributions
> thus
> > > far
> > > > >
> > > > > and look forward to his continued involvement.
> > > > >
> > > > >
> > > > >
> > > > > Congratulations and welcome, Guanghao!
> > > > >
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > *Best Regards,*
> > >  lijin bin
> > >
> >
>

Re: [ANNOUNCE] New HBase Committer Josh Elser

[ramkrishna vasudevan]

Congratulations Josh !!!

Regards
Ram

On Tue, Dec 13, 2016 at 4:18 AM, Enis Söztutar  wrote:

> Congrats Josh!
>
> Enis
>
> On Mon, Dec 12, 2016 at 11:39 AM, Esteban Gutierrez 
> wrote:
>
> > Congrats and welcome, Josh!
> >
> > esteban.
> >
> >
> > --
> > Cloudera, Inc.
> >
> >
> > On Sun, Dec 11, 2016 at 10:17 PM, Yu Li  wrote:
> >
> > > Congratulations and welcome!
> > >
> > > Best Regards,
> > > Yu
> > >
> > > On 12 December 2016 at 12:47, Mikhail Antonov 
> > > wrote:
> > >
> > > > Congratulations Josh!
> > > >
> > > > -Mikhail
> > > >
> > > > On Sun, Dec 11, 2016 at 5:20 PM, 张铎  wrote:
> > > >
> > > > > Congratulations!
> > > > >
> > > > > 2016-12-12 9:03 GMT+08:00 Jerry He :
> > > > >
> > > > > > Congratulations , Josh!
> > > > > >
> > > > > > Good work on the PQS too.
> > > > > >
> > > > > > Jerry
> > > > > >
> > > > > > On Sun, Dec 11, 2016 at 12:14 PM, Josh Elser 
> > > > wrote:
> > > > > >
> > > > > > > Thanks, all. I'm looking forward to continuing to work with you
> > > all!
> > > > > > >
> > > > > > >
> > > > > > > Nick Dimiduk wrote:
> > > > > > >
> > > > > > >> On behalf of the Apache HBase PMC, I am pleased to announce
> that
> > > > Josh
> > > > > > >> Elser
> > > > > > >> has accepted the PMC's invitation to become a committer on the
> > > > > project.
> > > > > > We
> > > > > > >> appreciate all of Josh's generous contributions thus far and
> > look
> > > > > > forward
> > > > > > >> to his continued involvement.
> > > > > > >>
> > > > > > >> Allow me to be the first to congratulate and welcome Josh into
> > his
> > > > new
> > > > > > >> role!
> > > > > > >>
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Thanks,
> > > > Michael Antonov
> > > >
> > >
> >
>

Re: Talks from hbaseconeast2016 have been posted

[ramkrishna vasudevan]

Thanks for the slides.

Regards
Ram

On Fri, Dec 9, 2016 at 5:49 AM, Stack  wrote:

> The slides were just pushed up on slideshare:
> http://www.slideshare.net/search/slideshow?searchfrom=
> header=hbaseconeast2016
>
> St.Ack
> P.S. Thanks Carter
>
> On Thu, Oct 27, 2016 at 9:56 AM, Stack  wrote:
>
> > Some good stuff in here:
> >
> > + BigTable Lead on some interesting tricks done to make the service more
> > robust
> > + Compare of SQL tiers by the boys from Splice
> > + Mikhail (looking like a gangster!) on the dead-but-not-dead
> RegionServer
> >
> > Then there's our JMS on HBase+Spark,  Joep and Sangjin on HBase as store
> > for Yarn Timeline Service v2 Lots of meaty material.
> >
> > Yours,
> > The Program Committee
> > P.S. Thanks Carter Page for assemblage.
> >
> >
> >
>

Re: meta region written by 2.0 cannot be opened by 1.x

[ramkrishna vasudevan]

If its going to be a pain then we need to support writing the older class
name in the FFT (trailer) and then use our internal mapping.

On Wed, Dec 7, 2016 at 3:22 PM, Anoop John <anoop.hb...@gmail.com> wrote:

> I read the comments in the issue 16189..  This concern was raised
> there already..   This is what I replied there.
> "Ya we may need to make such a need. User on 1.x has to first upgrade
> (rolling upgrade supported) to latest version in 1.x and then to 2.0.
> This was discussed some other place also. I believe in the mail thread
> where we discussed abt rolling upgrade support in 2.0"
>
> Not able to find the original mail thread which discussed abt rolling
> upgrade support in 2.0Pls correct if took some thing wrong way..
> Ya I agree that this way of indirection might be inconvenience for
> users.
>
>
> -Anoop-
>
> On Wed, Dec 7, 2016 at 3:13 PM, Ted Yu <yuzhih...@gmail.com> wrote:
> > bq. write old name only and within code we will have to map it with new
> > name.
> >
> > This is a viable approach.
> > Even if we document upgrading to 1.2.3+, some users may not perform this
> > step before upgrading to 2.0
> >
> > On Tue, Dec 6, 2016 at 10:04 PM, Anoop John <anoop.hb...@gmail.com>
> wrote:
> >
> >> Ya that bug raised by Enis was very valid..  So this means when
> >> rolling upgrade happens, if some of the other RSs with version <1.2.3
> >> , which is not having this fix,  this issue might come up!
> >> How to address then?   Do we need to enforce a 1.2.3+ upgrade 1st and
> >> then only 2.0 rolling upgrade?
> >>
> >> Or else we will need to fix in 2.0..  We write the new Comparator
> >> class name in FFT (trunk code)   To fix, we might have to write old
> >> name only and within code we will have to map it with new name. It
> >> will be ugly! It can be fixed only in 3.0 may be..  But that can make
> >> the rolling upgrade story easier for users..   Just saying the
> >> possibilities..
> >>
> >> Thanks Ted to bring it up again.
> >>
> >> -Anoop-
> >>
> >> On Wed, Dec 7, 2016 at 10:04 AM, ramkrishna vasudevan
> >> <ramkrishna.s.vasude...@gmail.com> wrote:
> >> > I think when Enis reported the issue (
> >> > https://issues.apache.org/jira/browse/HBASE-16189), in rolling
> upgrade
> >> > regions could move around. So a region created in 2.0 moved to a
> server
> >> > with 1.x.
> >> >
> >> > Regards
> >> > Ram
> >> >
> >> >
> >> > On Wed, Dec 7, 2016 at 1:27 AM, Stack <st...@duboce.net> wrote:
> >> >
> >> >> On Tue, Dec 6, 2016 at 10:19 AM, Ted Yu <yuzhih...@gmail.com> wrote:
> >> >>
> >> >> > Looking at http://hbase.apache.org/book.html#executing.the.0.96.
> >> upgrade
> >> >> ,
> >> >> > there is step of running "bin/hbase upgrade -check"
> >> >> >
> >> >> > How about adding a sample hfile which contains
> >> >> > CellComparator$MetaCellComparator
> >> >> > so that the upgrade check can read and verify ?
> >> >> >
> >> >> >
> >> >>
> >> >> We don't support downgrade. Never have.
> >> >> St.Ack
> >> >>
> >> >>
> >> >>
> >> >> > On Tue, Dec 6, 2016 at 8:50 AM, Ted Yu <yuzhih...@gmail.com>
> wrote:
> >> >> >
> >> >> > > The build I used yesterday didn't include HBASE-16189
> >> >> > > <https://issues.apache.org/jira/browse/HBASE-16189>
> >> >> > >
> >> >> > > Once it is included, the cluster can be downgraded fine.
> >> >> > >
> >> >> > > I wonder how users would know that their existing deployment has
> >> >> > > HBASE-16189 <https://issues.apache.org/jira/browse/HBASE-16189>
> >> before
> >> >> > > upgrading to 2.0 release.
> >> >> > >
> >> >> > > On Tue, Dec 6, 2016 at 2:29 AM, ramkrishna vasudevan <
> >> >> > > ramkrishna.s.vasude...@gmail.com> wrote:
> >> >> > >
> >> >> > >> @Ted
> >> >> > >> Does your version have this fix
> >> >> > >> https://issues.apache.org/jira/browse/HBASE-16189
> >> >> > >>
> >> &g

Re: meta region written by 2.0 cannot be opened by 1.x

[ramkrishna vasudevan]

I think when Enis reported the issue (
https://issues.apache.org/jira/browse/HBASE-16189), in rolling upgrade
regions could move around. So a region created in 2.0 moved to a server
with 1.x.

Regards
Ram


On Wed, Dec 7, 2016 at 1:27 AM, Stack <st...@duboce.net> wrote:

> On Tue, Dec 6, 2016 at 10:19 AM, Ted Yu <yuzhih...@gmail.com> wrote:
>
> > Looking at http://hbase.apache.org/book.html#executing.the.0.96.upgrade
> ,
> > there is step of running "bin/hbase upgrade -check"
> >
> > How about adding a sample hfile which contains
> > CellComparator$MetaCellComparator
> > so that the upgrade check can read and verify ?
> >
> >
>
> We don't support downgrade. Never have.
> St.Ack
>
>
>
> > On Tue, Dec 6, 2016 at 8:50 AM, Ted Yu <yuzhih...@gmail.com> wrote:
> >
> > > The build I used yesterday didn't include HBASE-16189
> > > <https://issues.apache.org/jira/browse/HBASE-16189>
> > >
> > > Once it is included, the cluster can be downgraded fine.
> > >
> > > I wonder how users would know that their existing deployment has
> > > HBASE-16189 <https://issues.apache.org/jira/browse/HBASE-16189> before
> > > upgrading to 2.0 release.
> > >
> > > On Tue, Dec 6, 2016 at 2:29 AM, ramkrishna vasudevan <
> > > ramkrishna.s.vasude...@gmail.com> wrote:
> > >
> > >> @Ted
> > >> Does your version have this fix
> > >> https://issues.apache.org/jira/browse/HBASE-16189
> > >>
> > >> Regards
> > >> Ram
> > >>
> > >> On Tue, Dec 6, 2016 at 3:56 PM, Ted Yu <yuzhih...@gmail.com> wrote:
> > >>
> > >> > Is the assumption that hbase:meta would not split ?
> > >> >
> > >> > In other thread, Francis Liu was proposing supporting splittable
> > >> > hbase:meta in 2.0 release.
> > >> >
> > >> > Cheers
> > >> >
> > >> > > On Dec 6, 2016, at 2:20 AM, 张铎 <palomino...@gmail.com> wrote:
> > >> > >
> > >> > > Could this be solved by hosting meta only on master?
> > >> > >
> > >> > > BTW, MetaCellComparator is introduced in HBASE-10800.
> > >> > >
> > >> > > Thanks.
> > >> > >
> > >> > > 2016-12-06 17:44 GMT+08:00 Ted Yu <yuzhih...@gmail.com>:
> > >> > >
> > >> > >> Hi,
> > >> > >> When I restarted a cluster with 1.1 , I found that hbase:meta
> > region
> > >> > >> (written to by the previously deployed 2.0) couldn't be opened:
> > >> > >>
> > >> > >> Caused by: java.io.IOException:
> > >> > >> org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem
> > >> reading
> > >> > >> HFile Trailer from file hdfs://yz1.xx.com:8020/apps/
> > >> hbase/data/data/
> > >> > >> hbase/meta/1588230740/info/599fc8a37311414e876803312009a986
> > >> > >>at
> > >> > >> org.apache.hadoop.hbase.regionserver.HStore.openStoreFiles(
> > >> HStore.java:
> > >> > >> 579)
> > >> > >>at
> > >> > >> org.apache.hadoop.hbase.regionserver.HStore.loadStoreFiles(
> > >> HStore.java:
> > >> > >> 534)
> > >> > >>at
> > >> > >> org.apache.hadoop.hbase.regionserver.HStore.(
> > HStore.java:275)
> > >> > >>at
> > >> > >> org.apache.hadoop.hbase.regionserver.HRegion.instantiateHSto
> > >> re(HRegion.
> > >> > >> java:5150)
> > >> > >>at
> > >> > >> org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.
> > >> java:912)
> > >> > >>at
> > >> > >> org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.
> > >> java:909)
> > >> > >>at java.util.concurrent.FutureTask.run(FutureTask.
> java:266)
> > >> > >>at
> > >> > >> java.util.concurrent.Executors$RunnableAdapter.call(
> > >> Executors.java:511)
> > >> > >>at java.util.concurrent.FutureTask.run(FutureTask.
> java:266)
> > >> > >>... 3 more
> > >> > >> Caused by: org.apache.hadoop.hbase.io.
>

Re: meta region written by 2.0 cannot be opened by 1.x

[ramkrishna vasudevan]

@Ted
Does your version have this fix
https://issues.apache.org/jira/browse/HBASE-16189

Regards
Ram

On Tue, Dec 6, 2016 at 3:56 PM, Ted Yu  wrote:

> Is the assumption that hbase:meta would not split ?
>
> In other thread, Francis Liu was proposing supporting splittable
> hbase:meta in 2.0 release.
>
> Cheers
>
> > On Dec 6, 2016, at 2:20 AM, 张铎  wrote:
> >
> > Could this be solved by hosting meta only on master?
> >
> > BTW, MetaCellComparator is introduced in HBASE-10800.
> >
> > Thanks.
> >
> > 2016-12-06 17:44 GMT+08:00 Ted Yu :
> >
> >> Hi,
> >> When I restarted a cluster with 1.1 , I found that hbase:meta region
> >> (written to by the previously deployed 2.0) couldn't be opened:
> >>
> >> Caused by: java.io.IOException:
> >> org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem reading
> >> HFile Trailer from file hdfs://yz1.xx.com:8020/apps/  hbase/data/data/
> >> hbase/meta/1588230740/info/599fc8a37311414e876803312009a986
> >>at
> >> org.apache.hadoop.hbase.regionserver.HStore.openStoreFiles(HStore.java:
> >> 579)
> >>at
> >> org.apache.hadoop.hbase.regionserver.HStore.loadStoreFiles(HStore.java:
> >> 534)
> >>at
> >> org.apache.hadoop.hbase.regionserver.HStore.(HStore.java:275)
> >>at
> >> org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.
> >> java:5150)
> >>at
> >> org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:912)
> >>at
> >> org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:909)
> >>at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> >>at
> >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> >>at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> >>... 3 more
> >> Caused by: org.apache.hadoop.hbase.io.hfile.CorruptHFileException:
> Problem
> >> reading HFile Trailer from file hdfs://
> >> yz1.xx.com:8020/apps/hbase/data/data/hbase/ meta/1588230740/
> >> info/599fc8a37311414e876803312009a986
> >>at
> >> org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(
> HFile.java:483)
> >>at
> >> org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:511)
> >>at
> >> org.apache.hadoop.hbase.regionserver.StoreFile$Reader.
> >> (StoreFile.java:1128)
> >>at
> >> org.apache.hadoop.hbase.regionserver.StoreFileInfo.
> >> open(StoreFileInfo.java:267)
> >>at
> >> org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:409)
> >>at
> >> org.apache.hadoop.hbase.regionserver.StoreFile.
> >> createReader(StoreFile.java:517)
> >>at
> >> org.apache.hadoop.hbase.regionserver.HStore.createStoreFileAndReader(
> >> HStore.java:687)
> >>at
> >> org.apache.hadoop.hbase.regionserver.HStore.access$000(HStore.java:130)
> >>at
> >> org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:554)
> >>at
> >> org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:551)
> >>... 6 more
> >> Caused by: java.io.IOException: java.lang.ClassNotFoundException:
> >> org.apache.hadoop.hbase.CellComparator$MetaCellComparator
> >>at
> >> org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.getComparatorClass(
> >> FixedFileTrailer.java:581)
> >>at
> >> org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.deserializeFromPB(
> >> FixedFileTrailer.java:300)
> >>at
> >> org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.
> >> deserialize(FixedFileTrailer.java:242)
> >>at
> >> org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(
> >> FixedFileTrailer.java:407)
> >>at
> >> org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(
> HFile.java:468)
> >>... 15 more
> >> Caused by: java.lang.ClassNotFoundException:
> >> org.apache.hadoop.hbase.CellComparator$MetaCellComparator
> >>at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> >>at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> >>at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
> >>at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> >>at java.lang.Class.forName0(Native Method)
> >>at java.lang.Class.forName(Class.java:264)
> >>at
> >> org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.getComparatorClass(
> >> FixedFileTrailer.java:579)
> >>
> >> When user does rolling upgrade from 1.1 to 2.0, the above may cause
> problem
> >> if hbase:meta region is updated by server running 2.0 but later
> assigned to
> >> a region server which still runs 1.1 (due to crash of the server running
> >> 2.0, e.g.)
> >>
> >> I want to get community feedback on the severity of this issue.
> >>
> >> Thanks
> >>
>

Re: [ANNOUNCE] New HBase committer Lijin Bin

[ramkrishna vasudevan]

Congratulations !! All the best !!!

On Tue, Nov 29, 2016 at 4:29 PM, Guanghao Zhang  wrote:

> Congratulations!
>
> 2016-11-29 17:48 GMT+08:00 Duo Zhang :
>
> > On behalf of the Apache HBase PMC, I am pleased to announce that Lijin
> > Bin(binlijin) has accepted the PMC's invitation to become a committer on
> > the project. We appreciate all of Lijin's generous contributions thus far
> > and look forward to his continued involvement.
> >
> > Congratulations and welcome, Lijin!
> >
>

Re: [ANNOUNCE] New HBase committer Phil Yang

[ramkrishna vasudevan]

Congratulations !! All the best !!!

On Tue, Nov 29, 2016 at 4:29 PM, Guanghao Zhang  wrote:

> Congratulations!
>
> 2016-11-29 17:49 GMT+08:00 Duo Zhang :
>
> > On behalf of the Apache HBase PMC, I am pleased to announce that Phil
> Yang
> > has accepted the PMC's invitation to become a committer on the project.
> We
> > appreciate all of Phil's generous contributions thus far and look forward
> > to his continued involvement.
> >
> > Congratulations and welcome, Phil!
> >
>

Re: HBASE 2.0 release progress

[ramkrishna vasudevan]

For the offheap write path related things,
The parent JIRA
https://issues.apache.org/jira/browse/HBASE-15179

We have raised subtasks under that. Many of them are closed and there are 3
main sub-jiras now. All the 3 are now patch available.

Subtasks include
-> Reading the requests in to a ByteBuffer pool of fixed size.
-> Making ChunkPool work with direct byte buffers
-> Sizing heuristics of onheap and offheap cells (along with memstore flush
sizing).
-> Async WAL support.


There were some POCs that we done before going with the patches against
trunk. The POC reports and design docs are available in the following links

https://docs.google.com/document/d/1fj5P8JeutQ-Uadb29ChDscMu
MaJqaMNRI86C4k5S1rQ/edit
https://docs.google.com/document/d/1bYIkwNVVyYk8bgik-C5x_yc_
2hWHyi2-9KMwEFWLfzY/edit

The POC reports suggest that we get around 24% improvement with 100 threads
when using the PE tool.

In these POCs we have also used the Y! compacting memstore and its variants
called Pipeline Memstore(which is under discussion) to prove how far their
work can help offheap because it could reduce the onheap overhead of the
cells in memstore. They can explain their current state of affairs when
they do write up here.

We are now testing with AsyncWAL to see if that could be made the default
WAL because currently in HDFS DFSOutputSTream we don't have support for
bytebuffers. We need to copy the cell contents to onheap byte[]. Currenly
as part of HBASE-15786 there is a workaround we are doing to over come that.

Sorry. I thought I had sent this mail but I just saw that it is in my
Drafts.
Regards
Ram


On Mon, Nov 14, 2016 at 5:27 AM, Stack  wrote:

> Thanks for the writeup Stephen.
>
> See below.
>
> On Fri, Nov 11, 2016 at 5:15 PM, Stephen Jiang 
> wrote:
>
> > Hello, fellow HBASE developers,
> >
> > We are making progress towards HBASE 2.0 releases.  I am using the
> > following queries to search for on-going HBASE 2.0 feature work items
> > (project = HBase AND (fixVersion = 2.0.0 OR affectedVersion = 2.0.0) AND
> > resolution is EMPTY AND (issuetype != Bug AND issuetype != Test AND
> > issuetype != Sub-task) ORDER BY issuetype DESC), at this time, we have
> > 247!  That is a lot.  At some time in near future, we definitely need to
> > trim down the list; otherwise, 2.0 will never be released.
> >
> >
> Agree. Suggest we all do our own review but at a certain stage, it is up to
> the RMs to make a call on what shape of 2.0 will be.
>
>
>
> > For now, Matteo and I are tracking some big projects that are on-going:
> >
> > (1). HBASE-14350 the stable Assignment Manager (using Procedure V2)
> > - This is a blocker to have branch-2 cut.  In the past few weeks, we made
> > good progress and majority of implementation is done.  The patches are
> > under review and testing.  Matto is drafting a document for review.
> >
> > (2). HBASE-14414 Backup/Restore Phase 3
> > - Currently it is blocked by HBASE-14123.  The giant HBASE-14123 patches
> > was discussed and reviewed from the community (see
> > http://www.mail-archive.com/dev@hbase.apache.org/msg41090.html for the
> > long
> > discussion); and all feedback were taken care of in the latest patch.
> > Currently I marked HBASE-14123 as 2.0 blocker, as without it, the further
> > develpment of backup/restore is blocked - the backup/restore is a key
> > enterprise feature for 2.0 release.  I think HBASE-14123 is ready for
> > another round of vote.
> >
> >
> IMO, not a blocker since it ancillary function but something we should get
> in.
>
>
>
> > (3). HBASE-15179 Offheap
> > - This is another important feature that I think it is MUST for 2.0.
> Since
> > stack works closely with Intel developers, he has some insight on this
> > project: "Intel are betting on this. Alibaba are using the offheap read
> > path and interested in write path too. This work is still very much up in
> > the air and being worked out as we go (especially the Y! Israel inmemory
> > compaction component).  It is a little shakey dependent on mslab pooling,
> > blockcache being on by default, async wal being default, and then
> dependent
> > on lots of perf and ITBLL testing."
> >
> >
> The above is from a private, hurried email that does not do the current
> state justice.
>
> Ram and Anoop are the authorities on offheaping and have been keeping up
> their state in an associated doc. The lads might be persuaded do a summary
> of current state here.
>
> Ditto for inmemory compaction. Our Anastasia and Eshcar are working hard at
> the moment doing laste pieces and perf testing. Maybe we can a status
> dumped here in dev.
>
>
>
> > (4). HBASE-16952 Protobuf3
> > - Good news, stack got the majority of work done already.  This is a MUST
> > for 2.0 release.  Now we only have a small sub-task HBASE-16967
> (findbugs)
> > left.
> >
> >
> To be clear, pb3 is under the offheaping umbrella. Let's add async WAL to
> this list, also needed by offheaping.
>
> Will be back after 

Re: [ANNOUNCE] Stephen Yuan Jiang joins Apache HBase PMC

[ramkrishna vasudevan]

Congrats Stephen!!

On Tue, Oct 18, 2016 at 2:37 AM, Stack  wrote:

> Wahoo!
>
> On Fri, Oct 14, 2016 at 11:27 AM, Enis Söztutar  wrote:
>
> > On behalf of the Apache HBase PMC, I am happy to announce that Stephen
> has
> > accepted our invitation to become a PMC member of the Apache HBase
> project.
> >
> > Stephen has been working on HBase for a couple of years, and is already a
> > committer for more than a year. Apart from his contributions in proc v2,
> > hbck and other areas, he is also helping for the 2.0 release which is the
> > most important milestone for the project this year.
> >
> > Welcome to the PMC Stephen,
> > Enis
> >
>