Re: [ANNOUNCE] Please welcome Francis Liu to the HBase PMC

[Gary Helmling]

Congrats and welcome, Francis!

On Wed, Apr 11, 2018 at 1:13 PM Ted Yu  wrote:

> Congratulations, Francis !
>
> On Wed, Apr 11, 2018 at 1:03 PM, Andrew Purtell 
> wrote:
>
> > On behalf of the Apache HBase PMC I am pleased to announce that Francis
> > Liu has accepted our invitation to become a PMC member on the Apache
> > HBase project. We appreciate Francis stepping up to take more
> > responsibility in the HBase project. He has been an active contributor to
> > HBase for many years and recently took over responsibilities as branch RM
> > for branch-1.3.
> >
> > Please join me in welcoming Francis to the HBase PMC!
> >
> > --
> > Best regards,
> > Andrew
> >
>

[jira] [Created] (HBASE-19332) DumpReplicationQueues misreports total WAL size

[Gary Helmling (JIRA)]

Gary Helmling created HBASE-19332:
-

 Summary: DumpReplicationQueues misreports total WAL size
 Key: HBASE-19332
 URL: https://issues.apache.org/jira/browse/HBASE-19332
 Project: HBase
  Issue Type: Bug
  Components: Replication
Reporter: Gary Helmling
Assignee: Gary Helmling
Priority: Trivial


DumpReplicationQueues uses an int to collect the total WAL size for a queue.  
Predictably, this overflows much of the time.  Let's use a long instead.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: [DISCUSSION] Removing the bypass semantic from the Coprocessor APIs

[Gary Helmling]

The Tephra TransactionProcessor CP makes use of bypass() in preDelete() to
override handling of delete tombstones in a transactional way:
https://github.com/apache/incubator-tephra/blob/master/tephra-hbase-compat-1.3/src/main/java/org/apache/tephra/hbase/coprocessor/TransactionProcessor.java#L244

The CDAP IncrementHandler CP also makes use of bypass() in preGetOp() and
preIncrementAfterRRowLock() to provide a transaction implementation of
readless increments:
https://github.com/caskdata/cdap/blob/develop/cdap-hbase-compat-1.1/src/main/java/co/cask/cdap/data2/increment/hbase11/IncrementHandler.java#L121

What would be the alternate approach for these applications?  In both cases
they need to impose their own semantics on the underlying KeyValue
storage.  Is there a different way this can be done?


On Tue, Oct 10, 2017 at 11:58 AM Anoop John  wrote:

> Wrap core scanners is different right?  That can be done in post
> hooks.  I have seen many use cases for this..  Its the question abt
> the pre hooks where we have not yet created the core object (like
> scanner).  The CP pre code itself doing the work of object creation
> and so the core code is been bypassed.Well the wrapping thing can
> be done in pre hook also. First create the core object by CP code
> itself and then do the wrapped object and return.. I have seen in one
> jira issue where the usage was this way..   The wrapping can be done
> in post also in such cases I believe.
>
> -Anoop-
>
> On Wed, Oct 11, 2017 at 12:23 AM, Andrew Purtell 
> wrote:
> > I think we should continue to support overriding function by object
> > inheritance. I didn't mention this and am not proposing more than
> removing
> > the bypass() sematic. No more no less. Phoenix absolutely depends on
> being
> > able to wrap core scanners and return the wrappers.
> >
> >
> > On Tue, Oct 10, 2017 at 11:50 AM, Anoop John 
> wrote:
> >
> >> When we say bypass the core code, it can be done today not only by
> >> calling bypass but by returning a not null object for some of the pre
> >> hooks.  Like preScannerOpen() if it return a scanner object, we will
> >> avoid the remaining core code execution for creation of the
> >> scanner(s).  So this proposal include this aspect also and remove any
> >> possible way of bypassing the core code by the CP hook code execution
> >> ?   Am +1.
> >>
> >> -Anoop-
> >>
> >> On Tue, Oct 10, 2017 at 11:40 PM, Andrew Purtell 
> >> wrote:
> >> > The coprocessor API provides an environment method, bypass(), that
> when
> >> > called from a preXXX hook will cause the core code to skip all
> remaining
> >> > processing. This capability was introduced on HBASE-3348. Since this
> >> time I
> >> > think we are more enlightened about the complications of this feature.
> >> (Or,
> >> > anyway, speaking for myself:)
> >> >
> >> > Not all hooks provide the bypass semantic. Where this is the case the
> >> > javadoc for the hook says so, but it can be missed. If you call
> bypass()
> >> in
> >> > a hook where it is not supported it is a no-op. This can lead to a
> poor
> >> > developer experience.
> >> >
> >> > Where bypass is supported what is being bypassed is all of the core
> code
> >> > implementing the remainder of the operation. In order to understand
> what
> >> > calling bypass() will skip, a coprocessor implementer should read and
> >> > understand all of the remaining code and its nuances. Although I think
> >> this
> >> > is good practice for coprocessor developers in general, it demands a
> >> lot. I
> >> > think it would provide a much better developer experience if we didn't
> >> > allow bypass, even though it means - in theory - a coprocessor would
> be a
> >> > lot more limited in some ways than before. What is skipped is
> extremely
> >> > version dependent. That core code will vary, perhaps significantly,
> even
> >> > between point releases. We do not provide the promise of consistent
> >> > behavior even between point releases for the bypass semantic. To
> achieve
> >> > that we could not change any code between hook points. Therefore the
> >> > coprocessor implementer becomes an HBase core developer in practice as
> >> soon
> >> > as they rely on bypass(). Every release of HBase may break the
> assumption
> >> > that the replacement for the bypassed code takes care of all necessary
> >> > skipped concerns. Because those concerns can change at any point,
> such an
> >> > assumption is never safe.
> >> >
> >> > I say "in theory" because I would be surprised if anyone is relying on
> >> the
> >> > bypass for the above reason. I seem to recall that Phoenix might use
> it
> >> in
> >> > one place to promote a normal mutation into an atomic operation, by
> >> > substituting one for the other, but if so that objective could be
> >> > reimplemented using their new locking manager.
> >> >
> >> > --
> >> > Best regards,
> >> > Andrew
> >>
> >
> >
> >
> > --
> > 

Re: [DISCUSS] 2.0 time to remove table owners?

[Gary Helmling]

>
>
> On 9/26/17 11:27 AM, Andrew Purtell wrote:
> > It would be a major functional change. Someone might be relying on the
> > table ownership semantic. However, 2.0 would be the next opportunity to
> > introduce a change of this type before 3.0.
> >
> > I don't think we need table owners. It is a shortcut in the permissions
> > model which is good for usability but bad for adding complexity. Removing
> > the shortcut would make it more likely we'd see odd situations like
> where a
> > user can create a table but surprisingly lack other permissions, but that
> > would be a consequence of mismanagement of grants by cluster admins, not
> a
> > bug or functional gap introduced by removing table ownership.
>
> +1 well put.
>
>
+1 as well.

Table owners were originally an approximation of delegated admin
functionality -- basically you can admin what you create.

The metadata aspect  of table owners (HBASE-11996) still seems useful.
But, like Andy says, the access control aspect just adds some implicit
complexity without much benefit.  For a delegated admin scenario, making
this explicit with ADMIN permission over a specific table or namespace
would be a more manageable and more transparent approach.

Removing the built-in grants generated with table owners seems like a good
step forward.

Re: JIRA Cleanup: 89-fb issues

[Gary Helmling]

+1 to closing these all out

On Thu, Sep 21, 2017 at 12:20 AM Chia-Ping Tsai  wrote:

> +1 to cleanup.
>
> On 2017-09-21 01:31, Mike Drob  wrote:
> > Hey devs,
> >
> > Was looking at old issues for cleanup and found a bunch of things for
> > 89-fb. I don't have enough context to know if any of them are still
> > relevant - some have comments asking to close but no responses.
> >
> > The filter I am using is this one:
> >
> https://issues.apache.org/jira/browse/HBASE-7266?jql=summary%20~%2089-fb%20AND%20status%20%3D%20open
> >
> > If you feel that any of these issues are relevant for master/branch-2,
> > please update them with detail. Otherwise, I will plan to close all of
> them
> > as obsolete, probably early next week.
> >
> > Thanks,
> > Mike
> > +
>

Re: Please congratulate our new PMC Chair Misty Stanley-Jones

[Gary Helmling]

Congrats, Misty, and thanks for all your efforts!

On Fri, Sep 22, 2017 at 3:57 PM Umesh Agashe  wrote:

> Congratulations Misty!
>
>
>
> On Fri, Sep 22, 2017 at 11:41 AM, Esteban Gutierrez 
> wrote:
>
> > Thats awesome! Congratulations, Misty!
> >
> >
> >
> > --
> > Cloudera, Inc.
> >
> >
> > On Fri, Sep 22, 2017 at 11:27 AM, Alexander Leblang <
> > alex.lebl...@cloudera.com> wrote:
> >
> > > Congrats Misty! Well done!
> > > On Fri, Sep 22, 2017 at 11:25 AM Wei-Chiu Chuang 
> > > wrote:
> > >
> > > > Congrats! Misty!!
> > > >
> > > > On Fri, Sep 22, 2017 at 7:50 AM, Jimmy Xiang 
> wrote:
> > > >
> > > > > Congrats! Misty!!
> > > > >
> > > > > On Fri, Sep 22, 2017 at 7:16 AM, Pankaj kr 
> > > wrote:
> > > > >
> > > > > > Congratulations Misty..!! :)
> > > > > >
> > > > > >
> > > > > > -Pankaj-
> > > > > >
> > > > > >
> > > > > > -Original Message-
> > > > > > From: Andrew Purtell [mailto:apurt...@apache.org]
> > > > > > Sent: Friday, September 22, 2017 3:08 AM
> > > > > > To: dev@hbase.apache.org; u...@hbase.apache.org
> > > > > > Subject: Please congratulate our new PMC Chair Misty
> Stanley-Jones
> > > > > >
> > > > > > At today's meeting of the Board, Special Resolution B changing
> the
> > > > HBase
> > > > > > project Chair to Misty Stanley-Jones was passed unanimously.
> > > > > >
> > > > > > Please join me in congratulating Misty on her new role!
> > > > > >
> > > > > > ​(If you need any help or advice please don't hesitate to ping
> me,
> > > > Misty,
> > > > > > but I suspect you'll do just fine and won't need it.)​
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Best regards,
> > > > > > Andrew
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > A very happy Hadoop contributor
> > > >
> > >
> >
>

Re: [ANNOUNCE] New HBase committer Huaxiang Sun

[Gary Helmling]

Congratulations and welcome!

On Mon, Jun 19, 2017 at 11:07 PM Huaxiang Sun  wrote:

> Thanks all for the warm welcome, it is a great honor for me!
>
> Huaxiang
>
> > On Jun 19, 2017, at 9:27 PM, Anoop John  wrote:
> >
> > Congratulations Huaxiang
> >
> > -Anoop-
> >
> > On Tue, Jun 20, 2017 at 9:36 AM, ramkrishna vasudevan
> >  wrote:
> >> Congratulations !!!
> >>
> >> On Tue, Jun 20, 2017 at 9:32 AM, Yu Li  wrote:
> >>
> >>> Congratulations and welcome, Huaxiang!
> >>>
> >>> Best Regards,
> >>> Yu
> >>>
> >>> On 20 June 2017 at 11:03, Allan Yang  wrote:
> >>>
>  Congratulations and welcome, Huaxiang!
> 
>  Best Regards
>  Allan Yang
> 
>  2017-06-20 10:32 GMT+08:00 Pankaj kr :
> 
> > Congratulations Huaxiang..!!
> >
> > Thanks & Regards,
> > Pankaj
> >
> > HUAWEI TECHNOLOGIES CO.LTD.
> > Huawei Tecnologies India Pvt. Ltd.
> > Near EPIP Industrial Area, Kundalahalli Village
> > Whitefield, Bangalore-560066
> > www.huawei.com
> > 
> > -
> > This e-mail and its attachments contain confidential information from
> > HUAWEI, which
> > is intended only for the person or entity whose address is listed
> >>> above.
> > Any use of the
> > information contained herein in any way (including, but not limited
> to,
> > total or partial
> > disclosure, reproduction, or dissemination) by persons other than the
> > intended
> > recipient(s) is prohibited. If you receive this e-mail in error,
> please
> > notify the sender by
> > phone or email immediately and delete it!
> >
> >
> > -Original Message-
> > From: Sean Busbey [mailto:bus...@apache.org]
> > Sent: Tuesday, June 20, 2017 3:31 AM
> > To: dev; u...@hbase.apache.org
> > Subject: [ANNOUNCE] New HBase committer Huaxiang Sun
> >
> > On behalf of the Apache HBase PMC, I am pleased to announce that
> >>> Huaxiang
> > Sun has accepted the PMC's invitation to become a committer on the
>  project.
> > We appreciate all of Huaxiang's great work thus far and look forward
> to
> > continued involvement.
> >
> > Please join me in congratulating Huaxiang!
> >
> 
> >>>
>
>

[ANNOUNCE] New Apache HBase committer Ashu Pachauri

[Gary Helmling]

On behalf of the Apache HBase PMC, I am pleased to announce that Ashu
Pachauri has accepted the PMC's invitation to become a committer on the
project.  We appreciate all of Ashu's generous contributions thus far and
look forward to his continued involvement.

Congratulations and welcome, Ashu!

[jira] [Created] (HBASE-18141) Regionserver fails to shutdown when abort triggered in RegionScannerImpl during RPC call

[Gary Helmling (JIRA)]

Gary Helmling created HBASE-18141:
-

 Summary: Regionserver fails to shutdown when abort triggered in 
RegionScannerImpl during RPC call
 Key: HBASE-18141
 URL: https://issues.apache.org/jira/browse/HBASE-18141
 Project: HBase
  Issue Type: Bug
  Components: regionserver, security
Affects Versions: 1.3.1
Reporter: Gary Helmling
Assignee: Gary Helmling
Priority: Critical
 Fix For: 1.3.2


When an abort is triggered within the RPC call path by 
HRegion.RegionScannerImpl, AccessController is incorrectly apply the RPC caller 
identity in the RegionServerObserver.preStopRegionServer() hook.  This leaves 
the regionserver in a non-responsive state, where its regions are not 
reassigned and it returns exceptions for all requests.

When an abort is triggered on the server side, we should not allow a 
coprocessor to reject the abort at all.

Here is a sample stack trace:
{noformat}
17/05/25 06:10:29 FATAL regionserver.HRegionServer: RegionServer abort: loaded 
coprocessors are: [org.apache.hadoop.hbase.security.access.AccessController, 
org.apache.hadoop.hbase.security.token.TokenProvider]
17/05/25 06:10:29 WARN regionserver.HRegionServer: The region server did not 
stop
org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient 
permissions for user 'rpcuser' (global, action=ADMIN)
at 
org.apache.hadoop.hbase.security.access.AccessController.requireGlobalPermission(AccessController.java:548)
at 
org.apache.hadoop.hbase.security.access.AccessController.requirePermission(AccessController.java:522)
at 
org.apache.hadoop.hbase.security.access.AccessController.preStopRegionServer(AccessController.java:2501)
at 
org.apache.hadoop.hbase.regionserver.RegionServerCoprocessorHost$1.call(RegionServerCoprocessorHost.java:86)
at 
org.apache.hadoop.hbase.regionserver.RegionServerCoprocessorHost.execShutdown(RegionServerCoprocessorHost.java:300)
at 
org.apache.hadoop.hbase.regionserver.RegionServerCoprocessorHost.preStop(RegionServerCoprocessorHost.java:82)
at 
org.apache.hadoop.hbase.regionserver.HRegionServer.stop(HRegionServer.java:1905)
at 
org.apache.hadoop.hbase.regionserver.HRegionServer.abort(HRegionServer.java:2118)
at 
org.apache.hadoop.hbase.regionserver.HRegionServer.abort(HRegionServer.java:2125)
at 
org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.abortRegionServer(HRegion.java:6326)
at 
org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.handleFileNotFound(HRegion.java:6319)
at 
org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:5941)
at 
org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:6084)
at 
org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:5858)
at 
org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2649)
at 
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:34950)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2320)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123)
at 
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188)
at 
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168)
{noformat}

I haven't yet evaluated which other release branches this might apply to.

I have a patch currently in progress, which I will post as soon as I complete a 
test case.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (HBASE-18072) Malformed Cell from client causes Regionserver abort on flush

[Gary Helmling (JIRA)]

Gary Helmling created HBASE-18072:
-

 Summary: Malformed Cell from client causes Regionserver abort on 
flush
 Key: HBASE-18072
 URL: https://issues.apache.org/jira/browse/HBASE-18072
 Project: HBase
  Issue Type: Bug
  Components: regionserver, rpc
Affects Versions: 1.3.0
Reporter: Gary Helmling
Assignee: Gary Helmling
Priority: Critical


When a client writes a mutation with a Cell with a corrupted value length 
field, it is possible for the corrupt cell to trigger an exception on memstore 
flush, which will trigger regionserver aborts until the region is manually 
recovered.

This boils down to a lack of validation on the client submitted byte[] backing 
the cell.

Consider the following sequence:

1. Client creates a new Put with a cell with value of byte[16]
2. When the backing KeyValue for the Put is created, we serialize 16 for the 
value length field in the backing array
3. Client calls Table.put()
4. RpcClientImpl calls KeyValueEncoder.encode() to serialize the Cell to the 
OutputStream
5. Memory corruption in the backing array changes the serialized contents of 
the value length field from 16 to 48
6. Regionserver handling the put uses KeyValueDecoder.decode() to create a 
KeyValue with the byte[] read directly off the InputStream.  The overall length 
of the array is correct, but the integer value serialized at the value length 
offset has been corrupted from the original value of 16 to 48.
7. The corrupt KeyValue is appended to the WAL and added to the memstore
8. After some time, the memstore flushes.  As HFileWriter is writing out the 
corrupted cell, it reads the serialized int from the value length position in 
the cell's byte[] to determine the number of bytes to write for the value.  
Because value offset + 48 is greater than the length of the cell's byte[], we 
hit an IndexOutOfBoundsException:
{noformat}
java.lang.IndexOutOfBoundsException
at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:151)
at java.io.DataOutputStream.write(DataOutputStream.java:107)
at 
org.apache.hadoop.hbase.io.hfile.NoOpDataBlockEncoder.encode(NoOpDataBlockEncoder.java:56)
at 
org.apache.hadoop.hbase.io.hfile.HFileBlock$Writer.write(HFileBlock.java:954)
at 
org.apache.hadoop.hbase.io.hfile.HFileWriterV2.append(HFileWriterV2.java:284)
at 
org.apache.hadoop.hbase.io.hfile.HFileWriterV3.append(HFileWriterV3.java:87)
at 
org.apache.hadoop.hbase.regionserver.StoreFile$Writer.append(StoreFile.java:1041)
at 
org.apache.hadoop.hbase.regionserver.StoreFlusher.performFlush(StoreFlusher.java:138)
at 
org.apache.hadoop.hbase.regionserver.DefaultStoreFlusher.flushSnapshot(DefaultStoreFlusher.java:75)
at 
org.apache.hadoop.hbase.regionserver.HStore.flushCache(HStore.java:937)
at 
org.apache.hadoop.hbase.regionserver.HStore$StoreFlusherImpl.flushCache(HStore.java:2413)
at 
org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2456)
{noformat}
9. Regionserver aborts due to the failed flush
10. The regionserver WAL is split into recovered.edits files, one of these 
containing the same corrupted cell
11. A new regionserver is assigned the region with the corrupted write
12. The new regionserver replays the recovered.edits entries into memstore and 
then tries to flush the memstore to an HFile
13. The flush triggers the same IndexOutOfBoundsException, causing us to go 
back to step #8 and loop on repeat until manual intervention is taken

The corrupted cell basically becomes a poison pill that aborts regionservers 
one at a time as the region with the problem edit is passed around.  This also 
means that a malicious client could easily construct requests allowing a denial 
of service attack against regionservers hosting any tables that the client has 
write access to.

At bare minimum, I think we need to do a sanity check on all the lengths for 
Cells read off the CellScanner for incoming requests.  This would allow us to 
reject corrupt cells before we append them to the WAL and succeed the request, 
putting us in a position where we cannot recover.  This would only detect the 
corruption of length fields which puts us in a bad state.

Whether or not Cells should carry some checksum generated at the time the Cell 
is created, which could then validated on the server-side, is a separate 
question.  This would allow detection of other parts of the backing cell 
byte[], such as within the key fields or the value field.  But the computer 
overhead of this may be too heavyweight to be practical.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: [DISCUSS] Status of the 0.98 release line

[Gary Helmling]

+1 to EOL, and thanks to Andrew for all of the RM'ing.

On Mon, Apr 10, 2017 at 12:27 PM Ted Yu  wrote:

> +1
>
> Andrew has done tremendous work.
>
> On Mon, Apr 10, 2017 at 12:17 PM, Mikhail Antonov 
> wrote:
>
> > +1 to EOL 0.98.
> >
> > Thanks Andrew for all the work maintaining it!
> >
> > -Mikhail
> >
> > On Mon, Apr 10, 2017 at 12:10 PM, Dima Spivak 
> > wrote:
> >
> > > +1
> > >
> > > -Dima
> > >
> > > On Mon, Apr 10, 2017 at 12:08 PM, Stack  wrote:
> > >
> > > > I agree we should EOL 0.98.
> > > > St.Ack
> > > >
> > > > On Mon, Apr 10, 2017 at 11:43 AM, Andrew Purtell <
> apurt...@apache.org>
> > > > wrote:
> > > >
> > > > > Please speak up if it is incorrect to interpret the lack of
> responses
> > > as
> > > > > indicating consensus on declaring 0.98 EOL.
> > > > >
> > > > > I believe we should declare 0.98 EOL.
> > > > >
> > > > >
> > > > > On Wed, Mar 29, 2017 at 6:56 AM, Sean Busbey 
> > > wrote:
> > > > >
> > > > > > Hi Folks!
> > > > > >
> > > > > > Back in January our Andrew Purtell stepped down as the release
> > > > > > manager for the 0.98 release line.
> > > > > >
> > > > > > On the resultant dev@hbase thread[1] folks seemed largely in
> favor
> > > of
> > > > > > declaring end-of-maintenance for the 0.98 line.
> > > > > >
> > > > > > Now that it's been a couple of months, does anyone have concerns
> > > about
> > > > > > pushing forward on that?
> > > > > >
> > > > > > Do folks who listen on user@hbase but not dev@hbase have any
> > > concerns?
> > > > > >
> > > > > > As with any end-of-maintenance branch, the PMC would consider on
> a
> > > > > > case-by-case basis doing a future release of the branch should a
> > > > > > critical security vulnerability show up.
> > > > > >
> > > > > >
> > > > > > [1]: https://s.apache.org/DjCi
> > > > > >
> > > > > > -busbey
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Best regards,
> > > > >
> > > > >- Andy
> > > > >
> > > > > If you are given a choice, you believe you have acted freely. -
> > Raymond
> > > > > Teller (via Peter Watts)
> > > > >
> > > >
> > >
> >
> >
> >
> > --
> > Thanks,
> > Michael Antonov
> >
>

[jira] [Created] (HBASE-17884) Backport HBASE-16217 to branch-1

[Gary Helmling (JIRA)]

Gary Helmling created HBASE-17884:
-

 Summary: Backport HBASE-16217 to branch-1
 Key: HBASE-17884
 URL: https://issues.apache.org/jira/browse/HBASE-17884
 Project: HBase
  Issue Type: Sub-task
Reporter: Gary Helmling


The change to add calling user to ObserverContext in HBASE-16217 should also be 
applied to branch-1 to avoid use of UserGroupInformation.doAs() for access 
control checks.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (HBASE-17827) Client tools relying on AuthUtil.getAuthChore() break credential cache login

[Gary Helmling (JIRA)]

Gary Helmling created HBASE-17827:
-

 Summary: Client tools relying on AuthUtil.getAuthChore() break 
credential cache login
 Key: HBASE-17827
 URL: https://issues.apache.org/jira/browse/HBASE-17827
 Project: HBase
  Issue Type: Bug
  Components: canary, security
Reporter: Gary Helmling
Assignee: Gary Helmling


Client tools, such as Canary, which make use of keytab based logins with 
AuthUtil.getAuthChore() do not allow any way to continue without a keytab-based 
login when security is enabled.  Currently, when security is enabled and the 
configuration lacks {{hbase.client.keytab.file}}, these tools would fail with:

{noformat}
ERROR hbase.AuthUtil: Error while trying to perform the initial login: Running 
in secure mode, but config doesn't have a keytab
java.io.IOException: Running in secure mode, but config doesn't have a keytab
at org.apache.hadoop.security.SecurityUtil.login(SecurityUtil.java:239)
at 
org.apache.hadoop.hbase.security.User$SecureHadoopUser.login(User.java:420)
at org.apache.hadoop.hbase.security.User.login(User.java:258)
at 
org.apache.hadoop.hbase.security.UserProvider.login(UserProvider.java:197)
at org.apache.hadoop.hbase.AuthUtil.getAuthChore(AuthUtil.java:98)
at org.apache.hadoop.hbase.tool.Canary.run(Canary.java:589)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.hbase.tool.Canary.main(Canary.java:1327)
Exception in thread "main" java.io.IOException: Running in secure mode, but 
config doesn't have a keytab
at org.apache.hadoop.security.SecurityUtil.login(SecurityUtil.java:239)
at 
org.apache.hadoop.hbase.security.User$SecureHadoopUser.login(User.java:420)
at org.apache.hadoop.hbase.security.User.login(User.java:258)
at 
org.apache.hadoop.hbase.security.UserProvider.login(UserProvider.java:197)
at org.apache.hadoop.hbase.AuthUtil.getAuthChore(AuthUtil.java:98)
at org.apache.hadoop.hbase.tool.Canary.run(Canary.java:589)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.hbase.tool.Canary.main(Canary.java:1327)
{noformat}

These tools should still work with the default credential-cache login, at least 
when a client keytab is not configured.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (HBASE-12579) Move obtainAuthTokenForJob() methods out of User

[Gary Helmling (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-12579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gary Helmling resolved HBASE-12579.
---
Resolution: Duplicate

The methods were deprecated and the existing usage was removed as part of 
HBASE-12493.  I guess I left this open for the final removal of the deprecated 
methods from the next major release.  The removal was done as part of 
HBASE-14208.

> Move obtainAuthTokenForJob() methods out of User
> 
>
> Key: HBASE-12579
> URL: https://issues.apache.org/jira/browse/HBASE-12579
> Project: HBase
>  Issue Type: Improvement
>  Components: security
>Reporter: Gary Helmling
>
> The {{User}} class currently contains some utility methods to obtain HBase 
> authentication tokens for the given user.  However, these methods initiate an 
> RPC to the {{TokenProvider}} coprocessor endpoint, an action which should not 
> be part of the User class' responsibilities.
> This leads to a couple of problems:
> # The way the methods are currently structured, it is impossible to integrate 
> them with normal connection management for the cluster (the TokenUtil class 
> constructs its own HTable instance internally).
> # The User class is logically part of the hbase-common module, but uses the 
> TokenUtil class (part of hbase-server, though it should probably be moved to 
> hbase-client) through reflection, leading to a hidden dependency.
> The {{obtainAuthTokenForJob()}} methods should be deprecated and the process 
> of obtaining authentication tokens should be moved to use the normal 
> connection lifecycle.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (HBASE-17611) Thrift 2 per-call latency metrics are capped at ~ 2 seconds

[Gary Helmling (JIRA)]

Gary Helmling created HBASE-17611:
-

 Summary: Thrift 2 per-call latency metrics are capped at ~ 2 
seconds
 Key: HBASE-17611
 URL: https://issues.apache.org/jira/browse/HBASE-17611
 Project: HBase
  Issue Type: Bug
  Components: metrics, Thrift
Reporter: Gary Helmling
Assignee: Gary Helmling
 Fix For: 1.3.1


Thrift 2 latency metrics are measured in nanoseconds.  However, the duration 
used for per-method latencies is cast to an int, meaning the values are capped 
at 2.147 seconds.  Let's use a long instead.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (HBASE-17604) Backport HBASE-15437 (fix request and response size metrics) to branch-1

[Gary Helmling (JIRA)]

Gary Helmling created HBASE-17604:
-

 Summary: Backport HBASE-15437 (fix request and response size 
metrics) to branch-1
 Key: HBASE-17604
 URL: https://issues.apache.org/jira/browse/HBASE-17604
 Project: HBase
  Issue Type: Bug
  Components: IPC/RPC, metrics
Reporter: Gary Helmling


HBASE-15437 fixed request and response size metrics in master.  We should apply 
the same to branch-1 and related release branches.

Prior to HBASE-15437, request and response size metrics were only calculated 
based on the protobuf message serialized size.  This isn't correct when the 
cell scanner payload is in use.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (HBASE-17578) Thrift per-method metrics should still update in the case of exceptions

[Gary Helmling (JIRA)]

Gary Helmling created HBASE-17578:
-

 Summary: Thrift per-method metrics should still update in the case 
of exceptions
 Key: HBASE-17578
 URL: https://issues.apache.org/jira/browse/HBASE-17578
 Project: HBase
  Issue Type: Bug
  Components: Thrift
Reporter: Gary Helmling
Assignee: Gary Helmling
 Fix For: 1.3.1


Currently, the InvocationHandler used to update per-method metrics in the 
Thrift server fails to update metrics if an exception occurs.  This causes us 
to miss outliers.  We should include exceptional cases in per-method latencies, 
and also look at adding specific exception rate metrics.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: [DISCUSS] Re: Replication resiliency

[Gary Helmling]

Thanks for the feedback, all.  Makes sense to me.  I'll follow up in the
issue to use the same UEH to abort or at least same abort handling as other
cases (the current UEH used for replication source worker threads only
logs).


On Fri, Jan 27, 2017 at 11:27 AM Josh Elser <els...@apache.org> wrote:

Cool.

Let me open an issue to scan the codebase to see if we can find any
instances where we are starting threads which aren't using the UEH.

Andrew Purtell wrote:
> Agreed, let's abort with an abundance of caution. That is the _least_
that should be done when a thread dies unexpectedly. Maybe we can improve
resiliency for specific cases later.
>
>
> On Jan 26, 2017, at 5:53 PM, Enis Söztutar<enis@gmail.com>  wrote:
>
>>> Do we have worker threads that we can't safely continue without
>> indefinitely? Can we solve the general problem of "unhandled exception
>> in threads cause a RS Abort"?
>> We have this already in the code base. We are injecting an
>> UncaughtExceptionhandler (which is calling Abortable.abort) to almost all
>> of the HRegionServer service threads (see HRS.startServiceThreads). But
>> I've also seen cases where some important thread got unmarked. I think it
>> is good idea to revisit that and make sure that all the threads are
>> injected with the UEH.
>>
>> The replication source threads are started on demand, that is why the UEH
>> is not injected I think. But agreed that we should do the safe route
here,
>> and abort the regionserver.
>>
>> Enis
>>
>>> On Thu, Jan 26, 2017 at 2:19 PM, Josh Elser<els...@apache.org>  wrote:
>>>
>>> +1 If any "worker" thread can't safely/reasonably retry some unexpected
>>> exception without a reasonable expectation of self-healing, tank the RS.
>>>
>>> Having those threads die but not the RS could go uncaught for indefinite
>>> period of time.
>>>
>>>
>>> Sean Busbey wrote:
>>>
>>>> I've noticed a few other places where we can lose a worker thread and
>>>> the RegionServer happily continues. One notable example is the worker
>>>> threads that handle syncs for the WAL. I'm generally a
>>>> fail-fast-and-loud advocate, so I like aborting when things look
>>>> wonky. I've also had to deal with a lot of pain around silent and thus
>>>> hard to see replication failures, so strong signals that the
>>>> replication system is in a bad way sound good to me atm.
>>>>
>>>> Do we have worker threads that we can't safely continue without
>>>> indefinitely? Can we solve the general problem of "unhandled exception
>>>> in threads cause a RS Abort"?
>>>>
>>>> As mentioned on the jira, I do worry a bit about cluster stability and
>>>> cascading failures, given the ability to have user-provided endpoints
>>>> in the replication process. Ultimately, I don't see that as different
>>>> than all the other places coprocessors can put the cluster at risk.
>>>>
>>>>> On Thu, Jan 26, 2017 at 2:48 PM, Sean Busbey<bus...@apache.org>
 wrote:
>>>>>
>>>>> (edited subject to ensure folks filtering for DISCUSS see this)
>>>>>
>>>>>
>>>>>
>>>>> On Thu, Jan 26, 2017 at 1:58 PM, Gary Helmling<ghelml...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Over in HBASE-17381 there has been some discussion around whether an
>>>>>> unhandled exception in a ReplicationSourceWorkerThread should
trigger a
>>>>>> regionserver abort.
>>>>>>
>>>>>> The current behavior in the case of an unexpected exception in
>>>>>> ReplicationSourceWorkerThread.run() is to log a message and simply
let
>>>>>> the
>>>>>> thread die, allowing replication for this source to back up.
>>>>>>
>>>>>> I've seen this happen in an OOME scenario, which seems like a clear
case
>>>>>> where we would be better off aborting the regionserver.
>>>>>>
>>>>>> However, in the case of any other unexpected exceptions out of the
run()
>>>>>> method, how do we want to handle this?
>>>>>>
>>>>>> I'm of the general opinion that where we would be better off
aborting on
>>>>>> all unexpected exceptions, as it means that:
>>>>>> a) we have some missing error handling
>>>>>> b) failing fast raises visibility and makes it easier to add any
error
>>>>>> handling that should be there
>>>>>> c) silently stopping up replication creates problems that are
difficult
>>>>>> for
>>>>>> our users to identify operationally and hard to troubleshoot.
>>>>>>
>>>>>> However, the current behavior has been there for quite a while, and
>>>>>> maybe
>>>>>> there are other situations or concerns I'm not seeing which would
>>>>>> justify
>>>>>> having regionserver stability over replication stability.
>>>>>>
>>>>>> What are folks thoughts on this?  Should the regionserver abort on
all
>>>>>> unexpected exceptions in the run method or should we more narrowly
focus
>>>>>> this on OOME's?
>>>>>>

Replication resiliency

[Gary Helmling]

Over in HBASE-17381 there has been some discussion around whether an
unhandled exception in a ReplicationSourceWorkerThread should trigger a
regionserver abort.

The current behavior in the case of an unexpected exception in
ReplicationSourceWorkerThread.run() is to log a message and simply let the
thread die, allowing replication for this source to back up.

I've seen this happen in an OOME scenario, which seems like a clear case
where we would be better off aborting the regionserver.

However, in the case of any other unexpected exceptions out of the run()
method, how do we want to handle this?

I'm of the general opinion that where we would be better off aborting on
all unexpected exceptions, as it means that:
a) we have some missing error handling
b) failing fast raises visibility and makes it easier to add any error
handling that should be there
c) silently stopping up replication creates problems that are difficult for
our users to identify operationally and hard to troubleshoot.

However, the current behavior has been there for quite a while, and maybe
there are other situations or concerns I'm not seeing which would justify
having regionserver stability over replication stability.

What are folks thoughts on this?  Should the regionserver abort on all
unexpected exceptions in the run method or should we more narrowly focus
this on OOME's?

Re: [DISCUSS] JIRA component "hbase"

[Gary Helmling]

I don't think I've used that one, so maybe I'm missing something, but +1 on
deleting.  I don't see any commonality to the issues.

On Thu, Jan 26, 2017 at 8:02 AM Sean Busbey  wrote:

> Anyone know what the intended scope of the "hbase" component is in JIRA?
>
> Right now we have ~77 issues that list it. AFAICT it's been used since
> at least Dec 2014.
>
> Here's a JIRA query with them: https://s.apache.org/CqpC
>
> Spot checking them, I'm not sure what the component is trying to
> cover. I'd like to either update it to a better name (and give it a
> description) or just delete it.
>
> thoughts?
>

[jira] [Created] (HBASE-17381) ReplicationSourceWorkerThread can die due to unhandled exceptions

[Gary Helmling (JIRA)]

Gary Helmling created HBASE-17381:
-

 Summary: ReplicationSourceWorkerThread can die due to unhandled 
exceptions
 Key: HBASE-17381
 URL: https://issues.apache.org/jira/browse/HBASE-17381
 Project: HBase
  Issue Type: Bug
Reporter: Gary Helmling


If a ReplicationSourceWorkerThread encounters an unexpected exception in the 
run() method (for example failure to allocate direct memory for the DFS 
client), the exception will be logged by the UncaughtExceptionHandler, but the 
thread will also die and the replication queue will back up indefinitely until 
the Regionserver is restarted.

We should make sure the worker thread is resilient to all exceptions that it 
can actually handle.  For those that it really can't, it seems better to abort 
the regionserver rather than just allow replication to stop with minimal signal.

Here is a sample exception:

{noformat}
ERROR regionserver.ReplicationSource: Unexpected exception in 
ReplicationSourceWorkerThread, 
currentPath=hdfs://.../hbase/WALs/XXXwalfilenameXXX
java.lang.OutOfMemoryError: Direct buffer memory
at java.nio.Bits.reserveMemory(Bits.java:693)
at java.nio.DirectByteBuffer.(DirectByteBuffer.java:123)
at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311)
at 
org.apache.hadoop.crypto.CryptoOutputStream.(CryptoOutputStream.java:96)
at 
org.apache.hadoop.crypto.CryptoOutputStream.(CryptoOutputStream.java:113)
at 
org.apache.hadoop.crypto.CryptoOutputStream.(CryptoOutputStream.java:108)
at 
org.apache.hadoop.hdfs.protocol.datatransfer.sasl.DataTransferSaslUtil.createStreamPair(DataTransferSaslUtil.java:344)
at 
org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.doSaslHandshake(SaslDataTransferClient.java:490)
at 
org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.getSaslStreams(SaslDataTransferClient.java:391)
at 
org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.send(SaslDataTransferClient.java:263)
at 
org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.checkTrustAndSend(SaslDataTransferClient.java:211)
at 
org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.peerSend(SaslDataTransferClient.java:160)
at 
org.apache.hadoop.hdfs.net.TcpPeerServer.peerFromSocketAndKey(TcpPeerServer.java:92)
at org.apache.hadoop.hdfs.DFSClient.newConnectedPeer(DFSClient.java:3444)
at 
org.apache.hadoop.hdfs.BlockReaderFactory.nextTcpPeer(BlockReaderFactory.java:778)
at 
org.apache.hadoop.hdfs.BlockReaderFactory.getRemoteBlockReaderFromTcp(BlockReaderFactory.java:695)
at org.apache.hadoop.hdfs.BlockReaderFactory.build(BlockReaderFactory.java:356)
at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:673)
at 
org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:882)
at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:934)
at java.io.DataInputStream.read(DataInputStream.java:100)
at org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:308)
at org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:276)
at org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:264)
at org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:423)
at 
org.apache.hadoop.hbase.replication.regionserver.ReplicationWALReaderManager.openReader(ReplicationWALReaderManager.java:70)
at 
org.apache.hadoop.hbase.replication.regionserver.ReplicationSource$ReplicationSourceWorkerThread.openReader(ReplicationSource.java:830)
at 
org.apache.hadoop.hbase.replication.regionserver.ReplicationSource$ReplicationSourceWorkerThread.run(ReplicationSource.java:572)
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HBASE-16337) Removing peers seem to be leaving spare queues

[Gary Helmling (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gary Helmling resolved HBASE-16337.
---
Resolution: Duplicate

Closing as a dupe, thanks for pointing it out.

> Removing peers seem to be leaving spare queues
> --
>
> Key: HBASE-16337
> URL: https://issues.apache.org/jira/browse/HBASE-16337
> Project: HBase
>  Issue Type: Sub-task
>  Components: Replication
>Reporter: Joseph
>
> I have been running IntegrationTestReplication repeatedly with the backported 
> Replication Table changes. Every other iteration of the test fails with, but 
> these queues should have been deleted when we removed the peers. I believe 
> this may be related to HBASE-16096, HBASE-16208, or HBASE-16081.
> 16/08/02 08:36:07 ERROR util.AbstractHBaseTool: Error running command-line 
> tool
> org.apache.hadoop.hbase.replication.ReplicationException: undeleted queue for 
> peerId: TestPeer, replicator: 
> hbase4124.ash2.facebook.com,16020,1470150251042, queueId: TestPeer
>   at 
> org.apache.hadoop.hbase.replication.ReplicationPeersZKImpl.checkQueuesDeleted(ReplicationPeersZKImpl.java:544)
>   at 
> org.apache.hadoop.hbase.replication.ReplicationPeersZKImpl.addPeer(ReplicationPeersZKImpl.java:127)
>   at 
> org.apache.hadoop.hbase.client.replication.ReplicationAdmin.addPeer(ReplicationAdmin.java:200)
>   at 
> org.apache.hadoop.hbase.test.IntegrationTestReplication$VerifyReplicationLoop.setupTablesAndReplication(IntegrationTestReplication.java:239)
>   at 
> org.apache.hadoop.hbase.test.IntegrationTestReplication$VerifyReplicationLoop.run(IntegrationTestReplication.java:325)
>   at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>   at 
> org.apache.hadoop.hbase.test.IntegrationTestReplication.runTestFromCommandLine(IntegrationTestReplication.java:418)
>   at 
> org.apache.hadoop.hbase.IntegrationTestBase.doWork(IntegrationTestBase.java:134)
>   at 
> org.apache.hadoop.hbase.util.AbstractHBaseTool.run(AbstractHBaseTool.java:112)
>   at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>   at 
> org.apache.hadoop.hbase.test.IntegrationTestReplication.main(IntegrationTestReplication.java:424)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] No regions on Master node in 2.0

[Gary Helmling]

On Wed, Nov 16, 2016 at 11:36 AM Ted Yu  wrote:

> HBASE-16025 is currently marked Critical.
>
> I think it should be a blocker.
>
>
Done.

Re: [DISCUSS] No regions on Master node in 2.0

[Gary Helmling]

On Tue, Nov 15, 2016 at 11:44 AM  wrote:

> > In the absence of more information, intuition says master carries meta
> to avoid a whole class of problems.
> Off-hand I think the class of problems we'll eliminate are problems that
> are well understood and being constantly dealt with and hardened to this
> day (ie puts to a region).
> > I think we have to evaluate whether the new pv2 master works with
> remote meta updates and the fact that those updates can fail partially or
> succeed without theI think failing meta updates need to be dealt with
> either way AFAIK eventually procedure state will be stored in HDFS which is
> also a distributed system.
>
>
I don't think these are really equivalent.  If you encounter a bad DN in
the write pipeline, you can construct a new write pipeline with a new set
of DNs.  If you encounter an error updating meta, which other copy of meta
do you try to write to?

Also, I don't really see Pv2 as being able to solve the whole reliability
problem here.  If it leads to more reliable region assignments that's
great.  But if a region is assigned, but clients can't see it because meta
can't be updated, I don't think that's a big improvement.  The region's
data is offline from the client perspective until it can actually see the
new region location.

Re: [DISCUSS] No regions on Master node in 2.0

[Gary Helmling]

Only answer to the above observation is demonstration that ITBLL with meta
not on master is as robust as runs that have master carrying meta.


Agree that this is a prerequisite.  Another useful measure might be the
delay before an assignment under load is visible to clients.



>
> But making RPC work for assignment here is precisely the problem.  There's
> no reason master should have to contend with user requests to meta in
order
> to be able to make updates.  And until clients can actually see the
change,
> it doesn't really matter if the master state has been updated or not.
>
>
In hbase-2.0.0, there'll be a new regime. hbase:meta writing will be single
writer by master only. No more contention on writes. Regards contention
reading, this is unavoidable.



In hbase-2.0.0, only the final publishing step, what we want clients to
see, will update hbase:meta. All other transitions will be internal.


There is still contention between readers and writers over available
handler threads.  With a local meta region, you don't have assignment
manager having to contend for handler threads in order to perform writes.
This is huge for reliability.

Without meta on master, it has not been hard to reproduce scenarios where
HBase _cannot_ start up from a cold roll with high client traffic.  Region
assignments just can not complete because master has to compete with all
the clients attempting to read new region locations from meta and can't get
in the queue.  With meta on master, this goes away completely.


> >
> >
> > Above is fine except for the bit where we need to be able to field
reads.
> > Lets distribute the data to be read over the cluster rather than treat
> meta
> > reads with kid gloves hosted on a 'special' server; let these 'reads' be
> > like any other read the cluster takes (see next point)
> >
> >
> In my opinion, the real "special" part here is the master bit -- which I
> think we should be working to make less special and more just a normal bit
> of housekeeping spread across nodes -- not the regionserver role.  It only
> looks special right now because the evolution has stopped in the middle.
I
> really don't think enshrining master as a separate process is the right
way
> forward for us.
>
>
I always liked this notion.

To be worked out is how Master and hbase:meta hosting would interplay (The
RS that is designated Master would also host hbase:meta? Would it be
exclusively hosting hbase;meta or hbase:meta would move with Master
function [Stuff we've talked about before]).


I think meta should be tied to the master function for the reasons
described above. It's key that the updates to meta be local.  I don't think
that has to mean that only meta regions are hosted by the regionserver
serving as master.  There could be other user regions hosted as well, given
the server has adequate headroom to handle the master functions.



> > >
> > Is this just because meta had a dedicated server?
> >
> >
> I'm sure that having dedicated resources for meta helps.  But I don't
think
> that's sufficient.  The key is that master writes to meta are local, and
do
> not have to contend with the user requests to meta.
>
> It seems premature to be discussing dropping a working implementation
which
> eliminates painful parts of distributed consensus, until we have a
complete
> working alternative to evaluate.  Until then, why are we looking at
> features that are in use and work well?
>
>
>
How to move forward here? The Pv2 master is almost done. An ITBLL bakeoff
of new Pv2 based assign vs a Master that exclusively hosts hbase:meta?


I think that's a necessary test for proving out the new AM implementation.
But remember that we are comparing a feature which is actively supporting
production workloads with a line of active development.  I think there
should also be additional testing around situations of high meta load and
end-to-end assignment latency.

Re: Coprocessor metrics

[Gary Helmling]

>
>
> My current feeling is that external API should never include
> classes/interfaces which you don't "own". Re-building the API that
> already exists is pedantic, but I think it's a really good way to pay
> down the maintenance debt (whenever the next metrics library "hotness"
> takes off).
>
>
+1 to this.  I'd be very hesitant to tie ourselves too strongly to a
specific implementation, even if it is just copying an interface.

For coprocessors specifically, I think we can start with a limited API
exposing common metric types and evolve it from there.  But starting simple
seems key.

So #2 seems like the right approach to me.

Re: [DISCUSS] EOL 1.1 Release Branch

[Gary Helmling]

>
> I'm not deeply familiar with the AssignmentManager. I see when we process
> split rollbacks in onRegionSplit() we only call regionOffline() on
> daughters if they are known to exist. However when processing merge
> rollbacks in the else case of onRegionMerge() we unconditionally call
> regionOffline() on the parent-being-merged. Shouldn't that likewise be
> conditional on regionStates holding a state for the parent-being-merged?
> Pardon if I've missed something.
>
>
> I'm really not familiar with the merge code, but this seems plausible to
> me.  I see that onRegionSplit() has an early out at the top of the method,
> but that will fail to evaluate if rs_a and rs_b are open and rs_p is null.
> So if it's called with a code of MERGE_REVERTED, I think we could wind up
> creating an offline meta entry for rs_p with no regioninfo, similar to
> HBASE-16093.  And that entry could wind up hiding the (still online)
> daughter regions.
>

s/onRegionSplit()/onRegionMerge()/ in that comment.

Re: [DISCUSS] EOL 1.1 Release Branch

[Gary Helmling]

>
> I'm not deeply familiar with the AssignmentManager. I see when we process
> split rollbacks in onRegionSplit() we only call regionOffline() on
> daughters if they are known to exist. However when processing merge
> rollbacks in the else case of onRegionMerge() we unconditionally call
> regionOffline() on the parent-being-merged. Shouldn't that likewise be
> conditional on regionStates holding a state for the parent-being-merged?
> Pardon if I've missed something.
>
>
I'm really not familiar with the merge code, but this seems plausible to
me.  I see that onRegionSplit() has an early out at the top of the method,
but that will fail to evaluate if rs_a and rs_b are open and rs_p is null.
So if it's called with a code of MERGE_REVERTED, I think we could wind up
creating an offline meta entry for rs_p with no regioninfo, similar to
HBASE-16093.  And that entry could wind up hiding the (still online)
daughter regions.

[jira] [Created] (HBASE-16964) Successfully archived files are not cleared from compacted store file list if archiving of any file fails

[Gary Helmling (JIRA)]

Gary Helmling created HBASE-16964:
-

 Summary: Successfully archived files are not cleared from 
compacted store file list if archiving of any file fails
 Key: HBASE-16964
 URL: https://issues.apache.org/jira/browse/HBASE-16964
 Project: HBase
  Issue Type: Bug
  Components: regionserver
Reporter: Gary Helmling
Assignee: Gary Helmling
Priority: Blocker
 Fix For: 1.3.0


In HStore.removeCompactedFiles(), we only clear archived files from 
StoreFileManager's list of compactedfiles if _all_ files were archived 
successfully.  If we encounter an error archiving any of the files, then any 
files which were already archived will remain in the list of compactedfiles.

Even worse, this means that all subsequent attempts to archive the list of 
compacted files will fail (as the previously successfully archived files still 
in the list will now throw FileNotFoundException), and the list of 
compactedfiles will never be cleared from that point on.

Finally, when the region closes, we will again throw an exception out of 
HStore.removeCompactedFiles(), in this case causing a regionserver abort.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HBASE-16958) Balancer recomputes block distributions every time balanceCluster() runs

[Gary Helmling (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gary Helmling resolved HBASE-16958.
---
   Resolution: Duplicate
 Assignee: (was: Gary Helmling)
Fix Version/s: (was: 1.3.0)

I re-opened HBASE-16570 to fix the issue that is described here.

> Balancer recomputes block distributions every time balanceCluster() runs
> 
>
> Key: HBASE-16958
> URL: https://issues.apache.org/jira/browse/HBASE-16958
> Project: HBase
>  Issue Type: Bug
>  Components: Balancer
>Reporter: Gary Helmling
>
> The change in HBASE-16570 modified the balancer to compute block 
> distributions in parallel with a pool of 5 threads.  However, because it does 
> this every time Cluster is instantiated, it effectively bypasses the cache of 
> block locations added in HBASE-14473:
> In the LoadBalancer.balanceCluster() implementations (in 
> StochasticLoadBalancer, SimpleLoadBalancer), we create a new Cluster instance.
> In Cluster., we call registerRegion() on every HRegionInfo.
> In registerRegion(), we do the following:
> {code}
> regionLocationFutures.set(regionIndex,
> regionFinder.asyncGetBlockDistribution(region));
> {code}
> Then, back in Cluster. we do a get() on each ListenableFuture in a loop.
> So while we are doing the calls to get block locations in parallel with 5 
> threads, we're recomputing them every time balanceCluster() is called and not 
> taking advantage of the cache at all.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Reopened] (HBASE-16570) Compute region locality in parallel at startup

[Gary Helmling (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gary Helmling reopened HBASE-16570:
---

I've reverted this from branch-1.3 for the moment, until the issue that I 
described can be addressed.  I don't see where this would impact master startup 
time.  If we need to pre-initialize this on startup, let's do it in a 
background thread only on startup.  We need to make sure that locality is not 
recomputed on every run and that we use the cache instead.

> Compute region locality in parallel at startup
> --
>
> Key: HBASE-16570
> URL: https://issues.apache.org/jira/browse/HBASE-16570
> Project: HBase
>  Issue Type: Sub-task
>Reporter: binlijin
>Assignee: binlijin
> Fix For: 2.0.0, 1.4.0, 1.3.1
>
> Attachments: HBASE-16570-master_V1.patch, 
> HBASE-16570-master_V2.patch, HBASE-16570-master_V3.patch, 
> HBASE-16570-master_V4.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-16958) Balancer recomputes block distributions every time balanceCluster() runs

[Gary Helmling (JIRA)]

Gary Helmling created HBASE-16958:
-

 Summary: Balancer recomputes block distributions every time 
balanceCluster() runs
 Key: HBASE-16958
 URL: https://issues.apache.org/jira/browse/HBASE-16958
 Project: HBase
  Issue Type: Bug
  Components: Balancer
Reporter: Gary Helmling
Assignee: Gary Helmling
 Fix For: 1.3.0


The change in HBASE-16570 modified the balancer to compute block distributions 
in parallel with a pool of 5 threads.  However, because it does this every time 
Cluster is instantiated, it effectively bypasses the cache of block locations 
added in HBASE-14473:

In the LoadBalancer.balanceCluster() implementations (in 
StochasticLoadBalancer, SimpleLoadBalancer), we create a new Cluster instance.

In Cluster., we call registerRegion() on every HRegionInfo.

In registerRegion(), we do the following:
{code}
regionLocationFutures.set(regionIndex,
regionFinder.asyncGetBlockDistribution(region));
{code}

Then, back in Cluster. we do a get() on each ListenableFuture in a loop.

So while we are doing the calls to get block locations in parallel with 5 
threads, we're recomputing them every time balanceCluster() is called and not 
taking advantage of the cache at all.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [RESULT] Re: [VOTE] First release candidate for HBase 1.2.4 (RC0) is available

[Gary Helmling]

>
>
> Is there anything in particular folks are looking for before I roll the
> next RC?
>
> HBASE-16754, perhaps?
>

I plan on committing HBASE-16754 later today or tomorrow, so if you do roll
a new RC, I'd be happy to get it in.

But on 1.2, I believe the window of opportunity for this issue to occur is
much more narrow than on 1.3+.  So if you want to extend the vote on the
current RC and it goes out without the fix, that is okay by me as well.

Re: [VOTE] First release candidate for HBase 1.2.4 (RC0) is available

[Gary Helmling]

I just posted a patch for HBASE-16754 (dropped compaction markers during
log split), which might qualify as a sinker for the RC.  This issue was
introduced by the change in HBASE-15441 (which fixed another log splitting
issue), which went in to 1.2.1.  It's a relatively rare distributed race.
The impact is that at least compactions can't run until the region is
reassigned.  It looks like the code in 1.2 may be slightly different -- I
don't see TestWALSplit for one -- but I think the same issue will apply in
branch-1.2.  I'll look more closely at the impact for the RC, but wanted to
give a heads up.

On Mon, Oct 10, 2016 at 7:58 AM Sean Busbey  wrote:

> The first release candidate for HBase 1.2.4 is available for download at:
>
> https://dist.apache.org/repos/dist/dev/hbase/hbase-1.2.4RC0/
>
> Maven artifacts are also available in a staging repository at:
>
> https://repository.apache.org/content/repositories/orgapachehbase-1155/
>
> Artifacts are signed with my key (0D80DB7C) published in our KEYS
> file at http://www.apache.org/dist/hbase/KEYS
>
> The RC corresponds to the signed tag 1.2.4RC0, which currently points
> to commit ref
>
> 04bd0ec8e4cb2f39e5f787126f9b6645dabf27df
>
> The detailed source and binary compatibility report vs 1.2.3 has been
> published for your review, at:
>
>
> https://dist.apache.org/repos/dist/dev/hbase/hbase-1.2.4RC0/1.2.3_1.2.4RC0_compat_report.html
>
> HBase 1.2.4 is the fourth patch release in the HBase 1.2 line, continuing
> on
> the theme of bringing a stable, reliable database to the Hadoop and NoSQL
> communities. This release includes about 30 bug fixes since 1.2.3.
> Critical fixes include:
>
> * HBASE-16340 Xerces implementation jars removed (incompatible change)
> * HBASE-15984 Given failure to parse a given WAL that was closed
> cleanly, replay the WAL.
> * HBASE-16721 Concurrency issue in WAL unflushed seqId tracking
>
> The full list of fixes included in this release is available at:
>
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12310753=12338116
>
> and in the CHANGES.txt file included in the distribution.
>
> Please try out this candidate and vote +/-1 by 12:00 Pacific Time on
> Monday, 2016-10-17 as to whether we should release these artifacts
> as HBase 1.2.4.
>
> thanks!
> -busbey
>

[jira] [Resolved] (HBASE-16146) Counters are expensive...

[Gary Helmling (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gary Helmling resolved HBASE-16146.
---
   Resolution: Fixed
 Assignee: Gary Helmling
 Hadoop Flags: Reviewed
Fix Version/s: 1.4.0
   1.3.0
   2.0.0

Committed to branch-1.3, branch-1, and master.  Counter is no longer used in 
master, but still present as a deprecated class, so included for consistency.

Thanks, [~stack], [~mantonov], and [~enis] for reviews.

> Counters are expensive...
> -
>
> Key: HBASE-16146
> URL: https://issues.apache.org/jira/browse/HBASE-16146
> Project: HBase
>  Issue Type: Sub-task
>Reporter: stack
>Assignee: Gary Helmling
> Fix For: 2.0.0, 1.3.0, 1.4.0
>
> Attachments: HBASE-16146.001.patch, HBASE-16146.branch-1.001.patch, 
> HBASE-16146.branch-1.3.001.patch, counters.patch, less_and_less_counters.png
>
>
> Doing workloadc, perf shows 10%+ of CPU being spent on counter#add. If I 
> disable some of the hot ones -- see patch -- I can get 10% more throughput 
> (390k to 440k). Figure something better.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-16788) Race in compacted file deletion between HStore close() and closeAndArchiveCompactedFiles()

[Gary Helmling (JIRA)]

Gary Helmling created HBASE-16788:
-

 Summary: Race in compacted file deletion between HStore close() 
and closeAndArchiveCompactedFiles()
 Key: HBASE-16788
 URL: https://issues.apache.org/jira/browse/HBASE-16788
 Project: HBase
  Issue Type: Bug
  Components: regionserver
Affects Versions: 1.3.0
Reporter: Gary Helmling
Assignee: Gary Helmling
Priority: Blocker


HBASE-13082 changed the way that compacted files are archived from being done 
inline on compaction completion to an async cleanup by the 
CompactedHFilesDischarger chore.  It looks like the changes to HStore to 
support this introduced a race condition in the compacted HFile archiving.

In the following sequence, we can wind up with two separate threads trying to 
archive the same HFiles, causing a regionserver abort:

# compaction completes normally and the compacted files are added to 
{{compactedfiles}} in HStore's DefaultStoreFileManager
# *threadA*: CompactedHFilesDischargeHandler runs in a RS executor service, 
calling closeAndArchiveCompactedFiles()
## obtains HStore readlock
## gets a copy of compactedfiles
## releases readlock
# *threadB*: calls HStore.close() as part of region close
## obtains HStore writelock
## calls DefaultStoreFileManager.clearCompactedfiles(), getting a copy of same 
compactedfiles
# *threadA*: calls HStore.removeCompactedfiles(compactedfiles)
## archives files in {compactedfiles} in HRegionFileSystem.removeStoreFiles()
## call HStore.clearCompactedFiles()
## waits on write lock
# *threadB*: continues with close()
## calls removeCompactedfiles(compactedfiles)
## calls HRegionFIleSystem.removeStoreFiles() -> 
HFileArchiver.archiveStoreFiles()
## receives FileNotFoundException because the files have already been archived 
by threadA
## throws IOException
# RS aborts

I think the combination of fetching the compactedfiles list and removing the 
files needs to be covered by locking.  Options I see are:
* Modify HStore.closeAndArchiveCompactedFiles(): use writelock instead of 
readlock and move the call to removeCompactedfiles() inside the lock.  This 
means the read operations will be blocked while the files are being archived, 
which is bad.
* Synchronize closeAndArchiveCompactedFiles() and modify close() to call it 
instead of calling removeCompactedfiles() directly
* Add a separate lock for compacted files removal and use in 
closeAndArchiveCompactedFiles() and close()





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-16754) Regions failing compaction due to referencing non-existent store file

[Gary Helmling (JIRA)]

Gary Helmling created HBASE-16754:
-

 Summary: Regions failing compaction due to referencing 
non-existent store file
 Key: HBASE-16754
 URL: https://issues.apache.org/jira/browse/HBASE-16754
 Project: HBase
  Issue Type: Bug
Reporter: Gary Helmling
Priority: Blocker
 Fix For: 1.3.0


Running a mixed read write workload on a recent build off branch-1.3, we are 
seeing compactions occasionally fail with errors like the following (actual 
filenames replaced with placeholders):

{noformat}
16/09/27 16:57:28 ERROR regionserver.CompactSplitThread: Compaction selection 
failed Store = XXX, pri = 116
java.io.FileNotFoundException: File does not exist: 
hdfs://.../hbase/data/ns/table/region/cf/XXfilenameXX
at 
org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1309)
at 
org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1301)
at 
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at 
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1317)
at 
org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:421)  
at 
org.apache.hadoop.hbase.regionserver.StoreFileInfo.getReferencedFileStatus(StoreFileInfo.java:342)
at 
org.apache.hadoop.hbase.regionserver.StoreFileInfo.getFileStatus(StoreFileInfo.java:355)
  
at 
org.apache.hadoop.hbase.regionserver.StoreFileInfo.getModificationTime(StoreFileInfo.java:360)
at 
org.apache.hadoop.hbase.regionserver.StoreFile.getModificationTimeStamp(StoreFile.java:321)
  
at 
org.apache.hadoop.hbase.regionserver.StoreUtils.getLowestTimestamp(StoreUtils.java:63)
at 
org.apache.hadoop.hbase.regionserver.compactions.RatioBasedCompactionPolicy.shouldPerformMajorCompaction(RatioBasedCompactionPolicy.java:63)
at 
org.apache.hadoop.hbase.regionserver.compactions.SortedCompactionPolicy.selectCompaction(SortedCompactionPolicy.java:82)
  
at 
org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.select(DefaultStoreEngine.java:107)
  
at 
org.apache.hadoop.hbase.regionserver.HStore.requestCompaction(HStore.java:1644)
at 
org.apache.hadoop.hbase.regionserver.CompactSplitThread.selectCompaction(CompactSplitThread.java:373)
at 
org.apache.hadoop.hbase.regionserver.CompactSplitThread.access$100(CompactSplitThread.java:59)
at 
org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.doCompaction(CompactSplitThread.java:498)
at 
org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.run(CompactSplitThread.java:568)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
16/09/27 17:01:31 ERROR regionserver.CompactSplitThread: Compaction selection 
failed Store = XXX, pri = 115
java.io.FileNotFoundException: File does not exist: 
hdfs://.../hbase/data/ns/table/region/cf/XXfilenameXX
at 
org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1309)
at 
org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1301)
at 
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at 
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1317)
at 
org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:421)  
at 
org.apache.hadoop.hbase.regionserver.StoreFileInfo.getReferencedFileStatus(StoreFileInfo.java:342)
at 
org.apache.hadoop.hbase.regionserver.StoreFileInfo.getFileStatus(StoreFileInfo.java:355)
  
at 
org.apache.hadoop.hbase.regionserver.StoreFileInfo.getModificationTime(StoreFileInfo.java:360)
at 
org.apache.hadoop.hbase.regionserver.StoreFile.getModificationTimeStamp(StoreFile.java:321)
  
at 
org.apache.hadoop.hbase.regionserver.StoreUtils.getLowestTimestamp(StoreUtils.java:63)
at 
org.apache.hadoop.hbase.regionserver.compactions.RatioBasedCompactionPolicy.shouldPerformMajorCompaction(RatioBasedCompactionPolicy.java:63)
at 
org.apache.hadoop.hbase.regionserver.compactions.SortedCompactionPolicy.selectCompaction(SortedCompactionPolicy.java:82)
  
at 
org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.select(DefaultStoreEngine.java:107)
  
at 
org.apache.hadoop.hbase.regionserver.HStore.requestCompaction(HStore.java:1644)
at 
org.apache.hadoop.hbase.regionserver.CompactSplitThread.selectCompaction(CompactSplitThread.java:373

[jira] [Created] (HBASE-16661) Add last major compaction age to per-region metrics

[Gary Helmling (JIRA)]

Gary Helmling created HBASE-16661:
-

 Summary: Add last major compaction age to per-region metrics
 Key: HBASE-16661
 URL: https://issues.apache.org/jira/browse/HBASE-16661
 Project: HBase
  Issue Type: Improvement
Reporter: Gary Helmling
Priority: Minor


After HBASE-12859, we can now track the last major compaction timestamp for 
each region.  However, this is only exposed through cluster status reporting 
and the admin API.

We have similar per-region metrics around storefile age, but none that filters 
on major compaction specifically.

Let's add a metric for last major compaction age.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-16657) Expose per-region last major compaction timestamp in RegionServer UI

[Gary Helmling (JIRA)]

Gary Helmling created HBASE-16657:
-

 Summary: Expose per-region last major compaction timestamp in 
RegionServer UI
 Key: HBASE-16657
 URL: https://issues.apache.org/jira/browse/HBASE-16657
 Project: HBase
  Issue Type: Improvement
  Components: regionserver, UI
Reporter: Gary Helmling


HBASE-12859 added some tracking for the last major compaction completed for 
each region.  However, this is currently only exposed through the cluster 
status reporting and the Admin API.  Since the regionserver is already 
reporting this information, it would be nice to fold it in somewhere to the 
region listing in the regionserver UI.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Consider adding the issue number in commit message

[Gary Helmling]

>
> Please never force push to any non-feature branch. I thought we had
> protections to stop force pushes on master and
> have filed INFRA-12602 to get them in place.
>
>
Yeah, I shouldn't have done it.  But I think the protection against force
pushes only applies to the "rel/*" namespace.


> To fix erroneous commit messages, please revert the offending commits
> and then reapply them with a correct commit message.
>
>
Honestly, I don't see the point of this.  In this case the original commit
is still there, so nothing is really fixed.  Instead we wind up with 3
commits muddying up the change history for the affected files.

I would much rather preserve a clean change history at the cost of a few
bad commit messages.  I don't think it's really that big a deal.

Re: Consider adding the issue number in commit message

[Gary Helmling]

Yes  this is one commit message where the JIRA number was left off.  I only
noticed it was missing after I pushed to master.  Mea culpa.

I've rebased to amend the commit message and force pushed.  However, there
were two intervening commits in the meantime, which also got included in
the rebase:

commit 77b327320a72ca01b35f655c42f8c13f659dff31
Author: anoopsamjohn 
Date:   Tue Sep 13 11:43:26 2016 +0530

HBASE-16229 Cleaning up size and heapSize calculation.

commit 2566cfeb60de644f287ac192d360f3fc15376c8f
Author: chenheng 
Date:   Tue Sep 13 10:07:45 2016 +0800

HBASE-16592 Unify Delete request with AP


I like a clear commit history as much as anyone, but let's consider the
consequences of "fixing" this.  Unfortunately anyone who pulled master
between the original commit and the fix may now need to re-checkout master
to avoid divergent commits.  In hindsight, I think the fix may create more
pain than the original problem.  My apologies if you have to re-checkout
master as a result.

Anoop and Heng, you will both likely fall into this camp.  Again, my
apologies.

This reaffirms my prejudice against force pushing to shared branches.


On Mon, Sep 12, 2016 at 6:24 PM Duo Zhang  wrote:

>
> https://github.com/apache/hbase/commit/1cdc5acfd456688f106287c19ae9af62fd03a3af
>
> I can not find the related issue of this commit...
>
> Thanks.
>

[jira] [Created] (HBASE-16540) Scan should do additional validation on start and stop row

[Gary Helmling (JIRA)]

Gary Helmling created HBASE-16540:
-

 Summary: Scan should do additional validation on start and stop row
 Key: HBASE-16540
 URL: https://issues.apache.org/jira/browse/HBASE-16540
 Project: HBase
  Issue Type: Bug
  Components: Client
Reporter: Gary Helmling


Scan.setStartRow() and setStopRow() should validate the byte[] passed to ensure 
it meets the criteria for a row key.  If the byte[] length is greater that 
Short.MAX_VALUE, we should throw an IllegalArgumentException in order to fast 
fail and prevent server-side errors being thrown and retried.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-16518) Remove old .arcconfig file

[Gary Helmling (JIRA)]

Gary Helmling created HBASE-16518:
-

 Summary: Remove old .arcconfig file
 Key: HBASE-16518
 URL: https://issues.apache.org/jira/browse/HBASE-16518
 Project: HBase
  Issue Type: Task
  Components: tooling
Reporter: Gary Helmling
Assignee: Gary Helmling
Priority: Trivial


The project .arcconfig file points to a project that no longer exists on a no 
longer supported phabricator instance.  Since it is no longer used for reviews, 
let's drop it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-16277) Improve CPU efficiency in VisibilityLabelsCache

[Gary Helmling (JIRA)]

Gary Helmling created HBASE-16277:
-

 Summary: Improve CPU efficiency in VisibilityLabelsCache
 Key: HBASE-16277
 URL: https://issues.apache.org/jira/browse/HBASE-16277
 Project: HBase
  Issue Type: Improvement
  Components: security
Reporter: Gary Helmling


For secure clusters where the VisibilityController coprocessor is loaded, 
regionservers sometimes degrade into very high CPU utilization, with many of 
the RPC handler threads stuck in:

{noformat}
"B.defaultRpcServer.handler=0,queue=0,port=16020" #114 daemon prio=5 os_prio=0 
tid=0x7f8a95bb7800 nid=0x382 runnable [0x7f8a3051f000]
   java.lang.Thread.State: RUNNABLE
at 
java.lang.ThreadLocal$ThreadLocalMap.expungeStaleEntry(ThreadLocal.java:617)
at java.lang.ThreadLocal$ThreadLocalMap.remove(ThreadLocal.java:499)
at java.lang.ThreadLocal$ThreadLocalMap.access$200(ThreadLocal.java:298)
at java.lang.ThreadLocal.remove(ThreadLocal.java:222)
at 
java.util.concurrent.locks.ReentrantReadWriteLock$Sync.tryReleaseShared(ReentrantReadWriteLock.java:426)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.releaseShared(AbstractQueuedSynchronizer.java:1341)
at 
java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.unlock(ReentrantReadWriteLock.java:881)
at 
org.apache.hadoop.hbase.security.visibility.VisibilityLabelsCache.getGroupAuths(VisibilityLabelsCache.java:237)
at 
org.apache.hadoop.hbase.security.visibility.FeedUserAuthScanLabelGenerator.getLabels(FeedUserAuthScanLabelGenerator.java:70)
at 
org.apache.hadoop.hbase.security.visibility.DefaultVisibilityLabelServiceImpl.getVisibilityExpEvaluator(DefaultVisibilityLabelServiceImpl.java:469)
at 
org.apache.hadoop.hbase.security.visibility.VisibilityUtils.createVisibilityLabelFilter(VisibilityUtils.java:284)
at 
org.apache.hadoop.hbase.security.visibility.VisibilityController.preGetOp(VisibilityController.java:684)
at 
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$26.call(RegionCoprocessorHost.java:849)
at 
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1673)
at 
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1749)
at 
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1705)
at 
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preGet(RegionCoprocessorHost.java:845)
at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:6748)
at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:6736)
at 
org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2029)
at 
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:33644)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2170)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:109)
at 
org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:137)
at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:112)
at java.lang.Thread.run(Thread.java:745)
{noformat}

In this case there are no visibility labels actually in use, so it appears that 
the locking overhead for the VisibilityLabelsCache can reach a tipping point 
where it does not degrade gracefully.

We should look at alternate approaches to the label caching in place of the 
current ReentrantReadWriteLock.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-16231) Integration tests should support client keytab login for secure clusters

[Gary Helmling (JIRA)]

Gary Helmling created HBASE-16231:
-

 Summary: Integration tests should support client keytab login for 
secure clusters
 Key: HBASE-16231
 URL: https://issues.apache.org/jira/browse/HBASE-16231
 Project: HBase
  Issue Type: Improvement
  Components: integration tests
Reporter: Gary Helmling
Assignee: Gary Helmling


Integration tests currently rely on an external kerberos login for secure 
clusters.  Elsewhere we use AuthUtil to login and refresh the credentials in a 
background thread.  We should do the same here.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-16218) Eliminate use of UGI.doAs() in AccessController testing

[Gary Helmling (JIRA)]

Gary Helmling created HBASE-16218:
-

 Summary: Eliminate use of UGI.doAs() in AccessController testing
 Key: HBASE-16218
 URL: https://issues.apache.org/jira/browse/HBASE-16218
 Project: HBase
  Issue Type: Sub-task
  Components: security
Reporter: Gary Helmling
Assignee: Gary Helmling


Many tests for AccessController observer coprocessor hooks make use of 
UGI.doAs() when the test user could simply be passed through.  Eliminate the 
unnecessary use of doAs().



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-16217) Identify calling user in ObserverContext

[Gary Helmling (JIRA)]

Gary Helmling created HBASE-16217:
-

 Summary: Identify calling user in ObserverContext
 Key: HBASE-16217
 URL: https://issues.apache.org/jira/browse/HBASE-16217
 Project: HBase
  Issue Type: Sub-task
  Components: Coprocessors, security
Reporter: Gary Helmling
Assignee: Gary Helmling


We already either explicitly pass down the relevant User instance initiating an 
action through the call path, or it is available through 
RpcServer.getRequestUser().  We should carry this through in the 
ObserverContext for coprocessor upcalls and make use of it for permissions 
checking.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HBASE-16202) Backport metric for CallQueueTooBigException to 1.3

[Gary Helmling (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gary Helmling resolved HBASE-16202.
---
Resolution: Invalid

No backport needed, I just did a simple cherry-pick to branch-1.3.

> Backport metric for CallQueueTooBigException to 1.3
> ---
>
> Key: HBASE-16202
> URL: https://issues.apache.org/jira/browse/HBASE-16202
> Project: HBase
>  Issue Type: Improvement
>  Components: IPC/RPC, metrics
>Reporter: Gary Helmling
>    Assignee: Gary Helmling
>
> HBASE-15353 added a separate metric for tracking the number of 
> CallQueueTooBigExceptions, but only went in to 1.4+.  Since CQTBE is already 
> in 1.2+, it would be nice to at least get this in the upcoming 1.3.0 release.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-16202) Backport metric for CallQueueTooBigException to 1.3

[Gary Helmling (JIRA)]

Gary Helmling created HBASE-16202:
-

 Summary: Backport metric for CallQueueTooBigException to 1.3
 Key: HBASE-16202
 URL: https://issues.apache.org/jira/browse/HBASE-16202
 Project: HBase
  Issue Type: Improvement
  Components: IPC/RPC, metrics
Reporter: Gary Helmling
Assignee: Gary Helmling


HBASE-15353 added a separate metric for tracking the number of 
CallQueueTooBigExceptions, but only went in to 1.4+.  Since CQTBE is already in 
1.2+, it would be nice to at least get this in the upcoming 1.3.0 release.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-16141) Unwind use of UserGroupInformation.doAs() to convey requester identity in coprocessor upcalls

[Gary Helmling (JIRA)]

Gary Helmling created HBASE-16141:
-

 Summary: Unwind use of UserGroupInformation.doAs() to convey 
requester identity in coprocessor upcalls
 Key: HBASE-16141
 URL: https://issues.apache.org/jira/browse/HBASE-16141
 Project: HBase
  Issue Type: Improvement
  Components: Coprocessors, security
Reporter: Gary Helmling
Assignee: Gary Helmling
 Fix For: 2.0.0, 1.4.0


In discussion on HBASE-16115, there is some discussion of whether 
UserGroupInformation.doAs() is the right mechanism for propagating the original 
requester's identify in certain system contexts (splits, compactions, some 
procedure calls).  It has the unfortunately of overriding the current user, 
which makes for very confusing semantics for coprocessor implementors.  We 
should instead find an alternate mechanism for conveying the caller identity, 
which does not override the current user context.

I think we should instead look at passing this through as part of the 
ObserverContext passed to every coprocessor hook.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-16097) Flushes and compactions fail on getting split point

[Gary Helmling (JIRA)]

Gary Helmling created HBASE-16097:
-

 Summary: Flushes and compactions fail on getting split point
 Key: HBASE-16097
 URL: https://issues.apache.org/jira/browse/HBASE-16097
 Project: HBase
  Issue Type: Bug
  Components: Compaction
Affects Versions: 1.2.1
Reporter: Gary Helmling
Assignee: Gary Helmling


We've seen a number of cases where flushes and compactions run, completely 
through, then throw an IndexOutOfBoundsException when getting the split point 
when checking if a split is needed.

For flushes, the stack trace looks something like:
{noformat}
ERROR regionserver.MemStoreFlusher: Cache flusher failed for entry [flush 
region ]
java.lang.IndexOutOfBoundsException: 131148
at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:139)
at 
org.apache.hadoop.hbase.util.ByteBufferUtils.toBytes(ByteBufferUtils.java:491)
at 
org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.midkey(HFileBlockIndex.java:351)
at 
org.apache.hadoop.hbase.io.hfile.HFileReaderV2.midkey(HFileReaderV2.java:520)
at 
org.apache.hadoop.hbase.regionserver.StoreFile$Reader.midkey(StoreFile.java:1510)
at 
org.apache.hadoop.hbase.regionserver.StoreFile.getFileSplitPoint(StoreFile.java:726)
at 
org.apache.hadoop.hbase.regionserver.DefaultStoreFileManager.getSplitPoint(DefaultStoreFileManager.java:127)
at 
org.apache.hadoop.hbase.regionserver.HStore.getSplitPoint(HStore.java:2036)
at 
org.apache.hadoop.hbase.regionserver.RegionSplitPolicy.getSplitPoint(RegionSplitPolicy.java:82)
at 
org.apache.hadoop.hbase.regionserver.HRegion.checkSplit(HRegion.java:7885)
at 
org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:513)
at 
org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:471)
at 
org.apache.hadoop.hbase.regionserver.MemStoreFlusher.access$900(MemStoreFlusher.java:75)
at 
org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:259)
at java.lang.Thread.run(Thread.java:745)
{noformat}

For compactions, the exception occurs in the same spot:
{noformat}
ERROR regionserver.CompactSplitThread: Compaction failed Request = 
regionName=X, storeName=X, fileCount=XX, fileSize=XXX M, priority=1, time=
java.lang.IndexOutOfBoundsException
at java.nio.Buffer.checkIndex(Buffer.java:540)
at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:139)
at 
org.apache.hadoop.hbase.util.ByteBufferUtils.toBytes(ByteBufferUtils.java:491)
at 
org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.midkey(HFileBlockIndex.java:351)
at 
org.apache.hadoop.hbase.io.hfile.HFileReaderV2.midkey(HFileReaderV2.java:520)
at 
org.apache.hadoop.hbase.regionserver.StoreFile$Reader.midkey(StoreFile.java:1510)
at 
org.apache.hadoop.hbase.regionserver.StoreFile.getFileSplitPoint(StoreFile.java:726)
at 
org.apache.hadoop.hbase.regionserver.DefaultStoreFileManager.getSplitPoint(DefaultStoreFileManager.java:127)
at 
org.apache.hadoop.hbase.regionserver.HStore.getSplitPoint(HStore.java:2036)
at 
org.apache.hadoop.hbase.regionserver.RegionSplitPolicy.getSplitPoint(RegionSplitPolicy.java:82)
at 
org.apache.hadoop.hbase.regionserver.HRegion.checkSplit(HRegion.java:7885)
at 
org.apache.hadoop.hbase.regionserver.CompactSplitThread.requestSplit(CompactSplitThread.java:241)
at 
org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.doCompaction(CompactSplitThread.java:540)
at 
org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.run(CompactSplitThread.java:566)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
{noformat}

This continues until a compaction runs through and rewrites whatever file is 
causing the problem, at which point a split can proceed successfully.

While compactions and flushes are successfully completing up until this point 
(it occurs after new store files have been moved into place), the exception 
thrown on flush causes us to exit prior to checking if a compaction is needed.  
So normal compactions wind up not being triggered and the effected regions 
accumulate a large number of store files.

No root cause yet, so I'm parking this info here for investigation.  Seems like 
we're either mis-writing part of the index or making some bad assumptions on 
the index blocks that we've read.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-16025) Cache table state to reduce load on META

[Gary Helmling (JIRA)]

Gary Helmling created HBASE-16025:
-

 Summary: Cache table state to reduce load on META
 Key: HBASE-16025
 URL: https://issues.apache.org/jira/browse/HBASE-16025
 Project: HBase
  Issue Type: Improvement
  Components: Client
Reporter: Gary Helmling
Priority: Critical
 Fix For: 2.0.0


HBASE-12035 moved keeping table enabled/disabled state from ZooKeeper into 
hbase:meta.  When we retry operations on the client, we check table state in 
order to return a specific message if the table is disabled.  This means that 
in master we will be going back to meta for every retry, even if a region's 
location has not changed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Adding JIRA contributors

[Gary Helmling]

Thanks for the tip.  I had tried the copy/paste but was still getting
tricked by the error message.  Clicking through that and going to "Update"
worked.

I'm guessing that for the contributor list and user info JIRA is using it's
own user db.  It seems kind of like wasted effort to have each project
duplicating the management of "non-spammer" validation by adding to a
contributor group.  I'm wondering if that could be handled through a shared
group across all projects instead.  Or alternately of course doing the spam
filtering a different way.

Anyone have a pointer to a JIRA where the long term spam solution is being
discussed?


On Fri, May 27, 2016 at 9:07 AM Sean Busbey <bus...@apache.org> wrote:

> Yes. If you paste in the full JIRA username  you'll get the "can't contact
> server" message, but it will just be from the autocomplete. If you close it
> and submit anyways, the user will be added.
>
> Using the rest API to add folks also works, but it's obviously more
> overhead.
> On May 27, 2016 07:43, "Stack" <st...@duboce.net> wrote:
>
> > On Fri, May 27, 2016 at 12:00 AM, Andrew Purtell <
> andrew.purt...@gmail.com
> > >
> > wrote:
> >
> > > This is a known problem.
> > >
> > > We solved this before by adding the 'jira-users' group to the
> Contributor
> > > role. However I had to recently remove it because it allows the
> spammers
> > > attacking Apache's JIRA to open issues with any new user they've signed
> > up
> > > for the purpose.
> > >
> > > Infrastructure is working on a long term solution for the problem we
> are
> > > having with spam attack on our JIRA instance. I don't have an ETA.
> > >
> > > One thing we can try is removing folks from the current Contributors
> list
> > > who haven't been active for a while to make room for new entries.
> > >
> > >
> > >
> > I have been doing the above.
> >
> > Adding the user's JIRA name seems to work though JIRA says it does not
> > (trick from Busbey).
> >
> > St.Ack
> >
> >
> >
> > > > On May 26, 2016, at 10:48 PM, Gary Helmling <ghelml...@gmail.com>
> > wrote:
> > > >
> > > > I can't seem to add new contributors to JIRA.  Going to the admin
> page
> > > and
> > > > editing the members of the contributors role, JIRA just displays the
> > > > message "The JIRA server could not be contacted".  This shows up for
> > > > type-ahead or just trying to paste in the username.
> > > >
> > > > Is anyone else able to make this work?
> > >
> >
>

Adding JIRA contributors

[Gary Helmling]

I can't seem to add new contributors to JIRA.  Going to the admin page and
editing the members of the contributors role, JIRA just displays the
message "The JIRA server could not be contacted".  This shows up for
type-ahead or just trying to paste in the username.

Is anyone else able to make this work?

Re: [ANNOUNCE] Mikhail Antonov joins the Apache HBase PMC

[Gary Helmling]

Welcome Mikhail!

On Thu, May 26, 2016 at 11:47 AM Ted Yu  wrote:

> Congratulations, Mikhail !
>
> On Thu, May 26, 2016 at 11:30 AM, Andrew Purtell 
> wrote:
>
> > On behalf of the Apache HBase PMC I am pleased to announce that Mikhail
> > Antonov has accepted our invitation to become a PMC member on the Apache
> > HBase project. Mikhail has been an active contributor in many areas,
> > including recently taking on the Release Manager role for the upcoming
> > 1.3.x code line. Please join me in thanking Mikhail for his contributions
> > to date and anticipation of many more contributions.
> >
> > Welcome to the PMC, Mikhail!
> >
> > --
> > Best regards,
> >
> >- Andy
> >
> > Problems worthy of attack prove their worth by hitting back. - Piet Hein
> > (via Tom White)
> >
>

[jira] [Created] (HBASE-15856) Cached Connection instances can wind up with addresses never resolved

[Gary Helmling (JIRA)]

Gary Helmling created HBASE-15856:
-

 Summary: Cached Connection instances can wind up with addresses 
never resolved
 Key: HBASE-15856
 URL: https://issues.apache.org/jira/browse/HBASE-15856
 Project: HBase
  Issue Type: Bug
  Components: Client
Reporter: Gary Helmling
Assignee: Gary Helmling
Priority: Critical


During periods where DNS is not working properly, we can wind up caching 
connections to master or regionservers where the initial hostname resolution 
and the resolution is never re-attempted.  This means that clients will forever 
get UnknownHostException for any calls.

When constructing a BlockingRpcChannelImplementation, we instantiate the 
InetSocketAddress to use for the connection.  This instance is then used in the 
rpc client connection, where we check isUnresolved() and throw an 
UnknownHostException if that returns true.  However, at this point the rpc 
channel is already cached in the HConnectionImplementation map of stubs.  So at 
this point it will never be resolved.

Setting the config for hbase.resolve.hostnames.on.failure masks this issue, 
since the stub key used is modified to contain the address.  However, even in 
that case, if DNS fails, an rpc channel instance with unresolved ISA will still 
be cached in the stubs under the hostname only key.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] Make AsyncFSWAL the default WAL in 2.0

[Gary Helmling]

>
>
> I was trying to avoid the below oft-repeated pattern at least for the case
> of critical developments:
>
> + New feature arrives after much work by developer, reviewers and testers
> accompanied by fanfare (blog, talks).
> + Developers and reviewers move on after getting it committed or it gets
> hacked into a deploy so it works in a frankenstein form
> + It sits in our code base across one or more releases marked as optional,
> 'experimental'
> + The 'experimental' bleamish discourages its exercise by users
> + The feature lags, rots
> + Or, the odd time, we go ahead and enable it as default in spite of the
> fact it was never tried when experimental.
>
> Distributed Log Replay sat in hbase across a few major versions. Only when
> the threat of our making an actual release with it on by default did it get
> serious attention where it was found flawed and is now being actively
> purged. This was after it made it past reviews, multiple attempts at
> testing at scale, and so on; i.e. we'd done it all by the book. The time in
> an 'experimental' state added nothing.
>
>
Those are all valid concerns as well. It's certainly a pattern that we've
seen repeated. That's also a broader concern I have about the farther we
push out 2.0, then the less exercised master is.

I don't really know how best to balance this with concerns about user
stability.  Enabling by default in master would certainly be a forcing
function and would help it get more testing before release.  I hear that
argument.  But I'm worried about the impact after release, where something
as simple as a bug-fix point release upgrade of Hadoop could result in
runtime breakage of an HBase install.  Will this happen in practice?  I
don't know.  It seems unlikely that the private variable names being used
for example would change in a point release.  But we're violating the
abstraction that Hadoop provides us which guarantees such breakage won't
occur.


> > Yes. 2.0 is a bit out there so we have some time to iron out issues is
> the
> thought. Yes, it could push out delivery of 2.0.
>
>
Having this on by default in an unreleased master doesn't actually worry me
that much.  It's just the question of what happens when we do release.  At
that point, this discussion will be ancient history and I don't think we'll
give any renewed consideration to what the impact of this change might be.
Ideally it would be great to see this work in HDFS by that point and for
that HDFS version this becomes a non-issue.


>
> I think the discussion here has been helpful. Holes have been found (and
> plugged), the risk involved has gotten a good airing out here on dev, and
> in spite of the back and forth, one of our experts in good standing is
> still against it being on by default.
>
> If you are not down w/ the arguments, I'd be fine not making it the
> default.
> St.Ack
>

I don't think it's right to block this by myself, since I'm clearly in the
minority.  Since others clearly support this change, have at it.

But let me pose an alternate question: what if HDFS flat out refuses to
adopt this change?  What are our options then with this already shipping as
a default?  Would we continue to endure breakage due to the use of HDFS
private internals?  Do we switch the default back?  Do we do something else?

Thanks for the discussion.

Re: [DISCUSS] Make AsyncFSWAL the default WAL in 2.0

[Gary Helmling]

>
> Yeah the 'push to upstream' work has been started already. See here
>
> https://issues.apache.org/jira/browse/HADOOP-12910
>
> But it is much harder to push code into HDFS than HBase. It is the core of
> all hadoop systems and I do not have many contacts in the hdfs community...
>
>
Yes, I'm familiar with the difficulty of getting even relatively small
change in to HDFS.

Is HADOOP-12910 really the right issue for this?  There is some good
discussion there, but it looks like it's primarily motivated by doing large
batches of async renames.  Isn't our issue more that we want a whole new
OutputStream implementation doing fan out instead of the regular pipeline
replication?

HDFS-9924 seems like it might be the better umbrella for this.  Maybe it
would be better to create a new top level issue and link it there and
comment in HDFS-9924 to try to get some traction.


> And it is more convincing if we make it default as it means that we will
> keep maintaining the code rather than make it stale and unstable.
>
>
I would agree with this reasoning if we were talking about making an
implementation inside HDFS the default.  That would indicate commitment to
contribute to and maintain the HDFS implementation.  Making a separate code
copy living in HBase the default I don't think really means anything for
HDFS.

The fact that this already needs to be updated for 2.8 just reinforces that
we're going to see maintainability issues with this.

Again, I appreciate all of the work that has gone in to this feature and
the big performance improvements it shows, but I think the sequencing of
events here is going to ultimately cause us and our users more pain than is
necessary.

Re: [DISCUSS] Make AsyncFSWAL the default WAL in 2.0

[Gary Helmling]

>
> The suggestion is that we make this new client the default now in master
> branch so we have plenty of time to find any issues with the
> implementation. We'd also enable it as the default because the improvement
> is dramatic (performance, less moving parts, comprehensible, etc.) and we
> think this async, lightweight, WAL-purposed client the way to go moving
> forward. We'd also spare our users having to make a choice; if optional, if
> they trip over the feature at all, they'll be wary enabling such a
> fundamental afraid that it experimental.
>
>
To me, having the opportunity to shake out issues sounds like an argument
for making it experimental, not making it the default.  I think that
features we're rolling out to our users actively enabled should have some
level of confidence that the issues are already shaken out.  I'm interested
in testing this out myself, but personally I would want to test this by
actively enabling it and not just having it show up unexpectedly in a new
release.

Maybe this is mitigated by the reality that a 2.0 release is not going to
happen for quite a while.  But this also becomes a self-fulfilling prophecy
if we continue to make disruptive changes in master.


> Going this route we are taking on risk as you call out Gary but the
> suggestion is that the benefits far outweigh the downsides (In mitigation,
> I don't think we've ever run against HDFS free of reflection code though,
> true, we are into a new level of violation with this client).
>
>
I'm not trying to argue for a perfect world. :)

But I do think there is a big difference in some of our other past use of
reflection to ride over changes in public APIs vs. reaching in to private
fields in private annotated classes.  What would we say to a coprocessor
that did the same with HBase?


> We are not arguing that it needs to be default to help get the async client
> upstreamed into HDFS. Maybe it would help but going by the issue cited by
> Duo below, it seems like there are a bunch of other concerns and dimensions
> to be considered first; it may take a while for an async DFS client to land
> (if at all). We should push on the upstreaming yes to close down the risk
> you note, but let us not predicate our use of async WAL on it first being
> committed to HDFS.
>
>
I don't think our ability to use the async WAL as an optional feature
should be predicated on inclusion in HDFS.  But I do think our use of it as
the default, and all the continuing support that that implies should be.
That is where we disagree.

I don't think we're really making progress with the discussion here.  I
don't agree with the arguments being put forward, but it seems like I'm in
the minority.

Re: [DISCUSS] Make AsyncFSWAL the default WAL in 2.0

[Gary Helmling]

Thanks for adding the tests and fixing up AES support.

My only real concern is the maintainability of this code as our own private
DFS client.  The SASL support, for example, is largely based on reflection
and reaches in to private fields of @InterfaceAudience.Private Hadoop
classes.  This seems bound to break with a future Hadoop release.  I
appreciate the parameterized testing wrapped around this because it doesn't
seem like we'll have much else in the way of safety checking.  This is not
a knock on the code -- it's a pretty clean reach into the HDFS guts, but a
reach it is.  For a component at the core of our data integrity, this seems
like a risk.

To me, it seems much safer to actively try to push this upstream into HDFS
right now, and still pointing to its optional, non-default use in HBase as
a compelling story.  I don't understand why making it the default in 2.0 is
necessary for this.  Do you really think it will make that big a difference
for upstreaming?  Once it's actually in Hadoop and maintained, it seems
like a no-brainer to make it the default.

On Mon, May 9, 2016 at 5:09 PM Stack <st...@duboce.net> wrote:

> Any other suggestions/objections here? If not, will make the cut over in
> next day or so.
> Thanks,
> St.Ack
>
> On Thu, May 5, 2016 at 10:02 PM, Stack <st...@duboce.net> wrote:
>
> > On Thu, May 5, 2016 at 7:39 PM, Yu Li <car...@gmail.com> wrote:
> >
> >> Almost miss the party...
> >>
> >> bq. Do you think it worth to backport this feature to branch-1 and
> release
> >> it in the next 1.x release? This may introduce a compatibility issue as
> >> said
> >> in HBASE-14949 that we need HBASE-14949 to make sure that the rolling
> >> upgrade
> >> does not lose data...
> >> From current perf data I think the effort is worthwhile, we already
> >> started
> >> some work here and will run it on production after some carefully
> testing
> >> (and of course, if the perf number confirmed, but I'm optimistic somehow
> >> :-P). Regarding HBASE-14949, I guess a two-step rolling upgrade will
> make
> >> it work, right? (And I guess this will also be a question when we
> upgrade
> >> from 1.x to 2.0 later?)
> >>
> >>
> > Or a clean shutdown and restart? Or a fresh install? I'd think backport
> > would be fine if you have to enable it and it has warnings and is clear
> on
> > circumstances under which there could be dataloss.
> >
> > St.Ack
> >
> >
> >
> >> btw, I'm +1 about making asyncfswal as default in 2.0 :-)
> >>
> >> Best Regards,
> >> Yu
> >>
> >> On 6 May 2016 at 09:49, Ted Yu <yuzhih...@gmail.com> wrote:
> >>
> >> > Thanks for your effort, Duo.
> >> >
> >> > I am in favor of turning AsyncWAL as default in master branch.
> >> >
> >> > Cheers
> >> >
> >> > On Thu, May 5, 2016 at 6:03 PM, 张铎 <palomino...@gmail.com> wrote:
> >> >
> >> > > Some progress.
> >> > >
> >> > > I have filed HBASE-15743 for the transparent encryption support,
> >> > > and HBASE-15754 for the AES encryption UT. Now both of them are
> >> resolved.
> >> > > Let's resume the discussion here.
> >> > >
> >> > > Thanks.
> >> > >
> >> > > 2016-05-03 10:09 GMT+08:00 张铎 <palomino...@gmail.com>:
> >> > >
> >> > > > Fine, will add the testcase.
> >> > > >
> >> > > > And for the RPC, we only implement a new client side DTP here and
> >> still
> >> > > > use the original RPC.
> >> > > >
> >> > > > Thanks.
> >> > > >
> >> > > > 2016-05-03 3:20 GMT+08:00 Gary Helmling <ghelml...@gmail.com>:
> >> > > >
> >> > > >> On Fri, Apr 29, 2016 at 6:24 PM 张铎 <palomino...@gmail.com>
> wrote:
> >> > > >>
> >> > > >> > Yes, it does. There is testcase that enumerates all the
> possible
> >> > > >> protection
> >> > > >> > level(authentication, integrity and privacy) and encryption
> >> > > >> algorithm(none,
> >> > > >> > 3des, rc4).
> >> > > >> >
> >> > > >> >
> >> > > >> >
> >> > > >>
> >> > >
> >> >
> >>
> https://github.com/apache/hbase/blob/master/hbase-server/src/test/java/org/apache/hadoop/hbase/io/asyncfs/TestSaslFanOutOneBlockAsyncDFSOutput.java
> >> > > >> >
> >> > > >> > I have also tested it in a secure cluster(hbase-2.0.0-SNAPSHOT
> >> and
> >> > > >> > hadoop-2.4.0).
> >> > > >> >
> >> > > >>
> >> > > >> Thanks.  Can you add in support for testing with AES
> >> > > >> (dfs.encrypt.data.transfer.cipher.suites=AES/CTR/NoPadding)?
> This
> >> is
> >> > > only
> >> > > >> available in Hadoop 2.6.0+, but I think is far more likely to be
> >> used
> >> > in
> >> > > >> production than 3des or rc4.
> >> > > >
> >> > > >
> >> > > >> Also, have you been following HADOOP-10768?  That is changing
> >> Hadoop
> >> > RPC
> >> > > >> encryption negotiation to support more performant AES wrapping,
> >> > similar
> >> > > to
> >> > > >> what is now supported in the data transfer pipeline.
> >> > > >>
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> >
> >
>

[jira] [Resolved] (HBASE-13707) CellCounter uses to many counters

[Gary Helmling (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-13707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gary Helmling resolved HBASE-13707.
---
Resolution: Duplicate
  Assignee: Gary Helmling  (was: NIDHI GAMBHIR)

Fixed in HBASE-15773

> CellCounter uses to many counters
> -
>
> Key: HBASE-13707
> URL: https://issues.apache.org/jira/browse/HBASE-13707
> Project: HBase
>  Issue Type: Bug
>  Components: mapreduce
>Affects Versions: 1.0.1
>Reporter: Jean-Marc Spaggiari
>    Assignee: Gary Helmling
>Priority: Minor
>  Labels: beginner
>
> CellCounters creates a counter per row... So it quickly becomes to many.
> We should provide an option to drop the statistic per rows and count only 
> cells overall for the table.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-15773) CellCounter improvements

[Gary Helmling (JIRA)]

Gary Helmling created HBASE-15773:
-

 Summary: CellCounter improvements
 Key: HBASE-15773
 URL: https://issues.apache.org/jira/browse/HBASE-15773
 Project: HBase
  Issue Type: Improvement
  Components: mapreduce
Reporter: Gary Helmling


Looking at the CellCounter map reduce, it seems like it can be improved in a 
few areas:

* it does not currently support setting scan batching.  This is important when 
we're fetching all versions for columns.  Actually, it would be nice to support 
all of the scan configuration currently provided in TableInputFormat.
* generating job counters containing row keys and column qualifiers is 
guaranteed to blow up on anything but the smallest table.  This is not usable 
and doesn't make any sense when the same counts are in the job output.  The row 
and qualifier specific counters should be dropped.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] Make AsyncFSWAL the default WAL in 2.0

[Gary Helmling]

On Fri, Apr 29, 2016 at 6:24 PM 张铎  wrote:

> Yes, it does. There is testcase that enumerates all the possible protection
> level(authentication, integrity and privacy) and encryption algorithm(none,
> 3des, rc4).
>
>
> https://github.com/apache/hbase/blob/master/hbase-server/src/test/java/org/apache/hadoop/hbase/io/asyncfs/TestSaslFanOutOneBlockAsyncDFSOutput.java
>
> I have also tested it in a secure cluster(hbase-2.0.0-SNAPSHOT and
> hadoop-2.4.0).
>

Thanks.  Can you add in support for testing with AES
(dfs.encrypt.data.transfer.cipher.suites=AES/CTR/NoPadding)?  This is only
available in Hadoop 2.6.0+, but I think is far more likely to be used in
production than 3des or rc4.

Also, have you been following HADOOP-10768?  That is changing Hadoop RPC
encryption negotiation to support more performant AES wrapping, similar to
what is now supported in the data transfer pipeline.

[jira] [Created] (HBASE-15741) TokenProvider coprocessor RPC incompatibile between 1.2 and 1.3

[Gary Helmling (JIRA)]

Gary Helmling created HBASE-15741:
-

 Summary: TokenProvider coprocessor RPC incompatibile between 1.2 
and 1.3
 Key: HBASE-15741
 URL: https://issues.apache.org/jira/browse/HBASE-15741
 Project: HBase
  Issue Type: Bug
  Components: Coprocessors
Affects Versions: 1.3.0
Reporter: Gary Helmling
Priority: Blocker


Attempting to run a map reduce job with a 1.3 client on a secure cluster 
running 1.2 is failing when making the coprocessor rpc to obtain a delegation 
token:
{noformat}
Exception in thread "main" 
org.apache.hadoop.hbase.exceptions.UnknownProtocolException: 
org.apache.hadoop.hbase.exceptions.UnknownProtocolException: No registered 
coprocessor service found for name hbase.pb.AuthenticationService in region 
hbase:meta,,1
at 
org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:7741)
at 
org.apache.hadoop.hbase.regionserver.RSRpcServices.execServiceOnRegion(RSRpcServices.java:1988)
at 
org.apache.hadoop.hbase.regionserver.RSRpcServices.execService(RSRpcServices.java:1970)
at 
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:33652)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2170)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:109)
at 
org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:137)
at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:112)
at java.lang.Thread.run(Thread.java:745)

at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at 
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at 
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
at 
org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
at 
org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)
at 
org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332)
at 
org.apache.hadoop.hbase.protobuf.ProtobufUtil.execService(ProtobufUtil.java:1631)
at 
org.apache.hadoop.hbase.ipc.RegionCoprocessorRpcChannel$1.call(RegionCoprocessorRpcChannel.java:104)
at 
org.apache.hadoop.hbase.ipc.RegionCoprocessorRpcChannel$1.call(RegionCoprocessorRpcChannel.java:94)
at 
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:137)
at 
org.apache.hadoop.hbase.ipc.RegionCoprocessorRpcChannel.callExecService(RegionCoprocessorRpcChannel.java:108)
at 
org.apache.hadoop.hbase.ipc.CoprocessorRpcChannel.callBlockingMethod(CoprocessorRpcChannel.java:73)
at 
org.apache.hadoop.hbase.protobuf.generated.AuthenticationProtos$AuthenticationService$BlockingStub.getAuthenticationToken(AuthenticationProtos.java:4512)
at 
org.apache.hadoop.hbase.security.token.TokenUtil.obtainToken(TokenUtil.java:86)
at 
org.apache.hadoop.hbase.security.token.TokenUtil$1.run(TokenUtil.java:111)
at 
org.apache.hadoop.hbase.security.token.TokenUtil$1.run(TokenUtil.java:108)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at 
org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:340)
at 
org.apache.hadoop.hbase.security.token.TokenUtil.obtainToken(TokenUtil.java:108)
at 
org.apache.hadoop.hbase.security.token.TokenUtil.addTokenForJob(TokenUtil.java:329)
at 
org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil.initCredentials(TableMapReduceUtil.java:490)
at 
org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil.initTableMapperJob(TableMapReduceUtil.java:209)
at 
org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil.initTableMapperJob(TableMapReduceUtil.java:162)
at 
org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil.initTableMapperJob(TableMapReduceUtil.java:285)
at 
org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil.initTableMapperJob(TableMapReduceUtil.java:86)
at 
org.apache.hadoop.hbase.mapreduce.CellCounter.createSubmittableJob(CellCounter.java:193)
at 
org.apache.hadoop.hbase.mapreduce.CellCounter.main(CellCounter.java:290)
Caused by: 
org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.exceptions.UnknownProtocolException):
 org.apache.hadoop.hbase.exceptions.UnknownProtocolException: No registered 
coprocessor service found for name hbase.pb.AuthenticationService in region 
hba

Re: [DISCUSS] Make AsyncFSWAL the default WAL in 2.0

[Gary Helmling]

How well has this been tested on secure clusters?  I know SASL support was
lacking initially, but I believe it had been added?  Does AsyncFSWAL
support all the HDFS transport encryption options?


On Fri, Apr 29, 2016 at 12:05 AM Stack  wrote:

> I'm +1 on enabling asyncfswal as default in 2.0:
>
> + We'll have plenty of time to figure issues if any if we get it in now,
> early.
> + The improvement in throughput is substantial
> + There are now less moving parts
> + A critical piece of our write path is much less opaque in its workings
> and no longer (effectively) immutable
>
> St.Ack
>
>
> On Thu, Apr 28, 2016 at 11:53 PM, 张铎  wrote:
>
> > I‘ve done dig in HDFS and HADOOP proejcts and found that there is an
> active
> > issue HADOOP-12910 that related to asynchronous FileSystem
> implementation.
> >
> > I have left some comments on it, maybe we could start from there.
> >
> > Thanks.
> >
> > 2016-04-29 14:42 GMT+08:00 Stack :
> >
> > > On Thu, Apr 28, 2016 at 8:47 PM, Ted Yu  wrote:
> > >
> > > > Last comment on HDFS-916 was from 2010.
> > > >
> > > > Suggest making a new issue or reviving discussion on HDFS-916
> > (currently
> > > > assigned to Todd).
> > > >
> > > >
> > > Duo is on it. Some mileage and confidence in the new code would be good
> > to
> > > have before going to HDFS (Getting stuff into HDFS is a PITA at the
> best
> > of
> > > times... lets have a good case when we go there).
> > >
> > >
> > > > bq. The fallback implementation is not aim to get a good performance
> > > >
> > > > For more than two weeks, I have been working with Azure Data Lake
> > > > developers so that all hbase system tests pass on ADLS - there were
> > > subtle
> > > > differences between ADLS and hdfs.
> > > >
> > > > If switching to AsyncWAL gives either WASB or ADLS subpar
> performance,
> > it
> > > > would make upgrading to hbase 2.x unacceptable for their users.
> > > >
> > > >
> > > Just use FSHLog instead of asyncfswal when up on WASB. Its just a
> config
> > > change.
> > >
> > > St.Ack
> > >
> > >
> > >
> > > > On Thu, Apr 28, 2016 at 8:39 PM, 张铎  wrote:
> > > >
> > > > > 2016-04-29 11:35 GMT+08:00 Ted Yu :
> > > > >
> > > > > > bq. AsyncFSOutput will be in HDFS-3.0
> > > > > >
> > > > > > Is there HDFS JIRA for the above ? Can you share the number ?
> > > > > >
> > > > > I have not filed a new one but there are bunch of related issues
> > > already,
> > > > > such as this one https://issues.apache.org/jira/browse/HDFS-916
> > > > >
> > > > > >
> > > > > > bq. Just wrap FSDataOutputStream to make it act like an
> > asynchronous
> > > > > output
> > > > > >
> > > > > > Can you be a bit more specific ?
> > > > > > HBase currently works with WASB and Azure Data Lake. Does the
> above
> > > > mean
> > > > > > their performance would suffer ?
> > > > > >
> > > > > Yes, the performance will suffer...
> > > > > The fallback implementation is not aim to get a good performance,
> > just
> > > > for
> > > > > compatibility with any FileSystem implementation.
> > > > >
> > > > > >
> > > > > > On Thu, Apr 28, 2016 at 8:30 PM, 张铎 
> wrote:
> > > > > >
> > > > > > > Inline comments.
> > > > > > > Thanks,
> > > > > > >
> > > > > > > 2016-04-29 10:57 GMT+08:00 Sean Busbey :
> > > > > > >
> > > > > > > > I am nervous about having default out-of-the-box new HBase
> > users
> > > > > > reliant
> > > > > > > on
> > > > > > > > a bespoke HDFS client, especially given Hadoop's
> compatibility
> > > > > > > > promises and history. Answers for these questions would make
> me
> > > > more
> > > > > > > > confident:
> > > > > > > >
> > > > > > > > 1) Where are we on getting the client-side changes to HDFS
> > pushed
> > > > > back
> > > > > > > > upstream?
> > > > > > > >
> > > > > > > No progress yet... Here I want to tell a good story that HBase
> is
> > > > > already
> > > > > > > use it as default :)
> > > > > > >
> > > > > > > >
> > > > > > > > 2) How well do we detect when our FS is not HDFS and what
> does
> > > > > > > > fallback look like?
> > > > > > > >
> > > > > > > Just wrap FSDataOutputStream to make it act like an
> asynchronous
> > > > > > > output(call hflush in a separated thread). The performance is
> not
> > > > good
> > > > > I
> > > > > > > think.
> > > > > > >
> > > > > > > >
> > > > > > > > 3) Will this mean altering the versions of Hadoop we label as
> > > > > > > > supported for HBase 2.y+?
> > > > > > > >
> > > > > > > I have tested with hadoop versions from 2.4.x to 2.7.x, so I
> > don't
> > > > > think
> > > > > > we
> > > > > > > need to change the supported versions?
> > > > > > >
> > > > > > > >
> > > > > > > > 4) How are we going to ensure our client remains compatible
> > with
> > > > > newer
> > > > > > > > Hadoop releases?
> > > > > > > >
> > > > > > > We can not ensure, HDFS always breaks HBase at a new release...
> > > > > > > I 

Re: [DISCUSS] No regions on Master node in 2.0

[Gary Helmling]

On Mon, Apr 25, 2016 at 11:20 AM Stack  wrote:

> On Fri, Apr 8, 2016 at 1:42 AM, Elliott Clark  wrote:
>
> > # Without meta on master, we double assign and lose data.
> >
> > That is currently a fact that I have seen over and over on multiple
> loaded
> > clusters. Some abstract clean up of deployment vs losing data is a
> > no-brainer for me. Master assignment, region split, region merge are all
> > risky, and all places that HBase can lose data. Meta being hosted on the
> > master makes communication easier and less flakey. Running ITBLL on a
> loop
> > that creates a new table every time, and without meta on master
> everything
> > will fail pretty reliably in ~2 days. With meta on master things pass
> MUCH
> > more.
> >
> >
> The above is a problem of branch-1?
>
> The discussion is what to do in 2.0 with the assumption that master state
> would be done up on procedure v2 making most of the transitions now done
> over zk and hbase:meta instead local to the master with only the final
> state published to a remote meta (an RPC but if we can't make RPC work
> reliably in our distributed system, thats a bigger problem).
>
>
But making RPC work for assignment here is precisely the problem.  There's
no reason master should have to contend with user requests to meta in order
to be able to make updates.  And until clients can actually see the change,
it doesn't really matter if the master state has been updated or not.

Sure, we could add more RPC priorities, even more handler pools and
additional queues for master requests to meta vs. user requests to meta.
Maybe with that plus adding in regionserver groups we actually start to
have something that comes close to what we already have today with meta on
master.  But why should we have to add all that complexity?  None of this
is an issue if master updates to meta are local and don't have to go
through RPC.


>
> > # Master hosting the system tables locates the system tables as close as
> > possible to the machine that will be mutating the data.
> >
> > Data locality is something that we all work for. Short circuit local
> reads,
> > Caching blocks in jvm, etc. Bringing data closer to the interested party
> > has a long history of making things faster and better. Master is in
> charge
> > of just about all mutations of all systems tables. It's in charge of
> > changing meta, changing acls, creating new namespaces, etc. So put the
> > memstore as close as possible to the system that's going to mutate meta.
> >
>
>
> Above is fine except for the bit where we need to be able to field reads.
> Lets distribute the data to be read over the cluster rather than treat meta
> reads with kid gloves hosted on a 'special' server; let these 'reads' be
> like any other read the cluster takes (see next point)
>
>
In my opinion, the real "special" part here is the master bit -- which I
think we should be working to make less special and more just a normal bit
of housekeeping spread across nodes -- not the regionserver role.  It only
looks special right now because the evolution has stopped in the middle.  I
really don't think enshrining master as a separate process is the right way
forward for us.


>
> > # If you want to make meta faster then moving it to other regionservers
> > makes things worse.
> >
> > Meta can get pretty hot. Putting it with other regions that clients will
> be
> > trying to access makes everything worse. It means that meta is competing
> > with user requests. If meta gets served and other requests don't, causing
> > more requests to meta; or requests to user regions get served and other
> > clients get starved.
> > At FB we've seen read throughput to meta doubled or more by swapping it
> to
> > master. Writes to meta are also much faster since there's no rpc hop, no
> > queueing, to fighting with reads. So far it has been the single biggest
> > thing to make meta faster.
> >
> >
> Is this just because meta had a dedicated server?
>
>
I'm sure that having dedicated resources for meta helps.  But I don't think
that's sufficient.  The key is that master writes to meta are local, and do
not have to contend with the user requests to meta.

It seems premature to be discussing dropping a working implementation which
eliminates painful parts of distributed consensus, until we have a complete
working alternative to evaluate.  Until then, why are we looking at
features that are in use and work well?



>

[jira] [Created] (HBASE-15678) Normalize RetryingCallable cache clearing and implementations

[Gary Helmling (JIRA)]

Gary Helmling created HBASE-15678:
-

 Summary: Normalize RetryingCallable cache clearing and 
implementations
 Key: HBASE-15678
 URL: https://issues.apache.org/jira/browse/HBASE-15678
 Project: HBase
  Issue Type: Sub-task
  Components: Client
Reporter: Gary Helmling
Assignee: Gary Helmling


This is a fair amount of duplication and inconsistency in the meta cache 
handling of RetryingCallable implementations:
* meta cache is often cleared in prepare() when reload=true, in addition to 
being cleared in throwable()
* each RetryingCallable implementation does this slightly differently, leading 
to inconsistencies and potential bugs
* RegionServerCallable and RegionAdminServiceCallable duplicate a lot of code, 
but with small, seemingly unnecessary inconsistencies.  We should clean these 
up into a common base with subclasses doing only the necessary differentiation.

The main goal here is to establish some common handling, to the extent 
possible, for the meta cache interactions by the different implementations.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-15641) Shell "alter" should do a single modifyTable operation

[Gary Helmling (JIRA)]

Gary Helmling created HBASE-15641:
-

 Summary: Shell "alter" should do a single modifyTable operation
 Key: HBASE-15641
 URL: https://issues.apache.org/jira/browse/HBASE-15641
 Project: HBase
  Issue Type: Improvement
  Components: shell
Reporter: Gary Helmling


When performing an "alter" on multiple column families in a table, then shell 
will perform a separate {{Admin.modifyColumn()}} call for each column family 
being modified, with all of the table regions being bulk-reopened each time.  
It would be much better to simply apply all the changes to the table 
descriptor, then do a single call to {{Admin.modifyTable()}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] No regions on Master node in 2.0

[Gary Helmling]

Sorry to be late to the party here.  I'll sprinkle my comments over the
thread where they make the most sense.


> Currently in the master branch, HMaster hosts 'system tables': e.g.
> hbase:meta. HMaster is doing more than just gardening the cluster,
> bootstrapping and keeping all up and serving healthy as in branch-1; in
> master branch, it is actually in the write path for the most critical
> system regions.
>
>
I think it's important to point out that this feature exists and is usable
in branch-1 as well, including in all 1.x releases.  It just disabled by
default branch-1 and enabled by default in master. So this is really a
comparison of an existing, shipping, feature that does work, and is being
used vs. ongoing development work in master.


>
> Let me provoke discussion by making the statement that we should undo
> HMaster carrying any regions ever; that the HMaster function is work enough
> for a single dedicated server and that it important enough that it cannot
> take a background role on a serving RegionServer (I could go back from this
> position if evidence HMaster role could be backgrounded). Notions of a
> Master carrying system tables only are just not on given system tables will
> be too big for a single server especially when hbase:meta is split (so we
> can scale).


If we really think that normal master housekeeping functions are work
enough that we shouldn't combine with region serving, then why do we think
that those will _not_ have to be scaled by splitting the metadata space
across multiple servers when we encounter meta-scaling issues that require
splitting meta to distribute it across multiple servers?  If we really want
to scale, then it seems like we need to tackle scaling the region metadata
in general across multiple active masters, in which case meta-on-master is
not really an argument either way.


> This simple distinction of HMaster and RegionServer roles is
> also what our users know and have gotten used to so needs to be a good
> reason to change it (We can still pursue the single binary that can do
> HMaster or HRegionServer role determined at runtime).
>

The distinction in roles in HBase has long been used as a criticism of
HBase's operational complexity.  I think we would be doing our users a
service by simplifying this and making it a detail they do not need to
worry about. If we can truly make this transparent to users and improve
operability at the same time, I think that would be the best outcome.

[jira] [Resolved] (HBASE-15573) Indefinite pause while trying to cleanup data

[Gary Helmling (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gary Helmling resolved HBASE-15573.
---
Resolution: Invalid

This JIRA instance is used for tracking development issues and bugs.

Please send an email to the u...@hbase.apache.org mailing list to ask any 
questions.  Sounds like a configuration issue with your client application.

> Indefinite pause while trying to cleanup data
> -
>
> Key: HBASE-15573
> URL: https://issues.apache.org/jira/browse/HBASE-15573
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.1.2, 1.1.4
>Reporter: Jorge Figueira
>Priority: Blocker
> Attachments: hbase-hadoop-master-HBASE.log, 
> hbase-hadoop-regionserver-HBASE.log, hbase-hadoop-zookeeper-HBASE.log
>
>
> Can't retrieve any information with hbase  rpc java client.
> With hbase shell its possible to scan data and retrieve all the information 
> normally.
> But with any rpc client region server don't retrieve data, all data come with 
> null values.
> Region Server log:
> DEBUG [RpcServer.reader=2,bindAddress=HBASE,port=16020] ipc.RpcServer: 
> RpcServer.listener,port=16020: DISCONNECTING client SERVER:37088 because read 
> count=-1
> DEBUG [RpcServer.reader=2,bindAddress=HBASE,port=16020] ipc.RpcServer: 
> RpcServer.listener,port=16020: DISCONNECTING client SERVER2:36997 because 
> read count=-1
> Master log:
> 2016-03-31 18:16:27,998 DEBUG [ProcedureExecutorTimeout] 
> procedure2.ProcedureExecutor$CompletedProcedureCleaner: No completed 
> procedures to cleanup.
> 2016-03-31 18:16:57,998 DEBUG [ProcedureExecutorTimeout] 
> procedure2.ProcedureExecutor$CompletedProcedureCleaner: No completed 
> procedures to cleanup.
> 2016-03-31 18:17:27,998 DEBUG [ProcedureExecutorTimeout] 
> procedure2.ProcedureExecutor$CompletedProcedureCleaner: No completed 
> procedures to cleanup



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Coprocessors causes data access delay?

[Gary Helmling]

Do you wait for the Put to return before you issue the Get?  Or have you
disabled auto flushing on your table instance -- HTable.setAutoFlush(false)?

Coprocessor hooks executed on the Put and Get paths are blocking, so unless
you're overriding normal processing in the prePut() hook and doing
something asynchronous, the Put should be there by the time it returns to
the client.  But it all depends on what is your coprocessor actually
doing.  Can you tell us more about that?

On Tue, Mar 29, 2016 at 6:22 AM Ted Yu  wrote:

> Is it possible to come up with unit test showing this behavior using
> simplified coprocessor ?
>
> Thanks
>
> > On Mar 29, 2016, at 5:42 AM, Jozef Vilcek  wrote:
> >
> > Hi,
> >
> > I need a help to shed some light on following observation:
> >
> > Through hbase shell I do PUT to a non existing cell (create) and
> immediate
> > GET.
> > When doing this against hbase table without a coprocessor, every GET
> shows
> > the value immediately. When I enable a coprocessor on the table and
> perform
> > the same check, I do not get a value for every GET. Sometimes GET returns
> > nothing, like cell is not there. If I delay GET few minutes after PUT,
> then
> > data are correctly retrieved.
> >
> > I want to understand what is going on.
> > Is coprocessor is doing something wrong? Is this somehow normal? or ... ?
> >
> > Coprocessor is a RegionObserver doing data filtering / expiration based
> on
> > predefined rules.
> > HBase version is 1.0.0 from cloudera cdh5.5.1 distibution.
> >
> > Thanks,
> > Jozo
>

Re: [DISCUSS] Backporting Regionserver Groups (HBASE-6721) to 1.x and 0.98

[Gary Helmling]

>
> Tho based on this discussion there's a bunch of good features that users
> want that won't fit the criteria for #1 and #2. So allowing a backport of
> the few that can fit into the criteria shouldn't significantly affect
> future release from trunk. This way we can have some progress on some
> features.
>
>
If we have features that don't fit criteria #1 (compatibility), then we
need a new major release according to our semver guidelines.  Criteria #2
(stability) I think needs to be a gating factor on any new feature coming
in.  We need to be able to demonstrate that we're not doing any harm for
existing users by adding new features.

In both cases, I'm not sure we're very well served by committing new
features to master, then back-porting to branch-1.  That means that
whatever stabilization is happening is likely from testing on branch-1
based releases and not master, giving more time for changes to diverge.  I
think we would be better served by releasing early and often from master
and not having a branch-1 at all.  That would serve as a forcing function
to make sure that change going in to master have to stabilize while "fresh"
before a release is made.

For more complex efforts, it would be better to stabilize in a feature
branch and then merge the completed effort as one.

I think we're already boxing ourselves in by looking at 2.0 as a feature
based release, which means it necessarily has an unbounded timeline.  That
means we have to do backports for other features ready before then, since
master can't be released.  If we instead focus on a releasable master, that
should help us release large changes more incrementally, and save ourselves
the hassle of so many backports at the same time.  Of course everyone is
free to invest their time where they choose.  But I think if we focus on
time-based releases instead it would be better for the project as a whole.

[jira] [Created] (HBASE-15363) Add client side metrics for SASL connection failures

[Gary Helmling (JIRA)]

Gary Helmling created HBASE-15363:
-

 Summary: Add client side metrics for SASL connection failures
 Key: HBASE-15363
 URL: https://issues.apache.org/jira/browse/HBASE-15363
 Project: HBase
  Issue Type: Improvement
  Components: Client, metrics, security
Reporter: Gary Helmling
Assignee: Gary Helmling


There are a number of cases where we can get SASL connection failures before 
getting to the server, like errors talking to the KDC/TGS and misconfiguration 
of kerberos principals.  Hence these will not show up in the server-side 
authentication_failures metric.

We should add client side metrics on SASL connection failures to capture these.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-15294) Document advanced replication configurations with security

[Gary Helmling (JIRA)]

Gary Helmling created HBASE-15294:
-

 Summary: Document advanced replication configurations with security
 Key: HBASE-15294
 URL: https://issues.apache.org/jira/browse/HBASE-15294
 Project: HBase
  Issue Type: Task
  Components: documentation
Reporter: Gary Helmling
Assignee: Gary Helmling


HBASE-14866 fixed handling of source and cluster replication configs for some 
replication tools, needed, for example, for correct handling of some 
cross-realm trust security configurations.

We need to document some examples in the reference guide.  One examle, to 
configure a replication peer with different server principals:

{noformat}
add_peer '1', CLUSTER_KEY => "server1.cie.com:2181:/hbase",
CONFIG => {
'hbase.master.kerberos.principal' => 'hbase/instan...@realm2.com',
'hbase.regionserver.kerberos.principal' => 'hbase/instan...@realm2.com',
}
{noformat}

Additional arguments to VerifyReplication should also be documented in the 
usage output.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-15234) ReplicationLogCleaner can abort due to transient ZK issues

[Gary Helmling (JIRA)]

Gary Helmling created HBASE-15234:
-

 Summary: ReplicationLogCleaner can abort due to transient ZK issues
 Key: HBASE-15234
 URL: https://issues.apache.org/jira/browse/HBASE-15234
 Project: HBase
  Issue Type: Bug
  Components: master, Replication
Reporter: Gary Helmling
Assignee: Gary Helmling
Priority: Critical


The ReplicationLogCleaner delegate for the LogCleaner chore can abort due to 
transient errors reading the replication znodes, leaving the log cleaner chore 
stopped, but the master still running.  This causes logs to build up in the 
oldWALs directory, which can even hit storage or file count limits in HDFS, 
causing problems.

We've seen this happen in a couple of clusters when a rolling restart was 
performed on the zk peers (only one restarted at a time).

The full stack trace when the log cleaner aborts is:
{noformat}
16/02/02 15:22:39 WARN zookeeper.ZKUtil: 
replicationLogCleaner-0x1522c8b93c2fbae, quorum=, 
baseZNode=/hbase Unable to get data of znode /hbase/replication/rs  
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = 
ConnectionLoss for /hbase/replication/rs
at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
at 
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:359)
at 
org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataNoWatch(ZKUtil.java:713)  
at 
org.apache.hadoop.hbase.replication.ReplicationQueuesClientZKImpl.getQueuesZNodeCversion(ReplicationQueuesClientZKImpl.java:80)
  
at 
org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner.loadWALsFromQueues(ReplicationLogCleaner.java:99)
  
at 
org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner.getDeletableFiles(ReplicationLogCleaner.java:70)
at 
org.apache.hadoop.hbase.master.cleaner.CleanerChore.checkAndDeleteFiles(CleanerChore.java:233)
at 
org.apache.hadoop.hbase.master.cleaner.CleanerChore.checkAndDeleteEntries(CleanerChore.java:157)
at 
org.apache.hadoop.hbase.master.cleaner.CleanerChore.chore(CleanerChore.java:124)
at org.apache.hadoop.hbase.ScheduledChore.run(ScheduledChore.java:185)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)  
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
  
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
  
at 
org.apache.hadoop.hbase.JitterScheduledThreadPoolExecutorImpl$JitteredRunnableScheduledFuture.run(JitterScheduledThreadPoolExecutorImpl.java:110)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)  
16/02/02 15:22:39 ERROR zookeeper.ZooKeeperWatcher: 
replicationLogCleaner-0x1522c8b93c2fbae, quorum=, 
baseZNode=/hbase Received unexpected KeeperException, re-throwing exception  
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = 
ConnectionLoss for /hbase/replication/rs
at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
at 
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:359)
at 
org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataNoWatch(ZKUtil.java:713)  
at 
org.apache.hadoop.hbase.replication.ReplicationQueuesClientZKImpl.getQueuesZNodeCversion(ReplicationQueuesClientZKImpl.java:80)
  
at 
org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner.loadWALsFromQueues(ReplicationLogCleaner.java:99)
  
at 
org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner.getDeletableFiles(ReplicationLogCleaner.java:70)
at 
org.apache.hadoop.hbase.master.cleaner.CleanerChore.checkAndDeleteFiles(CleanerChore.java:233)
at 
org.apache.hadoop.hbase.master.cleaner.CleanerChore.checkAndDeleteEntries(CleanerChore.java:157)
at 
org.apache.hadoop.hbase.master.cleaner.CleanerChore.chore(CleanerChore.java:124)
at org.apache.hadoop.hbase.ScheduledChore.run(ScheduledChore.java:185)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset

[jira] [Created] (HBASE-15111) "hbase version" should write to stdout

[Gary Helmling (JIRA)]

Gary Helmling created HBASE-15111:
-

 Summary: "hbase version" should write to stdout
 Key: HBASE-15111
 URL: https://issues.apache.org/jira/browse/HBASE-15111
 Project: HBase
  Issue Type: Improvement
  Components: util
Reporter: Gary Helmling
Assignee: Gary Helmling
Priority: Trivial


Calling {{hbase version}} currently outputs the version info by writing to 
{{LOG.info}}.  This means, if you change the default log level settings, you 
may get no output at all on the command line.

Since {{VersionInfo.main()}} is being called, it should really just output 
straight to stdout.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-15038) ExportSnapshot should support separate configurations for source and destination clusters

[Gary Helmling (JIRA)]

Gary Helmling created HBASE-15038:
-

 Summary: ExportSnapshot should support separate configurations for 
source and destination clusters
 Key: HBASE-15038
 URL: https://issues.apache.org/jira/browse/HBASE-15038
 Project: HBase
  Issue Type: Improvement
  Components: mapreduce, snapshots
Reporter: Gary Helmling
Assignee: Gary Helmling


Currently ExportSnapshot uses a single Configuration instance for both the 
source and destination FileSystem instances to use.  It should allow overriding 
properties for each filesystem connection separately.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-15025) Allow clients configured with insecure fallback to attempt SIMPLE auth when KRB fails

[Gary Helmling (JIRA)]

Gary Helmling created HBASE-15025:
-

 Summary: Allow clients configured with insecure fallback to 
attempt SIMPLE auth when KRB fails
 Key: HBASE-15025
 URL: https://issues.apache.org/jira/browse/HBASE-15025
 Project: HBase
  Issue Type: Improvement
  Components: security
Reporter: Gary Helmling
Assignee: Gary Helmling


We have separate configurations for both client and server allowing a 
"permissive" mode where connections to insecure servers and clients 
(respectively) are allowed.  However, if both client and server are configured 
for Kerberos authentication for a given cluster, and Kerberos authentication 
fails, the connection will still fail if the fallback configurations are set to 
true.

If the client is configured to allow insecure fallback, and Kerberos 
authentication fails, we could instead have the client retry with SIMPLE auth.  
If the server is also configured to allow insecure fallback, this would allow 
the connection to succeed in the case of transient problems with Kerberos 
infrastructure, for example.

There is of course a danger that this would allow misconfigurations of security 
to be silently ignored, but we can add some loud logging on the client side 
when fallback to SIMPLE auth occurs, plus we have metrics and logging on the 
server side for fallbacks.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-14886) ReplicationAdmin does not use full peer configuration

[Gary Helmling (JIRA)]

Gary Helmling created HBASE-14886:
-

 Summary: ReplicationAdmin does not use full peer configuration
 Key: HBASE-14886
 URL: https://issues.apache.org/jira/browse/HBASE-14886
 Project: HBase
  Issue Type: Bug
  Components: Replication
Reporter: Gary Helmling
Assignee: Gary Helmling
Priority: Critical
 Fix For: 2.0.0, 1.2.0, 1.3.0


In {{listValidReplicationPeers()}}, we're creating the peer {{Configuration}} 
based on the source connection configuration and simply applying the peer ZK 
cluster key.  This causes any additional properties present in the 
{{ReplicationPeerConfig}} configuration to not be applied.

We should instead be using the configuration returned by 
{{ReplicationPeers.getPeerConf()}}, which we already call in that method.





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-14866) VerifyReplication should use peer configuration in peer connection

[Gary Helmling (JIRA)]

Gary Helmling created HBASE-14866:
-

 Summary: VerifyReplication should use peer configuration in peer 
connection
 Key: HBASE-14866
 URL: https://issues.apache.org/jira/browse/HBASE-14866
 Project: HBase
  Issue Type: Improvement
  Components: Replication
Reporter: Gary Helmling
 Fix For: 2.0.0, 1.2.0, 1.3.0


VerifyReplication uses the replication peer's configuration to construct the 
ZooKeeper quorum address for the peer connection.  However, other configuration 
properties in the peer's configuration are dropped.  It should merge all 
configuration properties from the {{ReplicationPeerConfig}} when creating the 
peer connection and obtaining a credentials for the peer cluster.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-14821) CopyTable should allow overriding more config properties for peer cluster

[Gary Helmling (JIRA)]

Gary Helmling created HBASE-14821:
-

 Summary: CopyTable should allow overriding more config properties 
for peer cluster
 Key: HBASE-14821
 URL: https://issues.apache.org/jira/browse/HBASE-14821
 Project: HBase
  Issue Type: Improvement
  Components: mapreduce
Reporter: Gary Helmling
Assignee: Gary Helmling


When using CopyTable across two separate clusters, you can specify the ZK 
quorum for the destination cluster, but not much else in configuration 
overrides.  This can be a problem when the cluster configurations differ, such 
as when using security with different configurations for server principals.

We should provide a general way to override configuration properties for the 
peer / destination cluster.  One option would be to allow use of a prefix for 
command line properties ("peer.property.").  Properties matching this prefix 
will be stripped and merged to the peer configuration.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-14775) Replication can't authenticate with peer Zookeeper with different server principal

[Gary Helmling (JIRA)]

Gary Helmling created HBASE-14775:
-

 Summary: Replication can't authenticate with peer Zookeeper with 
different server principal
 Key: HBASE-14775
 URL: https://issues.apache.org/jira/browse/HBASE-14775
 Project: HBase
  Issue Type: Bug
Reporter: Gary Helmling
Assignee: Gary Helmling


When replication is setup with security, where the local ZK cluster and peer ZK 
cluster use different server principals, the source HBase cluster is unable to 
authenticate with the peer ZK cluster.

When ZK is configured for SASL authentication and a server principal other than 
the default ("zookeeper") is used, the correct server principal must be 
specified on the client as a system property -- the confusingly named 
{{zookeeper.sasl.client.username}}.  However, since this is given as a system 
property, authentication with the peer cluster breaks when it uses a different 
ZK server principal than the local cluster.

We need a way of tying this setting to the replication peer config and then 
setting the property when the peer's ZooKeeperWatcher is created.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Any interest in a meetup in October?

[Gary Helmling]

October 8th seems good to me.

On Wed, Sep 9, 2015 at 7:23 PM Mikhail Antonov  wrote:

> I'm tentatively in.
>
> -Mikhail
>
> On Wed, Sep 9, 2015 at 4:23 PM, Vladimir Rodionov
>  wrote:
> > Free beer is always good :)
> >
> > -Vlad
> >
> > On Wed, Sep 9, 2015 at 4:14 PM, Andrew Purtell 
> wrote:
> >
> >> It's been a while since we've had a dev/user meetup. I've speculatively
> >> reserved a large theater style room for the evening of October 8 at
> >> Salesforce HQ in downtown SF, if that could work for folks. What do
> >> you think? If it looks like we could get a critical mass I'll post it on
> >> the meetup group.
> >>
> >> --
> >> Best regards,
> >>
> >>- Andy
> >>
> >> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> >> (via Tom White)
> >>
>
>
>
> --
> Thanks,
> Michael Antonov
>

Re: Question on Reading Sequencefile data using mapreduce

[Gary Helmling]

I believe the recommended approach would be to use CellUtil.  It is marked
as Public/Evolving and exposes a number of static createCell() methods that
will generate Cell instances for different combinations of parameters.

On Tue, Apr 14, 2015 at 9:04 AM Nick Dimiduk ndimi...@gmail.com wrote:

 Heya devs,

 Looking for an appropriate answer for this question, it seems we don't have
 a public Cell implementation that can be used for generating HFiles. How
 are folks expected to generate Cells instances if KeyValue is
 @InterfaceAudience.Private ?

 -n

 On Tue, Apr 14, 2015 at 9:00 AM, Nick Dimiduk ndimi...@gmail.com wrote:

  What version of HBase are you using?
 
  In recent versions, HFileOutputFormat is a deprecated class (replaced by
  HFileOutputFormat2), and KeyValue is an internal API (see the class
  annotation @InterfaceAudience.Private, basically, use at your own peril).
  The javadoc on the KeyValue constructor you're using says Creates a
  KeyValue from the start of the specified byte array. *Presumes bytes
  content is formatted as a KeyValue blob*. (emphasis my own) It looks
  like the value you're using for bytes is not in the KeyValue blob format.
 
  Try instead one of the other KeyValue constructors, such as
  KeyValue(byte[] row, byte[] family, byte[] qualifier, byte[] value).
 
  Thanks,
  Nick
 
 
  On Thu, Apr 9, 2015 at 12:23 PM, yuantao peng pengyuan...@gmail.com
  wrote:
 
  I am learning how to upload binary data to HBASE using mapreduce.   Here
  are the steps I am following assuming my binary file is testlist
  (1) wrote a sequencefilewrite.java to read the local testlist file and
  save
  a sequence file to HDFS
  (2) wrote a MapReduce program to read the generated sequence file, and
  generate a HFile
  (3) bulk import this HFile to HBASE
 
  I am stuck at step (2) as I keep getting exception.  I am absolutely new
  to
  hadoop/hbase,   code is posted below,  any comments or suggestions are
  appreciated!!!
 
  Sequencewrite.java is like this:
 
  public class SequenceFileWrite
  {
  public static void main(String[] args) throws IOException {
  String uri = args[1];
  Configuration conf = new Configuration();
  FileSystem fs = FileSystem.get(conf);
  Path path = new Path(uri);
  File infile = new File(args[0]);
  SequenceFile.Writer writer = null;
  try
  {
 
BytesWritable key, value;
writer = SequenceFile.createWriter(fs, conf,  path,
  BytesWritable.class, BytesWritable.class);
FileInputStream fin = new FileInputStream(infile);
for(int i=0; i10; ++i) {
  key   = new BytesWritable();
  value = new BytesWritable();
  byte[] keybuf = new byte[2];
  byte[] valbuf = new byte[2];
  fin.read(keybuf);
  fin.read(valbuf);
  key.set(keybuf,0,2);
  value.set(valbuf,0,2);
  writer.append(key,value);
}
  } finally {
 IOUtils.closeStream(writer);
  }
  }
  }
 
  And my mapper is like this:
 
  public class HBaseTkrHdrMapper extends MapperBytesWritable,
  BytesWritable,
  ImmutableBytesWritable, KeyValue {
 
int tipOffSeconds = 0;
String tableName = ;
 
ImmutableBytesWritable hKey = new ImmutableBytesWritable();
KeyValue kv;
 
@Override
protected void setup(Context context) throws IOException,
   InterruptedException {
  Configuration c = context.getConfiguration();
  tipOffSeconds   = c.getInt(epoch.seconds.tipoff, 0);
  tableName   = c.get(hbase.table.mrtest);
}
 
@Override
protected void map(BytesWritable key, BytesWritable value, Context
  context)  throws IOException, InterruptedException {
  ImmutableBytesWritable hkey = new
  ImmutableBytesWritable(key.getBytes());
  KeyValue   hval = new KeyValue(value.getBytes());
  context.write(hkey, hval);
}
  }
 
  Driver code is as follows:
 
  public class Driver {
public static void main(String[] args) throws Exception {
  Configuration conf = new Configuration();
  args = new GenericOptionsParser(conf, args).getRemainingArgs();
 
  @SuppressWarnings(deprecation)
  Job job = new Job(conf, Bulk Import);
  job.setJarByClass(HBaseTkrHdrMapper.class);
 
  job.setMapperClass(HBaseTkrHdrMapper.class);
  job.setMapOutputKeyClass(ImmutableBytesWritable.class);
  job.setMapOutputValueClass(KeyValue.class);
  job.setInputFormatClass(SequenceFileInputFormat.class);
 
  HTable hTable = new HTable(conf, args[2]);
 
  // Auto configure partitioner and reducer
  HFileOutputFormat.configureIncrementalLoad(job, hTable);
 
  FileInputFormat.addInputPath(job, new Path(args[0]));
  FileOutputFormat.setOutputPath(job, new Path(args[1]));
 
  job.waitForCompletion(true);
}
  }
 
 
  The exception I got is :
 
 
  Error: java.lang.IllegalArgumentException: offset (0) + length (4)
 exceed
  the capacity of the array: 3
  at
 
 
 

Re: Private MVN repos in HBase's pom.xml

[Gary Helmling]

Sorry, I thought we had removed these from all the branches no longer using
the custom Surefire and JUnit builds long ago.

The repo was removed from master in
https://issues.apache.org/jira/browse/HBASE-8674

I'll go through the other branches and see where else it can be removed.
To remove the custom repo, we need backports of HBASE-4955 and HBASE-7574
to have been applied.



On Wed, Mar 25, 2015 at 4:29 PM Elliott Clark ecl...@apache.org wrote:

 Yeah we don't use it at all in branch-1 and master. I haven't checked 0.98
 lately though it used to need it.

 On Wed, Mar 25, 2015 at 3:57 PM, Andrew Purtell apurt...@apache.org
 wrote:

  I don't believe we've needed that since we moved away from custom
 versions
  of Surefire.
 
  On Wed, Mar 25, 2015 at 3:49 PM, Stack st...@duboce.net wrote:
 
   I tried it recently and doesn't seem like we need it anymore. I was
 going
   to remove it but...
   St.Ack
  
   On Wed, Mar 25, 2015 at 3:47 PM, lars hofhansl la...@apache.org
 wrote:
  
I just noticed that Gary's repo is still listed in our pom. Do we
 still
need that? Should we remove it?
Not that I do not trust Gary... :)  Just that we should limit
 ourselves
   to
the public repos.
   
-- Lars
   
repository
  idghelmling.testing/id
  nameGary Helmling test repo/name
  urlhttp://people.apache.org/~garyh/mvn//url
  snapshots
enabledtrue/enabled
  /snapshots
  releases
enabledtrue/enabled
  /releases
/repository
   
   
   
  
 
 
 
  --
  Best regards,
 
 - Andy
 
  Problems worthy of attack prove their worth by hitting back. - Piet Hein
  (via Tom White)
 

Re: [ANNOUNCE] Apache HBase 1.0.0 is now available for download

[Gary Helmling]

Fantastic work!  Congrats everyone!

On Tue Feb 24 2015 at 9:45:24 AM Esteban Gutierrez este...@cloudera.com
wrote:

 Wow! Congrats, all!

 --
 Cloudera, Inc.


 On Tue, Feb 24, 2015 at 9:41 AM, Jerry He jerry...@gmail.com wrote:

  Congratulations on the milestone!
 

Re: [VOTE] htrace-3.1.0, the thank you for your patience, eighth release candidate

[Gary Helmling]

Did you mean to send to d...@htrace.incubator.apache.org instead of
dev@hbase.apache.org?  Or are you cc'ing us here as an FYI of the new RC?

On Tue, Jan 13, 2015 at 9:04 PM, Stack st...@duboce.net wrote:

 I've posted the eighth htrace-3.1.0 release candidate here:

   http://people.apache.org/~stack/htrace-3.1.0-incubatingRC7/

 Maven artifacts here:

   https://repository.apache.org/content/repositories/orgapachehtrace-1011

 The only changes from RC7 are changes to address Billie and Masatake
 reviews.

 The release is a src-only tarball for now. Later we can add binary bundles
 to our release after we have better sense of what we as a project would
 like to deliver.

 This release is mainly a change of packaging from org.htrace to
 org.apache.htrace to suit our new home here in Apache Incubator but it does
 also includes 42 resolved issues [1] including the beginnings of an htrace
 daemon whose intent is to make it so there is a low barrier collecting
 cluster traces as well as a new flume receiver.

 Beware that Apache Incubator, org.apache.htrace is not compatible with its
 former self, org.htrace: the package name has changed but so has the JSON
 serialization format. See README.md for a few notes on how to migrate.

 Shall we make this release candidate our first incubator release?

 Lets keep the vote period short (We'll have to run another vote over in
 incubator general after this one if I understand the process properly).

 Please vote +1/0/-1 by Friday, January 15th, 2015.

 Thanks,
 St.Ack

 1.

 https://issues.apache.org/jira/issues/?jql=project%20%3D%20HTRACE%20AND%20status%20%3D%20resolved%20AND%20fixVersion%20%3D%203.1.0%20ORDER%20BY%20issuetype%20DESC

Re: NumTables/regions Quota management and RS groups - Core or coprocessors

[Gary Helmling]

I agree that quotas are a core issue.

It might be worth looking at why these features are easier to build as
coprocessors though (assuming that's true).  Could we be providing cleaner
separation in core APIs?

On Fri, Dec 5, 2014 at 1:35 PM, Francis Christopher Liu 
toffer@gmail.com wrote:

 Thanks for the response guys. Can I take it this is an affirmative for
 region server groups as well? HBASE-6721


 -Francis

 On Fri, Dec 5, 2014 at 10:17 AM, Ted Yu yuzhih...@gmail.com wrote:

  bq. something like quotas is a core issue
 
  +1
 
  On Fri, Dec 5, 2014 at 10:09 AM, Andrew Purtell apurt...@apache.org
  wrote:
 
   ​In my opinion something like quotas is a core issue. It might be nice
 to
   allow coprocessors to hook in, but that's a secondary consideration.​
  
   On Thu, Dec 4, 2014 at 6:50 PM, Enis Söztutar e...@apache.org wrote:
  
Hi,
   
We were chatting with Francis offline about these two features, which
  are
both related to NS and multi-tenant deployments.
   
The issue came up regarding whether these belong to core or they
 should
   be
left as coprocessor based implementations.
   
From a users perspective, if we leave these as co-processors, I feel
  that
they won't be used much and unless a very specific reason, most of
 the
users will not make use of them. However, especially for quotas for
example, I think that they should be enabled by default, and with
 some
reasonable default values (similar to how linux comes with default
  ulimit
settings).
   
We just wanted to bring this to dev to see what do you guys think.
   
https://issues.apache.org/jira/browse/HBASE-8410
https://issues.apache.org/jira/browse/HBASE-6721
   
  
  
  
   --
   Best regards,
  
  - Andy
  
   Problems worthy of attack prove their worth by hitting back. - Piet
 Hein
   (via Tom White)
  
 

[jira] [Created] (HBASE-12578) Change TokenProvider to a SingletonCoprocessorService

[Gary Helmling (JIRA)]

Gary Helmling created HBASE-12578:
-

 Summary: Change TokenProvider to a SingletonCoprocessorService
 Key: HBASE-12578
 URL: https://issues.apache.org/jira/browse/HBASE-12578
 Project: HBase
  Issue Type: Improvement
  Components: security
Reporter: Gary Helmling


The {{TokenProvider}} coprocessor service, which is responsible for issuing 
HBase delegation tokens, currently runs a region endpoint.  In the security 
documentation, we recommend configuring this coprocessor for all table regions, 
however, we only ever address delegation token requests to the META region.

When {{TokenProvider}} was first added, region coprocessors were the only way 
of adding endpoints.  But, since then, we've added support for endpoints for 
regionserver and master coprocessors.  This makes loading {{TokenProvider}} on 
all table regions unnecessarily wasteful.

We can reduce the overhead for {{TokenProvider}} and greatly improve it's 
scalability by doing the following:
# Convert {{TokenProvider}} to a {{SingletonCoprocessorService}} that is 
configured to run on all regionservers.  This will ensure a single instance per 
regionserver instead of one per region.
# Direct delegation token requests to a random running regionserver so that we 
don't hotspot any single instance with requests.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-12579) Move obtainAuthTokenForJob() methods out of User

[Gary Helmling (JIRA)]

Gary Helmling created HBASE-12579:
-

 Summary: Move obtainAuthTokenForJob() methods out of User
 Key: HBASE-12579
 URL: https://issues.apache.org/jira/browse/HBASE-12579
 Project: HBase
  Issue Type: Improvement
  Components: security
Reporter: Gary Helmling


The {{User}} class currently contains some utility methods to obtain HBase 
authentication tokens for the given user.  However, these methods initiate an 
RPC to the {{TokenProvider}} coprocessor endpoint, an action which should not 
be part of the User class' responsibilities.

This leads to a couple of problems:
# The way the methods are currently structured, it is impossible to integrate 
them with normal connection management for the cluster (the TokenUtil class 
constructs its own HTable instance internally).
# The User class is logically part of the hbase-common module, but uses the 
TokenUtil class (part of hbase-server, though it should probably be moved to 
hbase-client) through reflection, leading to a hidden dependency.

The {{obtainAuthTokenForJob()}} methods should be deprecated and the process of 
obtaining authentication tokens should be moved to use the normal connection 
lifecycle.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: A Chome extension for rendering patch on JIRA

[Gary Helmling]

Great stuff!  Thank you, David!

On Fri, Oct 10, 2014 at 5:21 PM, Stack st...@duboce.net wrote:
 On Fri, Oct 10, 2014 at 4:57 PM, David DENG davidden...@gmail.com wrote:


 https://chrome.google.com/webstore/detail/git-patch-viewer/hkoggakcdopbgnaeeidcmopfekipkleg

 Go to one of the JIRA patch, the raw text will be rendered as a Phabricator
 like page (Like http://review.facebook.net/ ).
 Recognized URL pattern:
 https://issues.apache.org/jira/secure/attachment/*/*.patch

 Source code and bug report: https://github.com/daviddengcn/patch-ext

 Have a try and enjoy!


 I've been using this a while now (Elliott showed it to me). Its very nice!
 Recommended!
 St.Ack

Re: hbase Observer POST PUT coprocesor not working

[Gary Helmling]

Do you see this log message from the start() method in your regionserver logs?

LOG.info((start));

If you don't see this, then there has been some problem loading your
coprocessor.  You can search for your coprocessor's classname to try
to track it down.

From your configuration and your code, of course, you should never see
this log line:

if (!Bytes.equals(table, Bytes.toBytes(SOURCE_TABLE))) {
 LOG.info(SOURCE_TABLE);

You've only configured the coprocessor to load on SOURCE_TABLE, so the
if condition should never return true.

If you are using HBase 0.98, then as Ted says, the postPut() method
signature is different.  You should uncomment the @Override annotation
on your method to enable compile time checking and fix the method
signature.  Without overriding the correct method, your method will
never be called.

On Mon, Oct 6, 2014 at 10:56 AM, Jasmin Guliani er.j...@gmail.com wrote:
 No, above logs are not showing up in region server. Also, I am using
 hbase-0.98

 On Mon, Oct 6, 2014 at 10:48 AM, Ted Yu yuzhih...@gmail.com wrote:

 Signature for postPut() in 0.98 is:
   public void postPut(final Put put, final WALEdit edit, final Durability
 durability)

 Which release are you using ?

 bq. LOG.info(SOURCE_TABLE);

 Did the above log show up in region server log ?

 On Mon, Oct 6, 2014 at 10:41 AM, Jasmin Guliani er.j...@gmail.com wrote:

 I have written coprocessor code with below code. After compiling this
 code,
 I have created the jar file. And then added this jar as a coprocessor to
 the table.

 alter 'SOURCE_TBL' , METHOD = 'table_att' , 'COPROCESSOR'=

 'hdfs:///apps/stsuid/HBASESamples-0.0.1-SNAPSHOT.jar|com.hbase.coprocessor.PutRegionObserverExample|1001'

 After this, when I am doing put operation on source table 'SOURCE_TBL'
 mentioned as below, I am expecting a record in index table 'INDEX_TBL' but
 there are no records in INDEX_TBL.

 put 'SOURCE_TBL' , '123' , 'data:CUST_ID' , 'RFRT'


 Please help me and let me know if my code of coprocessor is wrong or am I
 executing it in a wrong way.



 package com.hbase.coprocessor;

 import java.io.IOException;
 import java.util.Arrays;
 import java.util.List;
 import java.util.logging.Logger;

 import org.apache.hadoop.conf.Configuration;
 import org.apache.hadoop.hbase.Cell;
 import org.apache.hadoop.hbase.CoprocessorEnvironment;
 import org.apache.hadoop.hbase.HBaseConfiguration;
 import org.apache.hadoop.hbase.KeyValue;
 import org.apache.hadoop.hbase.client.Get;
 import org.apache.hadoop.hbase.client.HTable;
 import org.apache.hadoop.hbase.client.HTableInterface;
 import org.apache.hadoop.hbase.client.HTablePool;
 import org.apache.hadoop.hbase.client.Put;
 import org.apache.hadoop.hbase.coprocessor.BaseRegionObserver;
 import org.apache.hadoop.hbase.coprocessor.ObserverContext;
 import org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment;
 import org.apache.hadoop.hbase.regionserver.wal.WALEdit;
 import org.apache.hadoop.hbase.util.Bytes;

 public class PutRegionObserverExample extends BaseRegionObserver {

  public static final byte[] FIXED_ROW = Bytes.toBytes(@@@GETTIME@@@);
  private final static Logger LOG =
 Logger.getLogger(PutRegionObserverExample.class.getName());


  private HTablePool pool = null;

 private final static String  INDEX_TABLE  = INDEX_TBL;
 private final static String  SOURCE_TABLE = SOURCE_TBL;

 public void start(CoprocessorEnvironment env) throws IOException {

 LOG.info((start));
 pool = new HTablePool(env.getConfiguration(), 10);
  }

 //@Override
 public void postPut(
 final ObserverContextRegionCoprocessorEnvironment
 observerContext,
 final Put put,
 final WALEdit edit,
 final boolean writeToWAL)
 throws IOException {

 byte[] table =

 observerContext.getEnvironment().getRegion().getRegionInfo().getTableName();

 if (!Bytes.equals(table, Bytes.toBytes(SOURCE_TABLE))) {
  LOG.info(SOURCE_TABLE);
  return;
 }

 try {

  final ListCell filteredList = put.get(Bytes.toBytes
 (data),Bytes.toBytes(CUST_ID));
 byte [] id = put.getRow(); //Get the Entry ID

 Cell kv=filteredList.get( 0 ); //get Entry PARENT_ID
 byte[] parentId = kv.getValue();


HTableInterface htbl =
 pool.getTable(Bytes.toBytes(INDEX_TABLE));

//create row key for the index table
byte[] p1 = concatTwoByteArrays(parentId, :.getBytes());
 //Insert a semicolon between two UUIDs
byte[] rowkey = concatTwoByteArrays(p1, id);

Put indexput  = new Put(rowkey);

indexput.add(Bytes.toBytes(data), Bytes.toBytes(CUST_ID),
 Bytes.toBytes(tgt9));
htbl.put(indexput);

htbl.close();

 } catch ( IllegalArgumentException ex) {
 // handle excepion.
  LOG.info(ex.getMessage());
 }

 }


 @Override
 public void 

[ANNOUNCE] Tephra 0.3.0 Release

[Gary Helmling]

Hi all,

I'm happy to announce the 0.3.0 release of Tephra.

This release is a renaming of the project from Continuuity Tephra to
Cask Tephra, and includes the following changes:

* All packages have changed from com.continuuity.tephra to co.cask.tephra
* The Maven group ID has changed from com.continuuity.tephra to co.cask.tephra
* The github repository has moved to https://github.com/caskdata/tephra

If you have a current clone of the Tephra repository, please be sure
to re-clone from https://github.com/caskdata/tephra or to update your
git remote URL.

If you are currently using Tephra as a Maven dependency in any
project, please make note of the change to the groupId.  You will need
to update your dependency settings to something like the following:

dependency
  groupIdco.cask.tephra/groupId
  artifactIdtephra-api/artifactId
  version0.3.0/version
/dependency
dependency
  groupIdco.cask.tephra/groupId
  artifactIdtephra-core/artifactId
  version0.3.0/version
/dependency
dependency
  groupIdco.cask.tephra/groupId
  artifactIdtephra-hbase-compat-0.98/artifactId
  version0.3.0/version
/dependency

Release artifacts are available for download from:
https://github.com/caskdata/tephra/releases/tag/v0.3.0

For any questions or to get involved, please email the Tephra mailing
list at: tephra-...@googlegroups.com

[ANNOUNCE] Tephra 0.2.1 release

[Gary Helmling]

Hi all,

I'm happy to announce the 0.2.1 release of Tephra.

Tephra provides globally consistent transactions on top of Apache
HBase by leveraging HBase's native data versioning to provide
multi-versioned concurrency control (MVCC) for transactional reads and
writes. With MVCC capability, each transaction sees its own consistent
snapshot of data, providing snapshot isolation of concurrent
transactions.


This release fixes the following issues:

* TransactionProcessor use of FilterList on flush and compaction causes abort
* Support coverage report generation
* TransactionProcessor should use TransactionVisibilityFilter on flush
and compact
* Cleanup class and package names
* Use Configuration instead of HBaseConfiguration when possible
* TransactionVisibilityFilter should support additional filtering on
visible cells
* Assigned transaction IDs should reflect current timestamp
* Remove using of HBase Bytes in non-HBase related classes in tephra-core

Please note that a number of the Tephra packages and classes have been
renamed for clarity.  Any existing code will need to be updated.

Binary and source distributions of the release are available at:
https://github.com/continuuity/tephra/releases/tag/v0.2.1

For any questions or to get involved, please email the Tephra mailing
list at: tephra-...@googlegroups.com

[jira] [Created] (HBASE-11800) Coprocessor service methods in HTableInterface should be annotated public

[Gary Helmling (JIRA)]

Gary Helmling created HBASE-11800:
-

 Summary: Coprocessor service methods in HTableInterface should be 
annotated public
 Key: HBASE-11800
 URL: https://issues.apache.org/jira/browse/HBASE-11800
 Project: HBase
  Issue Type: Task
  Components: Client
Affects Versions: 0.96.0, 0.98.0
Reporter: Gary Helmling


The {{HTableInterface.coprocessorService(...)}} and 
{{HTableInterface.batchCoprocessorService(...)}} methods were made private in 
HBASE-9529, when the coprocessor APIs were seen as unstable and evolving.

However, these methods represent a standard way for clients to use custom APIs 
exposed via coprocessors.  In that sense, they are targeted at general HBase 
users (who may run but not develop coprocessors), as opposed to coprocessor 
developers who want to extend HBase.

The coprocessor endpoint API has also remained much more stable than the 
coprocessor Observer interfaces, which tend to change along with HBase 
internals.  So there should not be much difficulty in supporting these methods 
as part of the public API.

I think we should drop the {{@InterfaceAudience.Private}} annotation on these 
methods and support them as part of the public {{HTableInterface}}.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (HBASE-11653) RegionObserver coprocessor cannot override KeyValue values in prePut()

[Gary Helmling (JIRA)]

Gary Helmling created HBASE-11653:
-

 Summary: RegionObserver coprocessor cannot override KeyValue 
values in prePut()
 Key: HBASE-11653
 URL: https://issues.apache.org/jira/browse/HBASE-11653
 Project: HBase
  Issue Type: Bug
  Components: Coprocessors
Affects Versions: 0.94.21
Reporter: Gary Helmling
Assignee: Gary Helmling
Priority: Critical


Due to a bug in {{HRegion.internalPut()}}, any modifications that a 
{{RegionObserver}} makes to a Put's family map in the {{prePut()}} hook are 
lost.

This prevents coprocessors from modifying the values written by a {{Put}}.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

ANNOUNCE: Tephra for HBase transactions

[Gary Helmling]

Hi all,

I would like to introduce a new open source project, Continuuity Tephra,
which provides scalable, distributed transactions for Apache HBase.

Tephra provides snapshot isolation for concurrent transactions spanning
multiple regions, tables, and RPC calls.  A central transaction manager
provides globally unique, time-based transaction IDs and efficient conflict
detection.  Multiple transaction manager instances can be run to provide
automated failover that is transparent to clients. A simple client API,
including a drop-in replacement for HTable, makes it easy to work with
transactional operations.

Tephra was built to leverage core HBase features in order to operate
efficiently.  HBase cell versions provide multi-version concurrency control
for isolation.  Efficient transactional reads are performed using
server-side filtering, minimizing the amount of data scanned.  Coprocessors
hook into region flush and compaction operations to clean up data from
invalidated or no longer visible transactions.

Tephra is licensed under the Apache License, Version 2.0.

Tephra on Github (including docs):
https://github.com/continuuity/tephra

User and Developer Mailing List:
https://groups.google.com/d/forum/tephra-dev


Please take a look at Tephra and get involved to make it even better!


Gary Helmling

Re: Themis : implements cross-row/corss-table transaction on HBase.

[Gary Helmling]

Hi Jianwei,

You may also want to take a look at the generic client transaction API
being proposed in HBASE-11447:
https://issues.apache.org/jira/browse/HBASE-11447

I think it would be useful to have the Themis perspective there, and
whether the proposed API meets your needs and requirements.



On Tue, Jul 8, 2014 at 9:10 AM, Ted Yu yuzhih...@gmail.com wrote:

 Jianwei:
 You may want to update the comment for ThemisScan :

 //a wrapper class of Put in HBase which not expose timestamp to user
 public class ThemisScan extends ThemisRead {

 Is there plan to support append / increment as part of the transaction ?

 Currently Themis depends on 0.94.11
 Is there plan to support 0.96+ releases ?

 Thanks


 On Tue, Jul 8, 2014 at 12:34 AM, 崔建伟 cuijian...@xiaomi.com wrote:

  Hi everyone, I want to introduce our open-source project Themis which
  implements cross-row/corss-table transaction on HBase.
 
  Themis follows google's percolator algorithm(
  http://research.google.com/pubs/pub36726.html), which provides
  ACID-compliant transaction and snapshot isolation. The cross-row
  transaction is based on HBase's single-row atomic semantics and doesn't
 use
  a central transaction server, so that supports linear-scalability.
 
  Themis depends on a timestamp server to provides global strictly
  incremental timestamp to define the order of transactions, which will be
  used to resolve the write-write and read-write conflicts. The timestamp
  server is lightweight and could achieve hight throughput(500, 000 + qps),
  and Themis will batch timestamp requests across transactions in one Rpc,
 so
  that it won't become the bottleneck of the system even when processing
  billions of transactions every day.
 
  Although Themis could be implemented totally in client-side, we adopt
  coprocessor framework of HBase to achieve higher performance. Themis
  includes a client-side library to provides transaction APIs, such as
  themisPut/themisGet/themisScan/themisDelete, and a coprocessor library
  loaded on regionserver. Therefore, Themis could be used without changing
  the code and logic of HBase.
 
  We have been validating the correctness of Themis for a few months by a
  AccountTransfer simulation program, which concurrently does cross-row
  transactions by transferring money among different accounts(each account
 is
  a row in HBase) and verifies total money of all accounts doesn't change
 in
  the simulation. We have also run Themis on our production environment.
 
  We test the performance of Themis and get comparable result as
 percolator.
  The single-column transaction represents the worst performance case for
  Themis compared with HBase, the result is:
  1) For read, the performance of percolator is 90% of HBase;
  2) For write, the performance of percolator is 23% of HBase.
  The write performance drops a lot because Themis uses two-phase commit
  protocol to achieve ACID of transaction. For multi-row write, we improve
  the performance by paralleling all writes of pre-write phase. For
  single-row write, we are optimizing two-phase commit protocol to achieve
  better performance and will update the result when it is ready. The
 details
  of performance result could be found in github.
 
  The repository and introduction of Themis include:
  1. Themis github: https://github.com/XiaoMi/themis/. The source code,
  performance test result and user guide could be found here.
  2. Themis jira : https://issues.apache.org/jira/browse/HBASE-10999
  3. Chronos github: https://github.com/XiaoMi/chronos. Chronos is our
  open-source high-availability, high-performance timestamp server to
 provide
  global strictly incremental timestamp for Themis.
 
  If you find Themis interesting, please leave us comment in the mail, jira
  or github.
 
  Best
  cuijianwei
 

[jira] [Created] (HBASE-11292) Add an undelete operation

[Gary Helmling (JIRA)]

Gary Helmling created HBASE-11292:
-

 Summary: Add an undelete operation
 Key: HBASE-11292
 URL: https://issues.apache.org/jira/browse/HBASE-11292
 Project: HBase
  Issue Type: New Feature
  Components: Deletes
Reporter: Gary Helmling


While column families can be configured to keep deleted cells (allowing time 
range queries to still retrieve those cells), deletes are still somewhat unique 
in that they are irreversible operations.  Once a delete has been issued on a 
cell, the only way to undelete it is to rewrite the data with a timestamp 
newer than the delete.

The idea here is to add an undelete operation, that would make it possible to 
cancel a previous delete.  An undelete operation will be similar to a delete, 
in that it will be written as a marker (tombstone doesn't seem like the right 
word).  The undelete marker, however, will sort prior to a delete marker, 
canceling the effect of any following delete.

In the absence of a column family configured to KEEP_DELETED_CELLS, we can't be 
sure if a prior delete marker and the effected cells have already been garbage 
collected.  In this case (column family not configured with KEEP_DELETED_CELLS) 
it may be necessary for the server to reject undelete operations to avoid 
creating the appearance of a client contact for undeletes that can't reliably 
be honored.

I think there are additional subtleties of the implementation to be worked out, 
but I'm also interested in a broader discussion of interest in this capability.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: VOTE: Move to GIT

[Gary Helmling]

+1


On Mon, May 19, 2014 at 11:18 AM, Elliott Clark ecl...@apache.org wrote:

 +1


 On Mon, May 19, 2014 at 11:11 AM, yao yaosheng...@gmail.com wrote:

  +1 (can anyone +2 this ?)
 
 
  On Mon, May 19, 2014 at 9:38 AM, Jimmy Xiang jxi...@cloudera.com
 wrote:
 
   +1
  
  
   On Mon, May 19, 2014 at 9:24 AM, Andrew Purtell apurt...@apache.org
   wrote:
  
+1
   
   
On Mon, May 19, 2014 at 8:32 AM, Stack st...@duboce.net wrote:
   
 Following up on the old DISCUSSION thread on moving to git [1],
 lets
   vote
 (INFRA needs to see a vote).

 Move to GIT?

 [  ] +1 on yes, lets move to GIT
 [  ] 0 on don't care
 [  ] -1 on stay w/ current SVN setup.

 I'm +1.

 St.Ack
 P.S. The Mighty Talat Uyarer, who volunteered to run the migration
 a
while
 ago is back with a vengeance and up for the job.

 1. http://search-hadoop.com/m/WmXbV15hqqq

   
   
   
--
Best regards,
   
   - Andy
   
Problems worthy of attack prove their worth by hitting back. - Piet
  Hein
(via Tom White)