[jira] [Created] (HBASE-17002) [Master] Report quotas and active violations on Master UI

[Josh Elser (JIRA)]

Josh Elser created HBASE-17002:
--

 Summary: [Master] Report quotas and active violations on Master UI
 Key: HBASE-17002
 URL: https://issues.apache.org/jira/browse/HBASE-17002
 Project: HBase
  Issue Type: Sub-task
  Components: master, UI
Reporter: Josh Elser


We should be able to view the Master UI and determine what quotas exist and the 
quotas that are in violation to easily confirm/deny the state of 
tables/namespaces.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-17001) [RegionServer] Implement enforcement of quota violation policies

[Josh Elser (JIRA)]

Josh Elser created HBASE-17001:
--

 Summary: [RegionServer] Implement enforcement of quota violation 
policies
 Key: HBASE-17001
 URL: https://issues.apache.org/jira/browse/HBASE-17001
 Project: HBase
  Issue Type: Sub-task
  Components: regionserver
Reporter: Josh Elser


When the master enacts a quota violation policy, the RegionServers need to 
actually enforce that policy per its definition.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-17000) [RegionServer] Compute region filesystem space use

[Josh Elser (JIRA)]

Josh Elser created HBASE-17000:
--

 Summary: [RegionServer] Compute region filesystem space use
 Key: HBASE-17000
 URL: https://issues.apache.org/jira/browse/HBASE-17000
 Project: HBase
  Issue Type: Sub-task
  Components: regionserver
Reporter: Josh Elser


Each RegionServer needs to track how much space a Region takes up and roll this 
up to the table level.

Aggregation of this information in the Master will be covered by HBASE-16997.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-16999) [Master] Inform RegionServers on size quota violations

[Josh Elser (JIRA)]

Josh Elser created HBASE-16999:
--

 Summary: [Master] Inform RegionServers on size quota violations
 Key: HBASE-16999
 URL: https://issues.apache.org/jira/browse/HBASE-16999
 Project: HBase
  Issue Type: Sub-task
Reporter: Josh Elser


The Master, after computing the total usage across all regionservers, needs to 
inform the RegionServers about tables that need to change violation policy 
enforcement (enable or disable enforcement).

RPC, ZK, or ProcV2's notification bus are some examples of ways this could be 
implemented.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-16998) [Master] Analyze table use reports and update quota violations

[Josh Elser (JIRA)]

Josh Elser created HBASE-16998:
--

 Summary: [Master] Analyze table use reports and update quota 
violations
 Key: HBASE-16998
 URL: https://issues.apache.org/jira/browse/HBASE-16998
 Project: HBase
  Issue Type: Sub-task
Reporter: Josh Elser


Given the collected table usage reports from RegionServers, the Master needs to 
inspect all filesystem-use quotas and determine which need to move into 
violation and which need to move out of violation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-16997) Collect table use information from RegionServers in Master

[Josh Elser (JIRA)]

Josh Elser created HBASE-16997:
--

 Summary: Collect table use information from RegionServers in Master
 Key: HBASE-16997
 URL: https://issues.apache.org/jira/browse/HBASE-16997
 Project: HBase
  Issue Type: Sub-task
Reporter: Josh Elser


The Master will need to get reports of table usage in the cluster from 
RegionServers.

RegionServers could report this to the Master or the Master could poll this 
information from the RegionServers. Need to determine which model is more 
appropriate given the importance of applying quotas (e.g. we do not want 
quota-use calculation to impact region assignment).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-16996) Implement storage/retrieval of filesystem-use quotas into quota table

[Josh Elser (JIRA)]

Josh Elser created HBASE-16996:
--

 Summary: Implement storage/retrieval of filesystem-use quotas into 
quota table
 Key: HBASE-16996
 URL: https://issues.apache.org/jira/browse/HBASE-16996
 Project: HBase
  Issue Type: Sub-task
Reporter: Josh Elser


Provide read/write API for accessing the new filesystem-usage quotas in the 
existing {{hbase:quota}} table.

Make sure that both the client can read quotas the quotas in the table as well 
as the Master can perform the necessary update/delete actions per the quota 
RPCs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-16995) Build client Java API and client protobuf messages

[Josh Elser (JIRA)]

Josh Elser created HBASE-16995:
--

 Summary: Build client Java API and client protobuf messages
 Key: HBASE-16995
 URL: https://issues.apache.org/jira/browse/HBASE-16995
 Project: HBase
  Issue Type: Sub-task
  Components: Client
Reporter: Josh Elser
Assignee: Josh Elser


Extend the existing Java API and protobuf messages to allow the client to set 
filesystem-use quotas via the Master.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] FileSystem Quotas in HBase

[Josh Elser]

Thanks for the reviews so far, Ted and Stack. The comments were great 
and much appreciated.


Interpreting consensus from lack of objection, I'm going to move ahead 
in earnest starting to work on what was described in the doc. Expect to 
see some work break-out happening under HBASE-16961 and patches starting 
to land.


I'm also happy to entertain more discussion if anyone hasn't found the 
time to read/comment yet.


Thanks!

- Josh

Josh Elser wrote:

Sure thing, Ted.

https://docs.google.com/document/d/1VtLWDkB2tpwc_zgCNPE1ulZOeecF-YA2FYSK3TSs_bw/edit?usp=sharing


Let me open an umbrella issue for now. I can break up the work later.

https://issues.apache.org/jira/browse/HBASE-16961

Ted Yu wrote:

Josh:
Can you put the doc in google doc so that people can comment on it ?

Is there a JIRA opened for this work ?
Please open one if there is none.

Thanks

On Fri, Oct 28, 2016 at 9:00 AM, Josh Elser<els...@apache.org> wrote:


Hi folks,

I'd like to propose the introduction of FileSystem quotas to HBase.

Here's a design doc[1] available which (hopefully) covers all of the
salient points of what I think an initial version of such a feature
would
include.

tl;dr We can define quotas on tables and namespaces. Region size is
computed by RegionServers and sent to the Master. The Master inspects
the
sizes of Regions, rolling up to table and namespace sizes. Defined
quotas
in the quota table are evaluated given the computed sizes, and, for
those
tables/namespaces violating the quota, RegionServers are informed to
take
some action to limit any further filesystem growth by that
table/namespace.

I'd encourage you to give the document a read -- I tried to cover as
much
as I could without getting unnecessarily bogged down in implementation
details.

Feedback is, of course, welcomed. I'd like to start sketching out a
breakdown of the work (all writing and no programming makes Josh a sad
boy). I'm happy to field any/all questions. Thanks in advance.

- Josh

[1] http://home.apache.org/~elserj/hbase/FileSystemQuotasforApac
heHBase.pdf

[jira] [Created] (HBASE-16976) hbase-protocol-shaded module generates classes to wrong directory

[Josh Elser (JIRA)]

Josh Elser created HBASE-16976:
--

 Summary: hbase-protocol-shaded module generates classes to wrong 
directory
 Key: HBASE-16976
 URL: https://issues.apache.org/jira/browse/HBASE-16976
 Project: HBase
  Issue Type: Bug
  Components: build
Reporter: Josh Elser
Assignee: Josh Elser
 Fix For: 2.0.0


I've been fighting with getting master to import/build normally inside of 
Eclipse (a bit seems to have come in as a part of the shaded protocol work).

Once thing I see definitely wrong is that Eclipse is balking at the source 
files being compiled in {{target/}} instead of {{target/classes}}. This looks 
like an omission since there is no good reason to not use the standard 
convention of {{target/classes}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-16961) FileSystem Quotas

[Josh Elser (JIRA)]

Josh Elser created HBASE-16961:
--

 Summary: FileSystem Quotas
 Key: HBASE-16961
 URL: https://issues.apache.org/jira/browse/HBASE-16961
 Project: HBase
  Issue Type: New Feature
Reporter: Josh Elser
Assignee: Josh Elser


Umbrella issue for tracking the filesystem utilization of HBase data, defining 
quotas on that utilization, and enforcement when utilization exceeds the limits 
of the quota.

At a high level: we can define quotas on tables and namespaces. Region size is 
computed by RegionServers and sent to the Master. The Master inspects the sizes 
of Regions, rolling up to table and namespace sizes. Defined quotas in the 
quota table are evaluated given the computed sizes, and, for those 
tables/namespaces violating the quota, RegionServers are informed to take some 
action to limit any further filesystem growth by that table/namespace.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] FileSystem Quotas in HBase

[Josh Elser]

Sure thing, Ted.

https://docs.google.com/document/d/1VtLWDkB2tpwc_zgCNPE1ulZOeecF-YA2FYSK3TSs_bw/edit?usp=sharing

Let me open an umbrella issue for now. I can break up the work later.

https://issues.apache.org/jira/browse/HBASE-16961

Ted Yu wrote:

Josh:
Can you put the doc in google doc so that people can comment on it ?

Is there a JIRA opened for this work ?
Please open one if there is none.

Thanks

On Fri, Oct 28, 2016 at 9:00 AM, Josh Elser<els...@apache.org>  wrote:


Hi folks,

I'd like to propose the introduction of FileSystem quotas to HBase.

Here's a design doc[1] available which (hopefully) covers all of the
salient points of what I think an initial version of such a feature would
include.

tl;dr We can define quotas on tables and namespaces. Region size is
computed by RegionServers and sent to the Master. The Master inspects the
sizes of Regions, rolling up to table and namespace sizes. Defined quotas
in the quota table are evaluated given the computed sizes, and, for those
tables/namespaces violating the quota, RegionServers are informed to take
some action to limit any further filesystem growth by that table/namespace.

I'd encourage you to give the document a read -- I tried to cover as much
as I could without getting unnecessarily bogged down in implementation
details.

Feedback is, of course, welcomed. I'd like to start sketching out a
breakdown of the work (all writing and no programming makes Josh a sad
boy). I'm happy to field any/all questions. Thanks in advance.

- Josh

[1] http://home.apache.org/~elserj/hbase/FileSystemQuotasforApac
heHBase.pdf

[DISCUSS] FileSystem Quotas in HBase

[Josh Elser]

Hi folks,

I'd like to propose the introduction of FileSystem quotas to HBase.

Here's a design doc[1] available which (hopefully) covers all of the 
salient points of what I think an initial version of such a feature 
would include.


tl;dr We can define quotas on tables and namespaces. Region size is 
computed by RegionServers and sent to the Master. The Master inspects 
the sizes of Regions, rolling up to table and namespace sizes. Defined 
quotas in the quota table are evaluated given the computed sizes, and, 
for those tables/namespaces violating the quota, RegionServers are 
informed to take some action to limit any further filesystem growth by 
that table/namespace.


I'd encourage you to give the document a read -- I tried to cover as 
much as I could without getting unnecessarily bogged down in 
implementation details.


Feedback is, of course, welcomed. I'd like to start sketching out a 
breakdown of the work (all writing and no programming makes Josh a sad 
boy). I'm happy to field any/all questions. Thanks in advance.


- Josh

[1] http://home.apache.org/~elserj/hbase/FileSystemQuotasforApacheHBase.pdf

Re: [DISCUSS] Drop the support of jdk7 at a future 1.x release

[Josh Elser]

Sean Busbey wrote:

This would be a very big breaking change in release numbering that goes
against our compatibility guidelines. We only drop support for Java
versions on major releases.

If we want features that require newer jdks sooner, we should make major
releases sooner.


+1 On this exactly


On Sep 8, 2016 20:54, "Duo Zhang"  wrote:


The main reason is the asynchronous api we want to introduce in HBase
today. See HBASE-13784 and HBASE-16505.

The CompletableFuture in java8 is very suitable to use as the return value
of a async method. We can not use it if we still want to support java7, and
sadly, there is no candidate which is good enough to replace
CompletableFuture. ListenableFuture in guava or Promise in netty are good,
but we do not want to expose third-party classes in our public
API(especially guava, you know...). And we can also implement our own
ListenableFuture but it just a copy of guava. Or introduce a simple
Callback interface which does not need much code(for us) but this is a code
style around 2000s so people will not like it...

And luckily, I found that in our documentation

http://hbase.apache.org/book.html#basic.prerequisites

We only say that 1.3 will be compatible with jdk7, not all 1.x.

So here I propose that we drop the support of jdk7 in a future 1.x release,
maybe 1.4? Thus we can use CompletableFuture in both master and branch-1.

Thanks.

Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912

[Josh Elser]

So, given your stated concerns, Sean, the correctness-in-face-of-failure 
tests is the only WIP thing remaining that you'd like to see, right? Are 
there any other issues that you can think of now that would prevent you 
from +1'ing?


Regarding the move to a "higher-bar for new features" by the community, 
I'll stand back because I know that I have not been following closely 
enough to comment here. If this is a community decision that everyone is 
on board with, great. I honestly have not followed discussions closely 
enough to understand whether this is a "we'd like to do this" or "we are 
doing this".


This leads me to asking how can we address the concern on the amorphous 
state of what "HBase-2.0" and how to avoid this blocking contributions 
to 2.0. It seems like there's a chicken-and-egg problem with being 
unable to determine the expected level of testing while there's no 
concrete roadmap for 2.0. To me, it seems premature to raise the bar 
higher in this case (because there is no firm date where continued work 
must be completed by to avoid blocking the release), but, again, I type 
this with great hesitance (as I have not be diligent with my inbox).


Vladimir Rodionov wrote:

Sean,

* have docs

Agree. We have a doc and backup is the most documented feature :), we will
release it shortly to Apache.

* have sunny-day correctness tests

Feature has  close to 60 test cases, which run for approx 30 min. We can
add more, if community do not mind :)

* have correctness-in-face-of-failure tests

Any examples of these tests in existing features? In works, we have a clear
understanding of what should be done by the time of 2.0 release.
That is very close goal for us, to verify IT monkey for existing code.

* don't rely on things outside of HBase for normal operation (okay for
advanced operation)

We do not.

Enormous time has been spent already on the development and testing the
feature, it has passed our internal tests and many rounds of code reviews
by HBase committers. We do not mind if someone from HBase community
(outside of HW) will review the code, but it will probably takes forever to
wait for volunteer?, the feature is quite large (1MB+ cumulative patch)

2.0 branch is full of half baked features, most of them are in active
development, therefore I am not following you here, Sean? Why HBASE-7912 is
not good enough yet to be integrated into 2.0 branch?

-Vlad

On Wed, Sep 7, 2016 at 8:23 AM, Sean Busbey<bus...@apache.org>  wrote:


On Tue, Sep 6, 2016 at 10:36 PM, Josh Elser<josh.el...@gmail.com>  wrote:

So, the answer to Sean's original question is "as robust as snapshots
presently are"? (independence of backup/restore failure tolerance from
snapshot failure tolerance)

Is this just a question WRT context of the change, or is it means for a

veto

from you, Sean? Just trying to make sure I'm following along adequately.



I'd say ATM I'm -0, bordering on -1 but not for reasons I can articulate
well.

Here's an attempt.

We've been trying to move, as a community, towards minimizing risk to
downstream folks by getting "complete enough for use" gates in place
before we introduce new features. This was spurred by a some features
getting in half-baked and never making it to "can really use" status
(I'm thinking of distributed log replay and the zk-less assignment
stuff, I don't recall if there was more).

The gates, generally, included things like:

* have docs
* have sunny-day correctness tests
* have correctness-in-face-of-failure tests
* don't rely on things outside of HBase for normal operation (okay for
advanced operation)

As an example, we kept the MOB work off in a branch and out of master
until it could pass these criteria. The big exemption we've had to
this was the hbase-spark integration, where we all agreed it could
land in master because it was very well isolated (the slide away from
including docs as a first-class part of building up that integration
has led me to doubt the wisdom of this decision).

We've also been treating inclusion in a "probably will be released to
downstream" branches as a higher bar, requiring

* don't moderately impact performance when the feature isn't in use
* don't severely impact performance when the feature is in use
* either default-to-on or show enough demand to believe a non-trivial
number of folks will turn the feature on

The above has kept MOB and hbase-spark integration out of branch-1,
presumably while they've "gotten more stable" in master from the odd
vendor inclusion.

Are we going to have a 2.0 release before the end of the year? We're
coming up on 1.5 years since the release of version 1.0; seems like
it's about time, though I haven't seen any concrete plans this year.
Presuming we are going to have one by the end of the year, it seems a
bit close to still be adding in "features that need maturing" on the
branch.

The lack of

Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912

[Josh Elser]

So, the answer to Sean's original question is "as robust as snapshots 
presently are"? (independence of backup/restore failure tolerance from 
snapshot failure tolerance)


Is this just a question WRT context of the change, or is it means for a 
veto from you, Sean? Just trying to make sure I'm following along 
adequately.


Vladimir Rodionov wrote:

Snapshot robustness is better now with introduction of region splits/merges
on/off feature. Region splits during snapshots was the major problem.

-Vlad

On Fri, Sep 2, 2016 at 8:12 AM, Vladimir Rodionov
wrote:


Are they independent enough that we can get backup/restore tolerant to
failures prior to merge to master? Prior to backport to branch-1?

As we stated already, snapshots are not part of the feature, snapshots has
been merged into the master long time ago
and as far as I understood - without requiring them to be 100% robust and
fault tolerant and they are widely used in many production systems
nevertheless. https://issues.apache.org/jira/browse/HBASE-14415 relies on
Snapshots v2 but we can reconsider it, there are some thoughts how to make
backups snapshotless.

Backups are fault tolerant to some extent - in case of failure (and
failures can happen) we clean everything up and do not leave system table
in inconsistent state. Would it be enough, Sean Busbey?

-Vlad

On Fri, Sep 2, 2016 at 7:38 AM, Ted Yu  wrote:


We're continuing to make backup / restore more robust.
Work in progress (both are close to being integrated):

HBASE-15565 Rewrite restore with Procedure V2
HBASE-15449 Support physical table layout change

Since snapshot is dependency in the full backup, backup / restore wouldn't
be more robust than snapshot is.

On Fri, Sep 2, 2016 at 7:03 AM, Sean Busbey  wrote:


right, they're separate features but when asked about "robust
backup/restore" (which is what I care about for this feature getting
merged) things were pawned off on snapshots.

Are they independent enough that we can get backup/restore tolerant to
failures prior to merge to master? Prior to backport to branch-1?

On Thu, Sep 1, 2016 at 1:11 PM, Andrew Purtell
wrote:

I agree these are separate features FWIW

On Thu, Sep 1, 2016 at 11:09 AM, Vladimir Rodionov<

vladrodio...@gmail.com>

wrote:


Do we have JIRA issue(s) covering making snapshots robust in the

face

of monkeys?

I would like to mention that "robust snapshots" and "table

backup/restore"

are totally separate features, but we have separate JIRA for fault
tolerance (HBASE-14413).

-Vlad

On Thu, Sep 1, 2016 at 9:28 AM, Ted Yu  wrote:


Sean:
Please see HBASE-14413 for the last question.

FYI

On Thu, Sep 1, 2016 at 9:24 AM, Sean Busbey

wrote:

On Sat, Aug 20, 2016 at 12:59 PM, Vladimir Rodionov
  wrote:

Not sure what do you mean, Andrew by "trying out the branch via

the

IT",

but we do not recommend running this with monkey enabled.
It has not been tested in a such scenario yet and frankly

speaking it

is

not supposed to work (snapshots will fail anyway and we

depends on

snapshots)



Also won't have time to test out the branch this week, but if

we're

not going to handle failures do we have tools or guidance on
recovering in the case of things falling over?

Do we have JIRA issue(s) covering making snapshots robust in the

face

of monkeys?

--
busbey




--
Best regards,

- Andy

Problems worthy of attack prove their worth by hitting back. - Piet

Hein

(via Tom White)



--
busbey

Re: Source incompatibility between 1.2.[0,1] and 1.2.2 breaks Phoenix

[Josh Elser]

(top-post since I can't find a better place to respond to everyone who 
chimed in here)


Huge thanks, everyone! This was absolutely the best email thread (and 
JIRA issue) I could've come back to after not keeping up with email for 
the day.


Stack wrote:

On Tue, Aug 16, 2016 at 8:20 AM, Sean Busbey  wrote:


On Tue, Aug 16, 2016 at 6:40 AM, Stack  wrote:


On Tue, Aug 16, 2016 at 1:07 AM, Phil Yang  wrote:


I notice that HBASE-15645 is made up of several commits and revert in

git,

maybe it is more convenient to apply a new patch to remove the added
methods.

Created a new issue (HBASE-16420
) and waiting for

the

result of pre-commit build. The patch only fix the incompatibility of

1.1

and 1.2.  Do we need keep the compatibility between 1.x branches? If so

we

need also remove new methods from branch-1.3 and branch-1. And seems

some

other issues also added new methods to  non-Private  interface on
branch-1/1.3...



Patch looks good Phil. Thanks for putting it up.

On your question, adding API in a manner that breaks compile is allowed
going by our updated understanding.



This should be "is not allowed" right?



Thanks Sean. Yes (Doc says right thing in HBASE-16422 patch)




My reading of the consensus in the thread thus far is that adding methods
to IA.Public interfaces won't be possible in the 1.y line of releases due
to breaking source compatibility,




HBASE-16422 makes exclusion for patch release only. It does not preclude
our breaking on minor versions.




1) starting to have a distinction between places we expect folks to just
consume API vs extend it?

I used to be in favor of this, but Andrew's concern on bikeshedding has me
reconsidering. Simpler rules seem better both to reduce the complexity of
communicating downstream and what the RMs have to keep in their heads when
evaluating changes.



This and the below would be better on another thread I'd say Sean.

Lets keep this thread for handling this little compat episode.

Thanks,
St.Ack




2) dropping our use of IS.Stable, IS.Evolving, etc.

I've never found this distinction particularly useful since we aren't very
consistent on it. The compat guide only specifies what it means for "server
side APIs" without really defining what that means precisely, and we use it
in client APIs as well. I think a reasonable reading of our current guide
could take the IS.Evolving designation to mean that the breaking change to
Table was okay, but reading the comments on PHOENIX-3116 that
interpretation would clearly be surprising to at least one set of
downstream folks. (Plus none of the discussions I saw around this change
used this rationale, so its presence or lack thereof wouldn't have changed
the conversation.)

Re: Source incompatibility between 1.2.[0,1] and 1.2.2 breaks Phoenix

[Josh Elser]

Thanks for the great reply, Andrew!

Andrew Purtell wrote:

​
  I find the InterfaceAudience annotations on this really strange. How can
we have a Public audience Interface (o.a.h.h.c.Table) with Private methods?

I'm also not sure the Private annotations on the Table interface are that
useful. Any change to an interface renders it source incompatible, and
removal (or effective removal via signature change) of methods from an
interface introduces a binary incompatibility. I think the Private
annotations on methods of a Public interface imply we should refactor those
methods to a non-public interface.


I agree. This is how I would've expected to see such an 
non-public-facing method to have been added.



Now that we've had quite a few releases in the "not-quite-SemVer"
compatibility guide, is there any interest in trying to make the
compatibility guarantees more stringent?

I was looking at our compat guidelines (
http://hbase.apache.org/book.html#hbase.versioning) and think we could make
a few refinements.

We make several allowances for public interface changes that are binary
compatible but not source compatible in patch releases. I think we are only
taking into consideration callers but should also consider implementors of
public interfaces. Changing a public interface on a patch release renders
it source incompatible. Can we avoid doing that on *patch* releases, and
restrict this type of change to minors?

Although we make allowances for public interface changes that are binary
compatible we also say "A minor upgrade requires no application/client code
modification." which could imply source compatibility even across minors,
which I believe is not the intent.

We could add a source compatibility dimension for patch releases.


I like the way this sounds. Would it make sense to try to work on 
terminology to encapsulate this (after getting some more consensus that 
this is desirable)?



I would like to see us get to source-compatibility on patch releases, not
just binary compatibility

You mean source compatibility for Public annotated interfaces only, right?

In that case evaluating compliance would be easy: RMs would run the API
compat checker on a RC and if a patch release the number of binary and
source compat issues should both be zero.


Yes, sorry, I could've been more specified. Source-compatibility on the 
"public API" (defined presently by the Public audience annotation).

Source incompatibility between 1.2.[0,1] and 1.2.2 breaks Phoenix

[Josh Elser]

Hi folks,

I just noticed a ticket over in Phoenix [1] in which some method 
additions to the Table interface [2] breaks the source compatibility of 
Phoenix with HBase 1.2.2.


My understanding of the current guidelines is that these are allowed as 
they do not invalidate binary compatibility of clients using the API. 
Personally, I am very hard-pressed to use the word "compatible" in a 
sentence describing this change that isn't sarcastic :)


A couple of questions:

1) I find the InterfaceAudience annotations on this really strange. How 
can we have a Public audience Interface (o.a.h.h.c.Table) with Private 
methods? Is that just "how things are", or am I missing something? If 
this is not something that's meant to be public, I would think these new 
methods should be defined in a non-public interface.


2) Now that we've had quite a few releases in the "not-quite-SemVer" 
compatibility guide, is there any interest in trying to make the 
compatibility guarantees more stringent? I would like to see us get to 
source-compatibility on patch releases, not just binary compatibility. I 
am happy to try to help, but I know I don't have the time to devote to 
catch everything.


3) What do people think about changing this in a 1.2.3 with a quick 
turn-around?


Thanks!

- Josh

[1] https://issues.apache.org/jira/browse/PHOENIX-3116
[2] https://issues.apache.org/jira/browse/HBASE-15645

[jira] [Created] (HBASE-16376) Document implicit side-effects on partial results when calling Scan#setBatch(int)

[Josh Elser (JIRA)]

Josh Elser created HBASE-16376:
--

 Summary: Document implicit side-effects on partial results when 
calling Scan#setBatch(int)
 Key: HBASE-16376
 URL: https://issues.apache.org/jira/browse/HBASE-16376
 Project: HBase
  Issue Type: Task
  Components: API, documentation
Reporter: Josh Elser
Assignee: Josh Elser
 Fix For: 2.0.0, 1.3.0, 1.2.3, 0.98.22


It was brought to my attention that the javadoc on {{Scan#setBatch(int)}} does 
not inform the user that calling this method has the implicit side-effect that 
the user may see partial {{Result}}s.

While the side-effect isn't necessarily surprising for developers who know how 
it's implemented, but for API users, this might be a very jarring implication.

We should update the documentation on {{Scan#setBatch(int)}} to inform users 
that they may see partial results if they call this method (and perhaps refer 
them to the size-based {{Scan#setMaxResultSize(long)}} too).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-16043) Backport SPNEGO support to 0.98

[Josh Elser (JIRA)]

Josh Elser created HBASE-16043:
--

 Summary: Backport SPNEGO support to 0.98
 Key: HBASE-16043
 URL: https://issues.apache.org/jira/browse/HBASE-16043
 Project: HBase
  Issue Type: Sub-task
Reporter: Josh Elser
Assignee: Josh Elser
 Fix For: 0.98.21


The initial backport work of the parent issue to 0.98 was more difficult than I 
expected. 0.98 uses Hadoop's HttpServer whereas >=1.x uses a copied variant of 
that HttpServer class (that has significantly diverged since the copy).

Initial testing showed problems with the default configuration -- need to 
figure out what the difference is between the two (or just not backport this to 
0.98).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-15837) More gracefully handle a negative memstoreSize

[Josh Elser (JIRA)]

Josh Elser created HBASE-15837:
--

 Summary: More gracefully handle a negative memstoreSize
 Key: HBASE-15837
 URL: https://issues.apache.org/jira/browse/HBASE-15837
 Project: HBase
  Issue Type: Improvement
  Components: regionserver
Reporter: Josh Elser
Assignee: Josh Elser
 Fix For: 2.0.0


Over in PHOENIX-2883, I've been trying to figure out how to track down the root 
cause of an issue we were seeing where a negative memstoreSize was ultimately 
causing an RS to abort. The tl;dr version is

* Something causes memstoreSize to be negative (not sure what is doing this yet)
* All subsequent flushes short-circuit and don't run because they think there 
is no data to flush
* The region is eventually closed (commonly, for a move).
* A final flush is attempted on each store before closing (which also 
short-circuit for the same reason), leaving unflushed data in each store.
* The sanity check that each store's size is zero fails and the RS aborts.

I have a little patch which I think should improve our failure case around 
this, preventing the RS abort safely (forcing a flush when memstoreSize is 
negative) and logging a calltrace when an update to memstoreSize make it 
negative (to find culprits in the future).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-15660) Printing extra refs in ITBLL.Verify reducer can cause reducers to be killed due to lack of progress

[Josh Elser (JIRA)]

Josh Elser created HBASE-15660:
--

 Summary: Printing extra refs in ITBLL.Verify reducer can cause 
reducers to be killed due to lack of progress
 Key: HBASE-15660
 URL: https://issues.apache.org/jira/browse/HBASE-15660
 Project: HBase
  Issue Type: Bug
  Components: integration tests
Reporter: Josh Elser
Assignee: Josh Elser
Priority: Minor
 Fix For: 2.0.0, 1.1.5, 1.2.2


In debugging an ITBLL job which has numerous failures, I saw that instead of 
the Verify job completing and reporting that there were a large number of UNDEF 
nodes, the reducers in the Verify were failing due to lack of progress.

The reducer's syslog file was filled with information from the 
{{dumpExtraInfoOnRefs()}} method. I believe that when a reducer is repeatedly 
doing these lookups, the MR framework doesn't realize that any progress is 
being made (nothing is being written to the context) and eventually kills the 
reducer task. This ultimately causes the entire Verify job to fail because the 
reducer fails in the same manner each time.

We should make sure to invoke {{context.progress()}} when we do these lookups 
to let the framework know that we're still doing "our thing".



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Backporting to active branches

[Josh Elser]

Stack wrote:

...
>  Let's change our relationship slightly, dev community. If you're working on
>  a fix, please also post a patch for branch-1.1.



It is a bit tough. That'd be a patch for four branches (at least), three of
which have diverged in key areas (master, branch-1 and branch-1.2, and
branch-1.1). Each patch takes a bit of time, especially if some fixup
needed, and then, if doing the application, there is watching the build
subsequently to see if my application broke the build (which a seemingly
innocuous application does with some regularity). A critical fix should be
done in all branches, but for less-than-critical, I'd be for encouraging
folks to (rolling) upgrade to get new performance/feature?


It's a slippery slope trying to decide what has merits to get backported 
as well. After meeting compatibility guarantees, how do you decide if 
some change if critical enough to be considered for the non-EOL'ed 1.x 
branches?


Given that there's only now a 1.2.0 release in the works, it's also kind 
of crappy to try to restrict what can go out on a 1.1. I'm all for 
release early/often, but that needs to be measured against the cycles to 
make those new major/minor releases (so that we can subsequently 
encourage the community to adopt those new releases).

[jira] [Created] (HBASE-15232) Exceptions returned over multi RPC don't trigger region location reloads

[Josh Elser (JIRA)]

Josh Elser created HBASE-15232:
--

 Summary: Exceptions returned over multi RPC don't trigger region 
location reloads
 Key: HBASE-15232
 URL: https://issues.apache.org/jira/browse/HBASE-15232
 Project: HBase
  Issue Type: Sub-task
  Components: Client
Reporter: Josh Elser
Assignee: Josh Elser
 Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.4


Follow-on for HBASE-15221:

A work-around was added in HTableMultiplexer to work around an issue that 
AsyncProcess wasn't clearing the region location cache on Exception. This was 
stemming from the issue that the {{tableName}} is {{null}} because 
HTableMultiplexer is using the {{multi}} RPC. This causes an error that looks 
like:

{noformat}
[WARN] Coding error, see method javadoc. row=[B@1673eff, tableName=null
{noformat}

HBASE-15221 should fix HTableMultiplexer, but it would be good to push the fix 
down into AsyncProcess instead of using higher-level workarounds.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-15221) HTableMultiplexer improvements (stale region locations and resource leaks)

[Josh Elser (JIRA)]

Josh Elser created HBASE-15221:
--

 Summary: HTableMultiplexer improvements (stale region locations 
and resource leaks)
 Key: HBASE-15221
 URL: https://issues.apache.org/jira/browse/HBASE-15221
 Project: HBase
  Issue Type: Bug
  Components: Client
Reporter: Josh Elser
Assignee: Josh Elser
Priority: Critical
 Fix For: 2.0.0, 1.2.1, 1.1.4, 0.98.18


It looks like HTableMultiplexer has a couple of issues.

Upon failing to send a Put to the appropriate RS, the Put is re-queued back 
into the system. Normally this is fine as such an exception is transient and 
the Put would eventually succeed. However, in the case where the Put was 
rejected because of a NotServingRegionException (e.g. split, balance, merge), 
the re-queuing of the Put will end up using the same cached HRegionLocation. 
This means that the Put will just be repeatedly sent back to the same RS over 
and over again, eventually being dropped on the floor. Need to invalidate the 
location cache (or make sure we refresh it) when we re-queue the Put.

The internal ClusterConnection is also leaked. If a user creates many 
HTableMultiplexers, they'll eventually run into issues (memory, zk connections, 
etc) because they'll never get cleaned up. HTableMultiplexer needs a close 
method.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [VOTE] HBase 1.2.0 RC0

[Josh Elser]

I feel like I've seen this before myself on OSX, but didn't trace it 
down either.


Mikhail Antonov wrote:

Checked signatures, package content, RAT, ran tests, notice repeated
failure:

tests in error:
   TestRegionServerHostname.testRegionServerHostname:86 » TestTimedOut test
timed...

It failed once as part of suite, and now failing for me in individual runs
around every 3rd run (MacOSX). Anyone else saw that? Suspecting issue with
my env for now.

-Mikhail

On Mon, Jan 4, 2016 at 3:16 PM, Ted Yu  wrote:


+1

Ran unit test suite
Exercised basic shell commands

[INFO] BUILD SUCCESS
[INFO]

[INFO] Total time: 01:44 h
[INFO] Finished at: 2016-01-03T09:12:35-08:00

On Sun, Jan 3, 2016 at 4:29 AM, Sean Busbey  wrote:


Happy New Year everyone! I'm pleased to announce the first release
candidate for HBase 1.2.0.

Artifacts are available here:

https://dist.apache.org/repos/dist/dev/hbase/hbase-1.2.0RC0/

As of this vote, the relevant md5 hashes are:
41c499a325ae6449fe99bbe315d07e10  hbase-1.2.0-bin.tar.gz
537874610f1b53d4bce9d8a925661246  hbase-1.2.0-src.tar.gz

Maven artifacts are also available in the staging repository:

https://repository.apache.org/content/repositories/orgapachehbase-1122/

artifacts are signed with my code signing key 0D80DB7C, available in
the project KEYS file:

http://www.apache.org/dist/hbase/KEYS

these artifacts correspond to commit hash

20c4368065165ad49bdfe8172316e42566a6d6a0

which signed tag 1.2.0RC0 currently points to




https://git1-us-west.apache.org/repos/asf?p=hbase.git;a=tag;h=771e13960cf406185bb29447dffb6f2b5a56baec

HBase 1.2.0 is the second minor release in the HBase 1.x line, continuing
on
the theme of bringing a stable, reliable database to the Hadoop and NoSQL
communities. This release includes roughly 250 resolved issues not

covered

by previous 1.x releases.

Notable new features include:
- JDK8 is now supported
- Hadoop 2.6.1+ and Hadoop 2.7.1+ are now supported
- per column-family time ranges for scan (HBASE-14355)
- daemons respond to SIGHUP to reload configs (HBASE-14529)
- region location methods added to thrift2 proxy (HBASE-13698)
- table-level sync that sends deltas (HBASE-13639)
- client side metrics via JMX (HBASE-12911)

The full list of issues can be found at:




https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12310753=12332062

Please take a few minutes to verify the release[1] and vote on releasing
it:

[ ] +1 Release this package as Apache HBase 1.2.0
[ ] +0 no opinion
[ ] -1 Do not release this package because...

Vote will be subject to Majority Approval[2] and will close at 8:00PM
UTC on Monday, Jan 11th, 2016[3].

[1]: http://www.apache.org/info/verification.html
[2]: https://www.apache.org/foundation/glossary.html#MajorityApproval
[3]: to find this in your local timezone see:
http://s.apache.org/hbase-1.2.0-rc0-close

Re: Thrift versions and generated code

[Josh Elser]

Ahh, thanks, gentlemen.

Andrew Purtell wrote:

Yeah, let's finish that.


On Thu, Nov 12, 2015 at 11:49 AM, Ted Yu<yuzhih...@gmail.com>  wrote:


See https://issues.apache.org/jira/browse/HBASE-14172

On Thu, Nov 12, 2015 at 11:46 AM, Josh Elser<josh.el...@gmail.com>  wrote:


Hi,

In looking at https://issues.apache.org/jira/browse/HBASE-14800, I saw
that the current libthrift dependency on master was at 0.9.2, but the
generated code still has the 0.9.0 comments.

Is there a reason for that? Should the libthrift version defined in the
poms be the de-facto version used by that version of HBase?

Thanks.

- Josh

[jira] [Created] (HBASE-14800) Expose checkAndMutate via Thrift2

[Josh Elser (JIRA)]

Josh Elser created HBASE-14800:
--

 Summary: Expose checkAndMutate via Thrift2
 Key: HBASE-14800
 URL: https://issues.apache.org/jira/browse/HBASE-14800
 Project: HBase
  Issue Type: Improvement
  Components: Thrift
Reporter: Josh Elser
Assignee: Josh Elser
 Fix For: 2.0.0


Had a user ask why checkAndMutate wasn't exposed via Thrift2.

I see no good reason (since checkAndPut and checkAndDelete are already there), 
so let's add it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Thrift versions and generated code

[Josh Elser]

Hi,

In looking at https://issues.apache.org/jira/browse/HBASE-14800, I saw 
that the current libthrift dependency on master was at 0.9.2, but the 
generated code still has the 0.9.0 comments.


Is there a reason for that? Should the libthrift version defined in the 
poms be the de-facto version used by that version of HBase?


Thanks.

- Josh

Re: [VOTE] First release candidate for HBase 1.1.3 (RC0) is available

[Josh Elser]

+1 (non-binding)

* Built from source
* Ran tests (-PrunDevTests). o.a.h.h.r.TestRegionServerHostname was 
problematic, might have just been me.

* Checked sigs/xsums
* Checked the compat report (thanks for posting it, Nick)
* Skimmed release notes looking for anything that might introduce new 
deps for licensing concerns (found none)


Nick Dimiduk wrote:

I'm happy to announce the first release candidate of HBase 1.1.3 (HBase-1.1.
3RC0) is available for download at
https://dist.apache.org/repos/dist/dev/hbase/hbase-1.1.3RC0/

Maven artifacts are also available in the staging repository
https://repository.apache.org/content/repositories/orgapachehbase-1117

Artifacts are signed with my code signing subkey 0xAD9039071C3489BD,
available in the Apache keys directory
https://people.apache.org/keys/committer/ndimiduk.asc

There's also a signed tag for this release at
https://git-wip-us.apache.org/repos/asf?p=hbase.git;a=tag;h=16e905679e2dd5cb1b05ca8bc34a403e154a395f

The detailed source and binary compatibility report vs 1.1.0 has been
published for your review, at
http://people.apache.org/~ndimiduk/1.1.0_1.1.3RC0_compat_report.html

HBase 1.1.3 is the third patch release in the HBase 1.1 line, continuing on
the theme of bringing a stable, reliable database to the Hadoop and NoSQL
communities. This release includes over 120 bug fixes since the 1.1.2
release. Notable correctness fixes
include HBASE-14474, HBASE-14591, HBASE-14224,
HBASE-14431, HBASE-14407, HBASE-14313, HBASE-14621, HBASE-14501, and
HBASE-13250.

The full list of fixes included in this release is available at
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12310753=12333152
  and in the CHANGES.txt file included in the distribution.

Please try out this candidate and vote +/-1 by 23:59 Pacific time on
Friday, 2015-11-13 as to whether we should release these artifacts as HBase
1.1.3.

Thanks,
Nick

Re: On our unit tests...

[Josh Elser]

Huge kudos to you, Stack, for making the time to run these down.

As a contributor, I'm very moved by the thought of treating what Jenkins 
reports as truth.


Stack wrote:

Since I wrote the below, we've figured who the surefire-killer was
[HBASE-14589]. 9 of the last 10 1.2 builds passed (even though blue builds
are harder to achieve now since they are a compound of a jdk 1.7 and a jdk
1.8 run). 1.3 is failing on a few tests that seem legitimately flakey; I'm
looking into them. Trunk is settling down after being made into a jdk7/8
matrix; it should stabilize soon.

Repeating my petition from below, can we start putting our trust back in
apache builds and start relying on it again? It found a flakey end of last
week soon after it went in because builds mostly pass now so the flakey
shone through. It can find more if we all make the effort to keep it blue.
In particular, can we end the passes-locally-for-me practice since tests
that go zombie or hang usually run fine on boxes where there is no
contention.

Thanks,
St.Ack



On Fri, Oct 23, 2015 at 2:54 PM, Stack  wrote:


A few of us have been doing cleanup over the last month or so (see
HBASE-14420). As a project, we had let our unit test suite go to seed. It
was an anthology of mysterious crashes, zombies and flakes.

We are not done yet but tests are mostly stable again with patch builds
passing close to 100% of the time as long as the patch is good and trunk
and branch-1/branch-1.2 are tending back toward being blue always. Hanging
tests have been fixed and or disabled to be put back after scrubbing.
Mysterious surefire crashes/timeouts have been addressed by purging a
problematic test set that we intend to re-add after tuneup and fix. There
are still a few flakies in the mix.

This is a petition that we go out of our way going forward to keep OUR
test suite blue. We'll all be more productive if we can keep it this way.
Patches will land faster because there'll be less friction getting them in
(Landing big patches was taking me a week before starting in on this
effort). We'll catch a slew of problems before commit. New devs won't be
confounded by mysterious unrelated test fails. There'll be no need to keep
up an arcane knowledge of 'known flakies' or hanging tests or the need for
expending extra effort and resources doing 'look-it-works-locally-for-me'
test runs locally.

St.Ack

Below are some further notes for those interested in build and work done
to our test rig recently; ugly detail is over in HBASE-14420.

Until an alternative shows up, our Apache Jenkins needs to run blue always
if we want to do community development. True, Apache Jenkins is a trying
environment in which to run tests, but it is shared, public, and I have yet
to come across a hang or failure that was Apache-Jenkins-only; the only
difference I've seen is that the incidence of hangs and flakies is higher
on Apache.

The test-patch.sh script had some hacking done to it mostly removing code
that was finding and killing zombies. We were reporting ANY concurrent
build as a zombie, even those that were not hbase tests, and killing them
in the belief that they were leftovers from previous runs (the script had a
few different techniques for finding and executing adjacent processes).
This made some sense when we were supposed to be the only test running on
the box but this has not been true for a long time. Killing was
papering-over the fact that we were leaving zombies after us.

The Jenkins build configuration also had zombie code from test-patch.sh in
it (still does -- a TODO). Builds now dump out test machine load and
listing of what else is running on the box at test start to give a sense of
how loaded the test box is.

I feel particularly bad for the new contributors. They have it hard enough
already checking out a fat project with a slow build system with hours of
tests to run to verify changes. Lets spare them the added barrier of a
confounding experience when their nice patch throws up a mysterious jenkins
fail on submit.

[jira] [Created] (HBASE-14594) Use new DNS API introduced in HADOOP-12437

[Josh Elser (JIRA)]

Josh Elser created HBASE-14594:
--

 Summary: Use new DNS API introduced in HADOOP-12437
 Key: HBASE-14594
 URL: https://issues.apache.org/jira/browse/HBASE-14594
 Project: HBase
  Issue Type: Bug
Reporter: Josh Elser
Assignee: Josh Elser


HADOOP-12437 introduced a new API to {{org.apache.hadoop.net.DNS}}: 
{{getDefaultHost(String, String, boolean)}}.

The purpose of this method (the boolean argument really) is to change the 
functionality so that when rDNS fails, {{InetAddress#getCanonicalHostName()}} 
is consulted which includes resolution via the hosts file.

The direct application of this new method is relevant on hosts with multiples 
NICs and Kerberos enabled.

Sadly, this method only exists in 2.8.0-SNAPSHOT, so to benefit from the fix 
without great pain, some reflection is required.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: HBase and Accumulo

[Josh Elser]

IMO, they are very similar. Lots of smart people on both sides making 
really good changes.


I think HBase has a lot more instrumentation and understanding on the 
resources a cluster will use. For example, I think it's much clearer the 
resources and threads that the RPC server will use. This is much more 
obtuse (and grows/shrinks on its own in Accumulo). I think this also 
drifts into the HBase API usage too -- I have a much better 
understanding of what needs to be managed with HBase. This is a little 
more obtuse in Accumulo for new users.


I also think HBase has a much better understanding and tuning of the 
read path. I would trust consistently performing (SLA bound) workloads 
on HBase much more than Accumulo just because there hasn't been (public) 
work that is or has happened in Accumulo.


On the other side, it's been years since I've seen data loss or 
assignment bugs in Accumulo. Around 1.1.0, the bugs that Enis and Stack 
fixed shocked me. I was rather surprised to see these kinds of bugs crop 
up, and really worried me when I spent quite a lot of time trying to 
understand the bugs. Personally, I would trust Accumulo to be thrown off 
a cliff and still keep chugging (again, because I've done this myself). 
I don't have this confidence with HBase (yet).


Specifically WRT security since you brought it up. Last I tried to play 
with the cell-level security APIs in HBase, it seemed very obtuse to me. 
Perhaps I was just dense and didn't find the right sort of instructions. 
I think where security is critical, I would trust Accumulo more because 
it's been very fleshed out over many years and been a part of the core 
model since the start. I felt that HBase is still in a shake-down phase. 
(again, I don't want to be argumentative -- it's just my personal 
experience to date using the code and watching JIRA issues)


The HBase coprocessors and Accumulo iterators difference will still 
stand (they are not equivalent features and solve different problems, 
IMO). Coprocessors enable quite a bunch of interesting things (notably, 
Phoenix). At the same time, I like the functional-conciseness in how I 
can represent some problems using Accumulo iterators.


Ultimately, consider the use cases, evaluate the solutions and make your 
decision off of empirical evidence. That's the only way to really make a 
decision :)


Jerry He wrote:

Hi, folks

We have people that are evaluating HBase vs Accumulo.
Security is an important factor.

But I think after the Cell security was added in HBase, there is no more
real gap compared to Accumulo.

I know we have both HBase and Accumulo experts on this list.
Could someone shred more light?
I am looking for real gap comparing HBase to Accumulo if there is any so
that I can be prepared to address them. This is not limited to the security
area.

There are differences in some features and implementations. But they don't
see like real 'gaps'.

Any comments and feedbacks are welcome.

Thanks,

Jerry

Re: HBase and Accumulo

[Josh Elser]

Andrew Purtell wrote:

Last I tried to play with the cell-level security APIs in HBase, it

seemed very obtuse to me. Perhaps I was just dense and didn't find the
right sort of instructions.

I don't think anyone would debate that cell level security in HBase is a
work-in-progress. We'd really welcome your impressions and thoughts on any
use of those APIs if you're interested in providing that feedback. As
someone involved in their implementation, in my opinion they are not meant
to compete directly with Accumulo. They are an HBase-y take on similar
functionality meant to integrate with the HBase code base not rework HBase
to look more like Accumulo internals. So, there will be differences that
affect functionality and performance. Our aim is for these features to work
best when HBase use cases may need cell level security, and also adequately
if you were working with Accumulo for a while but now are in an environment
that uses HBase instead. This latter case needs investigation and
refinement no doubt. Our biggest issue I think is the lack of people with
Accumulo app dev experience in the HBase community (probably,
unsurprisingly).


It's been on my radar to make time to do a more thorough 
investigation/comparison :) It's definitely important to manage the 
expectations on the maturity of the security system. I think that's 
ultimately the biggest point I wanted to make WRT cell-level security.



I think where security is critical, I would trust Accumulo more because

it's been very fleshed out over many years and been a part of the core
model since the start. I felt that HBase is still in a shake-down phase.
(again, I don't want to be argumentative -- it's just my personal
experience to date using the code and watching JIRA issues)

This would depend on what features you need. For example, if you want only
the basic strong authentication and don't need the cell level features,
correct me if I'm wrong Josh but HBase had this quite some time before
Accumulo. Going feature by feature, here's a list in order of maturity:
- Strong authentication for RPC and the auth token provider for MapReduce
- Table and CF level ACL based access control
- Cell level ACL and 'visibility label' based access control

FWIW


Very valid point. I did limit security explicitly to the cell-level 
security which was short-sighted on my part. For example, only in the 
last major release of Accumulo can Kerberos-clients authenticate with 
Accumulo. As such, the same new feature tag should be applied to it (I 
wrote it, so I'm sure there are bugs). You're very right there.


CF ACLs are also another good point of reference. Accumulo's ACL support 
is limited to namespace and table, no per-family support.

Re: [VOTE] First release candidate for HBase 1.1.2 (RC0) is available

[Josh Elser]

Forgot to send out last night (figured I would now despite the -1)

Ignoring licensing issues, (non-binding) +1

* Checked sig/xsums
* Inspected compatibility report (thanks for including!)
* Built source
* Ran small+med tests (TestRegionServerHostname failed, passed on rerun)

Nick Dimiduk wrote:

I'm happy to announce the first release candidate of HBase 1.1.2
(HBase-1.1.2RC0) is available for download at
https://dist.apache.org/repos/dist/dev/hbase/hbase-1.1.2RC0/

Maven artifacts are also available in the staging repository
https://repository.apache.org/content/repositories/orgapachehbase-1090

Artifacts are signed with my code signing subkey 0xAD9039071C3489BD,
available in the Apache keys directory
https://people.apache.org/keys/committer/ndimiduk.asc

There's also a signed tag for this release at
https://git-wip-us.apache.org/repos/asf?p=hbase.git;a=tag;h=c4b4a91620639fd96e59171369c630dc51e3

The detailed source and binary compatibility report vs 1.1.0 has been
published for your review, at
http://people.apache.org/~ndimiduk/1.1.0_1.1.2RC0_compat_report.html

HBase 1.1.2 is the second patch release in the HBase 1.1 line, continuing
on the theme of bringing a stable, reliable database to the Hadoop and
NoSQL communities. This release includes over 70 bug fixes since the 1.1.1
release. Notable correctness fixes include HBASE-14054, HBASE-12865,
HBASE-13993, HBASE-13337, HBASE-14013, HBASE-13895.

The full list of fixes included in this release is available at
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12310753version=12332793
and in the CHANGES.txt file included in the distribution.

Please try out this candidate and vote +/-1 by midnight Pacific time on
Monday, 2015-08-17 as to whether we should release these artifacts as HBase
1.1.2.

Thanks,
Nick

Re: HBase and Accumulo

[Josh Elser]

Like I've said many times now, it's relative to your actual problem. If 
you don't have that much data (or intend to grow into that much data), 
it's not an issue. Obviously, this is the case for you.


However, it is an architectural difference between the two projects with 
known limitations for a single metadata region. It's a difference as 
what was asked for by Jerry.


Ted Malaska wrote:

I've been doing HBase for a long time and never had an issue with region
count limits and I have clusters with 10s of billions of records.  Many
there would be issues around a couple Trillion records, but never got that
high yet.

Ted Malaska

On Wed, Aug 19, 2015 at 2:24 PM, Josh Elserjosh.el...@gmail.com  wrote:


Oh, one other thing that I should mention (was prompted off-list).

(definition time since cross-list now: HBase regions == Accumulo tablets)

Accumulo will handle many more regions than HBase does now due to a
splittable metadata table. While I was told this was a very long and
arduous journey to implement correctly (WRT splitting, merges and bulk
loading), users with too many regions problems are extremely few and far
between for Accumulo.

I was very happy to see effort/design being put into this in HBase. And,
just to be fair in criticism/praises, HBase does appear to me to do
assignments of regions much faster than Accumulo does on a small cluster
(~5-10 nodes). Accumulo may take a few seconds to notice and reassign
tablets. I have yet to notice this with HBase (which also could be due to
lack of personal testing).


Jerry He wrote:


Hi, folks

We have people that are evaluating HBase vs Accumulo.
Security is an important factor.

But I think after the Cell security was added in HBase, there is no more
real gap compared to Accumulo.

I know we have both HBase and Accumulo experts on this list.
Could someone shred more light?
I am looking for real gap comparing HBase to Accumulo if there is any so
that I can be prepared to address them. This is not limited to the
security
area.

There are differences in some features and implementations. But they don't
see like real 'gaps'.

Any comments and feedbacks are welcome.

Thanks,

Jerry

Re: HBase and Accumulo

[Josh Elser]

Oh, one other thing that I should mention (was prompted off-list).

(definition time since cross-list now: HBase regions == Accumulo tablets)

Accumulo will handle many more regions than HBase does now due to a 
splittable metadata table. While I was told this was a very long and 
arduous journey to implement correctly (WRT splitting, merges and bulk 
loading), users with too many regions problems are extremely few and 
far between for Accumulo.


I was very happy to see effort/design being put into this in HBase. And, 
just to be fair in criticism/praises, HBase does appear to me to do 
assignments of regions much faster than Accumulo does on a small cluster 
(~5-10 nodes). Accumulo may take a few seconds to notice and reassign 
tablets. I have yet to notice this with HBase (which also could be due 
to lack of personal testing).


Jerry He wrote:

Hi, folks

We have people that are evaluating HBase vs Accumulo.
Security is an important factor.

But I think after the Cell security was added in HBase, there is no more
real gap compared to Accumulo.

I know we have both HBase and Accumulo experts on this list.
Could someone shred more light?
I am looking for real gap comparing HBase to Accumulo if there is any so
that I can be prepared to address them. This is not limited to the security
area.

There are differences in some features and implementations. But they don't
see like real 'gaps'.

Any comments and feedbacks are welcome.

Thanks,

Jerry

Re: HBase and Accumulo

[Josh Elser]

He didn't ask just about security, FWIW

I am looking for real gap comparing HBase to Accumulo if there is any 
so that I can be prepared to address them. This is not limited to the 
security area.


Sean Busbey wrote:

Let's please stick to the topic Jerry asked about: security features.

We can get into all sorts of discussions around scalability and read/write
performance in a different joint thread if folks want. We all have lots of
Opinions (and the YCSB community would love to see more of y'all show up to
improve it's suitability for comparing things ;) ). However, I think we're
all in agreement that both systems scale well enough for the vast
majority of use cases.


On Wed, Aug 19, 2015 at 1:37 PM, Ted Malaskated.mala...@cloudera.com
wrote:


Sorry Type-o

So there might be issues when you pass the Quadrillion.  But Like I said
never ran into that issue of region limits.

On Wed, Aug 19, 2015 at 2:29 PM, Ted Malaskated.mala...@cloudera.com
wrote:


Sorry 10 billion a day so that is 7 Trillion records.  So many issues
around 1000 Trillion

On Wed, Aug 19, 2015 at 2:28 PM, Ted Malaskated.mala...@cloudera.com
wrote:


I've been doing HBase for a long time and never had an issue with region
count limits and I have clusters with 10s of billions of records.  Many
there would be issues around a couple Trillion records, but never got

that

high yet.

Ted Malaska

On Wed, Aug 19, 2015 at 2:24 PM, Josh Elserjosh.el...@gmail.com

wrote:

Oh, one other thing that I should mention (was prompted off-list).

(definition time since cross-list now: HBase regions == Accumulo

tablets)

Accumulo will handle many more regions than HBase does now due to a
splittable metadata table. While I was told this was a very long and
arduous journey to implement correctly (WRT splitting, merges and bulk
loading), users with too many regions problems are extremely few and

far

between for Accumulo.

I was very happy to see effort/design being put into this in HBase.

And,

just to be fair in criticism/praises, HBase does appear to me to do
assignments of regions much faster than Accumulo does on a small

cluster

(~5-10 nodes). Accumulo may take a few seconds to notice and reassign
tablets. I have yet to notice this with HBase (which also could be due

to

lack of personal testing).


Jerry He wrote:


Hi, folks

We have people that are evaluating HBase vs Accumulo.
Security is an important factor.

But I think after the Cell security was added in HBase, there is no

more

real gap compared to Accumulo.

I know we have both HBase and Accumulo experts on this list.
Could someone shred more light?
I am looking for real gap comparing HBase to Accumulo if there is any

so

that I can be prepared to address them. This is not limited to the
security
area.

There are differences in some features and implementations. But they
don't
see like real 'gaps'.

Any comments and feedbacks are welcome.

Thanks,

Jerry

Re: HBase and Accumulo

[Josh Elser]

Alright, I have to ask... are you referring to the paper that cites 
Accumulo performance without write-ahead logs enabled? I have some 
serious reservations about the relevance of that paper to this 
conversation and just want to make sure people aren't led astray by what 
the actual takeaway should be.


Jeremy Kepner wrote:

A big difference between Accumulo and HBase is the published performance 
numbers.
The Accumulo community has done a good job of continuing to publish up-to-date 
performance
numbers in peer-reviewed venues which allow Accumulo to claim best in the world 
performance.

The HBase community hasn't been doing that so much.  It would be great if they 
did because
the HBase points on the graphs are old and it would be good to get new ones.


On Wed, Aug 19, 2015 at 02:30:58PM -0400, Josh Elser wrote:

Like I've said many times now, it's relative to your actual problem.
If you don't have that much data (or intend to grow into that much
data), it's not an issue. Obviously, this is the case for you.

However, it is an architectural difference between the two projects
with known limitations for a single metadata region. It's a
difference as what was asked for by Jerry.

Ted Malaska wrote:

I've been doing HBase for a long time and never had an issue with region
count limits and I have clusters with 10s of billions of records.  Many
there would be issues around a couple Trillion records, but never got that
high yet.

Ted Malaska

On Wed, Aug 19, 2015 at 2:24 PM, Josh Elserjosh.el...@gmail.com   wrote:


Oh, one other thing that I should mention (was prompted off-list).

(definition time since cross-list now: HBase regions == Accumulo tablets)

Accumulo will handle many more regions than HBase does now due to a
splittable metadata table. While I was told this was a very long and
arduous journey to implement correctly (WRT splitting, merges and bulk
loading), users with too many regions problems are extremely few and far
between for Accumulo.

I was very happy to see effort/design being put into this in HBase. And,
just to be fair in criticism/praises, HBase does appear to me to do
assignments of regions much faster than Accumulo does on a small cluster
(~5-10 nodes). Accumulo may take a few seconds to notice and reassign
tablets. I have yet to notice this with HBase (which also could be due to
lack of personal testing).


Jerry He wrote:


Hi, folks

We have people that are evaluating HBase vs Accumulo.
Security is an important factor.

But I think after the Cell security was added in HBase, there is no more
real gap compared to Accumulo.

I know we have both HBase and Accumulo experts on this list.
Could someone shred more light?
I am looking for real gap comparing HBase to Accumulo if there is any so
that I can be prepared to address them. This is not limited to the
security
area.

There are differences in some features and implementations. But they don't
see like real 'gaps'.

Any comments and feedbacks are welcome.

Thanks,

Jerry

Re: HBase and Accumulo

[Josh Elser]

Ah right, I did forgot about that paper. Thanks for clarifying.

Big +1 to Andy's comments, too.

Jeremy Kepner wrote:

Turning off the walog was mostly to shorten the benchmarking cycle
(it allowed us to go from zero to peak ingest in a few seconds).  BAH got
pretty much the same performance results in their paper,
it just took longer for their experiments to run.
So, in this case, we had two different teams doing things different
ways and getting the same result, which is what we like to see.

On Wed, Aug 19, 2015 at 03:27:07PM -0400, Josh Elser wrote:

Alright, I have to ask... are you referring to the paper that cites
Accumulo performance without write-ahead logs enabled? I have some
serious reservations about the relevance of that paper to this
conversation and just want to make sure people aren't led astray by
what the actual takeaway should be.

Jeremy Kepner wrote:

A big difference between Accumulo and HBase is the published performance 
numbers.
The Accumulo community has done a good job of continuing to publish up-to-date 
performance
numbers in peer-reviewed venues which allow Accumulo to claim best in the world 
performance.

The HBase community hasn't been doing that so much.  It would be great if they 
did because
the HBase points on the graphs are old and it would be good to get new ones.


On Wed, Aug 19, 2015 at 02:30:58PM -0400, Josh Elser wrote:

Like I've said many times now, it's relative to your actual problem.
If you don't have that much data (or intend to grow into that much
data), it's not an issue. Obviously, this is the case for you.

However, it is an architectural difference between the two projects
with known limitations for a single metadata region. It's a
difference as what was asked for by Jerry.

Ted Malaska wrote:

I've been doing HBase for a long time and never had an issue with region
count limits and I have clusters with 10s of billions of records.  Many
there would be issues around a couple Trillion records, but never got that
high yet.

Ted Malaska

On Wed, Aug 19, 2015 at 2:24 PM, Josh Elserjosh.el...@gmail.comwrote:


Oh, one other thing that I should mention (was prompted off-list).

(definition time since cross-list now: HBase regions == Accumulo tablets)

Accumulo will handle many more regions than HBase does now due to a
splittable metadata table. While I was told this was a very long and
arduous journey to implement correctly (WRT splitting, merges and bulk
loading), users with too many regions problems are extremely few and far
between for Accumulo.

I was very happy to see effort/design being put into this in HBase. And,
just to be fair in criticism/praises, HBase does appear to me to do
assignments of regions much faster than Accumulo does on a small cluster
(~5-10 nodes). Accumulo may take a few seconds to notice and reassign
tablets. I have yet to notice this with HBase (which also could be due to
lack of personal testing).


Jerry He wrote:


Hi, folks

We have people that are evaluating HBase vs Accumulo.
Security is an important factor.

But I think after the Cell security was added in HBase, there is no more
real gap compared to Accumulo.

I know we have both HBase and Accumulo experts on this list.
Could someone shred more light?
I am looking for real gap comparing HBase to Accumulo if there is any so
that I can be prepared to address them. This is not limited to the
security
area.

There are differences in some features and implementations. But they don't
see like real 'gaps'.

Any comments and feedbacks are welcome.

Thanks,

Jerry

Re: [DISCUSS] project for pre-commit patch testing (was Re: upstream jenkins build broken?)

[Josh Elser]

+1

(Have been talking to Sean in private on the subject -- seems 
appropriate to voice some public support)


I'd be interested in this for Accumulo and Slider. For Accumulo, we've 
come a far way without a pre-commit build, primarily due to a CTR 
process. We have seen the repeated questions of how do I run the tests 
which a more automated workflow would help with, IMO. I think Slider 
could benefit with the same reasons.


I'd also be giddy to see the recent improvements in Hadoop trickle down 
into the other projects that Allen already mentioned.


Take this as record that I'd be happy to try to help out where possible.

Sean Busbey wrote:

thank you for making a more digestible version Allen. :)

If you're interested in soliciting feedback from other projects, I created
ASF short links to this thread in common-dev and hbase:


* http://s.apache.org/yetus-discuss-hadoop
* http://s.apache.org/yetus-discuss-hbase

While I agree that it's important to get feedback from ASF projects that
might find this useful, I can say that recently I've been involved in the
non-ASF project YCSB and both the pretest and better shell stuff would be
immensely useful over there.

On Mon, Jun 15, 2015 at 10:36 PM, Allen Wittenauera...@altiscale.com  wrote:


 I'm clearly +1 on this idea.  As part of the rewrite in Hadoop of
test-patch, it was amazing to see how far and wide this bit of code as
spread.  So I see consolidating everyone's efforts as a huge win for a
large number of projects.  (esp considering how many I saw suffering from a
variety of identified bugs! )

 But….

 I think it's important for people involved in those other projects
to speak up and voice an opinion as to whether this is useful.

To summarize:

 In the short term, a single location to get/use a precommit patch
tester rather than everyone building/supporting their own in their spare
time.

  FWIW, we've already got the code base modified to be pluggable.
We've written some basic/simple plugins that support Hadoop, HBase, Tajo,
Tez, Pig, and Flink.  For HBase and Flink, this does include their custom
checks.  Adding support for other project shouldn't be hard.  Simple
projects take almost no time after seeing the basic pattern.

 I think it's worthwhile highlighting that means support for both
JIRA and GitHub as well as Ant and Maven from the same code base.

Longer term:

 Well, we clearly have ideas of things that we want to do. Adding
more features to test-patch (review board? gradle?) is obvious. But what
about teasing apart and generalizing some of the other shell bits from
projects? A common library for building CLI tools to fault injection to
release documentation creation tools to …  I'd even like to see us get as
advanced as a run this program to auto-generate daemon stop/start bits.

 I had a few chats with people about this idea at Hadoop Summit.
What's truly exciting are the ideas that people had once they realized what
kinds of problems we're trying to solve.  It's always amazing the problems
that projects have that could be solved by these types of solutions.  Let's
stop hiding our cool toys in this area.

 So, what feedback and ideas do you have in this area?  Are you a
yay or a nay?


On Jun 15, 2015, at 4:47 PM, Sean Busbeybus...@cloudera.com  wrote:


Oof. I had meant to push on this again but life got in the way and now

the

June board meeting is upon us. Sorry everyone. In the event that this

ends

up contentious, hopefully one of the copied communities can give us a
branch to work in.

I know everyone is busy, so here's the short version of this email: I'd
like to move some of the code currently in Hadoop (test-patch) into a new
TLP focused on QA tooling. I'm not sure what the best format for priming
this conversation is. ORC filled in the incubator project proposal
template, but I'm not sure how much that confused the issue. So to start,
I'll just write what I'm hoping we can accomplish in general terms here.

All software development projects that are community based (that is,
accepting outside contributions) face a common QA problem for vetting
in-coming contributions. Hadoop is fortunate enough to be sufficiently
popular that the weight of the problem drove tool development (i.e.
test-patch). That tool is generalizable enough that a bunch of other TLPs
have adopted their own forks. Unfortunately, in most projects this kind

of

QA work is an enabler rather than a primary concern, so often the tooling
is worked on ad-hoc and little shared improvements happen across
projects. Since
the tooling itself is never a primary concern, any made is rarely reused
outside of ASF projects.

Over the last couple months a few of us have been working on generalizing
the tooling present in the Hadoop code base (because it was the most

mature

out of all those in the various projects) and it's reached a point where

we

think we can start bringing on other downstream 

[jira] [Created] (HBASE-13892) Scanner with all results filtered out results in NPE

[Josh Elser (JIRA)]

Josh Elser created HBASE-13892:
--

 Summary: Scanner with all results filtered out results in NPE
 Key: HBASE-13892
 URL: https://issues.apache.org/jira/browse/HBASE-13892
 Project: HBase
  Issue Type: Bug
  Components: Client
Affects Versions: 1.1.0
Reporter: Josh Elser
Assignee: Josh Elser
Priority: Critical
 Fix For: 1.2.0, 1.1.1


Saw a failure during some testing with region_mover.rb

{code}
NativeException: java.lang.NullPointerException: null
__ensure__ at /usr/hdp/current/hbase-master/bin/region_mover.rb:110
  isSuccessfulScan at /usr/hdp/current/hbase-master/bin/region_mover.rb:109
  isSuccessfulScan at /usr/hdp/current/hbase-master/bin/region_mover.rb:104
 unloadRegions at /usr/hdp/current/hbase-master/bin/region_mover.rb:328
{code}

To try to get a real stacktrace, I wrote a simple test. Turns out, it was 
really simple to just produce the NPE within ClientScanner.

Patch with fix and test incoming.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Ruby shell versions for HBase 2.0

[Josh Elser]

Andrew Purtell wrote:

This is because the Accumulo shell is a custom built shell? If so, we had
one of those once and replaced it with the IRB based one. We didn't settle
on JRuby right away but, at the time, the consensus was we didn't want to
be in the business of maintaining yet another REPL when specialist open
source communities have already done that. Why has this changed? Has it?


Accumulo shell is just a Java program that uses JLine. It has no control 
flow structures, variables, or anything like that.



Why is the Accumulo shell easier to script with? Does it have control flow
constructs? Variable assignment? Or is it easier somehow because it does
not have those things? Even the system shell has those. Hmm... Are we
saying that any control flow or variables/substitution should be done by
calling our shell from a bash or sh script? Wouldn't that be super slow
given Java's startup time for every invocation of our shell? A bit like
trying to script the AWS command line utilities, which is excruciating.


This is what I was trying to get at: the lack of typical language 
constructs is both beneficial and irritating. I think it depends on what 
you really want to do. Does that mean trying to maintain two shells is 
worth it? I'm not sure, but I'm also not sure where the happy medium is.


If you exec the shell to just run one command at a time, yes. It is 
slow. You can redirect input or provide a file of commands to run at one 
time which don't suffer from the JVM startup problem.



I can see why a custom shell built with Java would easier to test where the
rest of the project is using Surefire/JUnit. That's a developer convenience
concern though, not a user convenience one.


That's only if you assume HBase users actually understand Ruby though, 
no? I would think that something which acts like your normal unix shell 
would be familiar to users w/o the need to understand some other language.




On Wed, May 13, 2015 at 10:41 AM, Sean Busbeybus...@cloudera.com  wrote:


Pros:

* It's easier to test and maintain
* it's easier to script with
* its interactive mode feels mode focused on the task of interacting with
a cluster to me (maybe this is just acting like a psql or mysql shell)

Cons

* adding custom commands requires knowing java

--
Sean
On May 13, 2015 12:31 PM, Andrew Purtellapurt...@apache.org  wrote:


Why is the Accumulo shell superior?

Is it scriptable?


On Wed, May 13, 2015 at 10:28 AM, Sean Busbeybus...@cloudera.com

wrote:

I would love to rip out the JRuby shell entirely and make something

closer

to the Accumulo shell, but I expect that will

1) be way more work

2) be even less compatible for those that rely on customizations.

I figured given time we could get a preview user shell (rather than
power shell via irb) together in 2.0 and aim for default in 3.0.

--
Sean
On May 13, 2015 12:19 PM, Stackst...@duboce.net  wrote:


Nice writeup Sean.

Yeah, +1 to new jruby in hbase 2.0. We'd need to be careful license

is

still amenable and hopefully jruby 9k will be slimmer than jruby

1.7+.

But if we are going to do a significant shell refactor for hbase 2.0,
should we consider doing something more radical; e.g. a new shell? If
interest, could start up a new thread so don't distract from this

one.

St.Ack




On Wed, May 13, 2015 at 9:22 AM, Sean Busbeybus...@cloudera.com

wrote:

Hi folks!

If you weren't aware, our current shell relies on Ruby,

specifically

the

REPL program IRB from JRuby. When we launched 1.0 we were on JRuby

1.6

with

defaults, which means we're stuck on Ruby 1.8.

For those that don't already know, Ruby 1.8 is super old and has

been

walking off into the sunset for a few years now. Most (but not

all!)

formal

support systems for running Ruby have EOLed 1.8 and there are

numerous

known issues with it.

Right now there's an open ticket to get us on JRuby 1.7 so that our

shell

can work on PPC systems[1]. That version of JRuby defaults to Ruby

1.9

but

can be run in Ruby 1.8 mode. There are some implementation details
outstanding, but I'm hoping that ticket can work out such that it

can

land

in branch-1.

For HBase 2.0, I'd like us to plan for a little farther out in the

future

than just updating to Ruby 1.9 (though that would be a fine

incremental

step with some non-trivial work attached). The current version of

Ruby

is

2.2. Much like the move from 1.8 -  1.9 it is not backwards

compatible.

JRuby's next major maintenance release line is version 9000[2]

and

it

will start out *only* supporting Ruby 2.2. Right now JRuby 9000 is

in

its

second preview release. They still have a few feature complete

items

to

address before they hit their first GA release.

I'd like us to move to Ruby 2.2 via JRuby 9000 for HBase 2.0.  This

will

cause operator pain to folks with advanced scripts based on Ruby

1.8,

but

it should allow us to update versions to avoid e.g. perf,

correctness,

and

security issues much more easily over the 

Re: Ruby shell versions for HBase 2.0

[Josh Elser]

Agreed on these points, Sean. Having used both, I think both approaches 
have their value and their drawbacks.


The ruby shell is _wonderful_ from having a full programming language to 
interact with. Accumulo's shell would force you to use your standard 
unix-toolbelt if you want to do any extra parsing/logic.


OTOH, general introspection and interaction with the system feels much 
more natural to me w/ Accumulo's shell. I know that's not a good way to 
quantify my feelings. I think it's mostly due to not having to write 
Ruby when I'm not actually scripting things. Another point is that 
Accumulo uses JLine under the hood which always comes with its own 
burden (things like C^c nuked the entire shell for the longest time 
because JLine didn't support it -- we sometimes have to fix JLine to get 
the functionality we want).


I also caught wind of someone actually using JavaScript via JSR-223 w/ 
the Accumulo shell, but it's not widely advertised at this point.


Finding a the right mix between ease of scripting and simplicity of 
administrative interactions is key if we want to move beyond what the 
HBase shell is now. My $0.02.


(and, to not completely derail the original conversation, moving to Ruby 
2.2 + JRuby 9000 would be what I think the right move to make is)


Sean Busbey wrote:

Pros:

* It's easier to test and maintain
* it's easier to script with
* its interactive mode feels mode focused on the task of interacting with
a cluster to me (maybe this is just acting like a psql or mysql shell)

Cons

* adding custom commands requires knowing java

[jira] [Created] (HBASE-13607) TestSplitLogManager.testGetPreviousRecoveryMode consistently failing

[Josh Elser (JIRA)]

Josh Elser created HBASE-13607:
--

 Summary: TestSplitLogManager.testGetPreviousRecoveryMode 
consistently failing
 Key: HBASE-13607
 URL: https://issues.apache.org/jira/browse/HBASE-13607
 Project: HBase
  Issue Type: Bug
  Components: test
Affects Versions: 1.1.0
Reporter: Josh Elser
Assignee: Josh Elser
Priority: Minor
 Fix For: 2.0.0, 1.2.0, 1.1.1


From Nick's 1.1.0 rc0 call for UT help:

https://builds.apache.org/job/HBase-1.1.0RC0-JDK7/72/testReport/org.apache.hadoop.hbase.master/TestSplitLogManager/testGetPreviousRecoveryMode/

{noformat}
java.lang.AssertionError: Mode4=LOG_SPLITTING
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.assertTrue(Assert.java:41)
at 
org.apache.hadoop.hbase.master.TestSplitLogManager.testGetPreviousRecoveryMode(TestSplitLogManager.java:661)
{noformat}

This is repeatedly failing locally for me.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-13603) Write test asserting desired priority of RS-Master RPCs

[Josh Elser (JIRA)]

Josh Elser created HBASE-13603:
--

 Summary: Write test asserting desired priority of RS-Master RPCs
 Key: HBASE-13603
 URL: https://issues.apache.org/jira/browse/HBASE-13603
 Project: HBase
  Issue Type: Test
  Components: rpc, test
Reporter: Josh Elser
Assignee: Josh Elser
Priority: Minor
 Fix For: 2.0.0, 1.2.0, 1.1.1


From HBASE-13351:

{quote}
Any way we can write a FT test to assert that the RS-Master APIs are treated 
with higher priority. I see your UT for asserting the annotation.
{quote}

Write a test that verifies expected RPCs are run on the correct pools in as 
real of an environment possible.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-13561) ITBLL.Verify doesn't actually evaluate counters after job completes

[Josh Elser (JIRA)]

Josh Elser created HBASE-13561:
--

 Summary: ITBLL.Verify doesn't actually evaluate counters after job 
completes
 Key: HBASE-13561
 URL: https://issues.apache.org/jira/browse/HBASE-13561
 Project: HBase
  Issue Type: Bug
  Components: integration tests
Affects Versions: 0.98.12, 1.0.0, 2.0.0, 1.1.0
Reporter: Josh Elser
Assignee: Josh Elser
 Fix For: 2.0.0, 1.1.0, 0.98.13, 1.0.2


Was digging through ITBLL and noticed this oddity:

The {{Verify}} Tool doesn't actually call {{Verify#verify(long)}} like the 
{{Loop}} Tool does. Granted, it doesn't know the total number of KVs that were 
written given the current arguments, it's not even checking to see if there 
things like UNDEFINED records found.

It seems to me that {{Verify}} should really be doing *some* checking on the 
counters like {{Loop}} does and not just leaving it up to the visual inspection 
of whomever launched the task.

Am I missing something?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-13554) Update book clarifying API stability guarantees

[Josh Elser (JIRA)]

Josh Elser created HBASE-13554:
--

 Summary: Update book clarifying API stability guarantees
 Key: HBASE-13554
 URL: https://issues.apache.org/jira/browse/HBASE-13554
 Project: HBase
  Issue Type: Task
  Components: documentation
Reporter: Josh Elser
Assignee: Josh Elser


From the Clarifying interface evolution freedom in patch releases 
[thread|http://mail-archives.apache.org/mod_mbox/hbase-dev/201504.mbox/%3CCA%2BRK%3D_CkTUfa3nWPsy2A0PZt07h%3D1stP72rcazBrw5U0JFdgXA%40mail.gmail.com%3E]
 on dev@h.a.o

Seems we have consensus that HBase uses Semantic Versioning isn't quite 
correct (or desired) at the moment. Update the documentation to make sure we're 
not misrepresenting any guarantees to users.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Clarifying interface evolution freedom in patch releases (was: Re: [VOTE] Third release candidate for HBase 1.0.1 (RC2))

[Josh Elser]

...@apache.org

wrote:


Anyone disagree with the point of view put forward by Josh

and

Sean?


On Wed, Apr 22, 2015 at 10:35 AM, Josh Elser

josh.el...@gmail.com

wrote:

Andy -- I understood your intent, but thanks for

clarifying.

(as

well

as

taking the time to break this discussion out in the first

place). I

agree

with your assessment.

re: Sean's comments, if it wasn't clear by me asking in

the

first

place,

I

also think sticking as close as possible to semver's

rules

is

the

best

approach, although I'm getting the impression that there

have

been

some

previous reservations to doing so (especially by your

comment

about

backporting features if there is demand is).

I've found adhering to the bug-fix release restrictions

can

be

a

very

painful and time-consuming task, so this is something to

get

a

representative sampling of those who do the work to make

sure

everyone

is

on board.


Sean Busbey wrote:


I'd much rather we stick with the definitions used in

Semantic

Versioning.

Our use is already confusing enough given our matrix of

compatibilities

that don't get major version for breaking protections.

We've previously discussed how we'll do additional minor

releases

when

there's sufficient interest in the new features present

there.

What's

building that demand if any backwards compatible change

can

go

back

into a

patch release?

Would we have an easier time restraining ourselves if we

had a

regular

schedule planned around new minor versions?


On Wed, Apr 22, 2015 at 12:03 PM, Josh Elser

josh.el...@gmail.com

wrote:

  While I can understand the desire to want to add

things,

I

do

think

it

makes things harder for users to reliably write code

against

versions

of

HBase which (by their view) should be completely

compatible

with

one

another.

Take this extremely hypothetical situation: I'm new to

HBase

and

start

writing some code against HBase 1.0.1 which was just

deployed

at

my

$job. I
don't _know_ what APIs are new, I just know what exists

and

treat

that

as

acceptable for me to be using. Meanwhile in production,

some

other

people

find a bug with HBase 1.0.1 and roll back to 1.0.0

which

they

had

been

previously using. My reaction would be of course my

code

should

work

with
HBase 1.0.0, I only used the public API when in fact

this

is

not

true.

Personally, I think it's a little bold to say semver is

even

in

use

if

this principal isn't being followed as it doesn't

follow

at

all

with

my

understanding on the guarantees defined by semver for

bug-fix

releases.

That being said, if the intent *is* to allow ourselves

to

make

these

sorts
of changes, I just think some sort of disclaimer should

be

present:

- HBase uses Semantic Versioning for its release

versioning

+ HBase uses Semantic Versioning for its release

versioning

with

a

caveat

that methods and members might be added in newer

bug-fix

releases

that

were
not present in the previous bug-fix release.


Andrew Purtell wrote:

  [Subject changed]

On Tue, Apr 21, 2015 at 8:47 PM, Josh Elser

josh.el...@gmail.com

  wrote:

   I was a little surprised when I noticed method

additions

to

InterfaceAudience.Public annotated classes. This

means

that a

user

could
write code against 1.0.1 that would not work against

1.0.0

which

seems

undesirable for a bugfix release. I read over the

book

section

on

compatibility and didn't see this addressed, so I

thought

I'd

ask.



Let's clarify this. It's not the first time this

question

has

been

asked.

To get things moving:

I propose the following addition to the Client API

compatibility

section
of Section 11.1:

+ APIs available in a patch version will be available

in

all

later

+ patch versions. However, new APIs may be added which

will

not

be

+ available in earlier patch versions.

I propose the following change to the Client Binary

compatibility

section
of Section 11.1:

- Old client code can run unchanged (no recompilation

needed)

against

new
jars.
+ Client code written to APIs available in a given

patch

release

+ can run unchanged (no recompilation needed) against

the

new

+ jars of later patch versions.


What do you think?

If these changes are (mostly) ok, then this clarifies

in

one

direction.

If these changes are not acceptable, I will propose

edits

that

clarify

toward the opposite meaning. ​

Re: Clarifying interface evolution freedom in patch releases (was: Re: [VOTE] Third release candidate for HBase 1.0.1 (RC2))

[Josh Elser]

Enis Söztutar wrote:

+1 to the proposal.

The problem is that we have a very big API surface especially with the
coprocessors included in the report. Even simple bug fixes can introduce
protected or public methods to base classes, which makes patch releases
very hard to maintain. I would not want to spend the effort to spend tons
of time trying to make a patch not introduce new methods in order to
backport. That effort can be spent elsewhere IMO.


Abso-friggin-lutely. Following bug-fix strictness is a _massive_ pain in 
the rear (been dealing with this over in Accumulo-landia and it sucks). 
IMO, the crux of what to consider here is finding the acceptable level 
to which HBase Devs want to hold themselves and how much knowledge is 
just expected of users.


Like you said, it is a lot of effort for what seems like ultimately very 
little effect on users. It will take constant effort by everyone landing 
patches in a bug-fix branch to not make the RM want to punch themself in 
the face.



Looking at the report
https://people.apache.org/~enis/1.0.0_1.0.1RC2_compat_report.html, nothing
strikes me as new functionality. Going from current 1.0.0 to 1.0.1RC2
should actually be as you would expect from upgrading a patch release.

Yes, adding new API in patch releases will make downgrading harder, but I
think that is an acceptable tradeoff. We can document that if your
application compiles (meaning that you are not using new API) with 1.0.0,
then you can swap your jars in a binary compat manner.


Like you pointed out in the other thread, though, choosing to follow 
semver is for the user. Your proposed solution adds in extra complexity 
on users that would otherwise be as simple as flipping the version on 
their Maven/Ivy configuration. Like you say, they should still be able 
to do things with the new jars against the old cluster, but it is a 
sharp-corner users would need to be aware of that isn't semver as 
advertised.


Really, I'm trying to play devil's advocate here. In the end, my only 
concern would be appropriate advertisement of what decision is made.



Enis

On Thu, Apr 23, 2015 at 10:03 AM, Andrew Purtellapurt...@apache.org
wrote:


Anyone disagree with the point of view put forward by Josh and Sean?

Re: Clarifying interface evolution freedom in patch releases (was: Re: [VOTE] Third release candidate for HBase 1.0.1 (RC2))

[Josh Elser]

Stack wrote:

On Thu, Apr 23, 2015 at 3:08 PM, Andrew Purtellapurt...@apache.org  wrote:


  So we like semver, not we use semver?



and Sean's


  No longer claiming semver would also have the added benefit of making it for

me to easier to explain our compatibility promises to people.

Yeah, 'we like semvar'/'we are almost semvar' + and are actively working
toward 'semvar' would give us leeway that we seem to need as we learn what
semvar implies.

Does this 'admission' help with the which-hadoop thread too?

St.Ack



+1

Re: Clarifying interface evolution freedom in patch releases (was: Re: [VOTE] Third release candidate for HBase 1.0.1 (RC2))

[Josh Elser]

While I can understand the desire to want to add things, I do think it 
makes things harder for users to reliably write code against versions of 
HBase which (by their view) should be completely compatible with one 
another.


Take this extremely hypothetical situation: I'm new to HBase and start 
writing some code against HBase 1.0.1 which was just deployed at my 
$job. I don't _know_ what APIs are new, I just know what exists and 
treat that as acceptable for me to be using. Meanwhile in production, 
some other people find a bug with HBase 1.0.1 and roll back to 1.0.0 
which they had been previously using. My reaction would be of course my 
code should work with HBase 1.0.0, I only used the public API when in 
fact this is not true.


Personally, I think it's a little bold to say semver is even in use if 
this principal isn't being followed as it doesn't follow at all with my 
understanding on the guarantees defined by semver for bug-fix releases.


That being said, if the intent *is* to allow ourselves to make these 
sorts of changes, I just think some sort of disclaimer should be present:


- HBase uses Semantic Versioning for its release versioning
+ HBase uses Semantic Versioning for its release versioning with a 
caveat that methods and members might be added in newer bug-fix releases 
that were not present in the previous bug-fix release.


Andrew Purtell wrote:

[Subject changed]

On Tue, Apr 21, 2015 at 8:47 PM, Josh Elserjosh.el...@gmail.com  wrote:


I was a little surprised when I noticed method additions to
InterfaceAudience.Public annotated classes. This means that a user could
write code against 1.0.1 that would not work against 1.0.0 which seems
undesirable for a bugfix release. I read over the book section on
compatibility and didn't see this addressed, so I thought I'd ask.



Let's clarify this. It's not the first time this question has been asked.

To get things moving:

I propose the following addition to the Client API compatibility section
of Section 11.1:

+ APIs available in a patch version will be available in all later
+ patch versions. However, new APIs may be added which will not be
+ available in earlier patch versions.

I propose the following change to the Client Binary compatibility section
of Section 11.1:

- Old client code can run unchanged (no recompilation needed) against new
jars.
+ Client code written to APIs available in a given patch release
+ can run unchanged (no recompilation needed) against the new
+ jars of later patch versions.


What do you think?

If these changes are (mostly) ok, then this clarifies in one direction.

If these changes are not acceptable, I will propose edits that clarify
toward the opposite meaning. ​

Re: Clarifying interface evolution freedom in patch releases (was: Re: [VOTE] Third release candidate for HBase 1.0.1 (RC2))

[Josh Elser]

Andy -- I understood your intent, but thanks for clarifying. (as well as 
taking the time to break this discussion out in the first place). I 
agree with your assessment.


re: Sean's comments, if it wasn't clear by me asking in the first place, 
I also think sticking as close as possible to semver's rules is the best 
approach, although I'm getting the impression that there have been some 
previous reservations to doing so (especially by your comment about 
backporting features if there is demand is).


I've found adhering to the bug-fix release restrictions can be a very 
painful and time-consuming task, so this is something to get a 
representative sampling of those who do the work to make sure everyone 
is on board.


Sean Busbey wrote:

I'd much rather we stick with the definitions used in Semantic Versioning.
Our use is already confusing enough given our matrix of compatibilities
that don't get major version for breaking protections.

We've previously discussed how we'll do additional minor releases when
there's sufficient interest in the new features present there. What's
building that demand if any backwards compatible change can go back into a
patch release?

Would we have an easier time restraining ourselves if we had a regular
schedule planned around new minor versions?


On Wed, Apr 22, 2015 at 12:03 PM, Josh Elserjosh.el...@gmail.com  wrote:


While I can understand the desire to want to add things, I do think it
makes things harder for users to reliably write code against versions of
HBase which (by their view) should be completely compatible with one
another.

Take this extremely hypothetical situation: I'm new to HBase and start
writing some code against HBase 1.0.1 which was just deployed at my $job. I
don't _know_ what APIs are new, I just know what exists and treat that as
acceptable for me to be using. Meanwhile in production, some other people
find a bug with HBase 1.0.1 and roll back to 1.0.0 which they had been
previously using. My reaction would be of course my code should work with
HBase 1.0.0, I only used the public API when in fact this is not true.

Personally, I think it's a little bold to say semver is even in use if
this principal isn't being followed as it doesn't follow at all with my
understanding on the guarantees defined by semver for bug-fix releases.

That being said, if the intent *is* to allow ourselves to make these sorts
of changes, I just think some sort of disclaimer should be present:

- HBase uses Semantic Versioning for its release versioning
+ HBase uses Semantic Versioning for its release versioning with a caveat
that methods and members might be added in newer bug-fix releases that were
not present in the previous bug-fix release.


Andrew Purtell wrote:


[Subject changed]

On Tue, Apr 21, 2015 at 8:47 PM, Josh Elserjosh.el...@gmail.com   wrote:

  I was a little surprised when I noticed method additions to

InterfaceAudience.Public annotated classes. This means that a user could
write code against 1.0.1 that would not work against 1.0.0 which seems
undesirable for a bugfix release. I read over the book section on
compatibility and didn't see this addressed, so I thought I'd ask.



Let's clarify this. It's not the first time this question has been asked.

To get things moving:

I propose the following addition to the Client API compatibility section
of Section 11.1:

+ APIs available in a patch version will be available in all later
+ patch versions. However, new APIs may be added which will not be
+ available in earlier patch versions.

I propose the following change to the Client Binary compatibility
section
of Section 11.1:

- Old client code can run unchanged (no recompilation needed) against new
jars.
+ Client code written to APIs available in a given patch release
+ can run unchanged (no recompilation needed) against the new
+ jars of later patch versions.


What do you think?

If these changes are (mostly) ok, then this clarifies in one direction.

If these changes are not acceptable, I will propose edits that clarify
toward the opposite meaning. ​

Re: [VOTE] Third release candidate for HBase 1.0.1 (RC2) is available. Please vote by April 24 2015

[Josh Elser]

+1 (non-binding)

* Checked sigs/xsums
* Looked for binary files in source tarball (found none unexpected)
* Built/ran-tests from source
* Started instance from binary tarball, ran shell commands
* Perused japi report

One note... in the japi report, I was a little surprised when I noticed 
method additions to InterfaceAudience.Public annotated classes. This 
means that a user could write code against 1.0.1 that would not work 
against 1.0.0 which seems undesirable for a bugfix release. I read over 
the book section on compatibility and didn't see this addressed, so I 
thought I'd ask.


Enis Söztutar wrote:

I am pleased to announce that the third release candidate for the release
1.0.1
(HBase-1.0.1RC2), is available for download at
https://dist.apache.org/repos/dist/dev/hbase/hbase-1.0.1RC2/

  Maven artifacts are also available in the temporary repository
https://repository.apache.org/content/repositories/orgapachehbase-1075

Signed with my code signing key E964B5FF. Can be found here:
https://people.apache.org/keys/committer/enis.asc

  Signed tag in the repository can be found here:
https://git-wip-us.apache.org/repos/asf?p=hbase.git;a=tag;h=e84dbc0b48861b6ff531a82321745f0607a7525f


HBase 1.0.1 is the next “patch” release in the 1.0.x release line and
supersedes 1.0.0.
According to the HBase’s semantic version guide (See [1]), the release
candidate is
source and binary compatible with 1.0.0 for client applications and server
side libraries
(coprocessors, filters, etc).

Binary / source compatibility report with 1.0.0 can be reached here:
https://people.apache.org/~enis/1.0.0_1.0.1RC2_compat_report.html


1.0.1 release contains 119 fixes on top of 1.0.0 release. Most of
the changes are
bug fixes except for the following:

[HBASE-13002] - Make encryption cipher configurable
[HBASE-13044] - Configuration option for disabling coprocessor loading
[HBASE-13054] - Provide more tracing information for locking/latching
events.
[HBASE-13059] - Set executable bit for scripts in dev-support
[HBASE-13086] - Show ZK root node on Master WebUI
[HBASE-13109] - Make better SEEK vs SKIP decisions during scanning
[HBASE-13120] - Allow disabling hadoop classpath and native library lookup
[HBASE-13132] - Improve RemoveColumn action debug message
[HBASE-13162] - Add capability for cleaning hbase acls to hbase cleanup
script.
[HBASE-13168] - Backport HBASE-12590 A solution for data skew in
HBase-Mapreduce Job
[HBASE-13183] - Make ZK tickTime configurable in standalone HBase
  [HBASE-13342] - Fix incorrect interface annotations
  [HBASE-12869] - Add a REST API implementation of the ClusterManager
interface
[HBASE-13380] - Cherry pick the HBASE-12808 compatibility checker tool back
to 0.98+

Full list of the issues can be found at:
  
https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12329042projectId=12310753
https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12329042projectId=12310753

  Compatibility
  -
This release (1.0.1) is source, wire and binary compatible with 1.0.0
release. Client
applications does not have to be recompiled with the new version (unless
new API is used)
if upgrading from 1.0.0. It is a drop-in replacement.

See release notes for 1.0.0 [2] for compatibility with earlier
versions (0.94, 0.96, 0.98).
Compatibility of 1.0.1 with earlier versions is the same as in 1.0.0.

Source Compatibility:
Client side code in HBase-1.0.x is (mostly) source compatible with 0.98.x
  versions. Some minor API changes might be needed from the client side.

Wire Compatibility:
HBase-1.0.x release is wire compatible with 0.98.x releases. Clients and
  servers running in different versions as long as new features are not used
  should be possible.
A rolling upgrade from 0.98.x clusters to 1.0.x is supported as well.
Rolling upgrade from 0.96 directly to 1.0.x is not supported.
1.0.x is NOT wire compatible with earlier releases (0.94, etc).

Binary Compatibility:
Binary compatibility at the Java API layer with earlier versions (0.98.x,
0.96.x and 0.94.x) is not supported. You may have to recompile your client
code and any server side code (coprocessors, filters etc) referring to
hbase jars.

Other Compatibility issues:
  - [HBASE-13479] - [branch-1.0] Master should not bind to region server
ports
 (Master now respects hbase.master.port and hbase.master.info.port)
  - [HBASE-13481] - Master should respect master (old) DNS/bind related
configurations
  - [HBASE-13289https://issues.apache.org/jira/browse/HBASE-13289] - typo
in splitSuccessCount metric
  - [HBASE-13275] - Setting hbase.security.authorization to false does not
disable authorization
  - [HBASE-13362] - Set max result size from client only (like scanner
caching)


Upgrading
  -
This release is rolling upgradable from 1.0.0 release.

See [2] and [3] for upgrade instructions from earlier versions. Upgrading
to 1.0.1 is similar
to upgrading to 1.0.0 as documented in [3].

 From 0.98.x : Upgrade from 0.98.x in regular 

[jira] [Created] (HBASE-13520) NullPointerException in TagRewriteCell

[Josh Elser (JIRA)]

Josh Elser created HBASE-13520:
--

 Summary: NullPointerException in TagRewriteCell
 Key: HBASE-13520
 URL: https://issues.apache.org/jira/browse/HBASE-13520
 Project: HBase
  Issue Type: Bug
Reporter: Josh Elser
Assignee: Josh Elser
 Fix For: 2.0.0, 1.1.0, 1.0.2


Found via running {{IntegrationTestIngestWithVisibilityLabels}} with Kerberos 
enabled.

{noformat}
2015-04-20 18:54:36,712 ERROR 
[B.defaultRpcServer.handler=17,queue=2,port=16020] ipc.RpcServer: Unexpected 
throwable object
java.lang.NullPointerException
at 
org.apache.hadoop.hbase.TagRewriteCell.getTagsLength(TagRewriteCell.java:157)
at 
org.apache.hadoop.hbase.TagRewriteCell.heapSize(TagRewriteCell.java:186)
at 
org.apache.hadoop.hbase.CellUtil.estimatedHeapSizeOf(CellUtil.java:568)
at 
org.apache.hadoop.hbase.regionserver.DefaultMemStore.heapSizeChange(DefaultMemStore.java:1024)
at 
org.apache.hadoop.hbase.regionserver.DefaultMemStore.internalAdd(DefaultMemStore.java:259)
at 
org.apache.hadoop.hbase.regionserver.DefaultMemStore.upsert(DefaultMemStore.java:567)
at 
org.apache.hadoop.hbase.regionserver.DefaultMemStore.upsert(DefaultMemStore.java:541)
at org.apache.hadoop.hbase.regionserver.HStore.upsert(HStore.java:2154)
at 
org.apache.hadoop.hbase.regionserver.HRegion.increment(HRegion.java:7127)
at 
org.apache.hadoop.hbase.regionserver.RSRpcServices.increment(RSRpcServices.java:504)
at 
org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2020)
at 
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:31967)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2106)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101)
at 
org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
at org.apache.hadoop.hbase.ipc.RpcExecutor$2.run(RpcExecutor.java:107)
at java.lang.Thread.run(Thread.java:745)
{noformat}

HBASE-11870 tried to be tricky when only the tags of a {{Cell}} need to be 
altered in the write-pipeline by creating a {{TagRewriteCell}} which avoided 
copying all components of the original {{Cell}}. In an attempt to help free the 
tags on the old cell that we wouldn't be referencing anymore, 
{{TagRewriteCell}} nulls out the original {{byte[] tags}}.

This causes a problem in that the implementation of {{heapSize()}} as it 
{{getTagsLength()}} on the original {{Cell}} instead of the on {{this}}. 
Because the tags on the passed in {{Cell}} (which was also a 
{{TagRewriteCell}}) were null'ed out in the constructor, this results in a NPE 
by the byte array is null.

I believe this isn't observed in normal, unsecure deployments because there is 
only one RegionObserver/Coprocessor loaded that gets invoked via 
{{postMutationBeforeWAL}}. When there is only one RegionObserver, the 
TagRewriteCell isn't passed another TagRewriteCell, but instead a cell from the 
wire/protobuf. This means that the optimization isn't performed. When we have 
two (or more) observers that a TagRewriteCell passes through (and a new 
TagRewriteCell is created and the old TagRewriteCell's tags array is nulled), 
this enables the described-above NPE.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-13480) ShortCircuitConnection doesn't short-circuit all calls as expected

[Josh Elser (JIRA)]

Josh Elser created HBASE-13480:
--

 Summary: ShortCircuitConnection doesn't short-circuit all calls as 
expected
 Key: HBASE-13480
 URL: https://issues.apache.org/jira/browse/HBASE-13480
 Project: HBase
  Issue Type: Bug
  Components: Client
Affects Versions: 1.0.0, 2.0.0, 1.1.0
Reporter: Josh Elser
Assignee: Josh Elser
 Fix For: 2.0.0, 1.1.0, 1.0.2


Noticed the following situation in debugging unexpected unit tests failures in 
HBASE-13351.

{{ConnectionUtils#createShortCircuitHConnection(Connection, ServerName, 
AdminService.BlockingInterface, ClientService.BlockingInterface)}} is intended 
to avoid the extra RPC by calling the server's instantiation of the protobuf 
rpc stub directly for the AdminService and ClientService.

The problem is that this is insufficient to actually avoid extra remote RPCs 
as all other calls to the Connection are routed to a real Connection 
instance. As such, any object created by the real Connection (such as an 
HTable) will use the real Connection, not the SSC.

The end result is that 
{{MasterRpcService#reportRegionStateTransition(RpcController, 
ReportRegionStateTransitionRequest)}} will make additional remote RPCs over 
what it thinks is an SSC through a {{Get}} on {{HTable}} which was constructed 
using the SSC, but the {{Get}} itself will use the underlying real Connection 
instead of the SSC. With insufficiently sized thread pools, this has been 
observed to result in RPC deadlock in the HMaster where an RPC attempts to make 
another RPC but there are no more threads available to service the second RPC 
so the first RPC blocks indefinitely.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-13458) Create/expand unit test to exercise htrace instrumentation

[Josh Elser (JIRA)]

Josh Elser created HBASE-13458:
--

 Summary: Create/expand unit test to exercise htrace instrumentation
 Key: HBASE-13458
 URL: https://issues.apache.org/jira/browse/HBASE-13458
 Project: HBase
  Issue Type: Test
  Components: test
Reporter: Josh Elser
Assignee: Josh Elser
Priority: Minor
 Fix For: 2.0.0, 1.1.0


From HBASE-13078, [~ndimiduk] suggested that we can also add a Medium/Large 
unit test that does some more assertions over the HTrace instrumentation. Some 
loose goals:

* Try to verify that spans continue from HBase into HDFS
* Ensure that user-created spans continue into HBase
* Validate expected API calls have expected instrumentation

Other ideas that people have? Any pain points experienced prior by people?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-13381) Expand TestSizeFailures to include small scans

[Josh Elser (JIRA)]

Josh Elser created HBASE-13381:
--

 Summary: Expand TestSizeFailures to include small scans
 Key: HBASE-13381
 URL: https://issues.apache.org/jira/browse/HBASE-13381
 Project: HBase
  Issue Type: Improvement
  Components: test
Reporter: Josh Elser
Assignee: Josh Elser
Priority: Minor
 Fix For: 1.1.0, 2.2.0, 0.98.14, 1.0.2


From Jonathan on HBASE-13335:

{quote}
It may also be a good idea to extend TestSizeFailures so that it also tests to 
ensure that all data is seen when the scan is small (e.g. perform that same 
scan near the end with but configure it with Scan.setSmall(true)). Even though 
that wouldn't be a small scan, it would test to make sure the fix behaves as 
expected.
{quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-13351) Annotate internal MasterRpcServices methods with admin priority

[Josh Elser (JIRA)]

Josh Elser created HBASE-13351:
--

 Summary: Annotate internal MasterRpcServices methods with admin 
priority
 Key: HBASE-13351
 URL: https://issues.apache.org/jira/browse/HBASE-13351
 Project: HBase
  Issue Type: Improvement
  Components: master
Reporter: Josh Elser
Assignee: Josh Elser
 Fix For: 2.0.0, 1.1.0


HBASE-12071, among other things, introduced annotating RPC methods to give 
certain methods priority over others. Namely, this helps ensure that client 
requests cannot starve out internal RPC between master and regionserver.

Similarly, we can do the same thing for Master RPC methods that are invoked by 
RS's.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-13262) ResultScanner doesn't return all rows in Scan

[Josh Elser (JIRA)]

Josh Elser created HBASE-13262:
--

 Summary: ResultScanner doesn't return all rows in Scan
 Key: HBASE-13262
 URL: https://issues.apache.org/jira/browse/HBASE-13262
 Project: HBase
  Issue Type: Bug
  Components: Client
 Environment: Single node, pseduo-distributed 1.1.0-SNAPSHOT
Reporter: Josh Elser
Assignee: Josh Elser
Priority: Critical
 Fix For: 1.1.0


Tried to write a simple Java client again 1.1.0-SNAPSHOT.

* Write 1M rows, each row with 1 family, and 10 qualifiers (values [0-9]), for 
a total of 10M cells written
* Read back the data from the table, ensure I saw 10M cells

Running it against {{04ac1891}} (and earlier) yesterday, I would get ~20% of 
the actual rows. Running against 1.0.0, returns all 10M records as expected.

[Code I was 
running|https://github.com/joshelser/hbase-hwhat/blob/master/src/main/java/hbase/HBaseTest.java]
 for the curious.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-13256) HBaseConfiguration#checkDefaultsVersion(Configuration) has spelling error

[Josh Elser (JIRA)]

Josh Elser created HBASE-13256:
--

 Summary: HBaseConfiguration#checkDefaultsVersion(Configuration) 
has spelling error
 Key: HBASE-13256
 URL: https://issues.apache.org/jira/browse/HBASE-13256
 Project: HBase
  Issue Type: Improvement
  Components: Client
Reporter: Josh Elser
Assignee: Josh Elser
Priority: Trivial
 Fix For: 2.0.0, 1.1.0


Happened to stumble onto the exception thrown by 
{{HBaseConfiguration#checkDefaultsVersion}}. This improves the spelling/grammar.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-13255) Bad grammar in RegionServer status page

[Josh Elser (JIRA)]

Josh Elser created HBASE-13255:
--

 Summary: Bad grammar in RegionServer status page
 Key: HBASE-13255
 URL: https://issues.apache.org/jira/browse/HBASE-13255
 Project: HBase
  Issue Type: Improvement
  Components: monitoring
Reporter: Josh Elser
Assignee: Josh Elser
Priority: Trivial
 Fix For: 2.0.0, 1.1.0


Noticed on the rs-status page, the blurb under the Regions section could use 
some grammatical improvements.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-13236) Clean up m2e-related warnings/errors from poms

[Josh Elser (JIRA)]

Josh Elser created HBASE-13236:
--

 Summary: Clean up m2e-related warnings/errors from poms
 Key: HBASE-13236
 URL: https://issues.apache.org/jira/browse/HBASE-13236
 Project: HBase
  Issue Type: Improvement
  Components: build
Reporter: Josh Elser
Priority: Minor
 Fix For: 2.0.0, 1.1.0


Pulled down HBase, imported into Eclipse (either directly with m2eclipse or by 
running {{mvn eclipse:eclipse}} to generate the projects), and this results in 
a bunch of red due to executions/goals of certain plugins being unable to run 
in the context of eclipse.

The lifecycle-mapping plugin can be used to get around these errors (and 
already exists in the pom).

Add more mappings to the configuration so that a fresh import into Eclipse is 
not hindered by a bunch of false' errors.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)