Re: [ANNOUNCE] New HBase committer Rushabh Shah

[Geoffrey Jacoby]

Congratulations, Rushabh!

Geoffrey

On Thu, Dec 15, 2022 at 10:26 AM Tak Lon (Stephen) Wu 
wrote:

> Congratulations, and welcome!
>
> -Stephen
>
> On Thu, Dec 15, 2022 at 5:39 AM Viraj Jasani  wrote:
>
> > Very well deserved! Congratulations and Welcome, Rushabh!!
> >
> >
> > On Wed, Dec 14, 2022 at 10:57 PM 张铎(Duo Zhang) 
> > wrote:
> >
> > > On behalf of the Apache HBase PMC, I am pleased to announce that
> > > Rushabh Shah(shahrs87)
> > > has accepted the PMC's invitation to become a committer on the
> > > project. We appreciate all
> > > of Rushabh's generous contributions thus far and look forward to his
> > > continued involvement.
> > >
> > > Congratulations and welcome, Rushabh Shah!
> > >
> > > 我很高兴代表 Apache HBase PMC 宣布 Rushabh Shah 已接受我们的邀请，成
> > > 为 Apache HBase 项目的 Committer。感谢 Rushabh Shah 一直以来为 HBase 项目
> > > 做出的贡献，并期待他在未来继续承担更多的责任。
> > >
> > > 欢迎 Rushabh Shah！
> > >
> >
>

[DISCUSS] HBase 2.5 / Hadoop 3 artifacts

[Geoffrey Jacoby]

I see that the next HBase 2.5 RC is imminent, and before that's set in
stone, I wanted to bring up the question of whether there will be official
HBase 2.5 binaries built with the Hadoop 3 profile and available in the
usual Maven repositories. (In addition to the usual Hadoop 2 profile
binaries)

The HBase 2.x line has a commitment to maintain support for Hadoop 2.x, but
Hadoop 3.3 is the current stable Hadoop line and the most recent release
notes [1] encourage all users of Hadoop  2.x to upgrade to Hadoop 3.

Without convenience artifacts built against Hadoop 3, no end-users with
Hadoop 3 clusters will be able to use the Apache-distributed binaries and
will instead have to recompile HBase from source themselves, or use a 3rd
party distribution that does so for them.

This is especially inconvenient for downstream projects such as Apache
Phoenix, which has never  officially supported the HBase 2.x / Hadoop 2.10
combination. (It currently supports only HBase 2.3 or 2.4 with Hadoop 3.
HBase 2.5 support will be added very shortly after its release as part of
Phoenix 5.2.)

To even run the Phoenix IT tests locally requires contributors to download
the HBase source release and manually mvn install to their local maven repo
using the Hadoop 3 profile, to avoid crashes in the HBase minicluster.[2]
This is a barrier to new contributors and confuses even veteran ones, and
has to be done again for every new HBase release.

In general, I expect the Hadoop 3 user base to grow and the Hadoop 2.10
user base to shrink with every future HBase 2 release, so I think this is a
worthwhile improvement.

Thanks,

Geoffrey

[1] https://hadoop.apache.org/release/3.3.4.html
[2] https://github.com/apache/phoenix/blob/master/BUILDING.md

[jira] [Resolved] (HBASE-24768) Clear cached service kerberos ticket in case of SASL failures thrown from server side

[Geoffrey Jacoby (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-24768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Geoffrey Jacoby resolved HBASE-24768.
-
Fix Version/s: 1.7.0
   Resolution: Fixed

This JIRA was merged back in October 2020 and seems to have been included in 
1.7.0, but wasn't resolved and didn't have a Fix Version. 

> Clear cached service kerberos ticket in case of SASL failures thrown from 
> server side
> -
>
> Key: HBASE-24768
> URL: https://issues.apache.org/jira/browse/HBASE-24768
> Project: HBase
>  Issue Type: Bug
>Reporter: Sandeep Guggilam
>Priority: Major
> Fix For: 1.7.0
>
>
> We setup a SASL connection using different mechanisms like Digest, Kerberos 
> from master to RS for various activities like region assignment etc. In case 
> of SASL connect failures, we try to dispose of the SaslRpcClient and try to 
> relogin from the keytab on the client side. However the relogin from keytab 
> method doesn't clear off the service ticket cached in memory unless TGT is 
> about to expire within a timeframe.
> This actually causes an issue where there is a keytab refresh that happens 
> because of expiry  on the RS server and throws a SASL connect error when 
> Master reaches out to the RS server with the cached service ticket that no 
> longer works with the new refreshed keytab. We might need to clear off the 
> service ticket cached as there could be a credential refresh on the RS server 
> side when handling connect failures



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [RESULT][VOTE] First release candidate for HBase 1.4.14 is available for download

[Geoffrey Jacoby]

I count 2 +1's (from Reid and Duo) and 1 +0 where Guanghao asked for a
discussion about whether a single failing unit test should block the
release. Was that changed later to a +1?

Geoffrey

On Mon, Oct 25, 2021 at 10:42 AM 张铎(Duo Zhang) 
wrote:

> With 3 binding +1s, no -1, the vote passes.
>
> Let me push the release out.
>
> 张铎(Duo Zhang)  于2021年10月25日周一 下午10:37写道：
>
> > Here is my own +1.
> >
> >
> >
> > Guanghao Zhang  于2021年10月25日周一 下午6:48写道：
> >
> >> +0 from me as there was one UT failed. Will change the vote if it is
> not a
> >> blocker.
> >> *Signature: ok
> >> *Checksum : ok
> >> *Rat check (1.8.0_301): ok
> >> - mvn clean apache-rat:check
> >> *Built from source (1.8.0_301): ok
> >> - mvn clean install -DskipTests
> >> *Unit tests (1.8.0_301): TestLogLevel always failed
> >> - mvn package -P runSmallTests
> >> *Started a mini cluster, UI check: ok
> >> *Shell check: ok
> >>
> >> Reid Chan  于2021年10月21日周四 下午12:01写道：
> >>
> >> > +1
> >> > * Signature: ok
> >> > * Checksum : ok
> >> > * Rat check (1.8.0_232): ok
> >> >  - mvn clean apache-rat:check
> >> > * Built from source (1.8.0_232): ok
> >> >  - mvn clean install  -DskipTests
> >> > * Unit tests pass (1.8.0_232): ok
> >> >  - mvn package -P runSmallTests
> >> > -Dsurefire.rerunFailingTestsCount=3
> >> >
> >> >
> >> > On Thu, Oct 21, 2021 at 9:27 AM Duo Zhang 
> wrote:
> >> >
> >> > > Please vote on this Apache HBase release candidate,
> >> > > hbase-1.4.14RC0
> >> > >
> >> > > The VOTE will remain open for at least 72 hours.
> >> > >
> >> > > [ ] +1 Release this package as Apache HBase 1.4.14
> >> > > [ ] -1 Do not release this package because ...
> >> > >
> >> > > The tag to be voted on is 1.4.14RC0:
> >> > >
> >> > >   https://github.com/apache/hbase/tree/1.4.14RC0
> >> > >
> >> > > This tag currently points to git reference
> >> > >
> >> > >   e7cbc2debc11a01dd4f3e6f6d6992b7bd307bbcb
> >> > >
> >> > > The release files, including signatures, digests included
> >> > > in this RC can be found at:
> >> > >
> >> > >   https://dist.apache.org/repos/dist/dev/hbase/1.4.14RC0/
> >> > >
> >> > > The release notes can be found at:
> >> > >
> >> > >
> >> > >
> >> > >
> >> >
> >>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12310753=12346995
> >> > >
> >> > > Maven artifacts are available in a staging repository at:
> >> > >
> >> > >
> >> >
> https://repository.apache.org/content/repositories/orgapachehbase-1467/
> >> > >
> >> > > Artifacts were signed with the 9AD2AE49 key which can be found in:
> >> > >
> >> > >   https://dist.apache.org/repos/dist/release/hbase/KEYS
> >> > >
> >> > > Notice that 1.4.14 will be the last release for the 1.4.x release
> >> line.
> >> > > Users
> >> > > are encouraged to upgrade to the 2.x stable release line.
> >> > >
> >> > > To learn more about Apache HBase, please see
> >> > >
> >> > >   http://hbase.apache.org/
> >> > >
> >> > > Thanks,
> >> > > Your HBase Release Manager
> >> > >
> >> >
> >>
> >
>

Re: [ANNOUNCE] New HBase committer Zhuoyue Huang(GeorryHuang)

[Geoffrey Jacoby]

Congratulations, Zhuoyue!

Geoffrey

On Thu, Oct 14, 2021 at 11:37 AM Andrew Purtell 
wrote:

> Congratulations and welcome!
>
> > On Oct 14, 2021, at 12:14 AM, Guanghao Zhang  wrote:
> >
> > Folks,
> >
> > On behalf of the Apache HBase PMC I am pleased to announce that Zhuoyue
> > Huang has accepted the PMC's invitation to become a committer on the
> > project.
> >
> > We appreciate all of the great contributions Zhuoyue Huang has made to
> the
> > community thus far and we look forward to his continued involvement.
> >
> > Allow me to be the first to congratulate Zhuoyue Huang on his new role!
> >
> > Thanks.
>

Re: [ANNOUNCE] New HBase PMC Bharath Vissapragada

[Geoffrey Jacoby]

Congratulations, Bharath!

On Mon, Aug 2, 2021 at 4:36 AM 张铎(Duo Zhang)  wrote:

> Congratulations!
>
> Peter Somogyi  于2021年8月2日周一 下午4:10写道：
> >
> > Congratulations!
> >
> > On Fri, Jul 30, 2021 at 3:26 PM Viraj Jasani  wrote:
> >
> > > On behalf of the Apache HBase PMC I am pleased to announce that Bharath
> > > Vissapragada has accepted our invitation to become a PMC member on the
> > > HBase project. We appreciate Bharath stepping up to take more
> > > responsibility for the project.
> > >
> > > Congratulations and welcome, Bharath!
> > >
>

[jira] [Created] (HBASE-26115) ServerTestHBaseCluster interface for testing coprocs

[Geoffrey Jacoby (Jira)]

Geoffrey Jacoby created HBASE-26115:
---

 Summary: ServerTestHBaseCluster interface for testing coprocs
 Key: HBASE-26115
 URL: https://issues.apache.org/jira/browse/HBASE-26115
 Project: HBase
  Issue Type: Test
Affects Versions: 3.0.0-alpha-1
Reporter: Geoffrey Jacoby
Assignee: Geoffrey Jacoby


The new TestingHBaseCluster introduced in HBASE-26080 provides a clean way for 
downstream developers writing features using the HBase client APIs to test 
their code. Its inner minicluster class, HBaseTestingUtil, was left unexposed 
with an interface audience of Phoenix, because coprocessors might need access 
to the internals of HBase itself to be tested.

Occasionally, a developer outside of HBase and Phoenix might need the same 
access to the internals. One way to do this would be to introduce a new 
interface, ServerTestHBaseCluster, that extends TestingHBaseCluster and exposes 
the HBaseTestingUtil, with an interface audience of COPROC, REPLICATION (for 
custom endpoints), PHOENIX.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [NOTICE] HBaseTestingUtility is now deprecated, starting from 3.0.0

[Geoffrey Jacoby]

It would of course be possible to solve any particular gap with a special
LP method or interface. I see three problems with that:

1. These special-purpose methods would increase in number over time, and
make the new interface less clean and more cluttered. I really like how
clean it is right now.
2. This requires us to know in advance what special-purpose methods are
needed. There are probably other developers out there with a very different
set of testing needs than mine that I'd never think of. This is
particularly difficult in an Apache-licensed project, where APIs are built
in public but often consumed in private organizations.
3. Some developers who need special-purpose LP methods won't be paying
attention to this thread, or only realize a gap later when building a new
feature, and will have to file a JIRA and either wait for the next HBase
release or just use the old minicluster anyway. That's not a good developer
experience.

Providing a normal mode (the new interface) and an "expert" mode (the
pointer to the inner HBaseTestingUtil for server-side code testing) keeps
the interface clean with just a single additional accessor, and allows
developers to adopt the new interface with confidence, knowing that if
there's a gap they can always work around it easily without a JIRA. Marking
the expert mode LP(COPROC, REPLICATION) makes it clear that it _is_ an
expert mode, just for server-side testing.

Or if you prefer, there could be a ServerTestingHBaseCluster interface that
extends TestingHBaseCluster and adds the extra HBTU accessor, with an
LP(COPROC, REPLICATION).

Could you please explain more about what you don't like about exposing HBTU
even for limited cases? There seems to be an assumption that downstream
developers will follow directions for an IA.Private on HBTU by not using
it, but not follow directions for IA.Private annotations on classes
reachable from HBTU?

Thanks,

Geoffrey



On Wed, Jul 21, 2021 at 1:21 AM 张铎(Duo Zhang)  wrote:

> Inline
>
> Geoffrey Jacoby  于2021年7月21日周三 上午4:43写道：
>
> > Thanks for creating the JIRA for adding the Configuration object.
> >
> > To make the discussion more concrete, I did an initial pass through many
> of
> > the minicluster tests at my dayjob, and here are the gaps I've found so
> > far. These don't count many cases where convenience methods on the HBTU
> are
> > used, but where the tests could be easily modified to use existing client
> > APIs instead.
> >
> > * An enormous number of both internal and Phoenix tests execute MapReduce
> > jobs (for example, Phoenix builds its secondary indexes via MapReduce).
> It
> > doesn't appear possible to run MapReduce jobs through the new testing
> > framework. (I filed HBASE-26102 for this.)
> >
> Usually, you do not need to start a mini MR client for running MR job, it
> will execute locally. Most HBase tests in hbase-mapreduce module do not
> start an actual mini MR cluster.
> And the current MiniMRCluster has already been deprecated in hadoop...
>
> >
> > * A test which uses two miniclusters which share the same mini-ZK quorum
> > under different znodes
> >
> What is the goal for these tests? In HBase, we do this when testing
> replication, but I think it is also OK to use different zk clusters? Unless
> there are problems when starting two zk clusters, then we should try to fix
> this problem.
>
> >
> > * An end-to-end test that verifies that a batch job that appends Cell
> Tags
> > to Delete markers it creates via a coproc in fact writes the Tags. This
> > requires getting a reference to the Region from the minicluster and
> > creating a RegionScanner because the HBase client APIs expressly prevent
> > reading Cell Tags. Supporting this would go against the
> > TestingHBaseCluster's design philosophy, so the old minicluster is still
> > necessary for it.
> >
> This is indeed a problem. We should find a way to let users verify the
> tags, as we do not leak it to client side. We could have something with
> IA.LP to expose the Region interface, maybe.
>
> >
> > So far, I think that with HBASE-26098 and HBASE-26102 the overwhelming
> > majority of internal tests would be fine with the new streamlined
> > TestingHBaseCluster. But not all of them, and there are valid reasons a
> > downstream developer might sometimes need the lower-level control the
> HBTU
> > gives.
> >
> > That's why I still believe that the premise that only HBase and Phoenix
> > need to directly test server-side components is incorrect, because
> > _downstream developers can create server-side components_ like coprocs
> and
> > replication endpoints.
> >
> > My suggested solution:
> > 1. Expose the inner HBaseTest

Re: [NOTICE] HBaseTestingUtility is now deprecated, starting from 3.0.0

[Geoffrey Jacoby]

c/main directory. You can
> > > still use it for several years...
> > >
> > > Thanks.
> > >
> > > Andrew Purtell  于2021年7月20日周二 上午3:26写道：
> > >
> > >>> Just leaving a reference to the old, lower-level HBTU as a public
> > >> property of the new
> > >>> interface seems lower-risk to me. What are the gains from hiding the
> > >>> existing HBTU?
> > >>
> > >> This would be similar to the strategy we adopted for the Admin
> > >> interface. Admin there for new users, and as a migration target, but
> > >> HBaseAdmin still available with deprecation annotations.
> > >>
> > >> I guess the question is if the consensus is HBTU was meant to be
> > >> adopted and consumed by downstream projects.
> > >>
> > >> In my opinion, nothing in any project's test/* source directories
> > >> should be considered public, supported, and supportable. Test
> > >> resources within a project exist to test that project, including its
> > >> private internals.
> > >>
> > >> What I would recommend, fwiw is two things:
> > >>
> > >> 1. Explicitly release a supported hbase-testing-util artifact, with
> > >> Public and LimitedPrivate interfaces, with code in src/ not test/.
> > >> 2. Bring back HBTU, but as a compatibility shim. Provide deprecated
> > >> access to HBTU for Phoenix, marked LP(Phoenix), with this deprecated
> > >> accessor to be removed in HBase 4.0, along with the HBTU interface.
> > >>
> > >>> On Mon, Jul 19, 2021 at 12:15 PM Geoffrey Jacoby  >
> > >>> wrote:
> > >>>
> > >>> Can this be a [DISCUSS] rather than a [NOTICE]? The implications for
> > >>> downstream projects (both Phoenix and many internal projects) are
> > large,
> > >>> and it seems like something that needs broader discussion before
> being
> > >> set
> > >>> in stone. The HBaseTestingUtility is used extensively in Phoenix, as
> > well
> > >>> as in many internal projects at my dayjob (some directly and some
> > through
> > >>> Phoenix's BaseTest wrapping of HBTU) -- it's quite useful.
> > >>>
> > >>> The idea of a better-encapsulated, easier-to-use HBase testing
> utility
> > >> is a
> > >>> good one, and the TestingHBaseCluster interface looks like a definite
> > >>> improvement. However, I notice at least one large gap right away:
> there
> > >>> doesn't appear to be a way to inject a custom Configuration object
> into
> > >> the
> > >>> test cluster, which is a very common pattern. (Example: run a test
> > suite
> > >>> twice with a new minicluster each time, once with a flag off and then
> > >> on.)
> > >>> This seems like a simple fix.
> > >>>
> > >>> More concerning is the underlying assumption of the change, that only
> > the
> > >>> HBase project, and perhaps Phoenix, will ever need to write a test of
> > >>> server-side components. That's simply not the case, because HBase has
> > >> many
> > >>> integration points that allow downstream developed code to run in
> > >>> server-side processes.
> > >>>
> > >>> These include:
> > >>> Coprocessor Observers and Endpoints
> > >>> Replication Endpoints
> > >>> MapReduce integration (which acts as a client from HBase's
> perspective
> > >> but
> > >>> runs within YARN services)
> > >>>
> > >>> In addition, Phoenix supports user-defined functions (UDFs) which I
> > >> believe
> > >>> can run server-side within a coproc in certain query plans.
> > >>>
> > >>> The change assumes that no one will ever need direct access to the
> > >> testing
> > >>> utility's internal ZooKeeper, MR, or DFS services, but this seems
> > >> relevant
> > >>> to failure scenario tests of both Replication Endpoints and MapReduce
> > >> jobs.
> > >>> The Admin API may be able to replace quite a lot of existing logic
> > going
> > >>> forward, and many existing tests already use it rather than the test
> > >>> utility directly.  But there are literally thousands of downstream
> > tests
> > >> to
> > >>> a

[jira] [Created] (HBASE-26102) TestingHBaseCluster should support running MapReduce jobs

[Geoffrey Jacoby (Jira)]

Geoffrey Jacoby created HBASE-26102:
---

 Summary: TestingHBaseCluster should support running MapReduce jobs
 Key: HBASE-26102
 URL: https://issues.apache.org/jira/browse/HBASE-26102
 Project: HBase
  Issue Type: Test
  Components: test
Affects Versions: 3.0.0-alpha-1
Reporter: Geoffrey Jacoby


The existing HBaseTestingUtility optionally allows users to start a mini-YARN 
cluster for the purpose of running MapReduce jobs, such as testing applications 
that use HBase's MapReduce integration. 

The new TestingHBaseCluster is a cleaner interface than HBTU, but exposes less 
functionality, and in particular does not seem to allow for starting a YARN 
minicluster. This would make it difficult to use in some common circumstances, 
such as testing Phoenix's index builds, which use MapReduce. 

Many downstream projects that depend on HBase may also have batch logic done as 
MapReduce or other kinds of YARN jobs.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [NOTICE] HBaseTestingUtility is now deprecated, starting from 3.0.0

[Geoffrey Jacoby]

Can this be a [DISCUSS] rather than a [NOTICE]? The implications for
downstream projects (both Phoenix and many internal projects) are large,
and it seems like something that needs broader discussion before being set
in stone. The HBaseTestingUtility is used extensively in Phoenix, as well
as in many internal projects at my dayjob (some directly and some through
Phoenix's BaseTest wrapping of HBTU) -- it's quite useful.

The idea of a better-encapsulated, easier-to-use HBase testing utility is a
good one, and the TestingHBaseCluster interface looks like a definite
improvement. However, I notice at least one large gap right away: there
doesn't appear to be a way to inject a custom Configuration object into the
test cluster, which is a very common pattern. (Example: run a test suite
twice with a new minicluster each time, once with a flag off and then on.)
This seems like a simple fix.

More concerning is the underlying assumption of the change, that only the
HBase project, and perhaps Phoenix, will ever need to write a test of
server-side components. That's simply not the case, because HBase has many
integration points that allow downstream developed code to run in
server-side processes.

These include:
Coprocessor Observers and Endpoints
Replication Endpoints
MapReduce integration (which acts as a client from HBase's perspective but
runs within YARN services)

In addition, Phoenix supports user-defined functions (UDFs) which I believe
can run server-side within a coproc in certain query plans.

The change assumes that no one will ever need direct access to the testing
utility's internal ZooKeeper, MR, or DFS services, but this seems relevant
to failure scenario tests of both Replication Endpoints and MapReduce jobs.
The Admin API may be able to replace quite a lot of existing logic going
forward, and many existing tests already use it rather than the test
utility directly.  But there are literally thousands of downstream tests to
analyze across many different organizations and institutions to verify that
nothing important is being lost, and that will take time. Just leaving a
reference to the old, lower-level HBTU as a public property of the new
interface seems lower-risk to me. What are the gains from hiding the
existing HBTU?

Geoffrey



On Sun, Jul 18, 2021 at 9:44 PM 张铎(Duo Zhang)  wrote:

> Please see the discussion in
> https://issues.apache.org/jira/browse/HBASE-13126
>
> And final work is done in
> https://issues.apache.org/jira/browse/HBASE-26081
> https://github.com/apache/hbase/pull/3478
>
> The original HBaseTestingUtility has been renamed to HBaseTestingUtil, and
> MiniHBaseCluster has been renamed to SingleProcessHBaseCluster. Now they
> are not expected to be used by end users any more. We marked it as
> IA.LimitedPrivate("Phoenix"), as maybe the Phoenix project may still need
> to test something internal to HBase.
>
> Anyway, we encourage every downstream projects(including Phoenix) to try to
> make use of the new TestingHBaseCluster introduced in
> https://issues.apache.org/jira/browse/HBASE-26080
>
> We can keep improving it if the current API set is not enough.
>
>  简略的中文版通知，非直译 
>
> HBaseTestingUtility 已经在 3.0.0 中被标记为 Deprecated，请所有用户尽量尝试使用在 HBASE-26080
> 中引入的 TestingHBaseCluster。有任何需求请随时反馈，我们会持续优化。
>

Re: [DISCUSS] 1.7.0.1/1.7.1 release

[Geoffrey Jacoby]

Thanks for taking this up, Bharath. Personally I'd lean toward doing the
1.7.1 release to get the other fixes released, since I'm not sure how many
1.7 line releases there will be, and some of those fixes are important for
replication correctness.

Geoffrey

On Thu, Jul 1, 2021 at 1:40 PM Bharath Vissapragada 
wrote:

> Hi,
>
> As some of you may know, we have an incompatible serialization backport
> that landed in 1.7.0 (courtesy of yours truly) that is causing upgrade
> issues (thanks to Viraj for noticing it, more details in HBASE-26021
> ). We have to withdraw
> the release so as to not let 1.x users upgrade to it and instead do another
> release in the same line. We can either do a 1.7.0.1 (= 1.7.0 + HBASE-26021
> fix) or do a 1.7.1 which includes all the commits since 1.7.0 which is
> fairly small (listed below).
>
>  delta since 1.7.0 release =
> 7d0a72be14 (origin/branch-1) HBASE-22923 min version of RegionServer to
> move system table regions (#3438)
> 28f36f4619 HBASE-26021: Undo the incompatible serialization change in
> HBASE-7767 (#3435)
> 395eb0c8e0 HBASE-25130 - Fix master in-memory server holding map after:
> (#3402)
> b2f8ec993e HBASE-26025 Add a flag to mark if the IOError can be solved by
> retry in thrift IOError (#3429)
> fd2f8a581f HBASE-26013 Get operations readRows metrics becomes zero after
> HBASE-25677 (#3410)
> 7e57fecda8 HBASE-21674:Port HBASE-21652 (Refactor ThriftServer making
> thrift2 server inherited from thrift1 server) to branch-1 (#2941)
> 5263b8cf40 HBASE-26004: port HBASE-26001 (cell level tags invisible in
> atomic operations when access control is on)to branch-1 (#3387)
> 2e24bad826 HBASE-25984: Avoid premature reuse of sync futures in FSHLog
> (#3371) (#3398)
> a40f4583e3 Set version on branch-1 to 1.7.1-SNAPSHOT
> 0fd6eeb012 HBASE-25910 - Fix port assignment test (#3308)
> 782e24bd9b HBASE-25924 Re-compute size of WAL file while removing from
> WALEntryStream (#3316)
> =
>
> One of these (marked in red above) is a critical fix that was causing us
> issues, so I'd prefer to include it in either of the paths we take. Andrew
> was suggesting 1.7.0.1 in the jira comments (correct me if your definition
> of 1.7.0.1 is different than mine) while I'm leaning towards doing a 1.7.1
> since the delta is fairly small. Thoughts?
>
> I'm happy to be an RM for this release unless there is any objection.
>
> Thanks,
> Bharath
>

Re: JDK version issue during doing 1.7.0 release

[Geoffrey Jacoby]

It appears that both thrift 0.12 (which HBase 1.6 uses) and Thrift 0.13
(which HBase 1.7 is targeted for and that Reid is having trouble building
for JDK 7) have CVEs attached to them, which is why later branches are
using Thrift 0.14.1.

(See CVE-2019-0205 [1] for Thrift 0.12, and CVE-2020-13949 [2] for both
Thrift 0.12 and 0.13)

Given that we need to support JDK 7 due to HBase 1.x compatibility
guidelines, and the 0.14 version of Thrift doesn't support JDK 7 [3], do we
have a way forward? I hope so, but I'm not seeing one offhand.

Geoffrey

[1] https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-0205
[2] https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-13949
[3] https://github.com/apache/thrift/blob/v0.14.0/LANGUAGES.md

On Tue, Apr 13, 2021 at 12:35 AM Reid Chan  wrote:

> Hi team and community:
>
> This is the error message when I tried to make a release 1.7.0:
>
> [INFO] Restricted to JDK 1.7 yet
> org.apache.thrift:libthrift:jar:0.13.0:compile contains
> org/apache/thrift/TNonblockingMultiFetchClient.class targeted to JDK 1.8
> HBase has unsupported dependencies.
>   HBase requires that all dependencies be compiled with version 1.7 or
> earlier
>   of the JDK to properly build from source.  You appear to be using a newer
> dependency. You can use
>   either "mvn -version" or "mvn enforcer:display-info" to verify what
> version is active.
>   Non-release builds can temporarily build with a newer JDK version by
> setting the
>   'compileSource' property (eg. mvn -DcompileSource=1.8 clean package).
> Found Banned Dependency: org.apache.thrift:libthrift:jar:0.13.0
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-enforcer-plugin:3.0.0-M1:enforce
> (enforce-maven-version) on project hbase-thrift: Some Enforcer rules have
> failed. Look above for specific messages explaining why the rule failed. ->
> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions,
> please read the following articles:
> [ERROR] [Help 1]
> http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
> [ERROR]
> [ERROR] After correcting the problems, you can resume the build with the
> command
> [ERROR]   mvn  -rf :hbase-thrift
>
>
> This happened at Thrift module, it seems that thrift-0.13.0 is targeted to
> JDK 1.8, but I need to use JDK 7 to do the release.
>
> Thus I couldn't run the make_rc.sh successfully, any hints or experiences
> about how to resolve this?
>
>
> --
> Best Regards,
> R.C
>

Re: [ANNOUNCE] New HBase PMC Huaxiang Sun

[Geoffrey Jacoby]

Congratulations, Huaxiang!

On Tue, Apr 13, 2021 at 1:20 PM Bharath Vissapragada 
wrote:

> Congrats Huaxiang!
>
> On Tue, Apr 13, 2021 at 1:16 PM Zach York 
> wrote:
>
> > Congrats Huaxiang!
> >
> > On Tue, Apr 13, 2021 at 12:01 PM Stack  wrote:
> >
> > > Hurray Huaxiang!
> > > S
> > >
> > > On Tue, Apr 13, 2021 at 12:39 AM Viraj Jasani 
> > wrote:
> > >
> > > > On behalf of the Apache HBase PMC I am pleased to announce that
> > Huaxiang
> > > > Sun has accepted our invitation to become a PMC member on the HBase
> > > > project. We appreciate Huaxiang stepping up to take more
> responsibility
> > > for
> > > > the project.
> > > >
> > > > Congratulations and welcome, Huaxiang!
> > > >
> > >
> >
>

Re: [ANNOUNCE] New HBase committer Geoffrey Jacoby

[Geoffrey Jacoby]

Thanks everyone!

On Tue, Apr 13, 2021 at 1:16 PM Zach York 
wrote:

> Congrats Geoffrey!
>
> On Tue, Apr 13, 2021 at 9:21 AM Esteban Gutierrez
>  wrote:
>
> > Congrats and welcome, Geoffrey!
> >
> > --
> > Cloudera, Inc.
> >
> >
> >
> > On Tue, Apr 13, 2021 at 8:18 AM zheng wang <18031...@qq.com> wrote:
> >
> > > Congratulations!
> > >
> > >
> > >
> > >
> > > --原始邮件--
> > > 发件人:
> > >   "user"
> > >     <
> > > vjas...@apache.org;
> > > 发送时间:2021年4月9日(星期五) 晚上7:53
> > > 收件人:"hbase-dev" > > u...@hbase.apache.org;
> > >
> > > 主题:[ANNOUNCE] New HBase committer Geoffrey Jacoby
> > >
> > >
> > >
> > > On behalf of the Apache HBase PMC I am pleased to announce that
> Geoffrey
> > > Jacoby has accepted the PMC's invitation to become a committer on the
> > > project.
> > >
> > > Thanks so much for the work you've been contributing. We look forward
> to
> > > your continued involvement.
> > >
> > > Congratulations and welcome, Geoffrey!
> >
>

[jira] [Created] (HBASE-25751) Add writable TimeToPurgeDeletes to ScanOptions

[Geoffrey Jacoby (Jira)]

Geoffrey Jacoby created HBASE-25751:
---

 Summary: Add writable TimeToPurgeDeletes to ScanOptions
 Key: HBASE-25751
 URL: https://issues.apache.org/jira/browse/HBASE-25751
 Project: HBase
  Issue Type: New Feature
Reporter: Geoffrey Jacoby


In HBase 1.x and before, it was possible to override all settings on the 
ScanInfo when overriding the flush and compaction scanner open coprocoessors. 
HBase 2.x changed the coprocessor model to instead allow changes to a limited 
set of options on the ScanOptions object.  

In HBASE-19895 and HBASE-24321, we added support for KeepDeletedCells and 
MinVersions to be overriden by ScanOptions, as needed by Phoenix. 

A 1.x coprocessor used at my day job overrides TimeToPurgeDeletes, and to 
convert it to HBase 2.x that property would need to be overridable from 
ScanOptions as well. This should be a straightforward extension of the previous 
work. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HBASE-25740) Backport HBASE-25629 to branch-1

[Geoffrey Jacoby (Jira)]

Geoffrey Jacoby created HBASE-25740:
---

 Summary: Backport HBASE-25629 to branch-1
 Key: HBASE-25740
 URL: https://issues.apache.org/jira/browse/HBASE-25740
 Project: HBase
  Issue Type: Test
Affects Versions: 1.7.0
Reporter: Geoffrey Jacoby
Assignee: Geoffrey Jacoby


HBASE-25629 recently fixed an issue where TestCurrentHourProvider consistently 
failed on certain OSes due to quirks in time zone implementations. This test is 
also failing in branch-1, so in order to expedite a potential 1.7.0 release we 
should backport to branch-1 as well. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HBASE-25731) TestConnectionImplementation BadHostname tests fail in branch-1

[Geoffrey Jacoby (Jira)]

Geoffrey Jacoby created HBASE-25731:
---

 Summary: TestConnectionImplementation BadHostname tests fail in 
branch-1
 Key: HBASE-25731
 URL: https://issues.apache.org/jira/browse/HBASE-25731
 Project: HBase
  Issue Type: Test
Affects Versions: 1.7.0
Reporter: Geoffrey Jacoby
Assignee: Geoffrey Jacoby


TestConnectionImplementation.testGetAdminBadHostname and 
testGetClientBadHostname are consistently failing in branch-1. 

This is because they're assuming that the validity of the host is checked 
immediately upon getting the protobuf service object, when instead the service 
code purposefully waits until the first service call to check. 

I'll revise the tests to make service calls and verify that they return the 
correct exceptions (ServiceException wrapping an UnknownHostException).



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: EOL branch-1 and all 1.x ?

[Geoffrey Jacoby]

Thanks, Reid, for volunteering to RM 1.7.0.  If you would like some
assistance in stabilizing the tests and getting branch-1 into a
release-ready state, please let me know. I see the flaky list is pretty
long and I'm happy to take some JIRAs.

Geoffrey

On Thu, Apr 1, 2021 at 11:22 AM Andrew Purtell  wrote:

> Reid Chan has committed to completing the 1.7.0 release, please see
> https://issues.apache.org/jira/browse/HBASE-25725 .
>
> So the execution of a public EOL statement for branch-1 can proceed, by the
> apparent consensus surfaced by this thread, but we should respect his wish
> to complete 1.7.0. This makes a nice end to branch-1, IMHO, flushing out
> all of the unreleased changes in that branch.
>
> On Thu, Apr 1, 2021 at 11:11 AM Nick Dimiduk  wrote:
>
> > Also big +1 from me.
> >
> > On Thu, Apr 1, 2021 at 09:08 Huaxiang Sun  wrote:
> >
> > > Per performance regression concern, we had one such issue for meta when
> > > upgrading from 1.2 to 2.3.
> > > It turned out to be default rpc scheduling changed from branch-1 to
> > > branch-2, and it causes performance regression.
> >
> >
> > Huaxiang, do you have a JIRA covering this change? What is your suggested
> > remediation for branch-2 releases? Can/should we make the fix the default
> > behavior, or is there good reason to keep us running in this “slow mode”
> by
> > default, require operators to opt-in to the perf boost?
> >
> > Thanks,
> > Nick
> >
> > On Thu, Apr 1, 2021 at 8:22 AM Pankaj Kumar 
> > > wrote:
> > >
> > > > +1 on EOL branch-1 and all branch-1.x.
> > > >
> > > > Regards,
> > > > Pankaj
> > > >
> > > > On Thu, Apr 1, 2021 at 3:34 AM Andrew Purtell 
> > > wrote:
> > > >
> > > > > Is it time to consider EOL of branch-1 and all 1.x releases ?
> > > > >
> > > > > There doesn't seem to be much developer interest in branch-1 beyond
> > > > > occasional maintenance. This is understandable. Per our
> compatibility
> > > > > guidelines, branch-1 commits must be compatible with Java 7, and
> the
> > > > range
> > > > > of acceptable versions of third party dependencies is also
> restricted
> > > due
> > > > > to Java 7 compatibility requirements. Most developers are writing
> > code
> > > > with
> > > > > Java 8+ idioms these days. For that reason and because the branch-1
> > > code
> > > > > base is generally aged at this point, all but trivial (or lucky!)
> > > > backports
> > > > > require substantial changes in order to integrate adequately. Let
> me
> > > also
> > > > > observe that branch-1 artifacts are not fully compatible with Java
> 11
> > > or
> > > > > later. (The shell is a good example of such issues: The version of
> > > > > jruby-complete required by branch-1 is not compatible with Java 11
> > and
> > > > > upgrading to the version used by branch-2 causes shell commands to
> > > error
> > > > > out due to Ruby language changes.)
> > > > >
> > > > > We can a priori determine there is insufficient motivation for
> > > production
> > > > > of release artifacts for the PMC to vote upon. Otherwise, someone
> > would
> > > > > have done it. We had 12 releases from branch-2 derived code in
> 2019,
> > 13
> > > > > releases from branch-2 derived code in 2020, and so far we have
> had 3
> > > > > releases from branch-2 derived code in 2021. In contrast, we had 8
> > > > releases
> > > > > from branch-1 derived code in 2019, 0 releases from branch-1 in
> 2020,
> > > and
> > > > > so far 0 releases from branch-1 in 2021.
> > > > >
> > > > > *  2021202020191.x0282.x31312*
> > > > >
> > > > > If there is someone interested in continuing branch-1, now is the
> > time
> > > to
> > > > > commit. However let me be clear that simply expressing an abstract
> > > desire
> > > > > to see continued branch-1 releases will not be that useful. It will
> > be
> > > > > noted, but will not have much real world impact. Apache is a
> > do-ocracy.
> > > > In
> > > > > the absence of intrinsic motivation of project participants, which
> is
> > > > what
> > > > > we seem to have here, you will need to do something: Fix the
> > > > compatibility
> > > > > issues, if any between the last release of 1.x and the current
> > branch-1
> > > > > head; fix any failing and flaky unit tests; produce release
> > artifacts;
> > > > and
> > > > > submit those artifacts to the PMC for voting. Or, convince someone
> > with
> > > > > commit rights and/or PMC membership to undertake these actions on
> > your
> > > > > behalf.
> > > > >
> > > > > Otherwise, I respectfully submit for your consideration, it is time
> > to
> > > > > declare  branch-1 and all 1.x code lines EOL, simply acknowledging
> > what
> > > > has
> > > > > effectively already happened.
> > > > >
> > > > > --
> > > > > Best regards,
> > > > > Andrew
> > > > >
> > > > > Words like orphans lost among the crosstalk, meaning torn from
> > truth's
> > > > > decrepit hands
> > > > >- A23, Crosstalk
> > > > >
> > > >
> > >
> >
>
>
> --
> Best regards,
> Andrew
>
> Words like orphans lost among the 

[jira] [Created] (HBASE-25525) WALKey Extended Attributes don't serialize to ReplicationSink

[Geoffrey Jacoby (Jira)]

Geoffrey Jacoby created HBASE-25525:
---

 Summary: WALKey Extended Attributes don't serialize to 
ReplicationSink
 Key: HBASE-25525
 URL: https://issues.apache.org/jira/browse/HBASE-25525
 Project: HBase
  Issue Type: Bug
  Components: Replication
Affects Versions: 1.6.0, 1.5.0
Reporter: Geoffrey Jacoby
Assignee: Geoffrey Jacoby
 Fix For: 1.7.0


HBASE-22622 introduced extended attributes on the WALKey object and protobuf, 
and HBASE-22623 created a coprocessor hook, preWALAppend, so that coprocessors 
can create and insert their own extended attributes. 

These attributes are readable on the source-side, such as in a custom 
ReplicationEndpoint. However, in branch-1 ReplicationProtbufUtil doesn't 
correctly populate the extended attributes on the WALKey protobuf. (In 2.1+, 
HBASE-20625 incidentally fixes this as part of a larger refactoring of 
WALCellCodec logic.)

This means that a custom ReplicationSink can't read extended attributes on a 
WALKey.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] Move the stable pointer to 2.3.x and EOM for branch-2.2?

[Geoffrey Jacoby]

+1 (non-binding)

On Tue, Dec 15, 2020 at 10:23 PM Nick Dimiduk  wrote:

> Good plan, +1 !
>
> On Tue, Dec 15, 2020 at 17:08 Guanghao Zhang  wrote:
>
> > Hi folks!
> >
> > Now we cut new branch-2.4 and the 2.4.0 will be released. We can EOM for
> > branch-2.2 to reduce the maintenance work. Meanwhile, I thought
> branch-2.3
> > is more stable than branch-2.2 because you guys take great test work when
> > release 2.3.0. And for branch-2.2, the 2.2.7 may be the last of the 2.2.x
> > releases.
> >
> > Any thoughts?
> >
>

Coprocs and Off-Heap Writes

[Geoffrey Jacoby]

I'm code-reviewing a Phoenix PR [1] right now, which adds Tags to a
mutation's Cells in a coproc. A question has come up regarding coprocs and
the optional off-heaping of the write path in HBase 2.x and up.

For what parts of the write path (and hence, which coproc hooks) is it safe
to change the underlying Cells of a batch mutation without leaking off-heap
memory?

The HBase book entry on off-heap writes [2] just discusses the ability to
make the MemStore off-heap, but HBASE-15179 and its design doc[3] say that
the entire write stack is off-heap.

Why this matters is if in a RegionObserver coproc hook (that's before the
MemStore commit) the mutation Cells can be assumed to be on-heap, then
clearing the internal family map of the mutation and replacing them with
new, altered Cells is safe. (Extra GC pressure aside, of course.) If not, I
presume the coproc would be leaking off-heap memory (unless there's magic
cleanup somewhere?)

If this is not a safe assumption, what would the recommended way be to
alter a Cell's Tags in a coproc, since Tags are explicitly not exposed to
the HBase client, Cells are immutable, and hence the only way to do so
would be to create new Cells in a coproc? My question's not how to create
the new Cells (that's been answered elsewhere) but how to dispose of the
old, original ones.

Also, if this is not a safe assumption, is there an accepted LP(Coproc) or
Public API that a coproc can check to see if it's in an "off-heap" mode or
not so that a leak can be avoided?

Thanks,

Geoffrey Jacoby

References:
[1] https://github.com/apache/phoenix/pull/978
[2] https://hbase.apache.org/book.html#regionserver.offheap.writepath
[3]
https://docs.google.com/document/d/1fj5P8JeutQ-Uadb29ChDscMuMaJqaMNRI86C4k5S1rQ/edit

Re: Recommended way of using HBase Cell Tags.

[Geoffrey Jacoby]

I completely understand why HBase wouldn't want to expose tags that it uses
for internal security purposes, like ACLs or visibility, to clients.
However, making _all_ tags be off-limits seems to me to limit quite a few
useful features.

Overloading the delete marker's value solves one particular problem, but
not the general case, because it can't be extended to Puts, which already
use their value field for real data. The motivating example in HBASE-25118
is distinguishing a bulk delete from customer operations. But there are
times we may want to distinguish an ETL or bulk write from customer
operations.

Let's say I have a batch job that does an ETL into a cluster at the same
time the cluster is taking other writes. I want to be really sure that all
my data got loaded properly, so I generate a checksum from the ETL dataset
before I load it. After the ETL, I want to generate a checksum for the
loaded data on the cluster and compare. So I need to write a Filter that
distinguishes the loaded data from any other operations going on at the
same time. (Let's assume I'm scanning raw and have major compaction
disabled so nothing gets purged, and there's nothing distinguishing about
the data itself)

The simplest way to do this would be to have a (hopefully tiny) Cell-level
annotation that identifies that it originally came from my ETL. That's
exactly what the Tag array field would provide. Now, I could hack something
into the Put value and change all my applications to ignore part of the
value array, but that assumes that I have full control over the value's
format (not true if I'm using, say, Phoenix). And like using the Delete
value, that's just hacking my own proprietary "Tag" capability into HBase
when a real one already exists.

So I'm curious why, so long as HBase internal tags continue to be
suppressed, is the Tag capability a bad thing to expose?

Geoffrey



On Fri, Oct 16, 2020 at 12:58 PM Andrew Purtell  wrote:

> I responded on the JIRA.
>
> You would be far better served adapting values for your proposal instead of
> tags. Tags are not a client side feature. Tags were and are designed for
> server side use only, and are stripped from client inbound and outbound
> RPCs.
>
> On Wed, Oct 14, 2020 at 9:40 AM Rushabh Shah
>  wrote:
>
> > Thank you Ram for your response !
> >
> > > For your case, is there a possibility to have yournew feature as a
> first
> > class feature using Tags? Just asking?
> >
> > Could you elaborate what you mean by first class feature ?
> >
> >
> > Rushabh Shah
> >
> >- Software Engineering SMTS | Salesforce
> >-
> >   - Mobile: 213 422 9052
> >
> >
> >
> > On Wed, Oct 14, 2020 at 9:35 AM ramkrishna vasudevan <
> > ramkrishna.s.vasude...@gmail.com> wrote:
> >
> > > Hi Rushabh
> > >
> > > If I remember correctly, the decision was not to expose tags for
> clients
> > > directly. All the tags were used as internal to the cell formation at
> the
> > > server side (for eg ACL and Visibility labels).
> > >
> > > For your case, is there a possibility to have yournew feature as a
> first
> > > class feature using Tags? Just asking?
> > >
> > > Regards
> > > Ram
> > >
> > > On Wed, Oct 14, 2020 at 8:17 PM Rushabh Shah
> > >  wrote:
> > >
> > > > Hi Everyone,
> > > > I want to understand how to use the Hbase Cell Tags feature. We have
> a
> > > use
> > > > case to identify the source of deletes (not the same as authenticated
> > > > kerberos user). I have added more details about my use case in
> > > HBASE-25118
> > > > . At my day job
> we
> > > use
> > > > Phoenix to interact with hbase and we are passing this information
> via
> > > > Phoenix ConnectionProperties. We are exploring the Cell Tags feature
> to
> > > add
> > > > this metadata to Hbase Cells (only to Delete Markers as of now).
> > > >
> > > > Via HBASE-18995 ,
> > we
> > > > have moved all the createCell methods which use Tag(s) as an argument
> > to
> > > > PrivateCellUtil class and made the InterfaceAudience of that class
> > > Private.
> > > > I saw some discussion on that jira
> > > > <
> > > >
> > >
> >
> https://issues.apache.org/jira/browse/HBASE-18995?focusedCommentId=16219960=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16219960
> > > > >]
> > > > to expose some methods as LimitedPrivate accessible to CP but was
> > decided
> > > > to do it later. We only expose CellBuilderFactory
> > > > <
> > > >
> > >
> >
> https://github.com/apache/hbase/blob/master/hbase-common/src/main/java/org/apache/hadoop/hbase/CellBuilderFactory.java
> > > > >
> > > > which returns which returns an instance of CellBuilder
> > > > <
> > > >
> > >
> >
> https://github.com/apache/hbase/blob/master/hbase-common/src/main/java/org/apache/hadoop/hbase/CellBuilder.java
> > > > >
> > > > which doesn't have a setTags method. Also the code is vastly
> different
> > in
> > > > branch-1.
> > > >
> > > > Could 

Re: [ANNOUNCE] Please welcome Viraj Jasani to the Apache HBase PMC

[Geoffrey Jacoby]

Congrats, Viraj!

Geoffrey

On Mon, Oct 5, 2020 at 10:03 PM Pankaj kr  wrote:

> Congratulations Viraj...!!
>
> Regards,
> Pankaj
>
> -Original Message-
> From: Andrew Purtell [mailto:apurt...@apache.org]
> Sent: Monday, October 5, 2020 10:28 PM
> To: dev ; Hbase-User 
> Subject: [ANNOUNCE] Please welcome Viraj Jasani to the Apache HBase PMC
>
> On behalf of the Apache HBase PMC I am pleased to announce that Viraj
> Jasani has accepted our invitation to become a PMC member on the HBase
> project. We appreciate Viraj stepping up to take more responsibility for
> the project.
>
> Please join me in welcoming Viraj to the HBase PMC!
>
>
> As a reminder, if anyone would like to nominate another person as a
> committer or PMC member, even if you are not currently a committer or PMC
> member, you can always drop a note to priv...@hbase.apache.org to let us
> know.
>
> --
> Best regards,
> Andrew
>

[jira] [Created] (HBASE-24909) Remove unnecessary ClassNotFoundException and NoSuchMethodException stack traces from HBase startup logs

[Geoffrey Jacoby (Jira)]

Geoffrey Jacoby created HBASE-24909:
---

 Summary: Remove unnecessary ClassNotFoundException and 
NoSuchMethodException stack traces from HBase startup logs
 Key: HBASE-24909
 URL: https://issues.apache.org/jira/browse/HBASE-24909
 Project: HBase
  Issue Type: Improvement
Reporter: Geoffrey Jacoby
Assignee: Geoffrey Jacoby


During cluster startup there are a couple of places where HBase needs to use 
reflection to determine what version of Hadoop or protobuf it's using, and then 
instantiate the right class. 

If its first guess is wrong, it logs a ClassNotFoundException (for protobuf 
MessageLite) or NoSuchMethodException (for 
DFSClient.decryptEncryptedDataEncryptionKey). While this is done at DEBUG 
level, the exception types and the stack traces can lead to operators thinking 
something is wrong. 

Since neither the exception nor the stack trace is actually of interest, we 
should just log a brief, clear message at DEBUG level about which decision was 
made and eat the exception without logging it. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: EOL 1.3.x?

[Geoffrey Jacoby]

+1 (non-binding)

Geoffrey

On Tue, Aug 11, 2020 at 8:19 AM Josh Elser  wrote:

> Sounds good to me.
>
> On 8/11/20 4:01 AM, 张铎(Duo Zhang) wrote:
> > The last release for 1.3.x is 2019.10.20, which means we do not have a
> > release for this release line for about 10 months.
> >
> > Let's make it EOL and tell users to at least upgrade to 1.4.x?
> >
> > Thanks.
> >
>

[jira] [Resolved] (HBASE-24780) Index rebuilds should record scan timestamp to job counter

[Geoffrey Jacoby (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-24780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Geoffrey Jacoby resolved HBASE-24780.
-
Resolution: Invalid

(sigh) Wrong project, apologies for the noise. 

> Index rebuilds should record scan timestamp to job counter
> --
>
> Key: HBASE-24780
> URL: https://issues.apache.org/jira/browse/HBASE-24780
> Project: HBase
>  Issue Type: Improvement
>    Reporter: Geoffrey Jacoby
>        Assignee: Geoffrey Jacoby
>Priority: Major
>
> The index tool output tables (PHOENIX_INDEX_TOOL and 
> PHOENIX_INDEX_TOOL_RESULT) are both keyed on the effective scan time of the 
> index rebuild. This makes the table quite difficult to query if you know the 
> index you're interested in but not the exact millisecond it was last rebuilt. 
> Short term: record this timestamp in the Phoenix index rebuild job counters 
> (this JIRA)
> Longer term: we might want to consider reversing the PK to both cut down on 
> hotspotting and make the table easier to query. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HBASE-24780) Index rebuilds should record scan timestamp to job counter

[Geoffrey Jacoby (Jira)]

Geoffrey Jacoby created HBASE-24780:
---

 Summary: Index rebuilds should record scan timestamp to job counter
 Key: HBASE-24780
 URL: https://issues.apache.org/jira/browse/HBASE-24780
 Project: HBase
  Issue Type: Improvement
Reporter: Geoffrey Jacoby
Assignee: Geoffrey Jacoby


The index tool output tables (PHOENIX_INDEX_TOOL and PHOENIX_INDEX_TOOL_RESULT) 
are both keyed on the effective scan time of the index rebuild. This makes the 
table quite difficult to query if you know the index you're interested in but 
not the exact millisecond it was last rebuilt. 

Short term: record this timestamp in the Phoenix index rebuild job counters 
(this JIRA)
Longer term: we might want to consider reversing the PK to both cut down on 
hotspotting and make the table easier to query. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] Removing problematic terms from our project

[Geoffrey Jacoby]

For most of the proposals (slave -> worker, blacklist -> denylist,
whitelist-> allowlist), I'm +1 (nonbinding). Denylist and acceptlist even
have the advantage of being clearer than the terms they're replacing.

However, I'm not convinced about changing "master" to "coordinator", or
something similar. Unlike "slave", which is negative in any context,
"master" has many definitions, including some common ones which do not
appear problematic. See https://www.merriam-webster.com/dictionary/master for
examples. In particular, the progression of an artisan was from
"apprentice" to "journeyman" to "master". A master smith, carpenter, or
artist would run a shop managing lots of workers and apprentices who would
hope to become masters of their own someday. So "master" and "worker" can
still go together.

Since it's the least problematic term, and by far the hardest term to
change (both within HBase and with effects on downstream projects such as
Ambari), I'm -0 (nonbinding) on changing "master".

Geoffrey

On Mon, Jun 22, 2020 at 1:32 PM Rushabh Shah
 wrote:

> +1 to renaming.
>
>
> Rushabh Shah
>
>- Software Engineering SMTS | Salesforce
>-
>   - Mobile: 213 422 9052
>
>
>
> On Mon, Jun 22, 2020 at 1:18 PM Josh Elser  wrote:
>
> > +1
> >
> > On 6/22/20 4:03 PM, Sean Busbey wrote:
> > > We should change our use of these terms. We can be equally or more
> clear
> > in
> > > what we are trying to convey where they are present.
> > >
> > > That they have been used historically is only useful if the advantage
> we
> > > gain from using them through that shared context outweighs the
> potential
> > > friction they add. They make me personally less enthusiastic about
> > > contributing. That's enough friction for me to advocate removing them.
> > >
> > > AFAICT reworking our replication stuff in terms of "active" and
> "passive"
> > > clusters did not result in a big spike of folks asking new questions
> > about
> > > where authority for state was.
> > >
> > > On Mon, Jun 22, 2020, 13:39 Andrew Purtell 
> wrote:
> > >
> > >> In response to renewed attention at the Foundation toward addressing
> > >> culturally problematic language and terms often used in technical
> > >> documentation and discussion, several projects have begun discussions,
> > or
> > >> made proposals, or started work along these lines.
> > >>
> > >> The HBase PMC began its own discussion on private@ on June 9, 2020
> > with an
> > >> observation of this activity and this suggestion:
> > >>
> > >> There is a renewed push back against classic technology industry terms
> > that
> > >> have negative modern connotations.
> > >>
> > >> In the case of HBase, the following substitutions might be proposed:
> > >>
> > >> - Coordinator instead of master
> > >>
> > >> - Worker instead of slave
> > >>
> > >> Recommendations for these additional substitutions also come up in
> this
> > >> type of discussion:
> > >>
> > >> - Accept list instead of white list
> > >>
> > >> - Deny list instead of black list
> > >>
> > >> Unfortunately we have Master all over our code base, baked into
> various
> > >> APIs and configuration variable names, so for us the necessary changes
> > >> amount to a new major release and deprecation cycle. It could well be
> > worth
> > >> it in the long run. We exist only as long as we draw a willing and
> > >> sufficient contributor community. It also wouldn’t be great to have an
> > >> activist fork appear somewhere, even if unlikely to be successful.
> > >>
> > >> Relevant JIRAs are:
> > >>
> > >> - HBASE-12677  >:
> > >> Update replication docs to clarify terminology
> > >> - HBASE-13852  >:
> > >> Replace master-slave terminology in book, site, and javadoc with a
> > more
> > >> modern vocabulary
> > >> - HBASE-24576  >:
> > >> Changing "whitelist" and "blacklist" in our docs and project
> > >>
> > >> In response to this proposal, a member of the PMC asked if the term
> > >> 'master' used by itself would be fine, because we only have use of
> > 'slave'
> > >> in replication documentation and that is easily addressed. In response
> > to
> > >> this question, others on the PMC suggested that even if only 'master'
> is
> > >> used, in this context it is still a problem.
> > >>
> > >> For folks who are surprised or lacking context on the details of this
> > >> discussion, one PMC member offered a link to this draft RFC as
> > background:
> > >> https://tools.ietf.org/id/draft-knodel-terminology-00.html
> > >>
> > >> There was general support for removing the term "master" / "hmaster"
> > from
> > >> our code base and using the terms "coordinator" or "leader" instead.
> In
> > the
> > >> context of replication, "worker" makes less sense and perhaps
> > "destination"
> > >> or "follower" would be more appropriate terms.
> > >>
> > >> One PMC member's 

Re: [DISCUSS] Change the IA for MutableSizeHistogram and MutableTimeHistogram to LImitedPrivate

[Geoffrey Jacoby]

Couple points:

1. I like Andrew's proposed solution, and we should do it, but I'm not sure
it's sufficient for Rushabh's purposes because of semver rules. Phoenix
supports HBase 1.3 -1.5 (soon to add 1.6) and HBase 2.0 (soon to gain 2.1
and 2.2, with 2.3 coming shortly after its release here.) If we add the new
sizeHistogram and timeHistogram methods to hbase-metrics, they'll be
available in Phoenix only in HBase 1.7 and 2.4. (since 2.3 is mostly-frozen)

 Since Phoenix will be supporting earlier versions of both HBase branches
for a good while, there will need to be a compatibility shim. And the
older-version instance of the shim will probably need to access the classes
directly. (Please correct me if I'm wrong, Rushabh or Andrew.) So it still
might need a LimitedPrivate IA.

2. I agree with Nick that it's better to use LimitedPrivate.COPROC rather
than LimitedPrivate.PHOENIX.

Geoffrey



On Thu, Jun 11, 2020 at 11:28 AM Josh Elser  wrote:

> Sounds reasonable to me!
>
> On 6/11/20 1:06 PM, Andrew Purtell wrote:
> > hbase-metrics-api is available for coprocessors already and interfaces
> > within are already LimitedPrivate(COPROC). However, that package is
> mostly
> > interface and seems geared toward consuming metrics instantiated and
> > registered via private stuff. Or, rather, I didn't see how Phoenix could
> choose
> > which of MutableSizeHistogram and MutableTimeHistogram to instantiate
> using
> > those interfaces, there is only Histogram MetricRegistry#histogram(String
> > name). So I think it is also worth some time to review the utility of
> > hbase-metrics-api and decide if more need be done there. Would the
> addition
> > of
> >
> > Histogram MetricRegistry#sizeHistogram(String name)
> > Histogram MetricRegistry#timeHistogram(String name)
> >
> > achieve the desired objective instead?
> >
> >
> > On Thu, Jun 11, 2020 at 9:16 AM Nick Dimiduk 
> wrote:
> >
> >> I was just about to reply with the same -- Josh is faster :) +1 on
> >> considering the full surface area of the APIs being exposed.
> >>
> >> I also wonder if exposing the metrics infrastructure is something of
> >> interest more broadly than Phoenix. Seems like any coprocessor might
> want
> >> to provide or monitor some metric value.
> >>
> >> On Thu, Jun 11, 2020 at 9:08 AM Josh Elser  wrote:
> >>
> >>> My only concern is that you can't just mark these two classes a
> >>> LimitedPrivate for Phoenix -- you would also have to mark
> >>> MutableRangeHistogram, MutableHistogram (and the rest of the class
> >>> hierarchy) to make sure that we don't make it super confusing as to
> what
> >>> comes from LimitedPrivate classes and what is coming from Private
> >> classes.
> >>>
> >>> Would it be better to just say: make
> >>> ./hbase-hadoop2-compat/src/main/java/org/apache/hadoop/metrics2/lib
> >>> LimitedPrivate?
> >>>
> >>> Do you also need the stuff in
> >>> hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase to push
> >>> metrics back through the HBase metrics subsystem?
> >>>
> >>> Sorry for the late reply. Just want to make sure we open up the
> >>> audience, we open it sufficiently.
> >>>
> >>> On 6/8/20 1:15 PM, Rushabh Shah wrote:
>  Hi,
>  Currently the IA for MutableSizeHistogram and MutableTimeHistogram is
>  private. We want to use these classes in PHOENIX project and I thought
> >> we
>  can leverage the existing implementation from hbase histo
> >> implementation.
>  IIUC the private IA can't be used in other projects. Proposing to make
> >> it
>  LimitedPrivate and mark HBaseInterfaceAudience.PHOENIX. Please
> suggest.
>  Related jira: https://issues.apache.org/jira/browse/HBASE-24520
> 
> >>>
> >>
> >
> >
>

[jira] [Created] (HBASE-24321) Add writable MinVersions and read-only Scan to coproc ScanOptions

[Geoffrey Jacoby (Jira)]

Geoffrey Jacoby created HBASE-24321:
---

 Summary: Add writable MinVersions and read-only Scan to coproc 
ScanOptions
 Key: HBASE-24321
 URL: https://issues.apache.org/jira/browse/HBASE-24321
 Project: HBase
  Issue Type: Improvement
Reporter: Geoffrey Jacoby
Assignee: Geoffrey Jacoby
 Fix For: 2.3.0


Between HBase 1.x and 2.0, the RegionObserver pre*ScannerOpen coprocessors were 
significantly changed so that the coproc implementer no longer has access to 
the actual Scanner, just a ScanOptions object that can be changed in limited 
ways. This is safer and prevents resource leaks and other bugs. 

While ScanOptions provides support for changing TTL, KeepDeletedCells, and 
MaxVersions, a fourth column family config parameter, MinVersions, appears to 
have been missed. This prevents coproc implementers from changing MinVersions 
dynamically. An example of this is PHOENIX-5645, which in the forthcoming 
Phoenix 4.16 (based on HBase 1.x) will allow users to configure a moving window 
where all versions are kept, and thus point-in-time queries are safe. This 
cannot be put in the forthcoming Phoenix 5.1 (based on HBase 2.1 and 2.2) 
because of the coproc changes.

Relatedly, preStoreScannerOpen lacks access to the Scan in HBase 2.0 and up. 
This prevents coprocs from reading the Scan parameters to check if, for 
example, a Scan has set the max time to a point in the past, and thus needs to 
override KeepDeletedCells. This can lead to incorrect behavior when doing 
point-in-time queries or using transactional engines that treat physically 
committed HBase writes as logically uncommitted parts of a transaction. It's 
also a correctness problem for PHOENIX-5645. Please note that only _read-only_ 
access to the Scan from the store scanner coproc hook is in scope for this 
change.  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Reopened] (HBASE-24018) Access check for getTableDescriptors is too restrictive

[Geoffrey Jacoby (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-24018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Geoffrey Jacoby reopened HBASE-24018:
-

> Access check for getTableDescriptors is too restrictive
> ---
>
> Key: HBASE-24018
> URL: https://issues.apache.org/jira/browse/HBASE-24018
> Project: HBase
>  Issue Type: Improvement
>Reporter: Abhishek Singh Chouhan
>Priority: Major
>
> Currently getTableDescriptor requires a user to have Admin or Create 
> permissions. A client might need to get table descriptors to act accordingly 
> eg. based on an attribute set or a CP loaded. It should not be necessary for 
> the client to have create or admin privileges just to read the descriptor, 
> execute and/or read permission should be sufficient? 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [ANNOUNCE] New HBase committer Bharath Vissapragada

[Geoffrey Jacoby]

Congratulations, Bharath!

On Wed, Feb 5, 2020 at 7:41 PM Viraj Jasani 
wrote:

> Congratulations Bharath..!! Well done :)
>
> On Thu, 6 Feb 2020 at 8:36 AM, Nick Dimiduk  wrote:
>
> > On behalf of the Apache HBase PMC I am pleased to announce that Bharath
> > Vissapragada has accepted the PMC's invitation to become a commiter on
> the
> > project. We appreciate all of Bharath's generous contributions thus far
> and
> > look forward to his continued involvement.
> >
> > Allow me to be the first to congratulate and welcome Bharath into his new
> > role!
> >
> > Thanks,
> > Nick
> >
> --
> Thanks,
> Viraj
>

[jira] [Created] (HBASE-23766) Support Point-In-Time Queries

[Geoffrey Jacoby (Jira)]

Geoffrey Jacoby created HBASE-23766:
---

 Summary: Support Point-In-Time Queries
 Key: HBASE-23766
 URL: https://issues.apache.org/jira/browse/HBASE-23766
 Project: HBase
  Issue Type: New Feature
Reporter: Geoffrey Jacoby
Assignee: Geoffrey Jacoby


HBase currently offers a snapshot feature which allows operators to capture the 
state of a table at a point in time in a way that can be cloned or queried in 
the future. It's quite useful in some circumstances, but limited because it's a 
heavyweight operation, and because it requires prior knowledge of the time you 
want to capture. 

Phoenix currently offers a feature called "SCN", which uses the max timestamp 
on Scans to provide the illusion of a "lookback" query at a point in time. It's 
imperfect, however, because of HBase's filtering and cleanup logic for deletes, 
max versions and TTLs can prevent users from seeing certain Cells they would 
have been able to see at a previous point in time. Even PHOENIX-5645, and the 
equivalent HBASE-23602, which try to control major compaction cleanup, don't 
cover all edge cases completely. (For example, you can't see rows whose TTL has 
expired now but hadn't back then. Same with max versions.) 

There are useful non-Phoenix applications as well, such as a change stream that 
shows before/after images, as DynamoDB offers. 

Since full support will require new configuration options added not just to 
major compaction, but also to the read pipeline, I'm filing this as an umbrella 
JIRA so we can have smaller sub-tasks, rather than trying to cram everything 
into HBASE-23602. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HBASE-23744) FastPathBalancedQueueRpcExecutor should enforce queue length of 0

[Geoffrey Jacoby (Jira)]

Geoffrey Jacoby created HBASE-23744:
---

 Summary: FastPathBalancedQueueRpcExecutor should enforce queue 
length of 0
 Key: HBASE-23744
 URL: https://issues.apache.org/jira/browse/HBASE-23744
 Project: HBase
  Issue Type: Bug
Reporter: Geoffrey Jacoby
Assignee: Geoffrey Jacoby


FastPathBalancedQueueRpcExecutor allows RPC requests to skip the RPC queue and 
get worked by an available handler under certain circumstances. 

Relatedly, the hbase.ipc.server.max.callqueue.length parameter can be set to 0, 
including dynamically. When this is the case the executor is supposed to block 
all dispatching. However, the FastPathBalancedQueueRpcExecutor will still 
dispatch the request if one of the "fast path" handlers is available on its 
stack. This both isn't the desired effect, and also makes 
TestSimpleRpcExecutor.testSoftAndHardQueueLimits unstable when it checks the 
queue length 0 behavior. 

A simple fix is just to check max queue length > 0 before 
FastPathBalancedQueueRpcExecutor pops the fast handler off the stack. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (HBASE-22908) Link to HBase 1.x Documentation on HBase Site

[Geoffrey Jacoby (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-22908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Geoffrey Jacoby resolved HBASE-22908.
-
Resolution: Fixed

I don't know which kind soul added the 1.4 documentation to the HBase website, 
but it's there now, so resolving this JIRA. 

> Link to HBase 1.x Documentation on HBase Site
> -
>
> Key: HBASE-22908
> URL: https://issues.apache.org/jira/browse/HBASE-22908
> Project: HBase
>  Issue Type: Improvement
>  Components: documentation
>        Reporter: Geoffrey Jacoby
>Priority: Minor
>
> Some other communities in the Hadoop stack, such as Hadoop itself and 
> ZooKeeper, maintain links to documentation for all of their active branches. 
> (ZooKeeper additionally has an archive to documentation on all their previous 
> releases)
> Since I believe the stable pointer is still on HBase 1.x, it's odd that the 
> HBase site does not actually provide a link to documentation for it, only 
> newer releases 2.0 and up. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HBASE-23711) Add test for MinVersions and KeepDeletedCells TTL

[Geoffrey Jacoby (Jira)]

Geoffrey Jacoby created HBASE-23711:
---

 Summary: Add test for MinVersions and KeepDeletedCells TTL
 Key: HBASE-23711
 URL: https://issues.apache.org/jira/browse/HBASE-23711
 Project: HBase
  Issue Type: Test
Reporter: Geoffrey Jacoby
Assignee: Geoffrey Jacoby


Recently I was researching how HBase handles the interactions between setting 
MinVersions and KeepDeletedCells = TTL, and I wrote a test to prove my 
assumptions about the behavior were correct. There doesn't seem to be an 
equivalent existing test in TestMinVersions, so I thought I'd contribute it. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HBASE-23710) Priority configuration for system coprocessors

[Geoffrey Jacoby (Jira)]

Geoffrey Jacoby created HBASE-23710:
---

 Summary: Priority configuration for system coprocessors
 Key: HBASE-23710
 URL: https://issues.apache.org/jira/browse/HBASE-23710
 Project: HBase
  Issue Type: New Feature
  Components: Coprocessors
Reporter: Geoffrey Jacoby
Assignee: Geoffrey Jacoby


Currenty HBase allows operators to set system region coprocessors via 
hbase-site.xml to be loaded on each table in a cluster (or alternately, all 
tables but system tables). HBase assumes that the first loaded system 
coprocessor gets the first, or SYSTEM priority, with each subsequent system 
coproc getting incremented by 1. As a reminder, in HBase _lower_ priorities go 
first. 

It can be useful for an operator to be able to define a coprocessor on each 
table that needs a different priority. For example, an operator might want a 
coproc to load on each table _last_, so that it can enforce some system 
invariant and know that no other coproc will interfere with it. 

I propose adding optional priority config to the hbase-site.xml configuration, 
separated from each coproc class in the comma-separated list by a special 
character (perhaps a colon) that's not used in class names. 

The region coprocessor host will parse the priority if present and use it when 
instantiating the coproc. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HBASE-23602) TTL Before Which No Data is Purged

[Geoffrey Jacoby (Jira)]

Geoffrey Jacoby created HBASE-23602:
---

 Summary: TTL Before Which No Data is Purged
 Key: HBASE-23602
 URL: https://issues.apache.org/jira/browse/HBASE-23602
 Project: HBase
  Issue Type: New Feature
Reporter: Geoffrey Jacoby


HBase currently offers operators a choice. They can set KEEP_DELETED_CELLS=true 
and VERSIONS to max value, plus no TTL, and they will always have a complete 
history of all changes (but high storage costs and penalties to read 
performance). Or they can have KEEP_DELETED_CELLS=false and VERSIONS/TTL set to 
some reasonable values, but that means that major compactions can destroy the 
ability to do a consistent snapshot read of any prior time. (This limits the 
usefulness and correctness of, for example, Phoenix's SCN lookback feature.) 

I propose having a new TTL property to give a minimum age that an expired or 
deleted Cell would have to achieve before it could be purged. (I see that 
HBASE-10118 already does something similar for the delete markers themselves.) 

This would allow operators to have a consistent history for some finite amount 
of recent time while still purging out the "long tail" of obsolete / deleted 
versions. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HBASE-23316) RegionServers should refuse to load Regions with malformed coprocs, but not crash

[Geoffrey Jacoby (Jira)]

Geoffrey Jacoby created HBASE-23316:
---

 Summary: RegionServers should refuse to load Regions with 
malformed coprocs, but not crash
 Key: HBASE-23316
 URL: https://issues.apache.org/jira/browse/HBASE-23316
 Project: HBase
  Issue Type: Improvement
Reporter: Geoffrey Jacoby
Assignee: Geoffrey Jacoby


Currently, a region server will crash if it tries to load a region with a 
coprocessor that is malformed (such as not being on the RS's classpath.) This 
can lead to a cascading "poison pill" in which  the HMaster keeps reassigning 
the region to different region servers, bringing down server after server and 
endangering the whole cluster.

We definitely can't load the Region if the coproc is wrong, but neither should 
that harm other, correctly configured regions on the same server. 

In this JIRA, I'll change the behavior to fail to load the region, and 
increment a metric for region load failures. Future JIRAs can build on this, 
such as by having the HMaster stop trying to load a malformed region absent 
user intervention after some number of retries. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] Deprecate Review Board in favor of Github reviews

[Geoffrey Jacoby]

+1 (non-binding), GitHub is a much better user experience, for both
reviewers and contributors

Geoffrey.

On Thu, Nov 14, 2019 at 5:48 AM Guangxu Cheng 
wrote:

> +1
> It will be more convenient to use Github PR.
>
> 张铎(Duo Zhang)  于2019年11月14日周四 下午6:17写道：
>
> > +1,  ReviewBoard is almost dead now as it is only available to
> > committers...
> >
> > Jan Hentschel  于2019年11月14日周四 下午6:11写道：
> >
> > > +1
> > >
> > > I also like the GitHub way much more compared to ReviewBoard.
> > >
> > > From: Peter Somogyi 
> > > Reply-To: "dev@hbase.apache.org" 
> > > Date: Wednesday, November 13, 2019 at 6:23 PM
> > > To: HBase Dev List 
> > > Subject: Re: [DISCUSS] Deprecate Review Board in favor of Github
> reviews
> > >
> > > +1
> > >
> > > Another issue with ReviewBoard is that it requires Apache ID so only
> > > committers are able to create new reviews or even comment.
> > >
> > > On Wed, Nov 13, 2019 at 5:21 PM Nick Dimiduk  >  > > ndimi...@apache.org>> wrote:
> > >
> > > Heya,
> > >
> > > Seems in the old days we were explicitly non-strict about where code
> > review
> > > were happening. I remember bouncing between Review Board and a
> > Phabricator
> > > instance (in addition to in-line patch reviews on JIRA). Now that we
> have
> > > this fancy Gitbox and integration with GitHub, it seems we're making a
> > > strong statement toward using Github PRs (in addition to in-line patch
> > > reviews on JIRA) for our code review system. Is it worth while
> supporting
> > > those older tools? I think maintaining the developer support tooling
> > around
> > > just these two mechanisms is plenty to keep up with.
> > >
> > > I propose we make the move to Github PR's "official". This
> > > basically involves updating the tome (here [0], here [1], probably
> > others)
> > > accordingly and sweeping the `dev-support` dir for old scripts.
> > >
> > > Thoughts?
> > >
> > > Thanks,
> > > Nick
> > >
> > > [0]: http://hbase.apache.org/book.html#developing
> > > [1]: http://hbase.apache.org/book.html#reviewboard
> > >
> > >
> > >
> >
>

[jira] [Created] (HBASE-23288) Backport HBASE-23251 (Add Column Family and Table Names to HFileContext) to branch-1

[Geoffrey Jacoby (Jira)]

Geoffrey Jacoby created HBASE-23288:
---

 Summary: Backport HBASE-23251 (Add Column Family and Table Names 
to HFileContext) to branch-1
 Key: HBASE-23288
 URL: https://issues.apache.org/jira/browse/HBASE-23288
 Project: HBase
  Issue Type: Sub-task
Reporter: Geoffrey Jacoby
Assignee: Geoffrey Jacoby






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HBASE-23251) Add Region and Table Names to HFileContext and use in HFileWriterImpl logging

[Geoffrey Jacoby (Jira)]

Geoffrey Jacoby created HBASE-23251:
---

 Summary: Add Region and Table Names to HFileContext and use in 
HFileWriterImpl logging
 Key: HBASE-23251
 URL: https://issues.apache.org/jira/browse/HBASE-23251
 Project: HBase
  Issue Type: Improvement
Reporter: Geoffrey Jacoby
Assignee: Geoffrey Jacoby


When something goes wrong in the Store / HFile write path, it would be very 
useful to know which region and table the error is coming from. Currently the 
HFileWriterImpl gets an HFileContext object with some useful state information, 
but the region and table aren't among them. 

For example, this would be very helpful diagnosing HBASE-23143 and similar 
issues. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (HBASE-22934) IndexedKeyValue creation fails after HBASE-22034

[Geoffrey Jacoby (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-22934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Geoffrey Jacoby resolved HBASE-22934.
-
Resolution: Invalid

> IndexedKeyValue creation fails after HBASE-22034
> 
>
> Key: HBASE-22934
> URL: https://issues.apache.org/jira/browse/HBASE-22934
> Project: HBase
>  Issue Type: Improvement
>    Reporter: Geoffrey Jacoby
>        Assignee: Geoffrey Jacoby
>Priority: Major
>
> HBASE-22034 backported to branch-1 HBASE-21401 (adding several validation 
> checks to KeyValue creation) and HBASE-22032 (adding a null check for the row 
> key to KeyValue creation). It will first be released in HBase 1.5.
> In Phoenix 4.14.2 we included PHOENIX-5188, which adds logic to initialize 
> IndexedKeyValues with the appropriate row key and offsets, so that it can 
> pass the new check in HBASE-22032. However, it did not correctly handle all 
> the checks in HBASE-21401. 
> When testing 4.14.3 of Phoenix with an HBase version containing HBASE-22034, 
> we see lots of errors like the following:
> java.lang.IllegalArgumentException: Overflow when reading key length at 
> position=0, KeyValueBytesHex=foo, offset=0, length=3
> This will affect the future 4.15-HBase-1.5, and as well as any future release 
> of a 4.14.3 based on HBase-1.5



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Created] (HBASE-22934) IndexedKeyValue creation fails after HBASE-22034

[Geoffrey Jacoby (Jira)]

Geoffrey Jacoby created HBASE-22934:
---

 Summary: IndexedKeyValue creation fails after HBASE-22034
 Key: HBASE-22934
 URL: https://issues.apache.org/jira/browse/HBASE-22934
 Project: HBase
  Issue Type: Improvement
Reporter: Geoffrey Jacoby
Assignee: Geoffrey Jacoby


HBASE-22034 backported to branch-1 HBASE-21401 (adding several validation 
checks to KeyValue creation) and HBASE-22032 (adding a null check for the row 
key to KeyValue creation). It will first be released in HBase 1.5.

In Phoenix 4.14.2 we included PHOENIX-5188, which adds logic to initialize 
IndexedKeyValues with the appropriate row key and offsets, so that it can pass 
the new check in HBASE-22032. However, it did not correctly handle all the 
checks in HBASE-21401. 

When testing 4.14.3 of Phoenix with an HBase version containing HBASE-22034, we 
see lots of errors like the following:

java.lang.IllegalArgumentException: Overflow when reading key length at 
position=0, KeyValueBytesHex=foo, offset=0, length=3

This will affect the future 4.15-HBase-1.5, and as well as any future release 
of a 4.14.3 based on HBase-1.5



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Created] (HBASE-22908) Link to HBase 1.x HBase Documentation on HBase Site

[Geoffrey Jacoby (Jira)]

Geoffrey Jacoby created HBASE-22908:
---

 Summary: Link to HBase 1.x HBase Documentation on HBase Site
 Key: HBASE-22908
 URL: https://issues.apache.org/jira/browse/HBASE-22908
 Project: HBase
  Issue Type: Improvement
  Components: documentation
Reporter: Geoffrey Jacoby


Some other communities in the Hadoop stack, such as Hadoop itself and 
ZooKeeper, maintain links to documentation for all of their active branches. 
(ZooKeeper additionally has an archive to documentation on all their previous 
releases)

Since I believe the stable pointer is still on HBase 1.x, it's odd that the 
HBase site does not actually provide a link to documentation for it, only newer 
releases 2.0 and up. 



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

Re: Coprocessors, clean ups, compatibility, deprecations, Phoenix... it's a bit of a mess

[Geoffrey Jacoby]

For coprocessors in general, I think Andrew sums the issues up well. I just
want to add that it's really important to have a mental model of who the
audience for a coprocessor is when designing. I remember when coprocessors
were first introduced, I read articles comparing them to triggers and
stored procedures, and the docs still describe them that way in section
112.1.  But they're really not like triggers and SPs, because a
user-written stored procedure shouldn't be able to crash your database. :-)
Andrew's comparison to kernel extensions is apt I think.

Writing a Linux kernel extension doesn't require you to be Linus or one of
his lieutenants, but it does require more sophistication that just writing
a user-space Linux program does. Their entire purpose is for people who
aren't at the level of Linus to affect the kernel behavior in limited ways.
Likewise, with HBase coprocessors. They should provide safe enough
abstractions that they don't require an HBase committer to implement
safely, but I think it's fine to require more HBase knowledge than what's
required to use the client API.

You can't say "a mutable WALEdit is too dangerous" without making
assumptions about the developer it's too dangerous _for_. Ideally these
assumptions should be explicit and agreed upon by the community. I think
differing audience assumptions are where at least some of the disagreements
are coming from.

Geoffrey


On Fri, Aug 9, 2019 at 8:55 AM Andrew Purtell 
wrote:

> The future of the coprocessor API is an interesting topic. I think we have
> a range of opinions in the community and I would like to hear more of them.
>
> Because of the compatibility headaches sometimes I’d like to rip them out.
> Sometimes they are essential for accomplishing something in Phoenix or our
> in house backup solution, for example. For me my opinion is very mixed.
>
> When first conceived the first use case was security and we took
> inspiration from OS kernel examples in Linux and TrustedBSD (now merged
> with FreeBSD) where upcall interfaces were made available where
> authoritative access control decisions are made. The set of hooks has
> expanded over time as users have requested extensions. This style of
> interface is powerful in that it enables a mixin approach to composition of
> functionality and extension. However in retrospect the maintenance burdens
> were not fully appreciated, at least by me. I had assumed we would be
> allowed more freedom to change but my experiences with Phoenix educated me
> on the kind of downstream headaches that result when we make those changes.
>
> If I were to do it again I would attempt an abstract and fluent interface
> where extensions would register intents and receive callbacks in a much
> more granular way, like:
>
> onRegion().onRPC().onGet().then(...)
>
> I suppose this looks kind of like Mockito. There would be no overlarge
> interfaces full of upcall methods that break source compatibility on every
> change (in branch-1). Although we could not avoid the complexity of
> ensuring the right callbacks are invoked at the right places on the right
> code paths the extension interface and its types would be decoupled from
> internals and the kind of compatibility headaches we impose on
> downstreamers (and ourselves) would mostly disappear. This has been
> proposed on an old JIRA somewhere...
>
> Of course a redo like this would be a very complex and time consuming
> project, and a port of say something like Phoenix would be a reboot of
> multi man year efforts, and as far as I know nobody working in the code
> today has that kind of sustained available time and attention. It’s too
> late. We have to make the best of the legacy of past engineering choices.
> If that is not correct then it would be a very pleasant surprise indeed.
>
> Given that we have the current model and we have downstreamers like
> Phoenix depending on them I think we are limited in the kind of clean ups
> we might like to do and need to be tolerant of the requirement to maintain
> these interfaces and their functionally across major versions. Deprecation
> is fine but if it is done without the input of the known consumer are we
> really being fair? Unclear.
>
>
> > On Aug 8, 2019, at 6:39 PM, 张铎(Duo Zhang)  wrote:
> >
> > When releasing 2.0.0 we faced a lot of problems because we exposes so
> many
> > internal classes to CPs. It is really hard to both consider the
> > compatibility and development on HBase. And then we have done lots of
> works
> > to abstract interfaces for CPs to use and hide the actual implementation
> > classes to be HBase only.
> >
> > The work is not fully done, so we still left some methods which exposes
> > internal classes there with a deprecated annotation, I think most of them
> > are for Phoenix. This is a trade off and I think it is also acceptable.
> > Phoenix could still use the deprecated methods, and we will not remove
> them
> > unless we find an alternate solution. And I think if anyone 

Re: Coprocessors, clean ups, compatibility, deprecations, Phoenix... it's a bit of a mess

[Geoffrey Jacoby]

There are a bunch of issues raised here, some micro and some macro. I'll
tackle them in two separate messages so they're short(ish) enough for
people to bother reading. :-) For this one I wear my "HBase contributor"
hat.

First the micro, which is my patch on HBASE-22623 which sparked a lot of
passionate disagreement. I want to start by saying that I really appreciate
the work that Duo and many others have done to improve the abstraction and
cleanliness of the code base -- implementing a change to the write path in
both branch-1 and master, you really feel the improvement in the later
code.

In a few places though, that cleanup left contradictions. One of them is
WALEdit, which is LimitedPrivate for coprocessors and replication, and is
already exposed in about 10 non-deprecated cooprocessor hooks between at
least two interfaces. But the comments say it should never be exposed to
coprocs, (I _think_ it means "never before WAL append", but that's not what
it says) and the add methods are marked IA.Private.

I think the fundamental mistake here is to have comments which contradict
the IA annotation, and to enforce the comments over the IA contract. If a
class is LimitedPrivate for coprocs and replication, then it's part of the
public API for those components, and should be safe to consume. If the
community later decides it's not safe, the IA can be changed and _all_
coproc + replication methods using WALEdit can be deprecated in favor of
some new safer interface. As I believe was once done for HLogKey vs WALKey.
But that hasn't (yet anyway) been done for WALEdit, and patches which honor
the IA contract, as I tried to do, shouldn't be rejected because they
violate a private understanding.

I'm (non-bindingly) strongly against any proposal to create an API which is
deprecated on the moment of its birth, as has been proposed above. That
seems nonsensical to me, since a deprecation means "don't use this", so
what's the point? Slipping my Phoenix hat on, I don't want Phoenix to be a
trespasser tolerated on sufferance, but based on good, clear abstractions
and APIs negotiated between the two communities, with those APIs open to
other projects as well.

But more on that in the next one, which I'll write in the (Pacific time)
morning.

Geoffrey

On Thu, Aug 8, 2019 at 6:39 PM 张铎(Duo Zhang)  wrote:

> When releasing 2.0.0 we faced a lot of problems because we exposes so many
> internal classes to CPs. It is really hard to both consider the
> compatibility and development on HBase. And then we have done lots of works
> to abstract interfaces for CPs to use and hide the actual implementation
> classes to be HBase only.
>
> The work is not fully done, so we still left some methods which exposes
> internal classes there with a deprecated annotation, I think most of them
> are for Phoenix. This is a trade off and I think it is also acceptable.
> Phoenix could still use the deprecated methods, and we will not remove them
> unless we find an alternate solution. And I think if anyone wants to remove
> them without replacment you will first jump out and give a -1 :)
>
> Specific to HBASE-22623, I still think we should add the deprecated
> annotation to keep the API consistent, otherwise users will be confused
> that whether they can use the WALEdit. And on the abstraction of WALEdit, I
> think we used to rely on a high level abstraction in HBASE-20952. But now
> since there is little progress there, I think we can start the work of
> abstracting WALEdit only.
>
> Thanks.
>
> Andrew Purtell  于2019年8月9日周五 上午3:21写道：
>
> > Please let me direct your attention to the tail of HBASE-22623 for a
> larger
> > discussion.  I tried to sum it up as follows:
> >
> > An opinion that we should have more and more coprocessor interfaces to
> > address new use cases is valid. An opinion that coprocessors are too
> > invasive and should be 'cleaned up' is also valid. An opinion that the
> > compatibility headaches of coprocessor interfaces are annoying is valid.
> An
> > opinion that Phoenix can be considered as a valid use case when
> considering
> > interface changes is valid. An opinion that only HBase level concerns
> > should motivate API changes is valid. These opinions are strawmen. I
> think
> > they approach actual positions in the community but I do not imply any
> > specific person has one of them. These strawmen are at least partially
> > contradictory. It is going to be an ongoing process to sort them out into
> > something that makes sense and can get consensus.
> >
> > So while as committer I am moving forward on HBASE-22623 because I don't
> > see a veto but instead a disagreement on the margins (deprecation or not)
> > motivated on larger principles, I also want to raise the visibility of
> the
> > disagreement because I think it impacts our relationship with another
> > project at Apache at a minimum, but also future technical directions of
> an
> > important subset of interfaces.
> >
> > For your consideration.
> >
> > --
> > 

Re: The note of the round table meeting after HBaseConAsia 2019

[Geoffrey Jacoby]

Just want to chime in with my Phoenix PMC hat on and say that there are no
current plans endorsed by the PMC to "split" Phoenix away from HBase. I'm
not even aware of any JIRAs proposing such a thing, though if the anonymous
Phoenix committer at HBaseCon Asia wants to make one, he or she is of
course welcome to do so. Since Phoenix is implemented as a series of HBase
coprocessor hooks, I'm not even sure what an HBase-less Phoenix would _be_.

As always, we welcome any and all suggestions on our dev list or JIRA on
how we can integrate with HBase better, improve stability, or build cool
new features together.

Thanks,

Geoffrey Jacoby




On Thu, Aug 8, 2019 at 11:25 AM Andrew Purtell  wrote:

> This is great, but in the future please refrain from borderline marketing
> of a commercial product on these lists. This is not the appropriate venue
> for that.
>
> It is especially poor form to dump on a fellow open source project, as you
> claim to be. This I think is the tell behind the commercial motivation.
>
> Also I should point out, being pretty familiar with Phoenix in operation
> where I work, and in my interactions with various Phoenix committers and
> PMC, that the particular group of HBasers in that group appeared to share a
> negative view - which I will not comment on, they are entitled to their
> opinions, and more choice in SQL access to HBase is good! - that should not
> be claimed to be universal or even representative.
>
>
>
> On Thu, Aug 8, 2019 at 9:42 AM Rohit Jain  wrote:
>
> > Hi folks,
> >
> > This is a nice write-up of the round-table meeting at HBaseConAsia.  I
> > would like to address the points I have pulled out from write-up (at the
> > bottom of this message).
> >
> > Many in the HBase community may not be aware that besides Apache Phoenix,
> > there has been a project called Apache Trafodion, contributed by
> > Hewlett-Packard in 2015 that has now been top-level project for a while.
> > Apache Trafodion is essentially technology from Tandem-Compaq-HP that
> > started its OLTP / Operational journey as NonStop SQL effectively in the
> > early 1990s.  Granted it is a C++ project, but it has 170+ patents as
> part
> > of it that were contributed to Apache.  These are capabilities that still
> > don’t exist in other databases.
> >
> > It is a full-fledged SQL relational database engine with the breadth of
> > ANSI SQL support, including OLAP functions mentioned, and including many
> de
> > facto standard functions from databases like Oracle.  You can go to the
> > Apache Trafodion wiki to see the documentation as to what all is
> supported
> > by Trafodion.
> >
> > When we introduced Apache Trafodion, we implemented a completely
> > distributed transaction management capability right into the HBase engine
> > using coprocessors, that is completely scalable with no bottlenecks
> > what-so-ever.  We have made this infrastructure very efficient over time,
> > e.g. reducing two-phase commit overhead for single region transactions.
> We
> > have presented this at HBaseCon.
> >
> > The engine also supports secondary indexes.  However, because of our
> > Multi-dimensional Access Method patented technology the need to use a
> > secondary index is substantially reduced.  All DDL and index updates are
> > completely protected by ACID transactions.
> >
> > Probably because of our own inability to create excitement about the
> > project, and potentially other reasons, we could not get community
> > involvement as we were expecting.  That is why you may see that while we
> > are maintaining the code base and introducing enhancements to it, much of
> > our focus has shifted to the commercial product based on Apache
> Trafodion,
> > namely EsgynDB.  But if the community involvement increases, we can
> > certainly refresh Trafodion with some of the additional functionality we
> > have added on the HBase side of the product.
> >
> > But let me be clear.  We are about 150 employees at Esgyn with 40 or so
> in
> > the US, mostly in Milpitas, and the rest in Shanghai, Beijing, and
> > Guiyang.  We cannot sustain the company on service revenue alone.  You
> have
> > seen companies that tried to do that have not been successful, unless
> they
> > have a way to leverage the open source project for a different business
> > model – enhanced capabilities, Cloud services, etc.
> >
> > To that end we have added to EsgynDB complete Disaster Recovery,
> > Point-in-Time, fuzzy Backup and Restore, Manageability via a Database
> > Manager, Multi-tenancy, and a large number of other capabilities for High
> > Availa

Re: Time for another round of branch-1 releases

[Geoffrey Jacoby]

Andrew,

What's the planned release cycle for branch-1? As you know, I'm working on
a pair of JIRAs (HBASE-22622 and 22623, which may or may not require
HBASE-18127) that need to go out in a minor release due to semver (added
methods to coprocessor interfaces), and they're definitely not all getting
committed by Friday. I'd been assuming that 1.5 was my branch-1 release
train.

HBase 1.2 was released Feb 2016; HBase 1.3 in Jan 2017; 1.4 in Dec 2017.
That shows that 1.5 is long overdue, but it also means that if 1.6 follows
suit, it won't come out until mid-to-late-2020, and it'll be a year or more
before I can make use of them. I can't build a Phoenix feature that relies
on an unreleased HBase one.

Are we moving towards more frequent minors on branch-1, as I know has been
discussed on this list before? If so, no problem, just let me know what
train I need to aim for. If not, it'd be great if this train could stay in
the station a bit longer.

Thanks,

Geoffrey


On Wed, Jul 10, 2019 at 6:35 PM Andrew Purtell  wrote:

> Now that HBASE-22627 (Port HBASE-22617 (Recovered WAL directories not
> getting cleaned up) to branch-1) has been committed it's time to roll a new
> set of 1.x releases for all releasing branches that needed this change.
>
> At a minimum this would be a 1.3.6 and 1.4.11.
>
> I would also like to try again to release 1.5.0.
>
> I'm back from vacation Friday. Planning on making the RCs and submitting
> them for consideration next week. If you have any pending work for branch-1
> please try to get it in before then.
>
> --
> Best regards,
> Andrew
>
> Words like orphans lost among the crosstalk, meaning torn from truth's
> decrepit hands
>- A23, Crosstalk
>

[jira] [Created] (HBASE-22623) Add coprocessor hooks for preWALAppend and postWALAppend

[Geoffrey Jacoby (JIRA)]

Geoffrey Jacoby created HBASE-22623:
---

 Summary: Add coprocessor hooks for preWALAppend and postWALAppend
 Key: HBASE-22623
 URL: https://issues.apache.org/jira/browse/HBASE-22623
 Project: HBase
  Issue Type: New Feature
Reporter: Geoffrey Jacoby
Assignee: Geoffrey Jacoby


While many coprocessor hooks expose the WALEdit to implementing coprocs, there 
aren't any that expose the WALKey before it's created and added to the 
WALEntry. 

It's sometimes useful for coprocessors to be able to edit the WALKey, for 
example to add extended attributes using the fields to be added in HBASE-22622. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HBASE-22622) WALKey Extended Attributes

[Geoffrey Jacoby (JIRA)]

Geoffrey Jacoby created HBASE-22622:
---

 Summary: WALKey Extended Attributes
 Key: HBASE-22622
 URL: https://issues.apache.org/jira/browse/HBASE-22622
 Project: HBase
  Issue Type: New Feature
  Components: wal
Reporter: Geoffrey Jacoby
Assignee: Geoffrey Jacoby


It would be useful if the WAL protobuf and WALKey class included an optional 
map of extended key/value attributes. While standard HBase replication would 
not use them, custom replication endpoints could use the data to make filtering 
decisions or take actions.

An example use case would be allowing a tool like Phoenix to annotate 
WAL.Entries to indicate that a given Entry is associated with a particular 
Phoenix view rather than the base table. A custom replication endpoint might 
choose to replicate some views but not others. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HBASE-22034) Backport HBASE-21404 and HBASE-22032 to branch-1

[Geoffrey Jacoby (JIRA)]

Geoffrey Jacoby created HBASE-22034:
---

 Summary: Backport HBASE-21404 and HBASE-22032 to branch-1
 Key: HBASE-22034
 URL: https://issues.apache.org/jira/browse/HBASE-22034
 Project: HBase
  Issue Type: Improvement
Reporter: Geoffrey Jacoby
Assignee: Geoffrey Jacoby


Branch-2 and master have good validation checks when constructing KeyValues. We 
should also have them on branch-1. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HBASE-22032) KeyValue validation should check for null byte array

[Geoffrey Jacoby (JIRA)]

Geoffrey Jacoby created HBASE-22032:
---

 Summary: KeyValue validation should check for null byte array
 Key: HBASE-22032
 URL: https://issues.apache.org/jira/browse/HBASE-22032
 Project: HBase
  Issue Type: Improvement
Affects Versions: 2.1.3, 2.0.4, 3.0.0
Reporter: Geoffrey Jacoby
Assignee: Geoffrey Jacoby


HBASE-21401 added some nice validation checks to throw precise errors if a 
KeyValue is constructed using invalid parameters. However it implicitly assumes 
that the KeyValue buffer is not null. It should validate this assumption and 
alert accordingly rather than throwing an NPE from an unrelated check. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [ANNOUNCE] New HBase committer Xu Cang

[Geoffrey Jacoby]

Congratulations, Xu!

On Tue, Feb 5, 2019 at 2:49 PM Andrew Purtell  wrote:

> On behalf of the Apache HBase PMC, I am pleased to announce that Xu
> Cang has accepted the PMC's invitation to become a committer on the
> project. We appreciate all of Xu's generous contributions thus far and
> look forward to his continued involvement.
>
> Congratulations and welcome, Xu!
>
> --
> Best regards,
> Andrew
>

[jira] [Created] (HBASE-21530) Abort_Procedure should be able to take a list of proc IDs

[Geoffrey Jacoby (JIRA)]

Geoffrey Jacoby created HBASE-21530:
---

 Summary: Abort_Procedure should be able to take a list of proc IDs
 Key: HBASE-21530
 URL: https://issues.apache.org/jira/browse/HBASE-21530
 Project: HBase
  Issue Type: Improvement
Reporter: Geoffrey Jacoby


As a convenience, it would be helpful if the HBase shell's abort_procedure call 
had the option of taking in multiple procedure ids at the same time, rather 
than relying on operators to use a loop in an external script. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HBASE-21529) Abort_Procedure in HBase shell fails when all handlers are taken up by Procedures

[Geoffrey Jacoby (JIRA)]

Geoffrey Jacoby created HBASE-21529:
---

 Summary: Abort_Procedure in HBase shell fails when all handlers 
are taken up by Procedures
 Key: HBASE-21529
 URL: https://issues.apache.org/jira/browse/HBASE-21529
 Project: HBase
  Issue Type: Bug
  Components: proc-v2
Affects Versions: 1.4.8, 1.3.2
Reporter: Geoffrey Jacoby


If all RPC handlers are taken up by stuck procedures, the operator will not be 
able to abort them, because the abort procedure call will try to use the same 
(exhausted) pool of RPC handlers. 

The abort procedure call should use a dedicated pool of RPC handlers. This pool 
can probably be small, but it needs to be separate. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HBASE-17912) Avoid major compactions on region server startup

[Geoffrey Jacoby (JIRA)]

Geoffrey Jacoby created HBASE-17912:
---

 Summary: Avoid major compactions on region server startup
 Key: HBASE-17912
 URL: https://issues.apache.org/jira/browse/HBASE-17912
 Project: HBase
  Issue Type: Improvement
  Components: Compaction
Affects Versions: 0.98.24, 1.3.1, 2.0.0
Reporter: Geoffrey Jacoby
Assignee: Geoffrey Jacoby


The HRegionServer.CompactionChecker chore wakes up every 10s and for each store 
in each region mods against a chore frequency (by default slightly under 3 
hours) to see if it's time to check if a major compaction is necessary for that 
store. 

The check to see if it needs to check for major compactions is calculated by 
if (iteration % multiplier != 0) continue;

where iteration is the number of times the chore has woken up. 

Because 0 % anything is 0, this will always check for necessary major 
compactions on each store when this chore is first run after the region server 
starts up. This can result in compaction storms when doing a rolling restart, 
because, for example, the new instance of the region server might get a lower 
jitter value than the old one had.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: [VOTE] Backup/Restore feature for HBase 2.0, vote closing 3/11/2017

[Geoffrey Jacoby]

I have no vote here, but I'd argue that HBASE-14417 and HBASE-14141
shouldn't be blockers. I agree that HBASE-15227 to add fault tolerance is a
blocker.

HBASE-14417 is support for incrementally backing up bulk loaded rows.
That's an important feature, but if you don't use bulk loads, or don't care
about _incremental_ backups of bulk loads, you'd be able to use backup
quite happily without it. Even if you bulk load occasionally, you can do a
full backup afterward. In the meantime, the docs and help text would just
need to make the limitation clear in big bold letters. :-)

HBASE-14141 is allowing HBase Backup to filter out unnecessary data from
its incremental backups. Since the backup tool allows you to specify that
only certain tables be backed up, incremental backups at WAL granularity
will accidentally backup some rows from unneeded tables. This doesn't
affect the correctness of the to-be-backed-up tables backups or restores,
only its storage cost. The feature works, but there's an important storage
optimization to be done.

As someone who works on clusters that don't use bulk load, and would want
to backup all tables, neither of these seems like a showstopper. However,
if HBASE-14141 would be a breaking change to the backup format, then that
would change my mind about it being a blocker.

Geoffrey



On Fri, Mar 10, 2017 at 3:01 PM, Andrew Purtell  wrote:

> Thanks for the offer but I like that you were honest about compiling a list
> of issues that you thought were blockers for release. Since this proposal
> is a merge into 2.0, and we are trying to release 2.0, I am -1 on this
> merge until those blockers are addressed.
>
> I had a look at the list.
>
> I think the documentation issue is important but not actually a blocker.
> That may be a controversial opinion, but documentation can be back-filled
> worst case. So take HBASE-17133 off the list.
>
> Remaining are effectively HBASE-14417, HBASE-14141, and HBASE-15227. They
> all have patches attached to the respective JIRAs so completing this work
> won't be onerous. Get these committed and I will lift my -1. The others who
> voted +1 on this thread surely can help with that.
>
> Thanks.
>
>
> On Fri, Mar 10, 2017 at 2:32 PM, Vladimir Rodionov  >
> wrote:
>
> > No problem I will downgrade Blockers to Majors if it scares you, Andrew
> 
> >
> > Sent from my iPhone
> >
> > > On Mar 10, 2017, at 1:52 PM, Andrew Purtell 
> wrote:
> > >
> > > ​I know the merge of this feature has lagged substantially. I think
> that
> > is
> > > regrettable but on another thread we are lamenting that 2.0 is already
> > > late. Unless I misunderstand, this is a proposal to merge something
> with
> > > known blockers into trunk before we branch it for 2.0 which will
> > > effectively prevent that release because these blockers will be there.
> I
> > am
> > > inclined to veto. Probably we should not propose branch merges into
> code
> > we
> > > are trying to get out the door with known blockers. Why not do that
> work
> > > first? It seems an obvious question. Perhaps I am missing something.
> > >
> > > If we can branch for 2.0 now and then merge this, and not into the 2.0
> > > branch, I would vote +1 for branch merge even with known blockers
> > pending.
> > > ​
> > >
> > > On Fri, Mar 10, 2017 at 1:42 PM, Vladimir Rodionov <
> > vladrodio...@gmail.com>
> > > wrote:
> > >
> > >> They are not blockers for merge - only for 2.0. GA
> > >> As I said already the feature is usable right now
> > >> We would like to continue working on master and we would like to see a
> > >> commitment from community
> > >>
> > >> Sent from my iPhone
> > >>
> > >> On Mar 10, 2017, at 11:16 AM, Andrew Purtell 
> > wrote:
> > >>
> >  Only BLOCKERs and CRITICALs are guaranteed for HBase 2.0 release.
> > >>>
> > >>> If we have identified blockers, why merge this before they are in?
> > >>> Otherwise we can't release 2.0, and it is overdue.
> > >>>
> > >>>
> > >>> On Wed, Mar 8, 2017 at 1:32 PM, Vladimir Rodionov <
> > >> vladrodio...@gmail.com>
> > >>> wrote:
> > >>>
> >  Hello, HBase folks
> > 
> >  For your consideration today is Backup/Restore feature for Apache
> > HBAse
> >  2.0.
> >  Backup code is available as a mega patch in HBASE-14123 (v61),
> applies
> >  cleanly to the current master, all test PASS, patch has no other
> > issues.
> > 
> >  The patch has gone through numerous rounds of code reviews and has
> > >> probably
> >  the most lengthy discussion thread on Apache JIRA (HBASE-14123) :)
> > 
> >  The work has been split into 3 phases (HBASE-14030, 14123, 14414)
> Two
> > >> first
> >  are complete, third one is still in progress.
> > 
> > 
> >  *** Summary of work HBASE-14123
> > 
> >  The new feature introduces new command-line extensions to the hbase
> > >> command
> >  and, from the client side, is accessible through 

[jira] [Created] (HBASE-17543) Create additional ReplicationEndpoint WALEntryFilters by configuration

[Geoffrey Jacoby (JIRA)]

Geoffrey Jacoby created HBASE-17543:
---

 Summary: Create additional ReplicationEndpoint WALEntryFilters by 
configuration
 Key: HBASE-17543
 URL: https://issues.apache.org/jira/browse/HBASE-17543
 Project: HBase
  Issue Type: Improvement
  Components: Replication
Reporter: Geoffrey Jacoby
Assignee: Geoffrey Jacoby


The existing BaseReplicationEndpoint creates a ChainWALEntryFilter containing a 
NamespaceTableCfWALEntryFilter and a ScopeWALEntryFilter. Adding a custom 
WALEntryFilter type to this chain requires creating an entirely new 
ReplicationEndpoint subclass and creating a new peer on the running cluster, 
which can be operationally complex to transition to without data loss in cases 
such as master/master.

For WALEntryFilters without constructor parameters, it would be straightforward 
to have a Configuration option to list additional WALEntryFilter classes the 
operator wants to include in the filter chain in the default endpoint, and then 
have the endpoint instantiate the filters via reflection. Then filter logic 
could be added (or removed) with only a hbase-site.xml change and a rolling 
restart. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-16576) Shell add_peer doesn't allow setting cluster_key for custom endpoints

[Geoffrey Jacoby (JIRA)]

Geoffrey Jacoby created HBASE-16576:
---

 Summary: Shell add_peer doesn't allow setting cluster_key for 
custom endpoints
 Key: HBASE-16576
 URL: https://issues.apache.org/jira/browse/HBASE-16576
 Project: HBase
  Issue Type: Bug
  Components: shell
Affects Versions: 0.98.22, 1.1.5, 2.0.0
Reporter: Geoffrey Jacoby
Assignee: Geoffrey Jacoby


The HBase shell allows a user to create a replication peer using the add_peer 
method, which can take a peer id and a Ruby hash. It creates a 
ReplicationPeerConfig and passes it through to the Java 
ReplicationAdmin#addPeer. 

The Ruby code makes an assumption that the Java API doesn't: that CLUSTER_KEY 
and ENDPOINT_CLASSNAME are mutually exclusive. If both are specified, it throws 
an error. If only ENDPOINT_CLASSNAME is set, the add_peer logic derives a local 
dummy cluster key based on the local cluster's configuration. 

CLUSTER_KEY shouldn't be required when an ENDPOINT_CLASSNAME is specified, 
because a custom endpoint might not need it. The dummy default logic is fine.  

But if an endpoint does require a remote cluster key, it shouldn't be forbidden 
to provide one, especially since the Java API permits it, and even the custom 
replication endpoint Java tests rely on this. (See 
TestReplicationEndpoint#testCustomReplicationEndpoint)

  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-16448) Custom metrics for custom replication endpoints

[Geoffrey Jacoby (JIRA)]

Geoffrey Jacoby created HBASE-16448:
---

 Summary: Custom metrics for custom replication endpoints
 Key: HBASE-16448
 URL: https://issues.apache.org/jira/browse/HBASE-16448
 Project: HBase
  Issue Type: Improvement
  Components: Replication
Affects Versions: 0.98.21, 1.2.2, 2.0.0
Reporter: Geoffrey Jacoby
Assignee: Geoffrey Jacoby


ReplicationEndpoints contain a MetricsSource class that can update a fixed set 
of key replication metrics. It does not, however, allow a developer 
implementing a custom ReplicationEndpoint to create/update custom metrics 
unique to the new endpoint's use case. 

As it turns out, MetricsSource wraps a class, which in turn wraps a class, that 
implements the Hadoop BaseSource interface, which does allow for custom 
metrics. Simply having the two wrappers implement BaseSource as well with 
pass-through methods should be straightforward, and would allow replication 
endpoints to use these generic metric methods.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-16108) RowCounter should support multiple key ranges

[Geoffrey Jacoby (JIRA)]

Geoffrey Jacoby created HBASE-16108:
---

 Summary: RowCounter should support multiple key ranges
 Key: HBASE-16108
 URL: https://issues.apache.org/jira/browse/HBASE-16108
 Project: HBase
  Issue Type: Improvement
Reporter: Geoffrey Jacoby


Currently, RowCounter only allows a single key range to be used as a filter. It 
would be useful in some cases to be able to specify multiple key ranges (or 
prefixes) in the same job. (For example, counting over a set of Phoenix tenant 
ids in an unsalted table)

This could be done by enhancing the existing key range parameter to take 
multiple start/stop row pairs. Alternately, a new --row-prefixes option could 
be added, similar to what HBASE-15847 did for VerifyReplication. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-15647) Backport HBASE-15507 to 0.98

[Geoffrey Jacoby (JIRA)]

Geoffrey Jacoby created HBASE-15647:
---

 Summary: Backport HBASE-15507 to 0.98
 Key: HBASE-15647
 URL: https://issues.apache.org/jira/browse/HBASE-15647
 Project: HBase
  Issue Type: Improvement
  Components: Replication
Affects Versions: 0.98.18
Reporter: Geoffrey Jacoby
Assignee: Geoffrey Jacoby


HBASE-15507 introduced an update_peer_config method to allow online changes to 
replication config. It depended on HBASE-11393, which so far exists only in 
master and can only be in 2.0 and above because of some incompatible changes to 
replication peer config serialization.
To get this in 0.98 will require two things:
1. Port the PeerConfigTracker in HBASE-11393 to 0.98 without taking the 
incompatible tableCF serialization changes.
2. Port HBASE-15507's logic on top of this.

Most of this work was done in HBASE-15633 for branch-1, but 0.98 is just 
different enough to need some tweaks of its own. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-15633) Backport HBASE-15507 to branch-1

[Geoffrey Jacoby (JIRA)]

Geoffrey Jacoby created HBASE-15633:
---

 Summary: Backport HBASE-15507 to branch-1
 Key: HBASE-15633
 URL: https://issues.apache.org/jira/browse/HBASE-15633
 Project: HBase
  Issue Type: New Feature
  Components: Replication, shell
Affects Versions: 1.3.0
Reporter: Geoffrey Jacoby
Assignee: Geoffrey Jacoby


HBase-15507 introduced an update_peer_config method to allow online changes to 
replication config. It depended on HBASE-11393, which so far exists only in 
master and can only be in 2.0 and above because of some incompatible changes to 
replication peer config serialization. 

To get this in branch-1 will require two things:
1. Port the PeerConfigTracker in HBASE-11393 to branch-1 without taking the 
incompatible tableCF serialization changes.
2. Port HBASE-15507's logic on top of this. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HBASE-15472) replication_admin_test creates a table it doesn't use

[Geoffrey Jacoby (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Geoffrey Jacoby resolved HBASE-15472.
-
Resolution: Duplicate

> replication_admin_test creates a table it doesn't use
> -
>
> Key: HBASE-15472
> URL: https://issues.apache.org/jira/browse/HBASE-15472
> Project: HBase
>  Issue Type: Bug
>  Components: Replication, shell
>Affects Versions: 2.0.0, 1.2.0
>    Reporter: Geoffrey Jacoby
>Assignee: Geoffrey Jacoby
>Priority: Minor
>  Labels: replication, shell
> Attachments: HBASE-15472.patch
>
>
> I noticed while adding tests to replication_admin_test.rb for HBASE-12940 
> that it creates an HBase table "hbase_shell_tests_table" that is never used 
> in any of the suite's tests. Removing the table creation statements speeds up 
> the suite locally from 1min 10s to 2s. 
> Note that until HBASE-14562 is worked, this test suite doesn't run as part of 
> the automatic test runs.  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-15567) TestReplicationShell broken by recent replication changes

[Geoffrey Jacoby (JIRA)]

Geoffrey Jacoby created HBASE-15567:
---

 Summary: TestReplicationShell broken by recent replication changes
 Key: HBASE-15567
 URL: https://issues.apache.org/jira/browse/HBASE-15567
 Project: HBase
  Issue Type: Bug
  Components: Replication, shell
Affects Versions: 2.0.0
Reporter: Geoffrey Jacoby
Assignee: Geoffrey Jacoby
Priority: Minor


Recent changes to the Ruby shell's add_peer method in HBASE-11393 have broken 
TestReplicationShell, which went unnoticed because it's currently Ignored as 
flaky. This test is useful when developing extensions to the replication shell 
commands, and should be kept working (and hopefully re-enabled in the near 
future.) 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-15507) Online modification of enabled ReplicationPeerConfig

[Geoffrey Jacoby (JIRA)]

Geoffrey Jacoby created HBASE-15507:
---

 Summary: Online modification of enabled ReplicationPeerConfig
 Key: HBASE-15507
 URL: https://issues.apache.org/jira/browse/HBASE-15507
 Project: HBase
  Issue Type: Improvement
  Components: Replication
Affects Versions: 1.2.0, 2.0.0, 1.3.0
Reporter: Geoffrey Jacoby
Assignee: Geoffrey Jacoby


It's currently possible to update the table CFs for a replication peer while 
it's running, but not the peer configuration or data maps introduced as part of 
custom replication endpoints in HBASE-11367. 

This means that if you have a custom endpoint that depends on some 
configuration parameter, you have to remove the peer and recreate it in order 
to change the param. In some use cases that's not possible without risking data 
loss. 

HBASE-11393, which will consolidate tableCFs in the same znode as the rest of 
ReplicationPeerConfig, may help here, but with or without it it still seems 
like there needs to be further work to add update config/data API support to 
ReplicationAdmin and a callback mechanism to notify the endpoints of config 
parameter changes. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-15472) replication_admin_test creates a table it doesn't use

[Geoffrey Jacoby (JIRA)]

Geoffrey Jacoby created HBASE-15472:
---

 Summary: replication_admin_test creates a table it doesn't use
 Key: HBASE-15472
 URL: https://issues.apache.org/jira/browse/HBASE-15472
 Project: HBase
  Issue Type: Bug
  Components: Replication, shell
Affects Versions: 1.2.0, 2.0.0
Reporter: Geoffrey Jacoby
Assignee: Geoffrey Jacoby
Priority: Minor


I noticed while adding tests to replication_admin_test.rb for HBASE-12940 that 
it creates an HBase table "hbase_shell_tests_table" that is never used in any 
of the suite's tests. Removing the table creation statements speeds up the 
suite locally from 1min 10s to 2s. 

Note that until HBASE-14562 is worked, this test suite doesn't run as part of 
the automatic test runs.  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-14985) TimeRange constructors should set allTime when appropriate

[Geoffrey Jacoby (JIRA)]

Geoffrey Jacoby created HBASE-14985:
---

 Summary: TimeRange constructors should set allTime when appropriate
 Key: HBASE-14985
 URL: https://issues.apache.org/jira/browse/HBASE-14985
 Project: HBase
  Issue Type: Improvement
Affects Versions: 0.98.16.1, 1.1.2
Reporter: Geoffrey Jacoby
Assignee: Geoffrey Jacoby
Priority: Minor


The default TimeRange constructor creates a range from 0 to Long.MAX_VALUE and 
sets an allTime flag to true. This flag allows some performance optimizations 
when comparing or using TimeRanges.

This flag is not set, however, if you call "new TimeRange(0L)" or "new 
TimeRange(0L, Long.MAX_VALUE)", even though both of these create a logically 
equivalent TimeRange to "new TimeRange()". Since TimeRanges are immutable and 
detecting this condition is trivial, we should set the flag automatically in 
the explicit constructors when appropriate. 




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-14416) HBase Backup/Restore Phase 2: Create plugin hooks for Backup tools

[Geoffrey Jacoby (JIRA)]

Geoffrey Jacoby created HBASE-14416:
---

 Summary: HBase Backup/Restore Phase 2: Create plugin hooks for 
Backup tools
 Key: HBASE-14416
 URL: https://issues.apache.org/jira/browse/HBASE-14416
 Project: HBase
  Issue Type: Sub-task
Reporter: Geoffrey Jacoby


Different organizations may already have their own backup tools for HBase, or 
wish to develop them in the future for their own particular use cases. It would 
be useful if HBase supported hooks to integrate those tools so that they could 
be configured and run in a standard way.

In particular, the administrative interface to start a backup, restore, or 
merge should be decoupled from any particular implementation as much as 
possible, so that any implementation with similar capabilities can be 
substituted via configuration without needing to fork or modify the code. 

Ideally, it will also be possible to decouple the various components so that 
implementations can be mixed and matched. (For example, one could use the 
standard backup's functionality to track what needs to be backed up, but use a 
custom plugin to change the logic or storage format of the backup operation 
itself.)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-13796) ZKUtil doesn't clean quorum setting properly

[Geoffrey Jacoby (JIRA)]

Geoffrey Jacoby created HBASE-13796:
---

 Summary: ZKUtil doesn't clean quorum setting properly
 Key: HBASE-13796
 URL: https://issues.apache.org/jira/browse/HBASE-13796
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.98.12, 1.1.0, 1.0.1
Reporter: Geoffrey Jacoby
Assignee: Geoffrey Jacoby
Priority: Minor


ZKUtil.getZooKeeperClusterKey is obviously trying to pull out the ZooKeeper 
quorum setting from the config object and remove several special characters 
from it. Due to a misplaced parenthesis, however, it's instead running the 
replace operation on the config setting _name_, HConstants.ZOOKEEPER_QUORUM, 
and not the config setting itself. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)