Re: [DISCUSS] deprecating jdk8, progress on LTS jdk support

[Duo Zhang]

I think this could be done by some module tricks, where we build a special
module with -source 8 and -target 8.We have done similar things in the past
to have hadoop1-compat and hadoop2-compact.

Let me have a try.

Sean Busbey  于2022年2月16日周三 13:31写道：

> Unfortunately if we want to keep jdk8 working we can't rely on building
> with the jdk release target option on newer jdks.
>
> We ran into this recently in HBASE-25465 where a JDK bug came up.
>
> On Tue, Feb 15, 2022 at 8:12 PM 张铎(Duo Zhang) 
> wrote:
>
> > +1 on moving the minimum required java version for compiling HBase to
> Java
> > 11. But we could still use --release=8 to still support jdk8 at runtime.
> >
> > Java 8 is still widely used, and I believe this will last for even more
> > years, as lots of big companies such as RedHat, Microsoft and Amazon are
> > still shipping new jdk8 releases, I do not think it is time to fully drop
> > the support of jdk8.
> >
> > Thanks.
> >
> > Sean Busbey  于2022年2月16日周三 09:57写道：
> >
> > > If we set the minJDK to 11 I believe that will effectively remove jdk8
> > > support, rather than "just" deprecate it.
> > >
> > > As we build out more robust testing I would be in favor of running less
> > of
> > > it on deprecated JDKs.
> > >
> > >
> > > On Tue, Feb 15, 2022, 16:10 Josh Elser  wrote:
> > >
> > > > Deprecating jdk8 for HBase 3 and requiring minJdk=11 seems reasonable
> > to
> > > > me.
> > > >
> > > > Gotta start pushing the issue somehow.
> > > >
> > > > On 2/15/22 1:47 PM, Sean Busbey wrote:
> > > > > Hi folks!
> > > > >
> > > > > It's been some time since we decided to stick to LTS JDK releases
> as
> > a
> > > > way
> > > > > of getting a handle on the JDK treadmill.
> > > > >
> > > > > What do folks think about deprecating JDK8? The openjdk8u project
> is
> > > > still
> > > > > going and there are commercial support options at least through
> 2030.
> > > > >
> > > > > Deprecating it in HBase 3 would mean we could remove it in HBase 4,
> > not
> > > > > that we would _have_ to remove it. The way I think about likely
> > timing
> > > of
> > > > > these events goes like this:
> > > > >
> > > > > * HBase 2 started alphas in June 2017, betas in January 2018, and
> > came
> > > > out
> > > > > in April 2018
> > > > > * HBase 3 started alphas in July 2021, and as of Feb 2022 we
> haven't
> > > > > discussed how close we are to our stated beta goals (upgrades from
> > > active
> > > > > 2.x releases and removal of not-ready features).
> > > > >
> > > > > Given the above, in the absence of us specifically pushing to roll
> > > > through
> > > > > major version numbers for some reason, I think a reasonably
> > > conservative
> > > > > estimate is for HBase 3 to arrive in late 2022 or early 2023 and
> then
> > > > HBase
> > > > > 4 to start alphas in ~2025. An HBase 5 prior to 2030 seems
> unlikely.
> > > > >
> > > > > That all said, our current reference guide section on java versions
> > > does
> > > > > not sound very confident about JDK11 support.
> > > > >
> > > > >> A Note on JDK11 *
> > > > >> Preliminary support for JDK11 is introduced with HBase 2.3.0. This
> > > > > support is limited to
> > > > >> compilation and running the full test suite. There are open
> > questions
> > > > > regarding the runtime
> > > > >> compatibility of JDK11 with Apache ZooKeeper and Apache Hadoop
> > > > > (HADOOP-15338).
> > > > >> Significantly, neither project has yet released a version with
> > > explicit
> > > > > runtime support for
> > > > >> JDK11. The remaining known issues in HBase are catalogued in
> > > > HBASE-22972.
> > > > >>
> > > > >
> > > > > Since that blurb was written, Hadoop has added JDK11 support [1] as
> > has
> > > > > ZooKeeper[2]. As a part of buttoning up our JDK11 support we could
> > > update
> > > > > our minimum supported versions of these projects to match that
> > support.
> > > > >
> > > > > What do folks think?
> > > > >
> > > > > [1]: https://hadoop.apache.org/docs/r3.3.0/index.html
> > > > > [2]:
> > > > >
> > >
> https://zookeeper.apache.org/doc/r3.6.0/zookeeperAdmin.html#sc_systemReq
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] deprecating jdk8, progress on LTS jdk support

[Sean Busbey]

Unfortunately if we want to keep jdk8 working we can't rely on building
with the jdk release target option on newer jdks.

We ran into this recently in HBASE-25465 where a JDK bug came up.

On Tue, Feb 15, 2022 at 8:12 PM 张铎(Duo Zhang)  wrote:

> +1 on moving the minimum required java version for compiling HBase to Java
> 11. But we could still use --release=8 to still support jdk8 at runtime.
>
> Java 8 is still widely used, and I believe this will last for even more
> years, as lots of big companies such as RedHat, Microsoft and Amazon are
> still shipping new jdk8 releases, I do not think it is time to fully drop
> the support of jdk8.
>
> Thanks.
>
> Sean Busbey  于2022年2月16日周三 09:57写道：
>
> > If we set the minJDK to 11 I believe that will effectively remove jdk8
> > support, rather than "just" deprecate it.
> >
> > As we build out more robust testing I would be in favor of running less
> of
> > it on deprecated JDKs.
> >
> >
> > On Tue, Feb 15, 2022, 16:10 Josh Elser  wrote:
> >
> > > Deprecating jdk8 for HBase 3 and requiring minJdk=11 seems reasonable
> to
> > > me.
> > >
> > > Gotta start pushing the issue somehow.
> > >
> > > On 2/15/22 1:47 PM, Sean Busbey wrote:
> > > > Hi folks!
> > > >
> > > > It's been some time since we decided to stick to LTS JDK releases as
> a
> > > way
> > > > of getting a handle on the JDK treadmill.
> > > >
> > > > What do folks think about deprecating JDK8? The openjdk8u project is
> > > still
> > > > going and there are commercial support options at least through 2030.
> > > >
> > > > Deprecating it in HBase 3 would mean we could remove it in HBase 4,
> not
> > > > that we would _have_ to remove it. The way I think about likely
> timing
> > of
> > > > these events goes like this:
> > > >
> > > > * HBase 2 started alphas in June 2017, betas in January 2018, and
> came
> > > out
> > > > in April 2018
> > > > * HBase 3 started alphas in July 2021, and as of Feb 2022 we haven't
> > > > discussed how close we are to our stated beta goals (upgrades from
> > active
> > > > 2.x releases and removal of not-ready features).
> > > >
> > > > Given the above, in the absence of us specifically pushing to roll
> > > through
> > > > major version numbers for some reason, I think a reasonably
> > conservative
> > > > estimate is for HBase 3 to arrive in late 2022 or early 2023 and then
> > > HBase
> > > > 4 to start alphas in ~2025. An HBase 5 prior to 2030 seems unlikely.
> > > >
> > > > That all said, our current reference guide section on java versions
> > does
> > > > not sound very confident about JDK11 support.
> > > >
> > > >> A Note on JDK11 *
> > > >> Preliminary support for JDK11 is introduced with HBase 2.3.0. This
> > > > support is limited to
> > > >> compilation and running the full test suite. There are open
> questions
> > > > regarding the runtime
> > > >> compatibility of JDK11 with Apache ZooKeeper and Apache Hadoop
> > > > (HADOOP-15338).
> > > >> Significantly, neither project has yet released a version with
> > explicit
> > > > runtime support for
> > > >> JDK11. The remaining known issues in HBase are catalogued in
> > > HBASE-22972.
> > > >>
> > > >
> > > > Since that blurb was written, Hadoop has added JDK11 support [1] as
> has
> > > > ZooKeeper[2]. As a part of buttoning up our JDK11 support we could
> > update
> > > > our minimum supported versions of these projects to match that
> support.
> > > >
> > > > What do folks think?
> > > >
> > > > [1]: https://hadoop.apache.org/docs/r3.3.0/index.html
> > > > [2]:
> > > >
> > https://zookeeper.apache.org/doc/r3.6.0/zookeeperAdmin.html#sc_systemReq
> > > >
> > >
> >
>

Re: [DISCUSS] deprecating jdk8, progress on LTS jdk support

[Sean Busbey]

> Regarding the original question, I would be in favor of the proposal. Time
> marches on. I assume just to state the obvious that our destination of
> minimum LTS would shift from 8 to 11.


Yes, sorry I should have expressly stated JDK11 would become the minimum
with some release after HBase 3.

I got here because I wanted to start working on qualifying JDK17 as a
runtime environment but then realized we were putting more caveats on JDK11
than I expected.

Hadoop 2 isn’t exactly dead, at least the source branch is still receiving
> occasional update, but is not releasing. We should probably consider it
> effectively EOL.


IIRC we've already dropped Hadoop 2 support for HBase 3.

The Hadoop minimum could become 3.3. The primary consideration to my mind
> is the state of S3A: in what version it can be said to be stable and
> feature complete. I think 3.3 is the appropriate code line for that
> criteria but perhaps 3.2 could serve as well.


I really like this as a criteria. Anyone else have an idea on this?



On Tue, Feb 15, 2022, 16:28 Andrew Purtell  wrote:

> The section in our docs on JDK11 seems out of date. In my experience using
> Java 11 to run ZooKeeper (3.5), Hadoop HDFS/YARN/MR (2.10 and 3.3), and
> HBase (2.4) is a nonissue. These software versions are stable under load.
> Most recently I tested scale ingest with 11.0.15+1, hot off the presses so
> to speak. No issues to report — although, I cannot claim this is the
> production configuration where I work yet. Perhaps near the end of 2022.
> Anyway, I think mine is likely not the only experience like this so the
> hedging language can be removed from the text, especially if someone else
> writes in with positive testimonial.
>
> Regarding the original question, I would be in favor of the proposal. Time
> matches on. I assume just to state the obvious that our destination of
> minimum LTS would shift from 8 to 11.
>
> I also agree it would be good to move up from EOL or soon to be EOL
> versions of our dependencies. Looks like ZooKeeper 3.6 will be the lowest
> live code line by the end of this year. Hadoop 2 isn’t exactly dead, at
> least the source branch is still receiving occasional update, but is not
> releasing. We should probably consider it effectively EOL. The Hadoop
> minimum could become 3.3. The primary consideration to my mind is the state
> of S3A: in what version it can be said to be stable and feature complete. I
> think 3.3 is the appropriate code line for that criteria but perhaps 3.2
> could serve as well.
>
> > On Feb 15, 2022, at 10:48 AM, Sean Busbey  wrote:
> >
> > Hi folks!
> >
> > It's been some time since we decided to stick to LTS JDK releases as a
> way
> > of getting a handle on the JDK treadmill.
> >
> > What do folks think about deprecating JDK8? The openjdk8u project is
> still
> > going and there are commercial support options at least through 2030.
> >
> > Deprecating it in HBase 3 would mean we could remove it in HBase 4, not
> > that we would _have_ to remove it. The way I think about likely timing of
> > these events goes like this:
> >
> > * HBase 2 started alphas in June 2017, betas in January 2018, and came
> out
> > in April 2018
> > * HBase 3 started alphas in July 2021, and as of Feb 2022 we haven't
> > discussed how close we are to our stated beta goals (upgrades from active
> > 2.x releases and removal of not-ready features).
> >
> > Given the above, in the absence of us specifically pushing to roll
> through
> > major version numbers for some reason, I think a reasonably conservative
> > estimate is for HBase 3 to arrive in late 2022 or early 2023 and then
> HBase
> > 4 to start alphas in ~2025. An HBase 5 prior to 2030 seems unlikely.
> >
> > That all said, our current reference guide section on java versions does
> > not sound very confident about JDK11 support.
> >
> >> A Note on JDK11 *
> >> Preliminary support for JDK11 is introduced with HBase 2.3.0. This
> > support is limited to
> >> compilation and running the full test suite. There are open questions
> > regarding the runtime
> >> compatibility of JDK11 with Apache ZooKeeper and Apache Hadoop
> > (HADOOP-15338).
> >> Significantly, neither project has yet released a version with explicit
> > runtime support for
> >> JDK11. The remaining known issues in HBase are catalogued in
> HBASE-22972.
> >>
> >
> > Since that blurb was written, Hadoop has added JDK11 support [1] as has
> > ZooKeeper[2]. As a part of buttoning up our JDK11 support we could update
> > our minimum supported versions of these projects to match that support.
> >
> > What do folks think?
> >
> > [1]: https://hadoop.apache.org/docs/r3.3.0/index.html
> > [2]:
> > https://zookeeper.apache.org/doc/r3.6.0/zookeeperAdmin.html#sc_systemReq
>

[jira] [Resolved] (HBASE-26742) Comparator of NOT_EQUAL NULL is invalid for checkAndMutate

[Xiaolin Ha (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-26742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiaolin Ha resolved HBASE-26742.

Fix Version/s: 2.5.0
   3.0.0-alpha-3
   2.4.10
   Resolution: Fixed

Merged to branch-2.4+, thanks [~zhangduo] for reviewing.

> Comparator of NOT_EQUAL NULL is invalid for checkAndMutate
> --
>
> Key: HBASE-26742
> URL: https://issues.apache.org/jira/browse/HBASE-26742
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 3.0.0-alpha-1, 2.0.0
>Reporter: Xiaolin Ha
>Assignee: Xiaolin Ha
>Priority: Major
> Fix For: 2.5.0, 3.0.0-alpha-3, 2.4.10
>
>
>  In server side, checkAndMutate ignores CompareOperator for null or empty 
> comparator value, but NOT_EQUAL should be treated specially.
> The check logic in HRegion#checkAndMutateInternal is as follows,
> {code:java}
> boolean valueIsNull =
>   comparator.getValue() == null || comparator.getValue().length == 0;
> if (result.isEmpty() && valueIsNull) {
>   matches = true;
> } else if (result.size() > 0 && result.get(0).getValueLength() == 0 && 
> valueIsNull) {
>   matches = true;
>   cellTs = result.get(0).getTimestamp();
> } else if (result.size() == 1 && !valueIsNull) {
>   Cell kv = result.get(0);
>   cellTs = kv.getTimestamp();
>   int compareResult = PrivateCellUtil.compareValue(kv, comparator);
>   matches = matches(op, compareResult);
> }{code}
> For current logics, here are some  counter examples(Comparator value is set 
> null),
>  # result is null, operator is NOT_EQUAL, but matches is true;
>  # result size >0, the value of the first cell is empty, operator is 
> NOT_EQUAL, but matches is true;
>  # result size is 1, operator is NOT_EQUAL, but matches is false;



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Re: [DISCUSS] deprecating jdk8, progress on LTS jdk support

[Duo Zhang]

+1 on moving the minimum required java version for compiling HBase to Java
11. But we could still use --release=8 to still support jdk8 at runtime.

Java 8 is still widely used, and I believe this will last for even more
years, as lots of big companies such as RedHat, Microsoft and Amazon are
still shipping new jdk8 releases, I do not think it is time to fully drop
the support of jdk8.

Thanks.

Sean Busbey  于2022年2月16日周三 09:57写道：

> If we set the minJDK to 11 I believe that will effectively remove jdk8
> support, rather than "just" deprecate it.
>
> As we build out more robust testing I would be in favor of running less of
> it on deprecated JDKs.
>
>
> On Tue, Feb 15, 2022, 16:10 Josh Elser  wrote:
>
> > Deprecating jdk8 for HBase 3 and requiring minJdk=11 seems reasonable to
> > me.
> >
> > Gotta start pushing the issue somehow.
> >
> > On 2/15/22 1:47 PM, Sean Busbey wrote:
> > > Hi folks!
> > >
> > > It's been some time since we decided to stick to LTS JDK releases as a
> > way
> > > of getting a handle on the JDK treadmill.
> > >
> > > What do folks think about deprecating JDK8? The openjdk8u project is
> > still
> > > going and there are commercial support options at least through 2030.
> > >
> > > Deprecating it in HBase 3 would mean we could remove it in HBase 4, not
> > > that we would _have_ to remove it. The way I think about likely timing
> of
> > > these events goes like this:
> > >
> > > * HBase 2 started alphas in June 2017, betas in January 2018, and came
> > out
> > > in April 2018
> > > * HBase 3 started alphas in July 2021, and as of Feb 2022 we haven't
> > > discussed how close we are to our stated beta goals (upgrades from
> active
> > > 2.x releases and removal of not-ready features).
> > >
> > > Given the above, in the absence of us specifically pushing to roll
> > through
> > > major version numbers for some reason, I think a reasonably
> conservative
> > > estimate is for HBase 3 to arrive in late 2022 or early 2023 and then
> > HBase
> > > 4 to start alphas in ~2025. An HBase 5 prior to 2030 seems unlikely.
> > >
> > > That all said, our current reference guide section on java versions
> does
> > > not sound very confident about JDK11 support.
> > >
> > >> A Note on JDK11 *
> > >> Preliminary support for JDK11 is introduced with HBase 2.3.0. This
> > > support is limited to
> > >> compilation and running the full test suite. There are open questions
> > > regarding the runtime
> > >> compatibility of JDK11 with Apache ZooKeeper and Apache Hadoop
> > > (HADOOP-15338).
> > >> Significantly, neither project has yet released a version with
> explicit
> > > runtime support for
> > >> JDK11. The remaining known issues in HBase are catalogued in
> > HBASE-22972.
> > >>
> > >
> > > Since that blurb was written, Hadoop has added JDK11 support [1] as has
> > > ZooKeeper[2]. As a part of buttoning up our JDK11 support we could
> update
> > > our minimum supported versions of these projects to match that support.
> > >
> > > What do folks think?
> > >
> > > [1]: https://hadoop.apache.org/docs/r3.3.0/index.html
> > > [2]:
> > >
> https://zookeeper.apache.org/doc/r3.6.0/zookeeperAdmin.html#sc_systemReq
> > >
> >
>

Re: [DISCUSS] deprecating jdk8, progress on LTS jdk support

[Sean Busbey]

If we set the minJDK to 11 I believe that will effectively remove jdk8
support, rather than "just" deprecate it.

As we build out more robust testing I would be in favor of running less of
it on deprecated JDKs.


On Tue, Feb 15, 2022, 16:10 Josh Elser  wrote:

> Deprecating jdk8 for HBase 3 and requiring minJdk=11 seems reasonable to
> me.
>
> Gotta start pushing the issue somehow.
>
> On 2/15/22 1:47 PM, Sean Busbey wrote:
> > Hi folks!
> >
> > It's been some time since we decided to stick to LTS JDK releases as a
> way
> > of getting a handle on the JDK treadmill.
> >
> > What do folks think about deprecating JDK8? The openjdk8u project is
> still
> > going and there are commercial support options at least through 2030.
> >
> > Deprecating it in HBase 3 would mean we could remove it in HBase 4, not
> > that we would _have_ to remove it. The way I think about likely timing of
> > these events goes like this:
> >
> > * HBase 2 started alphas in June 2017, betas in January 2018, and came
> out
> > in April 2018
> > * HBase 3 started alphas in July 2021, and as of Feb 2022 we haven't
> > discussed how close we are to our stated beta goals (upgrades from active
> > 2.x releases and removal of not-ready features).
> >
> > Given the above, in the absence of us specifically pushing to roll
> through
> > major version numbers for some reason, I think a reasonably conservative
> > estimate is for HBase 3 to arrive in late 2022 or early 2023 and then
> HBase
> > 4 to start alphas in ~2025. An HBase 5 prior to 2030 seems unlikely.
> >
> > That all said, our current reference guide section on java versions does
> > not sound very confident about JDK11 support.
> >
> >> A Note on JDK11 *
> >> Preliminary support for JDK11 is introduced with HBase 2.3.0. This
> > support is limited to
> >> compilation and running the full test suite. There are open questions
> > regarding the runtime
> >> compatibility of JDK11 with Apache ZooKeeper and Apache Hadoop
> > (HADOOP-15338).
> >> Significantly, neither project has yet released a version with explicit
> > runtime support for
> >> JDK11. The remaining known issues in HBase are catalogued in
> HBASE-22972.
> >>
> >
> > Since that blurb was written, Hadoop has added JDK11 support [1] as has
> > ZooKeeper[2]. As a part of buttoning up our JDK11 support we could update
> > our minimum supported versions of these projects to match that support.
> >
> > What do folks think?
> >
> > [1]: https://hadoop.apache.org/docs/r3.3.0/index.html
> > [2]:
> > https://zookeeper.apache.org/doc/r3.6.0/zookeeperAdmin.html#sc_systemReq
> >
>

Re: [DISCUSS] deprecating jdk8, progress on LTS jdk support

[Andrew Purtell]

The section in our docs on JDK11 seems out of date. In my experience using Java 
11 to run ZooKeeper (3.5), Hadoop HDFS/YARN/MR (2.10 and 3.3), and HBase (2.4) 
is a nonissue. These software versions are stable under load. Most recently I 
tested scale ingest with 11.0.15+1, hot off the presses so to speak. No issues 
to report — although, I cannot claim this is the production configuration where 
I work yet. Perhaps near the end of 2022. Anyway, I think mine is likely not 
the only experience like this so the hedging language can be removed from the 
text, especially if someone else writes in with positive testimonial. 

Regarding the original question, I would be in favor of the proposal. Time 
matches on. I assume just to state the obvious that our destination of minimum 
LTS would shift from 8 to 11. 

I also agree it would be good to move up from EOL or soon to be EOL versions of 
our dependencies. Looks like ZooKeeper 3.6 will be the lowest live code line by 
the end of this year. Hadoop 2 isn’t exactly dead, at least the source branch 
is still receiving occasional update, but is not releasing. We should probably 
consider it effectively EOL. The Hadoop minimum could become 3.3. The primary 
consideration to my mind is the state of S3A: in what version it can be said to 
be stable and feature complete. I think 3.3 is the appropriate code line for 
that criteria but perhaps 3.2 could serve as well. 

> On Feb 15, 2022, at 10:48 AM, Sean Busbey  wrote:
> 
> Hi folks!
> 
> It's been some time since we decided to stick to LTS JDK releases as a way
> of getting a handle on the JDK treadmill.
> 
> What do folks think about deprecating JDK8? The openjdk8u project is still
> going and there are commercial support options at least through 2030.
> 
> Deprecating it in HBase 3 would mean we could remove it in HBase 4, not
> that we would _have_ to remove it. The way I think about likely timing of
> these events goes like this:
> 
> * HBase 2 started alphas in June 2017, betas in January 2018, and came out
> in April 2018
> * HBase 3 started alphas in July 2021, and as of Feb 2022 we haven't
> discussed how close we are to our stated beta goals (upgrades from active
> 2.x releases and removal of not-ready features).
> 
> Given the above, in the absence of us specifically pushing to roll through
> major version numbers for some reason, I think a reasonably conservative
> estimate is for HBase 3 to arrive in late 2022 or early 2023 and then HBase
> 4 to start alphas in ~2025. An HBase 5 prior to 2030 seems unlikely.
> 
> That all said, our current reference guide section on java versions does
> not sound very confident about JDK11 support.
> 
>> A Note on JDK11 *
>> Preliminary support for JDK11 is introduced with HBase 2.3.0. This
> support is limited to
>> compilation and running the full test suite. There are open questions
> regarding the runtime
>> compatibility of JDK11 with Apache ZooKeeper and Apache Hadoop
> (HADOOP-15338).
>> Significantly, neither project has yet released a version with explicit
> runtime support for
>> JDK11. The remaining known issues in HBase are catalogued in HBASE-22972.
>> 
> 
> Since that blurb was written, Hadoop has added JDK11 support [1] as has
> ZooKeeper[2]. As a part of buttoning up our JDK11 support we could update
> our minimum supported versions of these projects to match that support.
> 
> What do folks think?
> 
> [1]: https://hadoop.apache.org/docs/r3.3.0/index.html
> [2]:
> https://zookeeper.apache.org/doc/r3.6.0/zookeeperAdmin.html#sc_systemReq

Re: [DISCUSS] deprecating jdk8, progress on LTS jdk support

[Josh Elser]

Deprecating jdk8 for HBase 3 and requiring minJdk=11 seems reasonable to me.

Gotta start pushing the issue somehow.

On 2/15/22 1:47 PM, Sean Busbey wrote:

Hi folks!

It's been some time since we decided to stick to LTS JDK releases as a way
of getting a handle on the JDK treadmill.

What do folks think about deprecating JDK8? The openjdk8u project is still
going and there are commercial support options at least through 2030.

Deprecating it in HBase 3 would mean we could remove it in HBase 4, not
that we would _have_ to remove it. The way I think about likely timing of
these events goes like this:

* HBase 2 started alphas in June 2017, betas in January 2018, and came out
in April 2018
* HBase 3 started alphas in July 2021, and as of Feb 2022 we haven't
discussed how close we are to our stated beta goals (upgrades from active
2.x releases and removal of not-ready features).

Given the above, in the absence of us specifically pushing to roll through
major version numbers for some reason, I think a reasonably conservative
estimate is for HBase 3 to arrive in late 2022 or early 2023 and then HBase
4 to start alphas in ~2025. An HBase 5 prior to 2030 seems unlikely.

That all said, our current reference guide section on java versions does
not sound very confident about JDK11 support.


A Note on JDK11 *
Preliminary support for JDK11 is introduced with HBase 2.3.0. This

support is limited to

compilation and running the full test suite. There are open questions

regarding the runtime

compatibility of JDK11 with Apache ZooKeeper and Apache Hadoop

(HADOOP-15338).

Significantly, neither project has yet released a version with explicit

runtime support for

JDK11. The remaining known issues in HBase are catalogued in HBASE-22972.



Since that blurb was written, Hadoop has added JDK11 support [1] as has
ZooKeeper[2]. As a part of buttoning up our JDK11 support we could update
our minimum supported versions of these projects to match that support.

What do folks think?

[1]: https://hadoop.apache.org/docs/r3.3.0/index.html
[2]:
https://zookeeper.apache.org/doc/r3.6.0/zookeeperAdmin.html#sc_systemReq

Re: [DISCUSS] operator tools, HBase 3 and StoreFileTracking

[Andrew Purtell]

Another option which I do not see mentioned yet is to extract the relevant 
common proto and source files from the ‘hbase’ repository into a new repository 
(‘hbase-storage’?), from which we would release artifacts to be consumed by 
both hbase and hbase-operator-tools. This maintains D.R.Y. through refactoring 
although it may down the road cause some complexity in coordinating evolution 
among the three (if not more) repositories and releases produced from them. 
This is like Josh’s Option 1 but without duplication. 

Regarding the option 2 issue… If it would help we can drop SFT into branch-2.5 
along with the log4j2 changes and release 2.5.0 afterward. We are taking the 
opportunity of this minor increment to accelerate log4j1 retirement, which is 
why it’s still waiting (but not for long). We can use the same opportunity to 
release SFT even if we designate it as an experimental feature if that would 
simplify some other logistics. For what it’s worth. 

> On Feb 15, 2022, at 7:44 AM, Josh Elser  wrote:
> 
> I was talking with Szabolcs prior to him sending this one, and it's a tricky 
> issue for sure.
> 
> To date, we've solved any HBase API issues by copying code into HBCK2 e.g. 
> HBCKMetaTableAccessor which copies parts of MetaTableAccessor, or we push the 
> logic down server-side to the HBase Master and invoke it over the Hbck RPC 
> interface.
> 
> I definitely want to avoid HBase version specific builds of the 
> operator-tools, so that is not an option in my mind for 2.x. The discussions 
> we had (that I remember) around HBCK2 were limited in scope to HBase 2.x.
> 
> Option 1: we copy the necessary proto files from HBase into the 
> operator-tools and try to remember that, if we make any change to the 
> serialization of the storefile list files, we have to copy that change to 
> HBCK2. Brittle on the surface but effective.
> 
> Option 2: We bump HBCK2 to hbase-2.6.0-SNAPSHOT. Problematic until we make an 
> HBase 2.6.0[-alpha] release. We should already have wire compat between all 
> of HBase 2.x which makes that a non-issue.
> 
> Option 3: We create an HBCK3 targeted for HBase 3.x. I'm not convinced we 
> need to do that (hbck for hbase 3.x would be just like hbck for hbase 2.x). 
> This would also not solve the problem for the SFT feature in hbase 2.6.
> 
> I think option 3 is a no-go. I am leaning towards option 1 at this point. 
> Hopefully my thought process is helpful for others to weigh in.
> 
> 
>> On 2/14/22 11:31 AM, Szabolcs Bukros wrote:
>> Hi Folks!
>> While working on adding tools to handle potential FileBased
>> StoreFileTracker issues to HBCK2 (HBASE-26624
>> ) I ran into multiple
>> problems I'm unsure how to solve.
>> First of all the tools would rely on files not yet available in any of the
>> released hbase artifacts. I tried to solve this without changing the hbase
>> dependency version to keep HBCK2 as hbase version independent as possible,
>> but none of the solutions I have found looked acceptable:
>>  - Pushing the logic to the hbase side (as far as I can tell) is not
>> feasible because it has to be able to repair meta which is easier when
>> hbase is down and the tool should be able to run without a working hbase.
>>  - The files tracking the store content are serialized proto objects so
>> while replicating those files in the operator tools is possible, it would
>> not be pretty.
>> Bumping operator tools to use hbase 2.6.0-SNAPSHOT (branch-2 has the SFT
>> changes) would mean that now we need that or a newer version to build the
>> project and a version check to avoid runtime problems with the new tools,
>> but otherwise this looks rather painless and backwards compatible. I know
>> operator tools tries to avoid having a hbase-specific release, but having
>> 2.6 as a min version to build against might be acceptable.
>> While looking into this I also checked what needs to be done to make
>> operator tools work with hbase 3.0.0-alpha-3-SNAPSHOT. Most of the changes
>> are backwards compatible but not all of them and the ones that aren't would
>> make a big chunk of Fsck unusable with older hbases. For me that looks
>> acceptable since this is a major version change, but that would mean I can
>> not rely on a potential HBCK3 to fix SFT issues, I would also need a
>> solution for HBCK2.
>> I tried to look for plans/direction regarding the new 1.3 operator tools
>> but could not find any.
>> Do you think it would be possible to bump the hbase version it uses to
>> 2.6.0-SNAPSHOT?
>> Do you think it would make sense to start working on a hbase3 compatible
>> branch or is it too early?
>> NOTE:
>> I'm aware hbase does not publish SNAPSHOT builds for years, but I do not
>> know how the internal build system works and if these artifacts would be
>> available for internal builds or not. I also do not know if necessary could
>> they be made available.

[DISCUSS] deprecating jdk8, progress on LTS jdk support

[Sean Busbey]

Hi folks!

It's been some time since we decided to stick to LTS JDK releases as a way
of getting a handle on the JDK treadmill.

What do folks think about deprecating JDK8? The openjdk8u project is still
going and there are commercial support options at least through 2030.

Deprecating it in HBase 3 would mean we could remove it in HBase 4, not
that we would _have_ to remove it. The way I think about likely timing of
these events goes like this:

* HBase 2 started alphas in June 2017, betas in January 2018, and came out
in April 2018
* HBase 3 started alphas in July 2021, and as of Feb 2022 we haven't
discussed how close we are to our stated beta goals (upgrades from active
2.x releases and removal of not-ready features).

Given the above, in the absence of us specifically pushing to roll through
major version numbers for some reason, I think a reasonably conservative
estimate is for HBase 3 to arrive in late 2022 or early 2023 and then HBase
4 to start alphas in ~2025. An HBase 5 prior to 2030 seems unlikely.

That all said, our current reference guide section on java versions does
not sound very confident about JDK11 support.

> A Note on JDK11 *
> Preliminary support for JDK11 is introduced with HBase 2.3.0. This
support is limited to
> compilation and running the full test suite. There are open questions
regarding the runtime
> compatibility of JDK11 with Apache ZooKeeper and Apache Hadoop
(HADOOP-15338).
> Significantly, neither project has yet released a version with explicit
runtime support for
> JDK11. The remaining known issues in HBase are catalogued in HBASE-22972.
>

Since that blurb was written, Hadoop has added JDK11 support [1] as has
ZooKeeper[2]. As a part of buttoning up our JDK11 support we could update
our minimum supported versions of these projects to match that support.

What do folks think?

[1]: https://hadoop.apache.org/docs/r3.3.0/index.html
[2]:
https://zookeeper.apache.org/doc/r3.6.0/zookeeperAdmin.html#sc_systemReq

Re: [DISCUSS] operator tools, HBase 3 and StoreFileTracking

[Josh Elser]

I was talking with Szabolcs prior to him sending this one, and it's a 
tricky issue for sure.


To date, we've solved any HBase API issues by copying code into HBCK2 
e.g. HBCKMetaTableAccessor which copies parts of MetaTableAccessor, or 
we push the logic down server-side to the HBase Master and invoke it 
over the Hbck RPC interface.


I definitely want to avoid HBase version specific builds of the 
operator-tools, so that is not an option in my mind for 2.x. The 
discussions we had (that I remember) around HBCK2 were limited in scope 
to HBase 2.x.


Option 1: we copy the necessary proto files from HBase into the 
operator-tools and try to remember that, if we make any change to the 
serialization of the storefile list files, we have to copy that change 
to HBCK2. Brittle on the surface but effective.


Option 2: We bump HBCK2 to hbase-2.6.0-SNAPSHOT. Problematic until we 
make an HBase 2.6.0[-alpha] release. We should already have wire compat 
between all of HBase 2.x which makes that a non-issue.


Option 3: We create an HBCK3 targeted for HBase 3.x. I'm not convinced 
we need to do that (hbck for hbase 3.x would be just like hbck for hbase 
2.x). This would also not solve the problem for the SFT feature in hbase 
2.6.


I think option 3 is a no-go. I am leaning towards option 1 at this 
point. Hopefully my thought process is helpful for others to weigh in.



On 2/14/22 11:31 AM, Szabolcs Bukros wrote:

Hi Folks!

While working on adding tools to handle potential FileBased
StoreFileTracker issues to HBCK2 (HBASE-26624
) I ran into multiple
problems I'm unsure how to solve.

First of all the tools would rely on files not yet available in any of the
released hbase artifacts. I tried to solve this without changing the hbase
dependency version to keep HBCK2 as hbase version independent as possible,
but none of the solutions I have found looked acceptable:
  - Pushing the logic to the hbase side (as far as I can tell) is not
feasible because it has to be able to repair meta which is easier when
hbase is down and the tool should be able to run without a working hbase.
  - The files tracking the store content are serialized proto objects so
while replicating those files in the operator tools is possible, it would
not be pretty.

Bumping operator tools to use hbase 2.6.0-SNAPSHOT (branch-2 has the SFT
changes) would mean that now we need that or a newer version to build the
project and a version check to avoid runtime problems with the new tools,
but otherwise this looks rather painless and backwards compatible. I know
operator tools tries to avoid having a hbase-specific release, but having
2.6 as a min version to build against might be acceptable.

While looking into this I also checked what needs to be done to make
operator tools work with hbase 3.0.0-alpha-3-SNAPSHOT. Most of the changes
are backwards compatible but not all of them and the ones that aren't would
make a big chunk of Fsck unusable with older hbases. For me that looks
acceptable since this is a major version change, but that would mean I can
not rely on a potential HBCK3 to fix SFT issues, I would also need a
solution for HBCK2.

I tried to look for plans/direction regarding the new 1.3 operator tools
but could not find any.

Do you think it would be possible to bump the hbase version it uses to
2.6.0-SNAPSHOT?
Do you think it would make sense to start working on a hbase3 compatible
branch or is it too early?

NOTE:
I'm aware hbase does not publish SNAPSHOT builds for years, but I do not
know how the internal build system works and if these artifacts would be
available for internal builds or not. I also do not know if necessary could
they be made available.

[jira] [Resolved] (HBASE-26434) Compact L0 files for cold regions using StripeCompactionPolicy

[Xiaolin Ha (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-26434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiaolin Ha resolved HBASE-26434.

Fix Version/s: 2.5.0
   3.0.0-alpha-3
   2.4.10
   Resolution: Fixed

Merged to branch-2.4+, thanks [~zhangduo] for reviewing.

> Compact L0 files for cold regions using StripeCompactionPolicy
> --
>
> Key: HBASE-26434
> URL: https://issues.apache.org/jira/browse/HBASE-26434
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 3.0.0-alpha-1, 2.0.0
>Reporter: Xiaolin Ha
>Assignee: Xiaolin Ha
>Priority: Major
> Fix For: 2.5.0, 3.0.0-alpha-3, 2.4.10
>
>
> For cold regions, the L0 files can be all expired after a long while. We 
> should compact and delete all TTL expired data in them if the count of them 
> is less than the configured min files to compact in L0.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Re: [DISCUSS] Replace log4j with reload4j for branch-2.x

[Duo Zhang]

Got it.

We could try it once log4j 2.17.2 is released.

Thanks Andrew for helping!

Andrew Purtell  于2022年2月15日周二 11:51写道：

> If eg 2.4.9 binaries could be replaced with 2.5.0 and configuration files
> won’t need updating, there is no concern.
>
> > On Feb 14, 2022, at 7:35 PM, 张铎  wrote:
> >
> > The new file format is for log4j2, which has a different name,
> > log4j2.properties.
> >
> > When you have log4j.properties and log4j-1.2-api.jar on the classpath,
> the
> > log4j1 bridge will take charge and initialize the logging system based on
> > your log4j.properties. This is my point.
> >
> > So in general, if you still want to use log4j.properties file, just
> remove
> > the log4j2.properties. If you want to fully migrate to log4j2, then you
> > need to learn the new grammar and change the log4j2.properties.
> >
> > We should mention this in the upgrade section.
> >
> > Thanks.
> >
> > Andrew Purtell 于2022年2月15日 周二10:52写道：
> >
> >>> In the new release version we will provide a new log4j2 format
> properties
> >> file
> >>
> >> How can we assume users have not changed their log4j properties file? I
> >> have done it quite often in production. I can't be the only one.
> >>
> >>> Even if users have provided their own version of log4j.properties, we
> >> have log4j-1.2-api, it will read the log4j.properties and initialize the
> >> logging system based on log4j1 properties file
> >>
> >> Pardon me, but based on the new properties file in this patch, that does
> >> not seem to be true.
> >>
> >> I the log4j.properties file we ship now, property names begin with
> 'log4j'.
> >> In the new example provided by the log4j2 upgrade patch, property names
> do
> >> not begin  with 'log4j.' .
> >>
> >> Operational compatibility in minor releases means existing configuration
> >> files should work with no changes required.
> >> Now, maybe this is a special case, but I do not like the idea of
> breaking
> >> users with something dumb like dropping the 'log4j.' prefix from
> property
> >> names, if it renders existing properties based configuration
> nonfunctional.
> >> So can we bridge this? That is my point.
> >>
> >>
> >>> On Sat, Feb 12, 2022 at 10:30 PM 张铎(Duo Zhang) 
> >>> wrote:
> >>>
> >>> Good news, another approach for implementing LOG4J2-3341 has been
> landed,
> >>> and this time it is OK for us to pass hbase.root.logger in. This means
> >> the
> >>> most blocker issue for backporting the log4j2 support to branch-2 is
> >> gone.
> >>> We could try it once 2.17.2 is out.
> >>>
> >>> Andrew Purtell  于2022年2月8日周二 09:39写道：
> >>>
>  When I was looking at the PR I also noticed a potentially annoying and
>  compatibility breaking problem with the properties file format. Is it
>  really true that with log4j1 the properties begin with 'log4j.' and
> >> with
>  log4j2 the 'log4j.' prefix is removed? If so user configured files
> >> won't
> >>> be
>  compatible unless we can intercept the parse of the file and map the
> >> one
> >>> to
>  the other, somehow. Pardon my ignorance if that would not be necessary
> >>> and
>  there is actually a path to compatibility. Anyway I mention it because
> >> if
>  it is an issue, for both of these issues, perhaps we can plug
> something
> >>> in
>  rather than rely on the log4j community? It is just a thought...
> 
> >>> I do not get the point here. You mean we could do a configuration file
> >>> translation? In the new release version we will provide a new log4j2
> >> format
> >>> properties file, and I guess for most users they do not need to modify
> >> this
> >>> file, just modify hbase-env.sh to control the level and appenders is
> >>> enough.
> >>> Even if users have provided their own version of log4j.properties, we
> >> have
> >>> log4j-1.2-api, it will read the log4j.properties and initialize the
> >> logging
> >>> system based on log4j1 properties file, unless you have implemented
> your
> >>> log4j1 appenders.
> >>>
> 
>  On Mon, Feb 7, 2022 at 4:26 PM 张铎(Duo Zhang) 
>  wrote:
> 
> > Yes, the current solution does not work, they do not want to do
> >>> variable
> > lookup before entering the format free configuration processing step.
> >
> > Will try to push the log4j community to give another solution.
> >
> > Thanks.
> >
> > Andrew Purtell  于2022年2月8日周二 03:54写道：
> >
> >> So it looks like LOG4J2-3341 is going to be reverted? Backport
> >> seems
> >> premature until this is resolved with the new solution.
> >>
> >> On Sat, Feb 5, 2022 at 12:04 PM Andrew Purtell <
> >> apurt...@apache.org>
> >> wrote:
> >>
> >>> Thank you for providing a PR, will review on Monday.
> >>>
> >>> On Sat, Feb 5, 2022 at 5:16 AM 张铎(Duo Zhang) <
> >>> palomino...@gmail.com>
> >>> wrote:
> >>>
>  https://github.com/apache/hbase/pull/4096
> 
>  The PR based on log4j 2.17.2-SNAPSHOT is ready.
> 
>  PTAL.
> 
>  

[jira] [Resolved] (HBASE-26688) Threads shared EMPTY_RESULT may lead to unexpected client job down.

[Duo Zhang (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-26688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Duo Zhang resolved HBASE-26688.
---
Resolution: Fixed

Merged to branch-2.4+.

Thanks [~xytss123] for contributing and all others for helping!

> Threads shared EMPTY_RESULT may lead to unexpected client job down.
> ---
>
> Key: HBASE-26688
> URL: https://issues.apache.org/jira/browse/HBASE-26688
> Project: HBase
>  Issue Type: Bug
>  Components: Client
>Reporter: Yutong Xiao
>Assignee: Yutong Xiao
>Priority: Major
> Fix For: 2.5.0, 3.0.0-alpha-3, 2.4.10
>
>
> Currently, we use a pre-created EMPTY_RESULT in the ProtoBuf.util to reduce 
> the object creation. But these objects could be shared by multi client 
> threads. The Result#cellScannerIndex related methods could throw confusing 
> exception and make the client job down. Could refine the logic of these 
> methods.
> The precreated objects in ProtoBufUtil.java:
> {code:java}
> private final static Cell[] EMPTY_CELL_ARRAY = new Cell[]{};
> private final static Result EMPTY_RESULT = Result.create(EMPTY_CELL_ARRAY);
> final static Result EMPTY_RESULT_EXISTS_TRUE = Result.create(null, true);
> final static Result EMPTY_RESULT_EXISTS_FALSE = Result.create(null, false);
> private final static Result EMPTY_RESULT_STALE = 
> Result.create(EMPTY_CELL_ARRAY, null, true);
> {code}
> Result#advance 
> {code:java}
> public boolean advance() {
> if (cells == null) return false;
> cellScannerIndex++;
> if (cellScannerIndex < this.cells.length) {
>   return true;
> } else if (cellScannerIndex == this.cells.length) {
>   return false;
> }
> // The index of EMPTY_RESULT could be incremented by multi threads and 
> throw this exception.
> throw new NoSuchElementException("Cannot advance beyond the last cell");
>   }
> {code}
> Result#current
> {code:java}
> public Cell current() {
> if (cells == null
> || cellScannerIndex == INITIAL_CELLSCANNER_INDEX
> || cellScannerIndex >= cells.length)
>   return null;
>// Although it is almost impossible,
>// We can arrive here when the client threads share the common reused 
> EMPTY_RESULT.
> return this.cells[cellScannerIndex];
>   }
> {code}
> In this case, the client can easily got confusing exceptions even if they use 
> different connections, tables in different threads.
> We should change the if condition cells == null to isEmpty() to avoid the 
> client crashed from these exception.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (HBASE-26757) hbase thrift2 FilterString not working

[banshee (Jira)]

banshee created HBASE-26757:
---

 Summary: hbase thrift2 FilterString not working
 Key: HBASE-26757
 URL: https://issues.apache.org/jira/browse/HBASE-26757
 Project: HBase
  Issue Type: Bug
  Components: hbase-connectors
Affects Versions: 2.4.9
Reporter: banshee


I am trying to connect hbase by thrift2 golang client.

Everything works well but the filterString field. No matter what value I set 
(just simply convert from string to []byte) it doesn't work and return an empty 
result.

And I had tried the thrift service of the old version and it worked. Is there 
some hint trick to generate the filterString []byte in thrift2?



--
This message was sent by Atlassian Jira
(v8.20.1#820001)