[jira] [Created] (HBASE-26342) cleaner supports custom paths of independent configuration and pool

[Xiaolin Ha (Jira)]

Xiaolin Ha created HBASE-26342:
--

 Summary: cleaner supports custom paths of independent 
configuration and pool
 Key: HBASE-26342
 URL: https://issues.apache.org/jira/browse/HBASE-26342
 Project: HBase
  Issue Type: Improvement
  Components: master
Affects Versions: 2.0.0, 3.0.0-alpha-2
Reporter: Xiaolin Ha
Assignee: Xiaolin Ha


With this, we can clean some paths more quickly.

We found in our cluster, when the very huge table with thousands of regions and 
high write throughputs and many snapshots tables on the same cluster, the speed 
of delete files in  archive path will lower than the speed of moved in files by 
compaction. Then archive may remains PB level data. 

The bottleneck is in cleaner but not in the thread pool size or queue size. It 
is because there is synchronized lock in SnapshotFileCache, and a batch of 
files need once SnapshotFileCache#refreshCache(), which look through all the 
snapshot dirs.

The speed of clear a path without the SnapshotHFileCleaner is thirty times 
faster.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (HBASE-26186) jenkins script for caching artifacts should verify cached file before relying on it

[Duo Zhang (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-26186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Duo Zhang resolved HBASE-26186.
---
Fix Version/s: 2.4.8
   2.3.7
   3.0.0-alpha-2
   2.5.0
 Hadoop Flags: Reviewed
   Resolution: Fixed

Pushed to branch-2.3+.

We do not have the script on branch-1 so skip cherry-picking.

Thanks [~busbey].

> jenkins script for caching artifacts should verify cached file before relying 
> on it
> ---
>
> Key: HBASE-26186
> URL: https://issues.apache.org/jira/browse/HBASE-26186
> Project: HBase
>  Issue Type: Task
>  Components: build, integration tests
>Affects Versions: 2.5.0
>Reporter: Nick Dimiduk
>Assignee: Sean Busbey
>Priority: Major
> Fix For: 2.5.0, 3.0.0-alpha-2, 2.3.7, 2.4.8
>
>
> Nightly builds of branch-2 have been failing on the "Packaging and 
> Integration" stage since 2021-07-22, commit 
> 8bc180c7737bbd1792851004625c552896bf57c6.
> The first failing build was 
> https://ci-hadoop.apache.org/blue/organizations/jenkins/HBase%2FHBase%20Nightly/detail/branch-2/304/pipeline/12



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HBASE-26341) Upload dashboard html for flaky find job to nightlies

[Duo Zhang (Jira)]

Duo Zhang created HBASE-26341:
-

 Summary: Upload dashboard html for flaky find job to nightlies
 Key: HBASE-26341
 URL: https://issues.apache.org/jira/browse/HBASE-26341
 Project: HBase
  Issue Type: Sub-task
  Components: flakies, jenkins, scripts
Reporter: Duo Zhang


This is an example

https://nightlies.apache.org/hbase/flaky.html

The nightlies server does not have strict CSP rules so it could display the 
html correctly.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

The nightly jobs for branch-1.x are failing

[Duo Zhang]

They all fail with docker image building failure.

10:50:04  Step 19/32 : RUN mkdir -p /opt/findbugs && curl -L -s -S
>
> https://sourceforge.net/projects/findbugs/files/findbugs/3.0.1/findbugs-noUpdateChecks-3.0.1.tar.gz/download
>  -o /opt/findbugs.tar.gz && tar xzf /opt/findbugs.tar.gz
> --strip-components 1 -C /opt/findbugs
> 10:50:05   ---> Running in d3ff60cd19b4
> 10:50:05   [91mcurl: (60) SSL certific [0m [91mate problem: certificate
> has expired
> 10:50:05  More details here: http://curl.haxx.se/docs/sslcerts.html
> 10:50:05
> 10:50:05  curl performs SSL certificate verification by default, using a
> "bundle"
> 10:50:05   of Certificate Authority (CA) public keys (CA certs). If the
> default
> 10:50:05   bundle file isn't adequate, you can specify [0m [91man
> alternate file
> 10:50:05   using the --cacert option.
> 10:50:05  If this HTTPS server uses a certificate signed by a CA
> represented in
> 10:50:05   the bundle, the certificate verification probably [0m [91m
> failed due to a
> 10:50:05   problem with the certificate (it might be expired, or the name
> might
> 10:50:05   not match the domain name in the U [0m [91mRL).
> 10:50:05  If you'd like to turn off curl's verification of the
> certificate, use
> 10:50:05   the -k (or --insecure) opt [0m [91mion.
> 10:50:06   [0mThe command '/bin/sh -c mkdir -p /opt/findbugs && curl
> -L -s -S
> https://sourceforge.net/projects/findbugs/files/findbugs/3.0.1/findbugs-noUpdateChecks-3.0.1.tar.gz/download
>  -o /opt/findbugs.tar.gz && tar xzf /opt/findbugs.tar.gz
> --strip-components 1 -C /opt/findbugs' returned a non-zero code: 60
> 10:50:06  ERROR: Docker failed to build yetus/hbase:633d966bfe.
>

But I tried the curl command locally and it could download the tarball
successfully.

Not sure what the problem is, did anyone face this problem before?

Thanks.

[jira] [Resolved] (HBASE-26312) Shell scan fails with timestamp

[Duo Zhang (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-26312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Duo Zhang resolved HBASE-26312.
---
Fix Version/s: 2.4.8
   3.0.0-alpha-2
   2.5.0
 Hadoop Flags: Reviewed
   Resolution: Fixed

Pushed to branch-2.4+.

Thanks [~xichaomin] for contributing.

> Shell scan fails with timestamp
> ---
>
> Key: HBASE-26312
> URL: https://issues.apache.org/jira/browse/HBASE-26312
> Project: HBase
>  Issue Type: Bug
>  Components: shell, test
>Reporter: xichaomin
>Assignee: xichaomin
>Priority: Major
> Fix For: 2.5.0, 3.0.0-alpha-2, 2.4.8
>
> Attachments: scan_setTimestamp.patch
>
>
> for removal of _setTimestamp_ in _Scan.java， scan with timestamp doesn't 
> work。_



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (HBASE-26339) SshPublisher will skip uploading artifacts if the build is failure

[Duo Zhang (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-26339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Duo Zhang resolved HBASE-26339.
---
Hadoop Flags: Reviewed
  Resolution: Fixed

Pushed to all active branches.

Thanks all for reviewing.

> SshPublisher will skip uploading artifacts if the build is failure
> --
>
> Key: HBASE-26339
> URL: https://issues.apache.org/jira/browse/HBASE-26339
> Project: HBase
>  Issue Type: Sub-task
>  Components: jenkins, scripts
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 1.4.14, 2.5.0, 3.0.0-alpha-2, 1.7.2, 2.3.7, 2.4.8
>
>
> This is not what we expect, need to force it always try to upload the 
> artifacts.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] use of Apache Yetus Audience Annotations

[Nick Dimiduk]

On the one hand, I agree that it's nice to have a common pattern and point
of reference across hadoop projects. On the other hand, we have deviated
from the meanings of the annotations, and so the javadocs on the yetus
classes do not match our usage. It's also strange and kind of a code smell
that we have another project namespace in our public API.

If we want to continue using the Yetus annotations, I can volunteer some
time to their care and feeding for JDK9+.

On Mon, Sep 27, 2021 at 9:13 AM Sean Busbey  wrote:

> Hi!
>
> Heads up that a discussion has started in Apache Yetus about dropping
> the Audience Annotations and associated javadoc tooling due to lack of
> community support[1]. The current cutting issue AFAICT is that things
> there haven't been updated for the changes in how doclets are handled
> in JDK9+.
>
> We still center all of our API scoping on this library. In general it
> has been solid and required relatively little investment on our part.
>
> Personally, I think this has worked really well for us so far and we
> ought to try to keep the shared resource going. Unfortunately, I don't
> currently have the cycles to personally step up in the Yetus project.
>
> What do folks think? Anyone able to help out in Yetus? Should we start
> moving to maintain this tooling internal to HBase?
>
> [1]:
> https://s.apache.org/ybdl6
> "[DISCUSS] Drop JDK8; audience-annotations" from d...@yetus.apache.org
>

Re: [DISCUSS] EOL 2.3

[Nick Dimiduk]

There is an existing DISCUSS thread re: moving the stable pointer,
mentioned here in an earlier post.
https://lists.apache.org/thread.html/r1cc4528a6a35cd1b0d38398aa61ad642a368901795d6970544d0a0a9%40%3Cdev.hbase.apache.org%3E

On Fri, Oct 8, 2021 at 9:07 AM Sean Busbey  wrote:

> please start a dedicated DISCUSS thread about moving the stable pointer.
>
> On Thu, Oct 7, 2021 at 11:33 PM Yu Li  wrote:
>
> > Hi all,
> >
> > Since 2.4.6 has been released on Sep. 13th and we plan to EOL 2.3.x,
> shall
> > we move the stable pointer to 2.4 (it's still on 2.3.6 for now on our
> > download page [1])? Or any concerns? Thanks.
> >
> > Best Regards,
> > Yu
> >
> > [1] https://hbase.apache.org/downloads.html
> >
> >
> > On Tue, 3 Aug 2021 at 06:01, Andrew Purtell 
> > wrote:
> >
> > > I’ve been working with 2.4 for months and it is stable enough for me to
> > +1
> > > moving the pointer at this time. This work includes many tests with 2.4
> > > using ITBLL with slowDeterministic and serverKilling policies and there
> > has
> > > not been an unstable result, as defined by crashes/loss of daemons by
> the
> > > IT framework, or data issues requiring hbck.
> > >
> > > > On Aug 2, 2021, at 2:27 PM, Nick Dimiduk 
> wrote:
> > > >
> > > > On Mon, Aug 2, 2021 at 1:22 PM Andrew Purtell <
> > andrew.purt...@gmail.com
> > > >
> > > > wrote:
> > > >
> > > >> EOL of 2.3 seems premature.
> > > >>
> > > >> The stable pointer is still pointing to 2.3.
> > > >>
> > > >> Advancing the stable pointer is an obvious prerequisite. If there
> are
> > > any
> > > >> concerns blocking advancing the pointer to 2.4, this is where we
> > should
> > > >> start the discussion.
> > > >>
> > > >
> > > > Point taken. I see the only conclusion of our thread [0] was to agree
> > > that
> > > > we should define "stable" criteria. I thought we'd already advanced
> the
> > > > marker.
> > > >
> > > > [0]:
> > > >
> > >
> >
> https://lists.apache.org/thread.html/r1cc4528a6a35cd1b0d38398aa61ad642a368901795d6970544d0a0a9%40%3Cdev.hbase.apache.org%3E
> > > >
> > > >>> On Aug 2, 2021, at 11:23 AM, Nick Dimiduk 
> > wrote:
> > > >>>
> > > >>> Heya,
> > > >>>
> > > >>> I'd like to start a discussion regarding the declaration of the end
> > of
> > > >> life
> > > >>> for branch-2.3. This release line has matured nicely, but with the
> > > stable
> > > >>> pointer moving forward, and our community interest in pushing on
> 2.5
> > > and
> > > >>> 3.0, I think it's worth considering whether we want to continue
> > putting
> > > >>> resources into this release line.
> > > >>>
> > > >>> I propose we make one final release, 2.3.7, whenever we have enough
> > > >> commits
> > > >>> to the branch, or the beginning of October, whichever comes first.
> > > >>>
> > > >>> Thoughts?
> > > >>>
> > > >>> Thanks,
> > > >>> Nick
> > > >>
> > >
> >
>

About time for 2.4.7

[Andrew Purtell]

The release of 2.4.6 was completed four weeks ago. It's time for 2.4.7. I
will spin the bits today. Let's vote next week.

-- 
Best regards,
Andrew

Words like orphans lost among the crosstalk, meaning torn from truth's
decrepit hands
   - A23, Crosstalk

Re: [hbase-connectors] - release to include Spark 3

[Sean Busbey]

Hi Jordan!

How do you currently pull in the dependency? Do you need us to publish an
artifact to maven central? Would a convenience binary built against spark-3
on downloads.apache.org suffice?

On Thu, Oct 7, 2021 at 7:55 PM Jordan Hambleton
 wrote:

> Hi Peter,
>
> We're seeing an uptick in usage with Spark 3. While hadoop-connectors have
> started supporting Spark 3 (HBASE-25326
> ) and while it's fine
> pulling down the repo and building from source, it would be a lot easier if
> we had a release for supporting Spark 3.
>
> Is it possible to get a next release of the hadoop-connectors out or is
> there another way others are using for Spark3 integration with HBase?
>
> Below is what I'm referring to in building.
>
> To build the connector with Spark 3.0, compile it with scala 2.12.
> Additional configurations that you can customize are the Spark version,
> HBase version, and Hadoop version. Example:
>
> $ mvn -Dspark.version=3.0.1 -Dscala.version=2.12.10
> -Dscala.binary.version=2.12 -Dhbase.version=2.2.4 -Dhadoop.profile=3.0
> -Dhadoop-three.version=3.2.0 -DskipTests clean package
>
> best regards,
> Jordan Hambleton
>

Re: [DISCUSS] EOL 2.3

[Sean Busbey]

please start a dedicated DISCUSS thread about moving the stable pointer.

On Thu, Oct 7, 2021 at 11:33 PM Yu Li  wrote:

> Hi all,
>
> Since 2.4.6 has been released on Sep. 13th and we plan to EOL 2.3.x, shall
> we move the stable pointer to 2.4 (it's still on 2.3.6 for now on our
> download page [1])? Or any concerns? Thanks.
>
> Best Regards,
> Yu
>
> [1] https://hbase.apache.org/downloads.html
>
>
> On Tue, 3 Aug 2021 at 06:01, Andrew Purtell 
> wrote:
>
> > I’ve been working with 2.4 for months and it is stable enough for me to
> +1
> > moving the pointer at this time. This work includes many tests with 2.4
> > using ITBLL with slowDeterministic and serverKilling policies and there
> has
> > not been an unstable result, as defined by crashes/loss of daemons by the
> > IT framework, or data issues requiring hbck.
> >
> > > On Aug 2, 2021, at 2:27 PM, Nick Dimiduk  wrote:
> > >
> > > On Mon, Aug 2, 2021 at 1:22 PM Andrew Purtell <
> andrew.purt...@gmail.com
> > >
> > > wrote:
> > >
> > >> EOL of 2.3 seems premature.
> > >>
> > >> The stable pointer is still pointing to 2.3.
> > >>
> > >> Advancing the stable pointer is an obvious prerequisite. If there are
> > any
> > >> concerns blocking advancing the pointer to 2.4, this is where we
> should
> > >> start the discussion.
> > >>
> > >
> > > Point taken. I see the only conclusion of our thread [0] was to agree
> > that
> > > we should define "stable" criteria. I thought we'd already advanced the
> > > marker.
> > >
> > > [0]:
> > >
> >
> https://lists.apache.org/thread.html/r1cc4528a6a35cd1b0d38398aa61ad642a368901795d6970544d0a0a9%40%3Cdev.hbase.apache.org%3E
> > >
> > >>> On Aug 2, 2021, at 11:23 AM, Nick Dimiduk 
> wrote:
> > >>>
> > >>> Heya,
> > >>>
> > >>> I'd like to start a discussion regarding the declaration of the end
> of
> > >> life
> > >>> for branch-2.3. This release line has matured nicely, but with the
> > stable
> > >>> pointer moving forward, and our community interest in pushing on 2.5
> > and
> > >>> 3.0, I think it's worth considering whether we want to continue
> putting
> > >>> resources into this release line.
> > >>>
> > >>> I propose we make one final release, 2.3.7, whenever we have enough
> > >> commits
> > >>> to the branch, or the beginning of October, whichever comes first.
> > >>>
> > >>> Thoughts?
> > >>>
> > >>> Thanks,
> > >>> Nick
> > >>
> >
>

[jira] [Resolved] (HBASE-26334) Upgrade commons-io to 2.11.0 in hbase-connectors

[Peter Somogyi (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-26334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Peter Somogyi resolved HBASE-26334.
---
Release Note: Upgraded commons-io to 2.11.0.  (was: Upgraded commons-io to 
2.8.0.)
  Resolution: Fixed

Merged to master.

> Upgrade commons-io to 2.11.0 in hbase-connectors
> 
>
> Key: HBASE-26334
> URL: https://issues.apache.org/jira/browse/HBASE-26334
> Project: HBase
>  Issue Type: Task
>  Components: hbase-connectors
>Reporter: Peter Somogyi
>Assignee: Peter Somogyi
>Priority: Major
> Fix For: hbase-connectors-1.0.1
>
>
> Upgrade commons-io in hbase-connectors to 2.11.0.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Reopened] (HBASE-26329) Upgrade commons-io to 2.11.0

[Peter Somogyi (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-26329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Peter Somogyi reopened HBASE-26329:
---

Reopening to backport to branch-2.4 and branch-2.3. Backport will include code 
changes from HBASE-25451.

> Upgrade commons-io to 2.11.0
> 
>
> Key: HBASE-26329
> URL: https://issues.apache.org/jira/browse/HBASE-26329
> Project: HBase
>  Issue Type: Task
>  Components: dependencies
>Reporter: Peter Somogyi
>Assignee: Peter Somogyi
>Priority: Major
> Fix For: 2.5.0, 3.0.0-alpha-2
>
>
> The 2.11.0 version got released from commons-io.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HBASE-26340) TableSplit returns false size under 1MB

[Jira]

Norbert Kalmár created HBASE-26340:
--

 Summary: TableSplit returns false size under 1MB
 Key: HBASE-26340
 URL: https://issues.apache.org/jira/browse/HBASE-26340
 Project: HBase
  Issue Type: Bug
Reporter: Norbert Kalmár


We calculate region size in the mapreduce package by getting the size in MB 
first and multiplying: 
https://github.com/apache/hbase/blob/39a20c528e2bf27cedf12734dbdb1b7b1e538076/hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/RegionSizeCalculator.java#L87

This will give a size of 0 until at least 1MB is reached. (And it will have an 
unwanted rounding affect as well). 
Spark for example can be tuned to do some performance tuning by eliminating the 
0 sized regions. This will eliminate any small regions which are not actually 
empty. The hadoop interface states the size is returned in bytes, and while 
this is true do to the multiplication, we multiply by 0 until 1MB is reached. 
I'm not sure why we get the size in MB units and not in bytes straight up. 
Should we fix this?



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Blog post series on "Evolution of Region assignment in HBase architecture"

[Viraj Jasani]

We have the "Part 3" of the blog series published.
Thanks to the co-writers: Duo Zhang and Andrew Purtell.

Part 3:
https://engineering.salesforce.com/evolution-of-region-assignment-in-the-apache-hbase-architecture-part-3-e03b814ae92

On Mon, Sep 13, 2021 at 10:53 PM Viraj Jasani  wrote:

> Thanks Duo for your offer to coordinate on writing "Part 3" of this
> series, sounds great!
> Although I see TRSP#assign being used by SCP directly while assigning the
> regions, I am yet to take a detailed look into HBASE-20881
>  and the relevant
> work. Let me reach out to you over Slack and we can take it from there.
>
> On Sun, Sep 12, 2021 at 7:02 PM 张铎(Duo Zhang) 
> wrote:
>
>> Thank you Viraj and Andrew, the blog posts are outstanding!
>>
>> And I think we'd better have a part 3, about the ServerCrashProcedure(SCP)
>> :)
>>
>> In 2.0 and 2.1, we use MoveRegionProcedure, AssignRegionProcedure and
>> UnassignRegionProcedure, and one of the reasons why we removed them all
>> and
>> introduced a single TRSP to do assign/unassign/move/reopen, is because of
>> SCP.
>>
>> If a region server crashed, obviously, we can not assign regions to it any
>> more, so we should have a way to stop the procedure which are still trying
>> to assign regions to the dead server. And even for unassigning a region,
>> we
>> still need to make it online first and then unassign it. For example, when
>> disabling a table, we must make sure that all the data in memstore have
>> been flushed to storage, so we will need make it online, and then do a
>> clean close.
>> In 2.0 and 2.1, we had 3 procedures for region assignment, and there were
>> lots of corner cases when we want to interrupt them from SCP, which made
>> the code really hard to understand and buggy. So finally, we introduced a
>> TRSP to replace them all. So SCP only needs to interrupt one type of
>> procedure.
>>
>> This is the story :)
>>
>> I could help if you guys want to write the part 3 about SCP :)
>>
>> Thanks.
>>
>> Viraj Jasani  于2021年9月8日周三 上午2:27写道：
>>
>> > As some of the HBase users are still running HBase 1.x versions in their
>> > production environment, and branch-1 is trending toward EOL, now is
>> really
>> > the right time to evaluate as well as understand the features and core
>> > design changes provided by HBase 2.x versions.
>> >
>> > As the majority of us are already aware, one of the key features with
>> > significant architectural changes provided by HBase 2 is
>> > AssignmentManagerV2 (AMv2).
>> > However, we don't seem to have one place explaining 1) *the evolution
>> > of AM* and
>> > 2) how it manages region assignments with better scalability,
>> reliability
>> > and fault-tolerance.
>> > Keeping this in mind, Andrew and I have published a series of two-part
>> blog
>> > posts explaining this evolution. Part 1 provides a) some basic
>> introduction
>> > to HBase concepts, and b) AM and it's shortcomings from previous
>> versions
>> > that AMv2 is trying to resolve. Part 2 provides detailed info about Pv2
>> and
>> > how AMv2 leverages it, and also state diagrams explaining some of the
>> > complex region assignment workflows. The intention of state diagrams is
>> for
>> > dev/users to be able to a) understand region assignment workflows
>> in-depth,
>> > b) easier code walk-through and c) debug and root cause issues with
>> > better knowledge.
>> >
>> > Part 1:
>> >
>> >
>> https://engineering.salesforce.com/evolution-of-region-assignment-in-the-apache-hbase-architecture-part-1-c43b1becc522
>> > Part 2:
>> >
>> >
>> https://engineering.salesforce.com/evolution-of-region-assignment-in-the-apache-hbase-architecture-part-2-9568fb3790b
>> >
>>
>