[jira] [Reopened] (HBASE-23340) hmaster /hbase/replication/rs session expired (hbase replication default value is true, we don't use ) causes logcleaner can not clean oldWALs, which resulits in oldW
[ https://issues.apache.org/jira/browse/HBASE-23340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Somogyi reopened HBASE-23340: --- Reopening to backport to branch-2.4. This fix can also be useful on that branch. > hmaster /hbase/replication/rs session expired (hbase replication default > value is true, we don't use ) causes logcleaner can not clean oldWALs, which > resulits in oldWALs too large (more than 2TB) > - > > Key: HBASE-23340 > URL: https://issues.apache.org/jira/browse/HBASE-23340 > Project: HBase > Issue Type: Improvement > Components: master >Affects Versions: 3.0.0-alpha-1, 2.2.3 >Reporter: jackylau >Assignee: Bo Cui >Priority: Major > Fix For: 3.0.0-alpha-1, 2.5.0 > > Attachments: Snipaste_2019-11-21_10-39-25.png, > Snipaste_2019-11-21_14-10-36.png > > > hmaster /hbase/replication/rs session expired (hbase replication default > value is true, we don't use ) causes logcleaner can not clean oldWALs, which > resulits in oldWALs too large (more than 2TB). > !Snipaste_2019-11-21_10-39-25.png! > > !Snipaste_2019-11-21_14-10-36.png! > > we can solve it by following : > 1) increase the session timeout(but i think it is not a good idea. because we > do not know how long to set is suitable) > 2) close the hbase replication. It is not a good idea too, when our user uses > this feature > 3) we need add retry times, for example when it has already happened three > times, we set the ReplicationLogCleaner and SnapShotCleaner stop > that is all my ideas, i do not konw it is suitable, If it is suitable, could > i commit a PR? > Does anynode have a good idea. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HBASE-27554) Test failures on branch-2.4 with corrupted exclude list
[
https://issues.apache.org/jira/browse/HBASE-27554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Somogyi resolved HBASE-27554.
---
Resolution: Fixed
Recent nightly tests were successful on branch-2.4 after the cleanup.
> Test failures on branch-2.4 with corrupted exclude list
> ---
>
> Key: HBASE-27554
> URL: https://issues.apache.org/jira/browse/HBASE-27554
> Project: HBase
> Issue Type: Bug
> Components: jenkins
>Affects Versions: 2.4.16
> Reporter: Peter Somogyi
> Assignee: Peter Somogyi
>Priority: Major
>
> Nightly builds and PRs on branch-2.4 are failing with an invalid exclude list.
> Executed unit test command:
> {code:java}
> /opt/maven/bin/mvn --batch-mode
> -Dmaven.repo.local=/home/jenkins/jenkins-home/workspace/Base-PreCommit-GitHub-PR_PR-4944/yetus-m2/hbase-branch-2.4-patch-0
> --threads=4
> -Djava.io.tmpdir=/home/jenkins/jenkins-home/workspace/Base-PreCommit-GitHub-PR_PR-4944/yetus-jdk8-hadoop2-check/src/target
> -DHBasePatchProcess -PrunAllTests
> -Dtest.exclude.pattern=**/replication.regionserver.TestMetaRegionReplicaReplicationEndpoint.java,**/client.TestMetaRegionLocationCache.java,**/master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks.java,**/replication.TestZKReplicationQueueStorageWARNING:
> All illegal access operations will be denied in a future
> release.java,**/replication.regionserver.TestBasicWALEntryStreamFSHLog.java
> -Dsurefire.firstPartForkCount=0.5C -Dsurefire.secondPartForkCount=0.5C clean
> test -fae {code}
> The latest exclude list contains "WARNING: All illegal access operations will
> be denied in a future release" and maven treats this as a new parameter. As a
> result unit tests are failing on CI that rely on the exclude list.
> [https://ci-hbase.apache.org/job/HBase-Find-Flaky-Tests/job/branch-2.4/lastSuccessfulBuild/artifact/output/excludes/*view*/]
> {noformat}
> **/replication.regionserver.TestMetaRegionReplicaReplicationEndpoint.java,**/client.TestMetaRegionLocationCache.java,**/master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks.java,**/replication.TestZKReplicationQueueStorageWARNING:
> All illegal access operations will be denied in a future
> release.java,**/replication.regionserver.TestBasicWALEntryStreamFSHLog.java
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Reopened] (HBASE-26320) Separate Log Cleaner DirScanPool to prevent the OLDWALs from filling up the disk when archive is large
[ https://issues.apache.org/jira/browse/HBASE-26320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Somogyi reopened HBASE-26320: --- Reopening to backport to branch-2.4. > Separate Log Cleaner DirScanPool to prevent the OLDWALs from filling up the > disk when archive is large > -- > > Key: HBASE-26320 > URL: https://issues.apache.org/jira/browse/HBASE-26320 > Project: HBase > Issue Type: Improvement > Components: Operability >Affects Versions: 1.7.1, 2.4.6 >Reporter: Zach York >Assignee: Zach York >Priority: Major > Fix For: 2.5.0, 3.0.0-alpha-3 > > > We currently share the DirScanPool (threadpool for scanning for files to > delete in the OldLogs and archive directories) between the LogCleaner and > HFileCleaner. This means that if the archive directory is large/has lots of > files/directories, the threads can get stuck scanning through the archive > directory, starving the LogCleaner. This is especially apparent on S3 where > list can be slower than on HDFS. > This JIRA creates separate DirScanPools for the LogCleaner and HFileCleaner -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HBASE-27506) Optionally disable sorting directories by size in CleanerChore
Peter Somogyi created HBASE-27506: - Summary: Optionally disable sorting directories by size in CleanerChore Key: HBASE-27506 URL: https://issues.apache.org/jira/browse/HBASE-27506 Project: HBase Issue Type: Improvement Reporter: Peter Somogyi Assignee: Peter Somogyi The CleanerChore computes the size of the directories to start the cleaning with the largest directory first. This computation can take a long time over object stores when the archive directory is huge. Make it configurable to skip the directory sorting in the CleanerChore. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HBASE-27507) Add configuration for disabling CleanerChore
Peter Somogyi created HBASE-27507: - Summary: Add configuration for disabling CleanerChore Key: HBASE-27507 URL: https://issues.apache.org/jira/browse/HBASE-27507 Project: HBase Issue Type: Improvement Reporter: Peter Somogyi Assignee: Peter Somogyi The CleanerChore can be disabled from the shell but not from the configuration. The Cleaner will start with HBase Master it is not possible to prevent it from the initial start. A configuration property could be added to disable the CleanerChore. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HBASE-27506) Optionally disable sorting directories by size in CleanerChore
[ https://issues.apache.org/jira/browse/HBASE-27506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Somogyi resolved HBASE-27506. --- Fix Version/s: 2.6.0 3.0.0-alpha-4 2.4.16 2.5.3 Resolution: Fixed Pushed to branch-2.4+. Thanks for the review [~wchevreuil]! > Optionally disable sorting directories by size in CleanerChore > -- > > Key: HBASE-27506 > URL: https://issues.apache.org/jira/browse/HBASE-27506 > Project: HBase > Issue Type: Improvement > Reporter: Peter Somogyi > Assignee: Peter Somogyi >Priority: Minor > Fix For: 2.6.0, 3.0.0-alpha-4, 2.4.16, 2.5.3 > > > The CleanerChore computes the size of the directories to start the cleaning > with the largest directory first. This computation can take a long time over > object stores when the archive directory is huge. > Make it configurable to skip the directory sorting in the CleanerChore. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HBASE-27001) The deleted variable cannot be printed out
[
https://issues.apache.org/jira/browse/HBASE-27001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Somogyi resolved HBASE-27001.
---
Fix Version/s: 2.4.16
Resolution: Fixed
> The deleted variable cannot be printed out
> --
>
> Key: HBASE-27001
> URL: https://issues.apache.org/jira/browse/HBASE-27001
> Project: HBase
> Issue Type: Bug
>Reporter: selina.yan
>Assignee: selina.yan
>Priority: Minor
> Fix For: 2.4.16, 3.0.0-alpha-3, 2.5.0
>
>
> In the deleteAction method of CleanerChore class, the delete variable cannot
> be printed out
> {code:java}
> LOG.trace("Finish deleting {} under {}, deleted=", type, dir, deleted); {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (HBASE-26320) Separate Log Cleaner DirScanPool to prevent the OLDWALs from filling up the disk when archive is large
[ https://issues.apache.org/jira/browse/HBASE-26320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Somogyi resolved HBASE-26320. --- Fix Version/s: 2.4.16 Resolution: Fixed Merged to branch-2.4. Thanks [~zhangduo] for the review. > Separate Log Cleaner DirScanPool to prevent the OLDWALs from filling up the > disk when archive is large > -- > > Key: HBASE-26320 > URL: https://issues.apache.org/jira/browse/HBASE-26320 > Project: HBase > Issue Type: Improvement > Components: Operability >Affects Versions: 1.7.1, 2.4.6 >Reporter: Zach York >Assignee: Zach York >Priority: Major > Fix For: 2.4.16, 3.0.0-alpha-3, 2.5.0 > > > We currently share the DirScanPool (threadpool for scanning for files to > delete in the OldLogs and archive directories) between the LogCleaner and > HFileCleaner. This means that if the archive directory is large/has lots of > files/directories, the threads can get stuck scanning through the archive > directory, starving the LogCleaner. This is especially apparent on S3 where > list can be slower than on HDFS. > This JIRA creates separate DirScanPools for the LogCleaner and HFileCleaner -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HBASE-27493) Allow namespace admins to clone snapshots created by them
[ https://issues.apache.org/jira/browse/HBASE-27493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Somogyi resolved HBASE-27493. --- Fix Version/s: 2.6.0 3.0.0-alpha-4 Resolution: Fixed Merged to branch-2 and master. [~bszabolcs] can you fill the release notes? > Allow namespace admins to clone snapshots created by them > - > > Key: HBASE-27493 > URL: https://issues.apache.org/jira/browse/HBASE-27493 > Project: HBase > Issue Type: Improvement > Components: snapshots >Affects Versions: 3.0.0-alpha-3, 2.5.1 >Reporter: Szabolcs Bukros >Assignee: Szabolcs Bukros >Priority: Major > Fix For: 2.6.0, 3.0.0-alpha-4 > > > Creating a snapshot requires table admin permissions. But cloning it requires > global admin permissions unless the user owns the snapshot and wants to > recreate the original table the snapshot was based on using the same table > name. This puts unnecessary load on the few people having global admin > permissions. I would like to relax this rule a bit and allow the owner of the > snapshot to clone it into any namespace where they have admin permissions > regardless of the table name used. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HBASE-27590) Change Iterable to List in CleanerChore
Peter Somogyi created HBASE-27590:
-
Summary: Change Iterable to List in CleanerChore
Key: HBASE-27590
URL: https://issues.apache.org/jira/browse/HBASE-27590
Project: HBase
Issue Type: Improvement
Reporter: Peter Somogyi
Assignee: Peter Somogyi
The HFileCleaners can have low performance on large /archive area when used
with slow storage like S3. The snapshot write lock in SnapshotFileCache is held
while the file metadata is fetched from S3. Due to this even with multiple
cleaner threads only a single cleaner can effectively delete files from the
archive.
File metadata collection is performed before SnapshotHFileCleaner just by
changing the passed parameter type in FileCleanerDelegate from Iterable to List.
Running with the below cleaner configurations I observed that the lock held in
SnapshotFileCache went down from 45000ms to 100ms when it was running for 1000
files in a directory. The complete evaluation and deletion for this folder took
the same time but since the file metadata fetch from S3 was done outside of the
lock the multiple cleaner threads were able to run concurrently.
{noformat}
hbase.cleaner.directory.sorting=false
hbase.cleaner.scan.dir.concurrent.size=0.75
hbase.regionserver.hfilecleaner.small.thread.count=16
hbase.regionserver.hfilecleaner.large.thread.count=8
{noformat}
The files to evaluate are already passed in a List to
CleanerChore.checkAndDeleteFiles but it is converted to an Iterable to run the
checks on the configured cleaners.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (HBASE-27554) Test failures on branch-2.4 with corrupted exclude list
Peter Somogyi created HBASE-27554:
-
Summary: Test failures on branch-2.4 with corrupted exclude list
Key: HBASE-27554
URL: https://issues.apache.org/jira/browse/HBASE-27554
Project: HBase
Issue Type: Bug
Components: jenkins
Affects Versions: 2.4.16
Reporter: Peter Somogyi
Assignee: Peter Somogyi
Nightly builds and PRs on branch-2.4 are failing with an invalid exclude list.
Executed unit test command:
{code:java}
/opt/maven/bin/mvn --batch-mode
-Dmaven.repo.local=/home/jenkins/jenkins-home/workspace/Base-PreCommit-GitHub-PR_PR-4944/yetus-m2/hbase-branch-2.4-patch-0
--threads=4
-Djava.io.tmpdir=/home/jenkins/jenkins-home/workspace/Base-PreCommit-GitHub-PR_PR-4944/yetus-jdk8-hadoop2-check/src/target
-DHBasePatchProcess -PrunAllTests
-Dtest.exclude.pattern=**/replication.regionserver.TestMetaRegionReplicaReplicationEndpoint.java,**/client.TestMetaRegionLocationCache.java,**/master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks.java,**/replication.TestZKReplicationQueueStorageWARNING:
All illegal access operations will be denied in a future
release.java,**/replication.regionserver.TestBasicWALEntryStreamFSHLog.java
-Dsurefire.firstPartForkCount=0.5C -Dsurefire.secondPartForkCount=0.5C clean
test -fae {code}
The latest exclude list contains "WARNING: All illegal access operations will
be denied in a future release" and maven treats this as a new parameter. As a
result unit tests are failing on CI that rely on the exclude list.
[https://ci-hbase.apache.org/job/HBase-Find-Flaky-Tests/job/branch-2.4/lastSuccessfulBuild/artifact/output/excludes/*view*/]
{noformat}
**/replication.regionserver.TestMetaRegionReplicaReplicationEndpoint.java,**/client.TestMetaRegionLocationCache.java,**/master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks.java,**/replication.TestZKReplicationQueueStorageWARNING:
All illegal access operations will be denied in a future
release.java,**/replication.regionserver.TestBasicWALEntryStreamFSHLog.java
{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Reopened] (HBASE-27001) The deleted variable cannot be printed out
[
https://issues.apache.org/jira/browse/HBASE-27001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Somogyi reopened HBASE-27001:
---
Reopening for branch-2.4 backport.
> The deleted variable cannot be printed out
> --
>
> Key: HBASE-27001
> URL: https://issues.apache.org/jira/browse/HBASE-27001
> Project: HBase
> Issue Type: Bug
>Reporter: selina.yan
>Assignee: selina.yan
>Priority: Minor
> Fix For: 2.5.0, 3.0.0-alpha-3
>
>
> In the deleteAction method of CleanerChore class, the delete variable cannot
> be printed out
> {code:java}
> LOG.trace("Finish deleting {} under {}, deleted=", type, dir, deleted); {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (HBASE-22048) Incorrect email links on website
[ https://issues.apache.org/jira/browse/HBASE-22048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Somogyi resolved HBASE-22048. --- Assignee: (was: Rishabh Jain) Resolution: Done Resolved by upgrading maven-project-info-reports-plugin. > Incorrect email links on website > > > Key: HBASE-22048 > URL: https://issues.apache.org/jira/browse/HBASE-22048 > Project: HBase > Issue Type: Bug > Components: website > Reporter: Peter Somogyi >Priority: Minor > Labels: beginner > > Project members email addresses has incorrect link. > [https://hbase.apache.org/team.html] > Instead of [apach...@apache.org|mailto:apach...@apache.org] it points to > [https://hbase.apache.org/apach...@apache.org] > This change might be related to ASF parent pom upgrade which changed the > maven-project-info-reports-plugin version. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HBASE-27745) Document protoc workarounds with Apple Silicon
[ https://issues.apache.org/jira/browse/HBASE-27745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Somogyi resolved HBASE-27745. --- Fix Version/s: 3.0.0-alpha-4 Resolution: Fixed Merged to master. Thanks for the review [~ndimiduk]! > Document protoc workarounds with Apple Silicon > -- > > Key: HBASE-27745 > URL: https://issues.apache.org/jira/browse/HBASE-27745 > Project: HBase > Issue Type: Sub-task > Components: documentation > Reporter: Peter Somogyi > Assignee: Peter Somogyi >Priority: Major > Fix For: 3.0.0-alpha-4 > > > Building hbase 2.x on Apple Silicon is difficult because there is no protoc > library available. > [~ndimiduk] added a solution in HBASE-27741 to use osx-x86_64 but it is also > possible to build protoc locally and use that. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HBASE-27745) Document protoc workarounds with Apple Silicon
Peter Somogyi created HBASE-27745: - Summary: Document protoc workarounds with Apple Silicon Key: HBASE-27745 URL: https://issues.apache.org/jira/browse/HBASE-27745 Project: HBase Issue Type: Sub-task Components: documentation Reporter: Peter Somogyi Assignee: Peter Somogyi Building hbase 2.x on Apple Silicon is difficult because there is no protoc library available. [~ndimiduk] added a solution in HBASE-27741 to use osx-x86_64 but it is also possible to build protoc locally and use that. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HBASE-27701) ZStdCodec codec implementation class documentation typo
[
https://issues.apache.org/jira/browse/HBASE-27701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Somogyi resolved HBASE-27701.
---
Fix Version/s: 3.0.0-alpha-4
Resolution: Fixed
Thanks for the contribution, [~frensjan]!
> ZStdCodec codec implementation class documentation typo
> ---
>
> Key: HBASE-27701
> URL: https://issues.apache.org/jira/browse/HBASE-27701
> Project: HBase
> Issue Type: Bug
> Components: documentation
>Reporter: Frens Jan Rumph
>Assignee: Frens Jan Rumph
>Priority: Minor
> Fix For: 3.0.0-alpha-4
>
>
> As mentioned in the [u...@hbase.apache.org|mailto:u...@hbase.apache.org]
> mailing list I noticed a small typo in the documentation on compression for
> Zstd. The codec implementation class in the documentation is listed as
> {{org.apache.hadoop.hbase.io.compress.zstd.ZStdCodec}} while the actual class
> is written with a lower case S: {{ZStdCodec.}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (HBASE-27685) Enable code coverage reporting to SonarQube in HBase
[ https://issues.apache.org/jira/browse/HBASE-27685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Somogyi resolved HBASE-27685. --- Fix Version/s: 2.6.0 3.0.0-alpha-4 2.4.17 2.5.4 Resolution: Fixed Merged to branch-2.4+. Thanks [~dora.horvath]! > Enable code coverage reporting to SonarQube in HBase > > > Key: HBASE-27685 > URL: https://issues.apache.org/jira/browse/HBASE-27685 > Project: HBase > Issue Type: Task >Reporter: Dóra Horváth >Assignee: Dóra Horváth >Priority: Minor > Fix For: 2.6.0, 3.0.0-alpha-4, 2.4.17, 2.5.4 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HBASE-27665) Update checkstyle in hbase-operator-tools
[ https://issues.apache.org/jira/browse/HBASE-27665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Somogyi resolved HBASE-27665. --- Fix Version/s: hbase-operator-tools-1.3.0 Resolution: Fixed Merged to master. Thanks [~zhangduo] for reviewing! > Update checkstyle in hbase-operator-tools > - > > Key: HBASE-27665 > URL: https://issues.apache.org/jira/browse/HBASE-27665 > Project: HBase > Issue Type: Task > Components: hbase-operator-tools >Affects Versions: hbase-operator-tools-1.2.0 > Reporter: Peter Somogyi > Assignee: Peter Somogyi >Priority: Major > Fix For: hbase-operator-tools-1.3.0 > > > The checkstyle dependency is vulnerable in hbase-operator-tools. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HBASE-27679) Bump junit to 4.13.2 in hbase-connectors
[ https://issues.apache.org/jira/browse/HBASE-27679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Somogyi resolved HBASE-27679. --- Fix Version/s: hbase-connectors-1.1.0 Resolution: Fixed Merged to master. > Bump junit to 4.13.2 in hbase-connectors > > > Key: HBASE-27679 > URL: https://issues.apache.org/jira/browse/HBASE-27679 > Project: HBase > Issue Type: Task > Components: hbase-connectors > Reporter: Peter Somogyi > Assignee: Peter Somogyi >Priority: Major > Fix For: hbase-connectors-1.1.0 > > > Dependapot reported an issue with junit. Move to the version we have in hbase > main repository. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HBASE-27665) Update checkstyle in hbase-operator-tools
Peter Somogyi created HBASE-27665: - Summary: Update checkstyle in hbase-operator-tools Key: HBASE-27665 URL: https://issues.apache.org/jira/browse/HBASE-27665 Project: HBase Issue Type: Task Components: hbase-operator-tools Affects Versions: hbase-operator-tools-1.2.0 Reporter: Peter Somogyi Assignee: Peter Somogyi The checkstyle dependency is vulnerable in hbase-operator-tools. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HBASE-25610) Support multiple tables as input in generateMissingTableDescriptorFile command in HBCK2
[ https://issues.apache.org/jira/browse/HBASE-25610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Somogyi resolved HBASE-25610. --- Fix Version/s: hbase-operator-tools-1.3.0 Resolution: Fixed Merged to master. Thanks for the patch [~sanjeetnishad]! > Support multiple tables as input in generateMissingTableDescriptorFile > command in HBCK2 > --- > > Key: HBASE-25610 > URL: https://issues.apache.org/jira/browse/HBASE-25610 > Project: HBase > Issue Type: Improvement > Components: hbase-operator-tools, hbck2 >Reporter: Sanjeet Nishad >Assignee: Sanjeet Nishad >Priority: Minor > Fix For: hbase-operator-tools-1.3.0 > > > Currently 'generateMissingTableDescriptorFile' command in HBCK2 supports only > 1 TableName as input. HBCK's _fixOrphanTables()_ had a support for fixing the > missing .tableinfo files for a list of tables and it also supported fixing > the missing descriptor for all tables if no tables were specified. > It looks like a convinient enhancement to have in > generateMissingTableDescriptorFile command of HBCK2. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HBASE-27678) Update checkstyle in hbase-connectors
Peter Somogyi created HBASE-27678: - Summary: Update checkstyle in hbase-connectors Key: HBASE-27678 URL: https://issues.apache.org/jira/browse/HBASE-27678 Project: HBase Issue Type: Task Components: hbase-connectors Reporter: Peter Somogyi Assignee: Peter Somogyi There is a known CVE on the used checkstyle in hbase-connectors. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HBASE-27679) Bump junit to 4.13.2
Peter Somogyi created HBASE-27679: - Summary: Bump junit to 4.13.2 Key: HBASE-27679 URL: https://issues.apache.org/jira/browse/HBASE-27679 Project: HBase Issue Type: Task Components: hbase-connectors Reporter: Peter Somogyi Assignee: Peter Somogyi Dependapot reported an issue with junit. Move to the version we have in hbase main repository. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HBASE-24452) Try github ci for hbase-connectors
[ https://issues.apache.org/jira/browse/HBASE-24452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Somogyi resolved HBASE-24452. --- Resolution: Won't Do > Try github ci for hbase-connectors > -- > > Key: HBASE-24452 > URL: https://issues.apache.org/jira/browse/HBASE-24452 > Project: HBase > Issue Type: Improvement >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HBASE-27678) Update checkstyle in hbase-connectors
[ https://issues.apache.org/jira/browse/HBASE-27678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Somogyi resolved HBASE-27678. --- Fix Version/s: hbase-connectors-1.1.0 Resolution: Fixed Merged to master. > Update checkstyle in hbase-connectors > - > > Key: HBASE-27678 > URL: https://issues.apache.org/jira/browse/HBASE-27678 > Project: HBase > Issue Type: Task > Components: hbase-connectors > Reporter: Peter Somogyi > Assignee: Peter Somogyi >Priority: Major > Fix For: hbase-connectors-1.1.0 > > > There is a known CVE on the used checkstyle in hbase-connectors. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HBASE-27639) Support hbase-connectors compilation with HBase 2.5.3, Hadoop 3.2.4 and Spark 3.2.3
[ https://issues.apache.org/jira/browse/HBASE-27639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Somogyi resolved HBASE-27639. --- Fix Version/s: hbase-connectors-1.1.0 Resolution: Fixed Merged to master. Thanks for the patch [~nihaljain.cs]. > Support hbase-connectors compilation with HBase 2.5.3, Hadoop 3.2.4 and Spark > 3.2.3 > --- > > Key: HBASE-27639 > URL: https://issues.apache.org/jira/browse/HBASE-27639 > Project: HBase > Issue Type: Improvement > Components: hbase-connectors, spark >Reporter: Nihal Jain >Assignee: Nihal Jain >Priority: Major > Fix For: hbase-connectors-1.1.0 > > > Goal is to allow hbase-connectors to compile with: > * HBase: 2.5.3 > * Hadoop: 3.2.4 and > * Spark: 3.2.3 > We could also discuss if we want to bump the versions of the above mentioned > in the pom itself, > or just want to let spark connector compile with above components as the JIRA > title says. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HBASE-27801) Remove redundant avro.version property from Kafka connector
[ https://issues.apache.org/jira/browse/HBASE-27801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Somogyi resolved HBASE-27801. --- Fix Version/s: hbase-connectors-1.1.0 Resolution: Fixed Merged to master. Thanks [~stoty]! > Remove redundant avro.version property from Kafka connector > --- > > Key: HBASE-27801 > URL: https://issues.apache.org/jira/browse/HBASE-27801 > Project: HBase > Issue Type: Bug > Components: hbase-connectors, kafka >Affects Versions: connector-1.0.0 >Reporter: Istvan Toth >Assignee: Istvan Toth >Priority: Minor > Fix For: hbase-connectors-1.1.0 > > > 1.7.7 > is defined both in the main connectors pom, and in the kafka module. > This is not useful. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HBASE-27776) Backport HBASE-27731 (Upgrade commons-validator to version 1.7) to branch-2.5
[ https://issues.apache.org/jira/browse/HBASE-27776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Somogyi resolved HBASE-27776. --- Resolution: Invalid > Backport HBASE-27731 (Upgrade commons-validator to version 1.7) to branch-2.5 > - > > Key: HBASE-27776 > URL: https://issues.apache.org/jira/browse/HBASE-27776 > Project: HBase > Issue Type: Task >Reporter: Wes Schuitema >Assignee: Wes Schuitema >Priority: Minor > > This is so we also fix these CVEs for 2.5 > - [CVE-2014-0114|https://nvd.nist.gov/vuln/detail/cve-2014-0114] > - [CVE-2019-10086|https://nvd.nist.gov/vuln/detail/cve-2019-10086] > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Reopened] (HBASE-27776) Backport HBASE-27731 (Upgrade commons-validator to version 1.7) to branch-2.5
[ https://issues.apache.org/jira/browse/HBASE-27776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Somogyi reopened HBASE-27776: --- > Backport HBASE-27731 (Upgrade commons-validator to version 1.7) to branch-2.5 > - > > Key: HBASE-27776 > URL: https://issues.apache.org/jira/browse/HBASE-27776 > Project: HBase > Issue Type: Task >Reporter: Wes Schuitema >Assignee: Wes Schuitema >Priority: Minor > > This is so we also fix these CVEs for 2.5 > - [CVE-2014-0114|https://nvd.nist.gov/vuln/detail/cve-2014-0114] > - [CVE-2019-10086|https://nvd.nist.gov/vuln/detail/cve-2019-10086] > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HBASE-27624) Cannot Specify Namespace via the hbase.table Option in Spark Connector
[
https://issues.apache.org/jira/browse/HBASE-27624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Somogyi resolved HBASE-27624.
---
Fix Version/s: hbase-connectors-1.0.1
Resolution: Fixed
Pushed to master. Thanks [~stoty]!
> Cannot Specify Namespace via the hbase.table Option in Spark Connector
> --
>
> Key: HBASE-27624
> URL: https://issues.apache.org/jira/browse/HBASE-27624
> Project: HBase
> Issue Type: Bug
> Components: hbase-connectors, spark
>Affects Versions: 1.0.1
>Reporter: Istvan Toth
>Assignee: Istvan Toth
>Priority: Major
> Fix For: hbase-connectors-1.0.1
>
>
> When using the old mapping format and specifying the HBase table via the
> _hbase.table_ option, the connector passes the namespaced string to HBase,
> and we get
> {noformat}
> Caused by: java.lang.IllegalArgumentException: Illegal character code:58, <:>
> at 7. User-space table qualifiers may only contain 'alphanumeric characters'
> and digits: staplesHbaseNamespace:staplesHbaseTableName
> at
> org.apache.hadoop.hbase.TableName.isLegalTableQualifierName(TableName.java:187)
> at
> org.apache.hadoop.hbase.TableName.isLegalTableQualifierName(TableName.java:138)
> at org.apache.hadoop.hbase.TableName.(TableName.java:320)
> at
> org.apache.hadoop.hbase.TableName.createTableNameIfNecessary(TableName.java:354)
> at org.apache.hadoop.hbase.TableName.valueOf(TableName.java:484){noformat}
> This seems to be related to the changes in HBASE-24276
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (HBASE-27590) Change Iterable to List in SnapshotFileCache
[
https://issues.apache.org/jira/browse/HBASE-27590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Somogyi resolved HBASE-27590.
---
Fix Version/s: 2.6.0
3.0.0-alpha-4
2.5.4
Resolution: Fixed
Merged to branch-2.5, branch-2, and master. Will merge to branch-2.4 once
HBASE-27627 gets resolved.
Thanks for the suggestion and review [~zhangduo]!
> Change Iterable to List in SnapshotFileCache
>
>
> Key: HBASE-27590
> URL: https://issues.apache.org/jira/browse/HBASE-27590
> Project: HBase
> Issue Type: Improvement
> Reporter: Peter Somogyi
> Assignee: Peter Somogyi
>Priority: Minor
> Fix For: 2.6.0, 3.0.0-alpha-4, 2.5.4
>
> Attachments: flame-1.html
>
>
> The HFileCleaners can have low performance on large /archive area when used
> with slow storage like S3. The snapshot write lock in SnapshotFileCache is
> held while the file metadata is fetched from S3. Due to this even with
> multiple cleaner threads only a single cleaner can effectively delete files
> from the archive.
> File metadata collection is performed before SnapshotHFileCleaner just by
> changing the passed parameter type in FileCleanerDelegate from Iterable to
> List.
> Running with the below cleaner configurations I observed that the lock held
> in SnapshotFileCache went down from 45000ms to 100ms when it was running for
> 1000 files in a directory. The complete evaluation and deletion for this
> folder took the same time but since the file metadata fetch from S3 was done
> outside of the lock the multiple cleaner threads were able to run
> concurrently.
> {noformat}
> hbase.cleaner.directory.sorting=false
> hbase.cleaner.scan.dir.concurrent.size=0.75
> hbase.regionserver.hfilecleaner.small.thread.count=16
> hbase.regionserver.hfilecleaner.large.thread.count=8
> {noformat}
> The files to evaluate are already passed in a List to
> CleanerChore.checkAndDeleteFiles but it is converted to an Iterable to run
> the checks on the configured cleaners.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (HBASE-27627) Backport HBASE-25899 to branch-2.4
Peter Somogyi created HBASE-27627: - Summary: Backport HBASE-25899 to branch-2.4 Key: HBASE-27627 URL: https://issues.apache.org/jira/browse/HBASE-27627 Project: HBase Issue Type: Sub-task Reporter: Peter Somogyi Assignee: Peter Somogyi Backport HBASE-25899 to branch-2.4 as it can increase the speed of archive cleanup. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HBASE-27629) Backport HBASE-27043 to branch-2.4
Peter Somogyi created HBASE-27629: - Summary: Backport HBASE-27043 to branch-2.4 Key: HBASE-27629 URL: https://issues.apache.org/jira/browse/HBASE-27629 Project: HBase Issue Type: Sub-task Reporter: Peter Somogyi Assignee: Peter Somogyi -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HBASE-27628) Spotbugs fix in RELEASENOTES.md
Peter Somogyi created HBASE-27628:
-
Summary: Spotbugs fix in RELEASENOTES.md
Key: HBASE-27628
URL: https://issues.apache.org/jira/browse/HBASE-27628
Project: HBase
Issue Type: Bug
Affects Versions: 2.5.3, 2.4.16
Reporter: Peter Somogyi
Assignee: Peter Somogyi
There is a whitespace vilotation in RELEASENOTES.md on branch-2.4 and
branch-2.5 causing the pre-commit and nightly builds to fail.
{noformat}
[ERROR] Failed to execute goal
com.diffplug.spotless:spotless-maven-plugin:2.27.2:check (default-cli) on
project hbase: The following files had format violations:
[ERROR] RELEASENOTES.md
[ERROR] @@ -85,7 +85,7 @@
[ERROR]
[ERROR]
*·[HBASE-27529](https://issues.apache.org/jira/browse/HBASE-27529)·|·*Major*·|·**Provide·RS·coproc·ability·to·attach·WAL·extended·attributes·to·mutations·at·replication·sink**
[ERROR]
[ERROR]
-New·regionserver·coproc·endpoints·that·can·be·used·by·coproc·at·the·replication·sink·cluster·if·WAL·has·extended·attributes.·
[ERROR]
+New·regionserver·coproc·endpoints·that·can·be·used·by·coproc·at·the·replication·sink·cluster·if·WAL·has·extended·attributes.
[ERROR]
Using·the·new·endpoints,·WAL·extended·attributes·can·be·transferred·to·Mutation·attributes·at·the·replication·sink·cluster.
{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (HBASE-27627) Backport HBASE-25899 to branch-2.4
[ https://issues.apache.org/jira/browse/HBASE-27627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Somogyi resolved HBASE-27627. --- Fix Version/s: 2.4.17 Resolution: Fixed Merged backport to branch-2.4. Thanks for the review [~taklwu] ! > Backport HBASE-25899 to branch-2.4 > -- > > Key: HBASE-27627 > URL: https://issues.apache.org/jira/browse/HBASE-27627 > Project: HBase > Issue Type: Sub-task > Reporter: Peter Somogyi > Assignee: Peter Somogyi >Priority: Major > Fix For: 2.4.17 > > > Backport HBASE-25899 to branch-2.4 as it can increase the speed of archive > cleanup. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HBASE-27628) Spotless fix in RELEASENOTES.md
[
https://issues.apache.org/jira/browse/HBASE-27628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Somogyi resolved HBASE-27628.
---
Fix Version/s: 2.4.17
2.5.4
Resolution: Fixed
Merged trivial fix to branch-2.4 and branch-2.5.
> Spotless fix in RELEASENOTES.md
> ---
>
> Key: HBASE-27628
> URL: https://issues.apache.org/jira/browse/HBASE-27628
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.4.16, 2.5.3
> Reporter: Peter Somogyi
> Assignee: Peter Somogyi
>Priority: Trivial
> Fix For: 2.4.17, 2.5.4
>
>
> There is a whitespace vilotation in RELEASENOTES.md on branch-2.4 and
> branch-2.5 causing the pre-commit and nightly builds to fail.
> {noformat}
> [ERROR] Failed to execute goal
> com.diffplug.spotless:spotless-maven-plugin:2.27.2:check (default-cli) on
> project hbase: The following files had format violations:
> [ERROR] RELEASENOTES.md
> [ERROR] @@ -85,7 +85,7 @@
> [ERROR]
> [ERROR]
> *·[HBASE-27529](https://issues.apache.org/jira/browse/HBASE-27529)·|·*Major*·|·**Provide·RS·coproc·ability·to·attach·WAL·extended·attributes·to·mutations·at·replication·sink**
> [ERROR]
> [ERROR]
> -New·regionserver·coproc·endpoints·that·can·be·used·by·coproc·at·the·replication·sink·cluster·if·WAL·has·extended·attributes.·
> [ERROR]
> +New·regionserver·coproc·endpoints·that·can·be·used·by·coproc·at·the·replication·sink·cluster·if·WAL·has·extended·attributes.
> [ERROR]
> Using·the·new·endpoints,·WAL·extended·attributes·can·be·transferred·to·Mutation·attributes·at·the·replication·sink·cluster.
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (HBASE-27629) Backport HBASE-27043 to branch-2.4
[ https://issues.apache.org/jira/browse/HBASE-27629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Somogyi resolved HBASE-27629. --- Fix Version/s: 2.4.17 Resolution: Fixed Merged to branch-2.4. > Backport HBASE-27043 to branch-2.4 > -- > > Key: HBASE-27629 > URL: https://issues.apache.org/jira/browse/HBASE-27629 > Project: HBase > Issue Type: Sub-task > Reporter: Peter Somogyi > Assignee: Peter Somogyi >Priority: Major > Fix For: 2.4.17 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Reopened] (HBASE-27590) Change Iterable to List in SnapshotFileCache
[
https://issues.apache.org/jira/browse/HBASE-27590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Somogyi reopened HBASE-27590:
---
> Change Iterable to List in SnapshotFileCache
>
>
> Key: HBASE-27590
> URL: https://issues.apache.org/jira/browse/HBASE-27590
> Project: HBase
> Issue Type: Improvement
> Reporter: Peter Somogyi
> Assignee: Peter Somogyi
>Priority: Minor
> Fix For: 2.6.0, 3.0.0-alpha-4, 2.5.4
>
> Attachments: flame-1.html
>
>
> The HFileCleaners can have low performance on large /archive area when used
> with slow storage like S3. The snapshot write lock in SnapshotFileCache is
> held while the file metadata is fetched from S3. Due to this even with
> multiple cleaner threads only a single cleaner can effectively delete files
> from the archive.
> File metadata collection is performed before SnapshotHFileCleaner just by
> changing the passed parameter type in FileCleanerDelegate from Iterable to
> List.
> Running with the below cleaner configurations I observed that the lock held
> in SnapshotFileCache went down from 45000ms to 100ms when it was running for
> 1000 files in a directory. The complete evaluation and deletion for this
> folder took the same time but since the file metadata fetch from S3 was done
> outside of the lock the multiple cleaner threads were able to run
> concurrently.
> {noformat}
> hbase.cleaner.directory.sorting=false
> hbase.cleaner.scan.dir.concurrent.size=0.75
> hbase.regionserver.hfilecleaner.small.thread.count=16
> hbase.regionserver.hfilecleaner.large.thread.count=8
> {noformat}
> The files to evaluate are already passed in a List to
> CleanerChore.checkAndDeleteFiles but it is converted to an Iterable to run
> the checks on the configured cleaners.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (HBASE-27590) Change Iterable to List in SnapshotFileCache
[
https://issues.apache.org/jira/browse/HBASE-27590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Somogyi resolved HBASE-27590.
---
Resolution: Fixed
Cherry-picked to branch-2.4.
> Change Iterable to List in SnapshotFileCache
>
>
> Key: HBASE-27590
> URL: https://issues.apache.org/jira/browse/HBASE-27590
> Project: HBase
> Issue Type: Improvement
> Reporter: Peter Somogyi
> Assignee: Peter Somogyi
>Priority: Minor
> Fix For: 2.6.0, 3.0.0-alpha-4, 2.5.4
>
> Attachments: flame-1.html
>
>
> The HFileCleaners can have low performance on large /archive area when used
> with slow storage like S3. The snapshot write lock in SnapshotFileCache is
> held while the file metadata is fetched from S3. Due to this even with
> multiple cleaner threads only a single cleaner can effectively delete files
> from the archive.
> File metadata collection is performed before SnapshotHFileCleaner just by
> changing the passed parameter type in FileCleanerDelegate from Iterable to
> List.
> Running with the below cleaner configurations I observed that the lock held
> in SnapshotFileCache went down from 45000ms to 100ms when it was running for
> 1000 files in a directory. The complete evaluation and deletion for this
> folder took the same time but since the file metadata fetch from S3 was done
> outside of the lock the multiple cleaner threads were able to run
> concurrently.
> {noformat}
> hbase.cleaner.directory.sorting=false
> hbase.cleaner.scan.dir.concurrent.size=0.75
> hbase.regionserver.hfilecleaner.small.thread.count=16
> hbase.regionserver.hfilecleaner.large.thread.count=8
> {noformat}
> The files to evaluate are already passed in a List to
> CleanerChore.checkAndDeleteFiles but it is converted to an Iterable to run
> the checks on the configured cleaners.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (HBASE-27630) hbase-spark bulkload stage directory limited to hdfs only
[ https://issues.apache.org/jira/browse/HBASE-27630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Somogyi resolved HBASE-27630. --- Resolution: Fixed Merged to master branch in the hbase-connectors repository. Thanks for the patch [~sergey.soldatov]! > hbase-spark bulkload stage directory limited to hdfs only > - > > Key: HBASE-27630 > URL: https://issues.apache.org/jira/browse/HBASE-27630 > Project: HBase > Issue Type: Bug > Components: spark >Affects Versions: connector-1.0.0 >Reporter: Sergey Soldatov >Assignee: Sergey Soldatov >Priority: Major > Fix For: hbase-connectors-1.1.0 > > > It's impossible to set up the staging directory for bulkload operation in > spark-hbase connector to any other filesystem different from hdfs. That might > be a problem for deployments where hbase.rootdir points to cloud storage. In > this case, an additional copy task from hdfs to cloud storage would be > required before loading hfiles to hbase. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HBASE-27933) Stable version outdated on https://hbase.apache.org/downloads.html
[ https://issues.apache.org/jira/browse/HBASE-27933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Somogyi resolved HBASE-27933. --- Fix Version/s: 3.0.0-beta-1 Resolution: Fixed Merged to master branch. Thanks for the contribution [~dieterdp_ng]! > Stable version outdated on https://hbase.apache.org/downloads.html > -- > > Key: HBASE-27933 > URL: https://issues.apache.org/jira/browse/HBASE-27933 > Project: HBase > Issue Type: Task >Affects Versions: 2.5.4 >Reporter: Dieter De Paepe >Assignee: Dieter De Paepe >Priority: Minor > Fix For: 3.0.0-beta-1 > > > In HBASE-27849, the stable version of HBase was updated to 2.5.x. This > updated the [https://downloads.apache.org/hbase/] page. > However, the download page ([https://hbase.apache.org/downloads.html)] still > refers to 2.4.x as the stable version. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Reopened] (HBASE-27897) ConnectionImplementation#locateRegionInMeta should pause and retry when taking user region lock failed
[ https://issues.apache.org/jira/browse/HBASE-27897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Somogyi reopened HBASE-27897: --- An incorrect cherry-pick to branch-2.4 causes test failures. > ConnectionImplementation#locateRegionInMeta should pause and retry when > taking user region lock failed > -- > > Key: HBASE-27897 > URL: https://issues.apache.org/jira/browse/HBASE-27897 > Project: HBase > Issue Type: Improvement > Components: Client >Affects Versions: 2.4.17, 2.5.4 >Reporter: Xiaolin Ha >Assignee: Xiaolin Ha >Priority: Major > Fix For: 2.6.0, 2.4.18, 2.5.6 > > > It just throws exception and skips the pause and retry logic when > ConnectionImplementation#takeUserRegionLock fails. In some circumstances, no > pause and retry by outer logic will make next > ConnectionImplementation#takeUserRegionLock still fails, since all the > threads simultaneously grab the lock. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HBASE-27897) ConnectionImplementation#locateRegionInMeta should pause and retry when taking user region lock failed
[ https://issues.apache.org/jira/browse/HBASE-27897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Somogyi resolved HBASE-27897. --- Resolution: Fixed Merted the addendum to branch-2.4. Thanks for the review, [~zhangduo]! > ConnectionImplementation#locateRegionInMeta should pause and retry when > taking user region lock failed > -- > > Key: HBASE-27897 > URL: https://issues.apache.org/jira/browse/HBASE-27897 > Project: HBase > Issue Type: Improvement > Components: Client >Affects Versions: 2.4.17, 2.5.4 >Reporter: Xiaolin Ha >Assignee: Xiaolin Ha >Priority: Major > Fix For: 2.6.0, 2.4.18, 2.5.6 > > > It just throws exception and skips the pause and retry logic when > ConnectionImplementation#takeUserRegionLock fails. In some circumstances, no > pause and retry by outer logic will make next > ConnectionImplementation#takeUserRegionLock still fails, since all the > threads simultaneously grab the lock. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HBASE-27884) [hbase-filesystem] Use log4j2 instead of log4j for logging
Peter Somogyi created HBASE-27884: - Summary: [hbase-filesystem] Use log4j2 instead of log4j for logging Key: HBASE-27884 URL: https://issues.apache.org/jira/browse/HBASE-27884 Project: HBase Issue Type: Task Components: Filesystem Integration Reporter: Peter Somogyi Move to log4j2 in hbase-filesystem. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HBASE-27883) [hbase-connectors] Use log4j2 instead of log4j for logging
Peter Somogyi created HBASE-27883: - Summary: [hbase-connectors] Use log4j2 instead of log4j for logging Key: HBASE-27883 URL: https://issues.apache.org/jira/browse/HBASE-27883 Project: HBase Issue Type: Task Components: hbase-connectors Reporter: Peter Somogyi Fix For: hbase-connectors-1.1.0 Move to log4j2 in hbase-connectors. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HBASE-27992) Bump exec-maven-plugin to 3.1.0
[
https://issues.apache.org/jira/browse/HBASE-27992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Somogyi resolved HBASE-27992.
---
Fix Version/s: 2.6.0
2.4.18
2.5.6
3.0.0-beta-1
4.0.0-alpha-1
Resolution: Fixed
Merged to branch-2.4+. Thanks for the review [~zhangduo]!
> Bump exec-maven-plugin to 3.1.0
> ---
>
> Key: HBASE-27992
> URL: https://issues.apache.org/jira/browse/HBASE-27992
> Project: HBase
> Issue Type: Task
> Components: build
> Reporter: Peter Somogyi
> Assignee: Peter Somogyi
>Priority: Trivial
> Fix For: 2.6.0, 2.4.18, 2.5.6, 3.0.0-beta-1, 4.0.0-alpha-1
>
>
> I frequently see IOException in hbase-shaded-with-hadoop-check-invariants
> make-sure-validation-files-are-in-sync phase. I'm not sure what is the root
> cause but it worth to bump the plugin version to the latest.
> {noformat}
> [INFO] --- exec-maven-plugin:1.6.0:exec
> (make-sure-validation-files-are-in-sync) @
> hbase-shaded-with-hadoop-check-invariants ---
> [ERROR] Command execution failed.
> java.io.IOException: Stream closed
> at java.lang.ProcessBuilder$NullOutputStream.write
> (ProcessBuilder.java:433)
> at java.io.OutputStream.write (OutputStream.java:116)
> at java.io.BufferedOutputStream.flushBuffer (BufferedOutputStream.java:82)
> at java.io.BufferedOutputStream.flush (BufferedOutputStream.java:140)
> at java.io.FilterOutputStream.close (FilterOutputStream.java:158)
> at org.apache.commons.exec.DefaultExecutor.closeProcessStreams
> (DefaultExecutor.java:306)
> at org.apache.commons.exec.DefaultExecutor.executeInternal
> (DefaultExecutor.java:387)
> at org.apache.commons.exec.DefaultExecutor.execute
> (DefaultExecutor.java:166)
> at org.codehaus.mojo.exec.ExecMojo.executeCommandLine (ExecMojo.java:804)
> at org.codehaus.mojo.exec.ExecMojo.executeCommandLine (ExecMojo.java:751)
> at org.codehaus.mojo.exec.ExecMojo.execute (ExecMojo.java:313)
> at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo
> (DefaultBuildPluginManager.java:137)
> at org.apache.maven.lifecycle.internal.MojoExecutor.execute
> (MojoExecutor.java:210)
> at org.apache.maven.lifecycle.internal.MojoExecutor.execute
> (MojoExecutor.java:156)
> at org.apache.maven.lifecycle.internal.MojoExecutor.execute
> (MojoExecutor.java:148)
> at
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject
> (LifecycleModuleBuilder.java:117)
> at
> org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call
> (MultiThreadedBuilder.java:190)
> at
> org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call
> (MultiThreadedBuilder.java:186)
> at java.util.concurrent.FutureTask.run (FutureTask.java:266)
> at java.util.concurrent.Executors$RunnableAdapter.call
> (Executors.java:511)
> at java.util.concurrent.FutureTask.run (FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker
> (ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run
> (ThreadPoolExecutor.java:624)
> at java.lang.Thread.run (Thread.java:748)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (HBASE-27992) Bump exec-maven-plugin to 3.1.0
Peter Somogyi created HBASE-27992:
-
Summary: Bump exec-maven-plugin to 3.1.0
Key: HBASE-27992
URL: https://issues.apache.org/jira/browse/HBASE-27992
Project: HBase
Issue Type: Task
Components: build
Reporter: Peter Somogyi
Assignee: Peter Somogyi
I frequently see IOException in hbase-shaded-with-hadoop-check-invariants
make-sure-validation-files-are-in-sync phase. I'm not sure what is the root
cause but it worth to bump the plugin version to the latest.
{noformat}
[INFO] --- exec-maven-plugin:1.6.0:exec
(make-sure-validation-files-are-in-sync) @
hbase-shaded-with-hadoop-check-invariants ---
[ERROR] Command execution failed.
java.io.IOException: Stream closed
at java.lang.ProcessBuilder$NullOutputStream.write (ProcessBuilder.java:433)
at java.io.OutputStream.write (OutputStream.java:116)
at java.io.BufferedOutputStream.flushBuffer (BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.flush (BufferedOutputStream.java:140)
at java.io.FilterOutputStream.close (FilterOutputStream.java:158)
at org.apache.commons.exec.DefaultExecutor.closeProcessStreams
(DefaultExecutor.java:306)
at org.apache.commons.exec.DefaultExecutor.executeInternal
(DefaultExecutor.java:387)
at org.apache.commons.exec.DefaultExecutor.execute
(DefaultExecutor.java:166)
at org.codehaus.mojo.exec.ExecMojo.executeCommandLine (ExecMojo.java:804)
at org.codehaus.mojo.exec.ExecMojo.executeCommandLine (ExecMojo.java:751)
at org.codehaus.mojo.exec.ExecMojo.execute (ExecMojo.java:313)
at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo
(DefaultBuildPluginManager.java:137)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute
(MojoExecutor.java:210)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute
(MojoExecutor.java:156)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute
(MojoExecutor.java:148)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject
(LifecycleModuleBuilder.java:117)
at
org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call
(MultiThreadedBuilder.java:190)
at
org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call
(MultiThreadedBuilder.java:186)
at java.util.concurrent.FutureTask.run (FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call (Executors.java:511)
at java.util.concurrent.FutureTask.run (FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker
(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run
(ThreadPoolExecutor.java:624)
at java.lang.Thread.run (Thread.java:748)
{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (HBASE-27884) [hbase-filesystem] Use log4j2 instead of log4j for logging
[ https://issues.apache.org/jira/browse/HBASE-27884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Somogyi resolved HBASE-27884. --- Release Note: Moved logging to log4j2. Resolution: Fixed Merged to master branch. Thanks [~subrat.mishra] for contributing! > [hbase-filesystem] Use log4j2 instead of log4j for logging > -- > > Key: HBASE-27884 > URL: https://issues.apache.org/jira/browse/HBASE-27884 > Project: HBase > Issue Type: Task > Components: Filesystem Integration > Reporter: Peter Somogyi >Assignee: Subrat Mishra >Priority: Major > Fix For: hbase-filesystem-1.0.0-alpha2 > > > Move to log4j2 in hbase-filesystem. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HBASE-27883) [hbase-connectors] Use log4j2 instead of log4j for logging
[ https://issues.apache.org/jira/browse/HBASE-27883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Somogyi resolved HBASE-27883. --- Resolution: Fixed Merged. Thanks [~subrat.mishra]! > [hbase-connectors] Use log4j2 instead of log4j for logging > -- > > Key: HBASE-27883 > URL: https://issues.apache.org/jira/browse/HBASE-27883 > Project: HBase > Issue Type: Task > Components: hbase-connectors > Reporter: Peter Somogyi >Assignee: Subrat Mishra >Priority: Blocker > Fix For: hbase-connectors-1.1.0 > > > Move to log4j2 in hbase-connectors. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HBASE-27892) Report memstore on-heap and off-heap size as jmx metrics
[ https://issues.apache.org/jira/browse/HBASE-27892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Somogyi resolved HBASE-27892. --- Fix Version/s: (was: 2.4.18) Resolution: Fixed Reverted from branch-2.4. [~jingyu] please reopen this ticket if you want this on branch-2.4. > Report memstore on-heap and off-heap size as jmx metrics > > > Key: HBASE-27892 > URL: https://issues.apache.org/jira/browse/HBASE-27892 > Project: HBase > Issue Type: Improvement > Components: metrics >Reporter: Bryan Beaudreault >Assignee: Jing Yu >Priority: Major > Fix For: 2.6.0, 2.5.6, 3.0.0-beta-1 > > > Currently we only report "memStoreSize" metric in sub=RegionServer bean. I've > noticed a big discrepancy between this metric and the RS UI's "Memstore > On-Heap Size". It seems like "memStoreSize" is the overall data size, while > the on-heap size is coming from our heap estimation which includes POJO heap > overhead, etc. > I have a regionserver with only 750mb of "memStoreSize", but the on-heap size > is over 1gb. This is non-trivial for estimating overall heap size necessary > for a regionserver. Since we have the data, let's report it. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Reopened] (HBASE-27892) Report memstore on-heap and off-heap size as jmx metrics
[ https://issues.apache.org/jira/browse/HBASE-27892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Somogyi reopened HBASE-27892: --- This commit causes compilation failure on branch-2.4. Reopening for the revert. > Report memstore on-heap and off-heap size as jmx metrics > > > Key: HBASE-27892 > URL: https://issues.apache.org/jira/browse/HBASE-27892 > Project: HBase > Issue Type: Improvement > Components: metrics >Reporter: Bryan Beaudreault >Assignee: Jing Yu >Priority: Major > Fix For: 2.6.0, 2.4.18, 2.5.6, 3.0.0-beta-1 > > > Currently we only report "memStoreSize" metric in sub=RegionServer bean. I've > noticed a big discrepancy between this metric and the RS UI's "Memstore > On-Heap Size". It seems like "memStoreSize" is the overall data size, while > the on-heap size is coming from our heap estimation which includes POJO heap > overhead, etc. > I have a regionserver with only 750mb of "memStoreSize", but the on-heap size > is over 1gb. This is non-trivial for estimating overall heap size necessary > for a regionserver. Since we have the data, let's report it. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HBASE-28252) Add sun.net.dns and sun.net.util to the JDK11+ module exports in the hbase script
[ https://issues.apache.org/jira/browse/HBASE-28252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Somogyi resolved HBASE-28252. --- Resolution: Fixed Merged to branch-3. The commit was partially committed in HBASE-28259 earlier. > Add sun.net.dns and sun.net.util to the JDK11+ module exports in the hbase > script > - > > Key: HBASE-28252 > URL: https://issues.apache.org/jira/browse/HBASE-28252 > Project: HBase > Issue Type: Bug > Components: scripts >Reporter: Istvan Toth >Assignee: Istvan Toth >Priority: Major > Fix For: 2.6.0, 2.4.18, 3.0.0-beta-1, 2.5.7 > > > As noted in HBASE-28247, HBase can run into module permission issues that are > not handled in the current JDK11 options in the hbase startup script. > The surefire test config also includes some JDK17 specific options, we should > also add those as needed. > We are not yet aware of any additional JVM options required by Java 21. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Reopened] (HBASE-28247) Add java.base/sun.net.dns and java.base/sun.net.util export to jdk11 JVM test flags
[
https://issues.apache.org/jira/browse/HBASE-28247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Somogyi reopened HBASE-28247:
---
Reopening because this was not committed to branch-3 by accident.
> Add java.base/sun.net.dns and java.base/sun.net.util export to jdk11 JVM
> test flags
>
>
> Key: HBASE-28247
> URL: https://issues.apache.org/jira/browse/HBASE-28247
> Project: HBase
> Issue Type: Bug
> Components: java
>Affects Versions: 2.6.0, 3.0.0-alpha-4, 2.4.17, 2.5.6
>Reporter: Istvan Toth
>Assignee: Istvan Toth
>Priority: Minor
> Fix For: 2.6.0, 2.4.18, 3.0.0-beta-1, 2.5.7
>
>
> While testing with JDK17 we have found that we need to add
> {noformat}
> --add-exports java.base/sun.net.dns=ALL-UNNAMED
> --add-exports java.base/sun.net.util=ALL-UNNAMED
> {noformat}
> on top of what is already defined in _hbase-surefire.jdk11.flags_ , otherwise
> RS and Master startup fails in the Hadoop security code.
> While this does not affect the test suite (at least not the commonly run
> tests), I consider hbase-surefire.jdk11.flags to be an unoffical resource to
> getting HBase to run on newer JDK versions.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (HBASE-28247) Add java.base/sun.net.dns and java.base/sun.net.util export to jdk11 JVM test flags
[
https://issues.apache.org/jira/browse/HBASE-28247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Somogyi resolved HBASE-28247.
---
Resolution: Fixed
Merged to branch-3.
> Add java.base/sun.net.dns and java.base/sun.net.util export to jdk11 JVM
> test flags
>
>
> Key: HBASE-28247
> URL: https://issues.apache.org/jira/browse/HBASE-28247
> Project: HBase
> Issue Type: Bug
> Components: java
>Affects Versions: 2.6.0, 3.0.0-alpha-4, 2.4.17, 2.5.6
>Reporter: Istvan Toth
>Assignee: Istvan Toth
>Priority: Minor
> Fix For: 2.6.0, 2.4.18, 3.0.0-beta-1, 2.5.7
>
>
> While testing with JDK17 we have found that we need to add
> {noformat}
> --add-exports java.base/sun.net.dns=ALL-UNNAMED
> --add-exports java.base/sun.net.util=ALL-UNNAMED
> {noformat}
> on top of what is already defined in _hbase-surefire.jdk11.flags_ , otherwise
> RS and Master startup fails in the Hadoop security code.
> While this does not affect the test suite (at least not the commonly run
> tests), I consider hbase-surefire.jdk11.flags to be an unoffical resource to
> getting HBase to run on newer JDK versions.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Reopened] (HBASE-28252) Add sun.net.dns and sun.net.util to the JDK11+ module exports in the hbase script
[ https://issues.apache.org/jira/browse/HBASE-28252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Somogyi reopened HBASE-28252: --- Reopening because the commit did not land on branch-3 by accident. > Add sun.net.dns and sun.net.util to the JDK11+ module exports in the hbase > script > - > > Key: HBASE-28252 > URL: https://issues.apache.org/jira/browse/HBASE-28252 > Project: HBase > Issue Type: Bug > Components: scripts >Reporter: Istvan Toth >Assignee: Istvan Toth >Priority: Major > Fix For: 2.6.0, 2.4.18, 3.0.0-beta-1, 2.5.7 > > > As noted in HBASE-28247, HBase can run into module permission issues that are > not handled in the current JDK11 options in the hbase startup script. > The surefire test config also includes some JDK17 specific options, we should > also add those as needed. > We are not yet aware of any additional JVM options required by Java 21. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HBASE-28517) Make properties dynamically configured
[ https://issues.apache.org/jira/browse/HBASE-28517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Somogyi resolved HBASE-28517. --- Fix Version/s: 2.6.0 2.4.18 4.0.0-alpha-1 3.0.0-beta-2 2.5.9 Release Note: Make the following properties dynamically configured: * hbase.rs.evictblocksonclose * hbase.rs.cacheblocksonwrite * hbase.block.data.cacheonread Resolution: Fixed Merged to all active branches. Thanks, [~kabhishek4] for the contribution! > Make properties dynamically configured > -- > > Key: HBASE-28517 > URL: https://issues.apache.org/jira/browse/HBASE-28517 > Project: HBase > Issue Type: Improvement >Reporter: Abhishek Kothalikar >Assignee: Abhishek Kothalikar >Priority: Major > Labels: pull-request-available > Fix For: 2.6.0, 2.4.18, 4.0.0-alpha-1, 3.0.0-beta-2, 2.5.9 > > > Make following properties dynamically configured, > hbase.rs.evictblocksonclose > hbase.rs.cacheblocksonwrite > hbase.block.data.cacheonread > for use case scenarios where configuring them dynamically would help in > achieving better throughput. -- This message was sent by Atlassian Jira (v8.20.10#820010)
