[jira] [Resolved] (HBASE-28517) Make properties dynamically configured
[ https://issues.apache.org/jira/browse/HBASE-28517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Somogyi resolved HBASE-28517. --- Fix Version/s: 2.6.0 2.4.18 4.0.0-alpha-1 3.0.0-beta-2 2.5.9 Release Note: Make the following properties dynamically configured: * hbase.rs.evictblocksonclose * hbase.rs.cacheblocksonwrite * hbase.block.data.cacheonread Resolution: Fixed Merged to all active branches. Thanks, [~kabhishek4] for the contribution! > Make properties dynamically configured > -- > > Key: HBASE-28517 > URL: https://issues.apache.org/jira/browse/HBASE-28517 > Project: HBase > Issue Type: Improvement >Reporter: Abhishek Kothalikar >Assignee: Abhishek Kothalikar >Priority: Major > Labels: pull-request-available > Fix For: 2.6.0, 2.4.18, 4.0.0-alpha-1, 3.0.0-beta-2, 2.5.9 > > > Make following properties dynamically configured, > hbase.rs.evictblocksonclose > hbase.rs.cacheblocksonwrite > hbase.block.data.cacheonread > for use case scenarios where configuring them dynamically would help in > achieving better throughput. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Reopened] (HBASE-28247) Add java.base/sun.net.dns and java.base/sun.net.util export to jdk11 JVM test flags
[
https://issues.apache.org/jira/browse/HBASE-28247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Somogyi reopened HBASE-28247:
---
Reopening because this was not committed to branch-3 by accident.
> Add java.base/sun.net.dns and java.base/sun.net.util export to jdk11 JVM
> test flags
>
>
> Key: HBASE-28247
> URL: https://issues.apache.org/jira/browse/HBASE-28247
> Project: HBase
> Issue Type: Bug
> Components: java
>Affects Versions: 2.6.0, 3.0.0-alpha-4, 2.4.17, 2.5.6
>Reporter: Istvan Toth
>Assignee: Istvan Toth
>Priority: Minor
> Fix For: 2.6.0, 2.4.18, 3.0.0-beta-1, 2.5.7
>
>
> While testing with JDK17 we have found that we need to add
> {noformat}
> --add-exports java.base/sun.net.dns=ALL-UNNAMED
> --add-exports java.base/sun.net.util=ALL-UNNAMED
> {noformat}
> on top of what is already defined in _hbase-surefire.jdk11.flags_ , otherwise
> RS and Master startup fails in the Hadoop security code.
> While this does not affect the test suite (at least not the commonly run
> tests), I consider hbase-surefire.jdk11.flags to be an unoffical resource to
> getting HBase to run on newer JDK versions.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (HBASE-28247) Add java.base/sun.net.dns and java.base/sun.net.util export to jdk11 JVM test flags
[
https://issues.apache.org/jira/browse/HBASE-28247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Somogyi resolved HBASE-28247.
---
Resolution: Fixed
Merged to branch-3.
> Add java.base/sun.net.dns and java.base/sun.net.util export to jdk11 JVM
> test flags
>
>
> Key: HBASE-28247
> URL: https://issues.apache.org/jira/browse/HBASE-28247
> Project: HBase
> Issue Type: Bug
> Components: java
>Affects Versions: 2.6.0, 3.0.0-alpha-4, 2.4.17, 2.5.6
>Reporter: Istvan Toth
>Assignee: Istvan Toth
>Priority: Minor
> Fix For: 2.6.0, 2.4.18, 3.0.0-beta-1, 2.5.7
>
>
> While testing with JDK17 we have found that we need to add
> {noformat}
> --add-exports java.base/sun.net.dns=ALL-UNNAMED
> --add-exports java.base/sun.net.util=ALL-UNNAMED
> {noformat}
> on top of what is already defined in _hbase-surefire.jdk11.flags_ , otherwise
> RS and Master startup fails in the Hadoop security code.
> While this does not affect the test suite (at least not the commonly run
> tests), I consider hbase-surefire.jdk11.flags to be an unoffical resource to
> getting HBase to run on newer JDK versions.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (HBASE-28252) Add sun.net.dns and sun.net.util to the JDK11+ module exports in the hbase script
[ https://issues.apache.org/jira/browse/HBASE-28252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Somogyi resolved HBASE-28252. --- Resolution: Fixed Merged to branch-3. The commit was partially committed in HBASE-28259 earlier. > Add sun.net.dns and sun.net.util to the JDK11+ module exports in the hbase > script > - > > Key: HBASE-28252 > URL: https://issues.apache.org/jira/browse/HBASE-28252 > Project: HBase > Issue Type: Bug > Components: scripts >Reporter: Istvan Toth >Assignee: Istvan Toth >Priority: Major > Fix For: 2.6.0, 2.4.18, 3.0.0-beta-1, 2.5.7 > > > As noted in HBASE-28247, HBase can run into module permission issues that are > not handled in the current JDK11 options in the hbase startup script. > The surefire test config also includes some JDK17 specific options, we should > also add those as needed. > We are not yet aware of any additional JVM options required by Java 21. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Reopened] (HBASE-28252) Add sun.net.dns and sun.net.util to the JDK11+ module exports in the hbase script
[ https://issues.apache.org/jira/browse/HBASE-28252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Somogyi reopened HBASE-28252: --- Reopening because the commit did not land on branch-3 by accident. > Add sun.net.dns and sun.net.util to the JDK11+ module exports in the hbase > script > - > > Key: HBASE-28252 > URL: https://issues.apache.org/jira/browse/HBASE-28252 > Project: HBase > Issue Type: Bug > Components: scripts >Reporter: Istvan Toth >Assignee: Istvan Toth >Priority: Major > Fix For: 2.6.0, 2.4.18, 3.0.0-beta-1, 2.5.7 > > > As noted in HBASE-28247, HBase can run into module permission issues that are > not handled in the current JDK11 options in the hbase startup script. > The surefire test config also includes some JDK17 specific options, we should > also add those as needed. > We are not yet aware of any additional JVM options required by Java 21. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HBASE-28135) Specify -Xms for tests
[ https://issues.apache.org/jira/browse/HBASE-28135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Somogyi resolved HBASE-28135. --- Fix Version/s: 2.6.0 2.4.18 2.5.6 3.0.0-beta-1 Resolution: Fixed Pushed to branch-2.4+. Thanks [~stoty]! > Specify -Xms for tests > -- > > Key: HBASE-28135 > URL: https://issues.apache.org/jira/browse/HBASE-28135 > Project: HBase > Issue Type: Improvement > Components: test >Reporter: Istvan Toth >Assignee: Istvan Toth >Priority: Major > Fix For: 2.6.0, 2.4.18, 2.5.6, 3.0.0-beta-1 > > > The default -Xms value is JVM dependent, but the host memory size is usually > included in the calculation. > -Xms in turn is used to calculate some GC parameters, for example NewSize and > OldSize, which affect the behaviour of tests. > As the memory consumption on the tests is not dependent on the host VM size, > we could set -Xms for the tests explictly, and enjoy more consistent test > results. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HBASE-28133) TestSyncTimeRangeTracker fails with OOM with small -Xms values
[
https://issues.apache.org/jira/browse/HBASE-28133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Somogyi resolved HBASE-28133.
---
Fix Version/s: 2.6.0
2.4.18
2.5.6
3.0.0-beta-1
Resolution: Fixed
Merged to all active branches. Thanks [~stoty]!
> TestSyncTimeRangeTracker fails with OOM with small -Xms values
> --
>
> Key: HBASE-28133
> URL: https://issues.apache.org/jira/browse/HBASE-28133
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.4.17
>Reporter: Istvan Toth
>Assignee: Istvan Toth
>Priority: Major
> Labels: Arm64, test
> Fix For: 2.6.0, 2.4.18, 2.5.6, 3.0.0-beta-1
>
>
> Edit2: It's not the OS, it's the -Xmx value determined from the host memory
> size.
> Edit: It's related to the OS and it's default java 8 , not to the processor
> architecture.
> This test seems to be cutting real close to the heap size.
> On ARM, it consistently fails on my RHEL8.8 Aarch64 VM with Java 8.
> {noformat}
> mvn test -P runDevTests -Dtest.build.data.basedirectory=/ram2G
> -Dhadoop.profile=3.0 -fn -B -Dtest=TestSyncTimeRangeTracker* -pl hbase-server
> ...
> [ERROR]
> org.apache.hadoop.hbase.regionserver.TestSyncTimeRangeTracker.testConcurrentIncludeTimestampCorrectness
> Time elapsed: 1.969 s <<< ERROR!
> java.lang.OutOfMemoryError: Java heap space
> {noformat}
> It seems that Java on ARM has some higher memory overhead than x86_64.
> Simply bumping -Xmx from the default 2200m to 2300m allows it to pass.
> {noformat}
> mvn test -P runDevTests -Dtest.build.data.basedirectory=/ram2G
> -Dhadoop.profile=3.0 -fn -B -Dtest=TestSyncTimeRangeTracker* -pl hbase-server
> -Dsurefire.Xmx=2300m
> ...
> [INFO] Running org.apache.hadoop.hbase.regionserver.TestSyncTimeRangeTracker
> [INFO] Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.395
> s - in org.apache.hadoop.hbase.regionserver.TestSyncTimeRangeTracker
> {noformat}
> However, the real solution should be reducing the memory usage for this test.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (HBASE-28059) Use correct units in RegionLoad#getStoreUncompressedSizeMB()
[
https://issues.apache.org/jira/browse/HBASE-28059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Somogyi resolved HBASE-28059.
---
Fix Version/s: 2.6.0
2.4.18
2.5.6
Resolution: Fixed
Meged to branch-2, branch-2.5, and branch-2.4. Thanks for the contribution,
[~charlesconnell]!
> Use correct units in RegionLoad#getStoreUncompressedSizeMB()
>
>
> Key: HBASE-28059
> URL: https://issues.apache.org/jira/browse/HBASE-28059
> Project: HBase
> Issue Type: Improvement
> Components: Admin
>Affects Versions: 2.5.5
>Reporter: Charles Connell
>Assignee: Charles Connell
>Priority: Major
> Fix For: 2.6.0, 2.4.18, 2.5.6
>
>
> When I run a snippet of code like this:
> {code:java}
> Map regionLoadMap = admin
> .getClusterStatus()
> .getLoad(
> ServerName.parseServerName(
> "my-server.my-company.net,60020,1693513660506"
> )
> )
> .getRegionsLoad();
> for (byte[] startKey : regionLoadMap.keySet()) {
> RegionLoad regionLoad = regionLoadMap.get(startKey);
> LOG.info("Region {}: {}", Bytes.toStringBinary(startKey), regionLoad);
> } {code}
> I get logs like this:
> {noformat}
> Region , key>,1659484033280.2b89407a1223720344bed425bf3c29b0.: numberOfStores=1,
> numberOfStorefiles=3, storeRefCount=0, storefileUncompressedSizeMB=17998848,
> lastMajorCompactionTimestamp=1693211464712, storefileSizeMB=5895,
> compressionRatio=0.0003, memstoreSizeMB=1, readRequestsCount=118899553,
> writeRequestsCount=731192, rootIndexSizeKB=9, totalStaticIndexSizeKB=10413,
> totalStaticBloomSizeKB=6592, totalCompactingKVs=0, currentCompactedKVs=0,
> compactionProgressPct=NaN, completeSequenceId=78093096,
> dataLocality=1.0{noformat}
> The {{storefileUncompressedSizeMB}} is vastly larger than the
> {{{}storefileSizeMB{}}}, so much more that it's not believable. I checked the
> store files in question in this instance. Adding up the uncompressed size
> reported in the HFile trailers sums to 5895 MiB, exactly 1024 times less than
> the reported 17998848.
> The reason for the misreporting is that
> {{RegionLoad#getStoreUncompressedSizeMB()}} converts the value from a
> {{Size}} object wrong.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Reopened] (HBASE-28025) Enhance ByteBufferUtils.findCommonPrefix to compare 8 bytes each time
[ https://issues.apache.org/jira/browse/HBASE-28025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Somogyi reopened HBASE-28025: --- Reopening to apply addendum for branch-2.4. > Enhance ByteBufferUtils.findCommonPrefix to compare 8 bytes each time > - > > Key: HBASE-28025 > URL: https://issues.apache.org/jira/browse/HBASE-28025 > Project: HBase > Issue Type: Improvement > Components: Performance >Affects Versions: 2.5.4 >Reporter: Becker Ewing >Assignee: Becker Ewing >Priority: Major > Fix For: 2.6.0, 2.4.18, 2.5.6, 3.0.0-beta-1 > > Attachments: HBASE-28025_jmh_benchmarks_src.zip, benchmark_results.txt > > > Currently, the `ByteBufferUtils.findCommonPrefix family of methods compare > two buffers a single byte at a time. In reviewing the patch for HBASE-28012, > [~zhangduo] suggested that the ByteBufferUtils.findCommonPrefix methods could > be enhanced to compare 8 bytes at a time like the > ByteBufferUtils.compareToUnsafe family of methods already does (which was > added in HBASE-12345) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HBASE-27883) [hbase-connectors] Use log4j2 instead of log4j for logging
[ https://issues.apache.org/jira/browse/HBASE-27883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Somogyi resolved HBASE-27883. --- Resolution: Fixed Merged. Thanks [~subrat.mishra]! > [hbase-connectors] Use log4j2 instead of log4j for logging > -- > > Key: HBASE-27883 > URL: https://issues.apache.org/jira/browse/HBASE-27883 > Project: HBase > Issue Type: Task > Components: hbase-connectors >Reporter: Peter Somogyi >Assignee: Subrat Mishra >Priority: Blocker > Fix For: hbase-connectors-1.1.0 > > > Move to log4j2 in hbase-connectors. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HBASE-27884) [hbase-filesystem] Use log4j2 instead of log4j for logging
[ https://issues.apache.org/jira/browse/HBASE-27884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Somogyi resolved HBASE-27884. --- Release Note: Moved logging to log4j2. Resolution: Fixed Merged to master branch. Thanks [~subrat.mishra] for contributing! > [hbase-filesystem] Use log4j2 instead of log4j for logging > -- > > Key: HBASE-27884 > URL: https://issues.apache.org/jira/browse/HBASE-27884 > Project: HBase > Issue Type: Task > Components: Filesystem Integration >Reporter: Peter Somogyi >Assignee: Subrat Mishra >Priority: Major > Fix For: hbase-filesystem-1.0.0-alpha2 > > > Move to log4j2 in hbase-filesystem. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HBASE-27992) Bump exec-maven-plugin to 3.1.0
[
https://issues.apache.org/jira/browse/HBASE-27992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Somogyi resolved HBASE-27992.
---
Fix Version/s: 2.6.0
2.4.18
2.5.6
3.0.0-beta-1
4.0.0-alpha-1
Resolution: Fixed
Merged to branch-2.4+. Thanks for the review [~zhangduo]!
> Bump exec-maven-plugin to 3.1.0
> ---
>
> Key: HBASE-27992
> URL: https://issues.apache.org/jira/browse/HBASE-27992
> Project: HBase
> Issue Type: Task
> Components: build
>Reporter: Peter Somogyi
>Assignee: Peter Somogyi
>Priority: Trivial
> Fix For: 2.6.0, 2.4.18, 2.5.6, 3.0.0-beta-1, 4.0.0-alpha-1
>
>
> I frequently see IOException in hbase-shaded-with-hadoop-check-invariants
> make-sure-validation-files-are-in-sync phase. I'm not sure what is the root
> cause but it worth to bump the plugin version to the latest.
> {noformat}
> [INFO] --- exec-maven-plugin:1.6.0:exec
> (make-sure-validation-files-are-in-sync) @
> hbase-shaded-with-hadoop-check-invariants ---
> [ERROR] Command execution failed.
> java.io.IOException: Stream closed
> at java.lang.ProcessBuilder$NullOutputStream.write
> (ProcessBuilder.java:433)
> at java.io.OutputStream.write (OutputStream.java:116)
> at java.io.BufferedOutputStream.flushBuffer (BufferedOutputStream.java:82)
> at java.io.BufferedOutputStream.flush (BufferedOutputStream.java:140)
> at java.io.FilterOutputStream.close (FilterOutputStream.java:158)
> at org.apache.commons.exec.DefaultExecutor.closeProcessStreams
> (DefaultExecutor.java:306)
> at org.apache.commons.exec.DefaultExecutor.executeInternal
> (DefaultExecutor.java:387)
> at org.apache.commons.exec.DefaultExecutor.execute
> (DefaultExecutor.java:166)
> at org.codehaus.mojo.exec.ExecMojo.executeCommandLine (ExecMojo.java:804)
> at org.codehaus.mojo.exec.ExecMojo.executeCommandLine (ExecMojo.java:751)
> at org.codehaus.mojo.exec.ExecMojo.execute (ExecMojo.java:313)
> at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo
> (DefaultBuildPluginManager.java:137)
> at org.apache.maven.lifecycle.internal.MojoExecutor.execute
> (MojoExecutor.java:210)
> at org.apache.maven.lifecycle.internal.MojoExecutor.execute
> (MojoExecutor.java:156)
> at org.apache.maven.lifecycle.internal.MojoExecutor.execute
> (MojoExecutor.java:148)
> at
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject
> (LifecycleModuleBuilder.java:117)
> at
> org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call
> (MultiThreadedBuilder.java:190)
> at
> org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call
> (MultiThreadedBuilder.java:186)
> at java.util.concurrent.FutureTask.run (FutureTask.java:266)
> at java.util.concurrent.Executors$RunnableAdapter.call
> (Executors.java:511)
> at java.util.concurrent.FutureTask.run (FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker
> (ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run
> (ThreadPoolExecutor.java:624)
> at java.lang.Thread.run (Thread.java:748)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (HBASE-27992) Bump exec-maven-plugin to 3.1.0
Peter Somogyi created HBASE-27992:
-
Summary: Bump exec-maven-plugin to 3.1.0
Key: HBASE-27992
URL: https://issues.apache.org/jira/browse/HBASE-27992
Project: HBase
Issue Type: Task
Components: build
Reporter: Peter Somogyi
Assignee: Peter Somogyi
I frequently see IOException in hbase-shaded-with-hadoop-check-invariants
make-sure-validation-files-are-in-sync phase. I'm not sure what is the root
cause but it worth to bump the plugin version to the latest.
{noformat}
[INFO] --- exec-maven-plugin:1.6.0:exec
(make-sure-validation-files-are-in-sync) @
hbase-shaded-with-hadoop-check-invariants ---
[ERROR] Command execution failed.
java.io.IOException: Stream closed
at java.lang.ProcessBuilder$NullOutputStream.write (ProcessBuilder.java:433)
at java.io.OutputStream.write (OutputStream.java:116)
at java.io.BufferedOutputStream.flushBuffer (BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.flush (BufferedOutputStream.java:140)
at java.io.FilterOutputStream.close (FilterOutputStream.java:158)
at org.apache.commons.exec.DefaultExecutor.closeProcessStreams
(DefaultExecutor.java:306)
at org.apache.commons.exec.DefaultExecutor.executeInternal
(DefaultExecutor.java:387)
at org.apache.commons.exec.DefaultExecutor.execute
(DefaultExecutor.java:166)
at org.codehaus.mojo.exec.ExecMojo.executeCommandLine (ExecMojo.java:804)
at org.codehaus.mojo.exec.ExecMojo.executeCommandLine (ExecMojo.java:751)
at org.codehaus.mojo.exec.ExecMojo.execute (ExecMojo.java:313)
at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo
(DefaultBuildPluginManager.java:137)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute
(MojoExecutor.java:210)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute
(MojoExecutor.java:156)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute
(MojoExecutor.java:148)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject
(LifecycleModuleBuilder.java:117)
at
org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call
(MultiThreadedBuilder.java:190)
at
org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call
(MultiThreadedBuilder.java:186)
at java.util.concurrent.FutureTask.run (FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call (Executors.java:511)
at java.util.concurrent.FutureTask.run (FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker
(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run
(ThreadPoolExecutor.java:624)
at java.lang.Thread.run (Thread.java:748)
{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (HBASE-27897) ConnectionImplementation#locateRegionInMeta should pause and retry when taking user region lock failed
[ https://issues.apache.org/jira/browse/HBASE-27897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Somogyi resolved HBASE-27897. --- Resolution: Fixed Merted the addendum to branch-2.4. Thanks for the review, [~zhangduo]! > ConnectionImplementation#locateRegionInMeta should pause and retry when > taking user region lock failed > -- > > Key: HBASE-27897 > URL: https://issues.apache.org/jira/browse/HBASE-27897 > Project: HBase > Issue Type: Improvement > Components: Client >Affects Versions: 2.4.17, 2.5.4 >Reporter: Xiaolin Ha >Assignee: Xiaolin Ha >Priority: Major > Fix For: 2.6.0, 2.4.18, 2.5.6 > > > It just throws exception and skips the pause and retry logic when > ConnectionImplementation#takeUserRegionLock fails. In some circumstances, no > pause and retry by outer logic will make next > ConnectionImplementation#takeUserRegionLock still fails, since all the > threads simultaneously grab the lock. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Reopened] (HBASE-27897) ConnectionImplementation#locateRegionInMeta should pause and retry when taking user region lock failed
[ https://issues.apache.org/jira/browse/HBASE-27897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Somogyi reopened HBASE-27897: --- An incorrect cherry-pick to branch-2.4 causes test failures. > ConnectionImplementation#locateRegionInMeta should pause and retry when > taking user region lock failed > -- > > Key: HBASE-27897 > URL: https://issues.apache.org/jira/browse/HBASE-27897 > Project: HBase > Issue Type: Improvement > Components: Client >Affects Versions: 2.4.17, 2.5.4 >Reporter: Xiaolin Ha >Assignee: Xiaolin Ha >Priority: Major > Fix For: 2.6.0, 2.4.18, 2.5.6 > > > It just throws exception and skips the pause and retry logic when > ConnectionImplementation#takeUserRegionLock fails. In some circumstances, no > pause and retry by outer logic will make next > ConnectionImplementation#takeUserRegionLock still fails, since all the > threads simultaneously grab the lock. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HBASE-27892) Report memstore on-heap and off-heap size as jmx metrics
[ https://issues.apache.org/jira/browse/HBASE-27892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Somogyi resolved HBASE-27892. --- Fix Version/s: (was: 2.4.18) Resolution: Fixed Reverted from branch-2.4. [~jingyu] please reopen this ticket if you want this on branch-2.4. > Report memstore on-heap and off-heap size as jmx metrics > > > Key: HBASE-27892 > URL: https://issues.apache.org/jira/browse/HBASE-27892 > Project: HBase > Issue Type: Improvement > Components: metrics >Reporter: Bryan Beaudreault >Assignee: Jing Yu >Priority: Major > Fix For: 2.6.0, 2.5.6, 3.0.0-beta-1 > > > Currently we only report "memStoreSize" metric in sub=RegionServer bean. I've > noticed a big discrepancy between this metric and the RS UI's "Memstore > On-Heap Size". It seems like "memStoreSize" is the overall data size, while > the on-heap size is coming from our heap estimation which includes POJO heap > overhead, etc. > I have a regionserver with only 750mb of "memStoreSize", but the on-heap size > is over 1gb. This is non-trivial for estimating overall heap size necessary > for a regionserver. Since we have the data, let's report it. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Reopened] (HBASE-27892) Report memstore on-heap and off-heap size as jmx metrics
[ https://issues.apache.org/jira/browse/HBASE-27892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Somogyi reopened HBASE-27892: --- This commit causes compilation failure on branch-2.4. Reopening for the revert. > Report memstore on-heap and off-heap size as jmx metrics > > > Key: HBASE-27892 > URL: https://issues.apache.org/jira/browse/HBASE-27892 > Project: HBase > Issue Type: Improvement > Components: metrics >Reporter: Bryan Beaudreault >Assignee: Jing Yu >Priority: Major > Fix For: 2.6.0, 2.4.18, 2.5.6, 3.0.0-beta-1 > > > Currently we only report "memStoreSize" metric in sub=RegionServer bean. I've > noticed a big discrepancy between this metric and the RS UI's "Memstore > On-Heap Size". It seems like "memStoreSize" is the overall data size, while > the on-heap size is coming from our heap estimation which includes POJO heap > overhead, etc. > I have a regionserver with only 750mb of "memStoreSize", but the on-heap size > is over 1gb. This is non-trivial for estimating overall heap size necessary > for a regionserver. Since we have the data, let's report it. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HBASE-27933) Stable version outdated on https://hbase.apache.org/downloads.html
[ https://issues.apache.org/jira/browse/HBASE-27933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Somogyi resolved HBASE-27933. --- Fix Version/s: 3.0.0-beta-1 Resolution: Fixed Merged to master branch. Thanks for the contribution [~dieterdp_ng]! > Stable version outdated on https://hbase.apache.org/downloads.html > -- > > Key: HBASE-27933 > URL: https://issues.apache.org/jira/browse/HBASE-27933 > Project: HBase > Issue Type: Task >Affects Versions: 2.5.4 >Reporter: Dieter De Paepe >Assignee: Dieter De Paepe >Priority: Minor > Fix For: 3.0.0-beta-1 > > > In HBASE-27849, the stable version of HBase was updated to 2.5.x. This > updated the [https://downloads.apache.org/hbase/] page. > However, the download page ([https://hbase.apache.org/downloads.html)] still > refers to 2.4.x as the stable version. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HBASE-27884) [hbase-filesystem] Use log4j2 instead of log4j for logging
Peter Somogyi created HBASE-27884: - Summary: [hbase-filesystem] Use log4j2 instead of log4j for logging Key: HBASE-27884 URL: https://issues.apache.org/jira/browse/HBASE-27884 Project: HBase Issue Type: Task Components: Filesystem Integration Reporter: Peter Somogyi Move to log4j2 in hbase-filesystem. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HBASE-27883) [hbase-connectors] Use log4j2 instead of log4j for logging
Peter Somogyi created HBASE-27883: - Summary: [hbase-connectors] Use log4j2 instead of log4j for logging Key: HBASE-27883 URL: https://issues.apache.org/jira/browse/HBASE-27883 Project: HBase Issue Type: Task Components: hbase-connectors Reporter: Peter Somogyi Fix For: hbase-connectors-1.1.0 Move to log4j2 in hbase-connectors. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HBASE-27801) Remove redundant avro.version property from Kafka connector
[ https://issues.apache.org/jira/browse/HBASE-27801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Somogyi resolved HBASE-27801. --- Fix Version/s: hbase-connectors-1.1.0 Resolution: Fixed Merged to master. Thanks [~stoty]! > Remove redundant avro.version property from Kafka connector > --- > > Key: HBASE-27801 > URL: https://issues.apache.org/jira/browse/HBASE-27801 > Project: HBase > Issue Type: Bug > Components: hbase-connectors, kafka >Affects Versions: connector-1.0.0 >Reporter: Istvan Toth >Assignee: Istvan Toth >Priority: Minor > Fix For: hbase-connectors-1.1.0 > > > 1.7.7 > is defined both in the main connectors pom, and in the kafka module. > This is not useful. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HBASE-27776) Backport HBASE-27731 (Upgrade commons-validator to version 1.7) to branch-2.5
[ https://issues.apache.org/jira/browse/HBASE-27776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Somogyi resolved HBASE-27776. --- Resolution: Invalid > Backport HBASE-27731 (Upgrade commons-validator to version 1.7) to branch-2.5 > - > > Key: HBASE-27776 > URL: https://issues.apache.org/jira/browse/HBASE-27776 > Project: HBase > Issue Type: Task >Reporter: Wes Schuitema >Assignee: Wes Schuitema >Priority: Minor > > This is so we also fix these CVEs for 2.5 > - [CVE-2014-0114|https://nvd.nist.gov/vuln/detail/cve-2014-0114] > - [CVE-2019-10086|https://nvd.nist.gov/vuln/detail/cve-2019-10086] > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Reopened] (HBASE-27776) Backport HBASE-27731 (Upgrade commons-validator to version 1.7) to branch-2.5
[ https://issues.apache.org/jira/browse/HBASE-27776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Somogyi reopened HBASE-27776: --- > Backport HBASE-27731 (Upgrade commons-validator to version 1.7) to branch-2.5 > - > > Key: HBASE-27776 > URL: https://issues.apache.org/jira/browse/HBASE-27776 > Project: HBase > Issue Type: Task >Reporter: Wes Schuitema >Assignee: Wes Schuitema >Priority: Minor > > This is so we also fix these CVEs for 2.5 > - [CVE-2014-0114|https://nvd.nist.gov/vuln/detail/cve-2014-0114] > - [CVE-2019-10086|https://nvd.nist.gov/vuln/detail/cve-2019-10086] > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HBASE-27745) Document protoc workarounds with Apple Silicon
[ https://issues.apache.org/jira/browse/HBASE-27745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Somogyi resolved HBASE-27745. --- Fix Version/s: 3.0.0-alpha-4 Resolution: Fixed Merged to master. Thanks for the review [~ndimiduk]! > Document protoc workarounds with Apple Silicon > -- > > Key: HBASE-27745 > URL: https://issues.apache.org/jira/browse/HBASE-27745 > Project: HBase > Issue Type: Sub-task > Components: documentation >Reporter: Peter Somogyi >Assignee: Peter Somogyi >Priority: Major > Fix For: 3.0.0-alpha-4 > > > Building hbase 2.x on Apple Silicon is difficult because there is no protoc > library available. > [~ndimiduk] added a solution in HBASE-27741 to use osx-x86_64 but it is also > possible to build protoc locally and use that. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HBASE-27745) Document protoc workarounds with Apple Silicon
Peter Somogyi created HBASE-27745: - Summary: Document protoc workarounds with Apple Silicon Key: HBASE-27745 URL: https://issues.apache.org/jira/browse/HBASE-27745 Project: HBase Issue Type: Sub-task Components: documentation Reporter: Peter Somogyi Assignee: Peter Somogyi Building hbase 2.x on Apple Silicon is difficult because there is no protoc library available. [~ndimiduk] added a solution in HBASE-27741 to use osx-x86_64 but it is also possible to build protoc locally and use that. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HBASE-27701) ZStdCodec codec implementation class documentation typo
[
https://issues.apache.org/jira/browse/HBASE-27701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Somogyi resolved HBASE-27701.
---
Fix Version/s: 3.0.0-alpha-4
Resolution: Fixed
Thanks for the contribution, [~frensjan]!
> ZStdCodec codec implementation class documentation typo
> ---
>
> Key: HBASE-27701
> URL: https://issues.apache.org/jira/browse/HBASE-27701
> Project: HBase
> Issue Type: Bug
> Components: documentation
>Reporter: Frens Jan Rumph
>Assignee: Frens Jan Rumph
>Priority: Minor
> Fix For: 3.0.0-alpha-4
>
>
> As mentioned in the [u...@hbase.apache.org|mailto:u...@hbase.apache.org]
> mailing list I noticed a small typo in the documentation on compression for
> Zstd. The codec implementation class in the documentation is listed as
> {{org.apache.hadoop.hbase.io.compress.zstd.ZStdCodec}} while the actual class
> is written with a lower case S: {{ZStdCodec.}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (HBASE-27685) Enable code coverage reporting to SonarQube in HBase
[ https://issues.apache.org/jira/browse/HBASE-27685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Somogyi resolved HBASE-27685. --- Fix Version/s: 2.6.0 3.0.0-alpha-4 2.4.17 2.5.4 Resolution: Fixed Merged to branch-2.4+. Thanks [~dora.horvath]! > Enable code coverage reporting to SonarQube in HBase > > > Key: HBASE-27685 > URL: https://issues.apache.org/jira/browse/HBASE-27685 > Project: HBase > Issue Type: Task >Reporter: Dóra Horváth >Assignee: Dóra Horváth >Priority: Minor > Fix For: 2.6.0, 3.0.0-alpha-4, 2.4.17, 2.5.4 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HBASE-27679) Bump junit to 4.13.2 in hbase-connectors
[ https://issues.apache.org/jira/browse/HBASE-27679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Somogyi resolved HBASE-27679. --- Fix Version/s: hbase-connectors-1.1.0 Resolution: Fixed Merged to master. > Bump junit to 4.13.2 in hbase-connectors > > > Key: HBASE-27679 > URL: https://issues.apache.org/jira/browse/HBASE-27679 > Project: HBase > Issue Type: Task > Components: hbase-connectors >Reporter: Peter Somogyi >Assignee: Peter Somogyi >Priority: Major > Fix For: hbase-connectors-1.1.0 > > > Dependapot reported an issue with junit. Move to the version we have in hbase > main repository. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HBASE-27639) Support hbase-connectors compilation with HBase 2.5.3, Hadoop 3.2.4 and Spark 3.2.3
[ https://issues.apache.org/jira/browse/HBASE-27639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Somogyi resolved HBASE-27639. --- Fix Version/s: hbase-connectors-1.1.0 Resolution: Fixed Merged to master. Thanks for the patch [~nihaljain.cs]. > Support hbase-connectors compilation with HBase 2.5.3, Hadoop 3.2.4 and Spark > 3.2.3 > --- > > Key: HBASE-27639 > URL: https://issues.apache.org/jira/browse/HBASE-27639 > Project: HBase > Issue Type: Improvement > Components: hbase-connectors, spark >Reporter: Nihal Jain >Assignee: Nihal Jain >Priority: Major > Fix For: hbase-connectors-1.1.0 > > > Goal is to allow hbase-connectors to compile with: > * HBase: 2.5.3 > * Hadoop: 3.2.4 and > * Spark: 3.2.3 > We could also discuss if we want to bump the versions of the above mentioned > in the pom itself, > or just want to let spark connector compile with above components as the JIRA > title says. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HBASE-27678) Update checkstyle in hbase-connectors
[ https://issues.apache.org/jira/browse/HBASE-27678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Somogyi resolved HBASE-27678. --- Fix Version/s: hbase-connectors-1.1.0 Resolution: Fixed Merged to master. > Update checkstyle in hbase-connectors > - > > Key: HBASE-27678 > URL: https://issues.apache.org/jira/browse/HBASE-27678 > Project: HBase > Issue Type: Task > Components: hbase-connectors >Reporter: Peter Somogyi >Assignee: Peter Somogyi >Priority: Major > Fix For: hbase-connectors-1.1.0 > > > There is a known CVE on the used checkstyle in hbase-connectors. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HBASE-27679) Bump junit to 4.13.2
Peter Somogyi created HBASE-27679: - Summary: Bump junit to 4.13.2 Key: HBASE-27679 URL: https://issues.apache.org/jira/browse/HBASE-27679 Project: HBase Issue Type: Task Components: hbase-connectors Reporter: Peter Somogyi Assignee: Peter Somogyi Dependapot reported an issue with junit. Move to the version we have in hbase main repository. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HBASE-24452) Try github ci for hbase-connectors
[ https://issues.apache.org/jira/browse/HBASE-24452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Somogyi resolved HBASE-24452. --- Resolution: Won't Do > Try github ci for hbase-connectors > -- > > Key: HBASE-24452 > URL: https://issues.apache.org/jira/browse/HBASE-24452 > Project: HBase > Issue Type: Improvement >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HBASE-27678) Update checkstyle in hbase-connectors
Peter Somogyi created HBASE-27678: - Summary: Update checkstyle in hbase-connectors Key: HBASE-27678 URL: https://issues.apache.org/jira/browse/HBASE-27678 Project: HBase Issue Type: Task Components: hbase-connectors Reporter: Peter Somogyi Assignee: Peter Somogyi There is a known CVE on the used checkstyle in hbase-connectors. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HBASE-27665) Update checkstyle in hbase-operator-tools
[ https://issues.apache.org/jira/browse/HBASE-27665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Somogyi resolved HBASE-27665. --- Fix Version/s: hbase-operator-tools-1.3.0 Resolution: Fixed Merged to master. Thanks [~zhangduo] for reviewing! > Update checkstyle in hbase-operator-tools > - > > Key: HBASE-27665 > URL: https://issues.apache.org/jira/browse/HBASE-27665 > Project: HBase > Issue Type: Task > Components: hbase-operator-tools >Affects Versions: hbase-operator-tools-1.2.0 >Reporter: Peter Somogyi >Assignee: Peter Somogyi >Priority: Major > Fix For: hbase-operator-tools-1.3.0 > > > The checkstyle dependency is vulnerable in hbase-operator-tools. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HBASE-25610) Support multiple tables as input in generateMissingTableDescriptorFile command in HBCK2
[ https://issues.apache.org/jira/browse/HBASE-25610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Somogyi resolved HBASE-25610. --- Fix Version/s: hbase-operator-tools-1.3.0 Resolution: Fixed Merged to master. Thanks for the patch [~sanjeetnishad]! > Support multiple tables as input in generateMissingTableDescriptorFile > command in HBCK2 > --- > > Key: HBASE-25610 > URL: https://issues.apache.org/jira/browse/HBASE-25610 > Project: HBase > Issue Type: Improvement > Components: hbase-operator-tools, hbck2 >Reporter: Sanjeet Nishad >Assignee: Sanjeet Nishad >Priority: Minor > Fix For: hbase-operator-tools-1.3.0 > > > Currently 'generateMissingTableDescriptorFile' command in HBCK2 supports only > 1 TableName as input. HBCK's _fixOrphanTables()_ had a support for fixing the > missing .tableinfo files for a list of tables and it also supported fixing > the missing descriptor for all tables if no tables were specified. > It looks like a convinient enhancement to have in > generateMissingTableDescriptorFile command of HBCK2. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HBASE-27665) Update checkstyle in hbase-operator-tools
Peter Somogyi created HBASE-27665: - Summary: Update checkstyle in hbase-operator-tools Key: HBASE-27665 URL: https://issues.apache.org/jira/browse/HBASE-27665 Project: HBase Issue Type: Task Components: hbase-operator-tools Affects Versions: hbase-operator-tools-1.2.0 Reporter: Peter Somogyi Assignee: Peter Somogyi The checkstyle dependency is vulnerable in hbase-operator-tools. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HBASE-27630) hbase-spark bulkload stage directory limited to hdfs only
[ https://issues.apache.org/jira/browse/HBASE-27630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Somogyi resolved HBASE-27630. --- Resolution: Fixed Merged to master branch in the hbase-connectors repository. Thanks for the patch [~sergey.soldatov]! > hbase-spark bulkload stage directory limited to hdfs only > - > > Key: HBASE-27630 > URL: https://issues.apache.org/jira/browse/HBASE-27630 > Project: HBase > Issue Type: Bug > Components: spark >Affects Versions: connector-1.0.0 >Reporter: Sergey Soldatov >Assignee: Sergey Soldatov >Priority: Major > Fix For: hbase-connectors-1.1.0 > > > It's impossible to set up the staging directory for bulkload operation in > spark-hbase connector to any other filesystem different from hdfs. That might > be a problem for deployments where hbase.rootdir points to cloud storage. In > this case, an additional copy task from hdfs to cloud storage would be > required before loading hfiles to hbase. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HBASE-27590) Change Iterable to List in SnapshotFileCache
[
https://issues.apache.org/jira/browse/HBASE-27590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Somogyi resolved HBASE-27590.
---
Resolution: Fixed
Cherry-picked to branch-2.4.
> Change Iterable to List in SnapshotFileCache
>
>
> Key: HBASE-27590
> URL: https://issues.apache.org/jira/browse/HBASE-27590
> Project: HBase
> Issue Type: Improvement
>Reporter: Peter Somogyi
>Assignee: Peter Somogyi
>Priority: Minor
> Fix For: 2.6.0, 3.0.0-alpha-4, 2.5.4
>
> Attachments: flame-1.html
>
>
> The HFileCleaners can have low performance on large /archive area when used
> with slow storage like S3. The snapshot write lock in SnapshotFileCache is
> held while the file metadata is fetched from S3. Due to this even with
> multiple cleaner threads only a single cleaner can effectively delete files
> from the archive.
> File metadata collection is performed before SnapshotHFileCleaner just by
> changing the passed parameter type in FileCleanerDelegate from Iterable to
> List.
> Running with the below cleaner configurations I observed that the lock held
> in SnapshotFileCache went down from 45000ms to 100ms when it was running for
> 1000 files in a directory. The complete evaluation and deletion for this
> folder took the same time but since the file metadata fetch from S3 was done
> outside of the lock the multiple cleaner threads were able to run
> concurrently.
> {noformat}
> hbase.cleaner.directory.sorting=false
> hbase.cleaner.scan.dir.concurrent.size=0.75
> hbase.regionserver.hfilecleaner.small.thread.count=16
> hbase.regionserver.hfilecleaner.large.thread.count=8
> {noformat}
> The files to evaluate are already passed in a List to
> CleanerChore.checkAndDeleteFiles but it is converted to an Iterable to run
> the checks on the configured cleaners.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Reopened] (HBASE-27590) Change Iterable to List in SnapshotFileCache
[
https://issues.apache.org/jira/browse/HBASE-27590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Somogyi reopened HBASE-27590:
---
> Change Iterable to List in SnapshotFileCache
>
>
> Key: HBASE-27590
> URL: https://issues.apache.org/jira/browse/HBASE-27590
> Project: HBase
> Issue Type: Improvement
>Reporter: Peter Somogyi
>Assignee: Peter Somogyi
>Priority: Minor
> Fix For: 2.6.0, 3.0.0-alpha-4, 2.5.4
>
> Attachments: flame-1.html
>
>
> The HFileCleaners can have low performance on large /archive area when used
> with slow storage like S3. The snapshot write lock in SnapshotFileCache is
> held while the file metadata is fetched from S3. Due to this even with
> multiple cleaner threads only a single cleaner can effectively delete files
> from the archive.
> File metadata collection is performed before SnapshotHFileCleaner just by
> changing the passed parameter type in FileCleanerDelegate from Iterable to
> List.
> Running with the below cleaner configurations I observed that the lock held
> in SnapshotFileCache went down from 45000ms to 100ms when it was running for
> 1000 files in a directory. The complete evaluation and deletion for this
> folder took the same time but since the file metadata fetch from S3 was done
> outside of the lock the multiple cleaner threads were able to run
> concurrently.
> {noformat}
> hbase.cleaner.directory.sorting=false
> hbase.cleaner.scan.dir.concurrent.size=0.75
> hbase.regionserver.hfilecleaner.small.thread.count=16
> hbase.regionserver.hfilecleaner.large.thread.count=8
> {noformat}
> The files to evaluate are already passed in a List to
> CleanerChore.checkAndDeleteFiles but it is converted to an Iterable to run
> the checks on the configured cleaners.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (HBASE-27629) Backport HBASE-27043 to branch-2.4
[ https://issues.apache.org/jira/browse/HBASE-27629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Somogyi resolved HBASE-27629. --- Fix Version/s: 2.4.17 Resolution: Fixed Merged to branch-2.4. > Backport HBASE-27043 to branch-2.4 > -- > > Key: HBASE-27629 > URL: https://issues.apache.org/jira/browse/HBASE-27629 > Project: HBase > Issue Type: Sub-task >Reporter: Peter Somogyi >Assignee: Peter Somogyi >Priority: Major > Fix For: 2.4.17 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HBASE-27627) Backport HBASE-25899 to branch-2.4
[ https://issues.apache.org/jira/browse/HBASE-27627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Somogyi resolved HBASE-27627. --- Fix Version/s: 2.4.17 Resolution: Fixed Merged backport to branch-2.4. Thanks for the review [~taklwu] ! > Backport HBASE-25899 to branch-2.4 > -- > > Key: HBASE-27627 > URL: https://issues.apache.org/jira/browse/HBASE-27627 > Project: HBase > Issue Type: Sub-task >Reporter: Peter Somogyi >Assignee: Peter Somogyi >Priority: Major > Fix For: 2.4.17 > > > Backport HBASE-25899 to branch-2.4 as it can increase the speed of archive > cleanup. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HBASE-27629) Backport HBASE-27043 to branch-2.4
Peter Somogyi created HBASE-27629: - Summary: Backport HBASE-27043 to branch-2.4 Key: HBASE-27629 URL: https://issues.apache.org/jira/browse/HBASE-27629 Project: HBase Issue Type: Sub-task Reporter: Peter Somogyi Assignee: Peter Somogyi -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HBASE-27628) Spotless fix in RELEASENOTES.md
[
https://issues.apache.org/jira/browse/HBASE-27628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Somogyi resolved HBASE-27628.
---
Fix Version/s: 2.4.17
2.5.4
Resolution: Fixed
Merged trivial fix to branch-2.4 and branch-2.5.
> Spotless fix in RELEASENOTES.md
> ---
>
> Key: HBASE-27628
> URL: https://issues.apache.org/jira/browse/HBASE-27628
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.4.16, 2.5.3
>Reporter: Peter Somogyi
>Assignee: Peter Somogyi
>Priority: Trivial
> Fix For: 2.4.17, 2.5.4
>
>
> There is a whitespace vilotation in RELEASENOTES.md on branch-2.4 and
> branch-2.5 causing the pre-commit and nightly builds to fail.
> {noformat}
> [ERROR] Failed to execute goal
> com.diffplug.spotless:spotless-maven-plugin:2.27.2:check (default-cli) on
> project hbase: The following files had format violations:
> [ERROR] RELEASENOTES.md
> [ERROR] @@ -85,7 +85,7 @@
> [ERROR]
> [ERROR]
> *·[HBASE-27529](https://issues.apache.org/jira/browse/HBASE-27529)·|·*Major*·|·**Provide·RS·coproc·ability·to·attach·WAL·extended·attributes·to·mutations·at·replication·sink**
> [ERROR]
> [ERROR]
> -New·regionserver·coproc·endpoints·that·can·be·used·by·coproc·at·the·replication·sink·cluster·if·WAL·has·extended·attributes.·
> [ERROR]
> +New·regionserver·coproc·endpoints·that·can·be·used·by·coproc·at·the·replication·sink·cluster·if·WAL·has·extended·attributes.
> [ERROR]
> Using·the·new·endpoints,·WAL·extended·attributes·can·be·transferred·to·Mutation·attributes·at·the·replication·sink·cluster.
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (HBASE-27628) Spotbugs fix in RELEASENOTES.md
Peter Somogyi created HBASE-27628:
-
Summary: Spotbugs fix in RELEASENOTES.md
Key: HBASE-27628
URL: https://issues.apache.org/jira/browse/HBASE-27628
Project: HBase
Issue Type: Bug
Affects Versions: 2.5.3, 2.4.16
Reporter: Peter Somogyi
Assignee: Peter Somogyi
There is a whitespace vilotation in RELEASENOTES.md on branch-2.4 and
branch-2.5 causing the pre-commit and nightly builds to fail.
{noformat}
[ERROR] Failed to execute goal
com.diffplug.spotless:spotless-maven-plugin:2.27.2:check (default-cli) on
project hbase: The following files had format violations:
[ERROR] RELEASENOTES.md
[ERROR] @@ -85,7 +85,7 @@
[ERROR]
[ERROR]
*·[HBASE-27529](https://issues.apache.org/jira/browse/HBASE-27529)·|·*Major*·|·**Provide·RS·coproc·ability·to·attach·WAL·extended·attributes·to·mutations·at·replication·sink**
[ERROR]
[ERROR]
-New·regionserver·coproc·endpoints·that·can·be·used·by·coproc·at·the·replication·sink·cluster·if·WAL·has·extended·attributes.·
[ERROR]
+New·regionserver·coproc·endpoints·that·can·be·used·by·coproc·at·the·replication·sink·cluster·if·WAL·has·extended·attributes.
[ERROR]
Using·the·new·endpoints,·WAL·extended·attributes·can·be·transferred·to·Mutation·attributes·at·the·replication·sink·cluster.
{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (HBASE-27590) Change Iterable to List in SnapshotFileCache
[
https://issues.apache.org/jira/browse/HBASE-27590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Somogyi resolved HBASE-27590.
---
Fix Version/s: 2.6.0
3.0.0-alpha-4
2.5.4
Resolution: Fixed
Merged to branch-2.5, branch-2, and master. Will merge to branch-2.4 once
HBASE-27627 gets resolved.
Thanks for the suggestion and review [~zhangduo]!
> Change Iterable to List in SnapshotFileCache
>
>
> Key: HBASE-27590
> URL: https://issues.apache.org/jira/browse/HBASE-27590
> Project: HBase
> Issue Type: Improvement
>Reporter: Peter Somogyi
>Assignee: Peter Somogyi
>Priority: Minor
> Fix For: 2.6.0, 3.0.0-alpha-4, 2.5.4
>
> Attachments: flame-1.html
>
>
> The HFileCleaners can have low performance on large /archive area when used
> with slow storage like S3. The snapshot write lock in SnapshotFileCache is
> held while the file metadata is fetched from S3. Due to this even with
> multiple cleaner threads only a single cleaner can effectively delete files
> from the archive.
> File metadata collection is performed before SnapshotHFileCleaner just by
> changing the passed parameter type in FileCleanerDelegate from Iterable to
> List.
> Running with the below cleaner configurations I observed that the lock held
> in SnapshotFileCache went down from 45000ms to 100ms when it was running for
> 1000 files in a directory. The complete evaluation and deletion for this
> folder took the same time but since the file metadata fetch from S3 was done
> outside of the lock the multiple cleaner threads were able to run
> concurrently.
> {noformat}
> hbase.cleaner.directory.sorting=false
> hbase.cleaner.scan.dir.concurrent.size=0.75
> hbase.regionserver.hfilecleaner.small.thread.count=16
> hbase.regionserver.hfilecleaner.large.thread.count=8
> {noformat}
> The files to evaluate are already passed in a List to
> CleanerChore.checkAndDeleteFiles but it is converted to an Iterable to run
> the checks on the configured cleaners.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (HBASE-27624) Cannot Specify Namespace via the hbase.table Option in Spark Connector
[
https://issues.apache.org/jira/browse/HBASE-27624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Somogyi resolved HBASE-27624.
---
Fix Version/s: hbase-connectors-1.0.1
Resolution: Fixed
Pushed to master. Thanks [~stoty]!
> Cannot Specify Namespace via the hbase.table Option in Spark Connector
> --
>
> Key: HBASE-27624
> URL: https://issues.apache.org/jira/browse/HBASE-27624
> Project: HBase
> Issue Type: Bug
> Components: hbase-connectors, spark
>Affects Versions: 1.0.1
>Reporter: Istvan Toth
>Assignee: Istvan Toth
>Priority: Major
> Fix For: hbase-connectors-1.0.1
>
>
> When using the old mapping format and specifying the HBase table via the
> _hbase.table_ option, the connector passes the namespaced string to HBase,
> and we get
> {noformat}
> Caused by: java.lang.IllegalArgumentException: Illegal character code:58, <:>
> at 7. User-space table qualifiers may only contain 'alphanumeric characters'
> and digits: staplesHbaseNamespace:staplesHbaseTableName
> at
> org.apache.hadoop.hbase.TableName.isLegalTableQualifierName(TableName.java:187)
> at
> org.apache.hadoop.hbase.TableName.isLegalTableQualifierName(TableName.java:138)
> at org.apache.hadoop.hbase.TableName.(TableName.java:320)
> at
> org.apache.hadoop.hbase.TableName.createTableNameIfNecessary(TableName.java:354)
> at org.apache.hadoop.hbase.TableName.valueOf(TableName.java:484){noformat}
> This seems to be related to the changes in HBASE-24276
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (HBASE-27627) Backport HBASE-25899 to branch-2.4
Peter Somogyi created HBASE-27627: - Summary: Backport HBASE-25899 to branch-2.4 Key: HBASE-27627 URL: https://issues.apache.org/jira/browse/HBASE-27627 Project: HBase Issue Type: Sub-task Reporter: Peter Somogyi Assignee: Peter Somogyi Backport HBASE-25899 to branch-2.4 as it can increase the speed of archive cleanup. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HBASE-27590) Change Iterable to List in CleanerChore
Peter Somogyi created HBASE-27590:
-
Summary: Change Iterable to List in CleanerChore
Key: HBASE-27590
URL: https://issues.apache.org/jira/browse/HBASE-27590
Project: HBase
Issue Type: Improvement
Reporter: Peter Somogyi
Assignee: Peter Somogyi
The HFileCleaners can have low performance on large /archive area when used
with slow storage like S3. The snapshot write lock in SnapshotFileCache is held
while the file metadata is fetched from S3. Due to this even with multiple
cleaner threads only a single cleaner can effectively delete files from the
archive.
File metadata collection is performed before SnapshotHFileCleaner just by
changing the passed parameter type in FileCleanerDelegate from Iterable to List.
Running with the below cleaner configurations I observed that the lock held in
SnapshotFileCache went down from 45000ms to 100ms when it was running for 1000
files in a directory. The complete evaluation and deletion for this folder took
the same time but since the file metadata fetch from S3 was done outside of the
lock the multiple cleaner threads were able to run concurrently.
{noformat}
hbase.cleaner.directory.sorting=false
hbase.cleaner.scan.dir.concurrent.size=0.75
hbase.regionserver.hfilecleaner.small.thread.count=16
hbase.regionserver.hfilecleaner.large.thread.count=8
{noformat}
The files to evaluate are already passed in a List to
CleanerChore.checkAndDeleteFiles but it is converted to an Iterable to run the
checks on the configured cleaners.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (HBASE-27493) Allow namespace admins to clone snapshots created by them
[ https://issues.apache.org/jira/browse/HBASE-27493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Somogyi resolved HBASE-27493. --- Fix Version/s: 2.6.0 3.0.0-alpha-4 Resolution: Fixed Merged to branch-2 and master. [~bszabolcs] can you fill the release notes? > Allow namespace admins to clone snapshots created by them > - > > Key: HBASE-27493 > URL: https://issues.apache.org/jira/browse/HBASE-27493 > Project: HBase > Issue Type: Improvement > Components: snapshots >Affects Versions: 3.0.0-alpha-3, 2.5.1 >Reporter: Szabolcs Bukros >Assignee: Szabolcs Bukros >Priority: Major > Fix For: 2.6.0, 3.0.0-alpha-4 > > > Creating a snapshot requires table admin permissions. But cloning it requires > global admin permissions unless the user owns the snapshot and wants to > recreate the original table the snapshot was based on using the same table > name. This puts unnecessary load on the few people having global admin > permissions. I would like to relax this rule a bit and allow the owner of the > snapshot to clone it into any namespace where they have admin permissions > regardless of the table name used. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HBASE-27565) Make the initial corePoolSize configurable for ChoreService
Peter Somogyi created HBASE-27565: - Summary: Make the initial corePoolSize configurable for ChoreService Key: HBASE-27565 URL: https://issues.apache.org/jira/browse/HBASE-27565 Project: HBase Issue Type: Improvement Components: conf Reporter: Peter Somogyi Assignee: Peter Somogyi The initial corePoolSize for ChoreService is set to 1. The pool size is increased when a scheduled task misses its start time. On a cluster where the archive size is large, the HFileCleaner could run for a very long time and block the rest of the chores to run. By making the initial pool size configurable we could solve the bottleneck of caused by a long-running HFileCleaner chore. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HBASE-27554) Test failures on branch-2.4 with corrupted exclude list
[
https://issues.apache.org/jira/browse/HBASE-27554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Somogyi resolved HBASE-27554.
---
Resolution: Fixed
Recent nightly tests were successful on branch-2.4 after the cleanup.
> Test failures on branch-2.4 with corrupted exclude list
> ---
>
> Key: HBASE-27554
> URL: https://issues.apache.org/jira/browse/HBASE-27554
> Project: HBase
> Issue Type: Bug
> Components: jenkins
>Affects Versions: 2.4.16
>Reporter: Peter Somogyi
>Assignee: Peter Somogyi
>Priority: Major
>
> Nightly builds and PRs on branch-2.4 are failing with an invalid exclude list.
> Executed unit test command:
> {code:java}
> /opt/maven/bin/mvn --batch-mode
> -Dmaven.repo.local=/home/jenkins/jenkins-home/workspace/Base-PreCommit-GitHub-PR_PR-4944/yetus-m2/hbase-branch-2.4-patch-0
> --threads=4
> -Djava.io.tmpdir=/home/jenkins/jenkins-home/workspace/Base-PreCommit-GitHub-PR_PR-4944/yetus-jdk8-hadoop2-check/src/target
> -DHBasePatchProcess -PrunAllTests
> -Dtest.exclude.pattern=**/replication.regionserver.TestMetaRegionReplicaReplicationEndpoint.java,**/client.TestMetaRegionLocationCache.java,**/master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks.java,**/replication.TestZKReplicationQueueStorageWARNING:
> All illegal access operations will be denied in a future
> release.java,**/replication.regionserver.TestBasicWALEntryStreamFSHLog.java
> -Dsurefire.firstPartForkCount=0.5C -Dsurefire.secondPartForkCount=0.5C clean
> test -fae {code}
> The latest exclude list contains "WARNING: All illegal access operations will
> be denied in a future release" and maven treats this as a new parameter. As a
> result unit tests are failing on CI that rely on the exclude list.
> [https://ci-hbase.apache.org/job/HBase-Find-Flaky-Tests/job/branch-2.4/lastSuccessfulBuild/artifact/output/excludes/*view*/]
> {noformat}
> **/replication.regionserver.TestMetaRegionReplicaReplicationEndpoint.java,**/client.TestMetaRegionLocationCache.java,**/master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks.java,**/replication.TestZKReplicationQueueStorageWARNING:
> All illegal access operations will be denied in a future
> release.java,**/replication.regionserver.TestBasicWALEntryStreamFSHLog.java
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (HBASE-23340) hmaster /hbase/replication/rs session expired (hbase replication default value is true, we don't use ) causes logcleaner can not clean oldWALs, which resulits in oldW
[ https://issues.apache.org/jira/browse/HBASE-23340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Somogyi resolved HBASE-23340. --- Resolution: Fixed Merged the subtask to branch-2.4. > hmaster /hbase/replication/rs session expired (hbase replication default > value is true, we don't use ) causes logcleaner can not clean oldWALs, which > resulits in oldWALs too large (more than 2TB) > - > > Key: HBASE-23340 > URL: https://issues.apache.org/jira/browse/HBASE-23340 > Project: HBase > Issue Type: Improvement > Components: master >Affects Versions: 3.0.0-alpha-1, 2.2.3 >Reporter: jackylau >Assignee: Bo Cui >Priority: Major > Fix For: 2.5.0, 3.0.0-alpha-1 > > Attachments: Snipaste_2019-11-21_10-39-25.png, > Snipaste_2019-11-21_14-10-36.png > > > hmaster /hbase/replication/rs session expired (hbase replication default > value is true, we don't use ) causes logcleaner can not clean oldWALs, which > resulits in oldWALs too large (more than 2TB). > !Snipaste_2019-11-21_10-39-25.png! > > !Snipaste_2019-11-21_14-10-36.png! > > we can solve it by following : > 1) increase the session timeout(but i think it is not a good idea. because we > do not know how long to set is suitable) > 2) close the hbase replication. It is not a good idea too, when our user uses > this feature > 3) we need add retry times, for example when it has already happened three > times, we set the ReplicationLogCleaner and SnapShotCleaner stop > that is all my ideas, i do not konw it is suitable, If it is suitable, could > i commit a PR? > Does anynode have a good idea. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HBASE-27556) Backport HBASE-23340 to branch-2.4
Peter Somogyi created HBASE-27556: - Summary: Backport HBASE-23340 to branch-2.4 Key: HBASE-27556 URL: https://issues.apache.org/jira/browse/HBASE-27556 Project: HBase Issue Type: Sub-task Affects Versions: 2.4.15 Reporter: Peter Somogyi Assignee: Peter Somogyi -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HBASE-27555) Process all patch-unit-root.txt in report-flakies.py
Peter Somogyi created HBASE-27555: - Summary: Process all patch-unit-root.txt in report-flakies.py Key: HBASE-27555 URL: https://issues.apache.org/jira/browse/HBASE-27555 Project: HBase Issue Type: Bug Components: test Reporter: Peter Somogyi The report-flakies.py which parses the output of the unit tests does not review all the patch-unit-root.txt files. The for loop exists on the first found unit test output file and ignores the rest. Earlier the build only had a single unit test run but the current setup runs multiple builds based on the JDK or Hadoop version. All of these outputs should be parsed by the script. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HBASE-27554) Test failures on branch-2.4 with corrupted exclude list
Peter Somogyi created HBASE-27554:
-
Summary: Test failures on branch-2.4 with corrupted exclude list
Key: HBASE-27554
URL: https://issues.apache.org/jira/browse/HBASE-27554
Project: HBase
Issue Type: Bug
Components: jenkins
Affects Versions: 2.4.16
Reporter: Peter Somogyi
Assignee: Peter Somogyi
Nightly builds and PRs on branch-2.4 are failing with an invalid exclude list.
Executed unit test command:
{code:java}
/opt/maven/bin/mvn --batch-mode
-Dmaven.repo.local=/home/jenkins/jenkins-home/workspace/Base-PreCommit-GitHub-PR_PR-4944/yetus-m2/hbase-branch-2.4-patch-0
--threads=4
-Djava.io.tmpdir=/home/jenkins/jenkins-home/workspace/Base-PreCommit-GitHub-PR_PR-4944/yetus-jdk8-hadoop2-check/src/target
-DHBasePatchProcess -PrunAllTests
-Dtest.exclude.pattern=**/replication.regionserver.TestMetaRegionReplicaReplicationEndpoint.java,**/client.TestMetaRegionLocationCache.java,**/master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks.java,**/replication.TestZKReplicationQueueStorageWARNING:
All illegal access operations will be denied in a future
release.java,**/replication.regionserver.TestBasicWALEntryStreamFSHLog.java
-Dsurefire.firstPartForkCount=0.5C -Dsurefire.secondPartForkCount=0.5C clean
test -fae {code}
The latest exclude list contains "WARNING: All illegal access operations will
be denied in a future release" and maven treats this as a new parameter. As a
result unit tests are failing on CI that rely on the exclude list.
[https://ci-hbase.apache.org/job/HBase-Find-Flaky-Tests/job/branch-2.4/lastSuccessfulBuild/artifact/output/excludes/*view*/]
{noformat}
**/replication.regionserver.TestMetaRegionReplicaReplicationEndpoint.java,**/client.TestMetaRegionLocationCache.java,**/master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks.java,**/replication.TestZKReplicationQueueStorageWARNING:
All illegal access operations will be denied in a future
release.java,**/replication.regionserver.TestBasicWALEntryStreamFSHLog.java
{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Reopened] (HBASE-23340) hmaster /hbase/replication/rs session expired (hbase replication default value is true, we don't use ) causes logcleaner can not clean oldWALs, which resulits in oldW
[ https://issues.apache.org/jira/browse/HBASE-23340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Somogyi reopened HBASE-23340: --- Reopening to backport to branch-2.4. This fix can also be useful on that branch. > hmaster /hbase/replication/rs session expired (hbase replication default > value is true, we don't use ) causes logcleaner can not clean oldWALs, which > resulits in oldWALs too large (more than 2TB) > - > > Key: HBASE-23340 > URL: https://issues.apache.org/jira/browse/HBASE-23340 > Project: HBase > Issue Type: Improvement > Components: master >Affects Versions: 3.0.0-alpha-1, 2.2.3 >Reporter: jackylau >Assignee: Bo Cui >Priority: Major > Fix For: 3.0.0-alpha-1, 2.5.0 > > Attachments: Snipaste_2019-11-21_10-39-25.png, > Snipaste_2019-11-21_14-10-36.png > > > hmaster /hbase/replication/rs session expired (hbase replication default > value is true, we don't use ) causes logcleaner can not clean oldWALs, which > resulits in oldWALs too large (more than 2TB). > !Snipaste_2019-11-21_10-39-25.png! > > !Snipaste_2019-11-21_14-10-36.png! > > we can solve it by following : > 1) increase the session timeout(but i think it is not a good idea. because we > do not know how long to set is suitable) > 2) close the hbase replication. It is not a good idea too, when our user uses > this feature > 3) we need add retry times, for example when it has already happened three > times, we set the ReplicationLogCleaner and SnapShotCleaner stop > that is all my ideas, i do not konw it is suitable, If it is suitable, could > i commit a PR? > Does anynode have a good idea. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HBASE-26320) Separate Log Cleaner DirScanPool to prevent the OLDWALs from filling up the disk when archive is large
[ https://issues.apache.org/jira/browse/HBASE-26320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Somogyi resolved HBASE-26320. --- Fix Version/s: 2.4.16 Resolution: Fixed Merged to branch-2.4. Thanks [~zhangduo] for the review. > Separate Log Cleaner DirScanPool to prevent the OLDWALs from filling up the > disk when archive is large > -- > > Key: HBASE-26320 > URL: https://issues.apache.org/jira/browse/HBASE-26320 > Project: HBase > Issue Type: Improvement > Components: Operability >Affects Versions: 1.7.1, 2.4.6 >Reporter: Zach York >Assignee: Zach York >Priority: Major > Fix For: 2.4.16, 3.0.0-alpha-3, 2.5.0 > > > We currently share the DirScanPool (threadpool for scanning for files to > delete in the OldLogs and archive directories) between the LogCleaner and > HFileCleaner. This means that if the archive directory is large/has lots of > files/directories, the threads can get stuck scanning through the archive > directory, starving the LogCleaner. This is especially apparent on S3 where > list can be slower than on HDFS. > This JIRA creates separate DirScanPools for the LogCleaner and HFileCleaner -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HBASE-27506) Optionally disable sorting directories by size in CleanerChore
[ https://issues.apache.org/jira/browse/HBASE-27506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Somogyi resolved HBASE-27506. --- Fix Version/s: 2.6.0 3.0.0-alpha-4 2.4.16 2.5.3 Resolution: Fixed Pushed to branch-2.4+. Thanks for the review [~wchevreuil]! > Optionally disable sorting directories by size in CleanerChore > -- > > Key: HBASE-27506 > URL: https://issues.apache.org/jira/browse/HBASE-27506 > Project: HBase > Issue Type: Improvement >Reporter: Peter Somogyi >Assignee: Peter Somogyi >Priority: Minor > Fix For: 2.6.0, 3.0.0-alpha-4, 2.4.16, 2.5.3 > > > The CleanerChore computes the size of the directories to start the cleaning > with the largest directory first. This computation can take a long time over > object stores when the archive directory is huge. > Make it configurable to skip the directory sorting in the CleanerChore. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HBASE-27507) Add configuration for disabling CleanerChore
Peter Somogyi created HBASE-27507: - Summary: Add configuration for disabling CleanerChore Key: HBASE-27507 URL: https://issues.apache.org/jira/browse/HBASE-27507 Project: HBase Issue Type: Improvement Reporter: Peter Somogyi Assignee: Peter Somogyi The CleanerChore can be disabled from the shell but not from the configuration. The Cleaner will start with HBase Master it is not possible to prevent it from the initial start. A configuration property could be added to disable the CleanerChore. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HBASE-27506) Optionally disable sorting directories by size in CleanerChore
Peter Somogyi created HBASE-27506: - Summary: Optionally disable sorting directories by size in CleanerChore Key: HBASE-27506 URL: https://issues.apache.org/jira/browse/HBASE-27506 Project: HBase Issue Type: Improvement Reporter: Peter Somogyi Assignee: Peter Somogyi The CleanerChore computes the size of the directories to start the cleaning with the largest directory first. This computation can take a long time over object stores when the archive directory is huge. Make it configurable to skip the directory sorting in the CleanerChore. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Reopened] (HBASE-26320) Separate Log Cleaner DirScanPool to prevent the OLDWALs from filling up the disk when archive is large
[ https://issues.apache.org/jira/browse/HBASE-26320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Somogyi reopened HBASE-26320: --- Reopening to backport to branch-2.4. > Separate Log Cleaner DirScanPool to prevent the OLDWALs from filling up the > disk when archive is large > -- > > Key: HBASE-26320 > URL: https://issues.apache.org/jira/browse/HBASE-26320 > Project: HBase > Issue Type: Improvement > Components: Operability >Affects Versions: 1.7.1, 2.4.6 >Reporter: Zach York >Assignee: Zach York >Priority: Major > Fix For: 2.5.0, 3.0.0-alpha-3 > > > We currently share the DirScanPool (threadpool for scanning for files to > delete in the OldLogs and archive directories) between the LogCleaner and > HFileCleaner. This means that if the archive directory is large/has lots of > files/directories, the threads can get stuck scanning through the archive > directory, starving the LogCleaner. This is especially apparent on S3 where > list can be slower than on HDFS. > This JIRA creates separate DirScanPools for the LogCleaner and HFileCleaner -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HBASE-27001) The deleted variable cannot be printed out
[
https://issues.apache.org/jira/browse/HBASE-27001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Somogyi resolved HBASE-27001.
---
Fix Version/s: 2.4.16
Resolution: Fixed
> The deleted variable cannot be printed out
> --
>
> Key: HBASE-27001
> URL: https://issues.apache.org/jira/browse/HBASE-27001
> Project: HBase
> Issue Type: Bug
>Reporter: selina.yan
>Assignee: selina.yan
>Priority: Minor
> Fix For: 2.4.16, 3.0.0-alpha-3, 2.5.0
>
>
> In the deleteAction method of CleanerChore class, the delete variable cannot
> be printed out
> {code:java}
> LOG.trace("Finish deleting {} under {}, deleted=", type, dir, deleted); {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Reopened] (HBASE-27001) The deleted variable cannot be printed out
[
https://issues.apache.org/jira/browse/HBASE-27001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Somogyi reopened HBASE-27001:
---
Reopening for branch-2.4 backport.
> The deleted variable cannot be printed out
> --
>
> Key: HBASE-27001
> URL: https://issues.apache.org/jira/browse/HBASE-27001
> Project: HBase
> Issue Type: Bug
>Reporter: selina.yan
>Assignee: selina.yan
>Priority: Minor
> Fix For: 2.5.0, 3.0.0-alpha-3
>
>
> In the deleteAction method of CleanerChore class, the delete variable cannot
> be printed out
> {code:java}
> LOG.trace("Finish deleting {} under {}, deleted=", type, dir, deleted); {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (HBASE-22048) Incorrect email links on website
[ https://issues.apache.org/jira/browse/HBASE-22048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Somogyi resolved HBASE-22048. --- Assignee: (was: Rishabh Jain) Resolution: Done Resolved by upgrading maven-project-info-reports-plugin. > Incorrect email links on website > > > Key: HBASE-22048 > URL: https://issues.apache.org/jira/browse/HBASE-22048 > Project: HBase > Issue Type: Bug > Components: website >Reporter: Peter Somogyi >Priority: Minor > Labels: beginner > > Project members email addresses has incorrect link. > [https://hbase.apache.org/team.html] > Instead of [apach...@apache.org|mailto:apach...@apache.org] it points to > [https://hbase.apache.org/apach...@apache.org] > This change might be related to ASF parent pom upgrade which changed the > maven-project-info-reports-plugin version. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HBASE-27431) Remove TestRemoteTable.testLimitedScan
Peter Somogyi created HBASE-27431: - Summary: Remove TestRemoteTable.testLimitedScan Key: HBASE-27431 URL: https://issues.apache.org/jira/browse/HBASE-27431 Project: HBase Issue Type: Task Components: REST, test Affects Versions: 2.4.15, 2.5.1, 2.6.0 Reporter: Peter Somogyi Assignee: Peter Somogyi The TestRemoteTable.testLimitedScan was originally added in HBASE-24267 for 3.0.0-alpha-1 but with the backport of HBASE-27401 the test was added to branch-2.4, branch-2.5, and branch-2 where the limit support is not available. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HBASE-27317) Correcting the Store file tracker tool help doc option
[ https://issues.apache.org/jira/browse/HBASE-27317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Somogyi resolved HBASE-27317. --- Fix Version/s: 2.6.0 3.0.0-alpha-4 Resolution: Fixed Merged to branch-2 and master. Thanks for the contribution [~abhradeep.kundu]! > Correcting the Store file tracker tool help doc option > -- > > Key: HBASE-27317 > URL: https://issues.apache.org/jira/browse/HBASE-27317 > Project: HBase > Issue Type: Sub-task >Reporter: Abhradeep Kundu >Assignee: Abhradeep Kundu >Priority: Minor > Fix For: 2.6.0, 3.0.0-alpha-4 > > > Column family is not an optional parameter, we need to correct the sysout msg > properly. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HBASE-26969) Eliminate MOB renames when SFT is enabled
[
https://issues.apache.org/jira/browse/HBASE-26969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Somogyi resolved HBASE-26969.
---
Fix Version/s: 2.5.0
(was: 2.6.0)
(was: 2.5.1)
Resolution: Fixed
Merged the backports to branch-2 and branch-2.5.
Thanks [~bszabolcs] for the contribution!
> Eliminate MOB renames when SFT is enabled
> -
>
> Key: HBASE-26969
> URL: https://issues.apache.org/jira/browse/HBASE-26969
> Project: HBase
> Issue Type: Sub-task
> Components: mob
>Affects Versions: 2.5.0, 3.0.0-alpha-3
>Reporter: Szabolcs Bukros
>Assignee: Szabolcs Bukros
>Priority: Major
> Fix For: 2.5.0, 3.0.0-alpha-4
>
>
> MOB file compaction and flush still relies on renames even when SFT is
> enabled.
> My proposed changes are:
> * when requireWritingToTmpDirFirst is false during mob flush/compact instead
> of using the temp writer we should create a different writer using a
> {color:#00}StoreFileWriterCreationTracker that writes directly to the mob
> store folder{color}
> * {color:#00}these StoreFileWriterCreationTracker should be stored in
> the MobStore. This would requires us to extend MobStore with a createWriter
> and a finalizeWriter method to handle this{color}
> * {color:#00}refactor {color}MobFileCleanerChore to run on the RS
> instead on Master to allow access to the
> {color:#00}StoreFileWriterCreationTracker{color}s to make sure the
> currently written files are not cleaned up
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (HBASE-25970) MOB data loss - incorrect concatenation of MOB_FILE_REFS
[
https://issues.apache.org/jira/browse/HBASE-25970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Somogyi resolved HBASE-25970.
---
Fix Version/s: 2.5.0
Resolution: Fixed
Merged to branch-2 and branch-2.5.
> MOB data loss - incorrect concatenation of MOB_FILE_REFS
> -
>
> Key: HBASE-25970
> URL: https://issues.apache.org/jira/browse/HBASE-25970
> Project: HBase
> Issue Type: Bug
> Components: mob
>Affects Versions: 3.0.0-alpha-1
>Reporter: Peter Somogyi
>Assignee: Peter Somogyi
>Priority: Critical
> Fix For: 2.5.0, 3.0.0-alpha-1
>
>
> Active MOB files can be deleted by MobFileCleanerChore. The MOB_FILE_REFS are
> wrongly concatenated when multiple files or tables are stored. During the MOB
> cleanup HBase wants to keep the incorrectly concatenated file and considers
> the real file as deletable.
> {noformat}
> 2021-06-03 15:03:09,876 TRACE
> org.apache.hadoop.hbase.mob.MobFileCleanerChore: Specific mob references
> found for
> store=hdfs://c1377-node2.coelab.cloudera.com:8020/hbase/data/default/IntegrationTestIngestWithMOB/1df6561244ecc434f505501e6bda9af0/test_cf/2956445a41184488b914e93199df905a
> :
> {IntegrationTestIngestWithMOB=[d41d8cd98f00b204e9800998ecf8427e20210603a6c8c0e5bf8548649b0009b156452c21_1df6561244ecc434f505501e6bda9af0d41d8cd98f00b204e9800998ecf8427e20210603497714eabbe5418c9d52382a4808e986_1df6561244ecc434f505501e6bda9af0,
>
> d41d8cd98f00b204e9800998ecf8427e202106037464cbddc86943a1bad77d52a3bd412c_1df6561244ecc434f505501e6bda9af0]}
>
> ...
> 2021-06-03 15:03:10,208 TRACE
> org.apache.hadoop.hbase.mob.MobFileCleanerChore: Archiving MOB file
> hdfs://c1377-node2.coelab.cloudera.com:8020/hbase/mobdir/data/default/IntegrationTestIngestWithMOB/e9b5d936e7f55a4f1c3246a8d5ce53c2/test_cf/d41d8cd98f00b204e9800998ecf8427e20210603497714eabbe5418c9d52382a4808e986_1df6561244ecc434f505501e6bda9af0
> creation time=1622729860319
> 2021-06-03 15:03:10,467 DEBUG
> org.apache.hadoop.hbase.mob.MobFileCleanerChore: MOB Cleaner is archiving:
> hdfs://c1377-node2.coelab.cloudera.com:8020/hbase/mobdir/data/default/IntegrationTestIngestWithMOB/e9b5d936e7f55a4f1c3246a8d5ce53c2/test_cf/d41d8cd98f00b204e9800998ecf8427e20210603497714eabbe5418c9d52382a4808e986_1df6561244ecc434f505501e6bda9af0
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Reopened] (HBASE-25970) MOB data loss - incorrect concatenation of MOB_FILE_REFS
[
https://issues.apache.org/jira/browse/HBASE-25970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Somogyi reopened HBASE-25970:
---
Reopening for branch-2 and branch-2.5 backports.
> MOB data loss - incorrect concatenation of MOB_FILE_REFS
> -
>
> Key: HBASE-25970
> URL: https://issues.apache.org/jira/browse/HBASE-25970
> Project: HBase
> Issue Type: Bug
> Components: mob
>Affects Versions: 3.0.0-alpha-1
>Reporter: Peter Somogyi
>Assignee: Peter Somogyi
>Priority: Critical
> Fix For: 3.0.0-alpha-1
>
>
> Active MOB files can be deleted by MobFileCleanerChore. The MOB_FILE_REFS are
> wrongly concatenated when multiple files or tables are stored. During the MOB
> cleanup HBase wants to keep the incorrectly concatenated file and considers
> the real file as deletable.
> {noformat}
> 2021-06-03 15:03:09,876 TRACE
> org.apache.hadoop.hbase.mob.MobFileCleanerChore: Specific mob references
> found for
> store=hdfs://c1377-node2.coelab.cloudera.com:8020/hbase/data/default/IntegrationTestIngestWithMOB/1df6561244ecc434f505501e6bda9af0/test_cf/2956445a41184488b914e93199df905a
> :
> {IntegrationTestIngestWithMOB=[d41d8cd98f00b204e9800998ecf8427e20210603a6c8c0e5bf8548649b0009b156452c21_1df6561244ecc434f505501e6bda9af0d41d8cd98f00b204e9800998ecf8427e20210603497714eabbe5418c9d52382a4808e986_1df6561244ecc434f505501e6bda9af0,
>
> d41d8cd98f00b204e9800998ecf8427e202106037464cbddc86943a1bad77d52a3bd412c_1df6561244ecc434f505501e6bda9af0]}
>
> ...
> 2021-06-03 15:03:10,208 TRACE
> org.apache.hadoop.hbase.mob.MobFileCleanerChore: Archiving MOB file
> hdfs://c1377-node2.coelab.cloudera.com:8020/hbase/mobdir/data/default/IntegrationTestIngestWithMOB/e9b5d936e7f55a4f1c3246a8d5ce53c2/test_cf/d41d8cd98f00b204e9800998ecf8427e20210603497714eabbe5418c9d52382a4808e986_1df6561244ecc434f505501e6bda9af0
> creation time=1622729860319
> 2021-06-03 15:03:10,467 DEBUG
> org.apache.hadoop.hbase.mob.MobFileCleanerChore: MOB Cleaner is archiving:
> hdfs://c1377-node2.coelab.cloudera.com:8020/hbase/mobdir/data/default/IntegrationTestIngestWithMOB/e9b5d936e7f55a4f1c3246a8d5ce53c2/test_cf/d41d8cd98f00b204e9800998ecf8427e20210603497714eabbe5418c9d52382a4808e986_1df6561244ecc434f505501e6bda9af0
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (HBASE-27285) Fix sonar report paths
[ https://issues.apache.org/jira/browse/HBASE-27285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Somogyi resolved HBASE-27285. --- Fix Version/s: hbase-connectors-1.1.0 Resolution: Fixed Merged to master. Thanks [~dora.horvath]! > Fix sonar report paths > -- > > Key: HBASE-27285 > URL: https://issues.apache.org/jira/browse/HBASE-27285 > Project: HBase > Issue Type: Task > Components: hbase-connectors >Reporter: Dóra Horváth >Assignee: Dóra Horváth >Priority: Minor > Fix For: hbase-connectors-1.1.0 > > > Sonar XML report paths are not set correctly for jacoco and scoverage. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HBASE-27281) Add default implementation for Connection$getClusterId
[ https://issues.apache.org/jira/browse/HBASE-27281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Somogyi resolved HBASE-27281. --- Hadoop Flags: Incompatible change Release Note: Adds a default null implementation for Connection$getClusterId. Downstream applications should implement this method. Resolution: Fixed Merged to branch-2.4+. Thanks [~zhangduo]! > Add default implementation for Connection$getClusterId > -- > > Key: HBASE-27281 > URL: https://issues.apache.org/jira/browse/HBASE-27281 > Project: HBase > Issue Type: Task > Components: Client >Affects Versions: 3.0.0-alpha-1, 2.5.0, 2.4.14 >Reporter: Peter Somogyi >Assignee: Peter Somogyi >Priority: Critical > Fix For: 2.5.0, 3.0.0-alpha-4, 2.4.14 > > > The getClusterId method was added to Connection for delegation token based > auth. This interface is InterfaceAudience.Public so it breaks client > compatibility. Hbase connections builds fail with the latest 2.4.14-SNAPSHOT > version and downstream applications relying on Connection interface will fail > to work on these versions. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HBASE-27281) Add default implementation for Connection$getClusterId
Peter Somogyi created HBASE-27281: - Summary: Add default implementation for Connection$getClusterId Key: HBASE-27281 URL: https://issues.apache.org/jira/browse/HBASE-27281 Project: HBase Issue Type: Task Components: Client Affects Versions: 3.0.0-alpha-1, 2.5.0, 2.4.14 Reporter: Peter Somogyi Assignee: Peter Somogyi Fix For: 2.5.0, 3.0.0-alpha-4, 2.4.14 The getClusterId method was added to Connection for delegation token based auth. This interface is InterfaceAudience.Public so it breaks client compatibility. Hbase connections builds fail with the latest 2.4.14-SNAPSHOT version and downstream applications relying on Connection interface will fail to work on these versions. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HBASE-27184) When Hbase uses GET to query data, some rowkeys cannot be queried，ERROR: java.io.TOException: java.lang.NoclassDefFoundError: org/apache/hadoop/hdfs/DFSInputStreamS2
[ https://issues.apache.org/jira/browse/HBASE-27184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Somogyi resolved HBASE-27184. --- Resolution: Not A Bug > When Hbase uses GET to query data, some rowkeys cannot be queried，ERROR: > java.io.TOException: java.lang.NoclassDefFoundError: > org/apache/hadoop/hdfs/DFSInputStreamS2 > - > > Key: HBASE-27184 > URL: https://issues.apache.org/jira/browse/HBASE-27184 > Project: HBase > Issue Type: Bug >Reporter: yangzhichao >Priority: Major > Attachments: image-2022-07-08-16-59-52-808.png > > > !image-2022-07-08-16-59-52-808.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HBASE-27196) Enable code coverage reporting to SonarQube in hbase-filesystem
[ https://issues.apache.org/jira/browse/HBASE-27196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Somogyi resolved HBASE-27196. --- Fix Version/s: hbase-filesystem-1.0.0-alpha2 Resolution: Fixed Thanks for the contribution, [~dora.horvath]! > Enable code coverage reporting to SonarQube in hbase-filesystem > --- > > Key: HBASE-27196 > URL: https://issues.apache.org/jira/browse/HBASE-27196 > Project: HBase > Issue Type: Task > Components: Filesystem Integration >Reporter: Dóra Horváth >Assignee: Dóra Horváth >Priority: Minor > Fix For: hbase-filesystem-1.0.0-alpha2 > > > We would like to be able to publish code coverage results to SonarQube and it > would be useful to create a script to do it. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HBASE-27051) TestReplicationSource.testReplicationSourceInitializingMetric is flaky
[
https://issues.apache.org/jira/browse/HBASE-27051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Somogyi resolved HBASE-27051.
---
Fix Version/s: 2.4.13
Resolution: Fixed
Pushed to branch-2.4.
> TestReplicationSource.testReplicationSourceInitializingMetric is flaky
> --
>
> Key: HBASE-27051
> URL: https://issues.apache.org/jira/browse/HBASE-27051
> Project: HBase
> Issue Type: Test
> Components: test
>Affects Versions: 2.5.0
>Reporter: Andrew Kyle Purtell
>Assignee: Andrew Kyle Purtell
>Priority: Minor
> Fix For: 2.5.0, 3.0.0-alpha-3, 2.4.13
>
>
> 1 second appears to be not enough time for slow or loaded test hosts:
> {noformat}
> TestReplicationSource.testReplicationSourceInitializingMetric Time elapsed:
> 1.021 s <<< FAILURE!
> java.lang.AssertionError: Waiting timed out after [1,000] msec
> at org.junit.Assert.fail(Assert.java:89)
> at org.apache.hadoop.hbase.Waiter.waitFor(Waiter.java:203)
> at org.apache.hadoop.hbase.Waiter.waitFor(Waiter.java:135)
> at
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSource.testReplicationSourceInitializingMetric(TestReplicationSource.java:576)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Reopened] (HBASE-27051) TestReplicationSource.testReplicationSourceInitializingMetric is flaky
[
https://issues.apache.org/jira/browse/HBASE-27051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Somogyi reopened HBASE-27051:
---
Reopening to backport to branch-2.4.
> TestReplicationSource.testReplicationSourceInitializingMetric is flaky
> --
>
> Key: HBASE-27051
> URL: https://issues.apache.org/jira/browse/HBASE-27051
> Project: HBase
> Issue Type: Test
> Components: test
>Affects Versions: 2.5.0
>Reporter: Andrew Kyle Purtell
>Assignee: Andrew Kyle Purtell
>Priority: Minor
> Fix For: 2.5.0, 3.0.0-alpha-3
>
>
> 1 second appears to be not enough time for slow or loaded test hosts:
> {noformat}
> TestReplicationSource.testReplicationSourceInitializingMetric Time elapsed:
> 1.021 s <<< FAILURE!
> java.lang.AssertionError: Waiting timed out after [1,000] msec
> at org.junit.Assert.fail(Assert.java:89)
> at org.apache.hadoop.hbase.Waiter.waitFor(Waiter.java:203)
> at org.apache.hadoop.hbase.Waiter.waitFor(Waiter.java:135)
> at
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSource.testReplicationSourceInitializingMetric(TestReplicationSource.java:576)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Resolved] (HBASE-26934) Publish code coverage reports to SonarQube
[ https://issues.apache.org/jira/browse/HBASE-26934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Somogyi resolved HBASE-26934. --- Fix Version/s: hbase-operator-tools-1.3.0 Resolution: Fixed Thanks [~dora.horvath]! > Publish code coverage reports to SonarQube > -- > > Key: HBASE-26934 > URL: https://issues.apache.org/jira/browse/HBASE-26934 > Project: HBase > Issue Type: Task > Components: hbase-connectors, hbase-operator-tools >Reporter: Dóra Horváth >Assignee: Dóra Horváth >Priority: Minor > Fix For: hbase-operator-tools-1.3.0 > > > We need to be able to publish code coverage results to SonarQube, it would be > useful to create a script to do it. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Resolved] (HBASE-27076) [HBOSS] compile against hadoop 3.3.2+ only
[ https://issues.apache.org/jira/browse/HBASE-27076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Somogyi resolved HBASE-27076. --- Fix Version/s: hbase-filesystem-1.0.0-alpha2 Release Note: Drop HBoss compatibility for Hadoop 3.2. Assignee: Steve Loughran Resolution: Fixed Merged to master. Thanks [~ste...@apache.org]! > [HBOSS] compile against hadoop 3.3.2+ only > -- > > Key: HBASE-27076 > URL: https://issues.apache.org/jira/browse/HBASE-27076 > Project: HBase > Issue Type: Improvement > Components: hboss >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Minor > Fix For: hbase-filesystem-1.0.0-alpha2 > > > to get openFile and other things to work safely, hboss needs to be changed so > it only builds against hadoopp 3.3 only. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Created] (HBASE-27023) Add protobuf to NOTICE file
Peter Somogyi created HBASE-27023: - Summary: Add protobuf to NOTICE file Key: HBASE-27023 URL: https://issues.apache.org/jira/browse/HBASE-27023 Project: HBase Issue Type: Task Reporter: Peter Somogyi Assignee: Peter Somogyi The spotless formatting removed the protobuf credit from the AbstractByteRange. It is currently not included in the NOTICE file. https://github.com/apache/hbase/commit/9c8c9e7fbf8005ea89fa9b13d6d063b9f0240443#diff-f5806f14849a23b9265b022f3f330b80d08bcc10fcf69d8ee2e1b0d5af266d52 -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Resolved] (HBASE-26896) list_quota_snapshots fails with ‘ERROR NameError: uninitialized constant Shell::Commands::ListQuotaSnapshots::TABLE’
[
https://issues.apache.org/jira/browse/HBASE-26896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Somogyi resolved HBASE-26896.
---
Fix Version/s: 2.5.0
2.6.0
3.0.0-alpha-3
2.4.12
Resolution: Fixed
Merged to branch-2.4+. Thanks [~stoty] for the fix!
> list_quota_snapshots fails with ‘ERROR NameError: uninitialized constant
> Shell::Commands::ListQuotaSnapshots::TABLE’
>
>
> Key: HBASE-26896
> URL: https://issues.apache.org/jira/browse/HBASE-26896
> Project: HBase
> Issue Type: Bug
> Components: shell
>Affects Versions: 2.5.0, 3.0.0-alpha-3, 2.4.11
>Reporter: Istvan Toth
>Assignee: Istvan Toth
>Priority: Major
> Fix For: 2.5.0, 2.6.0, 3.0.0-alpha-3, 2.4.12
>
>
> The list_quota_snapshots command fails with
> {noformat}
> ERROR NameError: uninitialized constant
> Shell::Commands::ListQuotaSnapshots::TABLE{noformat}
> regardless of the parameters it's called with.
> Apparently, it used to work in HBase 2.2.
> I don't know enough about Ruby to tell why it used to work, and what broke
> this exactly, but using qualified constants fixes the problem.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Created] (HBASE-26772) Shell suspended in background
Peter Somogyi created HBASE-26772:
-
Summary: Shell suspended in background
Key: HBASE-26772
URL: https://issues.apache.org/jira/browse/HBASE-26772
Project: HBase
Issue Type: Bug
Components: shell
Reporter: Peter Somogyi
Assignee: Peter Somogyi
The hbase shell process hangs when running in the background with piped input.
{noformat}
$ cat command.sh
#!/bin/bash
echo "list" | bin/hbase shell > shell.out 2>&1
$ ./command.sh &
[1] 62907
$
[1] + 62907 suspended (tty output) ./command.sh{noformat}
This regression was introduced with JRuby 9.1 -> 9.2 upgrade and the process
hangs on this line:
[https://github.com/apache/hbase/blob/master/hbase-shell/src/main/ruby/jar-bootstrap.rb#L41]
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Resolved] (HBASE-26741) Incorrect exception handling in shell
[
https://issues.apache.org/jira/browse/HBASE-26741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Somogyi resolved HBASE-26741.
---
Fix Version/s: 2.5.0
3.0.0-alpha-3
2.4.10
Resolution: Fixed
Merged to branch-2.4+. Thanks for the review [~elserj].
Filed HBASE-26751 to cover the shell exit behavior with tests.
> Incorrect exception handling in shell
> -
>
> Key: HBASE-26741
> URL: https://issues.apache.org/jira/browse/HBASE-26741
> Project: HBase
> Issue Type: Bug
> Components: shell
>Affects Versions: 2.5.0, 3.0.0-alpha-2, 2.4.10
>Reporter: Peter Somogyi
>Assignee: Peter Somogyi
>Priority: Critical
> Fix For: 2.5.0, 3.0.0-alpha-3, 2.4.10
>
>
> The exception handling changed in the shell compared to 2.2.
> {noformat}
> ➜ hbase-upstream git:(branch-2.4) ✗ cat commands.txt
> scan 'foo'
> exit
> ➜ hbase-upstream git:(branch-2.4) ✗ bin/hbase shell -n commands.txt
> 2022-02-07 16:21:33,654 WARN [main] util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> hbase:001:0> scan 'foo'
> ROW COLUMN+CELL
> Took 0.3890 seconds
> Traceback (most
> recent call last):
> RuntimeError (Unknown table foo!)
> hbase:002:0> exit
> ➜ hbase-upstream git:(branch-2.4) ✗ echo $?
> 0 {noformat}
> The execution continues even after an exception is thrown. In 2.2.7 the
> execution stops when an exception is thrown and the shell exits with error
> code.
> {noformat}
> ➜ hbase-2.2.7 bin/hbase shell -n commands.txt
> 2022-02-07 16:33:54,930 WARN [main] util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> ROW COLUMN+CELL
> Took 0.3082 seconds
> RuntimeError:
> Unknown table foo!
> translate_hbase_exceptions at
> /Users/petersomogyi/tmp/hbase-2.2.7/lib/ruby/shell/commands.rb:130
> command_safe at
> /Users/petersomogyi/tmp/hbase-2.2.7/lib/ruby/shell/commands.rb:49
> internal_command at
> /Users/petersomogyi/tmp/hbase-2.2.7/lib/ruby/shell.rb:148
> command at
> /Users/petersomogyi/tmp/hbase-2.2.7/lib/ruby/shell.rb:140
> scan at (eval):2
> at commands.txt:1
> load at org/jruby/RubyKernel.java:973
> at
> /Users/petersomogyi/tmp/hbase-2.2.7/bin/../bin/hirb.rb:186
> ➜ hbase-2.2.7 echo $?
> 1 {noformat}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Created] (HBASE-26751) Tests for shell exit behavior
Peter Somogyi created HBASE-26751: - Summary: Tests for shell exit behavior Key: HBASE-26751 URL: https://issues.apache.org/jira/browse/HBASE-26751 Project: HBase Issue Type: Test Components: shell, test Reporter: Peter Somogyi HBase shell exit behavior and exception handling is not covered with tests. The tests should cover the clean exits and also when a command throws exceptions in interactive and non-interactive modes. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (HBASE-26741) Incorrect exception handling in shell
Peter Somogyi created HBASE-26741:
-
Summary: Incorrect exception handling in shell
Key: HBASE-26741
URL: https://issues.apache.org/jira/browse/HBASE-26741
Project: HBase
Issue Type: Bug
Components: shell
Affects Versions: 3.0.0-alpha-2, 2.5.0, 2.4.10
Reporter: Peter Somogyi
The exception handling changed in the shell compared to 2.2.
{noformat}
➜ hbase-upstream git:(branch-2.4) ✗ cat commands.txt
scan 'foo'
exit
➜ hbase-upstream git:(branch-2.4) ✗ bin/hbase shell -n commands.txt
2022-02-07 16:21:33,654 WARN [main] util.NativeCodeLoader: Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
hbase:001:0> scan 'foo'
ROW COLUMN+CELL
Took 0.3890 seconds
Traceback (most
recent call last):
RuntimeError (Unknown table foo!)
hbase:002:0> exit
➜ hbase-upstream git:(branch-2.4) ✗ echo $?
0 {noformat}
The execution continues even after an exception is thrown. In 2.2.7 the
execution stops when an exception is thrown and the shell exits with error code.
{noformat}
➜ hbase-2.2.7 bin/hbase shell -n commands.txt
2022-02-07 16:33:54,930 WARN [main] util.NativeCodeLoader: Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
ROW COLUMN+CELL
Took 0.3082 seconds
RuntimeError:
Unknown table foo!
translate_hbase_exceptions at
/Users/petersomogyi/tmp/hbase-2.2.7/lib/ruby/shell/commands.rb:130
command_safe at
/Users/petersomogyi/tmp/hbase-2.2.7/lib/ruby/shell/commands.rb:49
internal_command at
/Users/petersomogyi/tmp/hbase-2.2.7/lib/ruby/shell.rb:148
command at
/Users/petersomogyi/tmp/hbase-2.2.7/lib/ruby/shell.rb:140
scan at (eval):2
at commands.txt:1
load at org/jruby/RubyKernel.java:973
at
/Users/petersomogyi/tmp/hbase-2.2.7/bin/../bin/hirb.rb:186
➜ hbase-2.2.7 echo $?
1 {noformat}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Created] (HBASE-26711) Remove 2.3 from Download page
Peter Somogyi created HBASE-26711: - Summary: Remove 2.3 from Download page Key: HBASE-26711 URL: https://issues.apache.org/jira/browse/HBASE-26711 Project: HBase Issue Type: Sub-task Reporter: Peter Somogyi Assignee: Peter Somogyi -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (HBASE-26710) Remove jenkins files from branch-2.3
Peter Somogyi created HBASE-26710: - Summary: Remove jenkins files from branch-2.3 Key: HBASE-26710 URL: https://issues.apache.org/jira/browse/HBASE-26710 Project: HBase Issue Type: Sub-task Components: community Reporter: Peter Somogyi Assignee: Peter Somogyi Remove jenkins files as 2.3 reached EOL. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Resolved] (HBASE-26665) Standalone unit test in hbase-examples
[ https://issues.apache.org/jira/browse/HBASE-26665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Somogyi resolved HBASE-26665. --- Resolution: Fixed Thanks [~andor]. Merged to the feature branch. > Standalone unit test in hbase-examples > -- > > Key: HBASE-26665 > URL: https://issues.apache.org/jira/browse/HBASE-26665 > Project: HBase > Issue Type: Sub-task >Reporter: Josh Elser >Assignee: Andor Molnar >Priority: Major > Fix For: HBASE-26553 > > > Andor is already working on this with nimbus, but filing this for him. > We should have a unit test which exercises the oauth bearer authentication > mechanism so that we know if the feature is functional at a basic level > (without having to set up on OAuth server). -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Resolved] (HBASE-24048) [Flakey Tests] Re-enable TestCustomSaslAuthenticationProvider#testNegativeAuthentication
[ https://issues.apache.org/jira/browse/HBASE-24048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Somogyi resolved HBASE-24048. --- Resolution: Duplicate Fixed with HBASE-26689. > [Flakey Tests] Re-enable > TestCustomSaslAuthenticationProvider#testNegativeAuthentication > > > Key: HBASE-24048 > URL: https://issues.apache.org/jira/browse/HBASE-24048 > Project: HBase > Issue Type: Sub-task > Components: flakies >Reporter: Michael Stack >Priority: Major > > See parent issue which disables flakey test. This task is to reenable once > flakey-ness has been cleared. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Resolved] (HBASE-26689) Backport HBASE-24443 Refactor TestCustomSaslAuthenticationProvider
[ https://issues.apache.org/jira/browse/HBASE-26689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Somogyi resolved HBASE-26689. --- Fix Version/s: 2.5.0 2.4.10 Resolution: Fixed Thanks for the review [~zhangduo]! Merged to branch-2, branch-2.5, branch-2.4, and branch-2.3. > Backport HBASE-24443 Refactor TestCustomSaslAuthenticationProvider > -- > > Key: HBASE-26689 > URL: https://issues.apache.org/jira/browse/HBASE-26689 > Project: HBase > Issue Type: Test > Components: test >Reporter: Peter Somogyi >Assignee: Peter Somogyi >Priority: Minor > Fix For: 2.5.0, 2.4.10 > > > Test execution times for TestCustomSaslAuthenticationProvider on branch-2 > takes 24 minutes. The refactored tests after HBASE-24443 on master succeed in > just under 2 * 2 minutes. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (HBASE-26689) Backport HBASE-24443 Refactor TestCustomSaslAuthenticationProvider
Peter Somogyi created HBASE-26689: - Summary: Backport HBASE-24443 Refactor TestCustomSaslAuthenticationProvider Key: HBASE-26689 URL: https://issues.apache.org/jira/browse/HBASE-26689 Project: HBase Issue Type: Test Components: test Reporter: Peter Somogyi Assignee: Peter Somogyi Test execution times for TestCustomSaslAuthenticationProvider on branch-2 takes 24 minutes. The refactored tests after HBASE-24443 on master succeed in just under 2 * 2 minutes. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Resolved] (HBASE-26682) Shakti Packers and Movers
[ https://issues.apache.org/jira/browse/HBASE-26682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Somogyi resolved HBASE-26682. --- > Shakti Packers and Movers > -- > > Key: HBASE-26682 > URL: https://issues.apache.org/jira/browse/HBASE-26682 > Project: HBase > Issue Type: Bug >Reporter: shaktimovers >Priority: Major > > Shakti movers and packers in Gurugram are a team of professionals who assist > their customers in having a stress-free moving experience. We ensure the > safety of your goods and provide prompt transportation at a low cost to our > clients. Our services are delivered on schedule and in accordance with the > client's requirements. > > [Packers and movers in gurugram|https://moversandpackersingurugram.com/] > [Packers and movers > gurugram|https://moversandpackersingurugram.com/movers-and-packers-in-gurugram/] > [Movers and packers in > gurugram|https://moversandpackersingurugram.com/packers-and-movers-in-gurugram/] > > [Movers and packers > gurugram|https://moversandpackersingurugram.com/packers-and-movers-gurugram/] -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Resolved] (HBASE-11914) YARN tests currently use more than 4GB of memory
[ https://issues.apache.org/jira/browse/HBASE-11914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Somogyi resolved HBASE-11914. --- Resolution: Incomplete Resolving stale issue. > YARN tests currently use more than 4GB of memory > > > Key: HBASE-11914 > URL: https://issues.apache.org/jira/browse/HBASE-11914 > Project: HBase > Issue Type: Sub-task >Reporter: Alex Newman >Assignee: Alex Newman >Priority: Major > > Much of the org.apache.hadoop.hbase.mapreduce patches use more than 4GB of > memory. There's no reason for this. This patch lowers the memory required so > that we can run the tests in wussy CI environments (circleci/travis-ci) -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Resolved] (HBASE-26653) [hbase-operator-tools] Upgrade log4j to 2.17.1
[ https://issues.apache.org/jira/browse/HBASE-26653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Somogyi resolved HBASE-26653. --- Release Note: Upgrade log4j to 2.17.1 to address CVE-2021-44832. Resolution: Fixed Merged to master. > [hbase-operator-tools] Upgrade log4j to 2.17.1 > -- > > Key: HBASE-26653 > URL: https://issues.apache.org/jira/browse/HBASE-26653 > Project: HBase > Issue Type: Task > Components: hbase-operator-tools, logging, security >Affects Versions: hbase-operator-tools-1.2.0 >Reporter: Peter Somogyi >Assignee: Peter Somogyi >Priority: Major > Fix For: hbase-operator-tools-2.0.0 > > > For addressing CVE-2021-44832. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (HBASE-26653) [hbase-operator-tools] Upgrade log4j to 2.17.1
Peter Somogyi created HBASE-26653: - Summary: [hbase-operator-tools] Upgrade log4j to 2.17.1 Key: HBASE-26653 URL: https://issues.apache.org/jira/browse/HBASE-26653 Project: HBase Issue Type: Task Components: hbase-operator-tools, logging, security Affects Versions: hbase-operator-tools-1.2.0 Reporter: Peter Somogyi Assignee: Peter Somogyi Fix For: hbase-operator-tools-2.0.0 For addressing CVE-2021-44832. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Resolved] (HBASE-26608) [hbase-operator-tools] Upgrade log4j2 to 2.17.0
[ https://issues.apache.org/jira/browse/HBASE-26608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Somogyi resolved HBASE-26608. --- Fix Version/s: hbase-operator-tools-1.2.0 Resolution: Fixed > [hbase-operator-tools] Upgrade log4j2 to 2.17.0 > --- > > Key: HBASE-26608 > URL: https://issues.apache.org/jira/browse/HBASE-26608 > Project: HBase > Issue Type: Task > Components: hbase-operator-tools, logging, security >Reporter: Duo Zhang >Assignee: Guangxu Cheng >Priority: Major > Fix For: hbase-operator-tools-1.2.0 > > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Resolved] (HBASE-26570) Upgrade to log4j 2.16.0
[ https://issues.apache.org/jira/browse/HBASE-26570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Somogyi resolved HBASE-26570. --- Resolution: Done > Upgrade to log4j 2.16.0 > --- > > Key: HBASE-26570 > URL: https://issues.apache.org/jira/browse/HBASE-26570 > Project: HBase > Issue Type: Umbrella > Components: dependencies >Reporter: Peter Somogyi >Assignee: Peter Somogyi >Priority: Major > > Log4j just release version 2.16.0 where jndi is turned off by default. Based > on the release announcement it is not required to fix CVE-2021-44228 but > recommended. > [https://lists.apache.org/thread/d6v4r6nosxysyq9rvnr779336yf0woz4] -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (HBASE-26572) Upgrade to log4j 2.16.0
Peter Somogyi created HBASE-26572: - Summary: Upgrade to log4j 2.16.0 Key: HBASE-26572 URL: https://issues.apache.org/jira/browse/HBASE-26572 Project: HBase Issue Type: Sub-task Components: dependencies Reporter: Peter Somogyi Assignee: Peter Somogyi Fix For: 3.0.0-alpha-2 -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (HBASE-26571) [hbase-operator-tools] Upgrade to log4j 2.16.0
Peter Somogyi created HBASE-26571: - Summary: [hbase-operator-tools] Upgrade to log4j 2.16.0 Key: HBASE-26571 URL: https://issues.apache.org/jira/browse/HBASE-26571 Project: HBase Issue Type: Sub-task Components: dependencies, hbase-operator-tools Reporter: Peter Somogyi Assignee: Peter Somogyi -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (HBASE-26570) Upgrade to log4j 2.16.0
Peter Somogyi created HBASE-26570: - Summary: Upgrade to log4j 2.16.0 Key: HBASE-26570 URL: https://issues.apache.org/jira/browse/HBASE-26570 Project: HBase Issue Type: Umbrella Components: dependencies, hbase-operator-tools Reporter: Peter Somogyi Assignee: Peter Somogyi Log4j just release version 2.16.0 where jndi is turned off by default. Based on the release announcement it is not required to fix CVE-2021-44228 but recommended. [https://lists.apache.org/thread/d6v4r6nosxysyq9rvnr779336yf0woz4] -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Resolved] (HBASE-26451) Hbase Export/Import via MapReduce job
[
https://issues.apache.org/jira/browse/HBASE-26451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Somogyi resolved HBASE-26451.
---
Resolution: Not A Bug
> Hbase Export/Import via MapReduce job
> -
>
> Key: HBASE-26451
> URL: https://issues.apache.org/jira/browse/HBASE-26451
> Project: HBase
> Issue Type: Bug
> Components: backuprestore
>Affects Versions: 2.0.1
>Reporter: Christos Dosis
>Priority: Major
>
> Hi Hbase support team,
>
> While using the MapReduce job with export/import commands we have the below
> behaviour.
>
> {+}Step1{+}: I have a hbase table(tsdb) with 3 rows. and export the table
> like below:
> /opt/hbase/bin/hbase org.apache.hadoop.hbase.mapreduce.Export tsdb
> file:///opt/hbase/backup/tsdb
> {+}Step2{+}: Then I delete 1 row.
> {+}Step3{+}: Then I import tsdb table from the exported data from Step 1.
> /opt/hbase/bin/hbase org.apache.hadoop.hbase.mapreduce.Import tsdb
> file:///opt/hbase/backup/tsdb
>
> There are still only 2 rows in the table. Is this a valid behaviour?
>
> Br,
> Chris
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
