[jira] [Created] (HBASE-25281) Bulkload split hfile too many times due to unreasonable split point
niuyulin created HBASE-25281: Summary: Bulkload split hfile too many times due to unreasonable split point Key: HBASE-25281 URL: https://issues.apache.org/jira/browse/HBASE-25281 Project: HBase Issue Type: Improvement Reporter: niuyulin Assignee: niuyulin [https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/tool/BulkLoadHFilesTool.java#L688] the split point is not in middle, If the hfile span N regions, we need split N-1 times now -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-25270) [hbase-thirdparty] Generate CHANGES.md and RELEASENOTES.md for 3.4.1
[ https://issues.apache.org/jira/browse/HBASE-25270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang resolved HBASE-25270. --- Hadoop Flags: Reviewed Resolution: Fixed Merged to master. Thanks [~zghao] for reviewing. > [hbase-thirdparty] Generate CHANGES.md and RELEASENOTES.md for 3.4.1 > > > Key: HBASE-25270 > URL: https://issues.apache.org/jira/browse/HBASE-25270 > Project: HBase > Issue Type: Sub-task > Components: thirdparty >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: thirdparty-3.4.1 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Reopened] (HBASE-25270) [hbase-thirdparty] Generate CHANGES.md and RELEASENOTES.md for 3.4.1
[ https://issues.apache.org/jira/browse/HBASE-25270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang reopened HBASE-25270: --- > [hbase-thirdparty] Generate CHANGES.md and RELEASENOTES.md for 3.4.1 > > > Key: HBASE-25270 > URL: https://issues.apache.org/jira/browse/HBASE-25270 > Project: HBase > Issue Type: Sub-task > Components: thirdparty >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: thirdparty-3.4.1 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-25270) [hbase-thirdparty] Generate CHANGES.md and RELEASENOTES.md for 3.4.1
[ https://issues.apache.org/jira/browse/HBASE-25270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang resolved HBASE-25270. --- Resolution: Fixed For generating changes and releasenotes > [hbase-thirdparty] Generate CHANGES.md and RELEASENOTES.md for 3.4.1 > > > Key: HBASE-25270 > URL: https://issues.apache.org/jira/browse/HBASE-25270 > Project: HBase > Issue Type: Sub-task > Components: thirdparty >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: thirdparty-3.4.1 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-25280) [meta replicas] ArrayIndexOutOfBoundsException in ZKConnectionRegistry
Michael Stack created HBASE-25280:
-
Summary: [meta replicas] ArrayIndexOutOfBoundsException in
ZKConnectionRegistry
Key: HBASE-25280
URL: https://issues.apache.org/jira/browse/HBASE-25280
Project: HBase
Issue Type: Bug
Components: meta replicas
Affects Versions: HBASE-18070
Reporter: Michael Stack
Assignee: Michael Stack
Fix For: 3.0.0-alpha-1, 2.4.0
ITBLL Testing HBASE-18070 feature, [~huaxiangsun] found this:
{code:java}
2020-11-12 19:48:12,358 ERROR org.apache.hadoop.hbase.util.FutureUtils:
Unexpected error caught when processing
CompletableFuturejava.lang.ArrayIndexOutOfBoundsException: Index 3 out of
bounds for length 3 at
org.apache.hadoop.hbase.client.ZKConnectionRegistry.lambda$getMetaRegionLocation$2(ZKConnectionRegistry.java:180)
at
org.apache.hadoop.hbase.util.FutureUtils.lambda$addListener$0(FutureUtils.java:68)
at
java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)
at
java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837)
at
java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
at
java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073)
at
org.apache.hadoop.hbase.client.ZKConnectionRegistry.lambda$getAndConvert$0(ZKConnectionRegistry.java:78)
at
org.apache.hadoop.hbase.util.FutureUtils.lambda$addListener$0(FutureUtils.java:68)
at
java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)
at
java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837)
at
java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
at
java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073)
at
org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:174)
at
org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:342)
at java.base/java.lang.Thread.run(Thread.java:834) {code}
The code has been this way a long time but his running with four replicas seems
to have revealed a race exposed by using replicaid as index.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HBASE-25279) Non-daemon thread in ZKWatcher
Josh Elser created HBASE-25279: -- Summary: Non-daemon thread in ZKWatcher Key: HBASE-25279 URL: https://issues.apache.org/jira/browse/HBASE-25279 Project: HBase Issue Type: Bug Reporter: Josh Elser Assignee: Josh Elser Fix For: 3.0.0-alpha-1 ZKWatcher spawns an ExecutorService which doesn't mark its threads as daemons which will prevent clean shut downs. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-25278) Add option to toggle CACHE_BLOCKS in count.rb
Josh Elser created HBASE-25278: -- Summary: Add option to toggle CACHE_BLOCKS in count.rb Key: HBASE-25278 URL: https://issues.apache.org/jira/browse/HBASE-25278 Project: HBase Issue Type: New Feature Components: shell Reporter: Josh Elser Assignee: Josh Elser Fix For: 3.0.0-alpha-1, 2.4.0 A trick I've found myself doing a couple of times (hat-tip to [~psomogyi]) is to edit table.rb so that the `count` shell command will not instruct RegionServers to not cache any data blocks. This is a quick+dirty way to force a table to be loaded into block cache (i.e. for performance testing). We can easily add another option to avoid having to edit the ruby files. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-25277) postScannerFilterRow impacts Scan performance a lot in HBase 2.x
Pankaj Kumar created HBASE-25277: Summary: postScannerFilterRow impacts Scan performance a lot in HBase 2.x Key: HBASE-25277 URL: https://issues.apache.org/jira/browse/HBASE-25277 Project: HBase Issue Type: Bug Components: Coprocessors, scan Reporter: Pankaj Kumar Assignee: Pankaj Kumar In our test we observed Scan performance is degraded by more than 60% in HBase-2.x as compared to 1.3.x. As per the flamegraph report, RS spent 31% of the time in postScannerFilterRow, however the coprocessors (AccessController, VisibilityController & ConstraintProcessor) does nothing in that hook. HBASE-14489 added the logic to avoid the call to postScannerFilterRow when not needed which is not working as expected in HBase 2.x. AccessController, VisibilityController & ConstraintProcessor override the postScannerFilterRow with dummy (same as RegionObserver) implementation, so RegionCoprocessorHost.hasCustomPostScannerFilterRow will be TRUE and call the hook for all configured CPs while processing each row . Suppose we have configured 5 region CPs and there are 1 M rows in table, then there will be 1 * 5 M dummy call to postScannerFilterRow during whole table scan. We need to remove postScannerFilterRow hook from these CPs as these are not doing anything. Another problem is in RegionCoprocessorHost.hasCustomPostScannerFilterRow init logic, currently it is always TRUE even though we remove postScannerFilterRow hook from AccessController, VisibilityController & ConstraintProcessor, because we are finding postScannerFilterRow until it is found (we look in configured CP's super class also) or clazz is NULL. https://github.com/apache/hbase/blob/035c192eb665469ce0c071db86c78f4a873c123b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionCoprocessorHost.java#L301 Supper class of Object (super class of AccessController) will be NULL, so RegionCoprocessorHost.hasCustomPostScannerFilterRow will be set as TRUE https://github.com/apache/hbase/blob/035c192eb665469ce0c071db86c78f4a873c123b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionCoprocessorHost.java#L279 -- This message was sent by Atlassian Jira (v8.3.4#803005)
How can I do a query for specific columns by regex-statement
Hi,
I have several columns by the schema family:qualifier. Here are the columns
with the values for the first row:
b'data:name': b'test33',
b'data:appender.0.more.deep.0.name': b'nested',
b'data:appender.0.more.deep.0.value': b'4711',
b'data:appender.0.more.deep.1.name': b'nested2',
b'data:appender.0.more.deep.1.value': b'42',
b'data:appender.0.more.name': b'Alex',
b'data:appender.0.more.process': b'buy',
b'data:appender.0.more.value': b'1.0',
b'data:appender.0.id': b'32542352352354432',
b'data:appender.1.more.name': b'Andrew',
b'data:appender.1.more.process': b'sell',
b'data:appender.1.more.value': b'20.0',
b'data:appender.1.id': b'deaf3232432',
…
b'data:appender.47110.more.deep.name': b'3rfsdf3',
b'data:appender.47110.more.deep.value': b'412',
b'data:appender.47110.more.name': b'Jenny',
b'data:appender.47110.more.process': b'buy',
b'data:appender.47110.more.value': b'10.0',
b'data:date': b'2020-11-02 12:18:49.479229',
b'data:ob.0.cd': b'902359874598459834985093',
b'data:ob.0.n': b'235a3ff4234234',
b'data:s.c': b'hco',
b'data:s.s.p': b'nm',
b'data:id': b'5345345'
The question is how can I query the row with happybase where at least one
column 'data:appender.NumberX.more.deep.NumberY.value' has for example the
value 42. The maximum number of NumberX or NumberY is not known to me and can
be different in each row. NumberY need not necessarily exist in each row as you
can see in the example above. The same applies to NumberX.
I don't know how do I query this because I am new to HBase. My Idea: First I
would to detect the numbers for the columnnames like this
re.findall(".[0-9]*.", key) and then replace it with a dot. I want to create a
query like this data:appender.more.deep.name that looks in all possible columns
(e.g. in 'data:appender.47110.more.deep.value' or
'data:appender.0.more.deep.0.value' or 'data:appender.0.more.deep.1.value'.
How is this (in python) possible to do this query or how the filter looks like?
Best thanks in advance.
Best regards,
John
[jira] [Resolved] (HBASE-25253) Deprecated master carrys regions related methods and configs
[ https://issues.apache.org/jira/browse/HBASE-25253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang resolved HBASE-25253. --- Hadoop Flags: Reviewed Resolution: Fixed Pushed to master and branch-2. Thanks [~wenfeiyi666] for contributing. Please fill the release note about which methods and configs are deprecated. > Deprecated master carrys regions related methods and configs > > > Key: HBASE-25253 > URL: https://issues.apache.org/jira/browse/HBASE-25253 > Project: HBase > Issue Type: Sub-task > Components: Balancer, master >Reporter: Duo Zhang >Assignee: wenfeiyi666 >Priority: Major > Fix For: 3.0.0-alpha-1, 2.4.0 > > > And also mention that this feature will be removed in 3.0.0. > This is the first step to undo master carrys regions. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-25272) Support scan on a specific replica
[ https://issues.apache.org/jira/browse/HBASE-25272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang resolved HBASE-25272. --- Hadoop Flags: Reviewed Resolution: Fixed Merged to branch-2. Thanks all for reviewing. > Support scan on a specific replica > -- > > Key: HBASE-25272 > URL: https://issues.apache.org/jira/browse/HBASE-25272 > Project: HBase > Issue Type: Improvement > Components: Client, scan >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Critical > Fix For: 2.4.0 > > > This is a missing part of the client library for sync client on branch-2, and > it is necessary when implementing meta replicas read. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-25267) Add SSL keystore type and truststore related configs for HBase RESTServer
[
https://issues.apache.org/jira/browse/HBASE-25267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Balazs Meszaros resolved HBASE-25267.
-
Fix Version/s: 2.3.4
2.2.7
2.4.0
3.0.0-alpha-1
Resolution: Fixed
> Add SSL keystore type and truststore related configs for HBase RESTServer
> -
>
> Key: HBASE-25267
> URL: https://issues.apache.org/jira/browse/HBASE-25267
> Project: HBase
> Issue Type: Improvement
> Components: REST
>Reporter: Mate Szalay-Beko
>Assignee: Mate Szalay-Beko
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.4.0, 2.2.7, 2.3.4
>
>
> The RESTServer currently relies on the following parameters to configure SSL
> on the REST API:
> * {{hbase.rest.ssl.enabled}}
> * {{hbase.rest.ssl.keystore.store}}
> * {{hbase.rest.ssl.keystore.password}}
> * {{hbase.rest.ssl.keystore.keypassword}}
> * {{hbase.rest.ssl.exclude.cipher.suites}}
> * {{hbase.rest.ssl.include.cipher.suites}}
> * {{hbase.rest.ssl.exclude.protocols}}
> * {{hbase.rest.ssl.include.protocols}}
> In this patch I want to introduce the following new parameters:
> * {{hbase.rest.ssl.keystore.type}}
> * {{hbase.rest.ssl.truststore.store}}
> * {{hbase.rest.ssl.truststore.password}}
> * {{hbase.rest.ssl.truststore.type}}
> If any of the new the parameter is not provided, then we should fall-back to
> the current behaviour (e.g. assuming JKS keystore/truststore types, or no
> passwords, or no custom trust store file).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Re: Failure: HBase Generate Website
I have a Pull Request to fix the website build waiting for reviews. https://issues.apache.org/jira/browse/HBASE-25275 On Wed, Nov 11, 2020 at 9:18 PM Apache Jenkins Server < jenk...@builds.apache.org> wrote: > > Build status: FAILURE > > The HBase website has not been updated to incorporate recent HBase changes. > > See > https://ci-hadoop.apache.org/job/HBase/job/hbase_generate_website/90/console >
[jira] [Resolved] (HBASE-25003) Backport HBASE-24350 and HBASE-24779 to branch-2.2 & branch-2.3
[ https://issues.apache.org/jira/browse/HBASE-25003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wellington Chevreuil resolved HBASE-25003. -- Resolution: Fixed > Backport HBASE-24350 and HBASE-24779 to branch-2.2 & branch-2.3 > --- > > Key: HBASE-25003 > URL: https://issues.apache.org/jira/browse/HBASE-25003 > Project: HBase > Issue Type: Improvement > Components: Replication >Affects Versions: 2.3.1, 2.2.5 >Reporter: Tamas Adami >Assignee: Tamas Adami >Priority: Major > Fix For: 2.2.7, 2.3.4 > > > Backport HBASE-24350 and HBASE-24779 to branch-2.2 & branch-2.3. > HBASE-24779 was written on top of HBASE-24350. Whilst it is possible to > manually resolve the conflict to exclude the piece of code introduced by > HBASE-24350, ideally we should also include HBASE-24350 in this package of > fixes, as it deals with some inconsistencies about table level metrics for > replication. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-25276) Need to throw the original exception in HRegion#openHRegion
[
https://issues.apache.org/jira/browse/HBASE-25276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guanghao Zhang resolved HBASE-25276.
Fix Version/s: 2.3.4
2.2.7
2.4.0
3.0.0-alpha-1
Resolution: Fixed
Pushed to branch-2.2+. Thanks [~niuyulin] for contributing.
> Need to throw the original exception in HRegion#openHRegion
> ---
>
> Key: HBASE-25276
> URL: https://issues.apache.org/jira/browse/HBASE-25276
> Project: HBase
> Issue Type: Bug
>Reporter: Guanghao Zhang
>Assignee: niuyulin
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.4.0, 2.2.7, 2.3.4
>
>
> {code:java}
> protected HRegion openHRegion(final CancelableProgressable reporter)
> throws IOException {
> try {
> ..
> } catch(Throwable t) {
> // By coprocessor path wrong region will open failed,
> // MetricsRegionWrapperImpl is already init and not close,
> // add region close when open failed
> this.close(); // This may throw IOException too.
> throw t;
> }
> return this;
> }
> {code}
> https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java#L8144
> this.close() may throw IOException too. The original exception will not be
> recorded.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
