[jira] [Resolved] (HBASE-24705) MetaFixer#fixHoles() does not include the case for read replicas (i.e, replica regions are not created)
[ https://issues.apache.org/jira/browse/HBASE-24705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huaxiang Sun resolved HBASE-24705. -- Fix Version/s: 2.4.0 2.3.1 3.0.0-alpha-1 Resolution: Fixed > MetaFixer#fixHoles() does not include the case for read replicas (i.e, > replica regions are not created) > --- > > Key: HBASE-24705 > URL: https://issues.apache.org/jira/browse/HBASE-24705 > Project: HBase > Issue Type: Bug > Components: read replicas >Affects Versions: 2.3.0 >Reporter: Huaxiang Sun >Assignee: Huaxiang Sun >Priority: Major > Fix For: 3.0.0-alpha-1, 2.3.1, 2.4.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] huaxiangsun merged pull request #2067: Backport: HBASE-24705 MetaFixer#fixHoles() does not include the case for read r…
huaxiangsun merged pull request #2067: URL: https://github.com/apache/hbase/pull/2067 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] huaxiangsun merged pull request #2068: Backport: HBASE-24705 MetaFixer#fixHoles() does not include the case for read r…
huaxiangsun merged pull request #2068: URL: https://github.com/apache/hbase/pull/2068 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-24742) Improve performance of SKIP vs SEEK logic
[ https://issues.apache.org/jira/browse/HBASE-24742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17157895#comment-17157895 ] Lars Hofhansl commented on HBASE-24742: --- Upon second thought. Perhaps a seek (not a reseek, but a seek that could actually goes backwards) could make it so that previousIndexedKey and nextIndexedKey are accidentally the same and we *still* would have to do the compare. In my test the majority of the improvement came from the first change. > Improve performance of SKIP vs SEEK logic > - > > Key: HBASE-24742 > URL: https://issues.apache.org/jira/browse/HBASE-24742 > Project: HBase > Issue Type: Bug > Components: Performance, regionserver >Affects Versions: 3.0.0-alpha-1, 1.7.0, 2.4.0 >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl >Priority: Major > Attachments: hbase-1.6-regression-flame-graph.png, > hbase-24742-branch-1.txt > > > In our testing of HBase 1.3 against the current tip of branch-1 we saw a 30% > slowdown in scanning scenarios. > We tracked it back to HBASE-17958 and HBASE-19863. > Both add comparisons to one of the tightest HBase has. > [~bharathv] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-24744) enable_table_replication command granting permissions on table automatically for the user
Dhanalakshmi Periyalwar created HBASE-24744:
---
Summary: enable_table_replication command granting permissions on
table automatically for the user
Key: HBASE-24744
URL: https://issues.apache.org/jira/browse/HBASE-24744
Project: HBase
Issue Type: Bug
Components: acl, security
Affects Versions: 2.1.0
Reporter: Dhanalakshmi Periyalwar
While enabling the table replication for the user table as an hbase user using
the "enable_table_replication" command, permission has been granted
automatically for the hbase user and getting listed in hbase:acl. The same
behaviour is applicable to other users too.
Issue Replication Steps:
hbase(main):001:0> whoami
dhana (auth:SIMPLE)
groups: dhana
Took 0.0214 seconds
hbase(main):002:0> list
TABLE
0 row(s)
Took 0.4268 seconds
=> []
hbase(main):003:0> create 'mytab','f1'
Created table mytab
Took 0.7834 seconds
=> Hbase::Table - mytab
hbase(main):004:0> describe 'mytab'
Table mytab is ENABLED
mytab
COLUMN FAMILIES DESCRIPTION
\{NAME => 'f1', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false',
NEW_VERSION_BEHAVIOR => 'false', KE
EP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false',
DATA_BLOCK_ENCODING => 'NONE', TTL => 'F
OREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW',
CACHE_INDEX_ON_WRITE =>
'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false',
PREFETCH_BLOCKS_ON_OPEN => 'false',
COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}
1 row(s)
Took 0.1319 seconds
hbase(main):005:0> scan 'hbase:acl'
ROWCOLUMN+CELL
hbase:acl column=l:dhana, timestamp=1593669605273, value=RWXCA
mytab column=l:dhana, timestamp=1593673200269, value=RWXCA
2 row(s)
Took 0.0969 seconds
hbase(main):006:0> exit
hbase(main):001:0> whoami
hbase (auth:SIMPLE)
groups: hbase
Took 0.0271 seconds
hbase(main):002:0> scan 'hbase:acl'
ROWCOLUMN+CELL
hbase:acl column=l:dhana, timestamp=1593669605273, value=RWXCA
mytab column=l:dhana, timestamp=1593673200269, value=RWXCA
2 row(s)
Took 0.5223 seconds
hbase(main):003:0> enable_table_replication 'mytab'
The replication of table 'mytab' successfully enabled
Took 16.0711 seconds
hbase(main):004:0> scan 'hbase:acl'
ROWCOLUMN+CELL
hbase:acl column=l:dhana, timestamp=1593669605273, value=RWXCA
mytab column=l:dhana, timestamp=1593673200269, value=RWXCA
mytab column=l:hbase, timestamp=1593673390976, value=RWXCA
<
2 row(s)
Took 0.0089 seconds
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] Reidddddd commented on pull request #2065: HBASE-24739 [Build] branch-1's build seems broken because of pylint
Reidd commented on pull request #2065: URL: https://github.com/apache/hbase/pull/2065#issuecomment-658552896 ping @busbey I'm not sure if I did it right, would you mind taking a look? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2065: HBASE-24739 [Build] branch-1's build seems broken because of pylint
Apache-HBase commented on pull request #2065: URL: https://github.com/apache/hbase/pull/2065#issuecomment-658552057 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 12m 29s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +0 :ok: | hadolint | 0m 0s | hadolint was not available. | | +0 :ok: | shelldocs | 0m 0s | Shelldocs was not available. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ branch-1 Compile Tests _ | | +0 :ok: | mvndep | 2m 23s | Maven dependency ordering for branch | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 8s | Maven dependency ordering for patch | | +1 :green_heart: | shellcheck | 0m 1s | There were no new shellcheck issues. | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | ||| _ Other Tests _ | | +0 :ok: | asflicense | 0m 0s | ASF License check generated no output? | | | | 15m 53s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2065/3/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2065 | | Optional Tests | dupname asflicense hadolint shellcheck shelldocs | | uname | Linux 9f1fd0cb07e2 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/Base-PreCommit-GitHub-PR_PR-2065/out/precommit/personality/provided.sh | | git revision | branch-1 / 0841f12 | | Max. process+thread count | 36 (vs. ulimit of 1) | | modules | C: U: | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2065/3/console | | versions | git=1.9.1 maven=3.0.5 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-24742) Improve performance of SKIP vs SEEK logic
[ https://issues.apache.org/jira/browse/HBASE-24742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17157890#comment-17157890 ] Lars Hofhansl commented on HBASE-24742: --- [~zhangduo] I'm with you there. The second part was harder to reason about, and I feel a bit less easy about it. In the end it's an optimization to save a comparison, previousIndexedKey and nextIndexedKey will never accidentally be the same (as in identical), so I *think* it should OK. I'm not sure how we can avoid passing the fake keys up, since it is designed to handle things and the "upper" heap. At least not without a lot refactoring. [~apurtell] I'll take a look at HBASE-24637 (I'm a bit thinly spread, though) > Improve performance of SKIP vs SEEK logic > - > > Key: HBASE-24742 > URL: https://issues.apache.org/jira/browse/HBASE-24742 > Project: HBase > Issue Type: Bug > Components: Performance, regionserver >Affects Versions: 3.0.0-alpha-1, 1.7.0, 2.4.0 >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl >Priority: Major > Attachments: hbase-1.6-regression-flame-graph.png, > hbase-24742-branch-1.txt > > > In our testing of HBase 1.3 against the current tip of branch-1 we saw a 30% > slowdown in scanning scenarios. > We tracked it back to HBASE-17958 and HBASE-19863. > Both add comparisons to one of the tightest HBase has. > [~bharathv] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #2065: HBASE-24739 [Build] branch-1's build seems broken because of pylint
Apache-HBase commented on pull request #2065: URL: https://github.com/apache/hbase/pull/2065#issuecomment-658547990 (!) A patch to the testing environment has been detected. Re-executing against the patched versions to perform further tests. The console is at https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2065/3/console in case of problems. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-24734) Wrong comparator opening Region when 'split-to-WAL' enabled.
[
https://issues.apache.org/jira/browse/HBASE-24734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17157873#comment-17157873
]
Michael Stack commented on HBASE-24734:
---
Oh, I missed that is new code that came in w/ 'HBASE-23741 Data loss when WAL
split to HFile enabled (#1254)'. I was looking at hfile creation instead of at
the actual stack trace. Thanks [~huaxiangsun] Let me address.
> Wrong comparator opening Region when 'split-to-WAL' enabled.
>
>
> Key: HBASE-24734
> URL: https://issues.apache.org/jira/browse/HBASE-24734
> Project: HBase
> Issue Type: Sub-task
> Components: HFile, MTTR
>Reporter: Michael Stack
>Priority: Major
>
> Came across this when we were testing the 'split-to-hfile' feature running
> ITBLL:
>
> {code:java}
> 2020-07-10 10:16:49,983 INFO org.apache.hadoop.hbase.regionserver.HRegion:
> Closing region hbase:meta,,1.15882307402020-07-10 10:16:49,997 INFO
> org.apache.hadoop.hbase.regionserver.HRegion: Closed
> hbase:meta,,1.15882307402020-07-10 10:16:49,998 WARN
> org.apache.hadoop.hbase.regionserver.handler.AssignRegionHandler: Fatal error
> occurred while opening region hbase:meta,,1.1588230740,
> aborting...java.lang.IllegalArgumentException: Invalid range:
> IntegrationTestBigLinkedList,,1594350463222.8f89e01a5245e79946e22d8a8ab4698b.
> >
> IntegrationTestBigLinkedList,\x10\x02J\xA1,1594349535271.be24dc276f686e6dcc7fb9d3f91c8387.
> at
> org.apache.hadoop.hbase.client.RegionInfoBuilder$MutableRegionInfo.containsRange(RegionInfoBuilder.java:300)
> at
> org.apache.hadoop.hbase.regionserver.HStore.tryCommitRecoveredHFile(HStore.java:)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.loadRecoveredHFilesIfAny(HRegion.java:5442)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals(HRegion.java:1010)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:950)
>at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7490)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegionFromTableDir(HRegion.java:7448)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7424)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7382)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7333)
> at
> org.apache.hadoop.hbase.regionserver.handler.AssignRegionHandler.process(AssignRegionHandler.java:135)
> at
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)2020-07-10
> 10:16:50,005 ERROR org.apache.hadoop.hbase.regionserver.HRegionServer: *
> ABORTING region server hbasedn149.example.org,16020,1594375563853: Failed to
> open region hbase:meta,,1.1588230740 and can not recover
> *java.lang.IllegalArgumentException: Invalid range:
> IntegrationTestBigLinkedList,,1594350463222.8f89e01a5245e79946e22d8a8ab4698b.
> >
> IntegrationTestBigLinkedList,\x10\x02J\xA1,1594349535271.be24dc276f686e6dcc7fb9d3f91c8387.
> {code}
> Seems basic case of wrong comparator. Below passes if I use the meta
> comparator
> {code:java}
> @Test
> public void testBinaryKeys() throws Exception {
> Set set = new TreeSet<>(CellComparatorImpl.COMPARATOR);
> final byte [] fam = Bytes.toBytes("col");
> final byte [] qf = Bytes.toBytes("umn");
> final byte [] nb = new byte[0];
> Cell [] keys = {
> createByteBufferKeyValueFromKeyValue(
> new KeyValue(Bytes.toBytes("a,\u\u,2"), fam, qf, 2,
> nb)),
> createByteBufferKeyValueFromKeyValue(
> new KeyValue(Bytes.toBytes("a,\u0001,3"), fam, qf, 3, nb)),
> createByteBufferKeyValueFromKeyValue(
> new KeyValue(Bytes.toBytes("a,,1"), fam, qf, 1, nb)),
> createByteBufferKeyValueFromKeyValue(
> new KeyValue(Bytes.toBytes("a,\u1000,5"), fam, qf, 5, nb)),
> createByteBufferKeyValueFromKeyValue(
> new KeyValue(Bytes.toBytes("a,a,4"), fam, qf, 4, nb)),
> createByteBufferKeyValueFromKeyValue(
> new KeyValue(Bytes.toBytes("a,a,0"), fam, qf, 0, nb)),
> };
> // Add to set with bad comparator
> Collections.addAll(set, keys);
> // This will output the keys incorrectly.
> boolean assertion = false;
> int count = 0;
> try {
> for (Cell k: set) {
> assertTrue("count=" + count + ", " + k.toString(), count++ ==
> k.getTimestamp());
> }
> } catch
[GitHub] [hbase] Apache-HBase commented on pull request #2068: Backport: HBASE-24705 MetaFixer#fixHoles() does not include the case for read r…
Apache-HBase commented on pull request #2068: URL: https://github.com/apache/hbase/pull/2068#issuecomment-658539784 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 3m 2s | Docker mode activated. | | -0 :warning: | yetus | 0m 7s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2 Compile Tests _ | | +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 51s | branch-2 passed | | +1 :green_heart: | compile | 1m 25s | branch-2 passed | | +1 :green_heart: | shadedjars | 5m 32s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 3s | branch-2 passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 52s | the patch passed | | +1 :green_heart: | compile | 1m 25s | the patch passed | | +1 :green_heart: | javac | 1m 25s | the patch passed | | +1 :green_heart: | shadedjars | 5m 33s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 0s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 2m 31s | hbase-client in the patch passed. | | +1 :green_heart: | unit | 197m 3s | hbase-server in the patch passed. | | | | 229m 2s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2068/1/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2068 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 81df1ad07eef 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / d3ec4886a1 | | Default Java | 1.8.0_232 | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2068/1/testReport/ | | Max. process+thread count | 2509 (vs. ulimit of 12500) | | modules | C: hbase-client hbase-server U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2068/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] bsglz commented on pull request #2054: HBASE-24664 Some changing of split region by overall region size rath…
bsglz commented on pull request #2054: URL: https://github.com/apache/hbase/pull/2054#issuecomment-658538577 @wchevreuil Could you help to merge this PR? Thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2064: HBASE-24735: Refactor ReplicationSourceManager: move logPositionAndCl…
Apache-HBase commented on pull request #2064: URL: https://github.com/apache/hbase/pull/2064#issuecomment-658529233 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 22s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ HBASE-24666 Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 13s | HBASE-24666 passed | | +1 :green_heart: | checkstyle | 1m 16s | HBASE-24666 passed | | +1 :green_heart: | spotbugs | 2m 12s | HBASE-24666 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 8s | the patch passed | | -0 :warning: | checkstyle | 1m 11s | hbase-server: The patch generated 1 new + 9 unchanged - 6 fixed = 10 total (was 15) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 12m 40s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. | | +1 :green_heart: | spotbugs | 2m 23s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 13s | The patch does not generate ASF License warnings. | | | | 37m 32s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2064/2/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2064 | | JIRA Issue | HBASE-24735 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle | | uname | Linux 4fed18631ec3 4.15.0-91-generic #92-Ubuntu SMP Fri Feb 28 11:09:48 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | HBASE-24666 / 9e8c930feb | | checkstyle | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2064/2/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | Max. process+thread count | 84 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2064/2/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (HBASE-11288) Splittable Meta
[ https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17157846#comment-17157846 ] Francis Christopher Liu edited comment on HBASE-11288 at 7/15/20, 3:39 AM: --- I am all for “collaborate and work out a path forward”. How do we go about doing that? My previous proposal was trying to add some rigor by tackling the most pressing issue (in terms of picking one over the other) which seemed based on earlier discussions and the documentation was tiering. Any proposal? I have some technical responses to the recents posts here but will try to hold off until we can agree on a path forward. (78 words) was (Author: toffer): I am all for “collaborate and work out a path forward”. How do we go about doing that? My previous proposal was tackling the most pressing issue which seemed based on earlier discussions and the documentation was tiering. Any proposal? I have some technical responses to the recents posts here but will try to hold off until we can agree on a path forward. (64 words) > Splittable Meta > --- > > Key: HBASE-11288 > URL: https://issues.apache.org/jira/browse/HBASE-11288 > Project: HBase > Issue Type: Umbrella > Components: meta >Reporter: Francis Christopher Liu >Assignee: Francis Christopher Liu >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #2067: Backport: HBASE-24705 MetaFixer#fixHoles() does not include the case for read r…
Apache-HBase commented on pull request #2067: URL: https://github.com/apache/hbase/pull/2067#issuecomment-658525655 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 47s | Docker mode activated. | | -0 :warning: | yetus | 0m 7s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2.3 Compile Tests _ | | +0 :ok: | mvndep | 0m 17s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 39s | branch-2.3 passed | | +1 :green_heart: | compile | 1m 25s | branch-2.3 passed | | +1 :green_heart: | shadedjars | 5m 15s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 59s | branch-2.3 passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 17s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 19s | the patch passed | | +1 :green_heart: | compile | 1m 23s | the patch passed | | +1 :green_heart: | javac | 1m 23s | the patch passed | | +1 :green_heart: | shadedjars | 5m 6s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 0s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 2m 10s | hbase-client in the patch passed. | | +1 :green_heart: | unit | 139m 53s | hbase-server in the patch passed. | | | | 168m 48s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2067/1/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2067 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 06d24e447551 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2.3 / 15c8c2f59c | | Default Java | 1.8.0_232 | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2067/1/testReport/ | | Max. process+thread count | 3918 (vs. ulimit of 12500) | | modules | C: hbase-client hbase-server U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2067/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-11288) Splittable Meta
[ https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17157846#comment-17157846 ] Francis Christopher Liu commented on HBASE-11288: - I am all for “collaborate and work out a path forward”. How do we go about doing that? My previous proposal was tackling the most pressing issue which seemed based on earlier discussions and the documentation was tiering. Any proposal? I have some technical responses to the recents posts here but will try to hold off until we can agree on a path forward. (64 words) > Splittable Meta > --- > > Key: HBASE-11288 > URL: https://issues.apache.org/jira/browse/HBASE-11288 > Project: HBase > Issue Type: Umbrella > Components: meta >Reporter: Francis Christopher Liu >Assignee: Francis Christopher Liu >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #2068: Backport: HBASE-24705 MetaFixer#fixHoles() does not include the case for read r…
Apache-HBase commented on pull request #2068: URL: https://github.com/apache/hbase/pull/2068#issuecomment-658523348 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 34s | Docker mode activated. | | -0 :warning: | yetus | 0m 6s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2 Compile Tests _ | | +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 8s | branch-2 passed | | +1 :green_heart: | compile | 1m 33s | branch-2 passed | | +1 :green_heart: | shadedjars | 5m 51s | branch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 30s | hbase-client in branch-2 failed. | | -0 :warning: | javadoc | 0m 39s | hbase-server in branch-2 failed. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 20s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 52s | the patch passed | | +1 :green_heart: | compile | 1m 35s | the patch passed | | +1 :green_heart: | javac | 1m 35s | the patch passed | | +1 :green_heart: | shadedjars | 5m 46s | patch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 27s | hbase-client in the patch failed. | | -0 :warning: | javadoc | 0m 39s | hbase-server in the patch failed. | ||| _ Other Tests _ | | +1 :green_heart: | unit | 2m 33s | hbase-client in the patch passed. | | +1 :green_heart: | unit | 128m 5s | hbase-server in the patch passed. | | | | 159m 24s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2068/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2068 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 8b8746fd6f22 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / d3ec4886a1 | | Default Java | 2020-01-14 | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2068/1/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-client.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2068/1/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2068/1/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-client.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2068/1/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2068/1/testReport/ | | Max. process+thread count | 3729 (vs. ulimit of 12500) | | modules | C: hbase-client hbase-server U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2068/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2067: Backport: HBASE-24705 MetaFixer#fixHoles() does not include the case for read r…
Apache-HBase commented on pull request #2067: URL: https://github.com/apache/hbase/pull/2067#issuecomment-658522935 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 29s | Docker mode activated. | | -0 :warning: | yetus | 0m 7s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2.3 Compile Tests _ | | +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 11s | branch-2.3 passed | | +1 :green_heart: | compile | 1m 33s | branch-2.3 passed | | +1 :green_heart: | shadedjars | 5m 44s | branch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 30s | hbase-client in branch-2.3 failed. | | -0 :warning: | javadoc | 0m 38s | hbase-server in branch-2.3 failed. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 21s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 53s | the patch passed | | +1 :green_heart: | compile | 1m 35s | the patch passed | | +1 :green_heart: | javac | 1m 35s | the patch passed | | +1 :green_heart: | shadedjars | 5m 47s | patch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 28s | hbase-client in the patch failed. | | -0 :warning: | javadoc | 0m 40s | hbase-server in the patch failed. | ||| _ Other Tests _ | | +1 :green_heart: | unit | 2m 30s | hbase-client in the patch passed. | | +1 :green_heart: | unit | 125m 54s | hbase-server in the patch passed. | | | | 157m 52s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2067/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2067 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 28fc80aef15f 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2.3 / 15c8c2f59c | | Default Java | 2020-01-14 | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2067/1/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-client.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2067/1/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2067/1/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-client.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2067/1/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2067/1/testReport/ | | Max. process+thread count | 3939 (vs. ulimit of 12500) | | modules | C: hbase-client hbase-server U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2067/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2069: HBASE-24615 MutableRangeHistogram#updateSnapshotRangeMetrics doesn't calculate the distribution for last bucket.
Apache-HBase commented on pull request #2069: URL: https://github.com/apache/hbase/pull/2069#issuecomment-658522077 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 3m 45s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ branch-2 Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 16s | branch-2 passed | | +1 :green_heart: | checkstyle | 0m 14s | branch-2 passed | | +1 :green_heart: | spotbugs | 0m 37s | branch-2 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 41s | the patch passed | | -0 :warning: | checkstyle | 0m 12s | hbase-hadoop2-compat: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 12m 59s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. | | +1 :green_heart: | spotbugs | 0m 39s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 12s | The patch does not generate ASF License warnings. | | | | 34m 47s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2069/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2069 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle | | uname | Linux caefadb3fdfa 4.15.0-91-generic #92-Ubuntu SMP Fri Feb 28 11:09:48 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / d3ec4886a1 | | checkstyle | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2069/1/artifact/yetus-general-check/output/diff-checkstyle-hbase-hadoop2-compat.txt | | Max. process+thread count | 84 (vs. ulimit of 12500) | | modules | C: hbase-hadoop2-compat U: hbase-hadoop2-compat | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2069/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (HBASE-24743) Reject to add a peer which replicate to itself earlier
[
https://issues.apache.org/jira/browse/HBASE-24743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guanghao Zhang reassigned HBASE-24743:
--
Assignee: Guanghao Zhang
> Reject to add a peer which replicate to itself earlier
> --
>
> Key: HBASE-24743
> URL: https://issues.apache.org/jira/browse/HBASE-24743
> Project: HBase
> Issue Type: Improvement
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Major
>
> Now there are one check in ReplicationSource#initialize method
> {code:java}
> // In rare case, zookeeper setting may be messed up. That leads to the
> incorrect
> // peerClusterId value, which is the same as the source clusterId
> if (clusterId.equals(peerClusterId) &&
> !replicationEndpoint.canReplicateToSameCluster()) {
> this.terminate("ClusterId " + clusterId + " is replicating to itself:
> peerClusterId "
> + peerClusterId + " which is not allowed by ReplicationEndpoint:"
> + replicationEndpoint.getClass().getName(), null, false);
> this.manager.removeSource(this);
> return;
> }
> {code}
> This check should move to AddPeerProcedure's precheck.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #2069: HBASE-24615 MutableRangeHistogram#updateSnapshotRangeMetrics doesn't calculate the distribution for last bucket.
Apache-HBase commented on pull request #2069: URL: https://github.com/apache/hbase/pull/2069#issuecomment-658521086 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 46s | Docker mode activated. | | -0 :warning: | yetus | 0m 6s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2 Compile Tests _ | | +1 :green_heart: | mvninstall | 5m 46s | branch-2 passed | | +1 :green_heart: | compile | 0m 22s | branch-2 passed | | +1 :green_heart: | shadedjars | 7m 45s | branch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 23s | hbase-hadoop2-compat in branch-2 failed. | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 5m 22s | the patch passed | | +1 :green_heart: | compile | 0m 25s | the patch passed | | +1 :green_heart: | javac | 0m 25s | the patch passed | | +1 :green_heart: | shadedjars | 8m 8s | patch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 21s | hbase-hadoop2-compat in the patch failed. | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 44s | hbase-hadoop2-compat in the patch passed. | | | | 31m 18s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2069/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2069 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 58b342dc0f54 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / d3ec4886a1 | | Default Java | 2020-01-14 | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2069/1/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-hadoop2-compat.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2069/1/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-hadoop2-compat.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2069/1/testReport/ | | Max. process+thread count | 267 (vs. ulimit of 12500) | | modules | C: hbase-hadoop2-compat U: hbase-hadoop2-compat | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2069/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (HBASE-24743) Reject to add a peer which replicate to itself earlier
Guanghao Zhang created HBASE-24743:
--
Summary: Reject to add a peer which replicate to itself earlier
Key: HBASE-24743
URL: https://issues.apache.org/jira/browse/HBASE-24743
Project: HBase
Issue Type: Improvement
Reporter: Guanghao Zhang
Now there are one check in ReplicationSource#initialize method
{code:java}
// In rare case, zookeeper setting may be messed up. That leads to the incorrect
// peerClusterId value, which is the same as the source clusterId
if (clusterId.equals(peerClusterId) &&
!replicationEndpoint.canReplicateToSameCluster()) {
this.terminate("ClusterId " + clusterId + " is replicating to itself:
peerClusterId "
+ peerClusterId + " which is not allowed by ReplicationEndpoint:"
+ replicationEndpoint.getClass().getName(), null, false);
this.manager.removeSource(this);
return;
}
{code}
This check should move to AddPeerProcedure's precheck.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-24578) [WAL] Add a parameter to config RingBufferEventHandler's SyncFuture count
[
https://issues.apache.org/jira/browse/HBASE-24578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17157842#comment-17157842
]
Reid Chan commented on HBASE-24578:
---
Yeah, i'm trying to fix it HBASE-24739
> [WAL] Add a parameter to config RingBufferEventHandler's SyncFuture count
> -
>
> Key: HBASE-24578
> URL: https://issues.apache.org/jira/browse/HBASE-24578
> Project: HBase
> Issue Type: Improvement
> Components: wal
>Affects Versions: 1.4.13, 2.2.5
>Reporter: Reid Chan
>Assignee: wenfeiyi666
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.3.1, 2.2.6
>
>
> The current value of RingBufferEventHandler's handler is the value of
> {{hbase.regionserver.handler.count}}, which works good in default wal
> provider --- one WAL per regionserver.
> When trying to use WAL group provider, either by group or wal per region, the
> default value is bad. If rs has 100 regions and wal per region strategy is
> used, then rs will allocate 100 *
> SyncFuture[$hbase.regionserver.handler.count] array
> {code}
> int maxHandlersCount = conf.getInt(HConstants.REGION_SERVER_HANDLER_COUNT,
> 200);
> this.ringBufferEventHandler = new RingBufferEventHandler(
> conf.getInt("hbase.regionserver.hlog.syncer.count", 5),
> maxHandlersCount);
> ...
>
> RingBufferEventHandler(final int syncRunnerCount, final int maxHandlersCount)
> {
> this.syncFutures = new SyncFuture[maxHandlersCount];
> ...
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] comnetwork commented on a change in pull request #2055: HBASE-24625 AsyncFSWAL.getLogFileSizeIfBeingWritten does not return the expected synced file length(addendum)
comnetwork commented on a change in pull request #2055: URL: https://github.com/apache/hbase/pull/2055#discussion_r454764681 ## File path: hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/AsyncProtobufLogWriter.java ## @@ -59,7 +59,7 @@ private final Class channelClass; - private AsyncFSOutput output; + private volatile AsyncFSOutput output; Review comment: Yes, you are right, I ignored other `write` methods also not test whether `output` is null. The problem here is it is meaningless to call `AsyncProtobufLogWriter.getSyncedLength` after `AsyncProtobufLogWriter.close` , which indeed occured when #1970 is applied to branch-2 previously caused by problematic state synchronization. If we do not test whether output is null, we can just leave some comments here. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2069: HBASE-24615 MutableRangeHistogram#updateSnapshotRangeMetrics doesn't calculate the distribution for last bucket.
Apache-HBase commented on pull request #2069: URL: https://github.com/apache/hbase/pull/2069#issuecomment-658519529 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 3m 33s | Docker mode activated. | | -0 :warning: | yetus | 0m 9s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2 Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 20s | branch-2 passed | | +1 :green_heart: | compile | 0m 17s | branch-2 passed | | +1 :green_heart: | shadedjars | 5m 39s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 17s | branch-2 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 43s | the patch passed | | +1 :green_heart: | compile | 0m 18s | the patch passed | | +1 :green_heart: | javac | 0m 18s | the patch passed | | +1 :green_heart: | shadedjars | 5m 32s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 14s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 28s | hbase-hadoop2-compat in the patch passed. | | | | 25m 37s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2069/1/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2069 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 8c7c68b9edf4 4.15.0-91-generic #92-Ubuntu SMP Fri Feb 28 11:09:48 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / d3ec4886a1 | | Default Java | 1.8.0_232 | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2069/1/testReport/ | | Max. process+thread count | 221 (vs. ulimit of 12500) | | modules | C: hbase-hadoop2-compat U: hbase-hadoop2-compat | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2069/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-24578) [WAL] Add a parameter to config RingBufferEventHandler's SyncFuture count
[
https://issues.apache.org/jira/browse/HBASE-24578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17157833#comment-17157833
]
wenfeiyi666 commented on HBASE-24578:
-
branch-1 build happen error "error in isort setup command: ('Invalid module
name', 'isort = isort.pylama_isort')"
> [WAL] Add a parameter to config RingBufferEventHandler's SyncFuture count
> -
>
> Key: HBASE-24578
> URL: https://issues.apache.org/jira/browse/HBASE-24578
> Project: HBase
> Issue Type: Improvement
> Components: wal
>Affects Versions: 1.4.13, 2.2.5
>Reporter: Reid Chan
>Assignee: wenfeiyi666
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.3.1, 2.2.6
>
>
> The current value of RingBufferEventHandler's handler is the value of
> {{hbase.regionserver.handler.count}}, which works good in default wal
> provider --- one WAL per regionserver.
> When trying to use WAL group provider, either by group or wal per region, the
> default value is bad. If rs has 100 regions and wal per region strategy is
> used, then rs will allocate 100 *
> SyncFuture[$hbase.regionserver.handler.count] array
> {code}
> int maxHandlersCount = conf.getInt(HConstants.REGION_SERVER_HANDLER_COUNT,
> 200);
> this.ringBufferEventHandler = new RingBufferEventHandler(
> conf.getInt("hbase.regionserver.hlog.syncer.count", 5),
> maxHandlersCount);
> ...
>
> RingBufferEventHandler(final int syncRunnerCount, final int maxHandlersCount)
> {
> this.syncFutures = new SyncFuture[maxHandlersCount];
> ...
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #2070: HBASE-24615 MutableRangeHistogram#updateSnapshotRangeMetrics doesn't calculate the distribution for last bucket.
Apache-HBase commented on pull request #2070: URL: https://github.com/apache/hbase/pull/2070#issuecomment-658515550 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 0s | Docker mode activated. | | -1 :x: | docker | 0m 8s | Docker failed to build yetus/hbase:7cf97016b9. | | Subsystem | Report/Notes | |--:|:-| | GITHUB PR | https://github.com/apache/hbase/pull/2070 | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2070/1/console | | versions | git=2.17.1 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] WenFeiYi opened a new pull request #2070: HBASE-24615 MutableRangeHistogram#updateSnapshotRangeMetrics doesn't calculate the distribution for last bucket.
WenFeiYi opened a new pull request #2070: URL: https://github.com/apache/hbase/pull/2070 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-24694) Support flush a single column family of table
[ https://issues.apache.org/jira/browse/HBASE-24694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17157830#comment-17157830 ] Zheng Wang commented on HBASE-24694: As discussed in HBASE-24404, flush table was implemented through execProcedure(use pv1), and compact table was implemented in client side by iterate all its regions, should flush table go same way? BTW, there is a preFlushTable cp-hook, so maybe we still need call execProcedure(just trigger the cp). > Support flush a single column family of table > - > > Key: HBASE-24694 > URL: https://issues.apache.org/jira/browse/HBASE-24694 > Project: HBase > Issue Type: New Feature >Reporter: Zheng Wang >Assignee: Zheng Wang >Priority: Major > > This is follow-on work of HBASE-24404, do it as a seprate issue could make it > easier to reveiw. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24615) MutableRangeHistogram#updateSnapshotRangeMetrics doesn't calculate the distribution for last bucket.
[
https://issues.apache.org/jira/browse/HBASE-24615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17157829#comment-17157829
]
wenfeiyi666 commented on HBASE-24615:
-
branch-1 branch-1.3 branch-1.4 exists the bug, branch-1.0 branch-1.1 branch-1.2
not exists.
> MutableRangeHistogram#updateSnapshotRangeMetrics doesn't calculate the
> distribution for last bucket.
>
>
> Key: HBASE-24615
> URL: https://issues.apache.org/jira/browse/HBASE-24615
> Project: HBase
> Issue Type: Bug
> Components: metrics
>Affects Versions: 2.3.0, master, 1.3.7, 2.2.6
>Reporter: Rushabh Shah
>Assignee: wenfeiyi666
>Priority: Major
>
> We are not processing the distribution for last bucket.
> https://github.com/apache/hbase/blob/master/hbase-hadoop-compat/src/main/java/org/apache/hadoop/metrics2/lib/MutableRangeHistogram.java#L70
> {code:java}
> public void updateSnapshotRangeMetrics(MetricsRecordBuilder
> metricsRecordBuilder,
> Snapshot snapshot) {
> long priorRange = 0;
> long cumNum = 0;
> final long[] ranges = getRanges();
> final String rangeType = getRangeType();
> for (int i = 0; i < ranges.length - 1; i++) { -> The bug lies
> here. We are not processing last bucket.
> long val = snapshot.getCountAtOrBelow(ranges[i]);
> if (val - cumNum > 0) {
> metricsRecordBuilder.addCounter(
> Interns.info(name + "_" + rangeType + "_" + priorRange + "-" +
> ranges[i], desc),
> val - cumNum);
> }
> priorRange = ranges[i];
> cumNum = val;
> }
> long val = snapshot.getCount();
> if (val - cumNum > 0) {
> metricsRecordBuilder.addCounter(
> Interns.info(name + "_" + rangeType + "_" + ranges[ranges.length -
> 1] + "-inf", desc),
> val - cumNum);
> }
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-24615) MutableRangeHistogram#updateSnapshotRangeMetrics doesn't calculate the distribution for last bucket.
[
https://issues.apache.org/jira/browse/HBASE-24615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17157826#comment-17157826
]
Hudson commented on HBASE-24615:
Results for branch branch-2
[build #2744 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2744/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2744/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2744/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2744/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2744/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> MutableRangeHistogram#updateSnapshotRangeMetrics doesn't calculate the
> distribution for last bucket.
>
>
> Key: HBASE-24615
> URL: https://issues.apache.org/jira/browse/HBASE-24615
> Project: HBase
> Issue Type: Bug
> Components: metrics
>Affects Versions: 2.3.0, master, 1.3.7, 2.2.6
>Reporter: Rushabh Shah
>Assignee: wenfeiyi666
>Priority: Major
>
> We are not processing the distribution for last bucket.
> https://github.com/apache/hbase/blob/master/hbase-hadoop-compat/src/main/java/org/apache/hadoop/metrics2/lib/MutableRangeHistogram.java#L70
> {code:java}
> public void updateSnapshotRangeMetrics(MetricsRecordBuilder
> metricsRecordBuilder,
> Snapshot snapshot) {
> long priorRange = 0;
> long cumNum = 0;
> final long[] ranges = getRanges();
> final String rangeType = getRangeType();
> for (int i = 0; i < ranges.length - 1; i++) { -> The bug lies
> here. We are not processing last bucket.
> long val = snapshot.getCountAtOrBelow(ranges[i]);
> if (val - cumNum > 0) {
> metricsRecordBuilder.addCounter(
> Interns.info(name + "_" + rangeType + "_" + priorRange + "-" +
> ranges[i], desc),
> val - cumNum);
> }
> priorRange = ranges[i];
> cumNum = val;
> }
> long val = snapshot.getCount();
> if (val - cumNum > 0) {
> metricsRecordBuilder.addCounter(
> Interns.info(name + "_" + rangeType + "_" + ranges[ranges.length -
> 1] + "-inf", desc),
> val - cumNum);
> }
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] WenFeiYi opened a new pull request #2069: HBASE-24615
WenFeiYi opened a new pull request #2069: URL: https://github.com/apache/hbase/pull/2069 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-24720) Meta replicas not cleaned when disabled
[
https://issues.apache.org/jira/browse/HBASE-24720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17157825#comment-17157825
]
Hudson commented on HBASE-24720:
Results for branch branch-2
[build #2744 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2744/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2744/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2744/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2744/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2744/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Meta replicas not cleaned when disabled
> ---
>
> Key: HBASE-24720
> URL: https://issues.apache.org/jira/browse/HBASE-24720
> Project: HBase
> Issue Type: Bug
> Components: read replicas
>Affects Versions: 3.0.0-alpha-1, 2.3.0, 2.4.0, 2.2.5
>Reporter: Szabolcs Bukros
>Assignee: Szabolcs Bukros
>Priority: Minor
> Fix For: 3.0.0-alpha-1, 2.3.1, 2.4.0, 2.2.6
>
>
> The assignMetaReplicas method works kinda like this:
> {code:java}
> void assignMetaReplicas(){
> if (numReplicas <= 1) return;
> //create if needed then assign meta replicas
> unassignExcessMetaReplica(numReplicas);
> }
> {code}
> Now this unassignExcessMetaReplica method is the one that gets rid of the
> replicas we no longer need. It closes them and deletes their zNode.
> Unfortunately this only happens if we decreased the replica number. If we
> disabled it, by setting the replica number to 1 assignMetaReplicas returns
> instantly without cleaning up the no longer needed replicas resulting in
> replicas lingering around.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-24578) [WAL] Add a parameter to config RingBufferEventHandler's SyncFuture count
[
https://issues.apache.org/jira/browse/HBASE-24578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17157823#comment-17157823
]
Hudson commented on HBASE-24578:
Results for branch branch-2
[build #2744 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2744/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2744/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2744/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2744/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2744/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> [WAL] Add a parameter to config RingBufferEventHandler's SyncFuture count
> -
>
> Key: HBASE-24578
> URL: https://issues.apache.org/jira/browse/HBASE-24578
> Project: HBase
> Issue Type: Improvement
> Components: wal
>Affects Versions: 1.4.13, 2.2.5
>Reporter: Reid Chan
>Assignee: wenfeiyi666
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.3.1, 2.2.6
>
>
> The current value of RingBufferEventHandler's handler is the value of
> {{hbase.regionserver.handler.count}}, which works good in default wal
> provider --- one WAL per regionserver.
> When trying to use WAL group provider, either by group or wal per region, the
> default value is bad. If rs has 100 regions and wal per region strategy is
> used, then rs will allocate 100 *
> SyncFuture[$hbase.regionserver.handler.count] array
> {code}
> int maxHandlersCount = conf.getInt(HConstants.REGION_SERVER_HANDLER_COUNT,
> 200);
> this.ringBufferEventHandler = new RingBufferEventHandler(
> conf.getInt("hbase.regionserver.hlog.syncer.count", 5),
> maxHandlersCount);
> ...
>
> RingBufferEventHandler(final int syncRunnerCount, final int maxHandlersCount)
> {
> this.syncFutures = new SyncFuture[maxHandlersCount];
> ...
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-24693) regioninfo#isLast() has a logic error
[
https://issues.apache.org/jira/browse/HBASE-24693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17157824#comment-17157824
]
Hudson commented on HBASE-24693:
Results for branch branch-2
[build #2744 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2744/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2744/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2744/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2744/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2744/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> regioninfo#isLast() has a logic error
> -
>
> Key: HBASE-24693
> URL: https://issues.apache.org/jira/browse/HBASE-24693
> Project: HBase
> Issue Type: Bug
>Affects Versions: master, 2.2.3
>Reporter: Bo Cui
>Assignee: Bo Cui
>Priority: Minor
> Fix For: 3.0.0-alpha-1, 2.3.1, 1.7.0, 2.4.0, 2.1.10, 2.2.6
>
>
> [https://github.com/apache/hbase/blob/90f4ff7d7c6997f61dbe40748b5c157879352acb/hbase-client/src/main/java/org/apache/hadoop/hbase/client/RegionInfo.java#L771]
> it should be
> {code:java}
> default boolean isLast() {
> return Bytes.equals(getEndKey(), HConstants.EMPTY_END_ROW);
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-24578) [WAL] Add a parameter to config RingBufferEventHandler's SyncFuture count
[
https://issues.apache.org/jira/browse/HBASE-24578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17157819#comment-17157819
]
Reid Chan commented on HBASE-24578:
---
Still trying to get it pushed to branch-1, that's why.
> [WAL] Add a parameter to config RingBufferEventHandler's SyncFuture count
> -
>
> Key: HBASE-24578
> URL: https://issues.apache.org/jira/browse/HBASE-24578
> Project: HBase
> Issue Type: Improvement
> Components: wal
>Affects Versions: 1.4.13, 2.2.5
>Reporter: Reid Chan
>Assignee: wenfeiyi666
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.3.1, 2.2.6
>
>
> The current value of RingBufferEventHandler's handler is the value of
> {{hbase.regionserver.handler.count}}, which works good in default wal
> provider --- one WAL per regionserver.
> When trying to use WAL group provider, either by group or wal per region, the
> default value is bad. If rs has 100 regions and wal per region strategy is
> used, then rs will allocate 100 *
> SyncFuture[$hbase.regionserver.handler.count] array
> {code}
> int maxHandlersCount = conf.getInt(HConstants.REGION_SERVER_HANDLER_COUNT,
> 200);
> this.ringBufferEventHandler = new RingBufferEventHandler(
> conf.getInt("hbase.regionserver.hlog.syncer.count", 5),
> maxHandlersCount);
> ...
>
> RingBufferEventHandler(final int syncRunnerCount, final int maxHandlersCount)
> {
> this.syncFutures = new SyncFuture[maxHandlersCount];
> ...
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] bsglz commented on pull request #2044: HBASE-24709 Support MoveCostFunction use a lower multiplier in offpea…
bsglz commented on pull request #2044: URL: https://github.com/apache/hbase/pull/2044#issuecomment-658504662 @virajjasani Seems no more comments, could you help merge it? Thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-24615) MutableRangeHistogram#updateSnapshotRangeMetrics doesn't calculate the distribution for last bucket.
[
https://issues.apache.org/jira/browse/HBASE-24615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17157817#comment-17157817
]
wenfeiyi666 commented on HBASE-24615:
-
ok, no problem
> MutableRangeHistogram#updateSnapshotRangeMetrics doesn't calculate the
> distribution for last bucket.
>
>
> Key: HBASE-24615
> URL: https://issues.apache.org/jira/browse/HBASE-24615
> Project: HBase
> Issue Type: Bug
> Components: metrics
>Affects Versions: 2.3.0, master, 1.3.7, 2.2.6
>Reporter: Rushabh Shah
>Assignee: wenfeiyi666
>Priority: Major
>
> We are not processing the distribution for last bucket.
> https://github.com/apache/hbase/blob/master/hbase-hadoop-compat/src/main/java/org/apache/hadoop/metrics2/lib/MutableRangeHistogram.java#L70
> {code:java}
> public void updateSnapshotRangeMetrics(MetricsRecordBuilder
> metricsRecordBuilder,
> Snapshot snapshot) {
> long priorRange = 0;
> long cumNum = 0;
> final long[] ranges = getRanges();
> final String rangeType = getRangeType();
> for (int i = 0; i < ranges.length - 1; i++) { -> The bug lies
> here. We are not processing last bucket.
> long val = snapshot.getCountAtOrBelow(ranges[i]);
> if (val - cumNum > 0) {
> metricsRecordBuilder.addCounter(
> Interns.info(name + "_" + rangeType + "_" + priorRange + "-" +
> ranges[i], desc),
> val - cumNum);
> }
> priorRange = ranges[i];
> cumNum = val;
> }
> long val = snapshot.getCount();
> if (val - cumNum > 0) {
> metricsRecordBuilder.addCounter(
> Interns.info(name + "_" + rangeType + "_" + ranges[ranges.length -
> 1] + "-inf", desc),
> val - cumNum);
> }
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-24742) Improve performance of SKIP vs SEEK logic
[ https://issues.apache.org/jira/browse/HBASE-24742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17157815#comment-17157815 ] Duo Zhang commented on HBASE-24742: --- And on the fake cell, maybe we should not pass it to upper layer? I think the intention here, is to avoid do a real seek for a store scanner, if it just needs to know its order in the KeyValueScanner, and delay the actual seek. Now I believe the code logic is to peek the kv and compare them directly. Maybe we could introduce something like getKeyForComparison? So we actually call peek or next, we will done the real seek and get a 'real' kv. In general, I think the open too many internal things to upper layer in KeyValueScanner, we have seek, reseek, requestSeek, realSeekDone, enforceSeek... Maybe when we introduced them at the first place, they were doing well and improving performance, but later when we added more code and fixed bugs, people will misuse them and cause performance regression... Looking at the POC, #1 is good, but I'm a bit nervous for #2, as in the discussion in HBASE-17958, We claimed that the index key could be changed during the next call. Will learn more when the PR is ready. Thanks. > Improve performance of SKIP vs SEEK logic > - > > Key: HBASE-24742 > URL: https://issues.apache.org/jira/browse/HBASE-24742 > Project: HBase > Issue Type: Bug > Components: Performance, regionserver >Affects Versions: 3.0.0-alpha-1, 1.7.0, 2.4.0 >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl >Priority: Major > Attachments: hbase-1.6-regression-flame-graph.png, > hbase-24742-branch-1.txt > > > In our testing of HBase 1.3 against the current tip of branch-1 we saw a 30% > slowdown in scanning scenarios. > We tracked it back to HBASE-17958 and HBASE-19863. > Both add comparisons to one of the tightest HBase has. > [~bharathv] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24742) Improve performance of SKIP vs SEEK logic
[ https://issues.apache.org/jira/browse/HBASE-24742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17157807#comment-17157807 ] Bharath Vissapragada commented on HBASE-24742: -- > I think the discussion in HBASE-17958 is enough to show that the logic is > necessary Yep, not suggesting that we undo that patch. Instead we should comprehensively fix the codepaths to not do extra byte compares. So agree with you. > I was/am planning a review of any commit that touches SQM and friends. This > was a bit daunting because (I am guessing) the number of commits from circa > 1.3 to 2.2 is more than a handful. [~apurtell] look at the flame graph I attached. I also noticed the bump in number of re-seeks. Based on my analysis I think HBASE-17958 is related, there are more cases where the skip hinting fails in the above code I pasted. Overall I think both the issues are related. We just tested a part of the fix (which is reduce the number of byte comparisons) but we need to analyze the code properly to see where the hinting fails and then re-seeks, which is essentially your jira. > Improve performance of SKIP vs SEEK logic > - > > Key: HBASE-24742 > URL: https://issues.apache.org/jira/browse/HBASE-24742 > Project: HBase > Issue Type: Bug > Components: Performance, regionserver >Affects Versions: 3.0.0-alpha-1, 1.7.0, 2.4.0 >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl >Priority: Major > Attachments: hbase-1.6-regression-flame-graph.png, > hbase-24742-branch-1.txt > > > In our testing of HBase 1.3 against the current tip of branch-1 we saw a 30% > slowdown in scanning scenarios. > We tracked it back to HBASE-17958 and HBASE-19863. > Both add comparisons to one of the tightest HBase has. > [~bharathv] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HBASE-24737) Find a way to resolve WALFileLengthProvider#getLogFileSizeIfBeingWritten problem
[ https://issues.apache.org/jira/browse/HBASE-24737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wenfeiyi666 reassigned HBASE-24737: --- Assignee: wenfeiyi666 > Find a way to resolve WALFileLengthProvider#getLogFileSizeIfBeingWritten > problem > > > Key: HBASE-24737 > URL: https://issues.apache.org/jira/browse/HBASE-24737 > Project: HBase > Issue Type: Sub-task >Reporter: Guanghao Zhang >Assignee: wenfeiyi666 >Priority: Major > > Now we use WALFileLengthProvider#getLogFileSizeIfBeingWritten to get the > synced wal length and prevent replicating unacked log entries. But after > offload ReplicationSource to new ReplicationServer, we need a new way to > resolve this problem. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-24742) Improve performance of SKIP vs SEEK logic
[ https://issues.apache.org/jira/browse/HBASE-24742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharath Vissapragada updated HBASE-24742: - Attachment: hbase-1.6-regression-flame-graph.png > Improve performance of SKIP vs SEEK logic > - > > Key: HBASE-24742 > URL: https://issues.apache.org/jira/browse/HBASE-24742 > Project: HBase > Issue Type: Bug > Components: Performance, regionserver >Affects Versions: 3.0.0-alpha-1, 1.7.0, 2.4.0 >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl >Priority: Major > Attachments: hbase-1.6-regression-flame-graph.png, > hbase-24742-branch-1.txt > > > In our testing of HBase 1.3 against the current tip of branch-1 we saw a 30% > slowdown in scanning scenarios. > We tracked it back to HBASE-17958 and HBASE-19863. > Both add comparisons to one of the tightest HBase has. > [~bharathv] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24578) [WAL] Add a parameter to config RingBufferEventHandler's SyncFuture count
[
https://issues.apache.org/jira/browse/HBASE-24578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17157799#comment-17157799
]
Guanghao Zhang commented on HBASE-24578:
This pushed to branch-2.2+. So we can close this jira?
> [WAL] Add a parameter to config RingBufferEventHandler's SyncFuture count
> -
>
> Key: HBASE-24578
> URL: https://issues.apache.org/jira/browse/HBASE-24578
> Project: HBase
> Issue Type: Improvement
> Components: wal
>Affects Versions: 1.4.13, 2.2.5
>Reporter: Reid Chan
>Assignee: wenfeiyi666
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.3.1, 2.2.6
>
>
> The current value of RingBufferEventHandler's handler is the value of
> {{hbase.regionserver.handler.count}}, which works good in default wal
> provider --- one WAL per regionserver.
> When trying to use WAL group provider, either by group or wal per region, the
> default value is bad. If rs has 100 regions and wal per region strategy is
> used, then rs will allocate 100 *
> SyncFuture[$hbase.regionserver.handler.count] array
> {code}
> int maxHandlersCount = conf.getInt(HConstants.REGION_SERVER_HANDLER_COUNT,
> 200);
> this.ringBufferEventHandler = new RingBufferEventHandler(
> conf.getInt("hbase.regionserver.hlog.syncer.count", 5),
> maxHandlersCount);
> ...
>
> RingBufferEventHandler(final int syncRunnerCount, final int maxHandlersCount)
> {
> this.syncFutures = new SyncFuture[maxHandlersCount];
> ...
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (HBASE-24742) Improve performance of SKIP vs SEEK logic
[ https://issues.apache.org/jira/browse/HBASE-24742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17157791#comment-17157791 ] Andrew Kyle Purtell edited comment on HBASE-24742 at 7/15/20, 1:35 AM: --- There is a regression in SKIP hint handling in branch-2, see HBASE-24637. Is this related? In the HBASE-24637 case the difference is not so much more comparisons on a hot path, which would be good to fix of course, no question, but a regression with wider scope with respect to reseeking (I/O) that causes, proportional to number of cells/columns, more work in the whole stack from file or blockcache to hfile reader up to SQM. I went out on vacation (and am still out) before tracking this down. I was/am planning a review of any commit that touches SQM and friends. This was a bit daunting because (I am guessing) the number of commits from circa 1.3 to 2.2 is more than a handful. Maybe not, that would be nice. Perhaps this issue finds it but I suspect more changes in this area were committed after HBASE-17958 and HBASE-19863. was (Author: apurtell): There is a regression in SKIP hint handling in branch-2, see HBASE-24637. Is this related? In the HBASE-24637 case the difference is not so much more comparisons on a hot path, which would be good to fix, but a regression with wider scope with respect to reseeking (I/O) that causes, proportional to number of cells/columns, more work in the whole stack from file or blockcache to hfile reader up to SQM. I went out on vacation (and am still out) before tracking this down. I was/am planning a review of any commit that touches SQM and friends. This was a bit daunting because (I am guessing) the number of commits from circa 1.3 to 2.2 is more than a handful. Maybe not, that would be nice. Perhaps this issue finds it but I suspect more changes in this area were committed after HBASE-17958 and HBASE-19863. > Improve performance of SKIP vs SEEK logic > - > > Key: HBASE-24742 > URL: https://issues.apache.org/jira/browse/HBASE-24742 > Project: HBase > Issue Type: Bug > Components: Performance, regionserver >Affects Versions: 3.0.0-alpha-1, 1.7.0, 2.4.0 >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl >Priority: Major > Attachments: hbase-24742-branch-1.txt > > > In our testing of HBase 1.3 against the current tip of branch-1 we saw a 30% > slowdown in scanning scenarios. > We tracked it back to HBASE-17958 and HBASE-19863. > Both add comparisons to one of the tightest HBase has. > [~bharathv] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #2068: Backport: HBASE-24705 MetaFixer#fixHoles() does not include the case for read r…
Apache-HBase commented on pull request #2068: URL: https://github.com/apache/hbase/pull/2068#issuecomment-658494351 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 2m 15s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ branch-2 Compile Tests _ | | +0 :ok: | mvndep | 0m 19s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 50s | branch-2 passed | | +1 :green_heart: | checkstyle | 2m 19s | branch-2 passed | | +1 :green_heart: | spotbugs | 3m 53s | branch-2 passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 27s | the patch passed | | -0 :warning: | checkstyle | 1m 18s | hbase-server: The patch generated 5 new + 227 unchanged - 0 fixed = 232 total (was 227) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 15m 12s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. | | +1 :green_heart: | spotbugs | 4m 19s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 25s | The patch does not generate ASF License warnings. | | | | 49m 1s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2068/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2068 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle | | uname | Linux 1e5c1334fb02 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / d3ec4886a1 | | checkstyle | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2068/1/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | Max. process+thread count | 84 (vs. ulimit of 12500) | | modules | C: hbase-client hbase-server U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2068/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2067: Backport: HBASE-24705 MetaFixer#fixHoles() does not include the case for read r…
Apache-HBase commented on pull request #2067: URL: https://github.com/apache/hbase/pull/2067#issuecomment-658494291 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 47s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ branch-2.3 Compile Tests _ | | +0 :ok: | mvndep | 0m 17s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 38s | branch-2.3 passed | | +1 :green_heart: | checkstyle | 1m 45s | branch-2.3 passed | | +1 :green_heart: | spotbugs | 3m 4s | branch-2.3 passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 16s | the patch passed | | -0 :warning: | checkstyle | 1m 7s | hbase-server: The patch generated 5 new + 228 unchanged - 0 fixed = 233 total (was 228) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 19m 14s | Patch does not cause any errors with Hadoop 2.10.0 or 3.1.2 3.2.1. | | +1 :green_heart: | spotbugs | 4m 26s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 30s | The patch does not generate ASF License warnings. | | | | 48m 46s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2067/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2067 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle | | uname | Linux 1f5af5035df5 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2.3 / 15c8c2f59c | | checkstyle | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2067/1/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | Max. process+thread count | 94 (vs. ulimit of 12500) | | modules | C: hbase-client hbase-server U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2067/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (HBASE-24742) Improve performance of SKIP vs SEEK logic
[ https://issues.apache.org/jira/browse/HBASE-24742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17157791#comment-17157791 ] Andrew Kyle Purtell edited comment on HBASE-24742 at 7/15/20, 1:30 AM: --- There is a regression in SKIP hint handling in branch-2, see HBASE-24637. Is this related? In the HBASE-24637 case the difference is not so much more comparisons on a hot path, which would be good to fix, but a regression with wider scope with respect to reseeking (I/O) that causes, proportional to number of cells/columns, more work in the whole stack from file or blockcache to hfile reader up to SQM. I went out on vacation (and am still out) before tracking this down. I was/am planning a review of any commit that touches SQM and friends. This was a bit daunting because (I am guessing) the number of commits from circa 1.3 to 2.2 is more than a handful. Maybe not, that would be nice. Perhaps this issue finds it but I suspect more changes in this area were committed after HBASE-17958 and HBASE-19863. was (Author: apurtell): There is a regression in SKIP hint handling in branch-2, see HBASE-24637. Is this related? In the HBASE-24637 case the difference is not so much more comparisons on a hot path, which would be good to fix, but a regression with wider scope with respect to reseeking (I/O) that causes, proportional to number of cells/columns, more work in the whole stack from file or blockcache to hfile reader up to SQM. I went out on vacation (and am still out) before tracking this down. I was/am planning a review of any commit that touches SQM and friends. This was a bit daunting because the number of commits from circa 1.3 to 2.2 is large. Perhaps this issue finds it but I suspect more changes in this area were committed after HBASE-17958 and HBASE-19863. > Improve performance of SKIP vs SEEK logic > - > > Key: HBASE-24742 > URL: https://issues.apache.org/jira/browse/HBASE-24742 > Project: HBase > Issue Type: Bug > Components: Performance, regionserver >Affects Versions: 3.0.0-alpha-1, 1.7.0, 2.4.0 >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl >Priority: Major > Attachments: hbase-24742-branch-1.txt > > > In our testing of HBase 1.3 against the current tip of branch-1 we saw a 30% > slowdown in scanning scenarios. > We tracked it back to HBASE-17958 and HBASE-19863. > Both add comparisons to one of the tightest HBase has. > [~bharathv] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HBASE-24742) Improve performance of SKIP vs SEEK logic
[ https://issues.apache.org/jira/browse/HBASE-24742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17157791#comment-17157791 ] Andrew Kyle Purtell edited comment on HBASE-24742 at 7/15/20, 1:29 AM: --- There is a regression in SKIP hint handling in branch-2, see HBASE-24637. Is this related? In the HBASE-24637 case the difference is not so much more comparisons on a hot path, which would be good to fix, but a regression with wider scope with respect to reseeking (I/O) that causes, proportional to number of cells/columns, more work in the whole stack from file or blockcache to hfile reader up to SQM. I went out on vacation (and am still out) before tracking this down. I was/am planning a review of any commit that touches SQM and friends. This was a bit daunting because the number of commits from circa 1.3 to 2.2 is large. Perhaps this issue finds it but I suspect more changes in this area were committed after HBASE-17958 and HBASE-19863. was (Author: apurtell): There is a regression in SKIP hint handling in branch-2, see HBASE-24637. Is this related? In the HBASE-24637 case the difference is not so much more comparisons on a hot path, which would be good to fix, but a regression with wider scope with respect to reseeking (I/O) that causes, proportional to number of cells/columns, more work in the whole stack from file or blockcache to hfile reader up to SQM. I went out on vacation (and am still out) before tracking this down. > Improve performance of SKIP vs SEEK logic > - > > Key: HBASE-24742 > URL: https://issues.apache.org/jira/browse/HBASE-24742 > Project: HBase > Issue Type: Bug > Components: Performance, regionserver >Affects Versions: 3.0.0-alpha-1, 1.7.0, 2.4.0 >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl >Priority: Major > Attachments: hbase-24742-branch-1.txt > > > In our testing of HBase 1.3 against the current tip of branch-1 we saw a 30% > slowdown in scanning scenarios. > We tracked it back to HBASE-17958 and HBASE-19863. > Both add comparisons to one of the tightest HBase has. > [~bharathv] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HBASE-24742) Improve performance of SKIP vs SEEK logic
[ https://issues.apache.org/jira/browse/HBASE-24742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17157791#comment-17157791 ] Andrew Kyle Purtell edited comment on HBASE-24742 at 7/15/20, 1:26 AM: --- There is a regression in SKIP hint handling in branch-2, see HBASE-24637. Is this related? In the HBASE-24637 case the difference is not so much more comparisons on a hot path, which would be good to fix, but a regression with wider scope with respect to reseeking (I/O) that causes, proportional to number of cells/columns, more work in the whole stack from file or blockcache to hfile reader up to SQM. I went out on vacation (and am still out) before tracking this down. was (Author: apurtell): There is a regression in SKIP hint handling in branch-2, see HBASE-24637. Is this related? In the HBASE-24637 case the difference is not so much more comparisons on a hot path but an actual serious regression with respect to reseeking (I/O). I went out on vacation (and am still out) before tracking this down. > Improve performance of SKIP vs SEEK logic > - > > Key: HBASE-24742 > URL: https://issues.apache.org/jira/browse/HBASE-24742 > Project: HBase > Issue Type: Bug > Components: Performance, regionserver >Affects Versions: 3.0.0-alpha-1, 1.7.0, 2.4.0 >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl >Priority: Major > Attachments: hbase-24742-branch-1.txt > > > In our testing of HBase 1.3 against the current tip of branch-1 we saw a 30% > slowdown in scanning scenarios. > We tracked it back to HBASE-17958 and HBASE-19863. > Both add comparisons to one of the tightest HBase has. > [~bharathv] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24742) Improve performance of SKIP vs SEEK logic
[ https://issues.apache.org/jira/browse/HBASE-24742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17157791#comment-17157791 ] Andrew Kyle Purtell commented on HBASE-24742: - There is an actual regression in SKIP hint handling in branch-2, see HBASE-24637. Is this related? In the HBASE-24637 case the difference is not so much more comparisons on a hot path but an actual serious regression with respect to reseeking (I/O). I went out on vacation (and am still out) before tracking this down. > Improve performance of SKIP vs SEEK logic > - > > Key: HBASE-24742 > URL: https://issues.apache.org/jira/browse/HBASE-24742 > Project: HBase > Issue Type: Bug > Components: Performance, regionserver >Affects Versions: 3.0.0-alpha-1, 1.7.0, 2.4.0 >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl >Priority: Major > Attachments: hbase-24742-branch-1.txt > > > In our testing of HBase 1.3 against the current tip of branch-1 we saw a 30% > slowdown in scanning scenarios. > We tracked it back to HBASE-17958 and HBASE-19863. > Both add comparisons to one of the tightest HBase has. > [~bharathv] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HBASE-24742) Improve performance of SKIP vs SEEK logic
[ https://issues.apache.org/jira/browse/HBASE-24742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17157791#comment-17157791 ] Andrew Kyle Purtell edited comment on HBASE-24742 at 7/15/20, 1:18 AM: --- There is a regression in SKIP hint handling in branch-2, see HBASE-24637. Is this related? In the HBASE-24637 case the difference is not so much more comparisons on a hot path but an actual serious regression with respect to reseeking (I/O). I went out on vacation (and am still out) before tracking this down. was (Author: apurtell): There is an actual regression in SKIP hint handling in branch-2, see HBASE-24637. Is this related? In the HBASE-24637 case the difference is not so much more comparisons on a hot path but an actual serious regression with respect to reseeking (I/O). I went out on vacation (and am still out) before tracking this down. > Improve performance of SKIP vs SEEK logic > - > > Key: HBASE-24742 > URL: https://issues.apache.org/jira/browse/HBASE-24742 > Project: HBase > Issue Type: Bug > Components: Performance, regionserver >Affects Versions: 3.0.0-alpha-1, 1.7.0, 2.4.0 >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl >Priority: Major > Attachments: hbase-24742-branch-1.txt > > > In our testing of HBase 1.3 against the current tip of branch-1 we saw a 30% > slowdown in scanning scenarios. > We tracked it back to HBASE-17958 and HBASE-19863. > Both add comparisons to one of the tightest HBase has. > [~bharathv] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24742) Improve performance of SKIP vs SEEK logic
[ https://issues.apache.org/jira/browse/HBASE-24742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17157787#comment-17157787 ] Duo Zhang commented on HBASE-24742: --- I think the discussion in HBASE-17958 is enough to show that the logic is necessary? Let’s not go back to let the filters deal with strange cells. Will take a look at the patch. Anyway, back to the problem, I agree we have done too many bytes comparison. We should find a general way to deal with it. Was imagine that, we add some methods in the ScannerContext, to record whether we have changed the row, or family, or column, or version, so for most cases we do not need to do bytes comparison again? Thanks. > Improve performance of SKIP vs SEEK logic > - > > Key: HBASE-24742 > URL: https://issues.apache.org/jira/browse/HBASE-24742 > Project: HBase > Issue Type: Bug > Components: Performance, regionserver >Affects Versions: 3.0.0-alpha-1, 1.7.0, 2.4.0 >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl >Priority: Major > Attachments: hbase-24742-branch-1.txt > > > In our testing of HBase 1.3 against the current tip of branch-1 we saw a 30% > slowdown in scanning scenarios. > We tracked it back to HBASE-17958 and HBASE-19863. > Both add comparisons to one of the tightest HBase has. > [~bharathv] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24734) Wrong comparator opening Region when 'split-to-WAL' enabled.
[
https://issues.apache.org/jira/browse/HBASE-24734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17157785#comment-17157785
]
Huaxiang Sun commented on HBASE-24734:
--
Yeah, it seems that it does not user meta comparator for meta hfile,
containsRange() does not consider the meta case.
> Wrong comparator opening Region when 'split-to-WAL' enabled.
>
>
> Key: HBASE-24734
> URL: https://issues.apache.org/jira/browse/HBASE-24734
> Project: HBase
> Issue Type: Sub-task
> Components: HFile, MTTR
>Reporter: Michael Stack
>Priority: Major
>
> Came across this when we were testing the 'split-to-hfile' feature running
> ITBLL:
>
> {code:java}
> 2020-07-10 10:16:49,983 INFO org.apache.hadoop.hbase.regionserver.HRegion:
> Closing region hbase:meta,,1.15882307402020-07-10 10:16:49,997 INFO
> org.apache.hadoop.hbase.regionserver.HRegion: Closed
> hbase:meta,,1.15882307402020-07-10 10:16:49,998 WARN
> org.apache.hadoop.hbase.regionserver.handler.AssignRegionHandler: Fatal error
> occurred while opening region hbase:meta,,1.1588230740,
> aborting...java.lang.IllegalArgumentException: Invalid range:
> IntegrationTestBigLinkedList,,1594350463222.8f89e01a5245e79946e22d8a8ab4698b.
> >
> IntegrationTestBigLinkedList,\x10\x02J\xA1,1594349535271.be24dc276f686e6dcc7fb9d3f91c8387.
> at
> org.apache.hadoop.hbase.client.RegionInfoBuilder$MutableRegionInfo.containsRange(RegionInfoBuilder.java:300)
> at
> org.apache.hadoop.hbase.regionserver.HStore.tryCommitRecoveredHFile(HStore.java:)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.loadRecoveredHFilesIfAny(HRegion.java:5442)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals(HRegion.java:1010)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:950)
>at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7490)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegionFromTableDir(HRegion.java:7448)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7424)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7382)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7333)
> at
> org.apache.hadoop.hbase.regionserver.handler.AssignRegionHandler.process(AssignRegionHandler.java:135)
> at
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)2020-07-10
> 10:16:50,005 ERROR org.apache.hadoop.hbase.regionserver.HRegionServer: *
> ABORTING region server hbasedn149.example.org,16020,1594375563853: Failed to
> open region hbase:meta,,1.1588230740 and can not recover
> *java.lang.IllegalArgumentException: Invalid range:
> IntegrationTestBigLinkedList,,1594350463222.8f89e01a5245e79946e22d8a8ab4698b.
> >
> IntegrationTestBigLinkedList,\x10\x02J\xA1,1594349535271.be24dc276f686e6dcc7fb9d3f91c8387.
> {code}
> Seems basic case of wrong comparator. Below passes if I use the meta
> comparator
> {code:java}
> @Test
> public void testBinaryKeys() throws Exception {
> Set set = new TreeSet<>(CellComparatorImpl.COMPARATOR);
> final byte [] fam = Bytes.toBytes("col");
> final byte [] qf = Bytes.toBytes("umn");
> final byte [] nb = new byte[0];
> Cell [] keys = {
> createByteBufferKeyValueFromKeyValue(
> new KeyValue(Bytes.toBytes("a,\u\u,2"), fam, qf, 2,
> nb)),
> createByteBufferKeyValueFromKeyValue(
> new KeyValue(Bytes.toBytes("a,\u0001,3"), fam, qf, 3, nb)),
> createByteBufferKeyValueFromKeyValue(
> new KeyValue(Bytes.toBytes("a,,1"), fam, qf, 1, nb)),
> createByteBufferKeyValueFromKeyValue(
> new KeyValue(Bytes.toBytes("a,\u1000,5"), fam, qf, 5, nb)),
> createByteBufferKeyValueFromKeyValue(
> new KeyValue(Bytes.toBytes("a,a,4"), fam, qf, 4, nb)),
> createByteBufferKeyValueFromKeyValue(
> new KeyValue(Bytes.toBytes("a,a,0"), fam, qf, 0, nb)),
> };
> // Add to set with bad comparator
> Collections.addAll(set, keys);
> // This will output the keys incorrectly.
> boolean assertion = false;
> int count = 0;
> try {
> for (Cell k: set) {
> assertTrue("count=" + count + ", " + k.toString(), count++ ==
> k.getTimestamp());
> }
> } catch (AssertionError e) {
> // Expected
> assertion = true;
> }
> assertTrue(assertion);
> // Make set with
[jira] [Updated] (HBASE-24742) Improve performance of SKIP vs SEEK logic
[ https://issues.apache.org/jira/browse/HBASE-24742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharath Vissapragada updated HBASE-24742: - Component/s: regionserver Performance > Improve performance of SKIP vs SEEK logic > - > > Key: HBASE-24742 > URL: https://issues.apache.org/jira/browse/HBASE-24742 > Project: HBase > Issue Type: Bug > Components: Performance, regionserver >Affects Versions: 3.0.0-alpha-1, 1.7.0, 2.4.0 >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl >Priority: Major > Attachments: hbase-24742-branch-1.txt > > > In our testing of HBase 1.3 against the current tip of branch-1 we saw a 30% > slowdown in scanning scenarios. > We tracked it back to HBASE-17958 and HBASE-19863. > Both add comparisons to one of the tightest HBase has. > [~bharathv] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-24742) Improve performance of SKIP vs SEEK logic
[ https://issues.apache.org/jira/browse/HBASE-24742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharath Vissapragada updated HBASE-24742: - Affects Version/s: 2.4.0 1.7.0 3.0.0-alpha-1 > Improve performance of SKIP vs SEEK logic > - > > Key: HBASE-24742 > URL: https://issues.apache.org/jira/browse/HBASE-24742 > Project: HBase > Issue Type: Bug >Affects Versions: 3.0.0-alpha-1, 1.7.0, 2.4.0 >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl >Priority: Major > Attachments: hbase-24742-branch-1.txt > > > In our testing of HBase 1.3 against the current tip of branch-1 we saw a 30% > slowdown in scanning scenarios. > We tracked it back to HBASE-17958 and HBASE-19863. > Both add comparisons to one of the tightest HBase has. > [~bharathv] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HBASE-24742) Improve performance of SKIP vs SEEK logic
[ https://issues.apache.org/jira/browse/HBASE-24742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17157774#comment-17157774 ] Lars Hofhansl edited comment on HBASE-24742 at 7/15/20, 12:43 AM: -- There are two observations: 1. We do not need to check for "fake" keys inserted by the ROWCOL BF logic if there are not ROWCOL BFs (or if they are not used) 2. We can extend the identity-compare of the nextIndexedKey across multiple calls. It's just an optimization and not for correctness. was (Author: lhofhansl): There are two observations: 1. We do not need to check for "fake" keys inserted by the ROWCOL BF logic if there are not ROWCOL BFs (or if they are not used) 2. We can extend the identify compare of the nextIndexedKey across multiple calls. It's just an optimization and not for correctness. > Improve performance of SKIP vs SEEK logic > - > > Key: HBASE-24742 > URL: https://issues.apache.org/jira/browse/HBASE-24742 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl >Priority: Major > Attachments: hbase-24742-branch-1.txt > > > In our testing of HBase 1.3 against the current tip of branch-1 we saw a 30% > slowdown in scanning scenarios. > We tracked it back to HBASE-17958 and HBASE-19863. > Both add comparisons to one of the tightest HBase has. > [~bharathv] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HBASE-24742) Improve performance of SKIP vs SEEK logic
[ https://issues.apache.org/jira/browse/HBASE-24742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17157774#comment-17157774 ] Lars Hofhansl edited comment on HBASE-24742 at 7/15/20, 12:42 AM: -- There are two observations: 1. We do not need to check for "fake" keys inserted by the ROWCOL BF logic if there are not ROWCOL BFs (or if they are not used) 2. We can extend the identify compare of the nextIndexedKey across multiple calls. It's just an optimization and not for correctness. was (Author: lhofhansl): There are two observations: 1. We do not need for "fake" keys inserted by the ROWCOL BF logic if there are not ROWCOL BFs (or if they are not used) 2. We can extend the identify compare of the nextIndexedKey across multiple calls. It's just an optimization and not for correctness. > Improve performance of SKIP vs SEEK logic > - > > Key: HBASE-24742 > URL: https://issues.apache.org/jira/browse/HBASE-24742 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl >Priority: Major > Attachments: hbase-24742-branch-1.txt > > > In our testing of HBase 1.3 against the current tip of branch-1 we saw a 30% > slowdown in scanning scenarios. > We tracked it back to HBASE-17958 and HBASE-19863. > Both add comparisons to one of the tightest HBase has. > [~bharathv] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HBASE-24742) Improve performance of SKIP vs SEEK logic
[ https://issues.apache.org/jira/browse/HBASE-24742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17157779#comment-17157779 ] Lars Hofhansl edited comment on HBASE-24742 at 7/15/20, 12:42 AM: -- Passes the test added in HBASE-19863 and brings the runtime of a test Phoenix query from 5.8s to 4.2s. (This is for a fully compacted table and VERSIONS=1, which represents the worst case, where the two linked jiras triple the number of comparisons per Cell). I'll post a PR tomorrow. was (Author: lhofhansl): Passes the test added in HBASE-19863 and brings the runtime of a test Phoenix query from 5.8s to 4.2s. (This is for a fully compacted table, which represents the worst case, where the two linked jiras triple the number of comparisons per Cell). I'll post a PR tomorrow. > Improve performance of SKIP vs SEEK logic > - > > Key: HBASE-24742 > URL: https://issues.apache.org/jira/browse/HBASE-24742 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl >Priority: Major > Attachments: hbase-24742-branch-1.txt > > > In our testing of HBase 1.3 against the current tip of branch-1 we saw a 30% > slowdown in scanning scenarios. > We tracked it back to HBASE-17958 and HBASE-19863. > Both add comparisons to one of the tightest HBase has. > [~bharathv] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HBASE-24742) Improve performance of SKIP vs SEEK logic
[ https://issues.apache.org/jira/browse/HBASE-24742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17157779#comment-17157779 ] Lars Hofhansl edited comment on HBASE-24742 at 7/15/20, 12:40 AM: -- Passes the test added in HBASE-19863 and brings the runtime of a test Phoenix query from 5.8s to 4.2s. (This is for a fully compacted table, which represents the worst case, where the two linked jiras triple the number of comparisons per Cell). I'll post a PR tomorrow. was (Author: lhofhansl): Passes the test added in HBASE-19863 and brings the runtime of a test Phoenix query from 5.8s to 4.2s. > Improve performance of SKIP vs SEEK logic > - > > Key: HBASE-24742 > URL: https://issues.apache.org/jira/browse/HBASE-24742 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl >Priority: Major > Attachments: hbase-24742-branch-1.txt > > > In our testing of HBase 1.3 against the current tip of branch-1 we saw a 30% > slowdown in scanning scenarios. > We tracked it back to HBASE-17958 and HBASE-19863. > Both add comparisons to one of the tightest HBase has. > [~bharathv] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] huaxiangsun opened a new pull request #2068: Backport: HBASE-24705 MetaFixer#fixHoles() does not include the case for read r…
huaxiangsun opened a new pull request #2068: URL: https://github.com/apache/hbase/pull/2068 …eplicas (i.e, replica regions are not created) (#2062) Signed-off-by: Viraj Jasani This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] huaxiangsun opened a new pull request #2067: Backport: HBASE-24705 MetaFixer#fixHoles() does not include the case for read r…
huaxiangsun opened a new pull request #2067: URL: https://github.com/apache/hbase/pull/2067 …eplicas (i.e, replica regions are not created) (#2062) Signed-off-by: Viraj Jasani This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-24742) Improve performance of SKIP vs SEEK logic
[ https://issues.apache.org/jira/browse/HBASE-24742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17157779#comment-17157779 ] Lars Hofhansl commented on HBASE-24742: --- Passes the test added in HBASE-19863 and brings the runtime of a test Phoenix query from 5.8s to 4.2s. > Improve performance of SKIP vs SEEK logic > - > > Key: HBASE-24742 > URL: https://issues.apache.org/jira/browse/HBASE-24742 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl >Priority: Major > Attachments: hbase-24742-branch-1.txt > > > In our testing of HBase 1.3 against the current tip of branch-1 we saw a 30% > slowdown in scanning scenarios. > We tracked it back to HBASE-17958 and HBASE-19863. > Both add comparisons to one of the tightest HBase has. > [~bharathv] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24742) Improve performance of SKIP vs SEEK logic
[
https://issues.apache.org/jira/browse/HBASE-24742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17157776#comment-17157776
]
Bharath Vissapragada commented on HBASE-24742:
--
To add more color, following is the tight loop that Lars is talking about
{noformat}
protected boolean trySkipToNextColumn(Cell cell) throws IOException {
Cell nextCell = null;
// used to guard against a changed next indexed key by doing a identity
comparison
// when the identity changes we need to compare the bytes again
Cell previousIndexedKey = null;
do {
Cell nextIndexedKey = getNextIndexedKey();
if (nextIndexedKey != null && nextIndexedKey !=
KeyValueScanner.NO_NEXT_INDEXED_KEY &&
(nextIndexedKey == previousIndexedKey ||
matcher.compareKeyForNextColumn(nextIndexedKey, cell) >= 0)) { <=
this.heap.next();
++kvsScanned;
previousIndexedKey = nextIndexedKey;
} else {
return false;
}
} while ((nextCell = this.heap.peek()) != null &&
CellUtil.matchingRowColumn(cell, nextCell));
// We need this check because it may happen that the new scanner that we get
// during heap.next() is requiring reseek due of fake KV previously
generated for
// ROWCOL bloom filter optimization. See HBASE-19863 for more details
if (nextCell != null && matcher.compareKeyForNextColumn(nextCell, cell) <
0) {. <===
return false;
}
return true;
}
{noformat}
Specifically that was added to prevent SQM from matching the skipped rows but
it turns out that it does may more compare checks than what it was before. To
test our theory we've undone the loop and let the SQM match the rows and we
gained almost ~30% back in scans with explicit column filters. But again as
discussed in HBASE-17958, that comes at an expense of correctness that filters
shouldn't see skipped rows.
[~zghao] [~zhangduo] FYI since you were involved in the original jira fix and
implementation.
> Improve performance of SKIP vs SEEK logic
> -
>
> Key: HBASE-24742
> URL: https://issues.apache.org/jira/browse/HBASE-24742
> Project: HBase
> Issue Type: Bug
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
>Priority: Major
> Attachments: hbase-24742-branch-1.txt
>
>
> In our testing of HBase 1.3 against the current tip of branch-1 we saw a 30%
> slowdown in scanning scenarios.
> We tracked it back to HBASE-17958 and HBASE-19863.
> Both add comparisons to one of the tightest HBase has.
> [~bharathv]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HBASE-24742) Improve performance of SKIP vs SEEK logic
[ https://issues.apache.org/jira/browse/HBASE-24742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl reassigned HBASE-24742: - Assignee: Lars Hofhansl > Improve performance of SKIP vs SEEK logic > - > > Key: HBASE-24742 > URL: https://issues.apache.org/jira/browse/HBASE-24742 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl >Priority: Major > Attachments: hbase-24742-branch-1.txt > > > In our testing of HBase 1.3 against the current tip of branch-1 we saw a 30% > slowdown in scanning scenarios. > We tracked it back to HBASE-17958 and HBASE-19863. > Both add comparisons to one of the tightest HBase has. > [~bharathv] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24742) Improve performance of SKIP vs SEEK logic
[ https://issues.apache.org/jira/browse/HBASE-24742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17157775#comment-17157775 ] Lars Hofhansl commented on HBASE-24742: --- Here's a patch. Please have a careful look, especially at the part that turns previousIndexedKey into a member. > Improve performance of SKIP vs SEEK logic > - > > Key: HBASE-24742 > URL: https://issues.apache.org/jira/browse/HBASE-24742 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Priority: Major > Attachments: hbase-24742-branch-1.txt > > > In our testing of HBase 1.3 against the current tip of branch-1 we saw a 30% > slowdown in scanning scenarios. > We tracked it back to HBASE-17958 and HBASE-19863. > Both add comparisons to one of the tightest HBase has. > [~bharathv] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-24742) Improve performance of SKIP vs SEEK logic
[ https://issues.apache.org/jira/browse/HBASE-24742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-24742: -- Attachment: hbase-24742-branch-1.txt > Improve performance of SKIP vs SEEK logic > - > > Key: HBASE-24742 > URL: https://issues.apache.org/jira/browse/HBASE-24742 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Priority: Major > Attachments: hbase-24742-branch-1.txt > > > In our testing of HBase 1.3 against the current tip of branch-1 we saw a 30% > slowdown in scanning scenarios. > We tracked it back to HBASE-17958 and HBASE-19863. > Both add comparisons to one of the tightest HBase has. > [~bharathv] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24742) Improve performance of SKIP vs SEEK logic
[ https://issues.apache.org/jira/browse/HBASE-24742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17157774#comment-17157774 ] Lars Hofhansl commented on HBASE-24742: --- There are two observations: 1. We do not need for "fake" keys inserted by the ROWCOL BF logic if there are not ROWCOL BFs (or if they are not used) 2. We can extend the identify compare of the nextIndexedKey across multiple calls. It's just an optimization and not for correctness. > Improve performance of SKIP vs SEEK logic > - > > Key: HBASE-24742 > URL: https://issues.apache.org/jira/browse/HBASE-24742 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Priority: Major > > In our testing of HBase 1.3 against the current tip of branch-1 we saw a 30% > slowdown in scanning scenarios. > We tracked it back to HBASE-17958 and HBASE-19863. > Both add comparisons to one of the tightest HBase has. > [~bharathv] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-24742) Improve performance of SKIP vs SEEK logic
Lars Hofhansl created HBASE-24742: - Summary: Improve performance of SKIP vs SEEK logic Key: HBASE-24742 URL: https://issues.apache.org/jira/browse/HBASE-24742 Project: HBase Issue Type: Bug Reporter: Lars Hofhansl In our testing of HBase 1.3 against the current tip of branch-1 we saw a 30% slowdown in scanning scenarios. We tracked it back to HBASE-17958 and HBASE-19863. Both add comparisons to one of the tightest HBase has. [~bharathv] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-24723) Distributing ROOT Region load
[ https://issues.apache.org/jira/browse/HBASE-24723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Stack updated HBASE-24723: -- Summary: Distributing ROOT Region load (was: Distributed cache of root table data (ROOT/hbase:meta)) > Distributing ROOT Region load > - > > Key: HBASE-24723 > URL: https://issues.apache.org/jira/browse/HBASE-24723 > Project: HBase > Issue Type: Bug >Reporter: Michael Stack >Priority: Major > > HBASE-11288 'Splittable Meta' progress keeps getting hung up on how to cache > the 'root' table – the first table in the hierarchy of lookups locating > user-data in a Table Region. The 'root' table is not splittable by design so > its load can not be spread about the cluster by splitting the hot Region as > we would usually do. > This issue is about listing out requirements, sketching possible approaches, > and then getting agreement ahead of implementation as subtasks. > In particular, an approach that would work whether the Region to be cached is > master-based or hosted by a RegionServer (see design attached to HBASE-11288 > for elaboration) would be preferred since then we can separate discussion of > cache from the HBASE-11288 'split meta' discussion. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] ndimiduk commented on pull request #2015: HBASE-24568 do-release need not wait for tag
ndimiduk commented on pull request #2015:
URL: https://github.com/apache/hbase/pull/2015#issuecomment-658464588
> I'm fine with this change as is, but longer term there's a code smell
around the ASF_REPO variable.
>
> it gets defined in `get_release_info` but does not appear to be
intentionally persisted or exported from there. so I suspect the
`check_for_tag` method only works there and I'm not sure we are gaining over
using `git ls-remote` directly
I could replicate the defaulting logic from `get_release_info`,
```
if [[ -z "${ASF_REPO}" ]]; then
ASF_REPO="https://gitbox.apache.org/repos/asf/${PROJECT}.git;
fi
```
into `check_for_tag`. That would make it safe for use outside of that
context.
I don't follow this comment -- the proposed patch _does_ use `git
ls-remote`. Do you mean to say, use `git ls-remote` _without_ specifying the
remote repository? The man page says `` is optional, so we could
omit it, though I'm not clear what the default remote is at this point. I like
keeping things explicit if we can.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] ndimiduk commented on pull request #2017: HBASE-24669 Logging of ppid should be consistent across all occurrences
ndimiduk commented on pull request #2017: URL: https://github.com/apache/hbase/pull/2017#issuecomment-658460798 I tried to make a patch for master; it looks like the code there logs slightly differently and this patch does not apply. @saintstack maybe your patch (or addendum) for HBASE-21078 hit master and branch-2 differently? I'd think this kind of thing would be in sync across branches. Maybe check something wasn't missed? I think @nyl3532016 's patch here (nicely logging both the id of the final child and the identity of the parent) is worth having everywhere. Do you folks agree? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] huaxiangsun merged pull request #1986: HBASE-24581 Skip compaction request/check for replica regions at the …
huaxiangsun merged pull request #1986: URL: https://github.com/apache/hbase/pull/1986 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] huaxiangsun merged pull request #2062: HBASE-24705 MetaFixer#fixHoles() does not include the case for read r…
huaxiangsun merged pull request #2062: URL: https://github.com/apache/hbase/pull/2062 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] ndimiduk commented on a change in pull request #2037: HBASE-24698 Turn OFF Canary WebUI as default
ndimiduk commented on a change in pull request #2037:
URL: https://github.com/apache/hbase/pull/2037#discussion_r454671159
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/tool/CanaryTool.java
##
@@ -128,14 +124,8 @@
public class CanaryTool implements Tool, Canary {
public static final String HBASE_CANARY_INFO_PORT = "hbase.canary.info.port";
- public static final int DEFAULT_CANARY_INFOPORT = 16050;
-
- public static final String HBASE_CANARY_INFO_BINDADDRESS =
"hbase.canary.info.bindAddress";
Review comment:
I think it's fine to remove these constants, at least to make them
private. Anyway, I see no reason for them to be public.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (HBASE-24698) Turn OFF Canary WebUI as default
[ https://issues.apache.org/jira/browse/HBASE-24698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17157721#comment-17157721 ] Nick Dimiduk commented on HBASE-24698: -- Maybe there's something lighter weight than Jetty that we could use? HTTP is not our critical data path at the moment, so i think we can be lighter on features. > Turn OFF Canary WebUI as default > > > Key: HBASE-24698 > URL: https://issues.apache.org/jira/browse/HBASE-24698 > Project: HBase > Issue Type: Sub-task > Components: canary >Reporter: Michael Stack >Assignee: Michael Stack >Priority: Major > Fix For: 3.0.0-alpha-1, 2.3.1, 2.4.0 > > > See parent issue. There is a CLASSPATH issue when running against hadoop3 > that needs resolving. Meantime, the canary fails to run with a cryptic > message. This will surprise operators. Let me make it so you ask for the > canary webui; by default, it does not come up. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] ndimiduk commented on pull request #2061: HBASE-24566 Add 2.3.0 to the downloads page
ndimiduk commented on pull request #2061: URL: https://github.com/apache/hbase/pull/2061#issuecomment-658428908 > ugh we don't save maven site output from precommit. poop. https://issues.apache.org/jira/browse/HBASE-24741 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (HBASE-24741) Preserve mvn site output in precommit jobs
Nick Dimiduk created HBASE-24741: Summary: Preserve mvn site output in precommit jobs Key: HBASE-24741 URL: https://issues.apache.org/jira/browse/HBASE-24741 Project: HBase Issue Type: Task Components: build Reporter: Nick Dimiduk It would be nice to see the result of site changes in PRs. This probably balloons the size of archived builds, but we don't (usually) keep PR builds around very long. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-24371) Add more details when print CompactionConfiguration info
[
https://issues.apache.org/jira/browse/HBASE-24371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Dimiduk updated HBASE-24371:
-
Fix Version/s: (was: 2.4.0)
> Add more details when print CompactionConfiguration info
>
>
> Key: HBASE-24371
> URL: https://issues.apache.org/jira/browse/HBASE-24371
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
>Reporter: Lijin Bin
>Assignee: Lijin Bin
>Priority: Minor
> Fix For: 3.0.0-alpha-1, 2.3.0, 2.2.6
>
>
> Now the CompactionConfiguration info logs look like
> {code}
> 2020-05-14 15:32:45,468 INFO
> [RpcServer.default.FPBQ.Fifo.handler=29,queue=2,port=16020]
> compactions.CompactionConfiguration: size [128 MB, 8.00 EB, 8.00 EB); files
> [3, 10); ratio 1.20; off-peak ratio 5.00; throttle point 2684354560;
> major period 60480, major jitter 0.50, min locality to compact
> 0.00; tiered compaction: max_age 9223372036854775807, incoming window min
> 6, compaction policy for tiered window
> org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy,
> single output for minor true, compaction window factory
> org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory
> 2020-05-14 15:32:45,468 INFO
> [RpcServer.default.FPBQ.Fifo.handler=29,queue=2,port=16020]
> compactions.CompactionConfiguration: size [128 MB, 8.00 EB, 8.00 EB); files
> [3, 10); ratio 1.20; off-peak ratio 5.00; throttle point 2684354560;
> major period 60480, major jitter 0.50, min locality to compact
> 0.00; tiered compaction: max_age 9223372036854775807, incoming window min
> 6, compaction policy for tiered window
> org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy,
> single output for minor true, compaction window factory
> org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory
> 2020-05-14 15:32:45,469 INFO
> [RpcServer.default.FPBQ.Fifo.handler=29,queue=2,port=16020]
> compactions.CompactionConfiguration: size [128 MB, 8.00 EB, 8.00 EB); files
> [3, 10); ratio 1.20; off-peak ratio 5.00; throttle point 2684354560;
> major period 60480, major jitter 0.50, min locality to compact
> 0.00; tiered compaction: max_age 9223372036854775807, incoming window min
> 6, compaction policy for tiered window
> org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy,
> single output for minor true, compaction window factory
> org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory
> 2020-05-14 15:32:45,469 INFO
> [RpcServer.default.FPBQ.Fifo.handler=29,queue=2,port=16020]
> compactions.CompactionConfiguration: size [128 MB, 8.00 EB, 8.00 EB); files
> [3, 10); ratio 1.20; off-peak ratio 5.00; throttle point 2684354560;
> major period 60480, major jitter 0.50, min locality to compact
> 0.00; tiered compaction: max_age 9223372036854775807, incoming window min
> 6, compaction policy for tiered window
> org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy,
> single output for minor true, compaction window factory
> org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory
> {code}
> We don't have any details on which region and which column family the
> CompactionConfiguration for. It would be better to include the region name
> and column family detail also in this log.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-8868) add metric to report client shortcircuit reads
[ https://issues.apache.org/jira/browse/HBASE-8868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-8868: Fix Version/s: (was: 2.4.0) > add metric to report client shortcircuit reads > -- > > Key: HBASE-8868 > URL: https://issues.apache.org/jira/browse/HBASE-8868 > Project: HBase > Issue Type: Improvement > Components: metrics, regionserver >Affects Versions: 0.94.8, 0.95.1 >Reporter: Viral Bajaria >Assignee: Wei-Chiu Chuang >Priority: Minor > Fix For: 3.0.0-alpha-1, 2.3.0, 1.7.0, 2.2.5 > > Attachments: HBASE-8868.master.001.patch > > > With the availability of shortcircuit reads, when the feature is enabled > there is no metric which exposes how many times the regionserver was able to > shortcircuit the read and not make a IPC to the datanode. > It will be great to add the metric and expose it via Ganglia. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-21406) "status 'replication'" should not show SINK if the cluster does not act as sink
[
https://issues.apache.org/jira/browse/HBASE-21406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Dimiduk updated HBASE-21406:
-
Fix Version/s: (was: 2.4.0)
> "status 'replication'" should not show SINK if the cluster does not act as
> sink
> ---
>
> Key: HBASE-21406
> URL: https://issues.apache.org/jira/browse/HBASE-21406
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 3.0.0-alpha-1, 2.3.0, 2.4.0, 2.2.5
>Reporter: Daisuke Kobayashi
>Assignee: Wellington Chevreuil
>Priority: Minor
> Fix For: 3.0.0-alpha-1, 2.3.0
>
> Attachments: HBASE-21406-branch-1.001.patch,
> HBASE-21406-master.001.patch, HBASE-21406-master.002.patch, Screen Shot
> 2018-10-31 at 18.12.54.png
>
>
> When replicating in 1 way, from source to target, {{status 'replication'}} on
> source always dumps SINK with meaningless metrics. It only makes sense when
> running the command on target cluster.
> {{status 'replication'}} on source, for example. {{AgeOfLastAppliedOp}} is
> always zero and {{TimeStampsOfLastAppliedOp}} does not get updated from the
> time the RS started since it's not acting as sink.
> {noformat}
> source-1.com
>SOURCE: PeerID=1, AgeOfLastShippedOp=0, SizeOfLogQueue=0,
> TimeStampsOfLastShippedOp=Mon Oct 29 23:44:14 PDT 2018, Replication Lag=0
>SINK : AgeOfLastAppliedOp=0, TimeStampsOfLastAppliedOp=Thu Oct 25
> 23:56:53 PDT 2018
> {noformat}
> {{status 'replication'}} on target works as expected. SOURCE is empty as it's
> not acting as source:
> {noformat}
> target-1.com
>SOURCE:
>SINK : AgeOfLastAppliedOp=70, TimeStampsOfLastAppliedOp=Mon Oct 29
> 23:44:08 PDT 2018
> {noformat}
> This is because {{getReplicationLoadSink}}, called in {{admin.rb}}, always
> returns a value (not null).
> 1.X
> https://github.com/apache/hbase/blob/rel/1.4.0/hbase-client/src/main/java/org/apache/hadoop/hbase/ServerLoad.java#L194-L204
> 2.X
> https://github.com/apache/hbase/blob/rel/2.0.0/hbase-client/src/main/java/org/apache/hadoop/hbase/ServerLoad.java#L392-L399
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-24301) Update Apache POM to version 23
[ https://issues.apache.org/jira/browse/HBASE-24301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-24301: - Fix Version/s: (was: 2.4.0) > Update Apache POM to version 23 > --- > > Key: HBASE-24301 > URL: https://issues.apache.org/jira/browse/HBASE-24301 > Project: HBase > Issue Type: Task >Reporter: Jan Hentschel >Assignee: Jan Hentschel >Priority: Minor > Fix For: 3.0.0-alpha-1, 2.3.0, 1.7.0, 2.2.5 > > > The most recent version of the Apache parent POM is v23. We should update to > this one. There should not be big changes, except that it updates the > rat-plugin to the version we already have. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-23980) Use enforcer plugin to print JVM info in maven output
[
https://issues.apache.org/jira/browse/HBASE-23980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Dimiduk updated HBASE-23980:
-
Fix Version/s: (was: 2.4.0)
> Use enforcer plugin to print JVM info in maven output
> -
>
> Key: HBASE-23980
> URL: https://issues.apache.org/jira/browse/HBASE-23980
> Project: HBase
> Issue Type: Task
> Components: build
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
>Priority: Minor
> Fix For: 3.0.0-alpha-1, 2.3.0, 1.7.0
>
>
> This is a little nice-to-have. We currently dump OS info using
> {{os-maven-plugin}}. Since we build on multiple JVM environments, would be
> nice to add that info too. Looking around, there wasn't an obvious source of
> this information, until I realized it's readily available via
> {{enforcer:display-info}}. Should add this to the top-level pom, if we can
> get it to run just once/build instead of with every module.
> {noformat}
> [INFO] --- maven-enforcer-plugin:3.0.0-M2:display-info (default-cli) @
> hbase-archetype-builder ---
> [INFO] Maven Version: 3.6.3
> [INFO] JDK Version: 1.8.0_222 normalized as: 1.8.0-222
> [INFO] OS Info: Arch: x86_64 Family: mac Name: mac os x Version: 10.14.6
> [INFO]
>
> {noformat}
> {noformat}
> [INFO] --- maven-enforcer-plugin:3.0.0-M2:display-info (default-cli) @
> hbase-archetype-builder ---
> [INFO] Maven Version: 3.6.3
> [INFO] JDK Version: 11.0.4 normalized as: 11.0.4
> [INFO] OS Info: Arch: x86_64 Family: mac Name: mac os x Version: 10.14.6
> [INFO]
>
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-24477) Move ConfigurationObserver and related classes to hbase-common
[ https://issues.apache.org/jira/browse/HBASE-24477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-24477: - Fix Version/s: (was: 2.4.0) > Move ConfigurationObserver and related classes to hbase-common > -- > > Key: HBASE-24477 > URL: https://issues.apache.org/jira/browse/HBASE-24477 > Project: HBase > Issue Type: Task > Components: conf >Reporter: Bharath Vissapragada >Assignee: Bharath Vissapragada >Priority: Minor > Fix For: 3.0.0-alpha-1, 2.3.0, 1.7.0 > > > It's a nice utility that all the modules can leverage. Putting it in common > makes it accessible. I want to use it in client for dynamic reconfiguration > of master addresses in HBASE-18095. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-24468) Add region info when log meessages in HStore.
[ https://issues.apache.org/jira/browse/HBASE-24468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-24468: - Fix Version/s: (was: 2.4.0) > Add region info when log meessages in HStore. > - > > Key: HBASE-24468 > URL: https://issues.apache.org/jira/browse/HBASE-24468 > Project: HBase > Issue Type: Improvement > Components: logging, regionserver >Affects Versions: 3.0.0-alpha-1 >Reporter: song XinCun >Assignee: song XinCun >Priority: Minor > Fix For: 3.0.0-alpha-1, 2.3.0 > > > Some log message do not have region info when log, need to add it. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-24524) SyncTable logging improvements
[ https://issues.apache.org/jira/browse/HBASE-24524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-24524: - Fix Version/s: (was: 2.4.0) Affects Version/s: (was: 2.4.0) > SyncTable logging improvements > -- > > Key: HBASE-24524 > URL: https://issues.apache.org/jira/browse/HBASE-24524 > Project: HBase > Issue Type: Improvement >Affects Versions: 3.0.0-alpha-1, 2.3.0, 2.2.5 >Reporter: Wellington Chevreuil >Assignee: Wellington Chevreuil >Priority: Minor > Fix For: 3.0.0-alpha-1, 2.3.0, 2.2.6 > > > While troubleshooting mismatches in replication deployment, SyncTable logging > can provide some insights on what is diverging between two clusters. One > caveat, though, is that it logs diverging row key as hexdecimal values, which > is not so useful for operators trying to figure out which rows are > mismatching, ideally, this info should be human readable, so that operators > could have the exact value they could use for querying the tables with other > tools, such as hbase shell. > Another issue is that while rows mismatches are logged as info, cell values > mismatches are only logged as debug. In general, any of the mismatches would > already be quite verbose, so ideally both should be logged in debug level. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-24115) Relocate test-only REST "client" from src/ to test/ and mark Private
[ https://issues.apache.org/jira/browse/HBASE-24115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-24115: - Fix Version/s: (was: 2.4.0) > Relocate test-only REST "client" from src/ to test/ and mark Private > > > Key: HBASE-24115 > URL: https://issues.apache.org/jira/browse/HBASE-24115 > Project: HBase > Issue Type: Test > Components: REST, security >Reporter: Andrew Kyle Purtell >Assignee: Andrew Kyle Purtell >Priority: Major > Fix For: 3.0.0-alpha-1, 2.3.0, 1.3.7, 1.7.0, 2.1.10, 1.4.14, 2.2.5 > > > Relocate test-only REST "client" from src/ to test/ and annotate as Private. > The classes o.a.h.h.rest.Remote* were developed to facilitate REST unit tests > and incorrectly committed to src/ . > Although this "breaks" compatibility by moving public classes to test jar and > marking them private, no attention has been paid to these classes with > respect to performance, convenience, or security. Consensus from various > discussions over the years is to move them to test/ as was intent of the > original committer, but misplaced by the same individual. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-24423) No need to get lock in canSplit because hasReferences will get lock too
[ https://issues.apache.org/jira/browse/HBASE-24423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-24423: - Fix Version/s: (was: 2.4.0) > No need to get lock in canSplit because hasReferences will get lock too > --- > > Key: HBASE-24423 > URL: https://issues.apache.org/jira/browse/HBASE-24423 > Project: HBase > Issue Type: Improvement > Components: regionserver >Reporter: Zheng Wang >Assignee: Zheng Wang >Priority: Minor > Fix For: 3.0.0-alpha-1, 2.3.0, 1.7.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-23999) [flakey test] TestTableOutputFormatConnectionExhaust
[
https://issues.apache.org/jira/browse/HBASE-23999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Dimiduk updated HBASE-23999:
-
Fix Version/s: (was: 2.4.0)
> [flakey test] TestTableOutputFormatConnectionExhaust
>
>
> Key: HBASE-23999
> URL: https://issues.apache.org/jira/browse/HBASE-23999
> Project: HBase
> Issue Type: Test
> Components: test
>Affects Versions: 2.3.0
>Reporter: Nick Dimiduk
>Assignee: Huaxiang Sun
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.3.0, 2.2.5
>
>
> Hit this during master startup sequence in the test.
> {noformat}
> 2020-03-16 23:40:37,298 ERROR [StoreOpener-1588230740-1]
> conf.Configuration(2980): error parsing conf hbase-site.xml
> com.ctc.wstx.exc.WstxEOFException: Unexpected EOF in prolog
> at [row,col,system-id]:
> [1,0,"file:/home/vagrant/repos/hbase/hbase-mapreduce/target/test-classes/hbase-site.xml"]
> at
> com.ctc.wstx.sr.StreamScanner.throwUnexpectedEOF(StreamScanner.java:687)
> at
> com.ctc.wstx.sr.BasicStreamReader.handleEOF(BasicStreamReader.java:2220)
> at
> com.ctc.wstx.sr.BasicStreamReader.nextFromProlog(BasicStreamReader.java:2126)
> at com.ctc.wstx.sr.BasicStreamReader.next(BasicStreamReader.java:1181)
> at
> org.apache.hadoop.conf.Configuration$Parser.parseNext(Configuration.java:3277)
> at
> org.apache.hadoop.conf.Configuration$Parser.parse(Configuration.java:3071)
> at
> org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2964)
> at
> org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2930)
> at
> org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2805)
> at org.apache.hadoop.conf.Configuration.get(Configuration.java:1199)
> at
> org.apache.hadoop.conf.Configuration.getTrimmed(Configuration.java:1253)
> at
> org.apache.hadoop.conf.Configuration.getBoolean(Configuration.java:1659)
> at
> org.apache.hadoop.hbase.HBaseConfiguration.checkDefaultsVersion(HBaseConfiguration.java:70)
> at
> org.apache.hadoop.hbase.HBaseConfiguration.addHbaseResources(HBaseConfiguration.java:84)
> at
> org.apache.hadoop.hbase.HBaseConfiguration.create(HBaseConfiguration.java:98)
> at org.apache.hadoop.hbase.io.crypto.Context.(Context.java:44)
> at
> org.apache.hadoop.hbase.io.crypto.Encryption$Context.(Encryption.java:64)
> at
> org.apache.hadoop.hbase.io.crypto.Encryption$Context.(Encryption.java:61)
> at org.apache.hadoop.hbase.regionserver.HStore.(HStore.java:228)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:5890)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:1096)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:1093)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)
> 2020-03-16 23:40:37,301 ERROR [master/bionic:0:becomeActiveMaster]
> regionserver.HRegion(1137): Could not initialize all stores for the
> region=hbase:meta,,1.1588230740
> {noformat}
> Looking at the file under {{target/test-classes}}, it looks like this is a
> file written by YARN.
> {noformat}
>
> yarn.log-aggregation.file-formatsTFilefalseyarn-default.xml
> hbase.master.mob.ttl.cleaner.period86400falsehbase-default.xml
> dfs.namenode.resource.check.interval5000falsehdfs-default.xml
> mapreduce.jobhistory.client.thread-count10falsemapred-default.xml
> ...
> {noformat}
> My guess is that we have something in the MR framework unconfigured, it's
> writing these temporary job files to some default (like the first class path
> location or something??) and parallel test runs are stomping on each other.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-24441) CacheConfig details logged at Store open is not really useful
[
https://issues.apache.org/jira/browse/HBASE-24441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Dimiduk updated HBASE-24441:
-
Fix Version/s: (was: 2.4.0)
> CacheConfig details logged at Store open is not really useful
> -
>
> Key: HBASE-24441
> URL: https://issues.apache.org/jira/browse/HBASE-24441
> Project: HBase
> Issue Type: Improvement
> Components: logging, regionserver
>Affects Versions: 3.0.0-alpha-1
>Reporter: Anoop Sam John
>Assignee: song XinCun
>Priority: Minor
> Fix For: 3.0.0-alpha-1, 2.3.0
>
>
> CacheConfig constructor is logging 'this' object at INFO level. This log
> comes during Store open(As CacheConfig instance for that store is created).
> As the log is at CacheConfig only, we don't get to know this is for which
> region:store. So not really useful log.
> {code}
> blockCache=org.apache.hadoop.hbase.io.hfile.CombinedBlockCache@7bc02941,
> cacheDataOnRead=true, cacheDataOnWrite=true, cacheIndexesOnWrite=false,
> cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false,
> prefetchOnOpen=false
> {code}
> Also during every compaction also this logs keeps coming. This is because
> during compaction we create new CacheConfig based on the HStore level
> CacheConfig object. We can avoid this log with every compaction happening.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-24019) Correct exception messages for table null and namespace unavailable.
[
https://issues.apache.org/jira/browse/HBASE-24019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Dimiduk updated HBASE-24019:
-
Fix Version/s: (was: 2.4.0)
> Correct exception messages for table null and namespace unavailable.
>
>
> Key: HBASE-24019
> URL: https://issues.apache.org/jira/browse/HBASE-24019
> Project: HBase
> Issue Type: Bug
>Reporter: Mohammad Arshad
>Assignee: Mohammad Arshad
>Priority: Minor
> Fix For: 2.3.0, 2.1.10, 2.2.5
>
>
> Exception message for following two scenarios should be corrected.
> 1. Change message to "The list of tables cannot be null." in below code
> {code:java}
> @Override
> public void moveTables(Set tables, String targetGroup) throws
> IOException {
> if (tables == null) {
> throw new ConstraintException("The list of servers cannot be null.");
> }
> {code}
> 2. Change the message to "Region server group "+group+" does not exist" in
> below code.
> {code:java}
> public void preCreateNamespace(ObserverContext
> ctx,
> NamespaceDescriptor ns) throws IOException {
> String group =
> ns.getConfigurationValue(RSGroupInfo.NAMESPACE_DESC_PROP_GROUP);
> if(group != null && groupAdminServer.getRSGroupInfo(group) == null) {
> throw new ConstraintException("Region server group "+group+" does not
> exit");
> }
> }
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-24302) Add an "ignoreTimestamps" option (defaulted to false) to HashTable/SyncTable tool
[ https://issues.apache.org/jira/browse/HBASE-24302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-24302: - Fix Version/s: (was: 2.4.0) > Add an "ignoreTimestamps" option (defaulted to false) to HashTable/SyncTable > tool > - > > Key: HBASE-24302 > URL: https://issues.apache.org/jira/browse/HBASE-24302 > Project: HBase > Issue Type: Improvement >Affects Versions: 3.0.0-alpha-1, 2.3.0, 2.2.5 >Reporter: Wellington Chevreuil >Assignee: Wellington Chevreuil >Priority: Major > Fix For: 3.0.0-alpha-1, 2.3.0, 2.2.5 > > > Currently, when hashing and comparing values between a source and a target > table, HashTable/SyncTable always consider cell timestamp values. However, > cell timestamp values are not always relevant for client applications, so > these use cases could benefit of a more flexible comparison logic where > timestamps could be ignored. > For such scenarios, HashTable/SyncTable could have better performance, since > cells with only timestamps diverging would not be copied. > Another case that would benefit from this option is when bulk deletes are > wrongly applied at target. At the moment, HashTable/SyncTable on it's own is > not capable of syncing back the clusters, as the source Puts would have an > older TS than the delete markers in the target. That would require target to > complete major compaction on the whole table before HashTable/SyncTable could > be run. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-24099) Use a fair ReentrantReadWriteLock for the region close lock
[
https://issues.apache.org/jira/browse/HBASE-24099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Dimiduk updated HBASE-24099:
-
Fix Version/s: (was: 2.4.0)
> Use a fair ReentrantReadWriteLock for the region close lock
> ---
>
> Key: HBASE-24099
> URL: https://issues.apache.org/jira/browse/HBASE-24099
> Project: HBase
> Issue Type: Improvement
>Reporter: Andrew Kyle Purtell
>Assignee: Andrew Kyle Purtell
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.3.0, 1.3.7, 1.7.0, 2.1.10, 1.4.14, 2.2.5
>
> Attachments: ltt_results.pdf, pe_results.pdf, ycsb_results.pdf
>
>
> Consider creating the region's ReentrantReadWriteLock with the fair locking
> policy. We have had a couple of production incidents where a regionserver
> stalled in shutdown for a very very long time, leading to RIT (FAILED_CLOSE).
> The latest example is a 43 minute shutdown, ~40 minutes (2465280 ms) of that
> time was spent waiting to acquire the write lock on the region in order to
> finish closing it.
> {quote}
> ...
> Finished memstore flush of ~66.92 MB/70167112, currentsize=0 B/0 for region
> . in 927ms, sequenceid=6091133815, compaction requested=false at
> 1585175635349 (+60 ms)
> Disabling writes for close at 1585178100629 (+2465280 ms)
> {quote}
> This time was spent in between the memstore flush and the task status change
> "Disabling writes for close at...". This is at HRegion.java:1481 in 1.3.6:
> {code}
> 1480: // block waiting for the lock for closing
> 1481: lock.writeLock().lock(); // FindBugs: Complains
> UL_UNRELEASED_LOCK_EXCEPTION_PATH but seems fine
> {code}
>
> The close lock is operating in unfair mode. The table in question is under
> constant high query load. When the close request was received, there were
> active readers. After the close request there were more active readers,
> near-continuous contention. Although the clients would receive
> RegionServerStoppingException and other error notifications, because the
> region could not be reassigned, they kept coming, region (re-)location would
> find the region still hosted on the stuck server. Finally the closing thread
> waiting for the write lock became no longer starved (by chance) after 40
> minutes.
> The ReentrantReadWriteLock javadoc is clear about the possibility of
> starvation when continuously contended: "_When constructed as non-fair (the
> default), the order of entry to the read and write lock is unspecified,
> subject to reentrancy constraints. A nonfair lock that is continuously
> contended may indefinitely postpone one or more reader or writer threads, but
> will normally have higher throughput than a fair lock._"
> We could try changing the acquisition semantics of this lock to fair. This is
> a one line change, where we call the RW lock constructor. Then:
> "_When constructed as fair, threads contend for entry using an approximately
> arrival-order policy. When the currently held lock is released, either the
> longest-waiting single writer thread will be assigned the write lock, or if
> there is a group of reader threads waiting longer than all waiting writer
> threads, that group will be assigned the read lock._"
> This could be better. The close process will have to wait until all readers
> and writers already waiting for acquisition either acquire and release or go
> away but won't be starved by future/incoming requests.
> There could be a throughput loss in request handling, though, because this is
> the global reentrant RW lock for the region.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-24252) Implement proxyuser/doAs mechanism for hbase-http
[ https://issues.apache.org/jira/browse/HBASE-24252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-24252: - Fix Version/s: (was: 2.4.0) > Implement proxyuser/doAs mechanism for hbase-http > - > > Key: HBASE-24252 > URL: https://issues.apache.org/jira/browse/HBASE-24252 > Project: HBase > Issue Type: Improvement > Components: security, UI >Reporter: Istvan Toth >Assignee: Istvan Toth >Priority: Major > Fix For: 3.0.0-alpha-1, 2.3.0, 2.2.5 > > > The REST and Thrift interfaces for HBase already implement the standard > hadoop ProxyUser mechanism for SPNEGO, but it is not implemented in > hbase-httpserver. > Implement it. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-23896) Snapshot owner cannot delete snapshot when ACL is enabled and Kerberos is not enabled
[
https://issues.apache.org/jira/browse/HBASE-23896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Dimiduk updated HBASE-23896:
-
Fix Version/s: (was: 2.4.0)
> Snapshot owner cannot delete snapshot when ACL is enabled and Kerberos is not
> enabled
> -
>
> Key: HBASE-23896
> URL: https://issues.apache.org/jira/browse/HBASE-23896
> Project: HBase
> Issue Type: Task
>Affects Versions: 3.0.0-alpha-1, 2.2.3
>Reporter: Guangxu Cheng
>Assignee: Guangxu Cheng
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.3.0, 2.2.5
>
> Attachments: HBASE-23896-branch-2.2-addendum.patch
>
>
> When ACL is enabled and Kerberos is not enabled, the snapshot owner cannot
> delete the snapshot. This is because the owner of the snapshot cannot be
> taken during permission verification. By investigation, found that only after
> HBase has enabled security authentication, the owner will be set when doing
> snapshot.
> SnapshotManager#takeSnapshotInternal
> {code:title=SnapshotManager.java|borderStyle=solid}
> RpcServer.getRequestUser().ifPresent(user -> {
> if (User.isHBaseSecurityEnabled(master.getConfiguration())) {
> builder.setOwner(user.getShortName());
> }
> });
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-24111) Enable CompactionTool executions on non-HDFS filesystems
[
https://issues.apache.org/jira/browse/HBASE-24111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Dimiduk updated HBASE-24111:
-
Fix Version/s: (was: 2.4.0)
> Enable CompactionTool executions on non-HDFS filesystems
>
>
> Key: HBASE-24111
> URL: https://issues.apache.org/jira/browse/HBASE-24111
> Project: HBase
> Issue Type: Improvement
> Components: Compaction, mapreduce, tooling
>Reporter: Peter Somogyi
>Assignee: Peter Somogyi
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.3.0, 2.2.5
>
>
> CompactionTool on non-HDFS filesystems does not work because the used
> filesystems are mixed up. To collect the StoreFiles the
> {{Filesystem.get(conf)}} method is used instead of relying on the root dir
> filesystem.
> With YARN the delegation tokens are not handled correctly with a different
> filesystem and the mappers only get delegation token for the staging
> directory's filesystem.
> Another issue is in MapReduce mode when YARN is used. In this case the TMP
> directory ({{hbase.tmp.dir}}) is created under
> {{yarn.nodemanager.local-dirs}} but this directory is created (even on HDFS a
> regular user can't create this directory) on the Storefile's filesystem where
> a local filesystem temp directory shouldn't be used.
> In this ticket I plan to clean up the used filesystems, remove the custom
> temp directory (use the regions' .tmp directory) and collect delegation
> tokens for staging dir + store dir paths' FileSystems.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-24021) Fail fast when bulkLoadHFiles method catch some IOException
[
https://issues.apache.org/jira/browse/HBASE-24021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Dimiduk updated HBASE-24021:
-
Fix Version/s: (was: 2.4.0)
> Fail fast when bulkLoadHFiles method catch some IOException
> ---
>
> Key: HBASE-24021
> URL: https://issues.apache.org/jira/browse/HBASE-24021
> Project: HBase
> Issue Type: Improvement
> Components: HFile, regionserver
>Reporter: niuyulin
>Assignee: niuyulin
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.3.0, 2.2.5
>
>
> In production environment, we usually do bulkload huge amount hfile . It
> reasonable fail fast when any IOException occur
>
> hbase/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
> {code:java}
> public Map> bulkLoadHFiles(Collection String>> familyPaths,
> boolean assignSeqId, BulkLoadListener bulkLoadListener,
> boolean copyFile, List clusterIds, boolean replicate) throws
> IOException {
> ..
> try {
> this.writeRequestsCount.increment();
> // There possibly was a split that happened between when the split keys
> // were gathered and before the HRegion's write lock was taken. We need
> // to validate the HFile region before attempting to bulk load all of them
> List ioes = new ArrayList<>();
> List> failures = new ArrayList<>();
> for (Pair p : familyPaths) {
> byte[] familyName = p.getFirst();
> String path = p.getSecond();
> HStore store = getStore(familyName);
> if (store == null) {
> IOException ioe = new org.apache.hadoop.hbase.DoNotRetryIOException(
> "No such column family " + Bytes.toStringBinary(familyName));
> ioes.add(ioe);
> } else {
> try {
> store.assertBulkLoadHFileOk(new Path(path));
> } catch (WrongRegionException wre) {
> // recoverable (file doesn't fit in region)
> failures.add(p);
> } catch (IOException ioe) {
> // unrecoverable (hdfs problem)
> ioes.add(ioe);
> }
> }
> }
> // validation failed because of some sort of IO problem.
> if (ioes.size() != 0) {
> IOException e = MultipleIOException.createIOException(ioes);
> LOG.error("There were one or more IO errors when checking if the bulk
> load is ok.", e);
> throw e;
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-24138) Ensure StochasticLoadBalancer can log details of decision to not run balancer
[
https://issues.apache.org/jira/browse/HBASE-24138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Dimiduk updated HBASE-24138:
-
Fix Version/s: (was: 2.4.0)
> Ensure StochasticLoadBalancer can log details of decision to not run balancer
> -
>
> Key: HBASE-24138
> URL: https://issues.apache.org/jira/browse/HBASE-24138
> Project: HBase
> Issue Type: Task
> Components: Balancer, Operability
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.3.0, 1.7.0, 2.2.5
>
>
> Ran into a customer case where the StochasticLoadBalancer was consistently
> deciding not to balance when bringing new region servers on line. Even
> setting the class to TRACE logging would only log a summary statement like:
> {code}
> 2020-04-03 00:29:55,133 TRACE
> org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer: Skipping load
> balancing because balanced cluster; total cost is 25.24853189705185, sum
> multiplier is 602.0 min cost which need balance is 0.05
> {code}
> Without any details about what went into that decision it's really hard to
> figure out what we need to tune to get the behavior we want.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-24515) batch Increment/Append fails when retrying the RPC
[
https://issues.apache.org/jira/browse/HBASE-24515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Dimiduk updated HBASE-24515:
-
Fix Version/s: (was: 2.4.0)
> batch Increment/Append fails when retrying the RPC
> --
>
> Key: HBASE-24515
> URL: https://issues.apache.org/jira/browse/HBASE-24515
> Project: HBase
> Issue Type: Bug
>Reporter: Toshihiro Suzuki
>Assignee: Toshihiro Suzuki
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.3.0, 2.1.10, 2.2.6
>
>
> When a client hits RPC timeout and sends a second RPC request for batch
> Increment/Append but the first RPC is already processed actually, the nonce
> of the RPC is saved in the RS.
> In this case, for the second RPC, the RS just reads the previous result and
> returns it to the client to avoid the Increment/Append is processed twice.
> At that time, for batch Increment/Append, we try to create a Get object from
> a CellScanner object in the following code:
>
> [https://github.com/apache/hbase/blob/66452afc09d8b82927e5e58565f97939faa22c7b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java#L773-L776]
> However, the CellScanner object is already consumed to create the
> Increment/Append object in the following code, and it fails with the
> following exception:
>
> [https://github.com/apache/hbase/blob/66452afc09d8b82927e5e58565f97939faa22c7b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java#L757]
> {code:java}
> 2020-06-06 14:09:06,153 WARN [hconnection-0x79c3903e-shared-pool3-t209]
> client.AsyncRequestFutureImpl: id=1, table=REF_Test, attempt=3/36,
> failureCount=1ops, last
> exception=org.apache.hadoop.hbase.DoNotRetryIOException:
> org.apache.hadoop.hbase.DoNotRetryIOException: Cell count of 1 but at index 0
> no cell returned: row: "xxx" mutate_type: INCREMENT timestamp:
> 9223372036854775807 durability: USE_DEFAULT time_range { from: 0 to:
> 9223372036854775807 } associated_cell_count: 1 nonce: 5281583076322914765
> at
> org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toGet(ProtobufUtil.java:934)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.increment(RSRpcServices.java:737)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:877)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2702)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42202)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-23998) Update license for jetty-client
[
https://issues.apache.org/jira/browse/HBASE-23998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Dimiduk updated HBASE-23998:
-
Fix Version/s: (was: 2.4.0)
> Update license for jetty-client
> ---
>
> Key: HBASE-23998
> URL: https://issues.apache.org/jira/browse/HBASE-23998
> Project: HBase
> Issue Type: Bug
> Components: build, dependencies
>Affects Versions: 3.0.0-alpha-1
> Environment: HBase master branch on Apache Hadoop 3.3.0
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.3.0, 2.2.5
>
>
> After HBASE-22103, compiling on Haddop 3.3.0 has the following error:
> {{mvn clean install -Dhadoop.profile=3.0 -Dhadoop.version=3.3.0-SNAPSHOT
> -DskipTests -Dmaven.javadoc.skip=true}}
> {noformat}
> This product includes Jetty :: Asynchronous HTTP Client licensed under the
> Apache Software License - Version 2.0.
> ERROR: Please check this License for acceptability here:
> https://www.apache.org/legal/resolved
> If it is okay, then update the list named 'non_aggregate_fine' in the
> LICENSE.vm file.
> If it isn't okay, then revert the change that added the dependency.
> More info on the dependency:
> org.eclipse.jetty
> jetty-client
> 9.4.20.v20190813
> {noformat}
> This is caused by YARN-8778 which added dependency on
> org.eclipse.jetty.websocket:websocket-client, and the Jetty 9.4 update in
> HADOOP-16152.
> {noformat}
> [INFO] +-
> org.apache.hadoop:hadoop-mapreduce-client-core:jar:3.3.0-SNAPSHOT:compile
> [INFO] | +- org.apache.hadoop:hadoop-yarn-client:jar:3.3.0-SNAPSHOT:compile
> [INFO] | | +-
> org.eclipse.jetty.websocket:websocket-client:jar:9.4.20.v20190813:compile
> [INFO] | | | +- org.eclipse.jetty:jetty-client:jar:9.4.20.v20190813:compile
> [INFO] | | | \-
> org.eclipse.jetty.websocket:websocket-common:jar:9.4.20.v20190813:compile
> [INFO] | | | \-
> org.eclipse.jetty.websocket:websocket-api:jar:9.4.20.v20190813:compile
> {noformat}
> Propose: update
> hbase-resource-bundle/src/main/resources/supplemental-models.xml to update
> the license text for jetty-client.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-24041) [regression] Increase RESTServer buffer size back to 64k
[
https://issues.apache.org/jira/browse/HBASE-24041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Dimiduk updated HBASE-24041:
-
Fix Version/s: (was: 2.4.0)
> [regression] Increase RESTServer buffer size back to 64k
> -
>
> Key: HBASE-24041
> URL: https://issues.apache.org/jira/browse/HBASE-24041
> Project: HBase
> Issue Type: Bug
> Components: REST
>Affects Versions: 3.0.0-alpha-1, 2.2.0, 2.3.0, 2.4.0
>Reporter: Esteban Gutierrez
>Assignee: Esteban Gutierrez
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.3.0, 2.2.5
>
>
> HBASE-14492 is not longer present in our current releases after HBASE-12894.
> Unfortunately our RESTServer is not extending HttpServer which means that
> {{DEFAULT_MAX_HEADER_SIZE}} is not being set and HTTP requests with a very
> large header can still cause connection issues for clients. A quick fix is
> just to add the settings to the {{HttpConfiguration}} configuration object. A
> long term solution should be to re-factor services that create an HTTP server
> and normalize all configuration settings across all of them.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
