[GitHub] [hbase] infraio commented on a change in pull request #2663: HBASE-24637 - Reseek regression related to filter SKIP hinting
infraio commented on a change in pull request #2663: URL: https://github.com/apache/hbase/pull/2663#discussion_r543113766 ## File path: hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java ## @@ -130,6 +132,13 @@ */ public static final String HBASE_CELLS_SCANNED_PER_HEARTBEAT_CHECK = "hbase.cells.scanned.per.heartbeat.check"; + /** + * The number of bytes after which we decide if to do only next() or continue + * with seek(). Currently it defaults to the STORESCANNER_PREAD_MAX_BYTES config + * To disable this feature put a value < 0. + */ + public static final String HBASE_SWITCH_TO_NEXT_AFTER_BYTES_READ = Review comment: I check your example in PPT. What this PR want to reduce is the compare times of "matcher.compareKeyForNextRow(nextIndexedKey, cell)", right? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache9 commented on pull request #2748: HBASE-25373 Remove HTrace completely in code base and try to make use…
Apache9 commented on pull request #2748: URL: https://github.com/apache/hbase/pull/2748#issuecomment-745112571 Any other concerns? @saintstack Thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] ramkrish86 commented on pull request #2747: HBASE-24850 CellComparator perf improvement
ramkrish86 commented on pull request #2747: URL: https://github.com/apache/hbase/pull/2747#issuecomment-745111755 Sure Stack let me close this. But before that I will ensure I read all your comments and address them in the other PR including some profiling. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-24850) CellComparator perf improvement
[ https://issues.apache.org/jira/browse/HBASE-24850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17249534#comment-17249534 ] ramkrishna.s.vasudevan commented on HBASE-24850: bq.Can we have compare(Cell) in ExtendedCell? Yes I looked into this option. What it will end up finally is that say if KV implements this compare(Cell). Now if the current cell (as pointed by 'this') and the incoming Cell are KVs ya we can handle it in KV. But if the incoming cell is BBKV then it means 'this' (referring to KV) needs to compare with BBKV. Then we should add that logic inside the compare(Cell) of KV. So it would be better to move such things to an Util method. that is why I felt that redirection from compare() to that Util method is not needed. > CellComparator perf improvement > --- > > Key: HBASE-24850 > URL: https://issues.apache.org/jira/browse/HBASE-24850 > Project: HBase > Issue Type: Improvement > Components: Performance, scan >Affects Versions: 2.0.0 >Reporter: Anoop Sam John >Assignee: ramkrishna.s.vasudevan >Priority: Critical > Fix For: 3.0.0-alpha-1, 2.5.0 > > > We have multiple perf issues in 2.x versions compared to 1.x. Eg: > HBASE-24754, HBASE-24637. > The pattern is clear that where ever we do more and more Cell compares, there > is some degrade. In HBASE-24754, with an old KVComparator style comparator, > we see much better perf for the PutSortReducer. (Again the gain is huge > because of large number of compare ops that test is doing). This issue is to > address and optimize compares generally in CellComparatorImpl itself. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #2745: [HBASE-25246] Backup/Restore hbase cell tags.
Apache-HBase commented on pull request #2745: URL: https://github.com/apache/hbase/pull/2745#issuecomment-745110530 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 33s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 29s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 26s | master passed | | +1 :green_heart: | checkstyle | 2m 3s | master passed | | +0 :ok: | refguide | 3m 23s | branch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect. | | +1 :green_heart: | spotbugs | 12m 21s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 29s | the patch passed | | +1 :green_heart: | checkstyle | 2m 29s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +0 :ok: | refguide | 3m 26s | patch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect. | | +1 :green_heart: | hadoopcheck | 17m 6s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | spotbugs | 15m 40s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 52s | The patch does not generate ASF License warnings. | | | | 73m 49s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2745/10/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2745 | | JIRA Issue | HBASE-25246 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle refguide | | uname | Linux 65e80cadad18 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / d50816fe44 | | refguide | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2745/10/artifact/yetus-general-check/output/branch-site/book.html | | refguide | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2745/10/artifact/yetus-general-check/output/patch-site/book.html | | Max. process+thread count | 139 (vs. ulimit of 3) | | modules | C: hbase-client hbase-server hbase-mapreduce . U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2745/10/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=3.1.12 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-24850) CellComparator perf improvement
[ https://issues.apache.org/jira/browse/HBASE-24850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17249533#comment-17249533 ] ramkrishna.s.vasudevan commented on HBASE-24850: The branching and the parsing of the cells parts are both the reasons. In 1.3.x we don't do branching (as only one cell type is available)but we are not parsing the cell parts indvidually. So that causes a performance issue. In 2.x we do branching and also don't do parsing. So it is having more impact. Now in the recent patch we do the parsing so it becomes faster but also in the branching we remove branches getting created again like inside compareRows, compareFam, compareCols etc. That helps. There is also a reason why the bulk load perf is slower based on these factors. > CellComparator perf improvement > --- > > Key: HBASE-24850 > URL: https://issues.apache.org/jira/browse/HBASE-24850 > Project: HBase > Issue Type: Improvement > Components: Performance, scan >Affects Versions: 2.0.0 >Reporter: Anoop Sam John >Assignee: ramkrishna.s.vasudevan >Priority: Critical > Fix For: 3.0.0-alpha-1, 2.5.0 > > > We have multiple perf issues in 2.x versions compared to 1.x. Eg: > HBASE-24754, HBASE-24637. > The pattern is clear that where ever we do more and more Cell compares, there > is some degrade. In HBASE-24754, with an old KVComparator style comparator, > we see much better perf for the PutSortReducer. (Again the gain is huge > because of large number of compare ops that test is doing). This issue is to > address and optimize compares generally in CellComparatorImpl itself. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache9 commented on pull request #2769: HBASE-25032 Wait for region server to become online before adding it to online servers in Master
Apache9 commented on pull request #2769: URL: https://github.com/apache/hbase/pull/2769#issuecomment-745110067 What about just add a flag in report duty to tell master whether the region server is ready to take regions? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-24850) CellComparator perf improvement
[ https://issues.apache.org/jira/browse/HBASE-24850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17249532#comment-17249532 ] ramkrishna.s.vasudevan commented on HBASE-24850: bq.I thought BBKV was default in branch-2? Sounds like its not? Ya it is not. When we use a file system based cache even there is i KV only. And anyway the keys that gets added to the memstore are purely KV only. We generally don't turn on offheap write path. > CellComparator perf improvement > --- > > Key: HBASE-24850 > URL: https://issues.apache.org/jira/browse/HBASE-24850 > Project: HBase > Issue Type: Improvement > Components: Performance, scan >Affects Versions: 2.0.0 >Reporter: Anoop Sam John >Assignee: ramkrishna.s.vasudevan >Priority: Critical > Fix For: 3.0.0-alpha-1, 2.5.0 > > > We have multiple perf issues in 2.x versions compared to 1.x. Eg: > HBASE-24754, HBASE-24637. > The pattern is clear that where ever we do more and more Cell compares, there > is some degrade. In HBASE-24754, with an old KVComparator style comparator, > we see much better perf for the PutSortReducer. (Again the gain is huge > because of large number of compare ops that test is doing). This issue is to > address and optimize compares generally in CellComparatorImpl itself. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #2766: [HBASE-25246] Backup/Restore hbase cell tags.
Apache-HBase commented on pull request #2766: URL: https://github.com/apache/hbase/pull/2766#issuecomment-745106747 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 49s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ branch-2 Compile Tests _ | | +0 :ok: | mvndep | 0m 30s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 28s | branch-2 passed | | +1 :green_heart: | checkstyle | 2m 14s | branch-2 passed | | +0 :ok: | refguide | 3m 30s | branch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect. | | +1 :green_heart: | spotbugs | 12m 11s | branch-2 passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 11s | the patch passed | | +1 :green_heart: | checkstyle | 2m 6s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +0 :ok: | refguide | 3m 36s | patch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect. | | +1 :green_heart: | hadoopcheck | 11m 37s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. | | +1 :green_heart: | spotbugs | 12m 56s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 48s | The patch does not generate ASF License warnings. | | | | 65m 11s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2766/6/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2766 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle refguide | | uname | Linux ce7546a308d9 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / 9c2f1f7a2b | | refguide | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2766/6/artifact/yetus-general-check/output/branch-site/book.html | | refguide | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2766/6/artifact/yetus-general-check/output/patch-site/book.html | | Max. process+thread count | 137 (vs. ulimit of 12500) | | modules | C: hbase-client hbase-server hbase-mapreduce . U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2766/6/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=3.1.12 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] ramkrish86 commented on a change in pull request #2663: HBASE-24637 - Reseek regression related to filter SKIP hinting
ramkrish86 commented on a change in pull request #2663: URL: https://github.com/apache/hbase/pull/2663#discussion_r543102351 ## File path: hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java ## @@ -130,6 +132,13 @@ */ public static final String HBASE_CELLS_SCANNED_PER_HEARTBEAT_CHECK = "hbase.cells.scanned.per.heartbeat.check"; + /** + * The number of bytes after which we decide if to do only next() or continue + * with seek(). Currently it defaults to the STORESCANNER_PREAD_MAX_BYTES config + * To disable this feature put a value < 0. + */ + public static final String HBASE_SWITCH_TO_NEXT_AFTER_BYTES_READ = Review comment: Currently am having 4 * blockSize as the default value now. I cannot have it less than one block size because unless I traverse multiple blocks I cannot make out this pattern. Within one block we cannot decide. I may be not getting your Q clearly - if that is the case do let me know I can try to explain it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] ramkrish86 commented on a change in pull request #2663: HBASE-24637 - Reseek regression related to filter SKIP hinting
ramkrish86 commented on a change in pull request #2663: URL: https://github.com/apache/hbase/pull/2663#discussion_r543101694 ## File path: hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java ## @@ -728,7 +739,7 @@ public boolean next(List outResult, ScannerContext scannerContext) throws break; case SEEK_NEXT_COL: -seekOrSkipToNextColumn(cell); +doSeekCol(cell); Review comment: Ok . I can see how can that be done. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache9 commented on pull request #2779: HBASE-23340 hmaster /hbase/replication/rs session expired (hbase repl…
Apache9 commented on pull request #2779: URL: https://github.com/apache/hbase/pull/2779#issuecomment-745103309 Please fix the checkstyle issue? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] sandeepvinayak commented on a change in pull request #2778: HBASE-25383: Ability to update and remove peer base config
sandeepvinayak commented on a change in pull request #2778: URL: https://github.com/apache/hbase/pull/2778#discussion_r543090129 ## File path: hbase-client/src/main/java/org/apache/hadoop/hbase/client/replication/ReplicationPeerConfigUtil.java ## @@ -64,6 +65,9 @@ public static final String HBASE_REPLICATION_PEER_BASE_CONFIG = "hbase.replication.peer.base.config"; + public static final String HBASE_REPLICATION_PEER_REMOVE_BASE_CONFIG = Review comment: @bharathv do you mean setting k1="" if we want to remove k1 configuration? Just setting "" doesn't really communicate the program what we want to remove because there might be other configs added from outside of hbase-site. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] sandeepvinayak commented on a change in pull request #2778: HBASE-25383: Ability to update and remove peer base config
sandeepvinayak commented on a change in pull request #2778: URL: https://github.com/apache/hbase/pull/2778#discussion_r543090129 ## File path: hbase-client/src/main/java/org/apache/hadoop/hbase/client/replication/ReplicationPeerConfigUtil.java ## @@ -64,6 +65,9 @@ public static final String HBASE_REPLICATION_PEER_BASE_CONFIG = "hbase.replication.peer.base.config"; + public static final String HBASE_REPLICATION_PEER_REMOVE_BASE_CONFIG = Review comment: @bharathv do you mean setting k1="" if we want to remove k1 configuration? Just setting "" doesn't really communicate the program what we want to remove because there might be other other configs added from outside of hbase-site. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] bharathv commented on a change in pull request #2778: HBASE-25383: Ability to update and remove peer base config
bharathv commented on a change in pull request #2778:
URL: https://github.com/apache/hbase/pull/2778#discussion_r543078007
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/replication/ReplicationPeerConfigUtil.java
##
@@ -64,6 +65,9 @@
public static final String HBASE_REPLICATION_PEER_BASE_CONFIG =
"hbase.replication.peer.base.config";
+ public static final String HBASE_REPLICATION_PEER_REMOVE_BASE_CONFIG =
Review comment:
I think this approach can be better, it seems confusing that one would
add a configuration to remove another configuration? (and the community is
generally against adding such esoteric configs)
What happens if someone forgets to undo this and that has some adverse
affect after a next restart..
Why not just consider "" config as a special case and then reset the configs
in the copiedConfigBuilder?
Something like
initially someone set it k1=v1;k2=v2
then they can set it to "" and restart master
detect that "" has been set (which is different from what we already have)
and remove whatever we already have populated?
To avoid config divergence, we can disallow
hbase.replication.peer.base.config to be done via updateConfig RPC (throw some
sensible exception)..
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/replication/ReplicationPeerConfigUtil.java
##
@@ -455,36 +458,45 @@ public static ReplicationPeerConfig
appendTableCFsToReplicationPeerConfig(
/**
* Helper method to add base peer configs from Configuration to
ReplicationPeerConfig
- * if not present in latter.
*
* This merges the user supplied peer configuration
* {@link org.apache.hadoop.hbase.replication.ReplicationPeerConfig} with
peer configs
* provided as property hbase.replication.peer.base.configs in hbase
configuration.
- * Expected format for this hbase configuration is "k1=v1;k2=v2,v2_1".
Original value
- * of conf is retained if already present in ReplicationPeerConfig.
+ * Expected format for this hbase configuration is "k1=v1;k2=v2,v2_1".
*
* @param conf Configuration
* @return ReplicationPeerConfig containing updated configs.
*/
- public static ReplicationPeerConfig
addBasePeerConfigsIfNotPresent(Configuration conf,
+ public static ReplicationPeerConfig
updateReplicationBasePeerConfigs(Configuration conf,
ReplicationPeerConfig receivedPeerConfig) {
-String basePeerConfigs = conf.get(HBASE_REPLICATION_PEER_BASE_CONFIG, "");
+String removeBasePeerConfigs =
conf.get(HBASE_REPLICATION_PEER_REMOVE_BASE_CONFIG, "");
ReplicationPeerConfigBuilder copiedPeerConfigBuilder =
ReplicationPeerConfig.
newBuilder(receivedPeerConfig);
-Map receivedPeerConfigMap =
receivedPeerConfig.getConfiguration();
+// remove the peer configurations specified in the conf
Review comment:
Sorry just got to this PR. Thats right, master branch replication
coordination is different from branch-1. This was specifically taken into
consideration in the original PR that added this feature (see the commit
message for branch-1)..
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] sandeepvinayak commented on a change in pull request #2778: HBASE-25383: Ability to update and remove peer base config
sandeepvinayak commented on a change in pull request #2778:
URL: https://github.com/apache/hbase/pull/2778#discussion_r543074326
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/replication/ReplicationPeerConfigUtil.java
##
@@ -455,36 +458,45 @@ public static ReplicationPeerConfig
appendTableCFsToReplicationPeerConfig(
/**
* Helper method to add base peer configs from Configuration to
ReplicationPeerConfig
- * if not present in latter.
*
* This merges the user supplied peer configuration
* {@link org.apache.hadoop.hbase.replication.ReplicationPeerConfig} with
peer configs
* provided as property hbase.replication.peer.base.configs in hbase
configuration.
- * Expected format for this hbase configuration is "k1=v1;k2=v2,v2_1".
Original value
- * of conf is retained if already present in ReplicationPeerConfig.
+ * Expected format for this hbase configuration is "k1=v1;k2=v2,v2_1".
*
* @param conf Configuration
* @return ReplicationPeerConfig containing updated configs.
*/
- public static ReplicationPeerConfig
addBasePeerConfigsIfNotPresent(Configuration conf,
+ public static ReplicationPeerConfig
updateReplicationBasePeerConfigs(Configuration conf,
ReplicationPeerConfig receivedPeerConfig) {
-String basePeerConfigs = conf.get(HBASE_REPLICATION_PEER_BASE_CONFIG, "");
+String removeBasePeerConfigs =
conf.get(HBASE_REPLICATION_PEER_REMOVE_BASE_CONFIG, "");
ReplicationPeerConfigBuilder copiedPeerConfigBuilder =
ReplicationPeerConfig.
newBuilder(receivedPeerConfig);
-Map receivedPeerConfigMap =
receivedPeerConfig.getConfiguration();
+// remove the peer configurations specified in the conf
Review comment:
Also, looking at the `master` branch code, looks like the RS restart has
nothing to do with this update config. Only master restart is able to update
configs.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] virajjasani commented on pull request #2756: HBASE-25378 Legacy comparator in Hfile trailer will fail to load
virajjasani commented on pull request #2756: URL: https://github.com/apache/hbase/pull/2756#issuecomment-745070903 > @virajjasani Good by you sir? All good sir. I just looked at Jira conversations and it seems this is not breaking data compatibility (good to know) but this PR can be merged for better handling of both scenarios: https://issues.apache.org/jira/browse/HBASE-25378 @pankaj72981 good to go ahead. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2779: HBASE-23340 hmaster /hbase/replication/rs session expired (hbase repl…
Apache-HBase commented on pull request #2779: URL: https://github.com/apache/hbase/pull/2779#issuecomment-745056148 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 42s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ branch-2 Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 24s | branch-2 passed | | +1 :green_heart: | checkstyle | 1m 19s | branch-2 passed | | +1 :green_heart: | spotbugs | 2m 13s | branch-2 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 20s | the patch passed | | -0 :warning: | checkstyle | 1m 7s | hbase-server: The patch generated 1 new + 93 unchanged - 0 fixed = 94 total (was 93) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 12m 18s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. | | +1 :green_heart: | spotbugs | 2m 9s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 14s | The patch does not generate ASF License warnings. | | | | 35m 22s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2779/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2779 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle | | uname | Linux ef312d14ebe7 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / 9c2f1f7a2b | | checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2779/1/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | Max. process+thread count | 94 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2779/1/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=3.1.12 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2778: HBASE-25383: Ability to update and remove peer base config
Apache-HBase commented on pull request #2778: URL: https://github.com/apache/hbase/pull/2778#issuecomment-745055055 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 37s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 24s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 49s | master passed | | +1 :green_heart: | checkstyle | 1m 52s | master passed | | +1 :green_heart: | spotbugs | 3m 41s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 47s | the patch passed | | +1 :green_heart: | checkstyle | 0m 27s | The patch passed checkstyle in hbase-client | | +1 :green_heart: | checkstyle | 0m 10s | hbase-replication: The patch generated 0 new + 0 unchanged - 1 fixed = 0 total (was 1) | | +1 :green_heart: | checkstyle | 1m 5s | The patch passed checkstyle in hbase-server | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 18m 29s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | spotbugs | 3m 58s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 35s | The patch does not generate ASF License warnings. | | | | 46m 44s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2778/2/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2778 | | JIRA Issue | HBASE-25383 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle | | uname | Linux 531bbba705d0 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / d50816fe44 | | Max. process+thread count | 94 (vs. ulimit of 3) | | modules | C: hbase-client hbase-replication hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2778/2/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=3.1.12 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] sandeepvinayak commented on a change in pull request #2778: HBASE-25383: Ability to update and remove peer base config
sandeepvinayak commented on a change in pull request #2778:
URL: https://github.com/apache/hbase/pull/2778#discussion_r543049473
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/replication/ReplicationPeerConfigUtil.java
##
@@ -455,36 +458,45 @@ public static ReplicationPeerConfig
appendTableCFsToReplicationPeerConfig(
/**
* Helper method to add base peer configs from Configuration to
ReplicationPeerConfig
- * if not present in latter.
*
* This merges the user supplied peer configuration
* {@link org.apache.hadoop.hbase.replication.ReplicationPeerConfig} with
peer configs
* provided as property hbase.replication.peer.base.configs in hbase
configuration.
- * Expected format for this hbase configuration is "k1=v1;k2=v2,v2_1".
Original value
- * of conf is retained if already present in ReplicationPeerConfig.
+ * Expected format for this hbase configuration is "k1=v1;k2=v2,v2_1".
*
* @param conf Configuration
* @return ReplicationPeerConfig containing updated configs.
*/
- public static ReplicationPeerConfig
addBasePeerConfigsIfNotPresent(Configuration conf,
+ public static ReplicationPeerConfig
updateReplicationBasePeerConfigs(Configuration conf,
ReplicationPeerConfig receivedPeerConfig) {
-String basePeerConfigs = conf.get(HBASE_REPLICATION_PEER_BASE_CONFIG, "");
+String removeBasePeerConfigs =
conf.get(HBASE_REPLICATION_PEER_REMOVE_BASE_CONFIG, "");
ReplicationPeerConfigBuilder copiedPeerConfigBuilder =
ReplicationPeerConfig.
newBuilder(receivedPeerConfig);
-Map receivedPeerConfigMap =
receivedPeerConfig.getConfiguration();
+// remove the peer configurations specified in the conf
Review comment:
That's a good point. Ideally IMO, the peer configuration should also not
be added from the region server restart. We should have a better home for this
(may be master restart?).
@virajjasani Do you think if it is standard to have conf upgrade before the
HBase rolling upgrade? If not, we might need to rework on updating the peer
configs
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2775: HBASE-25334 TestRSGroupsFallback.testFallback is flaky
Apache-HBase commented on pull request #2775: URL: https://github.com/apache/hbase/pull/2775#issuecomment-745047990 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 37s | Docker mode activated. | | -0 :warning: | yetus | 0m 7s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2 Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 29s | branch-2 passed | | +1 :green_heart: | compile | 0m 27s | branch-2 passed | | +1 :green_heart: | shadedjars | 7m 16s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 26s | branch-2 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 22s | the patch passed | | +1 :green_heart: | compile | 0m 29s | the patch passed | | +1 :green_heart: | javac | 0m 29s | the patch passed | | +1 :green_heart: | shadedjars | 7m 31s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 24s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 3m 8s | hbase-rsgroup in the patch passed. | | | | 30m 30s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2775/4/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2775 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 650c0c4f7f5f 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / 9c2f1f7a2b | | Default Java | AdoptOpenJDK-11.0.6+10 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2775/4/testReport/ | | Max. process+thread count | 3716 (vs. ulimit of 12500) | | modules | C: hbase-rsgroup U: hbase-rsgroup | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2775/4/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2768: HBASE-25293 Followup jira to address the client handling issue when c…
Apache-HBase commented on pull request #2768: URL: https://github.com/apache/hbase/pull/2768#issuecomment-745047603 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 35s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 22s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 10s | master passed | | +1 :green_heart: | compile | 1m 37s | master passed | | +1 :green_heart: | shadedjars | 7m 47s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 16s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 20s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 26s | the patch passed | | +1 :green_heart: | compile | 1m 41s | the patch passed | | +1 :green_heart: | javac | 1m 41s | the patch passed | | +1 :green_heart: | shadedjars | 8m 14s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 6s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 12s | hbase-client in the patch passed. | | +1 :green_heart: | unit | 142m 7s | hbase-server in the patch passed. | | | | 176m 54s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2768/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2768 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux ede6d8eb7d2c 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / d50816fe44 | | Default Java | AdoptOpenJDK-1.8.0_232-b09 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2768/2/testReport/ | | Max. process+thread count | 4474 (vs. ulimit of 3) | | modules | C: hbase-client hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2768/2/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2775: HBASE-25334 TestRSGroupsFallback.testFallback is flaky
Apache-HBase commented on pull request #2775: URL: https://github.com/apache/hbase/pull/2775#issuecomment-745047579 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 8s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ branch-2 Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 39s | branch-2 passed | | +1 :green_heart: | checkstyle | 0m 15s | branch-2 passed | | +1 :green_heart: | spotbugs | 0m 43s | branch-2 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 19s | the patch passed | | +1 :green_heart: | checkstyle | 0m 13s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 11m 38s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. | | +1 :green_heart: | spotbugs | 0m 51s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 14s | The patch does not generate ASF License warnings. | | | | 29m 19s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2775/4/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2775 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle | | uname | Linux 8fa7ec6c7b91 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / 9c2f1f7a2b | | Max. process+thread count | 94 (vs. ulimit of 12500) | | modules | C: hbase-rsgroup U: hbase-rsgroup | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2775/4/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=3.1.12 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2775: HBASE-25334 TestRSGroupsFallback.testFallback is flaky
Apache-HBase commented on pull request #2775: URL: https://github.com/apache/hbase/pull/2775#issuecomment-745047006 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 38s | Docker mode activated. | | -0 :warning: | yetus | 0m 8s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2 Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 30s | branch-2 passed | | +1 :green_heart: | compile | 0m 25s | branch-2 passed | | +1 :green_heart: | shadedjars | 6m 25s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 23s | branch-2 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 33s | the patch passed | | +1 :green_heart: | compile | 0m 27s | the patch passed | | +1 :green_heart: | javac | 0m 27s | the patch passed | | +1 :green_heart: | shadedjars | 6m 13s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 21s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 3m 57s | hbase-rsgroup in the patch passed. | | | | 27m 13s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2775/4/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2775 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 2172bef5cb5f 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / 9c2f1f7a2b | | Default Java | AdoptOpenJDK-1.8.0_232-b09 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2775/4/testReport/ | | Max. process+thread count | 3302 (vs. ulimit of 12500) | | modules | C: hbase-rsgroup U: hbase-rsgroup | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2775/4/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2768: HBASE-25293 Followup jira to address the client handling issue when c…
Apache-HBase commented on pull request #2768: URL: https://github.com/apache/hbase/pull/2768#issuecomment-745046853 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 38s | Docker mode activated. | | -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 23s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 36s | master passed | | +1 :green_heart: | compile | 1m 43s | master passed | | +1 :green_heart: | shadedjars | 7m 17s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 11s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 17s | the patch passed | | +1 :green_heart: | compile | 1m 40s | the patch passed | | +1 :green_heart: | javac | 1m 40s | the patch passed | | +1 :green_heart: | shadedjars | 7m 4s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 6s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 9s | hbase-client in the patch passed. | | -1 :x: | unit | 140m 49s | hbase-server in the patch failed. | | | | 174m 6s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2768/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2768 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 3d24a4660606 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / d50816fe44 | | Default Java | AdoptOpenJDK-11.0.6+10 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2768/2/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2768/2/testReport/ | | Max. process+thread count | 4411 (vs. ulimit of 3) | | modules | C: hbase-client hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2768/2/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] cuibo01 opened a new pull request #2779: HBASE-23340 hmaster /hbase/replication/rs session expired (hbase repl…
cuibo01 opened a new pull request #2779: URL: https://github.com/apache/hbase/pull/2779 …ication default value is true, we don't use ) causes logcleaner can not clean oldWALs, which resulits in oldWALs too large (more than 2TB) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] cuibo01 closed pull request #2774: HBASE-23340 hmaster /hbase/replication/rs session expired (hbase repl…
cuibo01 closed pull request #2774: URL: https://github.com/apache/hbase/pull/2774 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] anoopsjohn commented on pull request #2772: HBASE-25387 TagRewriteCell's getSerializedSize() method gets the size…
anoopsjohn commented on pull request #2772: URL: https://github.com/apache/hbase/pull/2772#issuecomment-745040994 Checking all related parts. Will report back.. Pls dont merge. I saw some diff from branch2.1 where it was tags len based write(as in patch) .. may be in master its changed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2775: HBASE-25334 TestRSGroupsFallback.testFallback is flaky
Apache-HBase commented on pull request #2775: URL: https://github.com/apache/hbase/pull/2775#issuecomment-745039560 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 28s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ branch-2 Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 52s | branch-2 passed | | +1 :green_heart: | checkstyle | 0m 15s | branch-2 passed | | +1 :green_heart: | spotbugs | 0m 42s | branch-2 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 11s | the patch passed | | -0 :warning: | checkstyle | 0m 13s | hbase-rsgroup: The patch generated 3 new + 0 unchanged - 0 fixed = 3 total (was 0) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 11m 41s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. | | +1 :green_heart: | spotbugs | 0m 50s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 14s | The patch does not generate ASF License warnings. | | | | 29m 58s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2775/3/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2775 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle | | uname | Linux 9a5fcc9df13c 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / 9c2f1f7a2b | | checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2775/3/artifact/yetus-general-check/output/diff-checkstyle-hbase-rsgroup.txt | | Max. process+thread count | 94 (vs. ulimit of 12500) | | modules | C: hbase-rsgroup U: hbase-rsgroup | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2775/3/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=3.1.12 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2775: HBASE-25334 TestRSGroupsFallback.testFallback is flaky
Apache-HBase commented on pull request #2775: URL: https://github.com/apache/hbase/pull/2775#issuecomment-745039226 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 39s | Docker mode activated. | | -0 :warning: | yetus | 0m 6s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2 Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 26s | branch-2 passed | | +1 :green_heart: | compile | 0m 28s | branch-2 passed | | +1 :green_heart: | shadedjars | 6m 52s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 25s | branch-2 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 2s | the patch passed | | +1 :green_heart: | compile | 0m 28s | the patch passed | | +1 :green_heart: | javac | 0m 28s | the patch passed | | +1 :green_heart: | shadedjars | 6m 54s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 22s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 2m 58s | hbase-rsgroup in the patch passed. | | | | 28m 49s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2775/3/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2775 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux cd05837b528f 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / 9c2f1f7a2b | | Default Java | AdoptOpenJDK-11.0.6+10 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2775/3/testReport/ | | Max. process+thread count | 3646 (vs. ulimit of 12500) | | modules | C: hbase-rsgroup U: hbase-rsgroup | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2775/3/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2775: HBASE-25334 TestRSGroupsFallback.testFallback is flaky
Apache-HBase commented on pull request #2775: URL: https://github.com/apache/hbase/pull/2775#issuecomment-745038941 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 37s | Docker mode activated. | | -0 :warning: | yetus | 0m 6s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2 Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 54s | branch-2 passed | | +1 :green_heart: | compile | 0m 24s | branch-2 passed | | +1 :green_heart: | shadedjars | 6m 13s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 23s | branch-2 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 33s | the patch passed | | +1 :green_heart: | compile | 0m 24s | the patch passed | | +1 :green_heart: | javac | 0m 24s | the patch passed | | +1 :green_heart: | shadedjars | 6m 30s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 23s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 3m 58s | hbase-rsgroup in the patch passed. | | | | 27m 38s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2775/3/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2775 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 22db568cc32d 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / 9c2f1f7a2b | | Default Java | AdoptOpenJDK-1.8.0_232-b09 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2775/3/testReport/ | | Max. process+thread count | 3394 (vs. ulimit of 12500) | | modules | C: hbase-rsgroup U: hbase-rsgroup | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2775/3/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (HBASE-25334) TestRSGroupsFallback.testFallback is flaky
[ https://issues.apache.org/jira/browse/HBASE-25334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17249465#comment-17249465 ] Xiaolin Ha edited comment on HBASE-25334 at 12/15/20, 3:54 AM: --- Thanks, [~stack] . Yes, they are the same reason. They all need the SCP be completed to make the table regions be reassigned. I have updated the PR. Since AssignmentTestingUtil.crashRs has already waited the RS to be dead on Master, it need to wait no DeadServersInProgress now. was (Author: xiaolin ha): [~stack] Yes, they are the same reason. They need the SCP be completed to make the table regions be reassigned. I have updated the PR. > TestRSGroupsFallback.testFallback is flaky > -- > > Key: HBASE-25334 > URL: https://issues.apache.org/jira/browse/HBASE-25334 > Project: HBase > Issue Type: Test >Reporter: Xiaolin Ha >Assignee: Xiaolin Ha >Priority: Major > Attachments: 1607918235175-image.png, > image-2020-12-13-10-15-55-445.png > > > Like in CI test results of PR [https://github.com/apache/hbase/pull/2699] > failed UTs site is > [https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2699/3/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt] > > In this unit test, it checks if all table regions assigned after balance, and > then assert for the RS group of regions. > But balance() uses aync move, and will throttle move regions, sleeping > between all the table regions are moved to its RSGroup. > If waiting time is not longer than the region movement duration, the > assertion will be fail. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25334) TestRSGroupsFallback.testFallback is flaky
[ https://issues.apache.org/jira/browse/HBASE-25334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17249465#comment-17249465 ] Xiaolin Ha commented on HBASE-25334: [~stack] Yes, they are the same reason. They need the SCP be completed to make the table regions be reassigned. I have updated the PR. > TestRSGroupsFallback.testFallback is flaky > -- > > Key: HBASE-25334 > URL: https://issues.apache.org/jira/browse/HBASE-25334 > Project: HBase > Issue Type: Test >Reporter: Xiaolin Ha >Assignee: Xiaolin Ha >Priority: Major > Attachments: 1607918235175-image.png, > image-2020-12-13-10-15-55-445.png > > > Like in CI test results of PR [https://github.com/apache/hbase/pull/2699] > failed UTs site is > [https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2699/3/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt] > > In this unit test, it checks if all table regions assigned after balance, and > then assert for the RS group of regions. > But balance() uses aync move, and will throttle move regions, sleeping > between all the table regions are moved to its RSGroup. > If waiting time is not longer than the region movement duration, the > assertion will be fail. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] infraio commented on a change in pull request #2663: HBASE-24637 - Reseek regression related to filter SKIP hinting
infraio commented on a change in pull request #2663: URL: https://github.com/apache/hbase/pull/2663#discussion_r543015759 ## File path: hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java ## @@ -728,7 +739,7 @@ public boolean next(List outResult, ScannerContext scannerContext) throws break; case SEEK_NEXT_COL: -seekOrSkipToNextColumn(cell); +doSeekCol(cell); Review comment: I thought it is not need to change this method name? You can add the new logic to seekOrSkipToNextColumn method directly? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] infraio commented on a change in pull request #2663: HBASE-24637 - Reseek regression related to filter SKIP hinting
infraio commented on a change in pull request #2663: URL: https://github.com/apache/hbase/pull/2663#discussion_r543015308 ## File path: hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java ## @@ -130,6 +132,13 @@ */ public static final String HBASE_CELLS_SCANNED_PER_HEARTBEAT_CHECK = "hbase.cells.scanned.per.heartbeat.check"; + /** + * The number of bytes after which we decide if to do only next() or continue + * with seek(). Currently it defaults to the STORESCANNER_PREAD_MAX_BYTES config + * To disable this feature put a value < 0. + */ + public static final String HBASE_SWITCH_TO_NEXT_AFTER_BYTES_READ = Review comment: The default HBASE_SWITCH_TO_NEXT_AFTER_BYTES_READ should smaller than one block size? But STORESCANNER_PREAD_MAX_BYTES is 4 times of block size. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] infraio commented on pull request #2772: HBASE-25387 TagRewriteCell's getSerializedSize() method gets the size…
infraio commented on pull request #2772: URL: https://github.com/apache/hbase/pull/2772#issuecomment-745014773 So this should be a bug. Any thoughts @anoopsjohn This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2768: HBASE-25293 Followup jira to address the client handling issue when c…
Apache-HBase commented on pull request #2768: URL: https://github.com/apache/hbase/pull/2768#issuecomment-745012173 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 38s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 24s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 56s | master passed | | +1 :green_heart: | checkstyle | 1m 34s | master passed | | +1 :green_heart: | spotbugs | 3m 14s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 30s | the patch passed | | -0 :warning: | checkstyle | 1m 11s | hbase-server: The patch generated 10 new + 0 unchanged - 0 fixed = 10 total (was 0) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 19m 4s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | spotbugs | 4m 8s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 28s | The patch does not generate ASF License warnings. | | | | 48m 24s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2768/2/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2768 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle | | uname | Linux c53a26fdffd1 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / d50816fe44 | | checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2768/2/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | Max. process+thread count | 94 (vs. ulimit of 3) | | modules | C: hbase-client hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2768/2/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=3.1.12 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-24850) CellComparator perf improvement
[ https://issues.apache.org/jira/browse/HBASE-24850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17249445#comment-17249445 ] Anoop Sam John commented on HBASE-24850: [~ram_krish] As per the perf test analysis what is causing the perf down for CellComparator path? is it too much branching and not getting inlined? Or is it the calls to decode int/short (much more than 1.x as KV methods not in Cell)? Its not just Comparator knowing abt the contiguous key part. KV had methods exposing this stuff which is missing in Cell and its sub classes. So bringing it back (not into Cell) another area. Can we have compare(Cell) in ExtendedCell? Sorry did not see the patch.. I saw Stack raised a concern over the branching and so asking. So in that case we need to see how to handle that area as well.. The perf test analysis will help here for decision making. > CellComparator perf improvement > --- > > Key: HBASE-24850 > URL: https://issues.apache.org/jira/browse/HBASE-24850 > Project: HBase > Issue Type: Improvement > Components: Performance, scan >Affects Versions: 2.0.0 >Reporter: Anoop Sam John >Assignee: ramkrishna.s.vasudevan >Priority: Critical > Fix For: 3.0.0-alpha-1, 2.5.0 > > > We have multiple perf issues in 2.x versions compared to 1.x. Eg: > HBASE-24754, HBASE-24637. > The pattern is clear that where ever we do more and more Cell compares, there > is some degrade. In HBASE-24754, with an old KVComparator style comparator, > we see much better perf for the PutSortReducer. (Again the gain is huge > because of large number of compare ops that test is doing). This issue is to > address and optimize compares generally in CellComparatorImpl itself. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #2745: [HBASE-25246] Backup/Restore hbase cell tags.
Apache-HBase commented on pull request #2745: URL: https://github.com/apache/hbase/pull/2745#issuecomment-745010105 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 44s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 30s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 50s | master passed | | +1 :green_heart: | compile | 2m 56s | master passed | | +1 :green_heart: | shadedjars | 8m 42s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 4m 16s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 37s | the patch passed | | +1 :green_heart: | compile | 2m 36s | the patch passed | | +1 :green_heart: | javac | 2m 36s | the patch passed | | +1 :green_heart: | shadedjars | 6m 31s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 3m 18s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 334m 37s | root in the patch failed. | | | | 376m 29s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2745/9/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2745 | | JIRA Issue | HBASE-25246 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 37406d813ce9 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / d50816fe44 | | Default Java | AdoptOpenJDK-1.8.0_232-b09 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2745/9/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-root.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2745/9/testReport/ | | Max. process+thread count | 6476 (vs. ulimit of 3) | | modules | C: hbase-client hbase-server hbase-mapreduce . U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2745/9/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] nyl3532016 commented on a change in pull request #2741: HBASE-25364 Redo the getMidPoint() in HFileWriterImpl to get rid of t…
nyl3532016 commented on a change in pull request #2741:
URL: https://github.com/apache/hbase/pull/2741#discussion_r542993866
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterImpl.java
##
@@ -380,143 +378,146 @@ public static Cell getMidpoint(final CellComparator
comparator, final Cell left,
if (comparator instanceof MetaCellComparator) {
return right;
}
-int diff = comparator.compareRows(left, right);
-if (diff > 0) {
- throw new IllegalArgumentException("Left row sorts after right row;
left="
- + CellUtil.getCellKeyAsString(left) + ", right=" +
CellUtil.getCellKeyAsString(right));
-}
byte[] midRow;
boolean bufferBacked = left instanceof ByteBufferExtendedCell
&& right instanceof ByteBufferExtendedCell;
-if (diff < 0) {
- // Left row is < right row.
- if (bufferBacked) {
-midRow = getMinimumMidpointArray(((ByteBufferExtendedCell)
left).getRowByteBuffer(),
-((ByteBufferExtendedCell) left).getRowPosition(),
left.getRowLength(),
-((ByteBufferExtendedCell) right).getRowByteBuffer(),
-((ByteBufferExtendedCell) right).getRowPosition(),
right.getRowLength());
- } else {
-midRow = getMinimumMidpointArray(left.getRowArray(),
left.getRowOffset(),
-left.getRowLength(), right.getRowArray(), right.getRowOffset(),
right.getRowLength());
- }
- // If midRow is null, just return 'right'. Can't do optimization.
- if (midRow == null) {
-return right;
- }
+if (bufferBacked) {
+ midRow = getMinimumMidpointArray(((ByteBufferExtendedCell)
left).getRowByteBuffer(),
+((ByteBufferExtendedCell) left).getRowPosition(), left.getRowLength(),
+((ByteBufferExtendedCell) right).getRowByteBuffer(),
+((ByteBufferExtendedCell) right).getRowPosition(),
right.getRowLength());
+} else {
+ midRow = getMinimumMidpointArray(left.getRowArray(),
left.getRowOffset(), left.getRowLength(),
+right.getRowArray(), right.getRowOffset(), right.getRowLength());
+}
+if (midRow != null) {
return PrivateCellUtil.createFirstOnRow(midRow);
}
-// Rows are same. Compare on families.
-diff = comparator.compareFamilies(left, right);
-if (diff > 0) {
- throw new IllegalArgumentException("Left family sorts after right
family; left="
- + CellUtil.getCellKeyAsString(left) + ", right=" +
CellUtil.getCellKeyAsString(right));
-}
-if (diff < 0) {
- if (bufferBacked) {
-midRow = getMinimumMidpointArray(((ByteBufferExtendedCell)
left).getFamilyByteBuffer(),
-((ByteBufferExtendedCell) left).getFamilyPosition(),
left.getFamilyLength(),
-((ByteBufferExtendedCell) right).getFamilyByteBuffer(),
-((ByteBufferExtendedCell) right).getFamilyPosition(),
right.getFamilyLength());
- } else {
-midRow = getMinimumMidpointArray(left.getFamilyArray(),
left.getFamilyOffset(),
-left.getFamilyLength(), right.getFamilyArray(),
right.getFamilyOffset(),
-right.getFamilyLength());
- }
- // If midRow is null, just return 'right'. Can't do optimization.
- if (midRow == null) {
-return right;
- }
- // Return new Cell where we use right row and then a mid sort family.
+//Rows are same. Compare on families.
+if (bufferBacked) {
+ midRow = getMinimumMidpointArray(((ByteBufferExtendedCell)
left).getFamilyByteBuffer(),
+((ByteBufferExtendedCell) left).getFamilyPosition(),
left.getFamilyLength(),
+((ByteBufferExtendedCell) right).getFamilyByteBuffer(),
+((ByteBufferExtendedCell) right).getFamilyPosition(),
right.getFamilyLength());
+} else {
+ midRow = getMinimumMidpointArray(left.getFamilyArray(),
left.getFamilyOffset(),
+left.getFamilyLength(), right.getFamilyArray(),
right.getFamilyOffset(),
+right.getFamilyLength());
+}
+if (midRow != null) {
return PrivateCellUtil.createFirstOnRowFamily(right, midRow, 0,
midRow.length);
}
// Families are same. Compare on qualifiers.
-diff = comparator.compareQualifiers(left, right);
-if (diff > 0) {
- throw new IllegalArgumentException("Left qualifier sorts after right
qualifier; left="
- + CellUtil.getCellKeyAsString(left) + ", right=" +
CellUtil.getCellKeyAsString(right));
-}
-if (diff < 0) {
- if (bufferBacked) {
-midRow = getMinimumMidpointArray(((ByteBufferExtendedCell)
left).getQualifierByteBuffer(),
-((ByteBufferExtendedCell) left).getQualifierPosition(),
left.getQualifierLength(),
-((ByteBufferExtendedCell) right).getQualifierByteBuffer(),
-((ByteBufferExtendedCell) right).getQualifierPosition(),
right.getQualifierLength());
- } else {
-midRow = getMinimumMidpointArray(left.getQualifierArray(),
[GitHub] [hbase] Apache-HBase commented on pull request #2745: [HBASE-25246] Backup/Restore hbase cell tags.
Apache-HBase commented on pull request #2745: URL: https://github.com/apache/hbase/pull/2745#issuecomment-744848653 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 54s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 23s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 5m 15s | master passed | | +1 :green_heart: | compile | 3m 28s | master passed | | +1 :green_heart: | shadedjars | 7m 47s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 4m 41s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 31s | the patch passed | | +1 :green_heart: | compile | 3m 19s | the patch passed | | +1 :green_heart: | javac | 3m 19s | the patch passed | | +1 :green_heart: | shadedjars | 7m 52s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 5m 14s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 187m 58s | root in the patch passed. | | | | 235m 18s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2745/9/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2745 | | JIRA Issue | HBASE-25246 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux f4a7dba55820 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / d50816fe44 | | Default Java | AdoptOpenJDK-11.0.6+10 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2745/9/testReport/ | | Max. process+thread count | 7160 (vs. ulimit of 3) | | modules | C: hbase-client hbase-server hbase-mapreduce . U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2745/9/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] GeorryHuang commented on pull request #2772: HBASE-25387 TagRewriteCell's getSerializedSize() method gets the size…
GeorryHuang commented on pull request #2772:
URL: https://github.com/apache/hbase/pull/2772#issuecomment-744808656
> So only TagRewriteCell forgot the check of tags.length? Can yuu paste some
code to show this. Thanks.
>
> And the HBase QA failed. Please take a look about this.
For almost 5 subclasses of 'ExtendedCell', when they write to stream:
```
// Tags length and tags byte array
if (withTags && getTagsLength() > 0) {
// Tags length
out.write((byte)(0xff & (getTagsLength() >> 8)));
out.write((byte)(0xff & getTagsLength()));
// Tags byte array
out.write(getTagsArray(), getTagsOffset(), getTagsLength());
}
```
When TagRewriteCell write to a ByteBuffer:
```
int tagsLen = this.tags == null ? 0 : this.tags.length;
if (tagsLen > 0) {
offset = ByteBufferUtils.putAsShort(buf, offset, tagsLen);
ByteBufferUtils.copyFromArrayToBuffer(buf, offset, this.tags, 0,
tagsLen);
}
```
However, when TagRewriteCell write to a stream:
```
if (withTags && this.tags != null) {
// Write the tagsLength 2 bytes
out.write((byte) (0xff & (this.tags.length >> 8)));
out.write((byte) (0xff & this.tags.length));
out.write(this.tags);
len += KeyValue.TAGS_LENGTH_SIZE + this.tags.length;
}
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] shahrs87 commented on pull request #2766: [HBASE-25246] Backup/Restore hbase cell tags.
shahrs87 commented on pull request #2766: URL: https://github.com/apache/hbase/pull/2766#issuecomment-744729370 The failing test org.apache.hadoop.hbase.replication.TestVerifyReplicationAdjunct passed locally. ``` [INFO] [INFO] --- [INFO] T E S T S [INFO] --- [INFO] Running org.apache.hadoop.hbase.replication.TestVerifyReplicationAdjunct [INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 165.302 s - in org.apache.hadoop.hbase.replication.TestVerifyReplicationAdjunct [INFO] [INFO] Results: [INFO] [INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0 ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2745: [HBASE-25246] Backup/Restore hbase cell tags.
Apache-HBase commented on pull request #2745: URL: https://github.com/apache/hbase/pull/2745#issuecomment-744721253 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 38s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 18s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 31s | master passed | | +1 :green_heart: | checkstyle | 2m 41s | master passed | | +0 :ok: | refguide | 4m 35s | branch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect. | | +1 :green_heart: | spotbugs | 16m 53s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 44s | the patch passed | | +1 :green_heart: | checkstyle | 2m 0s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +0 :ok: | refguide | 3m 24s | patch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect. | | +1 :green_heart: | hadoopcheck | 17m 13s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | spotbugs | 13m 9s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 53s | The patch does not generate ASF License warnings. | | | | 78m 25s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2745/9/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2745 | | JIRA Issue | HBASE-25246 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle refguide | | uname | Linux e84c716ba4af 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / d50816fe44 | | refguide | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2745/9/artifact/yetus-general-check/output/branch-site/book.html | | refguide | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2745/9/artifact/yetus-general-check/output/patch-site/book.html | | Max. process+thread count | 138 (vs. ulimit of 3) | | modules | C: hbase-client hbase-server hbase-mapreduce . U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2745/9/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=3.1.12 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2766: [HBASE-25246] Backup/Restore hbase cell tags.
Apache-HBase commented on pull request #2766: URL: https://github.com/apache/hbase/pull/2766#issuecomment-744720727 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 13s | Docker mode activated. | | -0 :warning: | yetus | 0m 8s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2 Compile Tests _ | | +0 :ok: | mvndep | 0m 32s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 23s | branch-2 passed | | +1 :green_heart: | compile | 2m 9s | branch-2 passed | | +1 :green_heart: | shadedjars | 5m 57s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 3m 34s | branch-2 passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 16s | the patch passed | | +1 :green_heart: | compile | 2m 15s | the patch passed | | +1 :green_heart: | javac | 2m 15s | the patch passed | | +1 :green_heart: | shadedjars | 6m 6s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 3m 41s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 307m 2s | root in the patch failed. | | | | 342m 0s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2766/5/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2766 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 3d68f0a7d284 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / 9c2f1f7a2b | | Default Java | AdoptOpenJDK-1.8.0_232-b09 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2766/5/artifact/yetus-jdk8-hadoop2-check/output/patch-unit-root.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2766/5/testReport/ | | Max. process+thread count | 4702 (vs. ulimit of 12500) | | modules | C: hbase-client hbase-server hbase-mapreduce . U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2766/5/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2778: HBASE-25383: Ability to update and remove peer base config
Apache-HBase commented on pull request #2778: URL: https://github.com/apache/hbase/pull/2778#issuecomment-744710565 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 12s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 22s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 45s | master passed | | +1 :green_heart: | compile | 1m 41s | master passed | | +1 :green_heart: | shadedjars | 6m 34s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 18s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 35s | the patch passed | | +1 :green_heart: | compile | 1m 42s | the patch passed | | +1 :green_heart: | javac | 1m 42s | the patch passed | | +1 :green_heart: | shadedjars | 6m 49s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 16s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 3s | hbase-client in the patch passed. | | +1 :green_heart: | unit | 0m 37s | hbase-replication in the patch passed. | | +1 :green_heart: | unit | 156m 11s | hbase-server in the patch passed. | | | | 188m 41s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2778/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2778 | | JIRA Issue | HBASE-25383 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 9a147988476d 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / d50816fe44 | | Default Java | AdoptOpenJDK-1.8.0_232-b09 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2778/1/testReport/ | | Max. process+thread count | 4300 (vs. ulimit of 3) | | modules | C: hbase-client hbase-replication hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2778/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] sandeepvinayak commented on a change in pull request #2778: HBASE-25383: Ability to update and remove peer base config
sandeepvinayak commented on a change in pull request #2778:
URL: https://github.com/apache/hbase/pull/2778#discussion_r542772730
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/replication/ReplicationPeerConfigUtil.java
##
@@ -455,36 +458,45 @@ public static ReplicationPeerConfig
appendTableCFsToReplicationPeerConfig(
/**
* Helper method to add base peer configs from Configuration to
ReplicationPeerConfig
- * if not present in latter.
*
* This merges the user supplied peer configuration
* {@link org.apache.hadoop.hbase.replication.ReplicationPeerConfig} with
peer configs
* provided as property hbase.replication.peer.base.configs in hbase
configuration.
- * Expected format for this hbase configuration is "k1=v1;k2=v2,v2_1".
Original value
- * of conf is retained if already present in ReplicationPeerConfig.
+ * Expected format for this hbase configuration is "k1=v1;k2=v2,v2_1".
*
* @param conf Configuration
* @return ReplicationPeerConfig containing updated configs.
*/
- public static ReplicationPeerConfig
addBasePeerConfigsIfNotPresent(Configuration conf,
+ public static ReplicationPeerConfig
updateReplicationBasePeerConfigs(Configuration conf,
ReplicationPeerConfig receivedPeerConfig) {
-String basePeerConfigs = conf.get(HBASE_REPLICATION_PEER_BASE_CONFIG, "");
+String removeBasePeerConfigs =
conf.get(HBASE_REPLICATION_PEER_REMOVE_BASE_CONFIG, "");
ReplicationPeerConfigBuilder copiedPeerConfigBuilder =
ReplicationPeerConfig.
newBuilder(receivedPeerConfig);
-Map receivedPeerConfigMap =
receivedPeerConfig.getConfiguration();
+// remove the peer configurations specified in the conf
+if (removeBasePeerConfigs.length() != 0) {
+ List removeBasePeerConfigList = Splitter.on(';').trimResults()
+.omitEmptyStrings().splitToList(removeBasePeerConfigs);
+ for (String peerConfigToRemove : removeBasePeerConfigList) {
+copiedPeerConfigBuilder.removeConfiguration(peerConfigToRemove);
+ }
+}
+
+Map receivedPeerConfigMap =
receivedPeerConfig.getConfiguration();
+String basePeerConfigs = conf.get(HBASE_REPLICATION_PEER_BASE_CONFIG, "");
Review comment:
@gjacoby126 That is correct. I will add a log here.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2778: HBASE-25383: Ability to update and remove peer base config
Apache-HBase commented on pull request #2778: URL: https://github.com/apache/hbase/pull/2778#issuecomment-744702928 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 31s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 21s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 22s | master passed | | +1 :green_heart: | compile | 1m 56s | master passed | | +1 :green_heart: | shadedjars | 6m 51s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 24s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 7s | the patch passed | | +1 :green_heart: | compile | 1m 58s | the patch passed | | +1 :green_heart: | javac | 1m 58s | the patch passed | | +1 :green_heart: | shadedjars | 6m 52s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 22s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 6s | hbase-client in the patch passed. | | +1 :green_heart: | unit | 0m 26s | hbase-replication in the patch passed. | | +1 :green_heart: | unit | 134m 7s | hbase-server in the patch passed. | | | | 167m 50s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2778/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2778 | | JIRA Issue | HBASE-25383 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux c5b125e7694e 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / d50816fe44 | | Default Java | AdoptOpenJDK-11.0.6+10 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2778/1/testReport/ | | Max. process+thread count | 4362 (vs. ulimit of 3) | | modules | C: hbase-client hbase-replication hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2778/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] gjacoby126 commented on a change in pull request #2778: HBASE-25383: Ability to update and remove peer base config
gjacoby126 commented on a change in pull request #2778:
URL: https://github.com/apache/hbase/pull/2778#discussion_r542727432
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/replication/ReplicationPeerConfigUtil.java
##
@@ -455,36 +458,45 @@ public static ReplicationPeerConfig
appendTableCFsToReplicationPeerConfig(
/**
* Helper method to add base peer configs from Configuration to
ReplicationPeerConfig
- * if not present in latter.
*
* This merges the user supplied peer configuration
* {@link org.apache.hadoop.hbase.replication.ReplicationPeerConfig} with
peer configs
* provided as property hbase.replication.peer.base.configs in hbase
configuration.
- * Expected format for this hbase configuration is "k1=v1;k2=v2,v2_1".
Original value
- * of conf is retained if already present in ReplicationPeerConfig.
+ * Expected format for this hbase configuration is "k1=v1;k2=v2,v2_1".
*
* @param conf Configuration
* @return ReplicationPeerConfig containing updated configs.
*/
- public static ReplicationPeerConfig
addBasePeerConfigsIfNotPresent(Configuration conf,
+ public static ReplicationPeerConfig
updateReplicationBasePeerConfigs(Configuration conf,
ReplicationPeerConfig receivedPeerConfig) {
-String basePeerConfigs = conf.get(HBASE_REPLICATION_PEER_BASE_CONFIG, "");
+String removeBasePeerConfigs =
conf.get(HBASE_REPLICATION_PEER_REMOVE_BASE_CONFIG, "");
ReplicationPeerConfigBuilder copiedPeerConfigBuilder =
ReplicationPeerConfig.
newBuilder(receivedPeerConfig);
-Map receivedPeerConfigMap =
receivedPeerConfig.getConfiguration();
+// remove the peer configurations specified in the conf
+if (removeBasePeerConfigs.length() != 0) {
+ List removeBasePeerConfigList = Splitter.on(';').trimResults()
+.omitEmptyStrings().splitToList(removeBasePeerConfigs);
+ for (String peerConfigToRemove : removeBasePeerConfigList) {
+copiedPeerConfigBuilder.removeConfiguration(peerConfigToRemove);
+ }
+}
+
+Map receivedPeerConfigMap =
receivedPeerConfig.getConfiguration();
+String basePeerConfigs = conf.get(HBASE_REPLICATION_PEER_BASE_CONFIG, "");
Review comment:
What happens if I try to add a peer config setting and remove the same
peer config setting in the same Configuration object? Looks like the add just
silently wins? I'd think that should produce an exception (or at the very least
a log).
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/replication/ReplicationPeerConfigUtil.java
##
@@ -455,36 +458,45 @@ public static ReplicationPeerConfig
appendTableCFsToReplicationPeerConfig(
/**
* Helper method to add base peer configs from Configuration to
ReplicationPeerConfig
- * if not present in latter.
*
* This merges the user supplied peer configuration
* {@link org.apache.hadoop.hbase.replication.ReplicationPeerConfig} with
peer configs
* provided as property hbase.replication.peer.base.configs in hbase
configuration.
- * Expected format for this hbase configuration is "k1=v1;k2=v2,v2_1".
Original value
- * of conf is retained if already present in ReplicationPeerConfig.
+ * Expected format for this hbase configuration is "k1=v1;k2=v2,v2_1".
*
* @param conf Configuration
* @return ReplicationPeerConfig containing updated configs.
*/
- public static ReplicationPeerConfig
addBasePeerConfigsIfNotPresent(Configuration conf,
+ public static ReplicationPeerConfig
updateReplicationBasePeerConfigs(Configuration conf,
ReplicationPeerConfig receivedPeerConfig) {
-String basePeerConfigs = conf.get(HBASE_REPLICATION_PEER_BASE_CONFIG, "");
+String removeBasePeerConfigs =
conf.get(HBASE_REPLICATION_PEER_REMOVE_BASE_CONFIG, "");
ReplicationPeerConfigBuilder copiedPeerConfigBuilder =
ReplicationPeerConfig.
newBuilder(receivedPeerConfig);
-Map receivedPeerConfigMap =
receivedPeerConfig.getConfiguration();
+// remove the peer configurations specified in the conf
Review comment:
Because configuration settings are often from hbase-site.xml, and
hbase-site.xml can be different on different region servers (e.g when upgrading
a cluster), it would seem like you could get into some unwelcome race
conditions with this approach.
Let's say that you're doing a rolling restart from Version 1 to Version 2.
Version 1 has a config C == true, Version 2 tries to remove C.
1. Server 1 is upgraded to Version 2 and restarts, and removes C from the
peer config
2. Server 2 is still on Version 1 and is restarted for an unrelated reason
(maybe it crashes and is restarted). It sees that C is not in the peer config,
and recreates it.
Assuming that you upgraded hbase-site.xml everywhere and restarted, the
cluster would _eventually_ converge on the right answer, but there's a lot of
opportunity for the
[GitHub] [hbase] Apache-HBase commented on pull request #2776: Using ContiguousCellFormat as a marker alone
Apache-HBase commented on pull request #2776: URL: https://github.com/apache/hbase/pull/2776#issuecomment-744677054 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 32s | Docker mode activated. | | -0 :warning: | yetus | 0m 8s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2.3 Compile Tests _ | | +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 30s | branch-2.3 passed | | +1 :green_heart: | compile | 1m 18s | branch-2.3 passed | | +1 :green_heart: | shadedjars | 5m 2s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 59s | branch-2.3 passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 17s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 13s | the patch passed | | +1 :green_heart: | compile | 1m 18s | the patch passed | | +1 :green_heart: | javac | 1m 18s | the patch passed | | +1 :green_heart: | shadedjars | 4m 58s | patch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 21s | hbase-common generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 29s | hbase-common in the patch passed. | | +1 :green_heart: | unit | 141m 2s | hbase-server in the patch passed. | | | | 167m 17s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2776/4/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2776 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 569df1b004f3 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2.3 / 94ce48b173 | | Default Java | AdoptOpenJDK-1.8.0_232-b09 | | javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2776/4/artifact/yetus-jdk8-hadoop2-check/output/diff-javadoc-javadoc-hbase-common.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2776/4/testReport/ | | Max. process+thread count | 3346 (vs. ulimit of 12500) | | modules | C: hbase-common hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2776/4/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2776: Using ContiguousCellFormat as a marker alone
Apache-HBase commented on pull request #2776: URL: https://github.com/apache/hbase/pull/2776#issuecomment-744670518 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 36s | Docker mode activated. | | -0 :warning: | yetus | 0m 8s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2.3 Compile Tests _ | | +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 10s | branch-2.3 passed | | +1 :green_heart: | compile | 1m 30s | branch-2.3 passed | | +1 :green_heart: | shadedjars | 5m 51s | branch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 19s | hbase-common in branch-2.3 failed. | | -0 :warning: | javadoc | 0m 40s | hbase-server in branch-2.3 failed. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 20s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 58s | the patch passed | | +1 :green_heart: | compile | 1m 37s | the patch passed | | +1 :green_heart: | javac | 1m 37s | the patch passed | | +1 :green_heart: | shadedjars | 5m 58s | patch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 18s | hbase-common in the patch failed. | | -0 :warning: | javadoc | 0m 40s | hbase-server in the patch failed. | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 49s | hbase-common in the patch passed. | | -1 :x: | unit | 124m 3s | hbase-server in the patch failed. | | | | 154m 40s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2776/4/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2776 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux b94154583c5f 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2.3 / 94ce48b173 | | Default Java | AdoptOpenJDK-11.0.6+10 | | javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2776/4/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-common.txt | | javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2776/4/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt | | javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2776/4/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-common.txt | | javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2776/4/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2776/4/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2776/4/testReport/ | | Max. process+thread count | 4274 (vs. ulimit of 12500) | | modules | C: hbase-common hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2776/4/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2766: [HBASE-25246] Backup/Restore hbase cell tags.
Apache-HBase commented on pull request #2766: URL: https://github.com/apache/hbase/pull/2766#issuecomment-744657689 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 33s | Docker mode activated. | | -0 :warning: | yetus | 0m 5s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2 Compile Tests _ | | +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 15s | branch-2 passed | | +1 :green_heart: | compile | 2m 56s | branch-2 passed | | +1 :green_heart: | shadedjars | 7m 46s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 5m 10s | branch-2 passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 42s | the patch passed | | +1 :green_heart: | compile | 3m 25s | the patch passed | | +1 :green_heart: | javac | 3m 25s | the patch passed | | +1 :green_heart: | shadedjars | 8m 32s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 4m 56s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 174m 33s | root in the patch passed. | | | | 220m 49s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2766/5/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2766 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 745ecbc25ec2 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / 9c2f1f7a2b | | Default Java | AdoptOpenJDK-11.0.6+10 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2766/5/testReport/ | | Max. process+thread count | 6056 (vs. ulimit of 12500) | | modules | C: hbase-client hbase-server hbase-mapreduce . U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2766/5/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2767: [HBASE-25246] Backup/Restore hbase cell tags.
Apache-HBase commented on pull request #2767: URL: https://github.com/apache/hbase/pull/2767#issuecomment-744653150 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 2m 15s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 3 new or modified test files. | ||| _ branch-1 Compile Tests _ | | +0 :ok: | mvndep | 2m 24s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 8m 37s | branch-1 passed | | +1 :green_heart: | compile | 2m 1s | branch-1 passed with JDK Azul Systems, Inc.-1.8.0_262-b19 | | +1 :green_heart: | compile | 2m 11s | branch-1 passed with JDK Azul Systems, Inc.-1.7.0_272-b10 | | +1 :green_heart: | checkstyle | 9m 44s | branch-1 passed | | +0 :ok: | refguide | 4m 33s | branch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect. | | +1 :green_heart: | shadedjars | 3m 30s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 4m 0s | branch-1 passed with JDK Azul Systems, Inc.-1.8.0_262-b19 | | +1 :green_heart: | javadoc | 6m 29s | branch-1 passed with JDK Azul Systems, Inc.-1.7.0_272-b10 | | +0 :ok: | spotbugs | 3m 11s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 18m 58s | branch-1 passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 19s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 18s | the patch passed | | +1 :green_heart: | compile | 2m 2s | the patch passed with JDK Azul Systems, Inc.-1.8.0_262-b19 | | +1 :green_heart: | javac | 2m 2s | the patch passed | | +1 :green_heart: | compile | 2m 0s | the patch passed with JDK Azul Systems, Inc.-1.7.0_272-b10 | | +1 :green_heart: | javac | 2m 0s | the patch passed | | -1 :x: | checkstyle | 6m 29s | root: The patch generated 2 new + 322 unchanged - 2 fixed = 324 total (was 324) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +0 :ok: | refguide | 3m 34s | patch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect. | | +1 :green_heart: | shadedjars | 3m 28s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | hadoopcheck | 5m 29s | Patch does not cause any errors with Hadoop 2.8.5 2.9.2. | | +1 :green_heart: | javadoc | 3m 51s | the patch passed with JDK Azul Systems, Inc.-1.8.0_262-b19 | | +1 :green_heart: | javadoc | 6m 9s | the patch passed with JDK Azul Systems, Inc.-1.7.0_272-b10 | | +1 :green_heart: | findbugs | 19m 52s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 155m 18s | root in the patch failed. | | +1 :green_heart: | asflicense | 1m 40s | The patch does not generate ASF License warnings. | | | | 285m 19s | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.hbase.client.TestReplicasClient | | | hadoop.hbase.regionserver.TestSplitTransactionOnCluster | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2767/6/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2767 | | Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile refguide | | uname | Linux 4a5085c80035 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-home/workspace/Base-PreCommit-GitHub-PR_PR-2767/out/precommit/personality/provided.sh | | git revision | branch-1 / 8e45165 | | Default Java | Azul Systems, Inc.-1.7.0_272-b10 | | Multi-JDK versions | /usr/lib/jvm/zulu-8-amd64:Azul Systems, Inc.-1.8.0_262-b19 /usr/lib/jvm/zulu-7-amd64:Azul Systems, Inc.-1.7.0_272-b10 | | refguide | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2767/6/artifact/out/branch-site/book.html | | checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2767/6/artifact/out/diff-checkstyle-root.txt |
[GitHub] [hbase] saintstack commented on pull request #2759: HBASE-25356 HBaseAdmin#getRegion() needs to filter out non-regionName…
saintstack commented on pull request #2759: URL: https://github.com/apache/hbase/pull/2759#issuecomment-744644569 You going to put up new changes @huaxiangsun ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] saintstack commented on pull request #2756: HBASE-25378 Legacy comparator in Hfile trailer will fail to load
saintstack commented on pull request #2756: URL: https://github.com/apache/hbase/pull/2756#issuecomment-744643074 @virajjasani Good by you sir? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] saintstack commented on pull request #2753: HBASE-25368 Filter out more invalid encoded name in isEncodedRegionNa…
saintstack commented on pull request #2753: URL: https://github.com/apache/hbase/pull/2753#issuecomment-744640880 Ok. Still +1 on this @huaxiangsun This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] saintstack commented on pull request #2747: HBASE-24850 CellComparator perf improvement
saintstack commented on pull request #2747: URL: https://github.com/apache/hbase/pull/2747#issuecomment-744638945 @ramkrish86 Should we close this in favor of #2776 ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-24754) Bulk load performance is degraded in HBase 2
[ https://issues.apache.org/jira/browse/HBASE-24754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-24754: Fix Version/s: 2.5.0 3.0.0-alpha-1 > Bulk load performance is degraded in HBase 2 > - > > Key: HBASE-24754 > URL: https://issues.apache.org/jira/browse/HBASE-24754 > Project: HBase > Issue Type: Bug > Components: Performance >Affects Versions: 2.2.3 >Reporter: Ajeet Rai >Assignee: ramkrishna.s.vasudevan >Priority: Major > Fix For: 3.0.0-alpha-1, 2.5.0 > > Attachments: Branc2_withComparator_atKeyValue.patch, > Branch1.3_putSortReducer_sampleCode.patch, > Branch2_putSortReducer_sampleCode.patch, flamegraph_branch-1_new.svg, > flamegraph_branch-2.svg, flamegraph_branch-2_afterpatch.svg > > > in our Test,It is observed that Bulk load performance is degraded in HBase 2 . > Test Input: > 1: Table with 500 region(300 column family) > 2: data =2 TB > Data Sample > 186000120150205100068110,1860001,20150205,5,404,735412,2938,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,1 > 3: Cluster: 7 node(2 master+5 Region Server) > 4: No of Container Launched are same in both case > HBase 2 took 10% more time then HBase 1.3 where test input is same for both > cluster > > |Feature|HBase 2.2.3 > Time(Sec)|HBase 1.3.1 > Time(Sec)|Diff%|Snappy lib: > | > |BulkLoad|21837|19686.16|-10.93|Snappy lib: > HBase 2.2.3: 1.4 > HBase 1.3.1: 1.4| -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] saintstack commented on a change in pull request #2742: HBASE-25365 The log in move_servers_rsgroup is incorrect
saintstack commented on a change in pull request #2742:
URL: https://github.com/apache/hbase/pull/2742#discussion_r542631191
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupInfoManagerImpl.java
##
@@ -1015,17 +1016,17 @@ private void moveServerRegionsFromGroup(Set
movedServers, Set
assignmentFutures.add(Pair.newPair(region, future));
} catch (IOException ioe) {
failedRegions.add(region.getRegionNameAsString());
- LOG.debug("Move region {} from group failed, will retry, current
retry time is {}",
-region.getShortNameToLog(), retry, ioe);
+ LOG.debug("Move region {} from server {} failed, will retry,
current retry time is {}",
Review comment:
nit: 'server' in the log message is redundant; we are always moving
between servers?
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupInfoManagerImpl.java
##
@@ -1015,17 +1016,17 @@ private void moveServerRegionsFromGroup(Set
movedServers, Set
assignmentFutures.add(Pair.newPair(region, future));
} catch (IOException ioe) {
failedRegions.add(region.getRegionNameAsString());
- LOG.debug("Move region {} from group failed, will retry, current
retry time is {}",
-region.getShortNameToLog(), retry, ioe);
+ LOG.debug("Move region {} from server {} failed, will retry,
current retry time is {}",
+region.getShortNameToLog(), owner.getHostname(), retry, ioe);
toThrow = ioe;
}
}
}
}
- waitForRegionMovement(assignmentFutures, failedRegions,
targetGrp.getName(), retry);
+ waitForRegionMovement(assignmentFutures, failedRegions, sourceGroupName,
retry);
if (failedRegions.isEmpty()) {
-LOG.info("All regions from server(s) {} moved to target group {}.",
movedServerNames,
- targetGrp.getName());
+LOG.info("All regions from server(s) {} moved to source group {}.",
movedServerNames,
Review comment:
Why not say, 'All regions from {} moved from {} to {}, movedServerNames,
sourceGroupName, targetGroupName?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] saintstack commented on pull request #2741: HBASE-25364 Redo the getMidPoint() in HFileWriterImpl to get rid of t…
saintstack commented on pull request #2741: URL: https://github.com/apache/hbase/pull/2741#issuecomment-744633516 @nyl3532016 You good w/ this change? If so, would like to add you as a sign-off. Thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] saintstack closed pull request #2733: HBASE-25353 [Flakey Tests] branch-2 TestShutdownBackupMaster
saintstack closed pull request #2733: URL: https://github.com/apache/hbase/pull/2733 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] saintstack commented on pull request #2733: HBASE-25353 [Flakey Tests] branch-2 TestShutdownBackupMaster
saintstack commented on pull request #2733: URL: https://github.com/apache/hbase/pull/2733#issuecomment-744631798 Manually merged. Closing This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] saintstack commented on pull request #2734: HBASE-25354 Update create-release scripts; rotted
saintstack commented on pull request #2734: URL: https://github.com/apache/hbase/pull/2734#issuecomment-744631562 Manually merged. Closing This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] saintstack closed pull request #2734: HBASE-25354 Update create-release scripts; rotted
saintstack closed pull request #2734: URL: https://github.com/apache/hbase/pull/2734 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] saintstack commented on pull request #2776: Using ContiguousCellFormat as a marker alone
saintstack commented on pull request #2776: URL: https://github.com/apache/hbase/pull/2776#issuecomment-744631056 Be careful @ramkrish86 .. you are missing the JIRA from this PR summary. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2778: HBASE-25383: Ability to update and remove peer base config
Apache-HBase commented on pull request #2778: URL: https://github.com/apache/hbase/pull/2778#issuecomment-744630875 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 33s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 22s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 48s | master passed | | +1 :green_heart: | checkstyle | 1m 42s | master passed | | +1 :green_heart: | spotbugs | 3m 32s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 30s | the patch passed | | +1 :green_heart: | checkstyle | 0m 27s | The patch passed checkstyle in hbase-client | | +1 :green_heart: | checkstyle | 0m 13s | hbase-replication: The patch generated 0 new + 0 unchanged - 1 fixed = 0 total (was 1) | | +1 :green_heart: | checkstyle | 1m 4s | The patch passed checkstyle in hbase-server | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 17m 32s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | spotbugs | 4m 3s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 37s | The patch does not generate ASF License warnings. | | | | 45m 30s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2778/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2778 | | JIRA Issue | HBASE-25383 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle | | uname | Linux 4daf166a2948 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / d50816fe44 | | Max. process+thread count | 94 (vs. ulimit of 3) | | modules | C: hbase-client hbase-replication hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2778/1/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=3.1.12 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2777: HBASE-25318 Configure where IntegrationTestImportTsv generates HFiles
Apache-HBase commented on pull request #2777: URL: https://github.com/apache/hbase/pull/2777#issuecomment-744615469 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 27s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 17s | master passed | | +1 :green_heart: | compile | 0m 28s | master passed | | +1 :green_heart: | shadedjars | 6m 43s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 17s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 1s | the patch passed | | +1 :green_heart: | compile | 0m 28s | the patch passed | | +1 :green_heart: | javac | 0m 28s | the patch passed | | +1 :green_heart: | shadedjars | 6m 46s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 15s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 51s | hbase-it in the patch passed. | | | | 26m 41s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2777/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2777 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux bc1a4626c74f 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / d50816fe44 | | Default Java | AdoptOpenJDK-11.0.6+10 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2777/1/testReport/ | | Max. process+thread count | 713 (vs. ulimit of 3) | | modules | C: hbase-it U: hbase-it | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2777/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] saintstack commented on a change in pull request #2776: Using ContiguousCellFormat as a marker alone
saintstack commented on a change in pull request #2776:
URL: https://github.com/apache/hbase/pull/2776#discussion_r542594504
##
File path: hbase-common/src/main/java/org/apache/hadoop/hbase/KeyValue.java
##
@@ -1396,7 +1403,14 @@ public int getQualifierOffset() {
* @return Qualifier offset
*/
private int getQualifierOffset(int foffset) {
-return foffset + getFamilyLength(foffset);
+return getQualifierOffset(foffset, getFamilyLength());
+ }
+
+ /**
+ * @return Qualifier offset
+ */
+ int getQualifierOffset(int foffset, int flength) {
Review comment:
So, these methods are not part of the Cell Interface. They are package
private. The CellComparator is in same package. It makes use of these package
private methods. I think that is fine. I don't think we need the marker
interface in this case given all current Cell implementations are 'contiguous'?
I say this because there are already too many Cell Interfaces. Lets worry about
adding a Marker interface later, when we are adding a new Cell type?
##
File path:
hbase-common/src/main/java/org/apache/hadoop/hbase/ContiguousCellFormat.java
##
@@ -0,0 +1,28 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase;
+
+import org.apache.yetus.audience.InterfaceAudience;
+
+/**
+ * A marker interface that indicates that the cells follow the KV
serialization pattern.
Review comment:
Say a bit more what you mean here... if you make a new patch (nit)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2777: HBASE-25318 Configure where IntegrationTestImportTsv generates HFiles
Apache-HBase commented on pull request #2777: URL: https://github.com/apache/hbase/pull/2777#issuecomment-744611861 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 34s | Docker mode activated. | | -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 54s | master passed | | +1 :green_heart: | compile | 0m 25s | master passed | | +1 :green_heart: | shadedjars | 7m 28s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 19s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 53s | the patch passed | | +1 :green_heart: | compile | 0m 31s | the patch passed | | +1 :green_heart: | javac | 0m 31s | the patch passed | | +1 :green_heart: | shadedjars | 8m 57s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 15s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 7s | hbase-it in the patch passed. | | | | 29m 41s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2777/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2777 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 9406e8c89a4c 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / d50816fe44 | | Default Java | AdoptOpenJDK-1.8.0_232-b09 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2777/1/testReport/ | | Max. process+thread count | 701 (vs. ulimit of 3) | | modules | C: hbase-it U: hbase-it | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2777/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2777: HBASE-25318 Configure where IntegrationTestImportTsv generates HFiles
Apache-HBase commented on pull request #2777: URL: https://github.com/apache/hbase/pull/2777#issuecomment-744611660 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 30s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 1s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 56s | master passed | | +1 :green_heart: | checkstyle | 0m 17s | master passed | | +1 :green_heart: | spotbugs | 0m 0s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 32s | the patch passed | | +1 :green_heart: | checkstyle | 0m 15s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 17m 21s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | spotbugs | 0m 0s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 14s | The patch does not generate ASF License warnings. | | | | 33m 13s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2777/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2777 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle | | uname | Linux 98425d942fa3 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / d50816fe44 | | Max. process+thread count | 78 (vs. ulimit of 3) | | modules | C: hbase-it U: hbase-it | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2777/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-24791) Improve HFileOutputFormat2 to avoid always call getTableRelativePath method
[ https://issues.apache.org/jira/browse/HBASE-24791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17249172#comment-17249172 ] Huaxiang Sun commented on HBASE-24791: -- Sorry, was trying to watch and clicked "Assignee" by accident. > Improve HFileOutputFormat2 to avoid always call getTableRelativePath method > --- > > Key: HBASE-24791 > URL: https://issues.apache.org/jira/browse/HBASE-24791 > Project: HBase > Issue Type: Improvement > Components: mapreduce >Affects Versions: 3.0.0-alpha-1 >Reporter: Yechao Chen >Assignee: Yechao Chen >Priority: Critical > Labels: HFileOutputFormat, bulkload > Fix For: 3.0.0-alpha-1 > > > Bulkload use HFileOutputFormat2 to write HFile > In the HFileOutputFormat2.RecordWriter > in the write method always called the getTableRelativePath method each time > This is unnecessary -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HBASE-24791) Improve HFileOutputFormat2 to avoid always call getTableRelativePath method
[ https://issues.apache.org/jira/browse/HBASE-24791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huaxiang Sun reassigned HBASE-24791: Assignee: Yechao Chen (was: Huaxiang Sun) > Improve HFileOutputFormat2 to avoid always call getTableRelativePath method > --- > > Key: HBASE-24791 > URL: https://issues.apache.org/jira/browse/HBASE-24791 > Project: HBase > Issue Type: Improvement > Components: mapreduce >Affects Versions: 3.0.0-alpha-1 >Reporter: Yechao Chen >Assignee: Yechao Chen >Priority: Critical > Labels: HFileOutputFormat, bulkload > Fix For: 3.0.0-alpha-1 > > > Bulkload use HFileOutputFormat2 to write HFile > In the HFileOutputFormat2.RecordWriter > in the write method always called the getTableRelativePath method each time > This is unnecessary -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HBASE-24791) Improve HFileOutputFormat2 to avoid always call getTableRelativePath method
[ https://issues.apache.org/jira/browse/HBASE-24791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huaxiang Sun reassigned HBASE-24791: Assignee: Huaxiang Sun (was: Yechao Chen) > Improve HFileOutputFormat2 to avoid always call getTableRelativePath method > --- > > Key: HBASE-24791 > URL: https://issues.apache.org/jira/browse/HBASE-24791 > Project: HBase > Issue Type: Improvement > Components: mapreduce >Affects Versions: 3.0.0-alpha-1 >Reporter: Yechao Chen >Assignee: Huaxiang Sun >Priority: Critical > Labels: HFileOutputFormat, bulkload > Fix For: 3.0.0-alpha-1 > > > Bulkload use HFileOutputFormat2 to write HFile > In the HFileOutputFormat2.RecordWriter > in the write method always called the getTableRelativePath method each time > This is unnecessary -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #2776: Using ContiguousCellFormat as a marker alone
Apache-HBase commented on pull request #2776: URL: https://github.com/apache/hbase/pull/2776#issuecomment-744608307 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 13s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ branch-2.3 Compile Tests _ | | +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 32s | branch-2.3 passed | | +1 :green_heart: | checkstyle | 1m 30s | branch-2.3 passed | | +1 :green_heart: | spotbugs | 2m 45s | branch-2.3 passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 17s | the patch passed | | -0 :warning: | checkstyle | 0m 26s | hbase-common: The patch generated 6 new + 118 unchanged - 0 fixed = 124 total (was 118) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 17m 8s | Patch does not cause any errors with Hadoop 2.10.0 or 3.1.2 3.2.1. | | +1 :green_heart: | spotbugs | 3m 2s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 25s | The patch does not generate ASF License warnings. | | | | 42m 10s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2776/4/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2776 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle | | uname | Linux feaac7433d49 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2.3 / 94ce48b173 | | checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2776/4/artifact/yetus-general-check/output/diff-checkstyle-hbase-common.txt | | Max. process+thread count | 94 (vs. ulimit of 12500) | | modules | C: hbase-common hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2776/4/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=3.1.12 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-25383) HBase doesn't update and remove the peer config from hbase.replication.source.custom.walentryfilters if the config is already set on the peer.
[ https://issues.apache.org/jira/browse/HBASE-25383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandeep Pal updated HBASE-25383: Description: Currently, we cannot update the peer-based config even if we change the value of the config. Secondly, once the configuration is added, there is no smooth way to remove the peer config. Summary: HBase doesn't update and remove the peer config from hbase.replication.source.custom.walentryfilters if the config is already set on the peer. (was: HBase doesn't update the peer config from hbase.replication.source.custom.walentryfilters if the config is already set on the peer. ) > HBase doesn't update and remove the peer config from > hbase.replication.source.custom.walentryfilters if the config is already set > on the peer. > --- > > Key: HBASE-25383 > URL: https://issues.apache.org/jira/browse/HBASE-25383 > Project: HBase > Issue Type: Bug >Reporter: Sandeep Pal >Assignee: Sandeep Pal >Priority: Major > > Currently, we cannot update the peer-based config even if we change the value > of the config. > Secondly, once the configuration is added, there is no smooth way to remove > the peer config. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] sandeepvinayak opened a new pull request #2778: HBASE-25383: Ability to update and remove peer base config
sandeepvinayak opened a new pull request #2778: URL: https://github.com/apache/hbase/pull/2778 Currently, we cannot update the peer based config even if we change the value of config. Secondly, once the configuration is added, there is not smooth way to remove the peer config. This patch adds an ability to update and remove the peer config if required. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (HBASE-25334) TestRSGroupsFallback.testFallback is flaky
[ https://issues.apache.org/jira/browse/HBASE-25334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17249154#comment-17249154 ] Michael Stack edited comment on HBASE-25334 at 12/14/20, 5:39 PM: -- [~Xiaolin Ha] is the failure like that in HBASE-25389 ? In HBASE-25389, we are trying to verify that the meta Region comes up in a new location after we kill its current server. The test was failing because we were checking for the new location before the old location server had crashed down. was (Author: stack): [~Xiaolin Ha] is the failure like that in HBASE-25389 ? > TestRSGroupsFallback.testFallback is flaky > -- > > Key: HBASE-25334 > URL: https://issues.apache.org/jira/browse/HBASE-25334 > Project: HBase > Issue Type: Test >Reporter: Xiaolin Ha >Assignee: Xiaolin Ha >Priority: Major > Attachments: 1607918235175-image.png, > image-2020-12-13-10-15-55-445.png > > > Like in CI test results of PR [https://github.com/apache/hbase/pull/2699] > failed UTs site is > [https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2699/3/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt] > > In this unit test, it checks if all table regions assigned after balance, and > then assert for the RS group of regions. > But balance() uses aync move, and will throttle move regions, sleeping > between all the table regions are moved to its RSGroup. > If waiting time is not longer than the region movement duration, the > assertion will be fail. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-25399) Write HBCK tools to rebuild the tracking
Tak-Lon (Stephen) Wu created HBASE-25399:
Summary: Write HBCK tools to rebuild the tracking
Key: HBASE-25399
URL: https://issues.apache.org/jira/browse/HBASE-25399
Project: HBase
Issue Type: Sub-task
Reporter: Tak-Lon (Stephen) Wu
{color:#00}support online region repairing, and maybe offline repairing as
well
{color}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25334) TestRSGroupsFallback.testFallback is flaky
[ https://issues.apache.org/jira/browse/HBASE-25334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17249154#comment-17249154 ] Michael Stack commented on HBASE-25334: --- [~Xiaolin Ha] is the failure like that in HBASE-25389 ? > TestRSGroupsFallback.testFallback is flaky > -- > > Key: HBASE-25334 > URL: https://issues.apache.org/jira/browse/HBASE-25334 > Project: HBase > Issue Type: Test >Reporter: Xiaolin Ha >Assignee: Xiaolin Ha >Priority: Major > Attachments: 1607918235175-image.png, > image-2020-12-13-10-15-55-445.png > > > Like in CI test results of PR [https://github.com/apache/hbase/pull/2699] > failed UTs site is > [https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2699/3/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt] > > In this unit test, it checks if all table regions assigned after balance, and > then assert for the RS group of regions. > But balance() uses aync move, and will throttle move regions, sleeping > between all the table regions are moved to its RSGroup. > If waiting time is not longer than the region movement duration, the > assertion will be fail. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-25398) Cleanup unused/unneeded temporarily HFiles
Tak-Lon (Stephen) Wu created HBASE-25398: Summary: Cleanup unused/unneeded temporarily HFiles Key: HBASE-25398 URL: https://issues.apache.org/jira/browse/HBASE-25398 Project: HBase Issue Type: Sub-task Reporter: Tak-Lon (Stephen) Wu Cleanup records and write a new cleaner to move unneeded files from data directory into archives (other than compaction) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-25397) Migrate hbase:storefile system table into editable meta
Tak-Lon (Stephen) Wu created HBASE-25397:
Summary: Migrate hbase:storefile system table into editable meta
Key: HBASE-25397
URL: https://issues.apache.org/jira/browse/HBASE-25397
Project: HBase
Issue Type: Sub-task
Reporter: Tak-Lon (Stephen) Wu
{color:#00}editable meta supported by HBASE-23055 allow us to write new
column family to hbase:meta, such that we will introduce a new column family
cf:storefile to hbase:meta with different trackable columns to served as the
in-memory tracking of e.g. `storefiles` and `compactedfile` in
StoreFileManager.{color}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HBASE-25396) Integtration with Direct insert into data/cf directory
Tak-Lon (Stephen) Wu created HBASE-25396: Summary: Integtration with Direct insert into data/cf directory Key: HBASE-25396 URL: https://issues.apache.org/jira/browse/HBASE-25396 Project: HBase Issue Type: Sub-task Reporter: Tak-Lon (Stephen) Wu Utilizes direct insert compactor, file writer, committer and etc while constructing the PersistedStoreEngine -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-25395) Introduce PersistedStoreEngine and PersistedStoreFileManager
Tak-Lon (Stephen) Wu created HBASE-25395: Summary: Introduce PersistedStoreEngine and PersistedStoreFileManager Key: HBASE-25395 URL: https://issues.apache.org/jira/browse/HBASE-25395 Project: HBase Issue Type: Sub-task Reporter: Tak-Lon (Stephen) Wu Create a new wrapper store engine on top of the default store engine, as well as, introducing a new store file manager that keep tracking path of HFiles of user tables into a hbase self-managed system table (further will have a different tasks to merge into meta table and check throughput do not change much) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-25318) Configure where IntegrationTestImportTsv generates HFiles
[
https://issues.apache.org/jira/browse/HBASE-25318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mate Szalay-Beko updated HBASE-25318:
-
Description:
Currently IntegrationTestImportTsv is generating HFiles under the working
directory of the hdfs user executing the tool, before bulkloading it into HBase.
Assuming you encrypt the HBase root directory within HDFS (using HDFS
Transparent Encryption), you can bulkload HFiles only if they sit in the same
encryption zone in HDFS as the HBase root directory itself.
When IntegrationTestImportTsv is executed against a real distributed cluster
and the working directory of the current user (e.g. /user/hbase) is not in the
same encryption zone as the HBase root directory (e.g. /hbase/data) then you
will get an exception:
{code:java}
ERROR org.apache.hadoop.hbase.regionserver.HRegion: There was a partial failure
due to IO when attempting to load d :
hdfs://mycluster/user/hbase/test-data/22d8460d-04cc-e032-88ca-2cc20a7dd01c/IntegrationTestImportTsv/hfiles/d/74655e3f8da142cb94bc31b64f0475cc
org.apache.hadoop.ipc.RemoteException(java.io.IOException):
/user/hbase/test-data/22d8460d-04cc-e032-88ca-2cc20a7dd01c/IntegrationTestImportTsv/hfiles/d/74655e3f8da142cb94bc31b64f0475cc
can't be moved into an encryption zone.
{code}
In this ticket I make it configurable where the IntegrationTestImportTsv
generates the HFiles. From now, one can execute this integration test on HDFS
Transparent Encryption enabled clusters, like:
{code:java}
./bin/hbase org.apache.hadoop.hbase.mapreduce.IntegrationTestImportTsv -D
IntegrationTestImportTsv.generatedHFileFolder=//testdata{code}
was:
IntegrationTestImportTsv is generating HFiles under the working directory of
the current hdfs user executing the tool, before bulkloading it into HBase.
Assuming you encrypt the HBase root directory within HDFS (using HDFS
Transparent Encryption), you can bulkload HFiles only if they sit in the same
encryption zone in HDFS as the HBase root directory itself.
When IntegrationTestImportTsv is executed against a real distributed cluster
and the working directory of the current user (e.g. /user/hbase) is not in the
same encryption zone as the HBase root directory (e.g. /hbase/data) then you
will get an exception:
{code:java}
2020-11-21 01:06:28,963 ERROR org.apache.hadoop.hbase.regionserver.HRegion:
There was a partial failure due to IO when attempting to load d :
hdfs://mycluster/user/hbase/test-data/22d8460d-04cc-e032-88ca-2cc20a7dd01c/IntegrationTestImportTsv/hfiles/d/74655e3f8da142cb94bc31b64f0475cc
org.apache.hadoop.ipc.RemoteException(java.io.IOException):
/user/hbase/test-data/22d8460d-04cc-e032-88ca-2cc20a7dd01c/IntegrationTestImportTsv/hfiles/d/74655e3f8da142cb94bc31b64f0475cc
can't be moved into an encryption zone.
{code}
We should make it configurable where the IntegrationTestImportTsv generates the
HFiles.
> Configure where IntegrationTestImportTsv generates HFiles
> -
>
> Key: HBASE-25318
> URL: https://issues.apache.org/jira/browse/HBASE-25318
> Project: HBase
> Issue Type: Improvement
> Components: integration tests
>Reporter: Mate Szalay-Beko
>Assignee: Mate Szalay-Beko
>Priority: Minor
>
> Currently IntegrationTestImportTsv is generating HFiles under the working
> directory of the hdfs user executing the tool, before bulkloading it into
> HBase.
> Assuming you encrypt the HBase root directory within HDFS (using HDFS
> Transparent Encryption), you can bulkload HFiles only if they sit in the same
> encryption zone in HDFS as the HBase root directory itself.
> When IntegrationTestImportTsv is executed against a real distributed cluster
> and the working directory of the current user (e.g. /user/hbase) is not in
> the same encryption zone as the HBase root directory (e.g. /hbase/data) then
> you will get an exception:
>
> {code:java}
> ERROR org.apache.hadoop.hbase.regionserver.HRegion: There was a partial
> failure due to IO when attempting to load d :
> hdfs://mycluster/user/hbase/test-data/22d8460d-04cc-e032-88ca-2cc20a7dd01c/IntegrationTestImportTsv/hfiles/d/74655e3f8da142cb94bc31b64f0475cc
> org.apache.hadoop.ipc.RemoteException(java.io.IOException):
> /user/hbase/test-data/22d8460d-04cc-e032-88ca-2cc20a7dd01c/IntegrationTestImportTsv/hfiles/d/74655e3f8da142cb94bc31b64f0475cc
> can't be moved into an encryption zone.
> {code}
>
> In this ticket I make it configurable where the IntegrationTestImportTsv
> generates the HFiles. From now, one can execute this integration test on HDFS
> Transparent Encryption enabled clusters, like:
> {code:java}
> ./bin/hbase org.apache.hadoop.hbase.mapreduce.IntegrationTestImportTsv -D
> IntegrationTestImportTsv.generatedHFileFolder=/ path>/testdata{code}
--
This message was sent by
[jira] [Work started] (HBASE-25318) Configure where IntegrationTestImportTsv generates HFiles
[
https://issues.apache.org/jira/browse/HBASE-25318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HBASE-25318 started by Mate Szalay-Beko.
> Configure where IntegrationTestImportTsv generates HFiles
> -
>
> Key: HBASE-25318
> URL: https://issues.apache.org/jira/browse/HBASE-25318
> Project: HBase
> Issue Type: Improvement
> Components: integration tests
>Reporter: Mate Szalay-Beko
>Assignee: Mate Szalay-Beko
>Priority: Minor
>
> Currently IntegrationTestImportTsv is generating HFiles under the working
> directory of the hdfs user executing the tool, before bulkloading it into
> HBase.
> Assuming you encrypt the HBase root directory within HDFS (using HDFS
> Transparent Encryption), you can bulkload HFiles only if they sit in the same
> encryption zone in HDFS as the HBase root directory itself.
> When IntegrationTestImportTsv is executed against a real distributed cluster
> and the working directory of the current user (e.g. /user/hbase) is not in
> the same encryption zone as the HBase root directory (e.g. /hbase/data) then
> you will get an exception:
>
> {code:java}
> ERROR org.apache.hadoop.hbase.regionserver.HRegion: There was a partial
> failure due to IO when attempting to load d :
> hdfs://mycluster/user/hbase/test-data/22d8460d-04cc-e032-88ca-2cc20a7dd01c/IntegrationTestImportTsv/hfiles/d/74655e3f8da142cb94bc31b64f0475cc
> org.apache.hadoop.ipc.RemoteException(java.io.IOException):
> /user/hbase/test-data/22d8460d-04cc-e032-88ca-2cc20a7dd01c/IntegrationTestImportTsv/hfiles/d/74655e3f8da142cb94bc31b64f0475cc
> can't be moved into an encryption zone.
> {code}
>
> In this ticket I make it configurable where the IntegrationTestImportTsv
> generates the HFiles. From now, one can execute this integration test on HDFS
> Transparent Encryption enabled clusters, like:
> {code:java}
> ./bin/hbase org.apache.hadoop.hbase.mapreduce.IntegrationTestImportTsv -D
> IntegrationTestImportTsv.generatedHFileFolder=/ path>/testdata{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] symat opened a new pull request #2777: HBASE-25318 Configure where IntegrationTestImportTsv generates HFiles
symat opened a new pull request #2777: URL: https://github.com/apache/hbase/pull/2777 Currently IntegrationTestImportTsv is generating HFiles under the working directory of the hdfs user executing the tool, before bulkloading it into HBase. Assuming you encrypt the HBase root directory within HDFS (using HDFS Transparent Encryption), you can bulkload HFiles only if they sit in the same encryption zone in HDFS as the HBase root directory itself. When IntegrationTestImportTsv is executed against a real distributed cluster and the working directory of the current user (e.g. /user/hbase) is not in the same encryption zone as the HBase root directory (e.g. /hbase/data) then you will get an exception: ``` ERROR org.apache.hadoop.hbase.regionserver.HRegion: There was a partial failure due to IO when attempting to load d : hdfs://mycluster/user/hbase/test-data/22d8460d-04cc-e032-88ca-2cc20a7dd01c/IntegrationTestImportTsv/hfiles/d/74655e3f8da142cb94bc31b64f0475cc org.apache.hadoop.ipc.RemoteException(java.io.IOException): /user/hbase/test-data/22d8460d-04cc-e032-88ca-2cc20a7dd01c/IntegrationTestImportTsv/hfiles/d/74655e3f8da142cb94bc31b64f0475cc can't be moved into an encryption zone. ``` In this commit I mmake it configurable where the IntegrationTestImportTsv generates the HFiles. From now, one can execute this integration test on HDFS Transparent Encryption enabled clusters, like: ``` ./bin/hbase org.apache.hadoop.hbase.mapreduce.IntegrationTestImportTsv -D IntegrationTestImportTsv.generatedHFileFolder=//testdata ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-25318) Configure where IntegrationTestImportTsv generates HFiles
[
https://issues.apache.org/jira/browse/HBASE-25318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mate Szalay-Beko updated HBASE-25318:
-
Summary: Configure where IntegrationTestImportTsv generates HFiles (was:
Config option for IntegrationTestImportTsv where to generate HFiles to bulkload)
> Configure where IntegrationTestImportTsv generates HFiles
> -
>
> Key: HBASE-25318
> URL: https://issues.apache.org/jira/browse/HBASE-25318
> Project: HBase
> Issue Type: Improvement
> Components: integration tests
>Reporter: Mate Szalay-Beko
>Assignee: Mate Szalay-Beko
>Priority: Minor
>
> IntegrationTestImportTsv is generating HFiles under the working directory of
> the current hdfs user executing the tool, before bulkloading it into HBase.
> Assuming you encrypt the HBase root directory within HDFS (using HDFS
> Transparent Encryption), you can bulkload HFiles only if they sit in the same
> encryption zone in HDFS as the HBase root directory itself.
> When IntegrationTestImportTsv is executed against a real distributed cluster
> and the working directory of the current user (e.g. /user/hbase) is not in
> the same encryption zone as the HBase root directory (e.g. /hbase/data) then
> you will get an exception:
>
> {code:java}
> 2020-11-21 01:06:28,963 ERROR org.apache.hadoop.hbase.regionserver.HRegion:
> There was a partial failure due to IO when attempting to load d :
> hdfs://mycluster/user/hbase/test-data/22d8460d-04cc-e032-88ca-2cc20a7dd01c/IntegrationTestImportTsv/hfiles/d/74655e3f8da142cb94bc31b64f0475cc
> org.apache.hadoop.ipc.RemoteException(java.io.IOException):
> /user/hbase/test-data/22d8460d-04cc-e032-88ca-2cc20a7dd01c/IntegrationTestImportTsv/hfiles/d/74655e3f8da142cb94bc31b64f0475cc
> can't be moved into an encryption zone.
> {code}
> We should make it configurable where the IntegrationTestImportTsv generates
> the HFiles.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HBASE-25394) Support Snapshot related operation with direct insert HFiles into data/CF directory
Tak-Lon (Stephen) Wu created HBASE-25394:
Summary: Support Snapshot related operation with direct insert
HFiles into data/CF directory
Key: HBASE-25394
URL: https://issues.apache.org/jira/browse/HBASE-25394
Project: HBase
Issue Type: Sub-task
Reporter: Tak-Lon (Stephen) Wu
{color:#00}Support restore snapshot, clone snapshot with direct insert into
data directory{color}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25334) TestRSGroupsFallback.testFallback is flaky
[ https://issues.apache.org/jira/browse/HBASE-25334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17249144#comment-17249144 ] Xiaolin Ha commented on HBASE-25334: Oh, in the CI test result that [~stack] attached, it failed at java.lang.AssertionError at org.apache.hadoop.hbase.rsgroup.TestRSGroupsFallback.lambda$assertRegionsInGroup$0(TestRSGroupsFallback.java:123) at org.apache.hadoop.hbase.rsgroup.TestRSGroupsFallback.assertRegionsInGroup(TestRSGroupsFallback.java:120) at org.apache.hadoop.hbase.rsgroup.TestRSGroupsFallback.testFallback(TestRSGroupsFallback.java:100) before line 100 it called crashRsInGroup(RSGroupInfo.DEFAULT_GROUP).. Logs show that the assertion of correct table group is before the SCP is executed. > TestRSGroupsFallback.testFallback is flaky > -- > > Key: HBASE-25334 > URL: https://issues.apache.org/jira/browse/HBASE-25334 > Project: HBase > Issue Type: Test >Reporter: Xiaolin Ha >Assignee: Xiaolin Ha >Priority: Major > Attachments: 1607918235175-image.png, > image-2020-12-13-10-15-55-445.png > > > Like in CI test results of PR [https://github.com/apache/hbase/pull/2699] > failed UTs site is > [https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2699/3/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt] > > In this unit test, it checks if all table regions assigned after balance, and > then assert for the RS group of regions. > But balance() uses aync move, and will throttle move regions, sleeping > between all the table regions are moved to its RSGroup. > If waiting time is not longer than the region movement duration, the > assertion will be fail. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-25393) Support split and merge region with direct insert into CF directory
Tak-Lon (Stephen) Wu created HBASE-25393:
Summary: Support split and merge region with direct insert into CF
directory
Key: HBASE-25393
URL: https://issues.apache.org/jira/browse/HBASE-25393
Project: HBase
Issue Type: Sub-task
Reporter: Tak-Lon (Stephen) Wu
{color:#00}Support region SPLIT and MERGE with direct insert HFiles into
column family directory{color}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HBASE-25392) Direct insert compacted HFiles into data directory.
Tak-Lon (Stephen) Wu created HBASE-25392:
Summary: Direct insert compacted HFiles into data directory.
Key: HBASE-25392
URL: https://issues.apache.org/jira/browse/HBASE-25392
Project: HBase
Issue Type: Sub-task
Reporter: Tak-Lon (Stephen) Wu
{color:#00}when performing compaction, write directory to the data
directory {color}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HBASE-25391) Flush directly into data directory, skip rename when committing flush
Tak-Lon (Stephen) Wu created HBASE-25391:
Summary: Flush directly into data directory, skip rename when
committing flush
Key: HBASE-25391
URL: https://issues.apache.org/jira/browse/HBASE-25391
Project: HBase
Issue Type: Sub-task
Reporter: Tak-Lon (Stephen) Wu
{color:#00}When flushing memstore snapshot to HFile, we write it directly
to the data directory.{color}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-24850) CellComparator perf improvement
[ https://issues.apache.org/jira/browse/HBASE-24850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17249134#comment-17249134 ] Michael Stack commented on HBASE-24850: --- I thought BBKV was default in branch-2? Sounds like its not? Talking offline w/ Ram, we only have 'contiguous' implementations of Cell currently so comparator for now should just presume it? When different implementation (do we have an issue for discussing this?), then we can worry about changing the comparator? > CellComparator perf improvement > --- > > Key: HBASE-24850 > URL: https://issues.apache.org/jira/browse/HBASE-24850 > Project: HBase > Issue Type: Improvement > Components: Performance, scan >Affects Versions: 2.0.0 >Reporter: Anoop Sam John >Assignee: ramkrishna.s.vasudevan >Priority: Critical > Fix For: 3.0.0-alpha-1, 2.5.0 > > > We have multiple perf issues in 2.x versions compared to 1.x. Eg: > HBASE-24754, HBASE-24637. > The pattern is clear that where ever we do more and more Cell compares, there > is some degrade. In HBASE-24754, with an old KVComparator style comparator, > we see much better perf for the PutSortReducer. (Again the gain is huge > because of large number of compare ops that test is doing). This issue is to > address and optimize compares generally in CellComparatorImpl itself. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #2775: HBASE-25334 TestRSGroupsFallback.testFallback is flaky
Apache-HBase commented on pull request #2775: URL: https://github.com/apache/hbase/pull/2775#issuecomment-744581079 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 37s | Docker mode activated. | | -0 :warning: | yetus | 0m 7s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2 Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 32s | branch-2 passed | | +1 :green_heart: | compile | 0m 28s | branch-2 passed | | +1 :green_heart: | shadedjars | 7m 12s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 23s | branch-2 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 18s | the patch passed | | +1 :green_heart: | compile | 0m 28s | the patch passed | | +1 :green_heart: | javac | 0m 28s | the patch passed | | +1 :green_heart: | shadedjars | 7m 14s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 23s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 0m 35s | hbase-rsgroup in the patch failed. | | | | 27m 30s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2775/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2775 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 3ed226fce089 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / 9c2f1f7a2b | | Default Java | AdoptOpenJDK-11.0.6+10 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2775/2/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-rsgroup.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2775/2/testReport/ | | Max. process+thread count | 223 (vs. ulimit of 12500) | | modules | C: hbase-rsgroup U: hbase-rsgroup | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2775/2/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2775: HBASE-25334 TestRSGroupsFallback.testFallback is flaky
Apache-HBase commented on pull request #2775: URL: https://github.com/apache/hbase/pull/2775#issuecomment-744579367 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 35s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ branch-2 Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 44s | branch-2 passed | | +1 :green_heart: | checkstyle | 0m 14s | branch-2 passed | | +1 :green_heart: | spotbugs | 0m 44s | branch-2 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 14s | the patch passed | | +1 :green_heart: | checkstyle | 0m 14s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 11m 39s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. | | +1 :green_heart: | spotbugs | 0m 49s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 14s | The patch does not generate ASF License warnings. | | | | 28m 52s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2775/2/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2775 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle | | uname | Linux 95f8e38ba5cb 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / 9c2f1f7a2b | | Max. process+thread count | 94 (vs. ulimit of 12500) | | modules | C: hbase-rsgroup U: hbase-rsgroup | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2775/2/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=3.1.12 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2775: HBASE-25334 TestRSGroupsFallback.testFallback is flaky
Apache-HBase commented on pull request #2775: URL: https://github.com/apache/hbase/pull/2775#issuecomment-744577308 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 35s | Docker mode activated. | | -0 :warning: | yetus | 0m 7s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2 Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 41s | branch-2 passed | | +1 :green_heart: | compile | 0m 24s | branch-2 passed | | +1 :green_heart: | shadedjars | 6m 2s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 24s | branch-2 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 18s | the patch passed | | +1 :green_heart: | compile | 0m 24s | the patch passed | | +1 :green_heart: | javac | 0m 24s | the patch passed | | +1 :green_heart: | shadedjars | 5m 56s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 22s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 3m 1s | hbase-rsgroup in the patch passed. | | | | 25m 27s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2775/2/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2775 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 6659dd3c0176 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / 9c2f1f7a2b | | Default Java | AdoptOpenJDK-1.8.0_232-b09 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2775/2/testReport/ | | Max. process+thread count | 3912 (vs. ulimit of 12500) | | modules | C: hbase-rsgroup U: hbase-rsgroup | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2775/2/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] saintstack commented on a change in pull request #2747: HBASE-24850 CellComparator perf improvement
saintstack commented on a change in pull request #2747:
URL: https://github.com/apache/hbase/pull/2747#discussion_r542540107
##
File path:
hbase-common/src/main/java/org/apache/hadoop/hbase/ContiguousCellFormat.java
##
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
contributor license
+ * agreements. See the NOTICE file distributed with this work for additional
information regarding
+ * copyright ownership. The ASF licenses this file to you under the Apache
License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the
License. You may obtain a
+ * copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless
required by applicable
+ * law or agreed to in writing, software distributed under the License is
distributed on an "AS IS"
+ * BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied. See the License
+ * for the specific language governing permissions and limitations under the
License.
+ */
+package org.apache.hadoop.hbase;
+
+import org.apache.yetus.audience.InterfaceAudience;
+import org.apache.yetus.audience.InterfaceStability;
+
+/**
+ * The interface represents any cell impl that is backed by contiguous layout
like {@link KeyValue},
+ * {@link ByteBufferKeyValue}. The APIs in this interface helps to easily
access the internals of
+ * the Cell structure so that we avoid fetching the offsets and lengths every
time from the internal
+ * backed memory structure. We can use this interface only if it follows the
standard KeyValue
+ * format. Helps in performance and this is jit friendly
+ */
+@InterfaceAudience.Private
+@InterfaceStability.Evolving
+public interface ContiguousCellFormat {
+
+ /**
+ * Return the family length position/offset.
+ * @param rowLen - the rowlength of the cell
+ * @return the family's offset position
+ */
+ int getFamilyLengthPosition(int rowLen);
+
+ /**
+ * Return the family length. Use this along with {@link
#getFamilyLengthPosition(int)}
+ * @param famLengthPosition - the family offset
+ * @return the family's length
+ */
+ byte getFamilyLength(int famLengthPosition);
+
+ /**
+ * Return the family position/offset. This can be used along with
+ * the result of {@link #getFamilyLengthPosition(int)}
+ * @param familyLengthPosition - the family length position
+ * @return the family's position
+ */
+ int getFamilyInternalPosition(int familyLengthPosition);
+
+ /**
+ * Return the qualifier position/offset
+ * @param famOffset - the family offset
+ * @param famLength - the family length
+ * @return the qualifier offset/position.
+ */
+ int getQualifierInternalPosition(int famOffset, byte famLength);
+
+ /**
+ * Return the qualifier length
+ * @param keyLength - the key length
+ * @param rowLen - the row length
+ * @param famLength - the family length
+ * @return the qualifier length
+ */
+ int getQualifierLength(int keyLength, int rowLen, int famLength);
+
+ /**
+ * Return the time stamp. Use this along with {@link #getKeyLength()}
+ * @param keyLength - the key length
+ * @return the timestamp
+ */
+ long getTimestamp(int keyLength);
+
+ /**
+ * Return the type byte. Use this along with {@link #getKeyLength()}
+ * @param keyLength - the key length
+ * @return - the type byte
+ */
+ byte getTypeByte(int keyLength);
+
+ /**
+ * The keylength of the cell
+ * @return the key length
+ */
+ int getKeyLength();
+}
Review comment:
nod
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] saintstack commented on a change in pull request #2747: HBASE-24850 CellComparator perf improvement
saintstack commented on a change in pull request #2747: URL: https://github.com/apache/hbase/pull/2747#discussion_r542538789 ## File path: hbase-common/src/main/java/org/apache/hadoop/hbase/CellComparatorImpl.java ## @@ -51,35 +53,38 @@ */ public static final CellComparatorImpl COMPARATOR = new CellComparatorImpl(); + private static final ContiguousCellFormatComparator contiguousCellComparator = Review comment: Haven't checked but might be worth adding this note on what contiguous means to your marker interface This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] saintstack commented on a change in pull request #2747: HBASE-24850 CellComparator perf improvement
saintstack commented on a change in pull request #2747:
URL: https://github.com/apache/hbase/pull/2747#discussion_r542537865
##
File path:
hbase-common/src/main/java/org/apache/hadoop/hbase/ByteBufferKeyOnlyKeyValue.java
##
@@ -292,4 +287,49 @@ public long heapSize() {
}
return ClassSize.align(FIXED_OVERHEAD);
}
+
+ @Override
+ public int getFamilyLengthPosition(int rowLen) {
Review comment:
Suggestion.
Main point is worry about exposing implementation detail especially after
all the work you lot did to change the KV so we could put another impl in place
(for example, one that caches sizes as they are calculated?)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2766: [HBASE-25246] Backup/Restore hbase cell tags.
Apache-HBase commented on pull request #2766: URL: https://github.com/apache/hbase/pull/2766#issuecomment-744567696 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 32s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ branch-2 Compile Tests _ | | +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 34s | branch-2 passed | | +1 :green_heart: | checkstyle | 2m 10s | branch-2 passed | | +0 :ok: | refguide | 3m 46s | branch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect. | | +1 :green_heart: | spotbugs | 11m 37s | branch-2 passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 12s | the patch passed | | +1 :green_heart: | checkstyle | 2m 11s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +0 :ok: | refguide | 3m 21s | patch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect. | | +1 :green_heart: | hadoopcheck | 11m 37s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. | | +1 :green_heart: | spotbugs | 12m 28s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 54s | The patch does not generate ASF License warnings. | | | | 63m 56s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2766/5/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2766 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle refguide | | uname | Linux 72a437eff3e2 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / 9c2f1f7a2b | | refguide | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2766/5/artifact/yetus-general-check/output/branch-site/book.html | | refguide | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2766/5/artifact/yetus-general-check/output/patch-site/book.html | | Max. process+thread count | 137 (vs. ulimit of 12500) | | modules | C: hbase-client hbase-server hbase-mapreduce . U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2766/5/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=3.1.12 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
