[GitHub] [hbase] Apache-HBase commented on pull request #3217: HBASE-25834 Remove balanceTable method from LoadBalancer interface
Apache-HBase commented on pull request #3217: URL: https://github.com/apache/hbase/pull/3217#issuecomment-832464032 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 9s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 26s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 10s | master passed | | +1 :green_heart: | compile | 1m 19s | master passed | | +1 :green_heart: | shadedjars | 9m 1s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 52s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 4s | the patch passed | | +1 :green_heart: | compile | 1m 19s | the patch passed | | +1 :green_heart: | javac | 1m 19s | the patch passed | | +1 :green_heart: | shadedjars | 8m 58s | patch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 14s | hbase-balancer generated 2 new + 1 unchanged - 2 fixed = 3 total (was 3) | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 28s | hbase-balancer in the patch passed. | | +1 :green_heart: | unit | 216m 32s | hbase-server in the patch passed. | | | | 251m 13s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3217/4/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3217 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux b49c8fe5abc2 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 17193dae58 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3217/4/artifact/yetus-jdk8-hadoop3-check/output/diff-javadoc-javadoc-hbase-balancer.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3217/4/testReport/ | | Max. process+thread count | 3203 (vs. ulimit of 3) | | modules | C: hbase-balancer hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3217/4/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Reidddddd merged pull request #3231: HBASE-25846 Backport 'HBASE-25825 RSGroupBasedLoadBalancer.onConfigurationChange should chain the request to internal balancer' to branch-1
Reidd merged pull request #3231: URL: https://github.com/apache/hbase/pull/3231 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (HBASE-25846) Backport 'HBASE-25825 RSGroupBasedLoadBalancer.onConfigurationChange should chain the request to internal balancer' to branch-1
[ https://issues.apache.org/jira/browse/HBASE-25846?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reid Chan resolved HBASE-25846. --- Hadoop Flags: Reviewed Resolution: Fixed > Backport 'HBASE-25825 RSGroupBasedLoadBalancer.onConfigurationChange should > chain the request to internal balancer' to branch-1 > --- > > Key: HBASE-25846 > URL: https://issues.apache.org/jira/browse/HBASE-25846 > Project: HBase > Issue Type: Improvement >Reporter: Caroline Zhou >Assignee: Caroline Zhou >Priority: Minor > Fix For: 1.7.0 > > > In branch-1, > [RSGroupBasedLoadBalancer#onConfigurationChange|https://github.com/apache/hbase/blob/branch-1/hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupBasedLoadBalancer.java#L452] > doesn't do anything – it should call the internal balancer's > onConfigurationChange(). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-25846) Backport 'HBASE-25825 RSGroupBasedLoadBalancer.onConfigurationChange should chain the request to internal balancer' to branch-1
[ https://issues.apache.org/jira/browse/HBASE-25846?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reid Chan updated HBASE-25846: -- Fix Version/s: 1.7.0 > Backport 'HBASE-25825 RSGroupBasedLoadBalancer.onConfigurationChange should > chain the request to internal balancer' to branch-1 > --- > > Key: HBASE-25846 > URL: https://issues.apache.org/jira/browse/HBASE-25846 > Project: HBase > Issue Type: Improvement >Reporter: Caroline Zhou >Assignee: Caroline Zhou >Priority: Minor > Fix For: 1.7.0 > > > In branch-1, > [RSGroupBasedLoadBalancer#onConfigurationChange|https://github.com/apache/hbase/blob/branch-1/hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupBasedLoadBalancer.java#L452] > doesn't do anything – it should call the internal balancer's > onConfigurationChange(). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] virajjasani commented on pull request #3230: HBASE-25847 More DEBUG and TRACE level logging in CatalogJanitor and HbckChore
virajjasani commented on pull request #3230: URL: https://github.com/apache/hbase/pull/3230#issuecomment-832472753 `TestCatalogJanitorInMemoryStates` failure might be relevant? Just retriggered the build, will get back to this after 2nd run completes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache9 merged pull request #3217: HBASE-25834 Remove balanceTable method from LoadBalancer interface
Apache9 merged pull request #3217: URL: https://github.com/apache/hbase/pull/3217 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (HBASE-25838) Use double instead of Double in StochasticLoadBalancer
[
https://issues.apache.org/jira/browse/HBASE-25838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duo Zhang resolved HBASE-25838.
---
Fix Version/s: 2.3.6
2.4.3
2.5.0
3.0.0-alpha-1
Hadoop Flags: Reviewed
Resolution: Fixed
Pushed to branch-2.3+.
Thanks [~niuyulin] for reviewing.
> Use double instead of Double in StochasticLoadBalancer
> --
>
> Key: HBASE-25838
> URL: https://issues.apache.org/jira/browse/HBASE-25838
> Project: HBase
> Issue Type: Sub-task
> Components: Balancer, Performance
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.3, 2.3.6
>
>
> {code}
> // to save and report costs to JMX
> private Double curOverallCost = 0d;
> private Double[] tempFunctionCosts;
> private Double[] curFunctionCosts;
> {code}
> It seems we never do null check on these field or the elements, so I do not
> see any reason we should use Double here. Just use double instead.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #3204: HBASE-25741: Replication Source still having the replication metrics for peer ID which doesn't exist
Apache-HBase commented on pull request #3204: URL: https://github.com/apache/hbase/pull/3204#issuecomment-832497548 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 33s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 2 new or modified test files. | ||| _ branch-1 Compile Tests _ | | -1 :x: | mvninstall | 9m 45s | root in branch-1 failed. | | +1 :green_heart: | compile | 0m 40s | branch-1 passed with JDK Azul Systems, Inc.-1.8.0_262-b19 | | +1 :green_heart: | compile | 0m 45s | branch-1 passed with JDK Azul Systems, Inc.-1.7.0_272-b10 | | +1 :green_heart: | checkstyle | 1m 41s | branch-1 passed | | +1 :green_heart: | shadedjars | 3m 0s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 49s | branch-1 passed with JDK Azul Systems, Inc.-1.8.0_262-b19 | | +1 :green_heart: | javadoc | 0m 41s | branch-1 passed with JDK Azul Systems, Inc.-1.7.0_272-b10 | | +0 :ok: | spotbugs | 3m 3s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 3m 0s | branch-1 passed | ||| _ Patch Compile Tests _ | | -1 :x: | mvninstall | 1m 43s | root in the patch failed. | | +1 :green_heart: | compile | 0m 44s | the patch passed with JDK Azul Systems, Inc.-1.8.0_262-b19 | | +1 :green_heart: | javac | 0m 44s | the patch passed | | +1 :green_heart: | compile | 0m 44s | the patch passed with JDK Azul Systems, Inc.-1.7.0_272-b10 | | +1 :green_heart: | javac | 0m 44s | the patch passed | | +1 :green_heart: | checkstyle | 1m 30s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedjars | 2m 51s | patch has no errors when building our shaded downstream artifacts. | | -1 :x: | hadoopcheck | 1m 29s | The patch causes 10 errors with Hadoop v2.8.5. | | -1 :x: | hadoopcheck | 3m 38s | The patch causes 10 errors with Hadoop v2.9.2. | | +1 :green_heart: | javadoc | 0m 31s | the patch passed with JDK Azul Systems, Inc.-1.8.0_262-b19 | | +1 :green_heart: | javadoc | 0m 41s | the patch passed with JDK Azul Systems, Inc.-1.7.0_272-b10 | | +1 :green_heart: | findbugs | 2m 50s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 135m 34s | hbase-server in the patch failed. | | +1 :green_heart: | asflicense | 0m 38s | The patch does not generate ASF License warnings. | | | | 173m 48s | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.hbase.regionserver.TestSplitTransactionOnCluster | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3204/16/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3204 | | JIRA Issue | HBASE-25741 | | Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile | | uname | Linux 4426542e8ba8 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-agent/workspace/Base-PreCommit-GitHub-PR_PR-3204/out/precommit/personality/provided.sh | | git revision | branch-1 / 633d966 | | Default Java | Azul Systems, Inc.-1.7.0_272-b10 | | Multi-JDK versions | /usr/lib/jvm/zulu-8-amd64:Azul Systems, Inc.-1.8.0_262-b19 /usr/lib/jvm/zulu-7-amd64:Azul Systems, Inc.-1.7.0_272-b10 | | mvninstall | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3204/16/artifact/out/branch-mvninstall-root.txt | | mvninstall | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3204/16/artifact/out/patch-mvninstall-root.txt | | hadoopcheck | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3204/16/artifact/out/patch-javac-2.8.5.txt | | hadoopcheck | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3204/16/artifact/out/patch-javac-2.9.2.txt | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3204/16/artifact/out/patch-unit-hbase-server.txt | | Test Results | https://ci-hadoop.apache.or
[GitHub] [hbase] Apache-HBase commented on pull request #3230: HBASE-25847 More DEBUG and TRACE level logging in CatalogJanitor and HbckChore
Apache-HBase commented on pull request #3230: URL: https://github.com/apache/hbase/pull/3230#issuecomment-832498562 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 6s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 58s | master passed | | +1 :green_heart: | compile | 3m 15s | master passed | | +1 :green_heart: | checkstyle | 1m 2s | master passed | | +1 :green_heart: | spotbugs | 2m 5s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 36s | the patch passed | | +1 :green_heart: | compile | 3m 12s | the patch passed | | +1 :green_heart: | javac | 3m 12s | the patch passed | | -0 :warning: | checkstyle | 1m 2s | hbase-server: The patch generated 1 new + 1 unchanged - 0 fixed = 2 total (was 1) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 18m 1s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | spotbugs | 2m 14s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 16s | The patch does not generate ASF License warnings. | | | | 47m 36s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3230/2/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3230 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux f5fae22b4931 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 17193dae58 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3230/2/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | Max. process+thread count | 95 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3230/2/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (HBASE-25834) Remove balanceTable method from LoadBalancer interface
[ https://issues.apache.org/jira/browse/HBASE-25834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang resolved HBASE-25834. --- Fix Version/s: 2.5.0 3.0.0-alpha-1 Hadoop Flags: Reviewed Release Note: Remove balanceTable method from LoadBalancer interface as we never call it outside balancer implementation. Mark balanceTable method as protected in BaseLoadBalancer. Mark balanceCluster method as final in BaseLoadBalancer, the implementation classes should not override it anymore, just implement the balanceTable method is enough. Resolution: Fixed Pushed to master and branch-2. Thanks [~niuyulin] for reviewing. > Remove balanceTable method from LoadBalancer interface > -- > > Key: HBASE-25834 > URL: https://issues.apache.org/jira/browse/HBASE-25834 > Project: HBase > Issue Type: Sub-task > Components: Balancer >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 3.0.0-alpha-1, 2.5.0 > > > The balanceTable method is never called outside balancer implementation, so > let's remove it from the interface to reduce the number of public methods, > which could also reduce the work of synchronization. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-25850) Fix spotbugs warnings on branch-2
Duo Zhang created HBASE-25850:
-
Summary: Fix spotbugs warnings on branch-2
Key: HBASE-25850
URL: https://issues.apache.org/jira/browse/HBASE-25850
Project: HBase
Issue Type: Bug
Reporter: Duo Zhang
{noformat}
Dodgy code Warnings
CodeWarning
UC Useless condition: it's known that major == true at this point
Bug type UC_USELESS_CONDITION (click for details)
In class org.apache.hadoop.hbase.mob.DefaultMobStoreCompactor
In method
org.apache.hadoop.hbase.mob.DefaultMobStoreCompactor.performCompaction(Compactor$FileDetails,
InternalScanner, CellSink, long, boolean, ThroughputController, boolean, int)
Value major == true
Unreachable code at DefaultMobStoreCompactor.java:[line 210]
At DefaultMobStoreCompactor.java:[line 210]
{noformat}
https://github.com/apache/hbase/blob/afa9836b87f9c909ac4fc18d904a5b5b0a3dfff6/hbase-server/src/main/java/org/apache/hadoop/hbase/mob/DefaultMobStoreCompactor.java#L211
{code}
if (major) {
try {
delFileWriter = mobStore.createDelFileWriterInTmp(new
Date(fd.latestPutTs),
fd.maxKeyCount, major ? majorCompactionCompression :
minorCompactionCompression,
store.getRegionInfo().getStartKey());
} catch (IOException e) {
LOG.warn(
"Failed to create del writer, "
+ "we will continue the compaction by writing delete markers
directly in store files",
e);
}
}
{code}
The 'major ? majorCompactionCompression : minorCompactionCompression' is
useless as we have a 'if (major)' test above.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-25850) Fix spotbugs warnings on branch-2
[
https://issues.apache.org/jira/browse/HBASE-25850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duo Zhang updated HBASE-25850:
--
Component/s: mob
findbugs
Compaction
> Fix spotbugs warnings on branch-2
> -
>
> Key: HBASE-25850
> URL: https://issues.apache.org/jira/browse/HBASE-25850
> Project: HBase
> Issue Type: Bug
> Components: Compaction, findbugs, mob
>Reporter: Duo Zhang
>Priority: Major
>
> {noformat}
> Dodgy code Warnings
> Code Warning
> UCUseless condition: it's known that major == true at this point
> Bug type UC_USELESS_CONDITION (click for details)
> In class org.apache.hadoop.hbase.mob.DefaultMobStoreCompactor
> In method
> org.apache.hadoop.hbase.mob.DefaultMobStoreCompactor.performCompaction(Compactor$FileDetails,
> InternalScanner, CellSink, long, boolean, ThroughputController, boolean, int)
> Value major == true
> Unreachable code at DefaultMobStoreCompactor.java:[line 210]
> At DefaultMobStoreCompactor.java:[line 210]
> {noformat}
> https://github.com/apache/hbase/blob/afa9836b87f9c909ac4fc18d904a5b5b0a3dfff6/hbase-server/src/main/java/org/apache/hadoop/hbase/mob/DefaultMobStoreCompactor.java#L211
> {code}
> if (major) {
> try {
> delFileWriter = mobStore.createDelFileWriterInTmp(new
> Date(fd.latestPutTs),
> fd.maxKeyCount, major ? majorCompactionCompression :
> minorCompactionCompression,
> store.getRegionInfo().getStartKey());
> } catch (IOException e) {
> LOG.warn(
> "Failed to create del writer, "
> + "we will continue the compaction by writing delete markers
> directly in store files",
> e);
> }
> }
> {code}
> The 'major ? majorCompactionCompression : minorCompactionCompression' is
> useless as we have a 'if (major)' test above.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-25850) Fix spotbugs warnings on branch-2
[
https://issues.apache.org/jira/browse/HBASE-25850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duo Zhang updated HBASE-25850:
--
Labels: beginner trivial (was: )
> Fix spotbugs warnings on branch-2
> -
>
> Key: HBASE-25850
> URL: https://issues.apache.org/jira/browse/HBASE-25850
> Project: HBase
> Issue Type: Bug
> Components: Compaction, findbugs, mob
>Reporter: Duo Zhang
>Priority: Major
> Labels: beginner, trivial
>
> {noformat}
> Dodgy code Warnings
> Code Warning
> UCUseless condition: it's known that major == true at this point
> Bug type UC_USELESS_CONDITION (click for details)
> In class org.apache.hadoop.hbase.mob.DefaultMobStoreCompactor
> In method
> org.apache.hadoop.hbase.mob.DefaultMobStoreCompactor.performCompaction(Compactor$FileDetails,
> InternalScanner, CellSink, long, boolean, ThroughputController, boolean, int)
> Value major == true
> Unreachable code at DefaultMobStoreCompactor.java:[line 210]
> At DefaultMobStoreCompactor.java:[line 210]
> {noformat}
> https://github.com/apache/hbase/blob/afa9836b87f9c909ac4fc18d904a5b5b0a3dfff6/hbase-server/src/main/java/org/apache/hadoop/hbase/mob/DefaultMobStoreCompactor.java#L211
> {code}
> if (major) {
> try {
> delFileWriter = mobStore.createDelFileWriterInTmp(new
> Date(fd.latestPutTs),
> fd.maxKeyCount, major ? majorCompactionCompression :
> minorCompactionCompression,
> store.getRegionInfo().getStartKey());
> } catch (IOException e) {
> LOG.warn(
> "Failed to create del writer, "
> + "we will continue the compaction by writing delete markers
> directly in store files",
> e);
> }
> }
> {code}
> The 'major ? majorCompactionCompression : minorCompactionCompression' is
> useless as we have a 'if (major)' test above.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HBASE-25850) Fix spotbugs warnings on branch-2
[
https://issues.apache.org/jira/browse/HBASE-25850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Zhang reassigned HBASE-25850:
--
Assignee: Xiao Zhang
> Fix spotbugs warnings on branch-2
> -
>
> Key: HBASE-25850
> URL: https://issues.apache.org/jira/browse/HBASE-25850
> Project: HBase
> Issue Type: Bug
> Components: Compaction, findbugs, mob
>Reporter: Duo Zhang
>Assignee: Xiao Zhang
>Priority: Major
> Labels: beginner, trivial
>
> {noformat}
> Dodgy code Warnings
> Code Warning
> UCUseless condition: it's known that major == true at this point
> Bug type UC_USELESS_CONDITION (click for details)
> In class org.apache.hadoop.hbase.mob.DefaultMobStoreCompactor
> In method
> org.apache.hadoop.hbase.mob.DefaultMobStoreCompactor.performCompaction(Compactor$FileDetails,
> InternalScanner, CellSink, long, boolean, ThroughputController, boolean, int)
> Value major == true
> Unreachable code at DefaultMobStoreCompactor.java:[line 210]
> At DefaultMobStoreCompactor.java:[line 210]
> {noformat}
> https://github.com/apache/hbase/blob/afa9836b87f9c909ac4fc18d904a5b5b0a3dfff6/hbase-server/src/main/java/org/apache/hadoop/hbase/mob/DefaultMobStoreCompactor.java#L211
> {code}
> if (major) {
> try {
> delFileWriter = mobStore.createDelFileWriterInTmp(new
> Date(fd.latestPutTs),
> fd.maxKeyCount, major ? majorCompactionCompression :
> minorCompactionCompression,
> store.getRegionInfo().getStartKey());
> } catch (IOException e) {
> LOG.warn(
> "Failed to create del writer, "
> + "we will continue the compaction by writing delete markers
> directly in store files",
> e);
> }
> }
> {code}
> The 'major ? majorCompactionCompression : minorCompactionCompression' is
> useless as we have a 'if (major)' test above.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #3217: HBASE-25834 Remove balanceTable method from LoadBalancer interface
Apache-HBase commented on pull request #3217: URL: https://github.com/apache/hbase/pull/3217#issuecomment-832515692 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 59s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 34s | master passed | | +1 :green_heart: | compile | 3m 35s | master passed | | +1 :green_heart: | checkstyle | 1m 15s | master passed | | +1 :green_heart: | spotbugs | 2m 31s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 38s | the patch passed | | +1 :green_heart: | compile | 3m 38s | the patch passed | | +1 :green_heart: | javac | 3m 38s | the patch passed | | +1 :green_heart: | checkstyle | 1m 16s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 17m 53s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | spotbugs | 2m 59s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 25s | The patch does not generate ASF License warnings. | | | | 50m 5s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3217/5/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3217 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux 79ac78a92c02 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 17193dae58 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Max. process+thread count | 96 (vs. ulimit of 3) | | modules | C: hbase-balancer hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3217/5/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Work started] (HBASE-25839) Bulk Import fails with java.io.IOException: Type mismatch in value from map
[
https://issues.apache.org/jira/browse/HBASE-25839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HBASE-25839 started by ZFabrik.
---
> Bulk Import fails with java.io.IOException: Type mismatch in value from map
> ---
>
> Key: HBASE-25839
> URL: https://issues.apache.org/jira/browse/HBASE-25839
> Project: HBase
> Issue Type: Bug
> Components: mapreduce
>Affects Versions: 2.2.5
>Reporter: ZFabrik
>Assignee: ZFabrik
>Priority: Major
>
> Bulk Import with {{-Dimport.bulk.output=/HFILES
> -Dimport.bulk.hasLargeResult=true}} always results in
> {noformat}
> Error: java.io.IOException: Type mismatch in value from map: expected
> org.apache.hadoop.hbase.util.MapReduceExtendedCell, received
> org.apache.hadoop.hbase.IndividualBytesFieldCell
> at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:1077)
> at
> org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:715)
> at
> org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89)
> at
> org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:112)
> at
> org.apache.hadoop.hbase.mapreduce.Import$CellSortImporter.map(Import.java:423)
> at
> org.apache.hadoop.hbase.mapreduce.Import$CellSortImporter.map(Import.java:394)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:175)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1844)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:169)
> {noformat}
> The Problem is that in
> {{org.apache.hadoop.hbase.mapreduce.Import.CellSortImporter#map}} the
> {{Cell}} coming from {{value.rawCells()}} (which is of type
> {{org.apache.hadoop.hbase.IndividualBytesFieldCell}}) is directly written to
> the context without wrapping it into {{MapReduceExtendedCell}} (like it
> happens at {{CellImporter#map}}).
> IMHO [line
> 423|https://github.com/apache/hbase/blob/f76a601273e834267b55c0cda12474590283fd4c/hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/Import.java#L423]
> must look like this:
> {code:java}
> context.write(new CellWritableComparable(ret), new
> MapReduceExtendedCell(ret));
> {code}
> Furthermore it seems to me that this is also wrong in all subsequent versions
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25839) Bulk Import fails with java.io.IOException: Type mismatch in value from map
[
https://issues.apache.org/jira/browse/HBASE-25839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17339549#comment-17339549
]
ZFabrik commented on HBASE-25839:
-
Hi Chevreuil,
Never did this before I'm not so familiar with the Github processes - must
admit that I'm a subversion user ;)
But I somehow manged to produce this:
[https://github.com/offermannu/hbase/pull/1] - I hope that this is want you
asked me to do.
> Bulk Import fails with java.io.IOException: Type mismatch in value from map
> ---
>
> Key: HBASE-25839
> URL: https://issues.apache.org/jira/browse/HBASE-25839
> Project: HBase
> Issue Type: Bug
> Components: mapreduce
>Affects Versions: 2.2.5
>Reporter: ZFabrik
>Assignee: ZFabrik
>Priority: Major
>
> Bulk Import with {{-Dimport.bulk.output=/HFILES
> -Dimport.bulk.hasLargeResult=true}} always results in
> {noformat}
> Error: java.io.IOException: Type mismatch in value from map: expected
> org.apache.hadoop.hbase.util.MapReduceExtendedCell, received
> org.apache.hadoop.hbase.IndividualBytesFieldCell
> at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:1077)
> at
> org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:715)
> at
> org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89)
> at
> org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:112)
> at
> org.apache.hadoop.hbase.mapreduce.Import$CellSortImporter.map(Import.java:423)
> at
> org.apache.hadoop.hbase.mapreduce.Import$CellSortImporter.map(Import.java:394)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:175)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1844)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:169)
> {noformat}
> The Problem is that in
> {{org.apache.hadoop.hbase.mapreduce.Import.CellSortImporter#map}} the
> {{Cell}} coming from {{value.rawCells()}} (which is of type
> {{org.apache.hadoop.hbase.IndividualBytesFieldCell}}) is directly written to
> the context without wrapping it into {{MapReduceExtendedCell}} (like it
> happens at {{CellImporter#map}}).
> IMHO [line
> 423|https://github.com/apache/hbase/blob/f76a601273e834267b55c0cda12474590283fd4c/hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/Import.java#L423]
> must look like this:
> {code:java}
> context.write(new CellWritableComparable(ret), new
> MapReduceExtendedCell(ret));
> {code}
> Furthermore it seems to me that this is also wrong in all subsequent versions
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (HBASE-25839) Bulk Import fails with java.io.IOException: Type mismatch in value from map
[
https://issues.apache.org/jira/browse/HBASE-25839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17339549#comment-17339549
]
ZFabrik edited comment on HBASE-25839 at 5/5/21, 9:26 AM:
--
Hi Chevreuil,
Never did this before I'm not so familiar with the Github processes - must
admit that I'm a subversion user ;)
But I somehow manged to produce this:
[https://github.com/offermannu/hbase/pull/1] - I hope that this is want you
asked me to do.
- Udo
was (Author: zfabrik):
Hi Chevreuil,
Never did this before I'm not so familiar with the Github processes - must
admit that I'm a subversion user ;)
But I somehow manged to produce this:
[https://github.com/offermannu/hbase/pull/1] - I hope that this is want you
asked me to do.
> Bulk Import fails with java.io.IOException: Type mismatch in value from map
> ---
>
> Key: HBASE-25839
> URL: https://issues.apache.org/jira/browse/HBASE-25839
> Project: HBase
> Issue Type: Bug
> Components: mapreduce
>Affects Versions: 2.2.5
>Reporter: ZFabrik
>Assignee: ZFabrik
>Priority: Major
>
> Bulk Import with {{-Dimport.bulk.output=/HFILES
> -Dimport.bulk.hasLargeResult=true}} always results in
> {noformat}
> Error: java.io.IOException: Type mismatch in value from map: expected
> org.apache.hadoop.hbase.util.MapReduceExtendedCell, received
> org.apache.hadoop.hbase.IndividualBytesFieldCell
> at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:1077)
> at
> org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:715)
> at
> org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89)
> at
> org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:112)
> at
> org.apache.hadoop.hbase.mapreduce.Import$CellSortImporter.map(Import.java:423)
> at
> org.apache.hadoop.hbase.mapreduce.Import$CellSortImporter.map(Import.java:394)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:175)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1844)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:169)
> {noformat}
> The Problem is that in
> {{org.apache.hadoop.hbase.mapreduce.Import.CellSortImporter#map}} the
> {{Cell}} coming from {{value.rawCells()}} (which is of type
> {{org.apache.hadoop.hbase.IndividualBytesFieldCell}}) is directly written to
> the context without wrapping it into {{MapReduceExtendedCell}} (like it
> happens at {{CellImporter#map}}).
> IMHO [line
> 423|https://github.com/apache/hbase/blob/f76a601273e834267b55c0cda12474590283fd4c/hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/Import.java#L423]
> must look like this:
> {code:java}
> context.write(new CellWritableComparable(ret), new
> MapReduceExtendedCell(ret));
> {code}
> Furthermore it seems to me that this is also wrong in all subsequent versions
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #3230: HBASE-25847 More DEBUG and TRACE level logging in CatalogJanitor and HbckChore
Apache-HBase commented on pull request #3230: URL: https://github.com/apache/hbase/pull/3230#issuecomment-832584044 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 28s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 36s | master passed | | +1 :green_heart: | compile | 1m 13s | master passed | | +1 :green_heart: | shadedjars | 8m 16s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 44s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 20s | the patch passed | | +1 :green_heart: | compile | 1m 14s | the patch passed | | +1 :green_heart: | javac | 1m 14s | the patch passed | | +1 :green_heart: | shadedjars | 8m 49s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 44s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 156m 27s | hbase-server in the patch failed. | | | | 188m 53s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3230/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3230 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 486137bc9384 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 17193dae58 | | Default Java | AdoptOpenJDK-11.0.10+9 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3230/2/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3230/2/testReport/ | | Max. process+thread count | 3930 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3230/2/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] virajjasani commented on pull request #3204: HBASE-25741: Replication Source still having the replication metrics for peer ID which doesn't exist
virajjasani commented on pull request #3204: URL: https://github.com/apache/hbase/pull/3204#issuecomment-832588050 mvninstall failure is known issue. However, hbase-server is built successfully and tests look good. @Reidd can we merged this since changes are limited to `hbase-server` module only? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3217: HBASE-25834 Remove balanceTable method from LoadBalancer interface
Apache-HBase commented on pull request #3217: URL: https://github.com/apache/hbase/pull/3217#issuecomment-832602039 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 30s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 21s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 30s | master passed | | +1 :green_heart: | compile | 1m 36s | master passed | | +1 :green_heart: | shadedjars | 11m 52s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 18s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 18s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 5m 59s | the patch passed | | +1 :green_heart: | compile | 1m 57s | the patch passed | | +1 :green_heart: | javac | 1m 57s | the patch passed | | +1 :green_heart: | shadedjars | 10m 26s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 20s | hbase-balancer generated 0 new + 2 unchanged - 2 fixed = 2 total (was 4) | | +1 :green_heart: | javadoc | 0m 49s | hbase-server in the patch passed. | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 36s | hbase-balancer in the patch passed. | | -1 :x: | unit | 156m 39s | hbase-server in the patch failed. | | | | 199m 29s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3217/5/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3217 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux bae2e1581ae4 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 17193dae58 | | Default Java | AdoptOpenJDK-11.0.10+9 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3217/5/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3217/5/testReport/ | | Max. process+thread count | 3773 (vs. ulimit of 3) | | modules | C: hbase-balancer hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3217/5/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3230: HBASE-25847 More DEBUG and TRACE level logging in CatalogJanitor and HbckChore
Apache-HBase commented on pull request #3230: URL: https://github.com/apache/hbase/pull/3230#issuecomment-832612958 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 7s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 18s | master passed | | +1 :green_heart: | compile | 1m 2s | master passed | | +1 :green_heart: | shadedjars | 8m 54s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 38s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 0s | the patch passed | | +1 :green_heart: | compile | 1m 2s | the patch passed | | +1 :green_heart: | javac | 1m 2s | the patch passed | | +1 :green_heart: | shadedjars | 8m 55s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 38s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 212m 5s | hbase-server in the patch failed. | | | | 244m 34s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3230/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3230 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux f83ce92d60e4 4.15.0-126-generic #129-Ubuntu SMP Mon Nov 23 18:53:38 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 17193dae58 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3230/2/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3230/2/testReport/ | | Max. process+thread count | 3243 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3230/2/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-25839) Bulk Import fails with java.io.IOException: Type mismatch in value from map
[
https://issues.apache.org/jira/browse/HBASE-25839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17339605#comment-17339605
]
Rushabh Shah commented on HBASE-25839:
--
[~zfabrik] This is the github repo where we accept pull requests.
https://github.com/apache/hbase
> Bulk Import fails with java.io.IOException: Type mismatch in value from map
> ---
>
> Key: HBASE-25839
> URL: https://issues.apache.org/jira/browse/HBASE-25839
> Project: HBase
> Issue Type: Bug
> Components: mapreduce
>Affects Versions: 2.2.5
>Reporter: ZFabrik
>Assignee: ZFabrik
>Priority: Major
>
> Bulk Import with {{-Dimport.bulk.output=/HFILES
> -Dimport.bulk.hasLargeResult=true}} always results in
> {noformat}
> Error: java.io.IOException: Type mismatch in value from map: expected
> org.apache.hadoop.hbase.util.MapReduceExtendedCell, received
> org.apache.hadoop.hbase.IndividualBytesFieldCell
> at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:1077)
> at
> org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:715)
> at
> org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89)
> at
> org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:112)
> at
> org.apache.hadoop.hbase.mapreduce.Import$CellSortImporter.map(Import.java:423)
> at
> org.apache.hadoop.hbase.mapreduce.Import$CellSortImporter.map(Import.java:394)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:175)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1844)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:169)
> {noformat}
> The Problem is that in
> {{org.apache.hadoop.hbase.mapreduce.Import.CellSortImporter#map}} the
> {{Cell}} coming from {{value.rawCells()}} (which is of type
> {{org.apache.hadoop.hbase.IndividualBytesFieldCell}}) is directly written to
> the context without wrapping it into {{MapReduceExtendedCell}} (like it
> happens at {{CellImporter#map}}).
> IMHO [line
> 423|https://github.com/apache/hbase/blob/f76a601273e834267b55c0cda12474590283fd4c/hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/Import.java#L423]
> must look like this:
> {code:java}
> context.write(new CellWritableComparable(ret), new
> MapReduceExtendedCell(ret));
> {code}
> Furthermore it seems to me that this is also wrong in all subsequent versions
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (HBASE-25839) Bulk Import fails with java.io.IOException: Type mismatch in value from map
[
https://issues.apache.org/jira/browse/HBASE-25839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17339605#comment-17339605
]
Rushabh Shah edited comment on HBASE-25839 at 5/5/21, 11:41 AM:
[~zfabrik] This is the github repo where we accept pull requests.
https://github.com/apache/hbase
Also this is a guideline which will help new contributors to hbase project.
https://hbase.apache.org/book.html#developer
was (Author: shahrs87):
[~zfabrik] This is the github repo where we accept pull requests.
https://github.com/apache/hbase
> Bulk Import fails with java.io.IOException: Type mismatch in value from map
> ---
>
> Key: HBASE-25839
> URL: https://issues.apache.org/jira/browse/HBASE-25839
> Project: HBase
> Issue Type: Bug
> Components: mapreduce
>Affects Versions: 2.2.5
>Reporter: ZFabrik
>Assignee: ZFabrik
>Priority: Major
>
> Bulk Import with {{-Dimport.bulk.output=/HFILES
> -Dimport.bulk.hasLargeResult=true}} always results in
> {noformat}
> Error: java.io.IOException: Type mismatch in value from map: expected
> org.apache.hadoop.hbase.util.MapReduceExtendedCell, received
> org.apache.hadoop.hbase.IndividualBytesFieldCell
> at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:1077)
> at
> org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:715)
> at
> org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89)
> at
> org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:112)
> at
> org.apache.hadoop.hbase.mapreduce.Import$CellSortImporter.map(Import.java:423)
> at
> org.apache.hadoop.hbase.mapreduce.Import$CellSortImporter.map(Import.java:394)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:175)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1844)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:169)
> {noformat}
> The Problem is that in
> {{org.apache.hadoop.hbase.mapreduce.Import.CellSortImporter#map}} the
> {{Cell}} coming from {{value.rawCells()}} (which is of type
> {{org.apache.hadoop.hbase.IndividualBytesFieldCell}}) is directly written to
> the context without wrapping it into {{MapReduceExtendedCell}} (like it
> happens at {{CellImporter#map}}).
> IMHO [line
> 423|https://github.com/apache/hbase/blob/f76a601273e834267b55c0cda12474590283fd4c/hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/Import.java#L423]
> must look like this:
> {code:java}
> context.write(new CellWritableComparable(ret), new
> MapReduceExtendedCell(ret));
> {code}
> Furthermore it seems to me that this is also wrong in all subsequent versions
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] virajjasani commented on pull request #3230: HBASE-25847 More DEBUG and TRACE level logging in CatalogJanitor and HbckChore
virajjasani commented on pull request #3230: URL: https://github.com/apache/hbase/pull/3230#issuecomment-832622591 The error is persistent and it is relevant ``` java.lang.AssertionError: Parent region should have been removed from RegionStates at org.apache.hadoop.hbase.master.janitor.TestCatalogJanitorInMemoryStates.testInMemoryParentCleanup(TestCatalogJanitorInMemoryStates.java:123) ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] offermannu opened a new pull request #3232: Fix HBASE-25839 Bulk Import fails with java.io.IOException: Type mismatch in value from map
offermannu opened a new pull request #3232: URL: https://github.com/apache/hbase/pull/3232 1. CellSortImporter emits values wrapped inside MapReduceExtendedCell 2. CellWritableComparable serializes given cells using standard KeyValue serialization methods so that it fits the existing deserilaization method -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3217: HBASE-25834 Remove balanceTable method from LoadBalancer interface
Apache-HBase commented on pull request #3217: URL: https://github.com/apache/hbase/pull/3217#issuecomment-832628347 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 5s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 12s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 2s | master passed | | +1 :green_heart: | compile | 1m 20s | master passed | | +1 :green_heart: | shadedjars | 8m 55s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 50s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 4s | the patch passed | | +1 :green_heart: | compile | 1m 21s | the patch passed | | +1 :green_heart: | javac | 1m 21s | the patch passed | | +1 :green_heart: | shadedjars | 8m 52s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 14s | hbase-balancer generated 0 new + 1 unchanged - 2 fixed = 1 total (was 3) | | +1 :green_heart: | javadoc | 0m 36s | hbase-server in the patch passed. | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 27s | hbase-balancer in the patch passed. | | +1 :green_heart: | unit | 215m 20s | hbase-server in the patch passed. | | | | 249m 24s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3217/5/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3217 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux bef21b620fce 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 17193dae58 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3217/5/testReport/ | | Max. process+thread count | 3481 (vs. ulimit of 3) | | modules | C: hbase-balancer hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3217/5/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3232: Fix HBASE-25839 Bulk Import fails with java.io.IOException: Type mismatch in value from map
Apache-HBase commented on pull request #3232: URL: https://github.com/apache/hbase/pull/3232#issuecomment-832648833 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 29s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 59s | master passed | | +1 :green_heart: | compile | 0m 50s | master passed | | +1 :green_heart: | checkstyle | 0m 21s | master passed | | +1 :green_heart: | spotbugs | 0m 47s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 38s | the patch passed | | +1 :green_heart: | compile | 0m 48s | the patch passed | | +1 :green_heart: | javac | 0m 48s | the patch passed | | -0 :warning: | checkstyle | 0m 18s | hbase-mapreduce: The patch generated 1 new + 15 unchanged - 0 fixed = 16 total (was 15) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 18m 1s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | spotbugs | 0m 55s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 15s | The patch does not generate ASF License warnings. | | | | 38m 4s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3232/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3232 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux 8286e7369f46 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 90f986497b | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3232/1/artifact/yetus-general-check/output/diff-checkstyle-hbase-mapreduce.txt | | Max. process+thread count | 96 (vs. ulimit of 3) | | modules | C: hbase-mapreduce U: hbase-mapreduce | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3232/1/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3232: Fix HBASE-25839 Bulk Import fails with java.io.IOException: Type mismatch in value from map
Apache-HBase commented on pull request #3232: URL: https://github.com/apache/hbase/pull/3232#issuecomment-832649619 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 6s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 55s | master passed | | +1 :green_heart: | compile | 0m 27s | master passed | | +1 :green_heart: | shadedjars | 8m 4s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 22s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 42s | the patch passed | | +1 :green_heart: | compile | 0m 28s | the patch passed | | +1 :green_heart: | javac | 0m 28s | the patch passed | | +1 :green_heart: | shadedjars | 8m 10s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 20s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 11m 27s | hbase-mapreduce in the patch passed. | | | | 39m 18s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3232/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3232 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 866ba9bc1fbe 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 90f986497b | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3232/1/testReport/ | | Max. process+thread count | 3528 (vs. ulimit of 3) | | modules | C: hbase-mapreduce U: hbase-mapreduce | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3232/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3232: Fix HBASE-25839 Bulk Import fails with java.io.IOException: Type mismatch in value from map
Apache-HBase commented on pull request #3232: URL: https://github.com/apache/hbase/pull/3232#issuecomment-832654150 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 13s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 59s | master passed | | +1 :green_heart: | compile | 0m 29s | master passed | | +1 :green_heart: | shadedjars | 9m 4s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 22s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 44s | the patch passed | | +1 :green_heart: | compile | 0m 29s | the patch passed | | +1 :green_heart: | javac | 0m 29s | the patch passed | | +1 :green_heart: | shadedjars | 9m 6s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 20s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 14m 35s | hbase-mapreduce in the patch passed. | | | | 46m 38s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3232/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3232 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 4d79fc4a9bf5 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 90f986497b | | Default Java | AdoptOpenJDK-11.0.10+9 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3232/1/testReport/ | | Max. process+thread count | 2798 (vs. ulimit of 3) | | modules | C: hbase-mapreduce U: hbase-mapreduce | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3232/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (HBASE-25851) Make LoadBalancer not extend Configured interface
Duo Zhang created HBASE-25851: - Summary: Make LoadBalancer not extend Configured interface Key: HBASE-25851 URL: https://issues.apache.org/jira/browse/HBASE-25851 Project: HBase Issue Type: Sub-task Environment: M Reporter: Duo Zhang Fix For: 3.0.0-alpha-1, 2.5.0 On master, LoadBalancer has a setClusterInfoProvider method and on branch-2, LoadBalancer has a setMasterServices method. Both ClusterInfoProvider and MasterServices could give a Configuration out so I do not think we need to expose another setConf method. When refactoring StochasticLoadBalancer, the initialization order is also a pain as after setConf, we will load a lot of configuration but at that time, the ClusterInfoProvider is not present, and easy to cause NPE... And it could also help simplity locking. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-25851) Make LoadBalancer not extend Configurable interface
[ https://issues.apache.org/jira/browse/HBASE-25851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-25851: -- Summary: Make LoadBalancer not extend Configurable interface (was: Make LoadBalancer not extend Configured interface) > Make LoadBalancer not extend Configurable interface > --- > > Key: HBASE-25851 > URL: https://issues.apache.org/jira/browse/HBASE-25851 > Project: HBase > Issue Type: Sub-task > Environment: M >Reporter: Duo Zhang >Priority: Major > Fix For: 3.0.0-alpha-1, 2.5.0 > > > On master, LoadBalancer has a setClusterInfoProvider method and on branch-2, > LoadBalancer has a setMasterServices method. Both ClusterInfoProvider and > MasterServices could give a Configuration out so I do not think we need to > expose another setConf method. > When refactoring StochasticLoadBalancer, the initialization order is also a > pain as after setConf, we will load a lot of configuration but at that time, > the ClusterInfoProvider is not present, and easy to cause NPE... > And it could also help simplity locking. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25835) Ignore duplicate split requests from regionserver reports
[
https://issues.apache.org/jira/browse/HBASE-25835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17339660#comment-17339660
]
Hudson commented on HBASE-25835:
Results for branch master
[build #283 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/283/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/283/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/283/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/283/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Ignore duplicate split requests from regionserver reports
> -
>
> Key: HBASE-25835
> URL: https://issues.apache.org/jira/browse/HBASE-25835
> Project: HBase
> Issue Type: Sub-task
>Affects Versions: 2.4.3
>Reporter: Andrew Kyle Purtell
>Assignee: Andrew Kyle Purtell
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.3
>
>
> A SplitTableRegionProcedure may already be running when a regionserver report
> is received that includes a split request. The outcome is multiple
> SplitTableRegionProcedure procedures scheduled for the split request, only
> one of which can succeed. The others error out. Perhaps it is only WARN level
> noise in the logs that will cause operator concern, but is clearly not
> correct.
> Do not create a split procedure in response to a region state change report
> if the region is not open or already splitting.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25842) Move regionserver http-related code into o.a.h.h.regionserver.http
[
https://issues.apache.org/jira/browse/HBASE-25842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17339661#comment-17339661
]
Hudson commented on HBASE-25842:
Results for branch master
[build #283 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/283/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/283/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/283/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/283/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Move regionserver http-related code into o.a.h.h.regionserver.http
> --
>
> Key: HBASE-25842
> URL: https://issues.apache.org/jira/browse/HBASE-25842
> Project: HBase
> Issue Type: Task
> Components: regionserver
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
>Priority: Minor
> Fix For: 3.0.0-alpha-1, 2.5.0
>
>
> Minor refactor. Group code related to the Region Server's Info Page into a
> common package.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25797) Batch version of checkAndMutate on AsyncTable always returns false
[
https://issues.apache.org/jira/browse/HBASE-25797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17339680#comment-17339680
]
Toshihiro Suzuki commented on HBASE-25797:
--
[~bryson] Thank you for sharing the code. But I just realized that you are
using different versions between client and server. Can you please use the same
version? And note that hbase-2.2.6 doesn't support batch version of
checkAndMutate(). We support that feature from hbase-2.4.0:
Please see the following Jira for the details:
https://issues.apache.org/jira/browse/HBASE-8458
> Batch version of checkAndMutate on AsyncTable always returns false
> --
>
> Key: HBASE-25797
> URL: https://issues.apache.org/jira/browse/HBASE-25797
> Project: HBase
> Issue Type: Bug
> Components: Client
>Affects Versions: 2.4.1
>Reporter: Hochul Shin
>Assignee: Toshihiro Suzuki
>Priority: Minor
> Labels: bug
>
> Batch version of CheckAndMutate always returns false whereas non-batch
> version works fine.
> The code is like:
>
> {code:java}
> AysncTable table = connection.getTable(tableName, executorService);
> ...
> // add r1
> table.put(Arrays.asList(
> new Put(Bytes.toBytes("r1")).addColumn(COL_FAMILY, Bytes.toBytes("q1"),
> Bytes.toBytes("v1";
> CheckAndMutate checkAndMutate1 =
> CheckAndMutate.newBuilder(Bytes.toBytes("r1"))
> .ifNotExists(COL_FAMILY, Bytes.toBytes("q1"))
> .build(new Put(Bytes.toBytes("r1")).addColumn(COL_FAMILY,
> Bytes.toBytes("q1"), Bytes.toBytes("v1")));
> CheckAndMutate checkAndMutate2 =
> CheckAndMutate.newBuilder(Bytes.toBytes("r2"))
> .ifNotExists(COL_FAMILY, Bytes.toBytes("q2"))
> .build(new Put(Bytes.toBytes("r2")).addColumn(COL_FAMILY,
> Bytes.toBytes("q2"), Bytes.toBytes("v2")));
> {code}
>
> With batch version of checkAndMutate
> {code:java}
> List> results =
> table.checkAndMutate(Arrays.asList(checkAndMutate1, checkAndMutate2));
> System.out.println("first: " + results.get(0).join().isSuccess()); // false
> System.out.println("second: " + results.get(1).join().isSuccess()); //false!
> {code}
>
>
> Even it returned false, the r1 was written to the table. (timestamp was
> updated)
>
> With non-batch version of checkAndMutate
>
> {code:java}
> CompletableFuture result1 =
> table.checkAndMutate(checkAndMutate1);
> CompletableFuture result2 =
> table.checkAndMutate(checkAndMutate2);
> System.out.println("first: " + result1.join().isSuccess()); // false
> System.out.println("second: " + result2.join().isSuccess()); // true as
> expected
> {code}
>
> As expected r1 wasn't written to the table.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (HBASE-25797) Batch version of checkAndMutate on AsyncTable always returns false
[
https://issues.apache.org/jira/browse/HBASE-25797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17339680#comment-17339680
]
Toshihiro Suzuki edited comment on HBASE-25797 at 5/5/21, 2:07 PM:
---
[~bryson] Thank you for sharing the code. But I just realized that you are
using different versions between client and server. Can you please use the same
version? And note that hbase-2.2.6 doesn't support batch version of
checkAndMutate(). We support that feature from hbase-2.4.0.
Please see the following Jira for the details:
https://issues.apache.org/jira/browse/HBASE-8458
was (Author: brfrn169):
[~bryson] Thank you for sharing the code. But I just realized that you are
using different versions between client and server. Can you please use the same
version? And note that hbase-2.2.6 doesn't support batch version of
checkAndMutate(). We support that feature from hbase-2.4.0:
Please see the following Jira for the details:
https://issues.apache.org/jira/browse/HBASE-8458
> Batch version of checkAndMutate on AsyncTable always returns false
> --
>
> Key: HBASE-25797
> URL: https://issues.apache.org/jira/browse/HBASE-25797
> Project: HBase
> Issue Type: Bug
> Components: Client
>Affects Versions: 2.4.1
>Reporter: Hochul Shin
>Assignee: Toshihiro Suzuki
>Priority: Minor
> Labels: bug
>
> Batch version of CheckAndMutate always returns false whereas non-batch
> version works fine.
> The code is like:
>
> {code:java}
> AysncTable table = connection.getTable(tableName, executorService);
> ...
> // add r1
> table.put(Arrays.asList(
> new Put(Bytes.toBytes("r1")).addColumn(COL_FAMILY, Bytes.toBytes("q1"),
> Bytes.toBytes("v1";
> CheckAndMutate checkAndMutate1 =
> CheckAndMutate.newBuilder(Bytes.toBytes("r1"))
> .ifNotExists(COL_FAMILY, Bytes.toBytes("q1"))
> .build(new Put(Bytes.toBytes("r1")).addColumn(COL_FAMILY,
> Bytes.toBytes("q1"), Bytes.toBytes("v1")));
> CheckAndMutate checkAndMutate2 =
> CheckAndMutate.newBuilder(Bytes.toBytes("r2"))
> .ifNotExists(COL_FAMILY, Bytes.toBytes("q2"))
> .build(new Put(Bytes.toBytes("r2")).addColumn(COL_FAMILY,
> Bytes.toBytes("q2"), Bytes.toBytes("v2")));
> {code}
>
> With batch version of checkAndMutate
> {code:java}
> List> results =
> table.checkAndMutate(Arrays.asList(checkAndMutate1, checkAndMutate2));
> System.out.println("first: " + results.get(0).join().isSuccess()); // false
> System.out.println("second: " + results.get(1).join().isSuccess()); //false!
> {code}
>
>
> Even it returned false, the r1 was written to the table. (timestamp was
> updated)
>
> With non-batch version of checkAndMutate
>
> {code:java}
> CompletableFuture result1 =
> table.checkAndMutate(checkAndMutate1);
> CompletableFuture result2 =
> table.checkAndMutate(checkAndMutate2);
> System.out.println("first: " + result1.join().isSuccess()); // false
> System.out.println("second: " + result2.join().isSuccess()); // true as
> expected
> {code}
>
> As expected r1 wasn't written to the table.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HBASE-25851) Make LoadBalancer not extend Configurable interface
[ https://issues.apache.org/jira/browse/HBASE-25851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang reassigned HBASE-25851: - Assignee: Duo Zhang > Make LoadBalancer not extend Configurable interface > --- > > Key: HBASE-25851 > URL: https://issues.apache.org/jira/browse/HBASE-25851 > Project: HBase > Issue Type: Sub-task > Environment: M >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 3.0.0-alpha-1, 2.5.0 > > > On master, LoadBalancer has a setClusterInfoProvider method and on branch-2, > LoadBalancer has a setMasterServices method. Both ClusterInfoProvider and > MasterServices could give a Configuration out so I do not think we need to > expose another setConf method. > When refactoring StochasticLoadBalancer, the initialization order is also a > pain as after setConf, we will load a lot of configuration but at that time, > the ClusterInfoProvider is not present, and easy to cause NPE... > And it could also help simplity locking. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work started] (HBASE-25851) Make LoadBalancer not extend Configurable interface
[ https://issues.apache.org/jira/browse/HBASE-25851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HBASE-25851 started by Duo Zhang. - > Make LoadBalancer not extend Configurable interface > --- > > Key: HBASE-25851 > URL: https://issues.apache.org/jira/browse/HBASE-25851 > Project: HBase > Issue Type: Sub-task > Environment: M >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 3.0.0-alpha-1, 2.5.0 > > > On master, LoadBalancer has a setClusterInfoProvider method and on branch-2, > LoadBalancer has a setMasterServices method. Both ClusterInfoProvider and > MasterServices could give a Configuration out so I do not think we need to > expose another setConf method. > When refactoring StochasticLoadBalancer, the initialization order is also a > pain as after setConf, we will load a lot of configuration but at that time, > the ClusterInfoProvider is not present, and easy to cause NPE... > And it could also help simplity locking. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-25851) Make LoadBalancer not extend Configurable interface
[ https://issues.apache.org/jira/browse/HBASE-25851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-25851: -- Parent Issue: HBASE-25826 (was: HBASE-25649) > Make LoadBalancer not extend Configurable interface > --- > > Key: HBASE-25851 > URL: https://issues.apache.org/jira/browse/HBASE-25851 > Project: HBase > Issue Type: Sub-task > Environment: M >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 3.0.0-alpha-1, 2.5.0 > > > On master, LoadBalancer has a setClusterInfoProvider method and on branch-2, > LoadBalancer has a setMasterServices method. Both ClusterInfoProvider and > MasterServices could give a Configuration out so I do not think we need to > expose another setConf method. > When refactoring StochasticLoadBalancer, the initialization order is also a > pain as after setConf, we will load a lot of configuration but at that time, > the ClusterInfoProvider is not present, and easy to cause NPE... > And it could also help simplity locking. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-25852) Move all the intialization work of LoadBalancer implementation to initialize method
Duo Zhang created HBASE-25852: - Summary: Move all the intialization work of LoadBalancer implementation to initialize method Key: HBASE-25852 URL: https://issues.apache.org/jira/browse/HBASE-25852 Project: HBase Issue Type: Sub-task Components: Balancer Reporter: Duo Zhang Assignee: Duo Zhang Fix For: 3.0.0-alpha-1, 2.5.0 To make sure that we have all the necessary properties set before initialization. And also, this could also help to make the synchronization more clear, as before initialization, we should not call balance or assignment methods yet, so we do not need to synchronize initialization method. We only need to synchronize when we want to change the configuration. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-21674) Port HBASE-21652 (Refactor ThriftServer making thrift2 server inherited from thrift1 server) to branch-1
[
https://issues.apache.org/jira/browse/HBASE-21674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17339701#comment-17339701
]
Sean Busbey commented on HBASE-21674:
-
I can make time to look at it. My time availability is up and down ATM so don't
feel you need to block on me.
{quote}
The warning seems about JDK complier 1.7, is it ok to ignore? I'm not sure
about that.
{quote}
I'll see if we can suppress it if we're not going to act on it.
> Port HBASE-21652 (Refactor ThriftServer making thrift2 server inherited from
> thrift1 server) to branch-1
>
>
> Key: HBASE-21674
> URL: https://issues.apache.org/jira/browse/HBASE-21674
> Project: HBase
> Issue Type: Sub-task
>Reporter: Andrew Kyle Purtell
>Assignee: Yutong Xiao
>Priority: Major
> Fix For: 1.8.0
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-22120) Replace HTrace with OpenTelemetry
[ https://issues.apache.org/jira/browse/HBASE-22120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-22120: -- Release Note: In this issue we change our tracing system from HTrace to OpenTelemetry. The HTrace dependencies are banned(transitive dependencies are still allowed as hadoop still depends on it), the imports of htrace related classes are also banned. We add OpenTelemtry support for our RPC system, which means all the rpc methods will be traced on both client side and server side. Most methods in Table interface are also traced, except scan and coprocessor related methods. As now the scan implementation is always 'async prefetch', we haven't find a suitable way to represent this relationship between the foreground and background spans. At server side, due to the same reason, we only use a span to record the time of the WAL sync operation, without tracing into the background sync thread. On how to enable tracing, please read the Tracing section in our refguide. https://hbase.apache.org/book.html#tracing > Replace HTrace with OpenTelemetry > - > > Key: HBASE-22120 > URL: https://issues.apache.org/jira/browse/HBASE-22120 > Project: HBase > Issue Type: New Feature > Components: tracing >Affects Versions: 3.0.0-alpha-1 >Reporter: Sergey Shelukhin >Assignee: Duo Zhang >Priority: Major > Fix For: 3.0.0-alpha-1 > > > h2. Deprecate HTrace usage in HBase > * HBase 1.x (branch-1) > * Declare HTrace (htrace 3.x) deprecated in the user doc. > * HBase 2.x (branch-2) > * Declare HTrace deprecated in the user doc. Furthermore, state that it is > known not working. > * Either fix the trace context propagation bug in HBase 2.x, or backport > OpenTracing support from the master branch. I am inclined to the latter. > * HBase 3.x (master branch) > * Remove HTrace entirely. > * Add OpenTracing APIs. Potentially backport to HBase 2.4. > * Replace OpenTracing API with OpenTelemetry when the latter stabilizes. > h1. Milestones > # Doc -- deprecation notice > # Replace existing HTrace code with OpenTracing code in the master branch > (3.x) > # Java (a [poc|https://github.com/jojochuang/hbase/tree/HBASE-22120] is > currently under way) > # HBase shell and scripts (Ruby, shell script) > # Doc > # Add new trace instrumentation code for new features not instrumented by > the existing HTace code. > # Propagate the traces to other systems such as HDFS and MapReduce. > # Support other OpenTracing tracers. > Update > As OpenTracing has now been replaced by OpenTelemetry, the goal finally > becomes replacing HTrace with OpenTelemetry. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-22120) Replace HTrace with OpenTelemetry
[ https://issues.apache.org/jira/browse/HBASE-22120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-22120: -- Release Note: In this issue we change our tracing system from HTrace to OpenTelemetry. The HTrace dependencies are banned(transitive dependencies are still allowed as hadoop still depends on them), the imports of htrace related classes are also banned. We add OpenTelemtry support for our RPC system, which means all the rpc methods will be traced on both client side and server side. Most methods in Table interface are also traced, except scan and coprocessor related methods. As now the scan implementation is always 'async prefetch', we haven't find a suitable way to represent this relationship between the foreground and background spans. At server side, due to the same reason, we only use a span to record the time of the WAL sync operation, without tracing into the background sync thread. On how to enable tracing, please read the Tracing section in our refguide. https://hbase.apache.org/book.html#tracing was: In this issue we change our tracing system from HTrace to OpenTelemetry. The HTrace dependencies are banned(transitive dependencies are still allowed as hadoop still depends on it), the imports of htrace related classes are also banned. We add OpenTelemtry support for our RPC system, which means all the rpc methods will be traced on both client side and server side. Most methods in Table interface are also traced, except scan and coprocessor related methods. As now the scan implementation is always 'async prefetch', we haven't find a suitable way to represent this relationship between the foreground and background spans. At server side, due to the same reason, we only use a span to record the time of the WAL sync operation, without tracing into the background sync thread. On how to enable tracing, please read the Tracing section in our refguide. https://hbase.apache.org/book.html#tracing > Replace HTrace with OpenTelemetry > - > > Key: HBASE-22120 > URL: https://issues.apache.org/jira/browse/HBASE-22120 > Project: HBase > Issue Type: New Feature > Components: tracing >Affects Versions: 3.0.0-alpha-1 >Reporter: Sergey Shelukhin >Assignee: Duo Zhang >Priority: Major > Fix For: 3.0.0-alpha-1 > > > h2. Deprecate HTrace usage in HBase > * HBase 1.x (branch-1) > * Declare HTrace (htrace 3.x) deprecated in the user doc. > * HBase 2.x (branch-2) > * Declare HTrace deprecated in the user doc. Furthermore, state that it is > known not working. > * Either fix the trace context propagation bug in HBase 2.x, or backport > OpenTracing support from the master branch. I am inclined to the latter. > * HBase 3.x (master branch) > * Remove HTrace entirely. > * Add OpenTracing APIs. Potentially backport to HBase 2.4. > * Replace OpenTracing API with OpenTelemetry when the latter stabilizes. > h1. Milestones > # Doc -- deprecation notice > # Replace existing HTrace code with OpenTracing code in the master branch > (3.x) > # Java (a [poc|https://github.com/jojochuang/hbase/tree/HBASE-22120] is > currently under way) > # HBase shell and scripts (Ruby, shell script) > # Doc > # Add new trace instrumentation code for new features not instrumented by > the existing HTace code. > # Propagate the traces to other systems such as HDFS and MapReduce. > # Support other OpenTracing tracers. > Update > As OpenTracing has now been replaced by OpenTelemetry, the goal finally > becomes replacing HTrace with OpenTelemetry. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-22120) Replace HTrace with OpenTelemetry
[ https://issues.apache.org/jira/browse/HBASE-22120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-22120: -- Release Note: In this issue we change our tracing system from HTrace to OpenTelemetry. The HTrace dependencies are banned(transitive dependencies are still allowed as hadoop still depends on them), the imports of htrace related classes are also banned. We add OpenTelemtry support for our RPC system, which means all the rpc methods will be traced on both client side and server side. Most methods in Table interface are also traced, except scan and coprocessor related methods. As now the scan implementation is always 'async prefetch', we haven't find a suitable way to represent this relationship between the foreground and background spans yet. At server side, due to the same reason, we only use a span to record the time of the WAL sync operation, without tracing into the background sync thread. On how to enable tracing, please read the Tracing section in our refguide. https://hbase.apache.org/book.html#tracing was: In this issue we change our tracing system from HTrace to OpenTelemetry. The HTrace dependencies are banned(transitive dependencies are still allowed as hadoop still depends on them), the imports of htrace related classes are also banned. We add OpenTelemtry support for our RPC system, which means all the rpc methods will be traced on both client side and server side. Most methods in Table interface are also traced, except scan and coprocessor related methods. As now the scan implementation is always 'async prefetch', we haven't find a suitable way to represent this relationship between the foreground and background spans. At server side, due to the same reason, we only use a span to record the time of the WAL sync operation, without tracing into the background sync thread. On how to enable tracing, please read the Tracing section in our refguide. https://hbase.apache.org/book.html#tracing > Replace HTrace with OpenTelemetry > - > > Key: HBASE-22120 > URL: https://issues.apache.org/jira/browse/HBASE-22120 > Project: HBase > Issue Type: New Feature > Components: tracing >Affects Versions: 3.0.0-alpha-1 >Reporter: Sergey Shelukhin >Assignee: Duo Zhang >Priority: Major > Fix For: 3.0.0-alpha-1 > > > h2. Deprecate HTrace usage in HBase > * HBase 1.x (branch-1) > * Declare HTrace (htrace 3.x) deprecated in the user doc. > * HBase 2.x (branch-2) > * Declare HTrace deprecated in the user doc. Furthermore, state that it is > known not working. > * Either fix the trace context propagation bug in HBase 2.x, or backport > OpenTracing support from the master branch. I am inclined to the latter. > * HBase 3.x (master branch) > * Remove HTrace entirely. > * Add OpenTracing APIs. Potentially backport to HBase 2.4. > * Replace OpenTracing API with OpenTelemetry when the latter stabilizes. > h1. Milestones > # Doc -- deprecation notice > # Replace existing HTrace code with OpenTracing code in the master branch > (3.x) > # Java (a [poc|https://github.com/jojochuang/hbase/tree/HBASE-22120] is > currently under way) > # HBase shell and scripts (Ruby, shell script) > # Doc > # Add new trace instrumentation code for new features not instrumented by > the existing HTace code. > # Propagate the traces to other systems such as HDFS and MapReduce. > # Support other OpenTracing tracers. > Update > As OpenTracing has now been replaced by OpenTelemetry, the goal finally > becomes replacing HTrace with OpenTelemetry. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-22120) Replace HTrace with OpenTelemetry
[ https://issues.apache.org/jira/browse/HBASE-22120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-22120: -- Release Note: In this issue we change our tracing system from HTrace to OpenTelemetry. The HTrace dependencies are banned(transitive dependencies are still allowed as hadoop still depends on them), the imports of htrace related classes are also banned. We add OpenTelemtry support for our RPC system, which means all the rpc methods will be traced on both client side and server side. Most methods in Table interface are also traced, except scan and coprocessor related methods. As now the scan implementation is always 'async prefetch', we haven't find a suitable way to represent this relationship between the foreground and background spans yet. At server side, due to the same reason, we only use a span to record the time of the WAL sync operation, without tracing into the background sync thread. And we do not trace the next method of RegionScanner, as a scan rpc call may lead to thousands of RegionScanner.next calls, which could slow down the rpc call even when tracing is disabled. On how to enable tracing, please read the Tracing section in our refguide. https://hbase.apache.org/book.html#tracing was: In this issue we change our tracing system from HTrace to OpenTelemetry. The HTrace dependencies are banned(transitive dependencies are still allowed as hadoop still depends on them), the imports of htrace related classes are also banned. We add OpenTelemtry support for our RPC system, which means all the rpc methods will be traced on both client side and server side. Most methods in Table interface are also traced, except scan and coprocessor related methods. As now the scan implementation is always 'async prefetch', we haven't find a suitable way to represent this relationship between the foreground and background spans yet. At server side, due to the same reason, we only use a span to record the time of the WAL sync operation, without tracing into the background sync thread. On how to enable tracing, please read the Tracing section in our refguide. https://hbase.apache.org/book.html#tracing > Replace HTrace with OpenTelemetry > - > > Key: HBASE-22120 > URL: https://issues.apache.org/jira/browse/HBASE-22120 > Project: HBase > Issue Type: New Feature > Components: tracing >Affects Versions: 3.0.0-alpha-1 >Reporter: Sergey Shelukhin >Assignee: Duo Zhang >Priority: Major > Fix For: 3.0.0-alpha-1 > > > h2. Deprecate HTrace usage in HBase > * HBase 1.x (branch-1) > * Declare HTrace (htrace 3.x) deprecated in the user doc. > * HBase 2.x (branch-2) > * Declare HTrace deprecated in the user doc. Furthermore, state that it is > known not working. > * Either fix the trace context propagation bug in HBase 2.x, or backport > OpenTracing support from the master branch. I am inclined to the latter. > * HBase 3.x (master branch) > * Remove HTrace entirely. > * Add OpenTracing APIs. Potentially backport to HBase 2.4. > * Replace OpenTracing API with OpenTelemetry when the latter stabilizes. > h1. Milestones > # Doc -- deprecation notice > # Replace existing HTrace code with OpenTracing code in the master branch > (3.x) > # Java (a [poc|https://github.com/jojochuang/hbase/tree/HBASE-22120] is > currently under way) > # HBase shell and scripts (Ruby, shell script) > # Doc > # Add new trace instrumentation code for new features not instrumented by > the existing HTace code. > # Propagate the traces to other systems such as HDFS and MapReduce. > # Support other OpenTracing tracers. > Update > As OpenTracing has now been replaced by OpenTelemetry, the goal finally > becomes replacing HTrace with OpenTelemetry. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-25853) Backport HBASE-22120 to branch-2
Duo Zhang created HBASE-25853: - Summary: Backport HBASE-22120 to branch-2 Key: HBASE-25853 URL: https://issues.apache.org/jira/browse/HBASE-25853 Project: HBase Issue Type: Task Components: tracing Reporter: Duo Zhang Fix For: 2.5.0 And also add tracing support for sync client on branch-2. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache9 opened a new pull request #3233: HBASE-25851 Make LoadBalancer not extend Configurable interface
Apache9 opened a new pull request #3233: URL: https://github.com/apache/hbase/pull/3233 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] ndimiduk merged pull request #3226: HBASE-25843 move master http-related code into o.a.h.h.master.http
ndimiduk merged pull request #3226: URL: https://github.com/apache/hbase/pull/3226 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-25843) Move master http-related code into o.a.h.h.master.http
[ https://issues.apache.org/jira/browse/HBASE-25843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-25843: - Fix Version/s: 3.0.0-alpha-1 > Move master http-related code into o.a.h.h.master.http > -- > > Key: HBASE-25843 > URL: https://issues.apache.org/jira/browse/HBASE-25843 > Project: HBase > Issue Type: Task > Components: master >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk >Priority: Minor > Fix For: 3.0.0-alpha-1 > > > Minor refactor. Group code related to the Master's Info Page into a common > package. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] ndimiduk merged pull request #3229: Backport "HBASE-25843 move master http-related code into o.a.h.h.master.http" to branch-2
ndimiduk merged pull request #3229: URL: https://github.com/apache/hbase/pull/3229 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-25843) Move master http-related code into o.a.h.h.master.http
[ https://issues.apache.org/jira/browse/HBASE-25843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-25843: - Fix Version/s: 2.5.0 > Move master http-related code into o.a.h.h.master.http > -- > > Key: HBASE-25843 > URL: https://issues.apache.org/jira/browse/HBASE-25843 > Project: HBase > Issue Type: Task > Components: master >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk >Priority: Minor > Fix For: 3.0.0-alpha-1, 2.5.0 > > > Minor refactor. Group code related to the Master's Info Page into a common > package. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-25843) Move master http-related code into o.a.h.h.master.http
[ https://issues.apache.org/jira/browse/HBASE-25843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-25843: - Resolution: Fixed Status: Resolved (was: Patch Available) > Move master http-related code into o.a.h.h.master.http > -- > > Key: HBASE-25843 > URL: https://issues.apache.org/jira/browse/HBASE-25843 > Project: HBase > Issue Type: Task > Components: master >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk >Priority: Minor > Fix For: 3.0.0-alpha-1, 2.5.0 > > > Minor refactor. Group code related to the Master's Info Page into a common > package. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25846) Backport 'HBASE-25825 RSGroupBasedLoadBalancer.onConfigurationChange should chain the request to internal balancer' to branch-1
[
https://issues.apache.org/jira/browse/HBASE-25846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17339723#comment-17339723
]
Hudson commented on HBASE-25846:
Results for branch branch-1
[build #121 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-1/121/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-1/121//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk7 checks{color}
-- For more information [see jdk7
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-1/121//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-1/121//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 source release artifact{color}
-- See build output for details.
> Backport 'HBASE-25825 RSGroupBasedLoadBalancer.onConfigurationChange should
> chain the request to internal balancer' to branch-1
> ---
>
> Key: HBASE-25846
> URL: https://issues.apache.org/jira/browse/HBASE-25846
> Project: HBase
> Issue Type: Improvement
>Reporter: Caroline Zhou
>Assignee: Caroline Zhou
>Priority: Minor
> Fix For: 1.7.0
>
>
> In branch-1,
> [RSGroupBasedLoadBalancer#onConfigurationChange|https://github.com/apache/hbase/blob/branch-1/hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupBasedLoadBalancer.java#L452]
> doesn't do anything – it should call the internal balancer's
> onConfigurationChange().
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25825) RSGroupBasedLoadBalancer.onConfigurationChange should chain the request to internal balancer
[
https://issues.apache.org/jira/browse/HBASE-25825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17339724#comment-17339724
]
Hudson commented on HBASE-25825:
Results for branch branch-1
[build #121 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-1/121/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-1/121//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk7 checks{color}
-- For more information [see jdk7
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-1/121//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-1/121//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 source release artifact{color}
-- See build output for details.
> RSGroupBasedLoadBalancer.onConfigurationChange should chain the request to
> internal balancer
>
>
> Key: HBASE-25825
> URL: https://issues.apache.org/jira/browse/HBASE-25825
> Project: HBase
> Issue Type: Bug
> Components: Balancer
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.3, 2.3.6
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-21674) Port HBASE-21652 (Refactor ThriftServer making thrift2 server inherited from thrift1 server) to branch-1
[ https://issues.apache.org/jira/browse/HBASE-21674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17339725#comment-17339725 ] Reid Chan commented on HBASE-21674: --- Thanks Sean! I think the thrift issue, what I'm struggling recently, is quite a tough task to me. When you have time, could you take a look at this [thread|https://lists.apache.org/thread.html/r118b08134676d9234362a28898249186fe73a1fb08535d6eec6a91d3%40%3Cdev.hbase.apache.org%3E], kind of related. > Port HBASE-21652 (Refactor ThriftServer making thrift2 server inherited from > thrift1 server) to branch-1 > > > Key: HBASE-21674 > URL: https://issues.apache.org/jira/browse/HBASE-21674 > Project: HBase > Issue Type: Sub-task >Reporter: Andrew Kyle Purtell >Assignee: Yutong Xiao >Priority: Major > Fix For: 1.8.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25831) [branch-1] remove thrift examples out of hbase-examples module for bypassing the thrift version check
[ https://issues.apache.org/jira/browse/HBASE-25831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17339730#comment-17339730 ] Reid Chan commented on HBASE-25831: --- cc [~busbey], It's better if I could get your review&help as well. > [branch-1] remove thrift examples out of hbase-examples module for bypassing > the thrift version check > - > > Key: HBASE-25831 > URL: https://issues.apache.org/jira/browse/HBASE-25831 > Project: HBase > Issue Type: Task > Components: Thrift >Reporter: Reid Chan >Assignee: Reid Chan >Priority: Blocker > > [ERROR] Failed to execute goal on project hbase-examples: Could not resolve > dependencies for project org.apache.hbase:hbase-examples:jar:1.7.0: Could not > find artifact org.apache.hbase:hbase-thrift:jar:1.7.0 in apache release > (https://repository.apache.org/content/repositories/releases/) -> [Help 1] > This is the msg when I tried to run make_rc.sh, we need to remove thrift > related codes from hbase-examples for making release successfully. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #3233: HBASE-25851 Make LoadBalancer not extend Configurable interface
Apache-HBase commented on pull request #3233: URL: https://github.com/apache/hbase/pull/3233#issuecomment-832826680 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 11s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 20s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 12s | master passed | | +1 :green_heart: | compile | 3m 51s | master passed | | +1 :green_heart: | checkstyle | 1m 24s | master passed | | +1 :green_heart: | spotbugs | 2m 42s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 11s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 3s | the patch passed | | +1 :green_heart: | compile | 3m 43s | the patch passed | | +1 :green_heart: | javac | 3m 43s | the patch passed | | +1 :green_heart: | checkstyle | 1m 24s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 20m 0s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | spotbugs | 3m 5s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 23s | The patch does not generate ASF License warnings. | | | | 55m 11s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3233/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3233 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux 5be652c6806b 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 90f986497b | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Max. process+thread count | 86 (vs. ulimit of 3) | | modules | C: hbase-balancer hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3233/1/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] apurtell commented on pull request #3230: HBASE-25847 More DEBUG and TRACE level logging in CatalogJanitor and HbckChore
apurtell commented on pull request #3230: URL: https://github.com/apache/hbase/pull/3230#issuecomment-832841115 With these changes the ingestion test completes successfully and without weird artifacts in the logging. All split regions are GCed by procedures. In memory state aligns with filesystem state. /hbase/logs/hbase-apurtell-master-ip-172-31-58-47.us-west-2.compute.internal.log:130972:2021-05-05 15:47:39,190 INFO [master/ip-172-31-58-47:8100.Chore.1] master.HbckChore: Loaded 230 regions (0 disabled, 0 split parents) from in-memory state /hbase/logs/hbase-apurtell-master-ip-172-31-58-47.us-west-2.compute.internal.log:130973:2021-05-05 15:47:39,190 DEBUG [master/ip-172-31-58-47:8100.Chore.1] master.HbckChore: Regions by state: OPEN=230 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] apurtell edited a comment on pull request #3230: HBASE-25847 More DEBUG and TRACE level logging in CatalogJanitor and HbckChore
apurtell edited a comment on pull request #3230: URL: https://github.com/apache/hbase/pull/3230#issuecomment-832841115 With these changes the ingestion test completes successfully and without weird artifacts in the logging. All split regions are GCed by procedures. In memory state aligns with filesystem state. /hbase/logs/hbase-apurtell-master-ip-172-31-58-47.us-west-2.compute.internal.log:130972:2021-05-05 15:47:39,190 INFO [master/ip-172-31-58-47:8100.Chore.1] master.HbckChore: Loaded 230 regions (0 disabled, 0 split parents) from in-memory state /hbase/logs/hbase-apurtell-master-ip-172-31-58-47.us-west-2.compute.internal.log:130973:2021-05-05 15:47:39,190 DEBUG [master/ip-172-31-58-47:8100.Chore.1] master.HbckChore: Regions by state: OPEN=230 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] apurtell commented on pull request #3230: HBASE-25847 More DEBUG and TRACE level logging in CatalogJanitor and HbckChore
apurtell commented on pull request #3230: URL: https://github.com/apache/hbase/pull/3230#issuecomment-832843132 Well since what I want here is "More DEBUG and TRACE level logging" I will remove the additional changes so the test failure is fixed and address that on a separate PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (HBASE-25854) Remove redundant AM in-memory state changes in CatalogJanitor
Andrew Kyle Purtell created HBASE-25854: --- Summary: Remove redundant AM in-memory state changes in CatalogJanitor Key: HBASE-25854 URL: https://issues.apache.org/jira/browse/HBASE-25854 Project: HBase Issue Type: Sub-task Reporter: Andrew Kyle Purtell Assignee: Andrew Kyle Purtell Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.3 In CatalogJanitor we schedule GCRegionProcedure to clean up both filesystem and in-memory state after a split, and GCMultipleMergedRegionsProcedure to do the same for merges. Both GCRegionProcedure and GCMultipleMergedRegionsProcedure clean up in-memory state, but CatalogJanitor also does this redundantly just after scheduling the procedures. The cleanup should be done in only one place. Presumably we are using the procedures to do it in a principled way. Remove the redundancy in CatalogJanitor (and fix any follow on issues, like test failures). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-25847) More DEBUG and TRACE level logging in CatalogJanitor and HbckChore
[ https://issues.apache.org/jira/browse/HBASE-25847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-25847: Status: Patch Available (was: Open) > More DEBUG and TRACE level logging in CatalogJanitor and HbckChore > -- > > Key: HBASE-25847 > URL: https://issues.apache.org/jira/browse/HBASE-25847 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.4.3 >Reporter: Andrew Kyle Purtell >Assignee: Andrew Kyle Purtell >Priority: Minor > Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.3 > > > Add DEBUG and TRACE level logging to CatalogJanitor and HbckChore to > facilitate debugging and verification of correct operation. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-25854) Remove redundant AM in-memory state changes in CatalogJanitor
[ https://issues.apache.org/jira/browse/HBASE-25854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-25854: Status: Patch Available (was: Open) > Remove redundant AM in-memory state changes in CatalogJanitor > - > > Key: HBASE-25854 > URL: https://issues.apache.org/jira/browse/HBASE-25854 > Project: HBase > Issue Type: Sub-task >Reporter: Andrew Kyle Purtell >Assignee: Andrew Kyle Purtell >Priority: Major > Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.3 > > > In CatalogJanitor we schedule GCRegionProcedure to clean up both filesystem > and in-memory state after a split, and GCMultipleMergedRegionsProcedure to do > the same for merges. > Both GCRegionProcedure and GCMultipleMergedRegionsProcedure clean up > in-memory state, but CatalogJanitor also does this redundantly just after > scheduling the procedures. > The cleanup should be done in only one place. Presumably we are using the > procedures to do it in a principled way. Remove the redundancy in > CatalogJanitor (and fix any follow on issues, like test failures). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] apurtell opened a new pull request #3234: HBASE-25854 Remove redundant AM in-memory state changes in CatalogJanitor
apurtell opened a new pull request #3234: URL: https://github.com/apache/hbase/pull/3234 In CatalogJanitor we schedule GCRegionProcedure to clean up both filesystem and in-memory state after a split, and GCMultipleMergedRegionsProcedure to do the same for merges. Both of these procedures clean up in-memory state, but CatalogJanitor also does this redundantly just after scheduling the procedures. The cleanup should be done in only one place. Presumably we are using the procedures to do it in a principled way. Remove the redundancy in CatalogJanitor and fix any follow on issues, like test failures. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] apurtell commented on pull request #3230: HBASE-25847 More DEBUG and TRACE level logging in CatalogJanitor and HbckChore
apurtell commented on pull request #3230: URL: https://github.com/apache/hbase/pull/3230#issuecomment-832853177 Just the logging changes now, and the bug fix to ensure log lines emit correct information. The CatalogJanitor cleanup proposal is now this PR: https://github.com/apache/hbase/pull/3234 I have submitted it without a test change so we get the same test results there and can carry the discussion forward. Will work on a test change too. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] apurtell commented on pull request #3234: HBASE-25854 Remove redundant AM in-memory state changes in CatalogJanitor
apurtell commented on pull request #3234: URL: https://github.com/apache/hbase/pull/3234#issuecomment-832854650 This change is moved here from https://github.com/apache/hbase/pull/3230. We know from the test report there that `TestCatalogJanitorInMemoryStates` will fail. Let's let it fail here too and then proceed. java.lang.AssertionError: Parent region should have been removed from RegionStates at org.apache.hadoop.hbase.master.janitor.TestCatalogJanitorInMemoryStates.testInMemoryParentCleanup(TestCatalogJanitorInMemoryStates.java:123) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] apurtell commented on pull request #3234: HBASE-25854 Remove redundant AM in-memory state changes in CatalogJanitor
apurtell commented on pull request #3234: URL: https://github.com/apache/hbase/pull/3234#issuecomment-832855720 @Apache9 added some context on https://github.com/apache/hbase/pull/3230 > Some background > > https://issues.apache.org/jira/browse/HBASE-24942 > > I've found a bug when refactoring CatalogJanitor because of we may clean the merge qualifiers in MergeTableRegionsProcedure. Theoretically we should only do this in GCRegion related procedures(as what you proposed here), but doing this in other places could speed up later process which could be blocked by merge qualifiers. > > For me I'm +1 on removing the redundant removal in CatalogJanitor, but let's wait for @saintstack 's opinon too? > > Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] apurtell edited a comment on pull request #3230: HBASE-25847 More DEBUG and TRACE level logging in CatalogJanitor and HbckChore
apurtell edited a comment on pull request #3230: URL: https://github.com/apache/hbase/pull/3230#issuecomment-832853177 Just the logging changes now, and the bug fix to ensure log lines emit correct information. The CatalogJanitor cleanup proposal is now this PR: https://github.com/apache/hbase/pull/3234 I have submitted it without a test change so we get the same test results there and can carry the discussion forward. Will work on a test change too. @virajjasani The next test report here should be good. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] apurtell edited a comment on pull request #3230: HBASE-25847 More DEBUG and TRACE level logging in CatalogJanitor and HbckChore
apurtell edited a comment on pull request #3230: URL: https://github.com/apache/hbase/pull/3230#issuecomment-832853177 Just the logging changes now, and the bug fix to ensure log lines emit correct information. The CatalogJanitor cleanup proposal is now this PR: https://github.com/apache/hbase/pull/3234 I have submitted it without a test change so we get the same test results there and can carry the discussion forward. Will work on a test change too. @virajjasani The next test report will be clean. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] apurtell commented on pull request #3234: HBASE-25854 Remove redundant AM in-memory state changes in CatalogJanitor
apurtell commented on pull request #3234: URL: https://github.com/apache/hbase/pull/3234#issuecomment-832857609 Although the unit test will fail, when I tried this change out on a cluster in a very write heavy ingestion test, the end result was good. The ingestion test completes successfully and without weird artifacts in the logging. All split regions are GCed by procedures. In memory state aligns with filesystem state. /hbase/logs/hbase-apurtell-master-ip-172-31-58-47.us-west-2.compute.internal.log:130972:2021-05-05 15:47:39,190 INFO [master/ip-172-31-58-47:8100.Chore.1] master.HbckChore: Loaded 230 regions (0 disabled, 0 split parents) from in-memory state /hbase/logs/hbase-apurtell-master-ip-172-31-58-47.us-west-2.compute.internal.log:130973:2021-05-05 15:47:39,190 DEBUG [master/ip-172-31-58-47:8100.Chore.1] master.HbckChore: Regions by state: OPEN=230 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] apurtell edited a comment on pull request #3234: HBASE-25854 Remove redundant AM in-memory state changes in CatalogJanitor
apurtell edited a comment on pull request #3234: URL: https://github.com/apache/hbase/pull/3234#issuecomment-832857609 Although the unit test will fail, when I tried this change out on a cluster in a very write heavy ingestion test, the end result was good. The ingestion test completes successfully and without weird artifacts in the logging. All split regions are GCed by procedures. In memory state aligns with filesystem state. /hbase/logs/hbase-apurtell-master-ip-172-31-58-47.us-west-2.compute.internal.log:130972:2021-05-05 15:47:39,190 INFO [master/ip-172-31-58-47:8100.Chore.1] master.HbckChore: Loaded 230 regions (0 disabled, 0 split parents) from in-memory state /hbase/logs/hbase-apurtell-master-ip-172-31-58-47.us-west-2.compute.internal.log:130973:2021-05-05 15:47:39,190 DEBUG [master/ip-172-31-58-47:8100.Chore.1] master.HbckChore: Regions by state: OPEN=230 /hbase/logs/hbase-apurtell-master-ip-172-31-58-47.us-west-2.compute.internal.log:130974:2021-05-05 15:47:39,190 INFO [master/ip-172-31-58-47:8100.Chore.1] master.HbckChore: Loaded 230 regions from 5 regionservers' reports and found 0 orphan regions /hbase/logs/hbase-apurtell-master-ip-172-31-58-47.us-west-2.compute.internal.log:130986:2021-05-05 15:47:39,200 INFO [master/ip-172-31-58-47:8100.Chore.1] master.HbckChore: Loaded 3 tables 230 regions from filesyetem and found 0 orphan regions -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] apurtell edited a comment on pull request #3230: HBASE-25847 More DEBUG and TRACE level logging in CatalogJanitor and HbckChore
apurtell edited a comment on pull request #3230: URL: https://github.com/apache/hbase/pull/3230#issuecomment-832841115 With these changes the ingestion test completes successfully and without weird artifacts in the logging. All split regions are GCed by procedures. In memory state aligns with filesystem state. /hbase/logs/hbase-apurtell-master-ip-172-31-58-47.us-west-2.compute.internal.log:130972:2021-05-05 15:47:39,190 INFO [master/ip-172-31-58-47:8100.Chore.1] master.HbckChore: Loaded 230 regions (0 disabled, 0 split parents) from in-memory state /hbase/logs/hbase-apurtell-master-ip-172-31-58-47.us-west-2.compute.internal.log:130973:2021-05-05 15:47:39,190 DEBUG [master/ip-172-31-58-47:8100.Chore.1] master.HbckChore: Regions by state: OPEN=230 /hbase/logs/hbase-apurtell-master-ip-172-31-58-47.us-west-2.compute.internal.log:130974:2021-05-05 15:47:39,190 INFO [master/ip-172-31-58-47:8100.Chore.1] master.HbckChore: Loaded 230 regions from 5 regionservers' reports and found 0 orphan regions /hbase/logs/hbase-apurtell-master-ip-172-31-58-47.us-west-2.compute.internal.log:130986:2021-05-05 15:47:39,200 INFO [master/ip-172-31-58-47:8100.Chore.1] master.HbckChore: Loaded 3 tables 230 regions from filesyetem and found 0 orphan regions -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-25829) SPLIT state detritus
[
https://issues.apache.org/jira/browse/HBASE-25829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17339779#comment-17339779
]
Andrew Kyle Purtell commented on HBASE-25829:
-
Latest ingest test looks good.
Two subtasks remaining:
* HBASE-25847: More DEBUG and TRACE level logging in CatalogJanitor and
HbckChore for easier debugging.
* HBASE-25854: Remove redundant AM in-memory state changes in CatalogJanitor.
At least a nit, but probably a source of future bugs.
> SPLIT state detritus
>
>
> Key: HBASE-25829
> URL: https://issues.apache.org/jira/browse/HBASE-25829
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.4.3
>Reporter: Andrew Kyle Purtell
>Assignee: Andrew Kyle Purtell
>Priority: Minor
> Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.3
>
>
> Seen after an integration test (see HBASE-25824) with 'calm' monkey, so this
> happened in the happy path.
> There were no errors accessing all loaded table data. The integration test
> writes a log to HDFS of every cell written to HBase and the verify phase uses
> that log to read each value and confirm it. That seems fine:
> {noformat}
> 2021-04-30 02:16:33,316 INFO [main]
> test.IntegrationTestLoadCommonCrawl$Verify: REFERENCED: 154943544
> 2021-04-30 02:16:33,316 INFO [main]
> test.IntegrationTestLoadCommonCrawl$Verify: UNREFERENCED: 0
> 2021-04-30 02:16:33,316 INFO [main]
> test.IntegrationTestLoadCommonCrawl$Verify: CORRUPT: 0
> {noformat}
> However whenever the balancer runs there are a number of concerning INFO
> level log messages printed of the form _assignment.RegionStates: Skipping, no
> server for state=SPLIT, location=null, table=TABLENAME_
> For example:
> {noformat}
> 2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
> assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
> table=IntegrationTestLoadCommonCrawl, region=087fb2f7847c2fc0a0b85eb30a97036e
> 2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
> assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
> table=IntegrationTestLoadCommonCrawl, region=0952b94a920454afe9c40becbb7bf205
> 2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
> assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
> table=IntegrationTestLoadCommonCrawl, region=f87a8b993f7eca2524bf2331b7ee3c06
> 2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
> assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
> table=IntegrationTestLoadCommonCrawl, region=74bb28864a120decdf0f4956741df745
> 2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
> assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
> table=IntegrationTestLoadCommonCrawl, region=bc918b609ade0ae4d5530f0467354cae
> 2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
> assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
> table=IntegrationTestLoadCommonCrawl, region=183a199984539f3917a2f8927fe01572
> 2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
> assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
> table=IntegrationTestLoadCommonCrawl, region=6cc5ce4fb4adc00445b3ec7dd8760ba8
> {noformat}
> The HBCK chore notices them but does nothing:
> "Loaded *80 regions* from in-memory state of AssignmentManager"
> "Loaded *73 regions from 5 regionservers' reports* and found 0 orphan regions"
> "Loaded 3 tables 80 regions from filesystem and found 0 orphan regions"
> Yes, there are exactly 7 region state records of SPLIT state with
> server=null.
> {noformat}
> 2021-04-30 02:02:09,300 INFO [master/ip-172-31-58-47:8100.Chore.1]
> master.HbckChore: Loaded 80 regions from in-memory state of AssignmentManager
> 2021-04-30 02:02:09,300 INFO [master/ip-172-31-58-47:8100.Chore.1]
> master.HbckChore: Loaded 73 regions from 5 regionservers' reports and found 0
> orphan regions
> 2021-04-30 02:02:09,306 INFO [master/ip-172-31-58-47:8100.Chore.1]
> master.HbckChore: Loaded 3 tables 80 regions from filesystem and found 0
> orphan regions
> {noformat}
> This repeats indefinitely.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-25854) Remove redundant AM in-memory state changes in CatalogJanitor
[ https://issues.apache.org/jira/browse/HBASE-25854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-25854: Description: In CatalogJanitor we schedule GCRegionProcedure to clean up both filesystem and in-memory state after a split, and GCMultipleMergedRegionsProcedure to do the same for merges. Both GCRegionProcedure and GCMultipleMergedRegionsProcedure clean up in-memory state, but CatalogJanitor also does this redundantly just after scheduling the procedures. The cleanup should be done in only one place. Presumably we are using the procedures to do it in a principled way. This is least a nit, but probably a source of future bugs. Remove the redundancy in CatalogJanitor and fix any follow on issues, like test failures. was: In CatalogJanitor we schedule GCRegionProcedure to clean up both filesystem and in-memory state after a split, and GCMultipleMergedRegionsProcedure to do the same for merges. Both GCRegionProcedure and GCMultipleMergedRegionsProcedure clean up in-memory state, but CatalogJanitor also does this redundantly just after scheduling the procedures. The cleanup should be done in only one place. Presumably we are using the procedures to do it in a principled way. Remove the redundancy in CatalogJanitor (and fix any follow on issues, like test failures). > Remove redundant AM in-memory state changes in CatalogJanitor > - > > Key: HBASE-25854 > URL: https://issues.apache.org/jira/browse/HBASE-25854 > Project: HBase > Issue Type: Sub-task >Reporter: Andrew Kyle Purtell >Assignee: Andrew Kyle Purtell >Priority: Major > Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.3 > > > In CatalogJanitor we schedule GCRegionProcedure to clean up both filesystem > and in-memory state after a split, and GCMultipleMergedRegionsProcedure to do > the same for merges. > Both GCRegionProcedure and GCMultipleMergedRegionsProcedure clean up > in-memory state, but CatalogJanitor also does this redundantly just after > scheduling the procedures. > The cleanup should be done in only one place. Presumably we are using the > procedures to do it in a principled way. This is least a nit, but probably a > source of future bugs. Remove the redundancy in CatalogJanitor and fix any > follow on issues, like test failures. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HBASE-25853) Backport HBASE-22120 to branch-2
[ https://issues.apache.org/jira/browse/HBASE-25853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell reassigned HBASE-25853: --- Assignee: Andrew Kyle Purtell > Backport HBASE-22120 to branch-2 > > > Key: HBASE-25853 > URL: https://issues.apache.org/jira/browse/HBASE-25853 > Project: HBase > Issue Type: Task > Components: tracing >Reporter: Duo Zhang >Assignee: Andrew Kyle Purtell >Priority: Major > Fix For: 2.5.0 > > > And also add tracing support for sync client on branch-2. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-21674) Port HBASE-21652 (Refactor ThriftServer making thrift2 server inherited from thrift1 server) to branch-1
[ https://issues.apache.org/jira/browse/HBASE-21674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17339783#comment-17339783 ] Andrew Kyle Purtell commented on HBASE-21674: - bq. My time availability is up and down ATM so don't feel you need to block on me. Welcome back [~busbey] > Port HBASE-21652 (Refactor ThriftServer making thrift2 server inherited from > thrift1 server) to branch-1 > > > Key: HBASE-21674 > URL: https://issues.apache.org/jira/browse/HBASE-21674 > Project: HBase > Issue Type: Sub-task >Reporter: Andrew Kyle Purtell >Assignee: Yutong Xiao >Priority: Major > Fix For: 1.8.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25797) Batch version of checkAndMutate on AsyncTable always returns false
[
https://issues.apache.org/jira/browse/HBASE-25797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17339786#comment-17339786
]
Hochul Shin commented on HBASE-25797:
-
Thanks a lot Toshihiro.
It's my bad. I had to check the version. I will close this ticket and test with
higher version.
Thanks again.
> Batch version of checkAndMutate on AsyncTable always returns false
> --
>
> Key: HBASE-25797
> URL: https://issues.apache.org/jira/browse/HBASE-25797
> Project: HBase
> Issue Type: Bug
> Components: Client
>Affects Versions: 2.4.1
>Reporter: Hochul Shin
>Assignee: Toshihiro Suzuki
>Priority: Minor
> Labels: bug
>
> Batch version of CheckAndMutate always returns false whereas non-batch
> version works fine.
> The code is like:
>
> {code:java}
> AysncTable table = connection.getTable(tableName, executorService);
> ...
> // add r1
> table.put(Arrays.asList(
> new Put(Bytes.toBytes("r1")).addColumn(COL_FAMILY, Bytes.toBytes("q1"),
> Bytes.toBytes("v1";
> CheckAndMutate checkAndMutate1 =
> CheckAndMutate.newBuilder(Bytes.toBytes("r1"))
> .ifNotExists(COL_FAMILY, Bytes.toBytes("q1"))
> .build(new Put(Bytes.toBytes("r1")).addColumn(COL_FAMILY,
> Bytes.toBytes("q1"), Bytes.toBytes("v1")));
> CheckAndMutate checkAndMutate2 =
> CheckAndMutate.newBuilder(Bytes.toBytes("r2"))
> .ifNotExists(COL_FAMILY, Bytes.toBytes("q2"))
> .build(new Put(Bytes.toBytes("r2")).addColumn(COL_FAMILY,
> Bytes.toBytes("q2"), Bytes.toBytes("v2")));
> {code}
>
> With batch version of checkAndMutate
> {code:java}
> List> results =
> table.checkAndMutate(Arrays.asList(checkAndMutate1, checkAndMutate2));
> System.out.println("first: " + results.get(0).join().isSuccess()); // false
> System.out.println("second: " + results.get(1).join().isSuccess()); //false!
> {code}
>
>
> Even it returned false, the r1 was written to the table. (timestamp was
> updated)
>
> With non-batch version of checkAndMutate
>
> {code:java}
> CompletableFuture result1 =
> table.checkAndMutate(checkAndMutate1);
> CompletableFuture result2 =
> table.checkAndMutate(checkAndMutate2);
> System.out.println("first: " + result1.join().isSuccess()); // false
> System.out.println("second: " + result2.join().isSuccess()); // true as
> expected
> {code}
>
> As expected r1 wasn't written to the table.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25831) [branch-1] remove thrift examples out of hbase-examples module for bypassing the thrift version check
[ https://issues.apache.org/jira/browse/HBASE-25831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17339785#comment-17339785 ] Andrew Kyle Purtell commented on HBASE-25831: - bq. Not sure whether I should merge the commit, but it looks like the PR's modifications didn't take effect? It does look that way. The change builds successfully for you locally? Then merge the PR. If there is a problem later we can always back it out. That would be one way around precommit limitations here. > [branch-1] remove thrift examples out of hbase-examples module for bypassing > the thrift version check > - > > Key: HBASE-25831 > URL: https://issues.apache.org/jira/browse/HBASE-25831 > Project: HBase > Issue Type: Task > Components: Thrift >Reporter: Reid Chan >Assignee: Reid Chan >Priority: Blocker > > [ERROR] Failed to execute goal on project hbase-examples: Could not resolve > dependencies for project org.apache.hbase:hbase-examples:jar:1.7.0: Could not > find artifact org.apache.hbase:hbase-thrift:jar:1.7.0 in apache release > (https://repository.apache.org/content/repositories/releases/) -> [Help 1] > This is the msg when I tried to run make_rc.sh, we need to remove thrift > related codes from hbase-examples for making release successfully. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-25797) Batch version of checkAndMutate on AsyncTable always returns false
[
https://issues.apache.org/jira/browse/HBASE-25797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hochul Shin resolved HBASE-25797.
-
Resolution: Fixed
> Batch version of checkAndMutate on AsyncTable always returns false
> --
>
> Key: HBASE-25797
> URL: https://issues.apache.org/jira/browse/HBASE-25797
> Project: HBase
> Issue Type: Bug
> Components: Client
>Affects Versions: 2.4.1
>Reporter: Hochul Shin
>Assignee: Toshihiro Suzuki
>Priority: Minor
> Labels: bug
>
> Batch version of CheckAndMutate always returns false whereas non-batch
> version works fine.
> The code is like:
>
> {code:java}
> AysncTable table = connection.getTable(tableName, executorService);
> ...
> // add r1
> table.put(Arrays.asList(
> new Put(Bytes.toBytes("r1")).addColumn(COL_FAMILY, Bytes.toBytes("q1"),
> Bytes.toBytes("v1";
> CheckAndMutate checkAndMutate1 =
> CheckAndMutate.newBuilder(Bytes.toBytes("r1"))
> .ifNotExists(COL_FAMILY, Bytes.toBytes("q1"))
> .build(new Put(Bytes.toBytes("r1")).addColumn(COL_FAMILY,
> Bytes.toBytes("q1"), Bytes.toBytes("v1")));
> CheckAndMutate checkAndMutate2 =
> CheckAndMutate.newBuilder(Bytes.toBytes("r2"))
> .ifNotExists(COL_FAMILY, Bytes.toBytes("q2"))
> .build(new Put(Bytes.toBytes("r2")).addColumn(COL_FAMILY,
> Bytes.toBytes("q2"), Bytes.toBytes("v2")));
> {code}
>
> With batch version of checkAndMutate
> {code:java}
> List> results =
> table.checkAndMutate(Arrays.asList(checkAndMutate1, checkAndMutate2));
> System.out.println("first: " + results.get(0).join().isSuccess()); // false
> System.out.println("second: " + results.get(1).join().isSuccess()); //false!
> {code}
>
>
> Even it returned false, the r1 was written to the table. (timestamp was
> updated)
>
> With non-batch version of checkAndMutate
>
> {code:java}
> CompletableFuture result1 =
> table.checkAndMutate(checkAndMutate1);
> CompletableFuture result2 =
> table.checkAndMutate(checkAndMutate2);
> System.out.println("first: " + result1.join().isSuccess()); // false
> System.out.println("second: " + result2.join().isSuccess()); // true as
> expected
> {code}
>
> As expected r1 wasn't written to the table.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #3230: HBASE-25847 More DEBUG and TRACE level logging in CatalogJanitor and HbckChore
Apache-HBase commented on pull request #3230: URL: https://github.com/apache/hbase/pull/3230#issuecomment-832878005 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 5s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 34s | master passed | | +1 :green_heart: | compile | 3m 11s | master passed | | +1 :green_heart: | checkstyle | 1m 3s | master passed | | +1 :green_heart: | spotbugs | 2m 2s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 38s | the patch passed | | +1 :green_heart: | compile | 3m 8s | the patch passed | | +1 :green_heart: | javac | 3m 8s | the patch passed | | -0 :warning: | checkstyle | 1m 4s | hbase-server: The patch generated 1 new + 1 unchanged - 0 fixed = 2 total (was 1) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 17m 51s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | spotbugs | 2m 11s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 13s | The patch does not generate ASF License warnings. | | | | 46m 45s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3230/3/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3230 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux d37628d83ab8 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / eb9b54304e | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3230/3/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | Max. process+thread count | 95 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3230/3/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3234: HBASE-25854 Remove redundant AM in-memory state changes in CatalogJanitor
Apache-HBase commented on pull request #3234: URL: https://github.com/apache/hbase/pull/3234#issuecomment-832880760 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 40s | Docker mode activated. | | -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 46s | master passed | | +1 :green_heart: | compile | 1m 19s | master passed | | +1 :green_heart: | shadedjars | 8m 40s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 45s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 29s | the patch passed | | +1 :green_heart: | compile | 1m 12s | the patch passed | | +1 :green_heart: | javac | 1m 12s | the patch passed | | +1 :green_heart: | shadedjars | 8m 17s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 41s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 10m 11s | hbase-server in the patch failed. | | | | 42m 23s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3234/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3234 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 061f988fd6d0 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / eb9b54304e | | Default Java | AdoptOpenJDK-11.0.10+9 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3234/1/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3234/1/testReport/ | | Max. process+thread count | 762 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3234/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] bharathv commented on a change in pull request #3230: HBASE-25847 More DEBUG and TRACE level logging in CatalogJanitor and HbckChore
bharathv commented on a change in pull request #3230:
URL: https://github.com/apache/hbase/pull/3230#discussion_r626763927
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/janitor/CatalogJanitor.java
##
@@ -165,13 +168,19 @@ public int scan() throws IOException {
this.lastReport = scanForReport();
if (!this.lastReport.isEmpty()) {
LOG.warn(this.lastReport.toString());
+ } else {
+LOG.debug(this.lastReport.toString());
Review comment:
nit: toString() might be an expensive call if debug is not enabled,
might want to switch to a isDebugEnabled() guard, (later versions of log4j
supports log.debug(Object)/log.debug(Supplier) so that toString() is eventually
evaluated after the debug guard check)..
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3234: HBASE-25854 Remove redundant AM in-memory state changes in CatalogJanitor
Apache-HBase commented on pull request #3234: URL: https://github.com/apache/hbase/pull/3234#issuecomment-832883932 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 2s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 57s | master passed | | +1 :green_heart: | compile | 3m 8s | master passed | | +1 :green_heart: | checkstyle | 1m 4s | master passed | | +1 :green_heart: | spotbugs | 2m 4s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 41s | the patch passed | | +1 :green_heart: | compile | 3m 11s | the patch passed | | +1 :green_heart: | javac | 3m 11s | the patch passed | | +1 :green_heart: | checkstyle | 1m 2s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 18m 1s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | spotbugs | 2m 12s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 16s | The patch does not generate ASF License warnings. | | | | 47m 28s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3234/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3234 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux 29f65b080f67 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / eb9b54304e | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Max. process+thread count | 96 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3234/1/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] saintstack commented on a change in pull request #3230: HBASE-25847 More DEBUG and TRACE level logging in CatalogJanitor and HbckChore
saintstack commented on a change in pull request #3230:
URL: https://github.com/apache/hbase/pull/3230#discussion_r626778648
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/HbckChore.java
##
@@ -223,15 +223,37 @@ private void loadRegionsFromInMemoryState() {
.isTableState(regionInfo.getTable(), TableState.State.DISABLED)) {
disabledTableRegions.add(regionInfo.getRegionNameAsString());
}
- if (regionInfo.isSplitParent()) {
+ if (regionState.isSplit()) {
Review comment:
+1 on this change.
Was always odd have a Regions' split state recorded in its RegionInfo;
RegionInfo does not keep internally if it is being merged or if open, etc.
Regions are manipulated by other entities so state belongs elsewhere. +1 on
move to regionstate checking instead.
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/janitor/CatalogJanitor.java
##
@@ -165,13 +168,19 @@ public int scan() throws IOException {
this.lastReport = scanForReport();
if (!this.lastReport.isEmpty()) {
LOG.warn(this.lastReport.toString());
+ } else {
+LOG.debug(this.lastReport.toString());
Review comment:
+1
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/janitor/CatalogJanitor.java
##
@@ -165,13 +168,19 @@ public int scan() throws IOException {
this.lastReport = scanForReport();
if (!this.lastReport.isEmpty()) {
LOG.warn(this.lastReport.toString());
+ } else {
+LOG.debug(this.lastReport.toString());
}
+
updateAssignmentManagerMetrics();
Map mergedRegions = this.lastReport.mergedRegions;
for (Map.Entry e : mergedRegions.entrySet()) {
if (this.services.isInMaintenanceMode()) {
// Stop cleaning if the master is in maintenance mode
+ if (LOG.isDebugEnabled()) {
+LOG.debug("In maintenence mode, not cleaning");
+ }
Review comment:
We need the if (LOG.isDebug ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] wchevreuil commented on a change in pull request #3232: HBASE-25839 Bulk Import fails with java.io.IOException: Type mismatch in value from map
wchevreuil commented on a change in pull request #3232:
URL: https://github.com/apache/hbase/pull/3232#discussion_r626785659
##
File path:
hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/Import.java
##
@@ -135,9 +135,7 @@ public CellWritableComparable(Cell kv) {
@Override
public void write(DataOutput out) throws IOException {
- out.writeInt(PrivateCellUtil.estimatedSerializedSizeOfKey(kv));
- out.writeInt(0);
- PrivateCellUtil.writeFlatKey(kv, out);
+KeyValueUtil.write(new KeyValue(kv), out);
Review comment:
`PrivateCellUtil.writeFlatKey` writes the key value in a different
format from `KeyValueUtil.write`. This breaks compatibility, can we just keep
writing in the same format as before?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (HBASE-25834) Remove balanceTable method from LoadBalancer interface
[
https://issues.apache.org/jira/browse/HBASE-25834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17339815#comment-17339815
]
Hudson commented on HBASE-25834:
Results for branch branch-2
[build #242 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/242/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/242/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/242/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/242/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/242/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Remove balanceTable method from LoadBalancer interface
> --
>
> Key: HBASE-25834
> URL: https://issues.apache.org/jira/browse/HBASE-25834
> Project: HBase
> Issue Type: Sub-task
> Components: Balancer
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0
>
>
> The balanceTable method is never called outside balancer implementation, so
> let's remove it from the interface to reduce the number of public methods,
> which could also reduce the work of synchronization.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25842) Move regionserver http-related code into o.a.h.h.regionserver.http
[
https://issues.apache.org/jira/browse/HBASE-25842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17339814#comment-17339814
]
Hudson commented on HBASE-25842:
Results for branch branch-2
[build #242 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/242/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/242/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/242/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/242/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/242/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Move regionserver http-related code into o.a.h.h.regionserver.http
> --
>
> Key: HBASE-25842
> URL: https://issues.apache.org/jira/browse/HBASE-25842
> Project: HBase
> Issue Type: Task
> Components: regionserver
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
>Priority: Minor
> Fix For: 3.0.0-alpha-1, 2.5.0
>
>
> Minor refactor. Group code related to the Region Server's Info Page into a
> common package.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #3233: HBASE-25851 Make LoadBalancer not extend Configurable interface
Apache-HBase commented on pull request #3233: URL: https://github.com/apache/hbase/pull/3233#issuecomment-832910388 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 29s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 24s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 55s | master passed | | +1 :green_heart: | compile | 1m 20s | master passed | | +1 :green_heart: | shadedjars | 8m 13s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 55s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 17s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 36s | the patch passed | | +1 :green_heart: | compile | 1m 20s | the patch passed | | +1 :green_heart: | javac | 1m 20s | the patch passed | | +1 :green_heart: | shadedjars | 8m 9s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 53s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 29s | hbase-balancer in the patch passed. | | +1 :green_heart: | unit | 147m 55s | hbase-server in the patch passed. | | | | 180m 28s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3233/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3233 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux f8b3e0cfd2a7 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 90f986497b | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3233/1/testReport/ | | Max. process+thread count | 4253 (vs. ulimit of 3) | | modules | C: hbase-balancer hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3233/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] offermannu commented on a change in pull request #3232: HBASE-25839 Bulk Import fails with java.io.IOException: Type mismatch in value from map
offermannu commented on a change in pull request #3232:
URL: https://github.com/apache/hbase/pull/3232#discussion_r626807264
##
File path:
hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/Import.java
##
@@ -135,9 +135,7 @@ public CellWritableComparable(Cell kv) {
@Override
public void write(DataOutput out) throws IOException {
- out.writeInt(PrivateCellUtil.estimatedSerializedSizeOfKey(kv));
- out.writeInt(0);
- PrivateCellUtil.writeFlatKey(kv, out);
+KeyValueUtil.write(new KeyValue(kv), out);
Review comment:
Whatever `CellWritableComparable.write()`writes must be compatible with
`CellWritableComparable.readFields`
which is currently not the case (see [this exception in
HBASE-25839](https://issues.apache.org/jira/browse/HBASE-25839?focusedCommentId=17338869&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17338869).
The phrase "keyLength=0" inside the exception message comes actually from the
statement `out.writeIn(0)`in line 139).
My proposal aligns the "write" with the existing
`CellWritableComparable.readFields`. Theoretically one can adjust
`CellWritableComparable.readFields` so that it becomes compatible with the
current `write` method but this looks more complicated to me.
AFAICS the key is only used during the map sorting phase. The
[reducer](https://github.com/apache/hbase/blob/f76a601273e834267b55c0cda12474590283fd4c/hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/Import.java#L267)
doesn't care about the key at all.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] offermannu commented on a change in pull request #3232: HBASE-25839 Bulk Import fails with java.io.IOException: Type mismatch in value from map
offermannu commented on a change in pull request #3232:
URL: https://github.com/apache/hbase/pull/3232#discussion_r626807264
##
File path:
hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/Import.java
##
@@ -135,9 +135,7 @@ public CellWritableComparable(Cell kv) {
@Override
public void write(DataOutput out) throws IOException {
- out.writeInt(PrivateCellUtil.estimatedSerializedSizeOfKey(kv));
- out.writeInt(0);
- PrivateCellUtil.writeFlatKey(kv, out);
+KeyValueUtil.write(new KeyValue(kv), out);
Review comment:
Whatever `CellWritableComparable.write()`writes must be compatible with
`CellWritableComparable.readFields`
which is currently not the case (see [this exception in
HBASE-25839](https://issues.apache.org/jira/browse/HBASE-25839?focusedCommentId=17338869&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17338869).
The phrase "keyLength=0" inside the exception message comes actually from the
statement `out.writeIn(0)`in line 139).
My proposal aligns the "write" with the existing
`CellWritableComparable.readFields`. Theoretically one can adjust
`CellWritableComparable.readFields` so that it becomes compatible with the
current `write` method but this looks more complicated to me.
AFAICS the key is only used during the mapper sorting phase. The
[reducer](https://github.com/apache/hbase/blob/f76a601273e834267b55c0cda12474590283fd4c/hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/Import.java#L267)
doesn't care about the key at all.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] offermannu commented on a change in pull request #3232: HBASE-25839 Bulk Import fails with java.io.IOException: Type mismatch in value from map
offermannu commented on a change in pull request #3232:
URL: https://github.com/apache/hbase/pull/3232#discussion_r626807264
##
File path:
hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/Import.java
##
@@ -135,9 +135,7 @@ public CellWritableComparable(Cell kv) {
@Override
public void write(DataOutput out) throws IOException {
- out.writeInt(PrivateCellUtil.estimatedSerializedSizeOfKey(kv));
- out.writeInt(0);
- PrivateCellUtil.writeFlatKey(kv, out);
+KeyValueUtil.write(new KeyValue(kv), out);
Review comment:
Whatever `CellWritableComparable.write()`writes must be compatible with
`CellWritableComparable.readFields`
which is currently not the case (see [this exception in
HBASE-25839](https://issues.apache.org/jira/browse/HBASE-25839?focusedCommentId=17338869&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17338869).
The phrase "keyLength=0" inside the exception message comes actually from the
statement `out.writeIn(0)`in line 139).
My proposal aligns the "write" with the existing
`CellWritableComparable.readFields`. Theoretically one can adjust
`CellWritableComparable.readFields` so that it becomes compatible with the
current `write` method but this looks more complicated to me.
AFAICS the key is only used during the mapper sorting phase (see
stacktrace). The
[reducer](https://github.com/apache/hbase/blob/f76a601273e834267b55c0cda12474590283fd4c/hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/Import.java#L267)
doesn't care about the key at all.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3233: HBASE-25851 Make LoadBalancer not extend Configurable interface
Apache-HBase commented on pull request #3233: URL: https://github.com/apache/hbase/pull/3233#issuecomment-832950290 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 9s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 22s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 56s | master passed | | +1 :green_heart: | compile | 1m 37s | master passed | | +1 :green_heart: | shadedjars | 8m 53s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 59s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 42s | the patch passed | | +1 :green_heart: | compile | 1m 38s | the patch passed | | +1 :green_heart: | javac | 1m 38s | the patch passed | | +1 :green_heart: | shadedjars | 8m 58s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 58s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 30s | hbase-balancer in the patch passed. | | +1 :green_heart: | unit | 205m 52s | hbase-server in the patch passed. | | | | 242m 56s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3233/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3233 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 89f405c113f6 4.15.0-128-generic #131-Ubuntu SMP Wed Dec 9 06:57:35 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 90f986497b | | Default Java | AdoptOpenJDK-11.0.10+9 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3233/1/testReport/ | | Max. process+thread count | 3407 (vs. ulimit of 3) | | modules | C: hbase-balancer hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3233/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3230: HBASE-25847 More DEBUG and TRACE level logging in CatalogJanitor and HbckChore
Apache-HBase commented on pull request #3230: URL: https://github.com/apache/hbase/pull/3230#issuecomment-832956987 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 29s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 31s | master passed | | +1 :green_heart: | compile | 1m 12s | master passed | | +1 :green_heart: | shadedjars | 8m 14s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 42s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 17s | the patch passed | | +1 :green_heart: | compile | 1m 12s | the patch passed | | +1 :green_heart: | javac | 1m 12s | the patch passed | | +1 :green_heart: | shadedjars | 8m 6s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 38s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 140m 8s | hbase-server in the patch passed. | | | | 171m 48s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3230/3/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3230 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 30a6e24d4786 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / eb9b54304e | | Default Java | AdoptOpenJDK-11.0.10+9 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3230/3/testReport/ | | Max. process+thread count | 4161 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3230/3/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (HBASE-25855) Fix typo in jersey relocation path
Nick Dimiduk created HBASE-25855: Summary: Fix typo in jersey relocation path Key: HBASE-25855 URL: https://issues.apache.org/jira/browse/HBASE-25855 Project: HBase Issue Type: Task Components: hbase-thirdparty Affects Versions: thirdparty-3.4.1 Reporter: Nick Dimiduk Assignee: Nick Dimiduk We shade to "jersery" instead of "jersey". -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase-thirdparty] ndimiduk opened a new pull request #50: HBASE-25855 Fix typo in jersey relocation path
ndimiduk opened a new pull request #50: URL: https://github.com/apache/hbase-thirdparty/pull/50 We shade to "jersery" instead of "jersey". -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase-thirdparty] ndimiduk commented on pull request #50: HBASE-25855 Fix typo in jersey relocation path
ndimiduk commented on pull request #50: URL: https://github.com/apache/hbase-thirdparty/pull/50#issuecomment-832967792 There's probably an hbase-connectors PR that accompanies this, fixing its use in the REST gateway. We don't currently seem to use jersey in hbase core. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase-thirdparty] ndimiduk commented on pull request #50: HBASE-25855 Fix typo in jersey relocation path
ndimiduk commented on pull request #50: URL: https://github.com/apache/hbase-thirdparty/pull/50#issuecomment-832970078 Never mind, hbase-rest hasn't moved yet. Nothing uses this yet? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-25855) Fix typo in jersey relocation path
[ https://issues.apache.org/jira/browse/HBASE-25855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-25855: - Status: Patch Available (was: Open) > Fix typo in jersey relocation path > -- > > Key: HBASE-25855 > URL: https://issues.apache.org/jira/browse/HBASE-25855 > Project: HBase > Issue Type: Task > Components: hbase-thirdparty >Affects Versions: thirdparty-3.4.1 >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk >Priority: Minor > > We shade to "jersery" instead of "jersey". -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase-thirdparty] Apache-HBase commented on pull request #50: HBASE-25855 Fix typo in jersey relocation path
Apache-HBase commented on pull request #50: URL: https://github.com/apache/hbase-thirdparty/pull/50#issuecomment-832982426 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 10s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | -0 :warning: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 0m 44s | master passed | | +1 :green_heart: | compile | 0m 5s | master passed | | +1 :green_heart: | javadoc | 0m 9s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 0m 9s | the patch passed | | +1 :green_heart: | compile | 0m 5s | the patch passed | | +1 :green_heart: | javac | 0m 5s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. | | +1 :green_heart: | javadoc | 0m 4s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 5s | hbase-shaded-jersey in the patch passed. | | +1 :green_heart: | asflicense | 0m 7s | The patch does not generate ASF License warnings. | | | | 2m 47s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-Thirdparty-PreCommit/job/PR-50/1/artifact/yetus-precommit-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase-thirdparty/pull/50 | | Optional Tests | dupname asflicense javac javadoc unit xml compile | | uname | Linux 6a62a205206e 5.4.0-1025-aws #25~18.04.1-Ubuntu SMP Fri Sep 11 12:03:04 UTC 2020 x86_64 GNU/Linux | | Build tool | maven | | git revision | master / ccc49e6 | | Default Java | Oracle Corporation-1.8.0_282-b08 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-Thirdparty-PreCommit/job/PR-50/1/testReport/ | | Max. process+thread count | 39 (vs. ulimit of 1000) | | modules | C: hbase-shaded-jersey U: hbase-shaded-jersey | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-Thirdparty-PreCommit/job/PR-50/1/console | | versions | git=2.20.1 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (HBASE-25856) Vulnerabilities found when serializing enum value
junwen yang created HBASE-25856: --- Summary: Vulnerabilities found when serializing enum value Key: HBASE-25856 URL: https://issues.apache.org/jira/browse/HBASE-25856 Project: HBase Issue Type: Improvement Reporter: junwen yang In hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/HdfsServerConstants.java, the way to serialize the enum ReplicaState is to write the ordinal to the output stream, which makes the serialization and deserialization vulnerable to the order of the values. If later in the new version, a value is added for the enum ReplicaState, then the cross-version interaction will cause problems, similar to HDFS-15624. An improvement is to either add comments to inform later developers not to change this enum, or add checking when trying to deserialize the enum to check whether the index is not out of index. -- This message was sent by Atlassian Jira (v8.3.4#803005)
