[GitHub] [hbase] Apache-HBase commented on pull request #3336: HBASE-25920 Support Hadoop 3.3.1
Apache-HBase commented on pull request #3336: URL: https://github.com/apache/hbase/pull/3336#issuecomment-855343208 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 7m 16s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 33s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 10s | master passed | | +1 :green_heart: | compile | 2m 48s | master passed | | +1 :green_heart: | shadedjars | 9m 6s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 3m 31s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 18s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 12s | the patch passed | | +1 :green_heart: | compile | 2m 57s | the patch passed | | +1 :green_heart: | javac | 2m 57s | the patch passed | | +1 :green_heart: | shadedjars | 9m 50s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 3m 35s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 374m 52s | root in the patch failed. | | | | 425m 17s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3336/7/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3336 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux cf163f2f735b 4.15.0-128-generic #131-Ubuntu SMP Wed Dec 9 06:57:35 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / de06e20e0a | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3336/7/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-root.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3336/7/testReport/ | | Max. process+thread count | 3578 (vs. ulimit of 3) | | modules | C: hbase-common hbase-client hbase-server . U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3336/7/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-20413) IntegrationTestRegionReplicaReplication fails
[ https://issues.apache.org/jira/browse/HBASE-20413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17358019#comment-17358019 ] Michael Stack commented on HBASE-20413: --- Unscheduling fix version. Probably still an issue but haven't been here in a while. > IntegrationTestRegionReplicaReplication fails > - > > Key: HBASE-20413 > URL: https://issues.apache.org/jira/browse/HBASE-20413 > Project: HBase > Issue Type: Bug > Components: test >Reporter: Michael Stack >Assignee: Sreeram Venkatasubramanian >Priority: Critical > > Following [~psomogyi]'s inspiration, I've been running unit tests on a GCE > instance. IntegrationTestRegionReplicaReplication fails always for > branch-2.0. It fails for branch-1 too but for a different looking reason. On > branch-2.0, its OOME'ing. Downing the size of ingested data, it seems to hang > unable to update because it is unable to flush. > Marking this a blocker for now. Fix or disable it for branch-2.0. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-20413) IntegrationTestRegionReplicaReplication fails
[ https://issues.apache.org/jira/browse/HBASE-20413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Stack updated HBASE-20413: -- Fix Version/s: (was: 3.0.0-alpha-1) > IntegrationTestRegionReplicaReplication fails > - > > Key: HBASE-20413 > URL: https://issues.apache.org/jira/browse/HBASE-20413 > Project: HBase > Issue Type: Bug > Components: test >Reporter: Michael Stack >Assignee: Sreeram Venkatasubramanian >Priority: Critical > > Following [~psomogyi]'s inspiration, I've been running unit tests on a GCE > instance. IntegrationTestRegionReplicaReplication fails always for > branch-2.0. It fails for branch-1 too but for a different looking reason. On > branch-2.0, its OOME'ing. Downing the size of ingested data, it seems to hang > unable to update because it is unable to flush. > Marking this a blocker for now. Fix or disable it for branch-2.0. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-18562) [AMv2] expireServers and ServerCrashProcedure cleanup
[ https://issues.apache.org/jira/browse/HBASE-18562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Stack resolved HBASE-18562. --- Resolution: Not A Problem Agree. Resolving as 'Not a problem' since so much has changed since. > [AMv2] expireServers and ServerCrashProcedure cleanup > - > > Key: HBASE-18562 > URL: https://issues.apache.org/jira/browse/HBASE-18562 > Project: HBase > Issue Type: Bug > Components: Region Assignment >Reporter: Michael Stack >Priority: Critical > > In review of HBASE-18551, [~uagashe] posed a scenario that revealed a hole in > our processing of unassigns; there is case where a UP might not get > notification from ServerCrashProcedure if the UP is scheduled AFTER a SCP has > gotten past its handleRIT call (No new SCP will be queued because > expireServer won't let it happen if crashed server is in dead server list > which it will be). > Chatting on it, expireServers is doing checks that belong inside > ServerCrashProcedure. expireServers scheduling an SCP each time it is called > would make it so SCP processing is serialized one behind the other. If the > first does the clean up all subsequent will do no work but Procedures > dependent on them will get their wakeup call. > This issue is about implementing the above cleanup. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-19562) Purge mirror writing of region and table info into fs at .tableinfo and .regioninfo
[ https://issues.apache.org/jira/browse/HBASE-19562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17358017#comment-17358017 ] Michael Stack commented on HBASE-19562: --- None. I undid fix version. > Purge mirror writing of region and table info into fs at .tableinfo and > .regioninfo > --- > > Key: HBASE-19562 > URL: https://issues.apache.org/jira/browse/HBASE-19562 > Project: HBase > Issue Type: Bug > Components: Filesystem Integration >Reporter: Michael Stack >Assignee: Michael Stack >Priority: Critical > Attachments: > 0002-HBASE-19562-Purge-mirror-writing-of-region-and-table.patch > > > We don't use these files in hbase2 yet we keep writing them when we create a > table or region. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-19562) Purge mirror writing of region and table info into fs at .tableinfo and .regioninfo
[ https://issues.apache.org/jira/browse/HBASE-19562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Stack updated HBASE-19562: -- Fix Version/s: (was: 3.0.0-alpha-1) > Purge mirror writing of region and table info into fs at .tableinfo and > .regioninfo > --- > > Key: HBASE-19562 > URL: https://issues.apache.org/jira/browse/HBASE-19562 > Project: HBase > Issue Type: Bug > Components: Filesystem Integration >Reporter: Michael Stack >Assignee: Michael Stack >Priority: Critical > Attachments: > 0002-HBASE-19562-Purge-mirror-writing-of-region-and-table.patch > > > We don't use these files in hbase2 yet we keep writing them when we create a > table or region. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-20179) [TESTING] Scale for 2.0.0
[ https://issues.apache.org/jira/browse/HBASE-20179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Stack resolved HBASE-20179. --- Fix Version/s: (was: 3.0.0-alpha-1) 2.0.0 Assignee: Michael Stack Resolution: Fixed Finished task. Set fix version as 2.0.0 though it released long ago. > [TESTING] Scale for 2.0.0 > - > > Key: HBASE-20179 > URL: https://issues.apache.org/jira/browse/HBASE-20179 > Project: HBase > Issue Type: Umbrella > Components: test >Reporter: Michael Stack >Assignee: Michael Stack >Priority: Critical > Fix For: 2.0.0 > > > Umbrella issue for scale testing for 2.0. > At least keep account of what testing has been done in here. > TODO as subtasks > * Long-running test > * ITBLL w/o killing Master > * Big ITBLL (1B works, 10B needs more verification, 100B, and 1T). > * Many Regions (43k regions takes about 12 minutes on a 6node cluster to > deploy and 3 1/2 minutes to go down -- needs tuning). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-20179) [TESTING] Scale for 2.0.0
[ https://issues.apache.org/jira/browse/HBASE-20179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Stack updated HBASE-20179: -- Summary: [TESTING] Scale for 2.0.0 (was: [TESTING] Scale) > [TESTING] Scale for 2.0.0 > - > > Key: HBASE-20179 > URL: https://issues.apache.org/jira/browse/HBASE-20179 > Project: HBase > Issue Type: Umbrella > Components: test >Reporter: Michael Stack >Priority: Critical > Fix For: 3.0.0-alpha-1 > > > Umbrella issue for scale testing for 2.0. > At least keep account of what testing has been done in here. > TODO as subtasks > * Long-running test > * ITBLL w/o killing Master > * Big ITBLL (1B works, 10B needs more verification, 100B, and 1T). > * Many Regions (43k regions takes about 12 minutes on a 6node cluster to > deploy and 3 1/2 minutes to go down -- needs tuning). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #3359: HBASE-25891 remove dependence storing wal filenames for backup
Apache-HBase commented on pull request #3359: URL: https://github.com/apache/hbase/pull/3359#issuecomment-855328228 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 58s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +0 :ok: | prototool | 0m 0s | prototool was not available. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 32s | Maven dependency ordering for branch | | -1 :x: | mvninstall | 5m 20s | root in master failed. | | +1 :green_heart: | compile | 2m 28s | master passed | | +1 :green_heart: | checkstyle | 0m 25s | master passed | | +1 :green_heart: | spotbugs | 5m 20s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 21s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 46s | the patch passed | | +1 :green_heart: | compile | 2m 23s | the patch passed | | +1 :green_heart: | cc | 2m 23s | the patch passed | | +1 :green_heart: | javac | 1m 40s | hbase-protocol-shaded in the patch passed. | | +1 :green_heart: | javac | 0m 43s | hbase-backup generated 0 new + 73 unchanged - 6 fixed = 73 total (was 79) | | +1 :green_heart: | checkstyle | 0m 27s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 24m 41s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | hbaseprotoc | 1m 29s | the patch passed | | +1 :green_heart: | spotbugs | 5m 14s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 21s | The patch does not generate ASF License warnings. | | | | 65m 57s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3359/2/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3359 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile cc hbaseprotoc prototool | | uname | Linux 787395b70867 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / de06e20e0a | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | mvninstall | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3359/2/artifact/yetus-general-check/output/branch-mvninstall-root.txt | | Max. process+thread count | 86 (vs. ulimit of 3) | | modules | C: hbase-protocol-shaded hbase-backup U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3359/2/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3359: HBASE-25891 remove dependence storing wal filenames for backup
Apache-HBase commented on pull request #3359: URL: https://github.com/apache/hbase/pull/3359#issuecomment-855326582 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 28s | Docker mode activated. | | -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 27s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 33s | master passed | | +1 :green_heart: | compile | 1m 27s | master passed | | +1 :green_heart: | shadedjars | 8m 15s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 37s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 17s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 15s | the patch passed | | +1 :green_heart: | compile | 1m 25s | the patch passed | | +1 :green_heart: | javac | 1m 25s | the patch passed | | +1 :green_heart: | shadedjars | 8m 11s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 34s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 0s | hbase-protocol-shaded in the patch passed. | | +1 :green_heart: | unit | 9m 48s | hbase-backup in the patch passed. | | | | 42m 49s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3359/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3359 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux eff342b1ac18 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / de06e20e0a | | Default Java | AdoptOpenJDK-11.0.10+9 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3359/2/testReport/ | | Max. process+thread count | 4829 (vs. ulimit of 3) | | modules | C: hbase-protocol-shaded hbase-backup U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3359/2/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3359: HBASE-25891 remove dependence storing wal filenames for backup
Apache-HBase commented on pull request #3359: URL: https://github.com/apache/hbase/pull/3359#issuecomment-855326553 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 28s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 23s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 6s | master passed | | +1 :green_heart: | compile | 1m 14s | master passed | | +1 :green_heart: | shadedjars | 8m 54s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 33s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 39s | the patch passed | | +1 :green_heart: | compile | 1m 10s | the patch passed | | +1 :green_heart: | javac | 1m 10s | the patch passed | | +1 :green_heart: | shadedjars | 8m 46s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 32s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 48s | hbase-protocol-shaded in the patch passed. | | +1 :green_heart: | unit | 9m 56s | hbase-backup in the patch passed. | | | | 42m 14s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3359/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3359 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux a105f4796664 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / de06e20e0a | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3359/2/testReport/ | | Max. process+thread count | 5051 (vs. ulimit of 3) | | modules | C: hbase-protocol-shaded hbase-backup U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3359/2/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3336: HBASE-25920 Support Hadoop 3.3.1
Apache-HBase commented on pull request #3336: URL: https://github.com/apache/hbase/pull/3336#issuecomment-855326307 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 27s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 32s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 12s | master passed | | +1 :green_heart: | compile | 3m 4s | master passed | | +1 :green_heart: | shadedjars | 8m 17s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 4m 16s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 21s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 9s | the patch passed | | +1 :green_heart: | compile | 3m 2s | the patch passed | | +1 :green_heart: | javac | 3m 2s | the patch passed | | +1 :green_heart: | shadedjars | 8m 14s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 4m 11s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 177m 32s | root in the patch passed. | | | | 221m 53s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3336/7/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3336 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 898402870ecd 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / de06e20e0a | | Default Java | AdoptOpenJDK-11.0.10+9 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3336/7/testReport/ | | Max. process+thread count | 6641 (vs. ulimit of 3) | | modules | C: hbase-common hbase-client hbase-server . U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3336/7/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-25891) Remove dependence storing WAL filenames for backup
[
https://issues.apache.org/jira/browse/HBASE-25891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17358007#comment-17358007
]
Mallikarjun commented on HBASE-25891:
-
I have updated the description above. Hopefully it can answer your questions
[~anoop.hbase]. Adding details specific to questions here.
{quote}Means the WAL files will get renamed with this prefix? When those files
become eligible for deletion then?
{quote}
No. They are cleaned up by cleanup chore. Similar to `TimeToLiveLogCleaner`
{quote}Now that we dont have this systen table at all, what happens when taking
a full/incremental snapshot?
{quote}
Full backup does snapshot and export. There is no dependence on WAL files.
Incremental backup continues to check `rslogts:` to see which regionserver was
backed up until what timestamp and based on which WAL files are generated to be
backed up.
{quote}How WAL files been retained when backup refers to it? When that become
eligible for deletion? (Backup deleted/ another full backup came?) And how we
make sure we allow WAL deletion then?
{quote}
We don't need to store list of WAL files for that. We have checkpoints until
what point WAL's are read for backup and all those WAL files created beyond
that timestamp are eligable for backup automatically. and those created before
that timestamp can be cleaned up.
> Remove dependence storing WAL filenames for backup
> --
>
> Key: HBASE-25891
> URL: https://issues.apache.org/jira/browse/HBASE-25891
> Project: HBase
> Issue Type: Improvement
> Components: backuprestore
>Affects Versions: 3.0.0-alpha-1
>Reporter: Mallikarjun
>Assignee: Mallikarjun
>Priority: Major
> Fix For: 3.0.0-alpha-1
>
>
> Context:
> Currently WAL logs are stored in `backup:system` meta table
> {code:java}
> // code placeholder
> wals:preprod-dn-1%2C16020%2C1614844389000.1621996160175 column=meta:backupId,
> timestamp=1622003479895, value=backup_1622003358258
> wals:preprod-dn-1%2C16020%2C1614844389000.1621996160175 column=meta:file,
> timestamp=1622003479895,
> value=hdfs://store/hbase/oldWALs/preprod-dn-1%2C16020%2C1614844389000.1621996160175
> wals:preprod-dn-1%2C16020%2C1614844389000.1621996160175 column=meta:root,
> timestamp=1622003479895, value=s3a://2021-05-25--21-45-00--full/set1
> wals:preprod-dn-1%2C16020%2C1614844389000.1621999760280 column=meta:backupId,
> timestamp=1622003479895, value=backup_1622003358258
> wals:preprod-dn-1%2C16020%2C1614844389000.1621999760280 column=meta:file,
> timestamp=1622003479895,
> value=hdfs://store/hbase/oldWALs/preprod-dn-1%2C16020%2C1614844389000.1621999760280
> wals:preprod-dn-1%2C16020%2C1614844389000.1621999760280 column=meta:root,
> timestamp=1622003479895, value=s3a://2021-05-25--21-45-00--full/set1
> {code}
> Also, Every backup (Incremental and Full) performs a log roll just before
> taking backup and stores what was the timestamp at which log roll was
> performed per regionserver per backup using following format.
>
> {code:java}
> // code placeholder
> rslogts:hdfs://xx.xx.xx.xx:8020/tmp/backup_yaktest\x00preprod-dn-2:16020
> column=meta:rs-log-ts, timestamp=1622887363301,value=\x00\x00\x01y\xDB\x81ar
> rslogts:hdfs://xx.xx.xx.xx:8020/tmp/backup_yaktest\x00preprod-dn-3:16020
> column=meta:rs-log-ts, timestamp=1622887363294, value=\x00\x00\x01y\xDB\x81aP
> rslogts:hdfs://xx.xx.xx.xx:8020/tmp/backup_yaktest\x00preprod-dn-1:16020
> column=meta:rs-log-ts, timestamp=1622887363275,
> value=\x00\x00\x01y\xDB\x81\x85
> {code}
>
>
> There are 2 cases for which WAL log refrences stored in `backup:system` and
> are being used.
> 1. To cleanup WAL's for which backup is already taken using
> `BackupLogCleaner`
> Since log roll timestamp is stored as part of backup per regionserver. We can
> check all previous successfull backup's and then identify which logs are to
> be retained and which ones are to be cleaned up as follows
> * Identify which are the latest successful backups performed per table.
> * Per backup identified above, identify what is the oldest log rolled
> timestamp perfomed per regionserver per table.
> * All those WAL's which are older than oldest log rolled timestamp perfomed
> for any table backed can be removed by `BackupLogCleaner`
>
> 2. During incremental backup, to check system table if there are any
> duplicate WAL's for which backup is taken again.
> * Incremental backup already identifies which all WAL's to be backed up
> using `rslogts:` mentioned above.
> * Additionally it checks `wals:` to ensure no logs are backuped for second
> time. And this is redundant and not seen any extra benefit.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #3360: HBASE-25975 Row Commit Sequencer
Apache-HBase commented on pull request #3360: URL: https://github.com/apache/hbase/pull/3360#issuecomment-855322836 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 26s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 24s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 48s | master passed | | +1 :green_heart: | compile | 1m 19s | master passed | | +1 :green_heart: | shadedjars | 8m 15s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 57s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 38s | the patch passed | | +1 :green_heart: | compile | 1m 20s | the patch passed | | +1 :green_heart: | javac | 1m 20s | the patch passed | | +1 :green_heart: | shadedjars | 8m 15s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 53s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 37s | hbase-hadoop-compat in the patch passed. | | -1 :x: | unit | 147m 49s | hbase-server in the patch failed. | | | | 180m 12s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3360/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3360 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 4e05b5e3d1fa 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / de06e20e0a | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3360/1/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3360/1/testReport/ | | Max. process+thread count | 4591 (vs. ulimit of 3) | | modules | C: hbase-hadoop-compat hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3360/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-25891) Remove dependence storing WAL filenames for backup
[
https://issues.apache.org/jira/browse/HBASE-25891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mallikarjun updated HBASE-25891:
Description:
Context:
Currently WAL logs are stored in `backup:system` meta table
{code:java}
// code placeholder
wals:preprod-dn-1%2C16020%2C1614844389000.1621996160175 column=meta:backupId,
timestamp=1622003479895, value=backup_1622003358258
wals:preprod-dn-1%2C16020%2C1614844389000.1621996160175 column=meta:file,
timestamp=1622003479895,
value=hdfs://store/hbase/oldWALs/preprod-dn-1%2C16020%2C1614844389000.1621996160175
wals:preprod-dn-1%2C16020%2C1614844389000.1621996160175 column=meta:root,
timestamp=1622003479895, value=s3a://2021-05-25--21-45-00--full/set1
wals:preprod-dn-1%2C16020%2C1614844389000.1621999760280 column=meta:backupId,
timestamp=1622003479895, value=backup_1622003358258
wals:preprod-dn-1%2C16020%2C1614844389000.1621999760280 column=meta:file,
timestamp=1622003479895,
value=hdfs://store/hbase/oldWALs/preprod-dn-1%2C16020%2C1614844389000.1621999760280
wals:preprod-dn-1%2C16020%2C1614844389000.1621999760280 column=meta:root,
timestamp=1622003479895, value=s3a://2021-05-25--21-45-00--full/set1
{code}
Also, Every backup (Incremental and Full) performs a log roll just before
taking backup and stores what was the timestamp at which log roll was performed
per regionserver per backup using following format.
{code:java}
// code placeholder
rslogts:hdfs://xx.xx.xx.xx:8020/tmp/backup_yaktest\x00preprod-dn-2:16020
column=meta:rs-log-ts, timestamp=1622887363301,value=\x00\x00\x01y\xDB\x81ar
rslogts:hdfs://xx.xx.xx.xx:8020/tmp/backup_yaktest\x00preprod-dn-3:16020
column=meta:rs-log-ts, timestamp=1622887363294, value=\x00\x00\x01y\xDB\x81aP
rslogts:hdfs://xx.xx.xx.xx:8020/tmp/backup_yaktest\x00preprod-dn-1:16020
column=meta:rs-log-ts, timestamp=1622887363275, value=\x00\x00\x01y\xDB\x81\x85
{code}
There are 2 cases for which WAL log refrences stored in `backup:system` and are
being used.
1. To cleanup WAL's for which backup is already taken using `BackupLogCleaner`
Since log roll timestamp is stored as part of backup per regionserver. We can
check all previous successfull backup's and then identify which logs are to be
retained and which ones are to be cleaned up as follows
* Identify which are the latest successful backups performed per table.
* Per backup identified above, identify what is the oldest log rolled
timestamp perfomed per regionserver per table.
* All those WAL's which are older than oldest log rolled timestamp perfomed
for any table backed can be removed by `BackupLogCleaner`
2. During incremental backup, to check system table if there are any duplicate
WAL's for which backup is taken again.
* Incremental backup already identifies which all WAL's to be backed up using
`rslogts:` mentioned above.
* Additionally it checks `wals:` to ensure no logs are backuped for second
time. And this is redundant and not seen any extra benefit.
was:
Currently WAL logs are stored in `backup:system` meta table
{code:java}
// code placeholder
wals:preprod-dn-1%2C16020%2C1614844389000.1621996160175 column=meta:backupId,
timestamp=1622003479895, value=backup_1622003358258
wals:preprod-dn-1%2C16020%2C1614844389000.1621996160175 column=meta:file,
timestamp=1622003479895,
value=hdfs://store/hbase/oldWALs/preprod-dn-1%2C16020%2C1614844389000.1621996160175
wals:preprod-dn-1%2C16020%2C1614844389000.1621996160175 column=meta:root,
timestamp=1622003479895, value=s3a://2021-05-25--21-45-00--full/set1
wals:preprod-dn-1%2C16020%2C1614844389000.1621999760280 column=meta:backupId,
timestamp=1622003479895, value=backup_1622003358258
wals:preprod-dn-1%2C16020%2C1614844389000.1621999760280 column=meta:file,
timestamp=1622003479895,
value=hdfs://store/hbase/oldWALs/preprod-dn-1%2C16020%2C1614844389000.1621999760280
wals:preprod-dn-1%2C16020%2C1614844389000.1621999760280 column=meta:root,
timestamp=1622003479895, value=s3a://2021-05-25--21-45-00--full/set1
{code}
Primarily used for following
* WAL's are stored in meta table to check if a particular log has been backed
up or not.
* Check during incremental backup if a particular WAL is being backed up was
covered during previous incremental backup or not.
Changes for above 2 use cases.
* Since log roll during incremental or full backup is stored with prefix
`trslm:`. Can be used to identify which log files can be cleaned up
* Check during incremental backup if a particular WAL is being backed up or
not is redundant. No such a check is required
> Remove dependence storing WAL filenames for backup
> --
>
> Key: HBASE-25891
> URL: https://issues.apache.org/jira/browse/HBASE-25891
> Project: HBase
> Issue Type: Improvement
> Components: backuprestore
>Affects Versions: 3.0.0-alpha-1
>
[GitHub] [hbase] Apache-HBase commented on pull request #3360: HBASE-25975 Row Commit Sequencer
Apache-HBase commented on pull request #3360: URL: https://github.com/apache/hbase/pull/3360#issuecomment-855322096 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 29s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 25s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 29s | master passed | | +1 :green_heart: | compile | 1m 34s | master passed | | +1 :green_heart: | shadedjars | 8m 13s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 4s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 13s | the patch passed | | +1 :green_heart: | compile | 1m 31s | the patch passed | | +1 :green_heart: | javac | 1m 31s | the patch passed | | +1 :green_heart: | shadedjars | 8m 9s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 2s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 41s | hbase-hadoop-compat in the patch passed. | | -1 :x: | unit | 136m 43s | hbase-server in the patch failed. | | | | 171m 20s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3360/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3360 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 12b4fd7d1f72 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / de06e20e0a | | Default Java | AdoptOpenJDK-11.0.10+9 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3360/1/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3360/1/testReport/ | | Max. process+thread count | 4040 (vs. ulimit of 3) | | modules | C: hbase-hadoop-compat hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3360/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3360: HBASE-25975 Row Commit Sequencer
Apache-HBase commented on pull request #3360: URL: https://github.com/apache/hbase/pull/3360#issuecomment-855311339 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 4m 22s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 23s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 6s | master passed | | +1 :green_heart: | compile | 3m 55s | master passed | | +1 :green_heart: | checkstyle | 1m 23s | master passed | | +1 :green_heart: | spotbugs | 2m 43s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 50s | the patch passed | | +1 :green_heart: | compile | 3m 54s | the patch passed | | -0 :warning: | javac | 0m 30s | hbase-hadoop-compat generated 1 new + 102 unchanged - 1 fixed = 103 total (was 103) | | +1 :green_heart: | checkstyle | 1m 18s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 19m 11s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | spotbugs | 3m 6s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 25s | The patch does not generate ASF License warnings. | | | | 57m 23s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3360/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3360 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux c28cc632d129 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / de06e20e0a | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | javac | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3360/1/artifact/yetus-general-check/output/diff-compile-javac-hbase-hadoop-compat.txt | | Max. process+thread count | 96 (vs. ulimit of 3) | | modules | C: hbase-hadoop-compat hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3360/1/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3352: HBASE-25913 Introduce EnvironmentEdge.Clock and Clock.currentTimeAdvancing
Apache-HBase commented on pull request #3352: URL: https://github.com/apache/hbase/pull/3352#issuecomment-855310882 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 7s | Docker mode activated. | | -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 31s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 4s | master passed | | +1 :green_heart: | compile | 1m 50s | master passed | | +1 :green_heart: | shadedjars | 8m 59s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 13s | master passed | | -0 :warning: | patch | 10m 39s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 1s | the patch passed | | +1 :green_heart: | compile | 1m 51s | the patch passed | | +1 :green_heart: | javac | 1m 51s | the patch passed | | +1 :green_heart: | shadedjars | 9m 1s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 13s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 58s | hbase-common in the patch passed. | | -1 :x: | unit | 237m 41s | hbase-server in the patch failed. | | -1 :x: | unit | 19m 34s | hbase-backup in the patch failed. | | | | 295m 36s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3352/3/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3352 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux b0f1db42a330 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / de06e20e0a | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3352/3/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3352/3/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-backup.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3352/3/testReport/ | | Max. process+thread count | 3378 (vs. ulimit of 3) | | modules | C: hbase-common hbase-server hbase-backup U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3352/3/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3336: HBASE-25920 Support Hadoop 3.3.1
Apache-HBase commented on pull request #3336: URL: https://github.com/apache/hbase/pull/3336#issuecomment-855310211 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 25s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 40s | Maven dependency ordering for branch | | -1 :x: | mvninstall | 4m 28s | root in master failed. | | +1 :green_heart: | compile | 9m 5s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for patch | | -1 :x: | mvninstall | 1m 8s | root in the patch failed. | | -1 :x: | compile | 1m 32s | root in the patch failed. | | -0 :warning: | javac | 1m 32s | root in the patch failed. | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | xml | 0m 5s | The patch has no ill-formed XML file. | | +1 :green_heart: | hadoopcheck | 26m 5s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 55s | The patch does not generate ASF License warnings. | | | | 48m 32s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3336/7/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3336 | | Optional Tests | dupname asflicense javac hadoopcheck xml compile | | uname | Linux fc371097e7ab 4.15.0-142-generic #146-Ubuntu SMP Tue Apr 13 01:11:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / de06e20e0a | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | mvninstall | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3336/7/artifact/yetus-general-check/output/branch-mvninstall-root.txt | | mvninstall | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3336/7/artifact/yetus-general-check/output/patch-mvninstall-root.txt | | compile | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3336/7/artifact/yetus-general-check/output/patch-compile-root.txt | | javac | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3336/7/artifact/yetus-general-check/output/patch-compile-root.txt | | Max. process+thread count | 126 (vs. ulimit of 3) | | modules | C: . hbase-client hbase-common hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3336/7/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] saintstack commented on a change in pull request #3356: HBASE-25973 Balancer should explain progress in a better way in log
saintstack commented on a change in pull request #3356:
URL: https://github.com/apache/hbase/pull/3356#discussion_r646046394
##
File path:
hbase-balancer/src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java
##
@@ -343,15 +343,15 @@ boolean needsBalance(TableName tableName,
BalancerClusterState cluster) {
sendRejectionReasonToRingBuffer(() -> getBalanceReason(calculatedTotal,
calculatedMultiplier),
costFunctions);
}
-if (LOG.isDebugEnabled()) {
- LOG.debug("{} {}; total cost={}, sum multiplier={}; cost/multiplier to
need a balance is {}",
- balanced ? "Skipping load balancing because balanced" : "We need to
load balance",
- isByTable ? String.format("table (%s)", tableName) : "cluster",
- total, sumMultiplier, minCostNeedBalance);
- if (LOG.isTraceEnabled()) {
-LOG.trace("Balance decision detailed function costs={}",
functionCost());
- }
+LOG.info("{} {}; total cost={}, sum multiplier={}; cost/multiplier to need
a balance is {}",
+ balanced ? "Skipping load balancing because balanced" :
+"Start calculating moving plan without logging info for up to {} secs",
Review comment:
Should be "Calculating plan. May take up to {} seconds to complete."
##
File path:
hbase-balancer/src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java
##
@@ -343,15 +343,15 @@ boolean needsBalance(TableName tableName,
BalancerClusterState cluster) {
sendRejectionReasonToRingBuffer(() -> getBalanceReason(calculatedTotal,
calculatedMultiplier),
costFunctions);
}
-if (LOG.isDebugEnabled()) {
- LOG.debug("{} {}; total cost={}, sum multiplier={}; cost/multiplier to
need a balance is {}",
- balanced ? "Skipping load balancing because balanced" : "We need to
load balance",
- isByTable ? String.format("table (%s)", tableName) : "cluster",
- total, sumMultiplier, minCostNeedBalance);
- if (LOG.isTraceEnabled()) {
-LOG.trace("Balance decision detailed function costs={}",
functionCost());
- }
+LOG.info("{} {}; total cost={}, sum multiplier={}; cost/multiplier to need
a balance is {}",
+ balanced ? "Skipping load balancing because balanced" :
+"Start calculating moving plan without logging info for up to {} secs",
+ isByTable ? String.format("table (%s)", tableName) : "cluster",
Review comment:
This string is a little confusing... We are balancing by table or by
cluster? Should this fact be in the previous string?... "Calculating a plan
for table=TABLENAME " or Calculating a plan for the full cluster..." Think
of the poor operator reading the log message.
##
File path:
hbase-balancer/src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java
##
@@ -470,17 +470,17 @@ private long calculateMaxSteps(BalancerClusterState
cluster) {
updateStochasticCosts(tableName, curOverallCost, curFunctionCosts);
if (initCost > currentCost) {
List plans = createRegionPlans(cluster);
- LOG.info("Finished computing new load balance plan. Computation took {}"
+
-" to try {} different iterations. Found a solution that moves " +
-"{} regions; Going from a computed cost of {}" +
-" to a new cost of {}", java.time.Duration.ofMillis(endTime -
startTime),
-step, plans.size(), initCost, currentCost);
+ LOG.info("Finished computing new load balance plan. Computation took {}
ms" +
Review comment:
We should name what we did here the same as what it was called in the
start message. Here it is 'new load balance plan'. In the start log it is a
'moving plan'.
##
File path:
hbase-balancer/src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java
##
@@ -470,17 +470,17 @@ private long calculateMaxSteps(BalancerClusterState
cluster) {
updateStochasticCosts(tableName, curOverallCost, curFunctionCosts);
if (initCost > currentCost) {
List plans = createRegionPlans(cluster);
- LOG.info("Finished computing new load balance plan. Computation took {}"
+
-" to try {} different iterations. Found a solution that moves " +
-"{} regions; Going from a computed cost of {}" +
-" to a new cost of {}", java.time.Duration.ofMillis(endTime -
startTime),
-step, plans.size(), initCost, currentCost);
+ LOG.info("Finished computing new load balance plan. Computation took {}
ms" +
+ " to try {} different iterations. Found a solution that moves " +
+ "{} regions; Going from a computed cost of {}" +
+ " to a new cost of {}. Move plan to be submitted to master to
execute",
Review comment:
Rather than 'move plan to be submitted...' just say 'Executing plan'.
Its called 'cost' but is that the right term? Is it our assessment of
'balance'?
[GitHub] [hbase] apurtell edited a comment on pull request #3360: HBASE-25975 Row Commit Sequencer
apurtell edited a comment on pull request #3360: URL: https://github.com/apache/hbase/pull/3360#issuecomment-855306586 The main issue with https://github.com/apache/hbase/pull/3352 is serializing commits with the act of getting the time requires that everything that might mutate the resource you are trying to protect must be within that scope. For a single row update the scope would be a row, and highly granular, so performance will not be affected that much. For batch mutations, which are typical, the scope is the region, and serializing commits to a region is expensive. I struggled for a while with various forms of hierarchical clocks where one might get a clock for the region, then use that clock to get child clocks for a row, and was not successful because they always ended up serializing *something* at region scope, but then reconsidered... What is it we are really tracking? Why not track that explicitly? It seemed much simpler in retrospect. Hence this PR, to explore that idea instead. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] apurtell commented on pull request #3360: HBASE-25975 Row Commit Sequencer
apurtell commented on pull request #3360: URL: https://github.com/apache/hbase/pull/3360#issuecomment-855306586 The main issue with https://github.com/apache/hbase/pull/3352 is serializing commits with the act of getting the time requires that everything that might mutate the resource you are trying to protect must be within that scope. For a single row update the scope would be a row, and highly granular, so performance will not be affected that much. For batch mutations, which are typical, the scope is the region, and serializing commits to a region is expensive. I struggled for a while with various forms of hierarchical clocks where one might get a clock for the region, then use that clock to get child clocks for a row, but then reconsidered... What is it we are really tracking? Why not track that explicitly? It seemed much simpler in retrospect. Hence this PR, to explore that idea instead. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-25975) Row commit sequencer
[ https://issues.apache.org/jira/browse/HBASE-25975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-25975: Status: Patch Available (was: Open) > Row commit sequencer > > > Key: HBASE-25975 > URL: https://issues.apache.org/jira/browse/HBASE-25975 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: Andrew Kyle Purtell >Assignee: Andrew Kyle Purtell >Priority: Major > Fix For: 3.0.0-alpha-1, 2.5.0 > > > Use a row commit sequencer in HRegion to ensure that only the operations that > mutate disjoint sets of rows are able to commit within the same clock tick. > This maintains the invariant that more than one mutation to a given row will > never be committed in the same clock tick. > Callers will first acquire row locks for the row(s) the pending mutation will > mutate. Then they will use RowCommitSequencer.getRowSequence to ensure that > the set of rows about to be mutated do not overlap with those for any other > pending mutations in the current clock tick. If an overlap is identified, > getRowSequence will yield and loop until there is no longer an overlap and > the caller's pending mutation can succeed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25975) Row commit sequencer
[ https://issues.apache.org/jira/browse/HBASE-25975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17357978#comment-17357978 ] Andrew Kyle Purtell commented on HBASE-25975: - Draft PR: https://github.com/apache/hbase/pull/3360 > Row commit sequencer > > > Key: HBASE-25975 > URL: https://issues.apache.org/jira/browse/HBASE-25975 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: Andrew Kyle Purtell >Assignee: Andrew Kyle Purtell >Priority: Major > Fix For: 3.0.0-alpha-1, 2.5.0 > > > Use a row commit sequencer in HRegion to ensure that only the operations that > mutate disjoint sets of rows are able to commit within the same clock tick. > This maintains the invariant that more than one mutation to a given row will > never be committed in the same clock tick. > Callers will first acquire row locks for the row(s) the pending mutation will > mutate. Then they will use RowCommitSequencer.getRowSequence to ensure that > the set of rows about to be mutated do not overlap with those for any other > pending mutations in the current clock tick. If an overlap is identified, > getRowSequence will yield and loop until there is no longer an overlap and > the caller's pending mutation can succeed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] apurtell opened a new pull request #3360: HBASE-25975 Row Commit Sequencer
apurtell opened a new pull request #3360: URL: https://github.com/apache/hbase/pull/3360 Use a row commit sequencer in HRegion to ensure that only the operations that mutate disjoint sets of rows are able to commit within the same clock tick. This maintains the invariant that more than one mutation to a given row will never be committed in the same clock tick. Callers will first acquire row locks for the row(s) the pending mutation will mutate. Then they will use RowCommitSequencer.getRowSequence to ensure that the set of rows about to be mutated do not overlap with those for any other pending mutations in the current clock tick. If an overlap is identified, getRowSequence will yield and loop until there is no longer an overlap and the caller's pending mutation can succeed. TODO: - Needs tests. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (HBASE-25975) Row commit sequencer
Andrew Kyle Purtell created HBASE-25975: --- Summary: Row commit sequencer Key: HBASE-25975 URL: https://issues.apache.org/jira/browse/HBASE-25975 Project: HBase Issue Type: Sub-task Components: regionserver Reporter: Andrew Kyle Purtell Assignee: Andrew Kyle Purtell Fix For: 3.0.0-alpha-1, 2.5.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] saintstack commented on pull request #3336: HBASE-25920 Support Hadoop 3.3.1
saintstack commented on pull request #3336: URL: https://github.com/apache/hbase/pull/3336#issuecomment-855305969 Same two tests, TestLogRolling and TestReplicationEndpointWithMultipleAsyncWAL keep showing up ... failing on jdk8 but passing on jdk11 and passing locally. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-25975) Row commit sequencer
[ https://issues.apache.org/jira/browse/HBASE-25975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-25975: Description: Use a row commit sequencer in HRegion to ensure that only the operations that mutate disjoint sets of rows are able to commit within the same clock tick. This maintains the invariant that more than one mutation to a given row will never be committed in the same clock tick. Callers will first acquire row locks for the row(s) the pending mutation will mutate. Then they will use RowCommitSequencer.getRowSequence to ensure that the set of rows about to be mutated do not overlap with those for any other pending mutations in the current clock tick. If an overlap is identified, getRowSequence will yield and loop until there is no longer an overlap and the caller's pending mutation can succeed. > Row commit sequencer > > > Key: HBASE-25975 > URL: https://issues.apache.org/jira/browse/HBASE-25975 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: Andrew Kyle Purtell >Assignee: Andrew Kyle Purtell >Priority: Major > Fix For: 3.0.0-alpha-1, 2.5.0 > > > Use a row commit sequencer in HRegion to ensure that only the operations that > mutate disjoint sets of rows are able to commit within the same clock tick. > This maintains the invariant that more than one mutation to a given row will > never be committed in the same clock tick. > Callers will first acquire row locks for the row(s) the pending mutation will > mutate. Then they will use RowCommitSequencer.getRowSequence to ensure that > the set of rows about to be mutated do not overlap with those for any other > pending mutations in the current clock tick. If an overlap is identified, > getRowSequence will yield and loop until there is no longer an overlap and > the caller's pending mutation can succeed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25666) Explain why balancer is skipping runs
[ https://issues.apache.org/jira/browse/HBASE-25666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17357956#comment-17357956 ] Michael Stack commented on HBASE-25666: --- [~claraxiong] what does the below comment refer too?? bq. Add more logging Perhaps a mistype here? > Explain why balancer is skipping runs > - > > Key: HBASE-25666 > URL: https://issues.apache.org/jira/browse/HBASE-25666 > Project: HBase > Issue Type: Improvement > Components: Balancer, master, UI >Reporter: Clara Xiong >Assignee: Zhuoyue Huang >Priority: Major > Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.4 > > > Balancer needs to find the balance of keeping cluster balanced and stable. > There is a configuration minCostNeedBalance as the threshold of imbalance to > try to balance. Since we use a single score to combine all the factors we > consider, it is hard for operation to understand why balancer is "stuck". We > should add to master-status UI to show balancer is skipping runs and explain > all factors considered, such as the weight and cost of all cost functions. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #3352: HBASE-25913 Introduce EnvironmentEdge.Clock and Clock.currentTimeAdvancing
Apache-HBase commented on pull request #3352: URL: https://github.com/apache/hbase/pull/3352#issuecomment-855303543 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 31s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 30s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 22s | master passed | | +1 :green_heart: | compile | 2m 14s | master passed | | +1 :green_heart: | shadedjars | 8m 33s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 24s | master passed | | -0 :warning: | patch | 10m 29s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 26s | the patch passed | | +1 :green_heart: | compile | 2m 11s | the patch passed | | +1 :green_heart: | javac | 2m 11s | the patch passed | | +1 :green_heart: | shadedjars | 8m 18s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 25s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 57s | hbase-common in the patch passed. | | -1 :x: | unit | 156m 24s | hbase-server in the patch failed. | | -1 :x: | unit | 19m 42s | hbase-backup in the patch failed. | | | | 215m 2s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3352/3/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3352 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux c2953c89f373 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / de06e20e0a | | Default Java | AdoptOpenJDK-11.0.10+9 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3352/3/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3352/3/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-backup.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3352/3/testReport/ | | Max. process+thread count | 5049 (vs. ulimit of 3) | | modules | C: hbase-common hbase-server hbase-backup U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3352/3/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3352: HBASE-25913 Introduce EnvironmentEdge.Clock and Clock.currentTimeAdvancing
Apache-HBase commented on pull request #3352: URL: https://github.com/apache/hbase/pull/3352#issuecomment-855287242 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 31s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 34s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 44s | master passed | | +1 :green_heart: | compile | 4m 45s | master passed | | +1 :green_heart: | checkstyle | 1m 39s | master passed | | +1 :green_heart: | spotbugs | 3m 25s | master passed | | -0 :warning: | patch | 1m 2s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 43s | the patch passed | | +1 :green_heart: | compile | 4m 45s | the patch passed | | -0 :warning: | javac | 0m 51s | hbase-common generated 1 new + 158 unchanged - 1 fixed = 159 total (was 159) | | -0 :warning: | checkstyle | 0m 24s | hbase-common: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) | | -0 :warning: | checkstyle | 1m 10s | hbase-server: The patch generated 4 new + 201 unchanged - 3 fixed = 205 total (was 204) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 21m 20s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | -1 :x: | spotbugs | 1m 14s | hbase-common generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 38s | The patch does not generate ASF License warnings. | | | | 63m 10s | | | Reason | Tests | |---:|:--| | FindBugs | module:hbase-common | | | Inconsistent synchronization of org.apache.hadoop.hbase.util.IncrementingEnvironmentEdge.timeIncrement; locked 71% of time Unsynchronized access at IncrementingEnvironmentEdge.java:71% of time Unsynchronized access at IncrementingEnvironmentEdge.java:[line 49] | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3352/3/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3352 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux 5975ce5cdb36 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / de06e20e0a | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | javac | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3352/3/artifact/yetus-general-check/output/diff-compile-javac-hbase-common.txt | | checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3352/3/artifact/yetus-general-check/output/diff-checkstyle-hbase-common.txt | | checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3352/3/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | spotbugs | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3352/3/artifact/yetus-general-check/output/new-spotbugs-hbase-common.html | | Max. process+thread count | 96 (vs. ulimit of 3) | | modules | C: hbase-common hbase-server hbase-backup U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3352/3/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] apurtell commented on pull request #3352: HBASE-25913 Introduce EnvironmentEdge.Clock and Clock.currentTimeAdvancing
apurtell commented on pull request #3352: URL: https://github.com/apache/hbase/pull/3352#issuecomment-855279953 Latest push fixes tests but I'm considering another approach as well. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3359: HBASE-25891 remove dependence storing wal filenames for backup
Apache-HBase commented on pull request #3359: URL: https://github.com/apache/hbase/pull/3359#issuecomment-855272036 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 8m 43s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +0 :ok: | prototool | 0m 0s | prototool was not available. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 28s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 54s | master passed | | +1 :green_heart: | compile | 2m 26s | master passed | | +1 :green_heart: | checkstyle | 0m 29s | master passed | | +1 :green_heart: | spotbugs | 5m 10s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 52s | the patch passed | | +1 :green_heart: | compile | 2m 18s | the patch passed | | +1 :green_heart: | cc | 2m 18s | the patch passed | | -0 :warning: | javac | 0m 39s | hbase-backup generated 1 new + 73 unchanged - 6 fixed = 74 total (was 79) | | +1 :green_heart: | checkstyle | 0m 26s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 24m 20s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | hbaseprotoc | 1m 40s | the patch passed | | +1 :green_heart: | spotbugs | 5m 37s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 23s | The patch does not generate ASF License warnings. | | | | 72m 21s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3359/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3359 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile cc hbaseprotoc prototool | | uname | Linux 8edc9748b32e 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / de06e20e0a | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | javac | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3359/1/artifact/yetus-general-check/output/diff-compile-javac-hbase-backup.txt | | Max. process+thread count | 86 (vs. ulimit of 3) | | modules | C: hbase-protocol-shaded hbase-backup U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3359/1/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-25891) Remove dependence storing WAL filenames for backup
[
https://issues.apache.org/jira/browse/HBASE-25891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17357933#comment-17357933
]
Anoop Sam John commented on HBASE-25891:
bq.Since log roll during incremental or full backup is stored with prefix
`trslm:`. Can be used to identify which log files can be cleaned up
Means the WAL files will get renamed with this prefix? When those files become
eligible for deletion then?
Can u pls give a detailed design consideration then? Now that we dont have
this systen table at all, what happens when taking a full/incremental snapshot?
How WAL files been retained when backup refers to it? When that become eligible
for deletion? (Backup deleted/ another full backup came?) And how we make sure
we allow WAL deletion then?
> Remove dependence storing WAL filenames for backup
> --
>
> Key: HBASE-25891
> URL: https://issues.apache.org/jira/browse/HBASE-25891
> Project: HBase
> Issue Type: Improvement
> Components: backuprestore
>Affects Versions: 3.0.0-alpha-1
>Reporter: Mallikarjun
>Assignee: Mallikarjun
>Priority: Major
> Fix For: 3.0.0-alpha-1
>
>
> Currently WAL logs are stored in `backup:system` meta table
> {code:java}
> // code placeholder
> wals:preprod-dn-1%2C16020%2C1614844389000.1621996160175 column=meta:backupId,
> timestamp=1622003479895, value=backup_1622003358258
> wals:preprod-dn-1%2C16020%2C1614844389000.1621996160175 column=meta:file,
> timestamp=1622003479895,
> value=hdfs://store/hbase/oldWALs/preprod-dn-1%2C16020%2C1614844389000.1621996160175
> wals:preprod-dn-1%2C16020%2C1614844389000.1621996160175 column=meta:root,
> timestamp=1622003479895, value=s3a://2021-05-25--21-45-00--full/set1
> wals:preprod-dn-1%2C16020%2C1614844389000.1621999760280 column=meta:backupId,
> timestamp=1622003479895, value=backup_1622003358258
> wals:preprod-dn-1%2C16020%2C1614844389000.1621999760280 column=meta:file,
> timestamp=1622003479895,
> value=hdfs://store/hbase/oldWALs/preprod-dn-1%2C16020%2C1614844389000.1621999760280
> wals:preprod-dn-1%2C16020%2C1614844389000.1621999760280 column=meta:root,
> timestamp=1622003479895, value=s3a://2021-05-25--21-45-00--full/set1
> {code}
> Primarily used for following
> * WAL's are stored in meta table to check if a particular log has been
> backed up or not.
> * Check during incremental backup if a particular WAL is being backed up was
> covered during previous incremental backup or not.
> Changes for above 2 use cases.
> * Since log roll during incremental or full backup is stored with prefix
> `trslm:`. Can be used to identify which log files can be cleaned up
> * Check during incremental backup if a particular WAL is being backed up or
> not is redundant. No such a check is required
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #3357: HBASE-25891 remove-dependence-storing-wal-filenames-for-backup
Apache-HBase commented on pull request #3357: URL: https://github.com/apache/hbase/pull/3357#issuecomment-855268878 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 27s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +0 :ok: | prototool | 0m 0s | prototool was not available. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 36s | master passed | | +1 :green_heart: | compile | 2m 5s | master passed | | +1 :green_heart: | checkstyle | 0m 26s | master passed | | +1 :green_heart: | spotbugs | 4m 20s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 40s | the patch passed | | +1 :green_heart: | compile | 2m 3s | the patch passed | | +1 :green_heart: | cc | 2m 3s | the patch passed | | -0 :warning: | javac | 0m 37s | hbase-backup generated 1 new + 73 unchanged - 6 fixed = 74 total (was 79) | | +1 :green_heart: | checkstyle | 0m 25s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 18m 1s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | hbaseprotoc | 1m 25s | the patch passed | | +1 :green_heart: | spotbugs | 4m 51s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 25s | The patch does not generate ASF License warnings. | | | | 50m 22s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3357/4/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3357 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile cc hbaseprotoc prototool | | uname | Linux 0178d93edfa3 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / de06e20e0a | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | javac | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3357/4/artifact/yetus-general-check/output/diff-compile-javac-hbase-backup.txt | | Max. process+thread count | 96 (vs. ulimit of 3) | | modules | C: hbase-protocol-shaded hbase-backup U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3357/4/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3357: HBASE-25891 remove-dependence-storing-wal-filenames-for-backup
Apache-HBase commented on pull request #3357: URL: https://github.com/apache/hbase/pull/3357#issuecomment-855268855 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 22s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 7s | master passed | | +1 :green_heart: | compile | 1m 14s | master passed | | +1 :green_heart: | shadedjars | 9m 44s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 34s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 54s | the patch passed | | +1 :green_heart: | compile | 1m 20s | the patch passed | | +1 :green_heart: | javac | 1m 20s | the patch passed | | +1 :green_heart: | shadedjars | 9m 59s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 35s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 55s | hbase-protocol-shaded in the patch passed. | | +1 :green_heart: | unit | 13m 43s | hbase-backup in the patch passed. | | | | 50m 24s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3357/4/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3357 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 3226933aa84f 4.15.0-142-generic #146-Ubuntu SMP Tue Apr 13 01:11:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / de06e20e0a | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3357/4/testReport/ | | Max. process+thread count | 3187 (vs. ulimit of 3) | | modules | C: hbase-protocol-shaded hbase-backup U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3357/4/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3359: HBASE-25891 remove dependence storing wal filenames for backup
Apache-HBase commented on pull request #3359: URL: https://github.com/apache/hbase/pull/3359#issuecomment-855268678 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 2s | Docker mode activated. | | -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 21s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 14s | master passed | | +1 :green_heart: | compile | 1m 11s | master passed | | +1 :green_heart: | shadedjars | 9m 2s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 30s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 5s | the patch passed | | +1 :green_heart: | compile | 1m 11s | the patch passed | | +1 :green_heart: | javac | 1m 11s | the patch passed | | +1 :green_heart: | shadedjars | 9m 3s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 28s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 48s | hbase-protocol-shaded in the patch passed. | | +1 :green_heart: | unit | 12m 58s | hbase-backup in the patch passed. | | | | 46m 28s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3359/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3359 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux b9434606e73b 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / de06e20e0a | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3359/1/testReport/ | | Max. process+thread count | 3090 (vs. ulimit of 3) | | modules | C: hbase-protocol-shaded hbase-backup U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3359/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3359: HBASE-25891 remove dependence storing wal filenames for backup
Apache-HBase commented on pull request #3359: URL: https://github.com/apache/hbase/pull/3359#issuecomment-855268153 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 27s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 23s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 28s | master passed | | +1 :green_heart: | compile | 1m 27s | master passed | | +1 :green_heart: | shadedjars | 8m 11s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 37s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 19s | the patch passed | | +1 :green_heart: | compile | 1m 26s | the patch passed | | +1 :green_heart: | javac | 1m 26s | the patch passed | | +1 :green_heart: | shadedjars | 8m 16s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 35s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 1s | hbase-protocol-shaded in the patch passed. | | +1 :green_heart: | unit | 9m 34s | hbase-backup in the patch passed. | | | | 42m 28s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3359/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3359 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 8950f4daeaa1 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / de06e20e0a | | Default Java | AdoptOpenJDK-11.0.10+9 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3359/1/testReport/ | | Max. process+thread count | 5025 (vs. ulimit of 3) | | modules | C: hbase-protocol-shaded hbase-backup U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3359/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3357: HBASE-25891 remove-dependence-storing-wal-filenames-for-backup
Apache-HBase commented on pull request #3357: URL: https://github.com/apache/hbase/pull/3357#issuecomment-855267609 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 28s | Docker mode activated. | | -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 14s | master passed | | +1 :green_heart: | compile | 1m 28s | master passed | | +1 :green_heart: | shadedjars | 8m 8s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 35s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 15s | the patch passed | | +1 :green_heart: | compile | 1m 25s | the patch passed | | +1 :green_heart: | javac | 1m 25s | the patch passed | | +1 :green_heart: | shadedjars | 8m 11s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 35s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 58s | hbase-protocol-shaded in the patch passed. | | +1 :green_heart: | unit | 8m 42s | hbase-backup in the patch passed. | | | | 40m 56s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3357/4/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3357 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 31e003cf7639 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / de06e20e0a | | Default Java | AdoptOpenJDK-11.0.10+9 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3357/4/testReport/ | | Max. process+thread count | 5044 (vs. ulimit of 3) | | modules | C: hbase-protocol-shaded hbase-backup U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3357/4/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-25891) Remove dependence storing WAL filenames for backup
[
https://issues.apache.org/jira/browse/HBASE-25891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17357912#comment-17357912
]
Mallikarjun commented on HBASE-25891:
-
[~anoop.hbase] I have made certain changes to this PR (not relating to the
multi tenancy scope earlier planned) and updated description accordingly what
the changes are. Please let me know if this is sufficient.
> Remove dependence storing WAL filenames for backup
> --
>
> Key: HBASE-25891
> URL: https://issues.apache.org/jira/browse/HBASE-25891
> Project: HBase
> Issue Type: Improvement
> Components: backuprestore
>Affects Versions: 3.0.0-alpha-1
>Reporter: Mallikarjun
>Assignee: Mallikarjun
>Priority: Major
> Fix For: 3.0.0-alpha-1
>
>
> Currently WAL logs are stored in `backup:system` meta table
> {code:java}
> // code placeholder
> wals:preprod-dn-1%2C16020%2C1614844389000.1621996160175 column=meta:backupId,
> timestamp=1622003479895, value=backup_1622003358258
> wals:preprod-dn-1%2C16020%2C1614844389000.1621996160175 column=meta:file,
> timestamp=1622003479895,
> value=hdfs://store/hbase/oldWALs/preprod-dn-1%2C16020%2C1614844389000.1621996160175
> wals:preprod-dn-1%2C16020%2C1614844389000.1621996160175 column=meta:root,
> timestamp=1622003479895, value=s3a://2021-05-25--21-45-00--full/set1
> wals:preprod-dn-1%2C16020%2C1614844389000.1621999760280 column=meta:backupId,
> timestamp=1622003479895, value=backup_1622003358258
> wals:preprod-dn-1%2C16020%2C1614844389000.1621999760280 column=meta:file,
> timestamp=1622003479895,
> value=hdfs://store/hbase/oldWALs/preprod-dn-1%2C16020%2C1614844389000.1621999760280
> wals:preprod-dn-1%2C16020%2C1614844389000.1621999760280 column=meta:root,
> timestamp=1622003479895, value=s3a://2021-05-25--21-45-00--full/set1
> {code}
> Primarily used for following
> * WAL's are stored in meta table to check if a particular log has been
> backed up or not.
> * Check during incremental backup if a particular WAL is being backed up was
> covered during previous incremental backup or not.
> Changes for above 2 use cases.
> * Since log roll during incremental or full backup is stored with prefix
> `trslm:`. Can be used to identify which log files can be cleaned up
> * Check during incremental backup if a particular WAL is being backed up or
> not is redundant. No such a check is required
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-25891) Remove dependence storing WAL filenames for backup
[
https://issues.apache.org/jira/browse/HBASE-25891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mallikarjun updated HBASE-25891:
Description:
Currently WAL logs are stored in `backup:system` meta table
{code:java}
// code placeholder
wals:preprod-dn-1%2C16020%2C1614844389000.1621996160175 column=meta:backupId,
timestamp=1622003479895, value=backup_1622003358258
wals:preprod-dn-1%2C16020%2C1614844389000.1621996160175 column=meta:file,
timestamp=1622003479895,
value=hdfs://store/hbase/oldWALs/preprod-dn-1%2C16020%2C1614844389000.1621996160175
wals:preprod-dn-1%2C16020%2C1614844389000.1621996160175 column=meta:root,
timestamp=1622003479895, value=s3a://2021-05-25--21-45-00--full/set1
wals:preprod-dn-1%2C16020%2C1614844389000.1621999760280 column=meta:backupId,
timestamp=1622003479895, value=backup_1622003358258
wals:preprod-dn-1%2C16020%2C1614844389000.1621999760280 column=meta:file,
timestamp=1622003479895,
value=hdfs://store/hbase/oldWALs/preprod-dn-1%2C16020%2C1614844389000.1621999760280
wals:preprod-dn-1%2C16020%2C1614844389000.1621999760280 column=meta:root,
timestamp=1622003479895, value=s3a://2021-05-25--21-45-00--full/set1
{code}
Primarily used for following
* WAL's are stored in meta table to check if a particular log has been backed
up or not.
* Check during incremental backup if a particular WAL is being backed up was
covered during previous incremental backup or not.
Changes for above 2 use cases.
* Since log roll during incremental or full backup is stored with prefix
`trslm:`. Can be used to identify which log files can be cleaned up
* Check during incremental backup if a particular WAL is being backed up or
not is redundant. No such a check is required
was:
Currently WAL logs are stored in `backup:system` meta table
{code:java}
// code placeholder
wals:preprod-dn-1%2C16020%2C1614844389000.1621996160175 column=meta:backupId,
timestamp=1622003479895, value=backup_1622003358258
wals:preprod-dn-1%2C16020%2C1614844389000.1621996160175 column=meta:file,
timestamp=1622003479895,
value=hdfs://store/hbase/oldWALs/preprod-dn-1%2C16020%2C1614844389000.1621996160175
wals:preprod-dn-1%2C16020%2C1614844389000.1621996160175 column=meta:root,
timestamp=1622003479895, value=s3a://2021-05-25--21-45-00--full/set1
wals:preprod-dn-1%2C16020%2C1614844389000.1621999760280 column=meta:backupId,
timestamp=1622003479895, value=backup_1622003358258
wals:preprod-dn-1%2C16020%2C1614844389000.1621999760280 column=meta:file,
timestamp=1622003479895,
value=hdfs://store/hbase/oldWALs/preprod-dn-1%2C16020%2C1614844389000.1621999760280
wals:preprod-dn-1%2C16020%2C1614844389000.1621999760280 column=meta:root,
timestamp=1622003479895, value=s3a://2021-05-25--21-45-00--full/set1
{code}
This has several problems
# Ever growing rows of wal's sourced for incremental backup is maintained and
never cleaned up.
# Unnecessary to have wal log listed for performing incremental backup or for
performing logcleaner.
# No support for rsgroup. Hence tables belonging to rsgroups which doesn't
have backup enabled also have to retain wals' and forever.
Proposed Solution:
> Remove dependence storing WAL filenames for backup
> --
>
> Key: HBASE-25891
> URL: https://issues.apache.org/jira/browse/HBASE-25891
> Project: HBase
> Issue Type: Improvement
> Components: backuprestore
>Affects Versions: 3.0.0-alpha-1
>Reporter: Mallikarjun
>Assignee: Mallikarjun
>Priority: Major
> Fix For: 3.0.0-alpha-1
>
>
> Currently WAL logs are stored in `backup:system` meta table
> {code:java}
> // code placeholder
> wals:preprod-dn-1%2C16020%2C1614844389000.1621996160175 column=meta:backupId,
> timestamp=1622003479895, value=backup_1622003358258
> wals:preprod-dn-1%2C16020%2C1614844389000.1621996160175 column=meta:file,
> timestamp=1622003479895,
> value=hdfs://store/hbase/oldWALs/preprod-dn-1%2C16020%2C1614844389000.1621996160175
> wals:preprod-dn-1%2C16020%2C1614844389000.1621996160175 column=meta:root,
> timestamp=1622003479895, value=s3a://2021-05-25--21-45-00--full/set1
> wals:preprod-dn-1%2C16020%2C1614844389000.1621999760280 column=meta:backupId,
> timestamp=1622003479895, value=backup_1622003358258
> wals:preprod-dn-1%2C16020%2C1614844389000.1621999760280 column=meta:file,
> timestamp=1622003479895,
> value=hdfs://store/hbase/oldWALs/preprod-dn-1%2C16020%2C1614844389000.1621999760280
> wals:preprod-dn-1%2C16020%2C1614844389000.1621999760280 column=meta:root,
> timestamp=1622003479895, value=s3a://2021-05-25--21-45-00--full/set1
> {code}
> Primarily used for following
> * WAL's are stored in meta table to check if a particular log has been
> backed up or not.
> * Check during incremental backup if a particular WAL is being backed up was
> covered
[GitHub] [hbase] rda3mon opened a new pull request #3359: HBASE-25891 remove dependence storing wal filenames for backup
rda3mon opened a new pull request #3359: URL: https://github.com/apache/hbase/pull/3359 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] rda3mon closed pull request #3358: HBASE-25891 remove dependence storing wal filenames for backup backedup
rda3mon closed pull request #3358: URL: https://github.com/apache/hbase/pull/3358 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] rda3mon opened a new pull request #3358: HBASE-25891 remove dependence storing wal filenames for backup backedup
rda3mon opened a new pull request #3358: URL: https://github.com/apache/hbase/pull/3358 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] rda3mon closed pull request #3357: HBASE-25891 remove-dependence-storing-wal-filenames-for-backup
rda3mon closed pull request #3357: URL: https://github.com/apache/hbase/pull/3357 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3357: HBASE-25891 remove-dependence-storing-wal-filenames-for-backup
Apache-HBase commented on pull request #3357: URL: https://github.com/apache/hbase/pull/3357#issuecomment-855256281 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 2m 42s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 17s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 31s | master passed | | +1 :green_heart: | compile | 1m 19s | master passed | | +1 :green_heart: | shadedjars | 9m 57s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 32s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 17s | the patch passed | | +1 :green_heart: | compile | 1m 23s | the patch passed | | +1 :green_heart: | javac | 1m 23s | the patch passed | | +1 :green_heart: | shadedjars | 9m 37s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 33s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 53s | hbase-protocol-shaded in the patch passed. | | +1 :green_heart: | unit | 12m 55s | hbase-backup in the patch passed. | | | | 50m 39s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3357/3/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3357 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux bfda52a16815 4.15.0-142-generic #146-Ubuntu SMP Tue Apr 13 01:11:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / de06e20e0a | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3357/3/testReport/ | | Max. process+thread count | 2986 (vs. ulimit of 3) | | modules | C: hbase-protocol-shaded hbase-backup U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3357/3/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3357: HBASE-25891 remove-dependence-storing-wal-filenames-for-backup
Apache-HBase commented on pull request #3357: URL: https://github.com/apache/hbase/pull/3357#issuecomment-855256247 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 28s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +0 :ok: | prototool | 0m 0s | prototool was not available. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 35s | master passed | | +1 :green_heart: | compile | 2m 2s | master passed | | +1 :green_heart: | checkstyle | 0m 27s | master passed | | +1 :green_heart: | spotbugs | 4m 18s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 38s | the patch passed | | +1 :green_heart: | compile | 2m 1s | the patch passed | | +1 :green_heart: | cc | 2m 1s | the patch passed | | -0 :warning: | javac | 0m 37s | hbase-backup generated 1 new + 73 unchanged - 6 fixed = 74 total (was 79) | | +1 :green_heart: | checkstyle | 0m 26s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 18m 3s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | hbaseprotoc | 1m 24s | the patch passed | | +1 :green_heart: | spotbugs | 4m 53s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 25s | The patch does not generate ASF License warnings. | | | | 50m 24s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3357/3/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3357 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile cc hbaseprotoc prototool | | uname | Linux 6560dfefff18 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / de06e20e0a | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | javac | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3357/3/artifact/yetus-general-check/output/diff-compile-javac-hbase-backup.txt | | Max. process+thread count | 96 (vs. ulimit of 3) | | modules | C: hbase-protocol-shaded hbase-backup U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3357/3/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3357: HBASE-25891 remove-dependence-storing-wal-filenames-for-backup
Apache-HBase commented on pull request #3357: URL: https://github.com/apache/hbase/pull/3357#issuecomment-855255267 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 27s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 23s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 25s | master passed | | +1 :green_heart: | compile | 1m 26s | master passed | | +1 :green_heart: | shadedjars | 8m 18s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 37s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 13s | the patch passed | | +1 :green_heart: | compile | 1m 26s | the patch passed | | +1 :green_heart: | javac | 1m 26s | the patch passed | | +1 :green_heart: | shadedjars | 8m 11s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 35s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 59s | hbase-protocol-shaded in the patch passed. | | +1 :green_heart: | unit | 8m 57s | hbase-backup in the patch passed. | | | | 41m 40s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3357/3/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3357 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 13081092d353 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / de06e20e0a | | Default Java | AdoptOpenJDK-11.0.10+9 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3357/3/testReport/ | | Max. process+thread count | 4986 (vs. ulimit of 3) | | modules | C: hbase-protocol-shaded hbase-backup U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3357/3/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (HBASE-5790) ZKUtil deleteRecursively should be a recoverable operation
[ https://issues.apache.org/jira/browse/HBASE-5790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang resolved HBASE-5790. -- Resolution: Implemented > ZKUtil deleteRecursively should be a recoverable operation > -- > > Key: HBASE-5790 > URL: https://issues.apache.org/jira/browse/HBASE-5790 > Project: HBase > Issue Type: Improvement >Reporter: Jesse Yates >Priority: Major > Labels: zookeeper > Attachments: java_HBASE-5790-v1.patch, java_HBASE-5790.patch > > > As of 3.4.3 Zookeeper now has full, multi-operation transaction. This means > we can wholesale delete chunks of the zk tree and ensure that we don't have > any pesky recursive delete issues where we delete the children of a node, but > then a child joins before deletion of the parent. Even without transactions, > this should be the behavior, but it is possible to make it much cleaner now > that we have this new feature in zk. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-22626) Master assigns the region successfully, but updates the state of region failed, and then keeping the state of the region is OPENNING in zookeeper, If master restarted
[
https://issues.apache.org/jira/browse/HBASE-22626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17357885#comment-17357885
]
Duo Zhang commented on HBASE-22626:
---
Have you try newer version of HBase? The assignment implementation has been
chagned lot in newer versions.
> Master assigns the region successfully, but updates the state of region
> failed, and then keeping the state of the region is OPENNING in zookeeper,
> If master restarted, those OPENNING regions will not be assign forever.
> ---
>
> Key: HBASE-22626
> URL: https://issues.apache.org/jira/browse/HBASE-22626
> Project: HBase
> Issue Type: Bug
> Components: Region Assignment
>Affects Versions: 2.0.0
>Reporter: chenwandong
>Priority: Critical
>
> Problem Description
> （1）One of the region server restarts causes all the regions that have been
> assigned to be migrated.
> （2）The master checks these regions and assigns them to other region servers,
> and assigns other region server assignments successfully, but the update
> state fails.
> 2019-06-22 16:44:20,065 INFO [PEWorker-8] procedure2.ProcedureExecutor:
> Finished pid=5038, ppid=4488, state=SUCCESS; AssignProcedure
> table=test061910, region=379c730490b4848f3db065fb25b87452 in 10.5680sec
> ... ...
> 2019-06-22 16:44:38,725 INFO [PEWorker-4]
> procedure.MasterProcedureScheduler: pid=5038, ppid=4488, state=SUCCESS;
> AssignProcedure table=test061910, region=379c730490b4848f3db065fb25b87452
> checking lock on 379c730490b4848f3db065fb25b87452
> 2019-06-22 16:44:38,725 ERROR [PEWorker-4] procedure2.ProcedureExecutor:
> CODE-BUG: Uncaught runtime exception for pid=5038, ppid=4488, state=SUCCESS;
> AssignProcedure table=test061910, region=379c730490b4848f3db065fb25b87452
> java.lang.UnsupportedOperationException: Unhandled state
> REGION_TRANSITION_FINISH; there is no rollback for assignment unless we
> cancel the operation by dropping/disabling the table
> at
> org.apache.hadoop.hbase.master.assignment.RegionTransitionProcedure.rollback(RegionTransitionProcedure.java:412)
> at
> org.apache.hadoop.hbase.master.assignment.RegionTransitionProcedure.rollback(RegionTransitionProcedure.java:95)
> at
> org.apache.hadoop.hbase.procedure2.Procedure.doRollback(Procedure.java:864)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecutor.java:1373)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecutor.java:1329)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1198)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$800(ProcedureExecutor.java:75)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1761)
> （3）The master restarted, and the old region state will be loaded after
> restarting. The region for the OPENNING state is not reassigned forever, and
> the following log is printed.
> 2019-06-22 16:46:26,210 INFO [master/hdp0:16000] assignment.RegionStateStore:
> Load hbase:meta entry region=379c730490b4848f3db065fb25b87452,
> regionState=OPENING, lastHost=hdp2,16020,1561190163476,
> regionLocation=hdp0,16020,1561190163887, openSeqNum=69448
> ... ...
> 2019-06-22 16:51:28,514 WARN [ProcExecTimeout] assignment.AssignmentManager:
> STUCK Region-In-Transition rit=OPENING, location=hdp0,16020,1561190163887,
> table=test061910, region=379c730490b4848f3db065fb25b87452
> ...
> 2019-06-22 16:49:28,483 WARN [ProcExecTimeout] assignment.AssignmentManager:
> STUCK Region-In-Transition rit=OPENING, location=hdp0,16020,1561190163887,
> table=test061910, region=379c730490b4848f3db065fb25b87452
> (4) The state in zookeeper
>
> test061910,0031457280,1560943902323.379c730490b4848f3db065fb25b87452.
> column=info:regioninfo, timestamp=1561192609882, value=\{ENCODED =>
> 379c730490b4848f3db065fb25b87452, NAME =>
> 'test061910,0031457280,1560943902323.379c730490b4848f3db065fb25b87452.',
> STARTKEY => '0031457280', ENDKEY =>
> '0041943040'}
>
> test061910,0031457280,1560943902323.379c730490b4848f3db065fb25b87452.
> column=info:seqnumDuringOpen, timestamp=1561190288613,
> value=\x00\x00\x00\x00\x00\x01\x0FH
>
> test061910,0031457280,1560943902323.379c730490b4848f3db065fb25b87452.
> column=info:server, timestamp=1561190288613, value=hdp2:16020
>
> test061910,0031457280,1560943902323.379c730490b4848f3db065fb25b87452.
> column=info:serverstartcode, timestamp=1561190288613, value=1561190163476
>
[jira] [Resolved] (HBASE-13814) AssignmentManager does not write the correct server name into Zookeeper when unassign region
[
https://issues.apache.org/jira/browse/HBASE-13814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duo Zhang resolved HBASE-13814.
---
Resolution: Won't Fix
Now we do not use zk assignment for 2.x.
> AssignmentManager does not write the correct server name into Zookeeper when
> unassign region
>
>
> Key: HBASE-13814
> URL: https://issues.apache.org/jira/browse/HBASE-13814
> Project: HBase
> Issue Type: Bug
> Components: Region Assignment
>Affects Versions: 0.94.27
>Reporter: Jianwei Cui
>Priority: Minor
> Attachments: HBASE-13814-0.94-v1.patch, HBASE-13814-0.94-v2.patch
>
>
> When moving region, the region will firstly be unassigned from corresponding
> region server by the method AssignmentManager#unassign(). AssignmentManager
> will write the region info and the server name into Zookeeper by the
> following code:
> {code}
> versionOfClosingNode = ZKAssign.createNodeClosing(
> master.getZooKeeper(), region, master.getServerName());
> {code}
> It seems that the AssignmentManager misuses the master's name as the server
> name. If the ROOT region is being moved and the region server holding the
> ROOT region is just crashed. The Master will try to start a
> MetaServerShutdownHandler if the server is judged as holding meta region. The
> judgment will be done by the method AssignmentManager#isCarryingRegion, and
> the method will firstly check the server name in Zookeeper:
> {code}
> ServerName addressFromZK = (data != null && data.getOrigin() != null) ?
> data.getOrigin() : null;
> if (addressFromZK != null) {
> // if we get something from ZK, we will use the data
> boolean matchZK = (addressFromZK != null &&
> addressFromZK.equals(serverName));
> {code}
> The wrong server name from Zookeeper will make the server not be judged as
> holding the ROOT region. Then, the master will start a ServerShutdownHandler.
> Unlike MetaServerShutdownHandler, the ServerShutdownHandler won't assign ROOT
> region firstly, making the ROOT region won't be assigned forever. In our test
> environment, we encounter this problem when moving ROOT region and stopping
> the region server concurrently.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (HBASE-13771) Replication clients should not access zookeeper directly
[ https://issues.apache.org/jira/browse/HBASE-13771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang resolved HBASE-13771. --- Resolution: Implemented > Replication clients should not access zookeeper directly > > > Key: HBASE-13771 > URL: https://issues.apache.org/jira/browse/HBASE-13771 > Project: HBase > Issue Type: Improvement >Affects Versions: 1.0.1, 1.1.0, 0.98.12, 1.2.0, 2.0.0 >Reporter: Andrew Kyle Purtell >Priority: Critical > > Replication client actions set and modify znodes directly. This is legacy > from an era before we had the AccessController available and missing coverage > of admin action in our security model. > All replication client actions should be mediated by the master, and hooked > up to the coprocessor framework for use by the AccessController. After adding > this functionality we should should restrict access to replication znodes to > only the HBase service principal, but provide a configuration option to relax > those permissions for as long as older admin clients are in use (with a stern > suggestion to upgrade ASAP). This type of functional change, with optional > backwards compatibility, should be fine for all branches. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-19993) Publish tests jar for hbase-zookeeper in bin tarball
[
https://issues.apache.org/jira/browse/HBASE-19993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duo Zhang resolved HBASE-19993.
---
Resolution: Not A Problem
> Publish tests jar for hbase-zookeeper in bin tarball
>
>
> Key: HBASE-19993
> URL: https://issues.apache.org/jira/browse/HBASE-19993
> Project: HBase
> Issue Type: Bug
>Reporter: Apekshit Sharma
>Assignee: Apekshit Sharma
>Priority: Major
>
> Since {{HBTU extends HBZKTU}} (such short forms! i know!), we need to publish
> hbase-zookeeper's tests jar too. Many IT tests use HBTU.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (HBASE-5133) Support zookeeper cluster key related parameters for LoadTestTool
[ https://issues.apache.org/jira/browse/HBASE-5133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang resolved HBASE-5133. -- Resolution: Won't Fix > Support zookeeper cluster key related parameters for LoadTestTool > - > > Key: HBASE-5133 > URL: https://issues.apache.org/jira/browse/HBASE-5133 > Project: HBase > Issue Type: Sub-task > Components: Client, regionserver >Reporter: Ted Yu >Priority: Major > > HBASE-5124 adds zookeeper cluster key related parameters for LoadTestTool > This task ports them to TRUNK. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-4914) Enhance MapReduce TableInputFormat to Support N-mappers per Region
[ https://issues.apache.org/jira/browse/HBASE-4914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang resolved HBASE-4914. -- Resolution: Won't Fix > Enhance MapReduce TableInputFormat to Support N-mappers per Region > -- > > Key: HBASE-4914 > URL: https://issues.apache.org/jira/browse/HBASE-4914 > Project: HBase > Issue Type: Sub-task > Components: Client, regionserver >Reporter: Nicolas Spiegelberg >Priority: Major > > Current TableInputFormat based MR jobs create exactly one mapper per region > where each mapper sets one Scan with appropriate start/stop row keys. This > change allows jobs to be run with any number of mappers per region, so that > when a mapper fails, there will be less data to be reprocessed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-4910) thrift scannerstopwithfilter not honoring stop row
[ https://issues.apache.org/jira/browse/HBASE-4910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang resolved HBASE-4910. -- Resolution: Won't Fix > thrift scannerstopwithfilter not honoring stop row > -- > > Key: HBASE-4910 > URL: https://issues.apache.org/jira/browse/HBASE-4910 > Project: HBase > Issue Type: Sub-task > Components: Client, regionserver, Thrift >Reporter: Nicolas Spiegelberg >Priority: Major > > Moving issue w/ no recent movement out of 0.95 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-10345) HMaster should not serve when disconnected with ZooKeeper
[
https://issues.apache.org/jira/browse/HBASE-10345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duo Zhang resolved HBASE-10345.
---
Resolution: Later
Now after we make use the master local region, the old master can not even
process a region assignment as it can not store procedures any more, so the
problem is less hurt now. Resolve for now. Feel free to reopen if you have new
ideas.
Thanks.
> HMaster should not serve when disconnected with ZooKeeper
> -
>
> Key: HBASE-10345
> URL: https://issues.apache.org/jira/browse/HBASE-10345
> Project: HBase
> Issue Type: Bug
> Components: master
>Affects Versions: 0.94.3
>Reporter: chendihao
>Priority: Major
>
> Refer to HBASE-9468(Previous active master can still serves RPC request when
> it is trying recovering expired zk session), we can fail fast to avoid
> existing double masters at the same time. But this problem may occur before
> session expired. When receive Disconnected event, we can't make sure of that
> this active master can communicate with zk later. And it doesn't know whether
> backup master has become the new active master or not until it receives
> Expired event(which may lose forever). During this
> unsure-who-is-active-master period, the current active master should not
> serve(maybe turn off RpcServer).
> Here is the statement from "ZooKeeper Distributed Process Coordination" P101
> {quote}
> If the developer is not careful, the old leader will continue to act as a
> leader and may take actions that conflict with those of the new leader. For
> this reason, when a process receives a Disconnected event, the process should
> suspend actions taken as a leader until it reconnects. Normally this
> reconnect happens very quickly.
> {quote}
> So it's equally necessary to handle Disconnected event and Expired event.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-15154) Master puts up a blockcache instance
[ https://issues.apache.org/jira/browse/HBASE-15154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17357882#comment-17357882 ] Duo Zhang commented on HBASE-15154: --- Is this still true? For now we need a memstore on master as we have a local region to store procedures. > Master puts up a blockcache instance > > > Key: HBASE-15154 > URL: https://issues.apache.org/jira/browse/HBASE-15154 > Project: HBase > Issue Type: Bug >Reporter: Michael Stack >Priority: Major > Fix For: 3.0.0-alpha-1 > > > Master is putting up a blockcache instance. It shouldn't. This issue comes of > the unification of Master and RegionServer where Master is a subclass of > RegionServer. Our [~jmspaggi] found out the hard way today when he tried to > bring up a cluster with a large offheap cache only the master member had been > given less memory. > Marking critical rather than blocker because there is a means of configuring > your way out of this little entanglement; set hbase.bucketcache.size to zero > in the master config only. > If you want to set direct memory for regionservers only and not on the > master, you should do the following: > Set HBASE_OFFHEAPSIZE=0G > ... and then turn it on for RegionServers only: > HBASE_REGIONSERVER_OPTS="HBASE_REGIONSERVER_OPTS > -XX:MaxDirectMemorySize=XXXG" where XXX is size of the direct memory you want > to allocate. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-15041) Clean up javadoc errors and reenable jdk8 linter
[ https://issues.apache.org/jira/browse/HBASE-15041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17357881#comment-17357881 ] Duo Zhang commented on HBASE-15041: --- I think this has already been fixed by [~ndimiduk]? > Clean up javadoc errors and reenable jdk8 linter > > > Key: HBASE-15041 > URL: https://issues.apache.org/jira/browse/HBASE-15041 > Project: HBase > Issue Type: Umbrella > Components: build, documentation >Affects Versions: 1.2.0, 1.3.0, 2.0.0 >Reporter: Sean Busbey >Assignee: Chia-Ping Tsai >Priority: Major > Fix For: 3.0.0-alpha-1 > > > umbrella to clean up our various errors according to the jdk8 javadoc linter. > plan is a sub-task per module. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-13643) Follow Google to get more 9's
[ https://issues.apache.org/jira/browse/HBASE-13643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-13643: -- Fix Version/s: (was: 3.0.0-alpha-1) 3.0.0-alpha-2 > Follow Google to get more 9's > - > > Key: HBASE-13643 > URL: https://issues.apache.org/jira/browse/HBASE-13643 > Project: HBase > Issue Type: Improvement >Reporter: Elliott Neil Clark >Priority: Major > Fix For: 3.0.0-alpha-2 > > > Ideas taken shamelessly from Google's HBasecon talk > (http://hbasecon.com/agenda/): > On failover all regions are unavailable for reads (and sometime writes) until > after all write ahead logs have been recovered. To combat that the last > flushed seqid is kept around. > Google took this one step farther and set some regions (Tablets in BigTable) > as read only. Setting a region as read only means there's no memstore. No > need to flush before move, split, or merge. > In addition to the wins that Google got, HBase would also be able to shed > some memory pressure. Right now every region gets a memstore and with that > memstore comes a mslab. Read only regions would not need these added object. > This should allow a regionserver to host lots of cold regions without too > much memory pressure. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-12943) Set sun.net.inetaddr.ttl in HBase
[ https://issues.apache.org/jira/browse/HBASE-12943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17357880#comment-17357880 ] Duo Zhang commented on HBASE-12943: --- Unschedule for 3.0.0-alpha-1, as it is not a problem for JDK8+ now. > Set sun.net.inetaddr.ttl in HBase > - > > Key: HBASE-12943 > URL: https://issues.apache.org/jira/browse/HBASE-12943 > Project: HBase > Issue Type: Bug >Reporter: Shaohui Liu >Assignee: Shaohui Liu >Priority: Major > Fix For: 1.8.0 > > Attachments: 12943-1-master.txt > > > The default value of config: sun.net.inetaddr.ttl is -1 and the java > processes will cache the mapping of hostname to ip address forever, See: > http://docs.oracle.com/javase/7/docs/technotes/guides/net/properties.html > But things go wrong when a regionserver with same hostname and different ip > address rejoins the hbase cluster. The HMaster will get wrong ip address of > the regionserver from this cache and every region assignment to this > regionserver will be blocked for a time because the HMaster can't communicate > with the regionserver. > A tradeoff is to set the sun.net.inetaddr.ttl to 10m or 1h and make the wrong > cache expired. > Suggestions are welcomed. Thanks~ -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-13503) Encryption improvements
[ https://issues.apache.org/jira/browse/HBASE-13503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-13503: -- Fix Version/s: (was: 3.0.0-alpha-1) 3.0.0-alpha-2 > Encryption improvements > --- > > Key: HBASE-13503 > URL: https://issues.apache.org/jira/browse/HBASE-13503 > Project: HBase > Issue Type: Umbrella > Components: encryption, security >Reporter: Andrew Kyle Purtell >Priority: Major > Fix For: 3.0.0-alpha-2 > > > Umbrella issue for a collection of related encryption improvements: > - Additional ciphers > - Better data key derivation > - Support master key rotation without process restarts (where applicable) > - Additional KeyProviders with backwards compatible interface evolution -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-12943) Set sun.net.inetaddr.ttl in HBase
[ https://issues.apache.org/jira/browse/HBASE-12943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-12943: -- Fix Version/s: (was: 3.0.0-alpha-1) > Set sun.net.inetaddr.ttl in HBase > - > > Key: HBASE-12943 > URL: https://issues.apache.org/jira/browse/HBASE-12943 > Project: HBase > Issue Type: Bug >Reporter: Shaohui Liu >Assignee: Shaohui Liu >Priority: Major > Fix For: 1.8.0 > > Attachments: 12943-1-master.txt > > > The default value of config: sun.net.inetaddr.ttl is -1 and the java > processes will cache the mapping of hostname to ip address forever, See: > http://docs.oracle.com/javase/7/docs/technotes/guides/net/properties.html > But things go wrong when a regionserver with same hostname and different ip > address rejoins the hbase cluster. The HMaster will get wrong ip address of > the regionserver from this cache and every region assignment to this > regionserver will be blocked for a time because the HMaster can't communicate > with the regionserver. > A tradeoff is to set the sun.net.inetaddr.ttl to 10m or 1h and make the wrong > cache expired. > Suggestions are welcomed. Thanks~ -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-11124) Pluggable major compaction strategy
[ https://issues.apache.org/jira/browse/HBASE-11124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-11124: -- Fix Version/s: (was: 3.0.0-alpha-1) 3.0.0-alpha-2 > Pluggable major compaction strategy > --- > > Key: HBASE-11124 > URL: https://issues.apache.org/jira/browse/HBASE-11124 > Project: HBase > Issue Type: New Feature >Reporter: Andrew Kyle Purtell >Priority: Major > Fix For: 3.0.0-alpha-2 > > > Consider adding to the master a schedulable internal task that manages major > compaction. The scheduler implementation should be pluggable. The default > plugin should implement the current policy, a simple interval trigger > governed by a site configuration property. Reserve a znode as a store for > major compaction scheduler state, allow the plugin access to (re)create it > and update as needed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-5313) Restructure hfiles layout for better compression
[ https://issues.apache.org/jira/browse/HBASE-5313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-5313: - Fix Version/s: (was: 3.0.0-alpha-1) 3.0.0-alpha-2 > Restructure hfiles layout for better compression > > > Key: HBASE-5313 > URL: https://issues.apache.org/jira/browse/HBASE-5313 > Project: HBase > Issue Type: Improvement > Components: HFile, io >Reporter: Dhruba Borthakur >Priority: Major > Fix For: 3.0.0-alpha-2 > > > A HFile block contain a stream of key-values. Can we can organize these kvs > on the disk in a better way so that we get much greater compression ratios? > One option (thanks Prakash) is to store all the keys in the beginning of the > block (let's call this the key-section) and then store all their > corresponding values towards the end of the block. This will allow us to > not-even decompress the values when we are scanning and skipping over rows in > the block. > Any other ideas? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-24362) Remove/Retain deprecated configs
[ https://issues.apache.org/jira/browse/HBASE-24362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-24362: -- Fix Version/s: (was: 3.0.0-alpha-1) 3.0.0-alpha-2 > Remove/Retain deprecated configs > > > Key: HBASE-24362 > URL: https://issues.apache.org/jira/browse/HBASE-24362 > Project: HBase > Issue Type: Umbrella >Reporter: Anoop Sam John >Priority: Critical > Fix For: 3.0.0-alpha-2 > > > This umbrella issue is to discuss and decide on each config which are > deprecated in 2.x line (Or even before itself). Whether we can completely > remove them from code. If so whether we need have some migration tool/step. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-21163) Support backup-and-restore operations without Hbase Super user privilege
[ https://issues.apache.org/jira/browse/HBASE-21163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17357879#comment-17357879 ] Duo Zhang commented on HBASE-21163: --- Any updates? > Support backup-and-restore operations without Hbase Super user privilege > > > Key: HBASE-21163 > URL: https://issues.apache.org/jira/browse/HBASE-21163 > Project: HBase > Issue Type: Improvement > Components: backuprestore >Reporter: Sujit P >Assignee: Vladimir Rodionov >Priority: Critical > Labels: Backup/Restore > Fix For: 3.0.0-alpha-1 > > > Hello Team, > I am opening this Apache Jira to request for an analysis on considering > following problem statement: > Currently backup-and-restore utility is designed to work with "hbase" > superuser privileges. > I see at-least couple concerns on that, may be more, will add more later on: > * For smaller organizations with less than 20 hbase tables or couple of > clusters, it is manageable, hbase admins. However, for larger organizations > or larger clusters, that would need providing hbase super user access to many > people to manage such operations which can be a security risk on source > cluster. > * In certain scenarios, it may be typical to have one DR Cluster in remote > data center to store backup tables, and having super privileges for all > tables in remote cluster is another risk for same reasons above. > I suggest to review into making backup and restore without hbase super > privileges . > Tenants or application admins may have certainly have admin access to > relevant tables/namespaces/snapshots. > Here is an example on what I am proposing from RDBMS : > [https://docs.oracle.com/cd/E16926_01/doc.121/e16564/configure_users_classes.htm#OBADM144] > Thanks > > PS: Forgive me if I hadn't opened my second apache Jira correct way, happy to > correct it. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-21112) [Auth] IPC client fallback to simple auth (forward-port to master)
[ https://issues.apache.org/jira/browse/HBASE-21112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17357878#comment-17357878 ] Duo Zhang commented on HBASE-21112: --- Any updates? > [Auth] IPC client fallback to simple auth (forward-port to master) > -- > > Key: HBASE-21112 > URL: https://issues.apache.org/jira/browse/HBASE-21112 > Project: HBase > Issue Type: Bug > Components: Client, IPC/RPC, security >Affects Versions: 2.1.0, 2.0.2 >Reporter: Jack Bearden >Priority: Critical > Fix For: 3.0.0-alpha-1 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-19855) Refactor RegionScannerImpl.nextInternal method
[ https://issues.apache.org/jira/browse/HBASE-19855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-19855: -- Status: Open (was: Patch Available) > Refactor RegionScannerImpl.nextInternal method > -- > > Key: HBASE-19855 > URL: https://issues.apache.org/jira/browse/HBASE-19855 > Project: HBase > Issue Type: Sub-task >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang >Priority: Major > Attachments: HBASE-19855.master.002.patch, > HBASE-19855.master.003.patch, HBASE-19855.master.004.patch, > HBASE-19855.master.004.patch > > > Now this method is too complicated and confusing... > https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-20525) Refactoring the code of read path
[ https://issues.apache.org/jira/browse/HBASE-20525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-20525: -- Fix Version/s: (was: 3.0.0-alpha-1) 3.0.0-alpha-2 > Refactoring the code of read path > - > > Key: HBASE-20525 > URL: https://issues.apache.org/jira/browse/HBASE-20525 > Project: HBase > Issue Type: Umbrella > Components: Scanners >Reporter: Duo Zhang >Priority: Critical > Fix For: 3.0.0-alpha-2 > > > The known problems of the current implementation: > 1. 'Seek or skip' should be decided at StoreFileScanner level, not > StoreScanner. > 2. As now we support creating multiple StoreFileReader instances for a single > HFile, we do not need to load the file info and other meta infos every time > when creating a new StoreFileReader instance. > 3. 'Pread or stream' should be decided at StoreFileScanner level, not > StoreScanner. > 4. Make sure that we can return at any point during a scan, at least when > filterRowKey we can not stop until we reach the next row, no matter how many > cells we need to skip... > 5. Doing bytes comparing everywhere, where we need to know if there is a row > change, a family change, a qualifier change, etc. This is a performance > killer. > And the most important thing is that, the code is way too complicated now and > become out of control... > This should be done before our 3.0.0 release. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-20179) [TESTING] Scale
[ https://issues.apache.org/jira/browse/HBASE-20179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17357876#comment-17357876 ] Duo Zhang commented on HBASE-20179: --- Let's resolve this issue? It is for 2.0? > [TESTING] Scale > --- > > Key: HBASE-20179 > URL: https://issues.apache.org/jira/browse/HBASE-20179 > Project: HBase > Issue Type: Umbrella > Components: test >Reporter: Michael Stack >Priority: Critical > Fix For: 3.0.0-alpha-1 > > > Umbrella issue for scale testing for 2.0. > At least keep account of what testing has been done in here. > TODO as subtasks > * Long-running test > * ITBLL w/o killing Master > * Big ITBLL (1B works, 10B needs more verification, 100B, and 1T). > * Many Regions (43k regions takes about 12 minutes on a 6node cluster to > deploy and 3 1/2 minutes to go down -- needs tuning). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-20413) IntegrationTestRegionReplicaReplication fails
[ https://issues.apache.org/jira/browse/HBASE-20413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17357877#comment-17357877 ] Duo Zhang commented on HBASE-20413: --- Any updates here? > IntegrationTestRegionReplicaReplication fails > - > > Key: HBASE-20413 > URL: https://issues.apache.org/jira/browse/HBASE-20413 > Project: HBase > Issue Type: Bug > Components: test >Reporter: Michael Stack >Assignee: Sreeram Venkatasubramanian >Priority: Critical > Fix For: 3.0.0-alpha-1 > > > Following [~psomogyi]'s inspiration, I've been running unit tests on a GCE > instance. IntegrationTestRegionReplicaReplication fails always for > branch-2.0. It fails for branch-1 too but for a different looking reason. On > branch-2.0, its OOME'ing. Downing the size of ingested data, it seems to hang > unable to update because it is unable to flush. > Marking this a blocker for now. Fix or disable it for branch-2.0. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-20324) Hbase master fails to become active in kerberos environment
[ https://issues.apache.org/jira/browse/HBASE-20324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang resolved HBASE-20324. --- Resolution: Incomplete Resolved as incomplete for now. Feel free to reopen if you have more details. Thanks. > Hbase master fails to become active in kerberos environment > --- > > Key: HBASE-20324 > URL: https://issues.apache.org/jira/browse/HBASE-20324 > Project: HBase > Issue Type: Bug > Environment: Hbase 2.0.0-beta2 > zookeeper-3.5.3-beta > 3 nodes Env > Kdc server on namenode > *hadoop-2.7.3* > *--Configured with keytabs(abhishekk1/2/3 are nodes)* > *abhishekk1 is namenode/hmaster* > *abhishekk2/3 are datanodes/regionservers* >Reporter: Abhishek Kulkarni >Priority: Critical > Attachments: hbase-root-master-.log, hbase-root-regionserver.log > > > > [^hbase-root-master-.log] > ^[^hbase-root-regionserver.log]^ > > ^^Trying to resolve this form last one month with different forums but not > able to resovleat all.^^ -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-20324) Hbase master fails to become active in kerberos environment
[ https://issues.apache.org/jira/browse/HBASE-20324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-20324: -- Fix Version/s: (was: 3.0.0-alpha-1) > Hbase master fails to become active in kerberos environment > --- > > Key: HBASE-20324 > URL: https://issues.apache.org/jira/browse/HBASE-20324 > Project: HBase > Issue Type: Bug > Environment: Hbase 2.0.0-beta2 > zookeeper-3.5.3-beta > 3 nodes Env > Kdc server on namenode > *hadoop-2.7.3* > *--Configured with keytabs(abhishekk1/2/3 are nodes)* > *abhishekk1 is namenode/hmaster* > *abhishekk2/3 are datanodes/regionservers* >Reporter: Abhishek Kulkarni >Priority: Critical > Attachments: hbase-root-master-.log, hbase-root-regionserver.log > > > > [^hbase-root-master-.log] > ^[^hbase-root-regionserver.log]^ > > ^^Trying to resolve this form last one month with different forums but not > able to resovleat all.^^ -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-20315) Document post release process steps for RM
[ https://issues.apache.org/jira/browse/HBASE-20315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-20315: -- Fix Version/s: (was: 3.0.0-alpha-1) 3.0.0-alpha-2 > Document post release process steps for RM > -- > > Key: HBASE-20315 > URL: https://issues.apache.org/jira/browse/HBASE-20315 > Project: HBase > Issue Type: Task > Components: build, documentation >Reporter: Andrew Kyle Purtell >Priority: Critical > Fix For: 3.0.0-alpha-2 > > > We should document post release steps that RMs have to take and add it to the > 'How To Release' section of the refguide. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-18562) [AMv2] expireServers and ServerCrashProcedure cleanup
[ https://issues.apache.org/jira/browse/HBASE-18562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17357871#comment-17357871 ] Duo Zhang commented on HBASE-18562: --- I think the method and the procedure have been changed a lot since then. Still needed? > [AMv2] expireServers and ServerCrashProcedure cleanup > - > > Key: HBASE-18562 > URL: https://issues.apache.org/jira/browse/HBASE-18562 > Project: HBase > Issue Type: Bug > Components: Region Assignment >Reporter: Michael Stack >Priority: Critical > > In review of HBASE-18551, [~uagashe] posed a scenario that revealed a hole in > our processing of unassigns; there is case where a UP might not get > notification from ServerCrashProcedure if the UP is scheduled AFTER a SCP has > gotten past its handleRIT call (No new SCP will be queued because > expireServer won't let it happen if crashed server is in dead server list > which it will be). > Chatting on it, expireServers is doing checks that belong inside > ServerCrashProcedure. expireServers scheduling an SCP each time it is called > would make it so SCP processing is serialized one behind the other. If the > first does the clean up all subsequent will do no work but Procedures > dependent on them will get their wakeup call. > This issue is about implementing the above cleanup. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-20118) AMv2: Next Steps
[ https://issues.apache.org/jira/browse/HBASE-20118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17357872#comment-17357872 ] Duo Zhang commented on HBASE-20118: --- Do we still need this issue? > AMv2: Next Steps > > > Key: HBASE-20118 > URL: https://issues.apache.org/jira/browse/HBASE-20118 > Project: HBase > Issue Type: Umbrella > Components: amv2 >Reporter: Michael Stack >Priority: Critical > Fix For: 3.0.0-alpha-1 > > > AMv2 is not done. There is more to do (I was reminded of it reading the TODOs > on this doc > https://docs.google.com/document/d/1eVKa7FHdeoJ1-9o8yZcOTAQbv0u0bblBlCCzVSIn69g/edit#heading=h.odwf1ja2iutv). > For example: > * ProcedureSyncWait, the main runner, is synchronous and is supposed to be > replaced with an async runner. It has us waiting on events. Unnecessary. > Replace with ProcedureYieldException. > Will fill out subtasks. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-19562) Purge mirror writing of region and table info into fs at .tableinfo and .regioninfo
[ https://issues.apache.org/jira/browse/HBASE-19562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17357870#comment-17357870 ] Duo Zhang commented on HBASE-19562: --- Any updates here? > Purge mirror writing of region and table info into fs at .tableinfo and > .regioninfo > --- > > Key: HBASE-19562 > URL: https://issues.apache.org/jira/browse/HBASE-19562 > Project: HBase > Issue Type: Bug > Components: Filesystem Integration >Reporter: Michael Stack >Assignee: Michael Stack >Priority: Critical > Fix For: 3.0.0-alpha-1 > > Attachments: > 0002-HBASE-19562-Purge-mirror-writing-of-region-and-table.patch > > > We don't use these files in hbase2 yet we keep writing them when we create a > table or region. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-19535) Introduce a new annotation to say there is no BC guarantee in the inheritance
[ https://issues.apache.org/jira/browse/HBASE-19535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-19535: -- Fix Version/s: (was: 3.0.0-alpha-1) 3.0.0-alpha-2 > Introduce a new annotation to say there is no BC guarantee in the inheritance > - > > Key: HBASE-19535 > URL: https://issues.apache.org/jira/browse/HBASE-19535 > Project: HBase > Issue Type: Task >Reporter: Chia-Ping Tsai >Priority: Critical > Fix For: 3.0.0-alpha-2 > > > We have added many Public "read-only" interface in 2.0 to save user from the > dangerous operations. For example, TableDescriptor, ColumnFamilyDescriptor, > Region, Store, etc. However, this change make us be hard to enhance these > interface (ie. add the new APIs) because the BC will be broken for the user > having the subclass of these interfaces. In contrast with Cell, Filter, and > coprocessor, these new interfaces are NOT designed to be a extendable class > for user but we have no explicit caveat. As I see it, it is necessary to > introduce an new annotation to explicitly say "you CAN extend this Public > class with the BC guarantee". On the other hand, user should not extend the > Public classes which don't have the such annotation. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-19533) How to do controlled shutdown in branch-2?
[ https://issues.apache.org/jira/browse/HBASE-19533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17357868#comment-17357868 ] Duo Zhang commented on HBASE-19533: --- Still a problem? > How to do controlled shutdown in branch-2? > -- > > Key: HBASE-19533 > URL: https://issues.apache.org/jira/browse/HBASE-19533 > Project: HBase > Issue Type: Task >Reporter: Michael Stack >Priority: Critical > Fix For: 3.0.0-alpha-1 > > > Before HBASE-18946, setting shutdown of a cluster, the Master would exit > immediately. RegionServers would run region closes and then try and notify > the Master of the close and would spew exceptions that the Master was > unreachable. > This is different to how branch-1 used to do it. It used to keep Master up > and it would be like the captain of the ship, the last to go down. As of > HBASE-18946, this is again the case but there are still open issues. > # Usually Master does all open and close of regions. On cluster shutdown, it > is the one time where the Regions run the region close. Currently, the > regions report the close to the Master which disregards the message since it > did not start the region closes. Should we do different? Try and update state > in hbase:meta setting it to CLOSE? We might not be able to write CLOSE for > all regions since hbase:meta will be closing too (the RS that is hosting > hbase:meta will close it last but that may not be enough). > # Should the Master run the cluster shutdown sending out close for all > regions? What if cluster of 1M regions? Untenable? Send a message per server? > That might be better. > Anyways, this needs attention. Filing issue in meantime. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #3357: HBASE-25891 remove-dependence-storing-wal-filenames-for-backup
Apache-HBase commented on pull request #3357: URL: https://github.com/apache/hbase/pull/3357#issuecomment-855249815 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 27s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. | | +0 :ok: | prototool | 0m 0s | prototool was not available. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 22s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 56s | master passed | | +1 :green_heart: | compile | 2m 5s | master passed | | +1 :green_heart: | checkstyle | 0m 28s | master passed | | +1 :green_heart: | spotbugs | 4m 16s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 36s | the patch passed | | +1 :green_heart: | compile | 2m 2s | the patch passed | | +1 :green_heart: | cc | 2m 2s | the patch passed | | -0 :warning: | javac | 0m 38s | hbase-backup generated 3 new + 72 unchanged - 7 fixed = 75 total (was 79) | | +1 :green_heart: | checkstyle | 0m 27s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 18m 19s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | hbaseprotoc | 1m 24s | the patch passed | | +1 :green_heart: | spotbugs | 4m 49s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 27s | The patch does not generate ASF License warnings. | | | | 51m 7s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3357/2/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3357 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile cc hbaseprotoc prototool | | uname | Linux d01bf5f47f86 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / de06e20e0a | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | javac | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3357/2/artifact/yetus-general-check/output/diff-compile-javac-hbase-backup.txt | | Max. process+thread count | 96 (vs. ulimit of 3) | | modules | C: hbase-protocol-shaded hbase-backup U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3357/2/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-19529) Handle null states in AM
[ https://issues.apache.org/jira/browse/HBASE-19529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17357867#comment-17357867 ] Duo Zhang commented on HBASE-19529: --- Do we still need to handle this? > Handle null states in AM > > > Key: HBASE-19529 > URL: https://issues.apache.org/jira/browse/HBASE-19529 > Project: HBase > Issue Type: Sub-task >Reporter: Apekshit Sharma >Priority: Critical > Fix For: 3.0.0-alpha-1 > > > From debugging in HBASE-19457, found some questions that need concrete > answers: > 1) What does a region state of null in meta means? Currently AM treats it as > OFFLINE > 2) What does a table state of null in meta means? Currently TSM treats it as > ENABLED. > More importantly, we need to fix holes in AM so that our state machine is > well defined and doesn't end up in these uncertainties. > Figuring out answers to above questions will help in that direction. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-23634) Enable "Split WAL to HFile" by default
[ https://issues.apache.org/jira/browse/HBASE-23634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-23634: -- Fix Version/s: (was: 3.0.0-alpha-1) 3.0.0-alpha-2 > Enable "Split WAL to HFile" by default > -- > > Key: HBASE-23634 > URL: https://issues.apache.org/jira/browse/HBASE-23634 > Project: HBase > Issue Type: Task >Affects Versions: 3.0.0-alpha-1, 2.3.0 >Reporter: Guanghao Zhang >Priority: Blocker > Fix For: 3.0.0-alpha-2 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-6189) Snappy build instructions are out of date
[ https://issues.apache.org/jira/browse/HBASE-6189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-6189: - Fix Version/s: (was: 3.0.0-alpha-1) 3.0.0-alpha-2 > Snappy build instructions are out of date > - > > Key: HBASE-6189 > URL: https://issues.apache.org/jira/browse/HBASE-6189 > Project: HBase > Issue Type: Bug > Components: build, documentation, Performance >Affects Versions: 0.92.1, 0.94.0, 0.95.2 >Reporter: Dave Revell >Assignee: Sean Busbey >Priority: Critical > Fix For: 3.0.0-alpha-2 > > > In the ref guide (http://hbase.apache.org/book.html#build.snappy), it says to > build snappy by passing -Dsnappy. Something's wrong here, because: > 1. this causes the build to fail because the hadoop-snappy tar artifact can't > be found by maven > 2. the snappy classes are already included in hadoop 1.0, so using the snappy > profile is unnecessary > It would be great if someone who knows when/why to use the snappy profile > could fix the instructions (and fix the POM if necessary). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #3357: HBASE-25891 remove-dependence-storing-wal-filenames-for-backup
Apache-HBase commented on pull request #3357: URL: https://github.com/apache/hbase/pull/3357#issuecomment-855249467 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 11s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 9s | master passed | | +1 :green_heart: | compile | 1m 12s | master passed | | +1 :green_heart: | shadedjars | 9m 24s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 34s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 28s | the patch passed | | +1 :green_heart: | compile | 1m 21s | the patch passed | | +1 :green_heart: | javac | 1m 21s | the patch passed | | +1 :green_heart: | shadedjars | 9m 4s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 28s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 51s | hbase-protocol-shaded in the patch passed. | | +1 :green_heart: | unit | 13m 46s | hbase-backup in the patch passed. | | | | 48m 22s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3357/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3357 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 4b2216417543 4.15.0-142-generic #146-Ubuntu SMP Tue Apr 13 01:11:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / de06e20e0a | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3357/2/testReport/ | | Max. process+thread count | 3186 (vs. ulimit of 3) | | modules | C: hbase-protocol-shaded hbase-backup U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3357/2/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3357: HBASE-25891 remove-dependence-storing-wal-filenames-for-backup
Apache-HBase commented on pull request #3357: URL: https://github.com/apache/hbase/pull/3357#issuecomment-855248597 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 27s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 11s | master passed | | +1 :green_heart: | compile | 1m 25s | master passed | | +1 :green_heart: | shadedjars | 8m 17s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 35s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 13s | the patch passed | | +1 :green_heart: | compile | 1m 27s | the patch passed | | +1 :green_heart: | javac | 1m 27s | the patch passed | | +1 :green_heart: | shadedjars | 8m 10s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 35s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 0s | hbase-protocol-shaded in the patch passed. | | +1 :green_heart: | unit | 8m 49s | hbase-backup in the patch passed. | | | | 41m 7s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3357/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3357 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 7b07e2f51770 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / de06e20e0a | | Default Java | AdoptOpenJDK-11.0.10+9 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3357/2/testReport/ | | Max. process+thread count | 5060 (vs. ulimit of 3) | | modules | C: hbase-protocol-shaded hbase-backup U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3357/2/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-25930) Thrift does not support requests in Kerberos environment
[
https://issues.apache.org/jira/browse/HBASE-25930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17357860#comment-17357860
]
Hudson commented on HBASE-25930:
Results for branch master
[build #315 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/315/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/315/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/315/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/315/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Thrift does not support requests in Kerberos environment
>
>
> Key: HBASE-25930
> URL: https://issues.apache.org/jira/browse/HBASE-25930
> Project: HBase
> Issue Type: Bug
> Components: Thrift
>Affects Versions: 3.0.0-alpha-1, 2.2.6, 2.5.0, 2.3.5, 2.4.3
>Reporter: haoning
>Assignee: haoning
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0, 2.3.6, 2.2.8, 2.4.4
>
> Attachments: image-2021-05-27-20-17-48-466.png
>
>
> In Kerberos environment, running DemoClient to request thrift server failed
>
> !image-2021-05-27-20-17-48-466.png!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #3357: HBASE-25891 remove-dependence-storing-wal-filenames-for-backup
Apache-HBase commented on pull request #3357: URL: https://github.com/apache/hbase/pull/3357#issuecomment-855242875 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 10m 10s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 22s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 30s | master passed | | +1 :green_heart: | compile | 1m 21s | master passed | | +1 :green_heart: | shadedjars | 9m 21s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 35s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 17s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 32s | the patch passed | | +1 :green_heart: | compile | 1m 20s | the patch passed | | +1 :green_heart: | javac | 1m 20s | the patch passed | | +1 :green_heart: | shadedjars | 9m 30s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 31s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 53s | hbase-protocol-shaded in the patch passed. | | -1 :x: | unit | 14m 12s | hbase-backup in the patch failed. | | | | 58m 58s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3357/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3357 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 0e0979f842b6 4.15.0-142-generic #146-Ubuntu SMP Tue Apr 13 01:11:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / de06e20e0a | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3357/1/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-backup.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3357/1/testReport/ | | Max. process+thread count | 3972 (vs. ulimit of 3) | | modules | C: hbase-protocol-shaded hbase-backup U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3357/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3357: HBASE-25891 remove-dependence-storing-wal-filenames-for-backup
Apache-HBase commented on pull request #3357: URL: https://github.com/apache/hbase/pull/3357#issuecomment-855241949 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 28s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +0 :ok: | prototool | 0m 1s | prototool was not available. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 24s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 50s | master passed | | +1 :green_heart: | compile | 2m 7s | master passed | | +1 :green_heart: | checkstyle | 0m 28s | master passed | | +1 :green_heart: | spotbugs | 4m 12s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 42s | the patch passed | | +1 :green_heart: | compile | 2m 6s | the patch passed | | +1 :green_heart: | cc | 2m 6s | the patch passed | | -0 :warning: | javac | 0m 38s | hbase-backup generated 3 new + 72 unchanged - 7 fixed = 75 total (was 79) | | +1 :green_heart: | checkstyle | 0m 27s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 18m 7s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | hbaseprotoc | 1m 25s | the patch passed | | +1 :green_heart: | spotbugs | 4m 56s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 26s | The patch does not generate ASF License warnings. | | | | 51m 3s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3357/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3357 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile cc hbaseprotoc prototool | | uname | Linux 457163287d11 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / de06e20e0a | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | javac | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3357/1/artifact/yetus-general-check/output/diff-compile-javac-hbase-backup.txt | | Max. process+thread count | 96 (vs. ulimit of 3) | | modules | C: hbase-protocol-shaded hbase-backup U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3357/1/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3357: HBASE-25891 remove-dependence-storing-wal-filenames-for-backup
Apache-HBase commented on pull request #3357: URL: https://github.com/apache/hbase/pull/3357#issuecomment-855240762 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 36s | Docker mode activated. | | -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 26s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 23s | master passed | | +1 :green_heart: | compile | 1m 25s | master passed | | +1 :green_heart: | shadedjars | 8m 14s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 36s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 16s | the patch passed | | +1 :green_heart: | compile | 1m 24s | the patch passed | | +1 :green_heart: | javac | 1m 23s | the patch passed | | +1 :green_heart: | shadedjars | 8m 10s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 35s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 0s | hbase-protocol-shaded in the patch passed. | | -1 :x: | unit | 9m 20s | hbase-backup in the patch failed. | | | | 42m 11s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3357/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3357 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 0b82c27169b5 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / de06e20e0a | | Default Java | AdoptOpenJDK-11.0.10+9 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3357/1/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-backup.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3357/1/testReport/ | | Max. process+thread count | 5431 (vs. ulimit of 3) | | modules | C: hbase-protocol-shaded hbase-backup U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3357/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] rda3mon opened a new pull request #3357: remove-dependence-storing-wal-filenames-for-backup
rda3mon opened a new pull request #3357: URL: https://github.com/apache/hbase/pull/3357 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] rda3mon closed pull request #3342: HBASE-25891 Remove dependence on storing wal filenames for incremental backup
rda3mon closed pull request #3342: URL: https://github.com/apache/hbase/pull/3342 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3353: HBASE-25969 Purge netty-all transitive includes
Apache-HBase commented on pull request #3353: URL: https://github.com/apache/hbase/pull/3353#issuecomment-855225595 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 11s | Docker mode activated. | | -0 :warning: | yetus | 0m 7s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2.4 Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 4s | branch-2.4 passed | | +1 :green_heart: | compile | 2m 24s | branch-2.4 passed | | +1 :green_heart: | shadedjars | 6m 38s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 2m 26s | branch-2.4 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 35s | the patch passed | | +1 :green_heart: | compile | 2m 24s | the patch passed | | +1 :green_heart: | javac | 2m 24s | the patch passed | | +1 :green_heart: | shadedjars | 6m 32s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 2m 21s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 383m 58s | root in the patch passed. | | | | 417m 52s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3353/3/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3353 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 9db2e4e82def 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2.4 / 7102dd0654 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3353/3/testReport/ | | Max. process+thread count | 2982 (vs. ulimit of 12500) | | modules | C: . U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3353/3/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (HBASE-25973) Balancer should explain progress in a better way in log
[ https://issues.apache.org/jira/browse/HBASE-25973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhuoyue Huang reassigned HBASE-25973: - Assignee: (was: Zhuoyue Huang) > Balancer should explain progress in a better way in log > --- > > Key: HBASE-25973 > URL: https://issues.apache.org/jira/browse/HBASE-25973 > Project: HBase > Issue Type: Bug > Components: Balancer >Affects Versions: 3.0.0-alpha-1 >Reporter: Clara Xiong >Priority: Major > > In the log, balancer logs at info level at the beginning of run: > > balancer.StochasticLoadBalancer: start StochasticLoadBalancer.balancer, > initCost=277.3479243125063, functionCost=RegionCountSkewCostFunction : > (500.0, 0.3749771215224234); ServerLocalityCostFunction : (25.0, > 0.5807483226644186); RackLocalityCostFunction : (15.0, 0.0); > TableSkewCostFunction : (1000.0, 0.0019704142954972883); > StoreFileCostFunction : (200.0, 0.3668512059459341); computedMaxSteps: > 42270438200 > the cost is reported without context, it is hard for operator to understand > how unbalanced the cluster is for balancer and how much progress we are > making. > For a large cluster, the calculation can take a long time, we also need to > let operator understand that it will take up to the max time to complete the > calculation. > At the end of computation: > balancer.StochasticLoadBalancer: Finished computing new load balance plan. > Computation took PT40M0.006S to try 1036409 different iterations. Found a > solution that moves 161926 regions; Going from a computed cost of > 118.75715593924485 to a new cost of 1.5509126920967042 > The time to compute the plan is also printed in a format that is not human > readable. we also need to let operator understand that balancer is just > submitting the plan and it be up to execution to complete the move. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HBASE-25973) Balancer should explain progress in a better way in log
[ https://issues.apache.org/jira/browse/HBASE-25973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhuoyue Huang reassigned HBASE-25973: - Assignee: Zhuoyue Huang > Balancer should explain progress in a better way in log > --- > > Key: HBASE-25973 > URL: https://issues.apache.org/jira/browse/HBASE-25973 > Project: HBase > Issue Type: Bug > Components: Balancer >Affects Versions: 3.0.0-alpha-1 >Reporter: Clara Xiong >Assignee: Zhuoyue Huang >Priority: Major > > In the log, balancer logs at info level at the beginning of run: > > balancer.StochasticLoadBalancer: start StochasticLoadBalancer.balancer, > initCost=277.3479243125063, functionCost=RegionCountSkewCostFunction : > (500.0, 0.3749771215224234); ServerLocalityCostFunction : (25.0, > 0.5807483226644186); RackLocalityCostFunction : (15.0, 0.0); > TableSkewCostFunction : (1000.0, 0.0019704142954972883); > StoreFileCostFunction : (200.0, 0.3668512059459341); computedMaxSteps: > 42270438200 > the cost is reported without context, it is hard for operator to understand > how unbalanced the cluster is for balancer and how much progress we are > making. > For a large cluster, the calculation can take a long time, we also need to > let operator understand that it will take up to the max time to complete the > calculation. > At the end of computation: > balancer.StochasticLoadBalancer: Finished computing new load balance plan. > Computation took PT40M0.006S to try 1036409 different iterations. Found a > solution that moves 161926 regions; Going from a computed cost of > 118.75715593924485 to a new cost of 1.5509126920967042 > The time to compute the plan is also printed in a format that is not human > readable. we also need to let operator understand that balancer is just > submitting the plan and it be up to execution to complete the move. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #3353: HBASE-25969 Purge netty-all transitive includes
Apache-HBase commented on pull request #3353: URL: https://github.com/apache/hbase/pull/3353#issuecomment-855198209 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 31s | Docker mode activated. | | -0 :warning: | yetus | 0m 5s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2.4 Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 57s | branch-2.4 passed | | +1 :green_heart: | compile | 2m 44s | branch-2.4 passed | | +1 :green_heart: | shadedjars | 6m 45s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 2m 46s | branch-2.4 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 57s | the patch passed | | +1 :green_heart: | compile | 2m 44s | the patch passed | | +1 :green_heart: | javac | 2m 44s | the patch passed | | +1 :green_heart: | shadedjars | 6m 49s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 2m 45s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 141m 54s | root in the patch failed. | | | | 177m 30s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3353/3/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3353 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux d1cb58dfe2a6 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2.4 / 7102dd0654 | | Default Java | AdoptOpenJDK-11.0.10+9 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3353/3/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-root.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3353/3/testReport/ | | Max. process+thread count | 4245 (vs. ulimit of 12500) | | modules | C: . U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3353/3/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (HBASE-25970) MOB data loss - incorrect concatenation of MOB_FILE_REFS
[
https://issues.apache.org/jira/browse/HBASE-25970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Somogyi resolved HBASE-25970.
---
Fix Version/s: 3.0.0-alpha-1
Resolution: Fixed
Merged to master. Thanks [~wchevreuil] and [~pankajkumar] for the reviews!
> MOB data loss - incorrect concatenation of MOB_FILE_REFS
> -
>
> Key: HBASE-25970
> URL: https://issues.apache.org/jira/browse/HBASE-25970
> Project: HBase
> Issue Type: Bug
> Components: mob
>Affects Versions: 3.0.0-alpha-1
>Reporter: Peter Somogyi
>Assignee: Peter Somogyi
>Priority: Critical
> Fix For: 3.0.0-alpha-1
>
>
> Active MOB files can be deleted by MobFileCleanerChore. The MOB_FILE_REFS are
> wrongly concatenated when multiple files or tables are stored. During the MOB
> cleanup HBase wants to keep the incorrectly concatenated file and considers
> the real file as deletable.
> {noformat}
> 2021-06-03 15:03:09,876 TRACE
> org.apache.hadoop.hbase.mob.MobFileCleanerChore: Specific mob references
> found for
> store=hdfs://c1377-node2.coelab.cloudera.com:8020/hbase/data/default/IntegrationTestIngestWithMOB/1df6561244ecc434f505501e6bda9af0/test_cf/2956445a41184488b914e93199df905a
> :
> {IntegrationTestIngestWithMOB=[d41d8cd98f00b204e9800998ecf8427e20210603a6c8c0e5bf8548649b0009b156452c21_1df6561244ecc434f505501e6bda9af0d41d8cd98f00b204e9800998ecf8427e20210603497714eabbe5418c9d52382a4808e986_1df6561244ecc434f505501e6bda9af0,
>
> d41d8cd98f00b204e9800998ecf8427e202106037464cbddc86943a1bad77d52a3bd412c_1df6561244ecc434f505501e6bda9af0]}
>
> ...
> 2021-06-03 15:03:10,208 TRACE
> org.apache.hadoop.hbase.mob.MobFileCleanerChore: Archiving MOB file
> hdfs://c1377-node2.coelab.cloudera.com:8020/hbase/mobdir/data/default/IntegrationTestIngestWithMOB/e9b5d936e7f55a4f1c3246a8d5ce53c2/test_cf/d41d8cd98f00b204e9800998ecf8427e20210603497714eabbe5418c9d52382a4808e986_1df6561244ecc434f505501e6bda9af0
> creation time=1622729860319
> 2021-06-03 15:03:10,467 DEBUG
> org.apache.hadoop.hbase.mob.MobFileCleanerChore: MOB Cleaner is archiving:
> hdfs://c1377-node2.coelab.cloudera.com:8020/hbase/mobdir/data/default/IntegrationTestIngestWithMOB/e9b5d936e7f55a4f1c3246a8d5ce53c2/test_cf/d41d8cd98f00b204e9800998ecf8427e20210603497714eabbe5418c9d52382a4808e986_1df6561244ecc434f505501e6bda9af0
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] petersomogyi merged pull request #3355: HBASE-25970 MOB data loss - incorrect concatenation of MOB_FILE_REFS
petersomogyi merged pull request #3355: URL: https://github.com/apache/hbase/pull/3355 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
