[jira] [Updated] (HBASE-26974) Introduce a LogRollProcedure
[ https://issues.apache.org/jira/browse/HBASE-26974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ruanhui updated HBASE-26974: Description: The current log-rolling for all regionservers is based in ZK. Here is an attempt to reimplement it with procedure v2. Here are some requirements about the implementation. The procedure can be introduced as a new feature. It should remain fully compatible with previous implementations. Also, this feature can be disabled by the configuration. Currently we only use the logroll procedure when taking a backup job, so I think all code logic should be implemented in the hbase-backup module as much as possible(I'm not sure if this is the right way to do it. If you have any suggestions, please let me know). Here are some details about the implementation. *LogRollProcedure* The LogRollProcedure is used to roll WAL for all the regionservers in the cluster. It acquires the shared lock of the backup system table. *RSLogRollProcedure* The RSLogRollProcedure is used to schedule a RSLogRollRemoteProcedure for each regionserver. When the subprocedure returns, the RSLogRollProcedure will check the logrolling result in the backup system table. If failed, The RSLogRollProcedure will schedule a new RSLogRollRemoteProcedure to retry. *RSLogRollRemoteProcedure* The RSLogRollRemoteProcedure is used to send the log roll request to the remote server. This is only the first version implementation, any suggestions and feedbacks are appreciated. was: The current log-rolling for all regionservers is based in ZK. Here is an attempt to reimplement it with procedure v2. Here are some requirements about the implementation. The procedure can be introduced as a new feature. It should remain fully compatible with previous implementations. Also, this feature can be disabled by the configuration. Currently we only use the logroll procedure when taking a backup job, so I think all code logic should be implemented in the hbase-backup module as much as possible(I'm not sure if this is the right way to do it. If you have any suggestions, please let me know). Here are some details about the implementation. LogRollProcedure The LogRollProcedure is used to roll WAL for all the regionservers in the cluster. It acquires the shared lock of the backup system table. RSLogRollProcedure The RSLogRollProcedure is used to schedule a RSLogRollRemoteProcedure for each regionserver. When the subprocedure returns, the RSLogRollProcedure will check the logrolling result in the backup system table. If failed, The RSLogRollProcedure will schedule a new RSLogRollRemoteProcedure to retry. RSLogRollRemoteProcedure The RSLogRollRemoteProcedure is used to send the log roll request to the remote server. This is only the first version implementation, any suggestions and feedbacks are appreciated. > Introduce a LogRollProcedure > > > Key: HBASE-26974 > URL: https://issues.apache.org/jira/browse/HBASE-26974 > Project: HBase > Issue Type: Improvement > Components: backuprestore, proc-v2 >Reporter: ruanhui >Assignee: ruanhui >Priority: Minor > Fix For: 3.0.0-alpha-3 > > > The current log-rolling for all regionservers is based in ZK. Here is an > attempt to reimplement it with procedure v2. > Here are some requirements about the implementation. > The procedure can be introduced as a new feature. It should remain fully > compatible with previous implementations. Also, this feature can be disabled > by the configuration. Currently we only use the logroll procedure when taking > a backup job, so I think all code logic should be implemented in the > hbase-backup module as much as possible(I'm not sure if this is the right way > to do it. If you have any suggestions, please let me know). > Here are some details about the implementation. > *LogRollProcedure* > The LogRollProcedure is used to roll WAL for all the regionservers in the > cluster. It acquires the shared lock of the backup system table. > *RSLogRollProcedure* > The RSLogRollProcedure is used to schedule a RSLogRollRemoteProcedure for > each regionserver. When the subprocedure returns, the RSLogRollProcedure will > check the logrolling result in the backup system table. If failed, The > RSLogRollProcedure will schedule a new RSLogRollRemoteProcedure to retry. > *RSLogRollRemoteProcedure* > The RSLogRollRemoteProcedure is used to send the log roll request to the > remote server. > This is only the first version implementation, any suggestions and feedbacks > are appreciated. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Created] (HBASE-26974) Introduce a LogRollProcedure
ruanhui created HBASE-26974: --- Summary: Introduce a LogRollProcedure Key: HBASE-26974 URL: https://issues.apache.org/jira/browse/HBASE-26974 Project: HBase Issue Type: Improvement Components: backuprestore, proc-v2 Reporter: ruanhui Assignee: ruanhui Fix For: 3.0.0-alpha-3 The current log-rolling for all regionservers is based in ZK. Here is an attempt to reimplement it with procedure v2. Here are some requirements about the implementation. The procedure can be introduced as a new feature. It should remain fully compatible with previous implementations. Also, this feature can be disabled by the configuration. Currently we only use the logroll procedure when taking a backup job, so I think all code logic should be implemented in the hbase-backup module as much as possible(I'm not sure if this is the right way to do it. If you have any suggestions, please let me know). Here are some details about the implementation. LogRollProcedure The LogRollProcedure is used to roll WAL for all the regionservers in the cluster. It acquires the shared lock of the backup system table. RSLogRollProcedure The RSLogRollProcedure is used to schedule a RSLogRollRemoteProcedure for each regionserver. When the subprocedure returns, the RSLogRollProcedure will check the logrolling result in the backup system table. If failed, The RSLogRollProcedure will schedule a new RSLogRollRemoteProcedure to retry. RSLogRollRemoteProcedure The RSLogRollRemoteProcedure is used to send the log roll request to the remote server. This is only the first version implementation, any suggestions and feedbacks are appreciated. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[GitHub] [hbase] Apache-HBase commented on pull request #4358: HBASE-26951 HMaster should exit gracefully, when stopped via hbase-daemon.sh
Apache-HBase commented on PR #4358: URL: https://github.com/apache/hbase/pull/4358#issuecomment-1107792207 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 3s | Docker mode activated. | | -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 31s | master passed | | +1 :green_heart: | compile | 0m 46s | master passed | | +1 :green_heart: | shadedjars | 3m 39s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 25s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 35s | the patch passed | | +1 :green_heart: | compile | 0m 47s | the patch passed | | +1 :green_heart: | javac | 0m 47s | the patch passed | | +1 :green_heart: | shadedjars | 3m 38s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 25s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 184m 59s | hbase-server in the patch passed. | | | | 202m 10s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4358/4/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4358 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 189da899561f 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / f5b10e0115 | | Default Java | AdoptOpenJDK-11.0.10+9 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4358/4/testReport/ | | Max. process+thread count | 2914 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4358/4/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4358: HBASE-26951 HMaster should exit gracefully, when stopped via hbase-daemon.sh
Apache-HBase commented on PR #4358: URL: https://github.com/apache/hbase/pull/4358#issuecomment-1107792185 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 2m 57s | Docker mode activated. | | -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 21s | master passed | | +1 :green_heart: | compile | 0m 39s | master passed | | +1 :green_heart: | shadedjars | 3m 41s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 25s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 17s | the patch passed | | +1 :green_heart: | compile | 0m 39s | the patch passed | | +1 :green_heart: | javac | 0m 39s | the patch passed | | +1 :green_heart: | shadedjars | 3m 39s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 24s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 183m 35s | hbase-server in the patch passed. | | | | 202m 4s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4358/4/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4358 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux b14ae93c6711 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / f5b10e0115 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4358/4/testReport/ | | Max. process+thread count | 3051 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4358/4/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-26959) Brotli compression support
[
https://issues.apache.org/jira/browse/HBASE-26959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17526991#comment-17526991
]
Hudson commented on HBASE-26959:
Results for branch branch-2
[build #523 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/523/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/523/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/523/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/523/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/523/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(x) {color:red}-1 client integration test{color}
--Failed when running client tests on top of Hadoop 2. [see log for
details|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/523//artifact/output-integration/hadoop-2.log].
(note that this means we didn't run on Hadoop 3)
> Brotli compression support
> --
>
> Key: HBASE-26959
> URL: https://issues.apache.org/jira/browse/HBASE-26959
> Project: HBase
> Issue Type: New Feature
>Reporter: Andrew Kyle Purtell
>Assignee: Andrew Kyle Purtell
>Priority: Minor
> Fix For: 2.5.0, 3.0.0-alpha-3
>
>
> I was about to document the compression improvements upcoming in 2.5.0 and
> realized we lacked support for one of the major compression formats in use
> today. (Arguably brotli is not often used to compress data at rest, but is
> widely supported among web browsers and HTTP servers and clients.)
> Brotli support for Java is provided by Apache 2 licensed
> [Brotli4j|https://github.com/hyperxpro/Brotli4j] available in Maven central.
> It is not difficult to add it, so let's add it. There might be some advantage
> to opting for brotli compression when storing web data given the precomputed
> static dictionary of such content included in its specification.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Commented] (HBASE-25768) Support an overall coarse and fast balance strategy for StochasticLoadBalancer
[ https://issues.apache.org/jira/browse/HBASE-25768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17526990#comment-17526990 ] kangTwang commented on HBASE-25768: --- Hi [~Xiaolin Ha] : Has this PR not been completed yet? > Support an overall coarse and fast balance strategy for StochasticLoadBalancer > -- > > Key: HBASE-25768 > URL: https://issues.apache.org/jira/browse/HBASE-25768 > Project: HBase > Issue Type: Improvement > Components: Balancer >Affects Versions: 3.0.0-alpha-1, 2.0.0, 1.4.13 >Reporter: Xiaolin Ha >Assignee: Xiaolin Ha >Priority: Major > > When we use StochasticLoadBalancer + balanceByTable, we could face two > difficulties. > # For each table, their regions are distributed uniformly, but for the > overall cluster, still exiting imbalance between RSes; > # When there are large-scaled restart of RSes, or expansion for groups or > cluster, we hope the balancer can execute as soon as possible, but the > StochasticLoadBalancer may need a lot of time to compute costs. > We can detect these circumstances in StochasticLoadBalancer(such as using the > percentage of skew tables), and before the normal balance steps trying, we > can add a strategy to let it just balance like the SimpleLoadBalancer or use > few light cost functions here. > > -- This message was sent by Atlassian Jira (v8.20.7#820007)
