[jira] [Resolved] (HBASE-28313) StorefileRefresherChore should not refresh readonly table
[ https://issues.apache.org/jira/browse/HBASE-28313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang resolved HBASE-28313. --- Fix Version/s: 2.6.0 2.4.18 2.5.8 3.0.0-beta-2 Hadoop Flags: Reviewed Resolution: Fixed Pushed to all active branches. Thanks [~haosun] for contributing! > StorefileRefresherChore should not refresh readonly table > - > > Key: HBASE-28313 > URL: https://issues.apache.org/jira/browse/HBASE-28313 > Project: HBase > Issue Type: Improvement > Components: regionserver >Affects Versions: 2.4.17 >Reporter: Hao Sun >Assignee: Hao Sun >Priority: Major > Labels: pull-request-available > Fix For: 2.6.0, 2.4.18, 2.5.8, 3.0.0-beta-2 > > > According to semantics, when StoreFile Refresher is turned on, only the area > with replicas should be refreshed. *Tables with read-only attributes will not > be refreshed.* > There are the following scenarios: Assume that some tables in the database > have replica enabled, while others do not. If a table that does not have > replica enabled needs to perform migration tasks - first enable readonly and > then export the snapshot, StorefileRefresherChore will always refresh the > table that does not have replica enabled but performs migration tasks. This > is obviously illogical. > Therefore, StorefileRefresherChore needs to determine whether the region has > a read-only copy, and refresh it if it has a read-only copy. And tables with > read-only properties are not refreshed. > patch:[HBASE-28313|https://github.com/apache/hbase/pull/5641] -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [VOTE] hbase-thirdparty 4.1.6 (RC0)
+1 - Signature: OK - Checksum : OK - Rat check: OK - Build from source: OK - Staging repositories : Ok - Change logs, release notes: Ok - Build master code with 4.1.6 thirdparty : Ok. Thanks, Rajeshbabu. On Wed, Feb 28, 2024 at 8:50 PM Nihal Jain wrote: > Ah thanks for clarifying. Updating my vote to +1. > > Regards > Nihal > > On Wed, 28 Feb 2024, 20:25 Nick Dimiduk, wrote: > > > On Wed, Feb 28, 2024 at 3:37 PM Nihal Jain > wrote: > > > > > But I see that > > > > > > > > > https://dist.apache.org/repos/dist/dev/hbase/hbase-thirdparty-4.1.6RC0/RELEASENOTES.md > > > is empty for the current release. > > > > > > Is this enough to sink an RC? Or can this be replaced in place? > Consider > > my > > > vote as +1, if we are good. > > > > > > > I noticed that. None of the issues in Jira have release notes, so there > was > > nothing generated for this release. I didn't look back through the > history > > to see if we omitted mention of a release version when the release notes > > section was empty. I think it's fine to point out the absence of release > > notes for a given version, but I don't feel strongly about it. Anyway, I > > recall we discussed removing release notes and changes from the product > > artifacts but I don't have that thread handy. > > > > Thanks, > > Nick > > > > On Tue, 27 Feb 2024, 21:37 Nick Dimiduk, wrote: > > > > > > > Please vote on this Apache hbase thirdparty release candidate, > > > > hbase-thirdparty-4.1.6RC0 > > > > > > > > The VOTE will remain open for at least 72 hours. > > > > > > > > [ ] +1 Release this package as Apache hbase thirdparty 4.1.6 > > > > [ ] -1 Do not release this package because ... > > > > > > > > The tag to be voted on is 4.1.6RC0: > > > > > > > > https://github.com/apache/hbase-thirdparty/tree/4.1.6RC0 > > > > > > > > This tag currently points to git reference > > > > > > > > b1a8d26f5eb9f45d4069d1fdd909d0d22123950c > > > > > > > > The release files, including signatures, digests, as well as > CHANGES.md > > > > and RELEASENOTES.md included in this RC can be found at: > > > > > > > > > > > > https://dist.apache.org/repos/dist/dev/hbase/hbase-thirdparty-4.1.6RC0/ > > > > > > > > Maven artifacts are available in a staging repository at: > > > > > > > > > > > > https://repository.apache.org/content/repositories/orgapachehbase-1533/ > > > > > > > > Artifacts were signed with the 0xEF4EBF27 key which can be found in: > > > > > > > > https://downloads.apache.org/hbase/KEYS > > > > > > > > To learn more about Apache hbase thirdparty, please see > > > > > > > > http://hbase.apache.org/ > > > > > > > > Thanks, > > > > Your HBase Release Manager > > > > > > > > > >
[jira] [Created] (HBASE-28408) Confusing logging during backup restore
Dieter De Paepe created HBASE-28408:
---
Summary: Confusing logging during backup restore
Key: HBASE-28408
URL: https://issues.apache.org/jira/browse/HBASE-28408
Project: HBase
Issue Type: Bug
Components: backuprestore
Affects Versions: 2.6.0
Reporter: Dieter De Paepe
Encountered this while experimenting with the backup/restore functionality.
My setup was as follows:
* Took several backups (Full1, inc2, inc3)
* Changed an entry in the "lily_tenant_acme:LILY_SETTINGS" table
* Attempt a restore (to test if my changed entry is reverted):
{code:java}
$ hbase restore -conf backup-conf.xml s3a://backuprestore-experiments/hbase
backup_1709123740345 -t "lily_tenant_acme:LILY_SETTINGS" -m
"lily_tenant_acme:LILY_SETTINGS-restored1" -o
24/02/28 16:15:41 WARN org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil:
The addDependencyJars(Configuration, Class...) method has been deprecated
since it is easy to use incorrectly. Most users should rely on
addDependencyJars(Job) instead. See HBASE-8386 for more details.
24/02/28 16:15:58 WARN org.apache.hadoop.hbase.tool.LoadIncrementalHFiles:
Skipping non-directory
hdfs://hdfsns/user/lily/hbase-staging/bulk_output-lily_tenant_acme-LILY_SETTINGS-restored1-1709136941410/_SUCCESS
24/02/28 16:15:59 WARN org.apache.hadoop.hbase.backup.impl.RestoreTablesClient:
Nothing has changed, so there is no need to restore
'lily_tenant_acme:LILY_SETTINGS'
{code}
Based on the final logging line, I presumed my restore operation had failed.
After some investigation however, I found that this was not the case: my change
was reverted as expected.
Some code investigation learned me this log message is shown because I was
restoring backup `inc3`, and there were no changes between `full1` and `inc3`.
I suggest rephrasing this log message, and changing it to a INFO level.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Reopened] (HBASE-28342) Decommissioned hosts should be rejected by the HMaster
[ https://issues.apache.org/jira/browse/HBASE-28342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk reopened HBASE-28342: -- Sorry, I missed something in review. Looking more closely at the other exceptions thrown, I think we should make two small changes. First, the DecommissionedHostRejectedException should be in the `org.apache.hadoop.hbase.ipc` package (still in the hbase-server module). Second, It should be annotated as `@InterfaceAudience.Public` because it's part of our RPC protocol. > Decommissioned hosts should be rejected by the HMaster > -- > > Key: HBASE-28342 > URL: https://issues.apache.org/jira/browse/HBASE-28342 > Project: HBase > Issue Type: Improvement > Components: master >Reporter: Ahmad Alhour >Assignee: Ahmad Alhour >Priority: Major > Labels: pull-request-available > Fix For: 2.6.0, 4.0.0-alpha-1, 3.0.0-beta-2 > > > We had an issue with a cluster, internally at HubSpot, where a decommissioned > RegionServer was still being picked up by the HMaster. The host the > RegionServer was living on was impaired, and we couldn't correctly kill the > RegionServer, so the HMaster would periodically hear back from the host and > remove it from its dead host's list. > We would like to implement a fix so that this doesn't happen. We're thinking > of adding a boolean flag to the Decommission RegionServer Admin API that > signifies ignoring the startcode of the servername, when the boolean is True > the host will be rejected every time it comes back even if it had a different > startcode. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HBASE-28321) RpcConnectionRegistry is broken when security is enabled and we use different principal for master and region server
[ https://issues.apache.org/jira/browse/HBASE-28321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang resolved HBASE-28321. --- Fix Version/s: 2.6.0 3.0.0-beta-2 Hadoop Flags: Reviewed Release Note: Introduced a preamble security call to let server respond its server principal so client knows which one to use if there are multiple server principal candidates. This requires changing the SecurityInfo to accept a list of server principal patterns for a given rpc service. Altough SecurityInfo is marked as IA.Private, since it is leaked in SaslClientAuthenticationProvider, we still keep the getServerPrincipal method and only mark it as deprecated since 2.6.0, and plan to remove it in 4.0.0. In SaslClientAuthenticationProvider, we marked the old createClient method as deprecated since 2.6.0 and plan to remove in 4.0.0. Now you should prefer the method which passes a String server principal, instead of passing a SecurityInfo. And notice that, if you do use different server principals for master and region server, then if you use 2.6.0+ hbase client, you can not connect a 2.6.0- cluster now, as we do not implement the fallback logic since it is not easy as the server will close the connection directly with unexpected header error. But anyway, you can not connect to the cluster with 2.6.0- client either, because there is no to provide two candidates in the old code base, and this is just what this issue fixes. Resolution: Fixed Pushed to branch-2.6+. Thanks [~bbeaudreault] for reviewing! > RpcConnectionRegistry is broken when security is enabled and we use different > principal for master and region server > > > Key: HBASE-28321 > URL: https://issues.apache.org/jira/browse/HBASE-28321 > Project: HBase > Issue Type: Sub-task > Components: Client, IPC/RPC, security >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Critical > Labels: pull-request-available > Fix For: 2.6.0, 3.0.0-beta-2 > > > After introducing RpcConnectionRegistry, we let master and region server both > implement ClientMetaService. > In our current client architecture, when security is enabled, we rely on the > record in SecurityInfo to determine the server principal to use, > unfortunately there is only one principal can be specified, so if we use > different principal for master and region server, either we can not connect > to master, or we can not connect to region server. > And just changing the server principal field in SecurityInfo to an array can > not solve the problem, as when connecting, we do not know whether the remote > server is a master or region server, so we still can not determine which > principal to use... > Anyway, since this has been in our code base since 2.5.0, it is not a new > problem, so just set it as critical, not a blocker. But we should find out > the solution ASAP. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [VOTE] hbase-thirdparty 4.1.6 (RC0)
Ah thanks for clarifying. Updating my vote to +1. Regards Nihal On Wed, 28 Feb 2024, 20:25 Nick Dimiduk, wrote: > On Wed, Feb 28, 2024 at 3:37 PM Nihal Jain wrote: > > > But I see that > > > > > https://dist.apache.org/repos/dist/dev/hbase/hbase-thirdparty-4.1.6RC0/RELEASENOTES.md > > is empty for the current release. > > > > Is this enough to sink an RC? Or can this be replaced in place? Consider > my > > vote as +1, if we are good. > > > > I noticed that. None of the issues in Jira have release notes, so there was > nothing generated for this release. I didn't look back through the history > to see if we omitted mention of a release version when the release notes > section was empty. I think it's fine to point out the absence of release > notes for a given version, but I don't feel strongly about it. Anyway, I > recall we discussed removing release notes and changes from the product > artifacts but I don't have that thread handy. > > Thanks, > Nick > > On Tue, 27 Feb 2024, 21:37 Nick Dimiduk, wrote: > > > > > Please vote on this Apache hbase thirdparty release candidate, > > > hbase-thirdparty-4.1.6RC0 > > > > > > The VOTE will remain open for at least 72 hours. > > > > > > [ ] +1 Release this package as Apache hbase thirdparty 4.1.6 > > > [ ] -1 Do not release this package because ... > > > > > > The tag to be voted on is 4.1.6RC0: > > > > > > https://github.com/apache/hbase-thirdparty/tree/4.1.6RC0 > > > > > > This tag currently points to git reference > > > > > > b1a8d26f5eb9f45d4069d1fdd909d0d22123950c > > > > > > The release files, including signatures, digests, as well as CHANGES.md > > > and RELEASENOTES.md included in this RC can be found at: > > > > > > > > https://dist.apache.org/repos/dist/dev/hbase/hbase-thirdparty-4.1.6RC0/ > > > > > > Maven artifacts are available in a staging repository at: > > > > > > > > https://repository.apache.org/content/repositories/orgapachehbase-1533/ > > > > > > Artifacts were signed with the 0xEF4EBF27 key which can be found in: > > > > > > https://downloads.apache.org/hbase/KEYS > > > > > > To learn more about Apache hbase thirdparty, please see > > > > > > http://hbase.apache.org/ > > > > > > Thanks, > > > Your HBase Release Manager > > > > > >
[jira] [Resolved] (HBASE-28384) Client ingegration tests fails for branch-2/branch-2.6
[ https://issues.apache.org/jira/browse/HBASE-28384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang resolved HBASE-28384. --- Fix Version/s: 2.6.0 3.0.0-beta-2 Hadoop Flags: Reviewed Resolution: Fixed Pushed to branch-2.6+. Thanks [~ndimiduk] and [~meiyi] for reviewing! > Client ingegration tests fails for branch-2/branch-2.6 > -- > > Key: HBASE-28384 > URL: https://issues.apache.org/jira/browse/HBASE-28384 > Project: HBase > Issue Type: Bug > Components: hadoop3, jenkins >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Labels: pull-request-available > Fix For: 2.6.0, 3.0.0-beta-2 > > > In HBASE-28331, we fixed the problem that we should use the latest hadoop3 > tarballs, for generating hdfs-site.xml. > But for branch-2.x, we still use hadoop2 when compiling hbase by default, so > there is no org.apache.hadoop.ipc.ProtobufRpcEngine2 class. But the hadoop > 3.3.x will generate hdfs-site.xml with this config, then leads to the failure > of hbase start up. > Should we change to build with hadoop3 while running on top of hadoop3? Since > for branch-2.x, we will publish artifacts for hadoop3. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [VOTE] hbase-thirdparty 4.1.6 (RC0)
On Wed, Feb 28, 2024 at 3:37 PM Nihal Jain wrote: > But I see that > > https://dist.apache.org/repos/dist/dev/hbase/hbase-thirdparty-4.1.6RC0/RELEASENOTES.md > is empty for the current release. > > Is this enough to sink an RC? Or can this be replaced in place? Consider my > vote as +1, if we are good. > I noticed that. None of the issues in Jira have release notes, so there was nothing generated for this release. I didn't look back through the history to see if we omitted mention of a release version when the release notes section was empty. I think it's fine to point out the absence of release notes for a given version, but I don't feel strongly about it. Anyway, I recall we discussed removing release notes and changes from the product artifacts but I don't have that thread handy. Thanks, Nick On Tue, 27 Feb 2024, 21:37 Nick Dimiduk, wrote: > > > Please vote on this Apache hbase thirdparty release candidate, > > hbase-thirdparty-4.1.6RC0 > > > > The VOTE will remain open for at least 72 hours. > > > > [ ] +1 Release this package as Apache hbase thirdparty 4.1.6 > > [ ] -1 Do not release this package because ... > > > > The tag to be voted on is 4.1.6RC0: > > > > https://github.com/apache/hbase-thirdparty/tree/4.1.6RC0 > > > > This tag currently points to git reference > > > > b1a8d26f5eb9f45d4069d1fdd909d0d22123950c > > > > The release files, including signatures, digests, as well as CHANGES.md > > and RELEASENOTES.md included in this RC can be found at: > > > > > https://dist.apache.org/repos/dist/dev/hbase/hbase-thirdparty-4.1.6RC0/ > > > > Maven artifacts are available in a staging repository at: > > > > > https://repository.apache.org/content/repositories/orgapachehbase-1533/ > > > > Artifacts were signed with the 0xEF4EBF27 key which can be found in: > > > > https://downloads.apache.org/hbase/KEYS > > > > To learn more about Apache hbase thirdparty, please see > > > > http://hbase.apache.org/ > > > > Thanks, > > Your HBase Release Manager > > >
Re: [VOTE] hbase-thirdparty 4.1.6 (RC0)
+0 * Signature: OK * Checksum : OK * Rat check (1.8.0_381): OK - mvn clean apache-rat:check * Build from source (1.8.0_381): OK - mvn clean install -DskipTests * Unit Tests (1.8.0_381): OK - mvn clean package -P release -Dsurefire.rerunFailingTestsCount=3 But I see that https://dist.apache.org/repos/dist/dev/hbase/hbase-thirdparty-4.1.6RC0/RELEASENOTES.md is empty for the current release. Is this enough to sink an RC? Or can this be replaced in place? Consider my vote as +1, if we are good. Regards Nihal On Tue, 27 Feb 2024, 21:37 Nick Dimiduk, wrote: > Please vote on this Apache hbase thirdparty release candidate, > hbase-thirdparty-4.1.6RC0 > > The VOTE will remain open for at least 72 hours. > > [ ] +1 Release this package as Apache hbase thirdparty 4.1.6 > [ ] -1 Do not release this package because ... > > The tag to be voted on is 4.1.6RC0: > > https://github.com/apache/hbase-thirdparty/tree/4.1.6RC0 > > This tag currently points to git reference > > b1a8d26f5eb9f45d4069d1fdd909d0d22123950c > > The release files, including signatures, digests, as well as CHANGES.md > and RELEASENOTES.md included in this RC can be found at: > > https://dist.apache.org/repos/dist/dev/hbase/hbase-thirdparty-4.1.6RC0/ > > Maven artifacts are available in a staging repository at: > > https://repository.apache.org/content/repositories/orgapachehbase-1533/ > > Artifacts were signed with the 0xEF4EBF27 key which can be found in: > > https://downloads.apache.org/hbase/KEYS > > To learn more about Apache hbase thirdparty, please see > > http://hbase.apache.org/ > > Thanks, > Your HBase Release Manager >
