[jira] [Created] (HBASE-25921) Fix Wrong FileSystem when running `filesystem` on non-HDFS storage
Tak-Lon (Stephen) Wu created HBASE-25921:
Summary: Fix Wrong FileSystem when running `filesystem` on
non-HDFS storage
Key: HBASE-25921
URL: https://issues.apache.org/jira/browse/HBASE-25921
Project: HBase
Issue Type: Bug
Components: hbase-operator-tools
Affects Versions: 1.2.0, 1.1.0, 1.0.0
Reporter: Tak-Lon (Stephen) Wu
Assignee: Tak-Lon (Stephen) Wu
recently we found a bug that when running {{hbck filesystem -f}} on a non-HDFS
and caused {{Wrong FS}} error and cannot move forward.
this change fix this problem and use {{FSUtils.getCurrentFileSystem}} to
respect the root directory of what we set with {{hbase.rootdir}} in stead of
just using {{fs.defaultFS}}
e.g. this is one of the captured ERROR message
{code}
Exception in thread "main" java.lang.IllegalArgumentException: Wrong FS:
s3://xyz/data/default/abc, expected: hdfs://xyz/
at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:737)
at
org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:233)
at
org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:1045)
at
org.apache.hadoop.hdfs.DistributedFileSystem.access$1000(DistributedFileSystem.java:131)
at
org.apache.hadoop.hdfs.DistributedFileSystem$24.doCall(DistributedFileSystem.java:1112)
at
org.apache.hadoop.hdfs.DistributedFileSystem$24.doCall(DistributedFileSystem.java:1109)
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at
org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:1119)
at
org.apache.hadoop.hbase.util.FSUtils.listStatusWithStatusFilter(FSUtils.java:1530)
at
org.apache.hbase.hbck1.HFileCorruptionChecker.checkTableDir(HFileCorruptionChecker.java:320)
at
org.apache.hbase.hbck1.HFileCorruptionChecker.checkTables(HFileCorruptionChecker.java:431)
at org.apache.hbase.FileSystemFsck.fsck(FileSystemFsck.java:87)
at org.apache.hbase.HBCK2.doCommandLine(HBCK2.java:721)
at org.apache.hbase.HBCK2.run(HBCK2.java:631)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
at org.apache.hbase.HBCK2.main(HBCK2.java:865)
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[DISCUSS] Breakout discussion on storefile tracking storage solutions
Hi folks, This is a follow-on for the HBASE-24749 discussion on storefile tracking, specifically focusing on where/how do we store the list of files for each Store. I tried to capture my thoughts and the suggestions by Duo and Wellington in this google doc [1]. Please feel free to ask for edit permission (and send me a note if your email address isn't one that I would otherwise recognize :) ) to correct, improve, or expand on any other sections. FWIW, I was initially not super excited about a per-Store file, but, the more I think about it, the more I'm coming around to that idea. I think it will be more "exception-handling", but avoid the long-term operational burden of yet-another-important-system-table. - Josh [1] https://docs.google.com/document/d/1yzjvQvQfnT-M8ZgKdcQNedF8HssTnQR2loPkZtlJGVg/edit?usp=sharing
[jira] [Resolved] (HBASE-25890) [branch-1] add -U for maven build to force a check for updated releases and snapshots on remote repositories
[
https://issues.apache.org/jira/browse/HBASE-25890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Reid Chan resolved HBASE-25890.
---
Resolution: Fixed
> [branch-1] add -U for maven build to force a check for updated releases and
> snapshots on remote repositories
>
>
> Key: HBASE-25890
> URL: https://issues.apache.org/jira/browse/HBASE-25890
> Project: HBase
> Issue Type: Task
> Components: build
>Reporter: Reid Chan
>Assignee: Reid Chan
>Priority: Blocker
> Fix For: 1.7.0
>
>
> {code}
> [ERROR] Failed to execute goal on project hbase-assembly: Could not resolve
> dependencies for project org.apache.hbase:hbase-assembly:pom:1.7.0: Could not
> find artifact org.apache.hbase:hbase-thrift:jar:1.7.0 in apache release
> (https://repository.apache.org/content/repositories/releases/) -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please
> read the following articles:
> [ERROR] [Help 1]
> http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException
> [ERROR]
> [ERROR] After correcting the problems, you can resume the build with the
> command
> [ERROR] mvn -rf :hbase-assembly
> {code}
> {code}
> [ERROR] Failed to execute goal on project hbase-assembly: Could not resolve
> dependencies for project org.apache.hbase:hbase-assembly:pom:1.7.0: Failure
> to find org.apache.hbase:hbase-thrift:jar:1.7.0 in
> https://repository.apache.org/content/repositories/releases/ was cached in
> the local repository, resolution will not be reattempted until the update
> interval of apache release has elapsed or updates are forced -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please
> read the following articles:
> [ERROR] [Help 1]
> http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException
> [ERROR]
> [ERROR] After correcting the problems, you can resume the build with the
> command
> [ERROR] mvn -rf :hbase-assembly
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (HBASE-25887) Corrupt wal while region server is aborting.
[
https://issues.apache.org/jira/browse/HBASE-25887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Reid Chan resolved HBASE-25887.
---
Hadoop Flags: Reviewed
Resolution: Fixed
> Corrupt wal while region server is aborting.
>
>
> Key: HBASE-25887
> URL: https://issues.apache.org/jira/browse/HBASE-25887
> Project: HBase
> Issue Type: Improvement
> Components: regionserver, wal
>Affects Versions: 1.6.0
>Reporter: Rushabh Shah
>Assignee: Rushabh Shah
>Priority: Major
> Fix For: 1.7.0
>
>
> We have seen a case in our production cluster where we ended up in corrupt
> wal. WALSplitter logged the below error
> {noformat}
> 2021-05-12 00:42:46,786 FATAL [:60020-1] regionserver.HRegionServer -
> ABORTING region server HOST-B,60020,16207794418
> 88: Caught throwable while processing event RS_LOG_REPLAY
> java.lang.NullPointerException
> at org.apache.hadoop.hbase.CellUtil.matchingFamily(CellUtil.java:411)
> at
> org.apache.hadoop.hbase.regionserver.wal.WALEdit.isMetaEditFamily(WALEdit.java:145)
> at
> org.apache.hadoop.hbase.regionserver.wal.WALEdit.isMetaEdit(WALEdit.java:150)
> at
> org.apache.hadoop.hbase.wal.WALSplitter.splitLogFile(WALSplitter.java:408)
> at
> org.apache.hadoop.hbase.wal.WALSplitter.splitLogFile(WALSplitter.java:261)
> at
> org.apache.hadoop.hbase.regionserver.SplitLogWorker$1.exec(SplitLogWorker.java:105)
> at
> org.apache.hadoop.hbase.regionserver.handler.WALSplitterHandler.process(WALSplitterHandler.java:72)
> at
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:129)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
> Looking at the raw wal file, we could see that the last WALEdit contains the
> region id, tablename and sequence number but cells were not persisted.
> Looking at the logs of the RS that generated that corrupt wal file,
> {noformat}
> 2021-05-11 23:29:22,114 DEBUG [/HOST-A:60020] wal.FSHLog - Closing WAL writer
> in /hbase/WALs/HOST-A,60020,1620774393046
> 2021-05-11 23:29:22,196 DEBUG [/HOST-A:60020] ipc.AbstractRpcClient -
> Stopping rpc client
> 2021-05-11 23:29:22,198 INFO [/HOST-A:60020] regionserver.Leases -
> regionserver/HOST-A/:60020 closing leases
> 2021-05-11 23:29:22,198 INFO [/HOST-A:60020] regionserver.Leases -
> regionserver/HOST-A:/HOST-A:60020 closed leases
> 2021-05-11 23:29:22,198 WARN [0020.append-pool8-t1] wal.FSHLog - Append
> sequenceId=7147823, requesting roll of WAL
> java.nio.channels.ClosedChannelException
> at
> org.apache.hadoop.hdfs.DataStreamer$LastExceptionInStreamer.throwException4Close(DataStreamer.java:331)
> at
> org.apache.hadoop.hdfs.DFSOutputStream.checkClosed(DFSOutputStream.java:151)
> at org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:105)
> at
> org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:58)
> at java.io.DataOutputStream.write(DataOutputStream.java:107)
> at org.apache.hadoop.hbase.KeyValue.write(KeyValue.java:2543)
> at
> org.apache.phoenix.hbase.index.wal.KeyValueCodec.write(KeyValueCodec.java:104)
> at
> org.apache.hadoop.hbase.regionserver.wal.IndexedWALEditCodec$IndexKeyValueEncoder.write(IndexedWALEditCodec.java:218)
> at
> org.apache.hadoop.hbase.regionserver.wal.ProtobufLogWriter.append(ProtobufLogWriter.java:128)
> at
> org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.append(FSHLog.java:2083)
> at
> org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1941)
> at
> org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1857)
> at
> com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:129)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
> These 2 lines are interesting.
> {quote}2021-05-11 23:29:22,114 DEBUG [/HOST-A:60020] wal.FSHLog - Closing WAL
> writer in /hbase/WALs/HOST-A,60020,1620774393046
>
>
> 2021-05-11 23:29:22,198 WARN [0020.append-pool8-t1] wal.FSHLog - Append
> sequenceId=7147823, requesting roll of WAL
> java.nio.channels.ClosedChannelException
> {quote}
> The append thread encountered java.nio.channels.ClosedChannelException while
> writing to wal file because the wal file was already
[jira] [Resolved] (HBASE-25919) Support Hadoop 3.3.1
[ https://issues.apache.org/jira/browse/HBASE-25919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang resolved HBASE-25919. --- Resolution: Duplicate > Support Hadoop 3.3.1 > > > Key: HBASE-25919 > URL: https://issues.apache.org/jira/browse/HBASE-25919 > Project: HBase > Issue Type: Task >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Major > > The Hadoop 3.3.1 is a big release, quite different from 3.3.0. > File this jira to track the support for Hadoop 3.3.1. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-25919) Support Hadoop 3.3.1
Wei-Chiu Chuang created HBASE-25919: --- Summary: Support Hadoop 3.3.1 Key: HBASE-25919 URL: https://issues.apache.org/jira/browse/HBASE-25919 Project: HBase Issue Type: Task Reporter: Wei-Chiu Chuang Assignee: Wei-Chiu Chuang The Hadoop 3.3.1 is a big release, quite different from 3.3.0. File this jira to track the support for Hadoop 3.3.1. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-25920) Support Hadoop 3.3.1
Wei-Chiu Chuang created HBASE-25920: --- Summary: Support Hadoop 3.3.1 Key: HBASE-25920 URL: https://issues.apache.org/jira/browse/HBASE-25920 Project: HBase Issue Type: Task Reporter: Wei-Chiu Chuang Assignee: Wei-Chiu Chuang The Hadoop 3.3.1 is a big release, quite different from 3.3.0. File this jira to track the support for Hadoop 3.3.1. -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: [VOTE] Second release candidate for HBase 2.4.3 (RC1) is available
+1 * Signature: ok * Checksum : ok * Compat Report: ok * Rat check (1.8.0_282): ok - mvn clean apache-rat:check -D skipTests * Built from source (1.8.0_282): ok - mvn clean install -D skipTests -DskipTests * Rat check (11.0.10): ok - mvn clean apache-rat:check -D skipTests -D hadoop.profile=3.0 * Built from source (11.0.10): ok - mvn clean install -D skipTests -D hadoop.profile=3.0 -DskipTests Also keeping an eye on tests in Jenkins. Things have been a little rocky lately, but have been stable overall. On Thu, May 20, 2021 at 12:11 PM Andrew Purtell wrote: > Please vote on this Apache HBase release candidate, hbase-2.4.3RC1. > > The VOTE will remain open for at least 72 hours. > > [ ] +1 Release this package as Apache HBase 2.4.3 > [ ] -1 Do not release this package because ... > > The tag to be voted on is 2.4.3RC1: > > https://github.com/apache/hbase/tree/2.4.3RC1 > > The release files, including signatures, digests, as well as CHANGES.md > and RELEASENOTES.md included in this RC can be found at: > > https://dist.apache.org/repos/dist/dev/hbase/2.4.3RC1/ > > These sources correspond with the git tag "2.4.3RC1" (401b60b217). > > Temporary Maven artifacts are available in the staging repository: > > > https://repository.apache.org/content/repositories/orgapachehbase-1447/ > > Artifacts were signed with the apurt...@apache.org key which can be found > in: > > https://dist.apache.org/repos/dist/release/hbase/KEYS > > The API compatibility report for this RC can be found at: > > > > https://dist.apache.org/repos/dist/dev/hbase/2.4.3RC1/api_compare_2.4.2_to_2.4.3RC1.html > > We performed the following successful pre-flight checks before > announcing the previous RC, RC0: > > - Unit tests > > - 10 TB Common Crawl data load via IntegrationTestLoadCommonCrawl, > slowDeterministic policy > > To learn more about Apache HBase, please see > > http://hbase.apache.org/ > > Thanks, > Your HBase Release Manager >
[jira] [Created] (HBASE-25918) Upgrade hbase-thirdparty dependency to 3.5.0
Pankaj Kumar created HBASE-25918: Summary: Upgrade hbase-thirdparty dependency to 3.5.0 Key: HBASE-25918 URL: https://issues.apache.org/jira/browse/HBASE-25918 Project: HBase Issue Type: Bug Components: dependencies Reporter: Pankaj Kumar Assignee: Pankaj Kumar Recently we have fixed multiple CVEs from jetty & netty as part of HBASE-25728 & HBASE-25746. This Jira is to upgrade hbase-thirdpaty jar version. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-25875) RegionServer failed to start due to IllegalThreadStateException in AuthenticationTokenSecretManager.start
[
https://issues.apache.org/jira/browse/HBASE-25875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pankaj Kumar resolved HBASE-25875.
--
Resolution: Fixed
> RegionServer failed to start due to IllegalThreadStateException in
> AuthenticationTokenSecretManager.start
> -
>
> Key: HBASE-25875
> URL: https://issues.apache.org/jira/browse/HBASE-25875
> Project: HBase
> Issue Type: Bug
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0, 2.3.6, 2.4.4
>
>
> RegionServer failed to complete initialization and aborted during
> AuthenticationTokenSecretManager#leaderElector start.
> Observed following WARN log,
> {noformat}
> 2021-05-03 07:59:01,848 | WARN | RS-EventLoopGroup-1-6 | Thread
> leaderElector[ZKSecretWatcher-leaderElector:56] is stopped or not alive |
> org.apache.hadoop.hbase.security.token.AuthenticationTokenSecretManager.retrievePassword(AuthenticationTokenSecretManager.java:153)
> 2021-05-03 07:59:01,848 | INFO | RS-EventLoopGroup-1-6 | Thread
> leaderElector [ZKSecretWatcher-leaderElector:56] is started |
> org.apache.hadoop.hbase.security.token.AuthenticationTokenSecretManager.retrievePassword(AuthenticationTokenSecretManager.java:156)
> 2021-05-03 07:59:01,854 | INFO | ZKSecretWatcher-leaderElector | Found
> existing leader with ID: RS-IP-PORT-StartCode |
> org.apache.hadoop.hbase.zookeeper.ZKLeaderManager.waitToBecomeLeader(ZKLeaderManager.java:130)
> {noformat}
> As per the code, AuthenticationTokenSecretManager#leaderElector is started
> while retrieving password before AuthenticationTokenSecretManager#start,
>
> [https://github.com/apache/hbase/blob/8c2332d46532135723cc7a6084a2a125f3d9d8db/hbase-server/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java#L155]
> So IllegalThreadStateException occured during
> AuthenticationTokenSecretManager#start,
>
> [https://github.com/apache/hbase/blob/8c2332d46532135723cc7a6084a2a125f3d9d8db/hbase-server/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java#L107]
> {noformat}
> 2021-05-03 07:59:02,066 | ERROR | main | Failed construction RegionServer |
> org.apache.hadoop.hbase.regionserver.HRegionServer.(HRegionServer.java:775)
> java.lang.IllegalThreadStateException
> at java.lang.Thread.start(Thread.java:708)
> at
> org.apache.hadoop.hbase.security.token.AuthenticationTokenSecretManager.start(AuthenticationTokenSecretManager.java:107)
> at
> org.apache.hadoop.hbase.ipc.NettyRpcServer.start(NettyRpcServer.java:131)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.start(RSRpcServices.java:1695)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.(HRegionServer.java:756)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.constructRegionServer(HRegionServer.java:3270)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.start(HRegionServerCommandLine.java:63)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.run(HRegionServerCommandLine.java:87)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (HBASE-25841) Add basic jshell support
[
https://issues.apache.org/jira/browse/HBASE-25841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Dimiduk resolved HBASE-25841.
--
Resolution: Fixed
Pushed addendums to both branches.
> Add basic jshell support
>
>
> Key: HBASE-25841
> URL: https://issues.apache.org/jira/browse/HBASE-25841
> Project: HBase
> Issue Type: New Feature
> Components: shell, Usability
>Affects Versions: 3.0.0-alpha-1, 2.3.0, 2.4.0, 2.5.0
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
>Priority: Minor
> Fix For: 3.0.0-alpha-1, 2.5.0
>
>
> Let's make it easy to start a {{jshell}} session that includes HBase jars and
> dependencies on the class path.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (HBASE-25791) UI of master-status to show a recent history of that why balancer was rejected to run
[ https://issues.apache.org/jira/browse/HBASE-25791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Somogyi resolved HBASE-25791. --- Resolution: Fixed > UI of master-status to show a recent history of that why balancer was > rejected to run > --- > > Key: HBASE-25791 > URL: https://issues.apache.org/jira/browse/HBASE-25791 > Project: HBase > Issue Type: Sub-task > Components: Balancer, master, UI >Reporter: Zhuoyue Huang >Assignee: Zhuoyue Huang >Priority: Major > Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.4 > > Attachments: image-2021-05-18-11-31-21-001.png > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Reopened] (HBASE-25791) UI of master-status to show a recent history of that why balancer was rejected to run
[ https://issues.apache.org/jira/browse/HBASE-25791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Somogyi reopened HBASE-25791: --- > UI of master-status to show a recent history of that why balancer was > rejected to run > --- > > Key: HBASE-25791 > URL: https://issues.apache.org/jira/browse/HBASE-25791 > Project: HBase > Issue Type: Sub-task > Components: Balancer, master, UI >Reporter: Zhuoyue Huang >Assignee: Zhuoyue Huang >Priority: Major > Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.4 > > Attachments: image-2021-05-18-11-31-21-001.png > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: [DISCUSS] Implement and release HBASE-24749 (an hfile tracker that allows for avoiding renames)
Coming full circle on the "makes me worry" comment I left: I asked the question in work channels about my concern and SteveL did confirm that the "S3 strong consistency" feature does apply generally to CRUD operations. I believe this means, if we assume there is exactly one RegionServer which is hosting a Region at one time, that one RegionServer is capable of ensuring that the gaps which do exist in S3 are a non-issue (without the need for an HBOSS-like solution). Taking the suggested on a file-per-store which enumerates the committed files: the RegionServer can make sure that operates which concurrently want to update that file are exclusive, e.g. a bulk load, a memstore flush, a compaction commit. On my plate today is to incorporate this into a design doc specifically for storefile metadata (from the other message in this broader thread) On 5/24/21 1:39 PM, Josh Elser wrote: I got pulled into a call with some folks from S3 at the last minute late week. There was a comment made in passing about reading the latest, written version of a file. At the moment, I didn't want to digress into that because of immutable HFiles. However, if we're tracking files-per-store in a file, that makes me worry. To the nice digging both Duo and Andrew have shared here already and Nick's point about design, I definitely think stating what we expect and mapping that to the "platforms" which provide that "today" (as we know each will change) is the only way to insulate ourselves. The Hadoop FS contract tests are also a great thing we can adopt. On 5/21/21 9:53 PM, 张铎(Duo Zhang) wrote: So maybe we could introduce a .hfilelist directory, and put the hflielist files under this directory, so we do not need to list all the files under the region directory. And considering the possible implementation for typical object storages, listing the last directory on the whole path will be less expensive. Andrew Purtell 于2021年5月22日周六 上午9:35 写道： On May 21, 2021, at 6:07 PM, 张铎 wrote: Since we just make use of the general FileSystem API to do listing, is it possible to make use of ' bucket index listing'? Yes, those words mean the same thing. Andrew Purtell 于2021年5月22日周六 上午 6:34写道： On May 20, 2021, at 4:00 AM, Wellington Chevreuil < wellington.chevre...@gmail.com> wrote: IMO it should be a file per store. Per region is not suitable here as compaction is per store. Per file means we still need to list all the files. And usually, after compaction, we need to do an atomic operation to remove several old files and add a new file, or even several files for stripe compaction. It will be easy if we just write one file to commit these changes. Fine for me if it's simpler. Mentioned the per file approach because I thought it could be easier/faster to do that, rather than having to update the store file list on every flush. AFAIK, append is out of the table, so updating this file would mean read it, write original content plus new hfile to a temp file, delete original file, rename it). That sounds right to be. A minor potential optimization is the filename could have a timestamp component, so a bucket index listing at that path would pick up a list including the latest, and the latest would be used as the manifest of valid store files. The cloud object store is expected to provide an atomic listing semantic where the file is written and closed and only then is it visible, and it is visible at once to everyone. (I think this is available on most.) Old manifest file versions could be lazily deleted. Em qui., 20 de mai. de 2021 às 02:57, 张铎(Duo Zhang) < palomino...@gmail.com> escreveu: IIRC S3 is the only object storage which does not guarantee read-after-write consistency in the past... This is the quick result after googling AWS [1] Amazon S3 delivers strong read-after-write consistency automatically for all applications Azure[2] Azure Storage was designed to embrace a strong consistency model that guarantees that after the service performs an insert or update operation, subsequent read operations return the latest update. Aliyun[3] A feature requires that object operations in OSS be atomic, which indicates that operations can only either succeed or fail without intermediate states. To ensure that users can access only complete data, OSS does not return corrupted or partial data. Object operations in OSS are highly consistent. For example, when a user receives an upload (PUT) success response, the uploaded object can be read immediately, and copies of the object are written to multiple devices for redundancy. Therefore, the situations where data is not obtained when you perform the read-after-write operation do not exist. The same is true for delete operations. After you delete an object, the object and its copies no longer exist. GCP[4] Cloud Storage provides strong global consistency for the following operations,
[jira] [Reopened] (HBASE-25841) Add basic jshell support
[
https://issues.apache.org/jira/browse/HBASE-25841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Dimiduk reopened HBASE-25841:
--
This breaks master process startup.
> Add basic jshell support
>
>
> Key: HBASE-25841
> URL: https://issues.apache.org/jira/browse/HBASE-25841
> Project: HBase
> Issue Type: New Feature
> Components: shell, Usability
>Affects Versions: 3.0.0-alpha-1, 2.3.0, 2.4.0, 2.5.0
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
>Priority: Minor
> Fix For: 3.0.0-alpha-1, 2.5.0
>
>
> Let's make it easy to start a {{jshell}} session that includes HBase jars and
> dependencies on the class path.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HBASE-25916) Move FavoredNodeLoadBalancer to hbase-balancer module
Duo Zhang created HBASE-25916: - Summary: Move FavoredNodeLoadBalancer to hbase-balancer module Key: HBASE-25916 URL: https://issues.apache.org/jira/browse/HBASE-25916 Project: HBase Issue Type: Sub-task Components: Balancer, FavoredNodes Reporter: Duo Zhang -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-25917) Move RSGroupBasedLoadBalancer to hbase-balancer
Duo Zhang created HBASE-25917: - Summary: Move RSGroupBasedLoadBalancer to hbase-balancer Key: HBASE-25917 URL: https://issues.apache.org/jira/browse/HBASE-25917 Project: HBase Issue Type: Sub-task Reporter: Duo Zhang -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-25818) Move StochasticLoadBalancer to hbase-balancer module
[ https://issues.apache.org/jira/browse/HBASE-25818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang resolved HBASE-25818. --- Fix Version/s: 3.0.0-alpha-1 Hadoop Flags: Reviewed Resolution: Fixed Merged to master. Thanks [~meiyi] for reviewing. > Move StochasticLoadBalancer to hbase-balancer module > > > Key: HBASE-25818 > URL: https://issues.apache.org/jira/browse/HBASE-25818 > Project: HBase > Issue Type: Sub-task > Components: Balancer >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 3.0.0-alpha-1 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-25534) Honor TableDescriptor settings earlier in normalization
[
https://issues.apache.org/jira/browse/HBASE-25534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Dimiduk resolved HBASE-25534.
--
Resolution: Fixed
Thank you [~DeanZ]!
> Honor TableDescriptor settings earlier in normalization
> ---
>
> Key: HBASE-25534
> URL: https://issues.apache.org/jira/browse/HBASE-25534
> Project: HBase
> Issue Type: Improvement
> Components: Normalizer
>Affects Versions: 3.0.0-alpha-1, 2.4.2
>Reporter: Baiqiang Zhao
>Assignee: Baiqiang Zhao
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.2
>
>
> -Table can be enabled and disabled merge/split in TableDescriptor, we should
> judge before calculating the plan.-
> The normalizer configuration can be set by table level. For example,
> hbase.normalizer.min.region.count can be set by "alter ‘table’,
> CONFIGURATION=>\{'hbase.normalizer.min.region.count' => '5'}". If the table
> is not set, then use the global configuration which is set in hbase-site.xml.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (HBASE-25894) Improve the performance for region load and region count related cost functions
[ https://issues.apache.org/jira/browse/HBASE-25894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang resolved HBASE-25894. --- Hadoop Flags: Reviewed Resolution: Fixed Pushed to master and branch-2. Thanks [~meiyi] for reviewing. > Improve the performance for region load and region count related cost > functions > --- > > Key: HBASE-25894 > URL: https://issues.apache.org/jira/browse/HBASE-25894 > Project: HBase > Issue Type: Sub-task > Components: Balancer, Performance >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 3.0.0-alpha-1, 2.5.0 > > > For a large cluster, we have a lot of regions, so computing the whole cost > will be expensive. We should try to remove the unnecessary calculation as > much as possible. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-25915) ‘mobdir’ overlaps in HFileLink.mobPath
wenfeiyi666 created HBASE-25915: --- Summary: ‘mobdir’ overlaps in HFileLink.mobPath Key: HBASE-25915 URL: https://issues.apache.org/jira/browse/HBASE-25915 Project: HBase Issue Type: Bug Components: mob Affects Versions: 2.4.3, 1.4.13, 3.0.0-alpha-1 Reporter: wenfeiyi666 Assignee: wenfeiyi666 Fix For: 3.0.0-alpha-1, 1.4.14, 2.4.4 ‘mobdir’ overlaps in HFileLink.mobPath, like '/root/mobdir/mobdir/data/table/...' -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-25906) UI of master-status to show recent history of balancer desicion
[ https://issues.apache.org/jira/browse/HBASE-25906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Viraj Jasani resolved HBASE-25906. -- Hadoop Flags: Reviewed Resolution: Fixed Thanks for the contribution [~GeorryHuang]. > UI of master-status to show recent history of balancer desicion > --- > > Key: HBASE-25906 > URL: https://issues.apache.org/jira/browse/HBASE-25906 > Project: HBase > Issue Type: Improvement > Components: Balancer, master, UI >Affects Versions: 3.0.0-alpha-1, 2.5.0 >Reporter: Zhuoyue Huang >Assignee: Zhuoyue Huang >Priority: Major > Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.4 > > Attachments: screenshot-1.png > > > HBASE-24528 provide ‘Balancer Decision’ to display the history that includes > decision factor details and weights and costs while running balancer. > This issue implement 'Balancer Decision' UI web page -- This message was sent by Atlassian Jira (v8.3.4#803005)
