[jira] [Resolved] (HBASE-26480) Close NamedQueueRecorder to allow HMaster/RS to shutdown gracefully
[
https://issues.apache.org/jira/browse/HBASE-26480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaolin Ha resolved HBASE-26480.
Fix Version/s: 1.8.0
Resolution: Fixed
Merged to branch-1, thanks [~shahrs87] for contributing, and thanks
[~zhangduo] for reviewing.
> Close NamedQueueRecorder to allow HMaster/RS to shutdown gracefully
> ---
>
> Key: HBASE-26480
> URL: https://issues.apache.org/jira/browse/HBASE-26480
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.7.0
>Reporter: Rushabh Shah
>Assignee: Rushabh Shah
>Priority: Major
> Fix For: 1.8.0
>
>
> Saw one case in our production cluster where RS was not exiting. Saw this
> non-daemon thread in hung RS stack trace:
> {noformat}
> "main.slowlog.append-pool-pool1-t1" #26 prio=5 os_prio=31
> tid=0x7faf23bf7800 nid=0x6d07 waiting on condition [0x73f4d000]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x0004039e3840> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> com.lmax.disruptor.BlockingWaitStrategy.waitFor(BlockingWaitStrategy.java:47)
> at
> com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
> at
> com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
> at
> com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
>
> This is coming from
> [NamedQueueRecorder|https://github.com/apache/hbase/blob/branch-1/hbase-server/src/main/java/org/apache/hadoop/hbase/namequeues/NamedQueueRecorder.java#L65]
> implementation.
> This bug doesn't exists in branch-2 and master since the Disruptor
> initialization has changed and we set daemon=true also. See [this
> code|https://github.com/apache/hbase/blob/branch-2/hbase-server/src/main/java/org/apache/hadoop/hbase/namequeues/NamedQueueRecorder.java#L68]
>
> FYI [~vjasani] [~zhangduo]
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [hbase] sunhelly merged pull request #3871: HBASE-26480 Close NamedQueueRecorder to allow HMaster/RS to shutdown gracefully
sunhelly merged pull request #3871: URL: https://github.com/apache/hbase/pull/3871 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] zhengzhuobinzzb commented on a change in pull request #3876: HBASE-26482 HMaster may clean wals that is replicating in rare cases
zhengzhuobinzzb commented on a change in pull request #3876: URL: https://github.com/apache/hbase/pull/3876#discussion_r755772858 ## File path: hbase-replication/src/main/java/org/apache/hadoop/hbase/replication/ZKReplicationQueueStorage.java ## @@ -417,6 +417,11 @@ public long getWALPosition(ServerName serverName, String queueId, String fileNam } // add delete op for peer listOfOps.add(ZKUtilOp.deleteNodeFailSilent(oldQueueNode)); + // Append new queue id for prevent lock competition in zookeeper server. + String claimLockZNode = ZNodePaths.joinZNode(queuesZNode, "cversion_" + newQueueId); + // update cversion for queuesZNode + listOfOps.add(ZKUtilOp.createAndFailSilent(claimLockZNode, HConstants.EMPTY_BYTE_ARRAY)); Review comment: You are right, I added more comments and documentation to explain why we update the root queuesZNode and the relationship between claimQueue and getAllWALs -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (HBASE-26452) Table name and Column Family Name not always initialized in the HFileContext
[
https://issues.apache.org/jira/browse/HBASE-26452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kulwant Singh reassigned HBASE-26452:
-
Assignee: Kulwant Singh
> Table name and Column Family Name not always initialized in the HFileContext
>
>
> Key: HBASE-26452
> URL: https://issues.apache.org/jira/browse/HBASE-26452
> Project: HBase
> Issue Type: Bug
> Components: BucketCache, regionserver
>Affects Versions: 3.0.0-alpha-1, 3.0.0-alpha-2, 2.4.8
>Reporter: Kulwant Singh
>Assignee: Kulwant Singh
>Priority: Minor
>
> {noformat}
> Example:{noformat}
> {noformat}
> 2021-10-21 01:48:42,860 WARN [main-BucketCacheWriter-0] bucket.BucketCache:
> Most recent failed allocation in 6 milliseconds; Table
> Name = null, Column Family = null, HFile Name :
> 1919692f80b841b7a3d4453ec01186f1
> org.apache.hadoop.hbase.io.hfile.bucket.BucketAllocatorException: Allocation
> too big size=1050637; adjust BucketCache sizes hbase.bucket
> cache.bucket.sizes to accomodate if size seems reasonable and you want it
> cached.
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (HBASE-26458) Value of hbase.master.snapshot.ttl is not used
[
https://issues.apache.org/jira/browse/HBASE-26458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Viraj Jasani updated HBASE-26458:
-
Fix Version/s: 3.0.0-alpha-2
1.7.2
(was: 1.7.1)
(was: 2.4.8)
Hadoop Flags: Reviewed
Release Note: (was: Fixed bug where hbase.master.snapshot.ttl was not
respected.)
Resolution: Fixed
Status: Resolved (was: Patch Available)
The code fix is merged to branch-1 and the documentation is improved on master
branch.
Thanks for the contribution [~jswi].
> Value of hbase.master.snapshot.ttl is not used
> --
>
> Key: HBASE-26458
> URL: https://issues.apache.org/jira/browse/HBASE-26458
> Project: HBase
> Issue Type: Bug
> Components: snapshots
>Affects Versions: 1.7.1, 2.4.8
>Reporter: Joel Swiatek
>Assignee: Joel Swiatek
>Priority: Minor
> Fix For: 3.0.0-alpha-2, 1.7.2
>
> Attachments:
> branch-1-0001-HBASE-22458-Add-UNSET_SNAPSHOT_PROP-and-fix-TTL-defa.patch,
> branch-2-0001-HBASE-22458-Add-UNSET_SNAPSHOT_PROP-and-fix-TTL-defa.patch,
> master-0001-HBASE-22458-Add-UNSET_SNAPSHOT_PROP-and-fix-TTL-defa.patch
>
> Original Estimate: 120h
> Remaining Estimate: 120h
>
> When creating a snapshot, users can explicitly specify the TTL to be used. If
> no TTL is specified, then the SnapshotDescription is initially created with a
> TTL of -1 to indicate FOREVER.
> When the SnapshotDescription runs through SnapshotDescriptionUtils#validate,
> the TTL is checked to see if the default value of hbase.master.snapshot.ttl
> should be applied. The value from the config is only applied if the TTL == 0,
> but it should be -1.
> This has another nasty side-effect: any user who creates a snapshot and
> explicitly sets \{TTL => 0} will find that their snapshot gets its TTL from
> hbase.master.snapshot.TTL.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [hbase] virajjasani merged pull request #3857: HBASE-26458 Add UNSET_SNAPSHOT_PROP and fix TTL defaulting
virajjasani merged pull request #3857: URL: https://github.com/apache/hbase/pull/3857 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] virajjasani merged pull request #3852: HBASE-26458 Update Snapshot TTL doc
virajjasani merged pull request #3852: URL: https://github.com/apache/hbase/pull/3852 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-26475) The flush and compact methods in HTU should skip processing secondary replicas
[
https://issues.apache.org/jira/browse/HBASE-26475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17448335#comment-17448335
]
Hudson commented on HBASE-26475:
Results for branch branch-2.4
[build #245 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/245/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/245/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/245/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/245/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/245/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> The flush and compact methods in HTU should skip processing secondary replicas
> --
>
> Key: HBASE-26475
> URL: https://issues.apache.org/jira/browse/HBASE-26475
> Project: HBase
> Issue Type: Improvement
> Components: test
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 2.5.0, 3.0.0-alpha-2, 2.4.9
>
>
> To avoid generating bunch of "NOT flushing because not writable" warning logs.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [hbase] comnetwork commented on a change in pull request #3859: HBASE-26465 MemStoreLAB may be released early when its SegmentScanner…
comnetwork commented on a change in pull request #3859:
URL: https://github.com/apache/hbase/pull/3859#discussion_r755648268
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java
##
@@ -1222,6 +1225,14 @@ private boolean updateStorefiles(List sfs,
long snapshotId) throws I
this.lock.writeLock().lock();
try {
this.storeEngine.getStoreFileManager().insertNewFiles(sfs);
+ /**
+ * NOTE:we should keep clearSnapshot method inside the write lock
because clearSnapshot may
+ * close {@link DefaultMemStore#snapshot}, which may be used by
+ * {@link DefaultMemStore#getScanners}.
+ */
+ if (snapshotId > 0) {
Review comment:
@anoopsjohn, thank you for review, yes, you are right ,that is the
actual fix part.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Assigned] (HBASE-26482) HMaster may clean wals that is replicating in rare cases
[
https://issues.apache.org/jira/browse/HBASE-26482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duo Zhang reassigned HBASE-26482:
-
Assignee: zhuobin zheng
> HMaster may clean wals that is replicating in rare cases
>
>
> Key: HBASE-26482
> URL: https://issues.apache.org/jira/browse/HBASE-26482
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Reporter: zhuobin zheng
>Assignee: zhuobin zheng
>Priority: Critical
>
> In our cluster, i can found some FileNotFoundException when
> ReplicationSourceWALReader running for replication recovery queue.
> I guss the wal most likely removed by hmaste. And i found something to
> support it.
> The method getAllWALs:
> [https://github.com/apache/hbase/blob/master/hbase-replication/src/main/java/org/apache/hadoop/hbase/replication/ZKReplicationQueueStorage.java#L509
>
> |https://github.com/apache/hbase/blob/master/hbase-replication/src/main/java/org/apache/hadoop/hbase/replication/ZKReplicationQueueStorage.java#L509]Use
> zk cversion of /hbase/replication/rs as an optimistic lock to control
> concurrent ops.
> But, zk cversion *only can only reflect the changes of child nodes, but not
> the changes of grandchildren.*
> So, HMaster may loss some wal from this method in follow situation.
> # HMaster do log clean , and invoke getAllWALs to filter log which should
> not be deleted.
> # HMaster cache current cversion of /hbase/replication/rs as *v0*
> # HMaster cache all RS server name, and traverse them, get the WAL in each
> Queue
> # *RS2* dead after HMaster traverse {*}RS1{*}, and before traverse *RS2*
> # *RS1* claim one queue of *RS2,* which named *peerid-RS2* now
> # By the way , the cversion of /hbase/replication/rs not changed before all
> of *RS2* queue is removed, because the children of /hbase/replication/rs not
> change.
> # So, Hmaster will lost the wals in *peerid-RS2,* because we have already
> traversed *RS1 ,* and ** this queue not exists in *RS2*
> The above expression is currently only speculation, not confirmed
> Flie Not Found Log.
>
> {code:java}
> // code placeholder
> 2021-11-22 15:18:39,593 ERROR
> [ReplicationExecutor-0.replicationSource,peer_id-hostname,60020,1636802867348.replicationSource.wal-reader.hostname%2C60020%2C1636802867348,peer_id-hostname,60020,1636802867348]
> regionserver.WALEntryStream: Couldn't locate log:
> hdfs://namenode/hbase/oldWALs/hostname%2C60020%2C1636802867348.1636944748704
> 2021-11-22 15:18:39,593 ERROR
> [ReplicationExecutor-0.replicationSource,peer_id-hostname,60020,1636802867348.replicationSource.wal-reader.hostname%2C60020%2C1636802867348,peer_id-hostname,60020,1636802867348]
> regionserver.ReplicationSourceWALReader: Failed to read stream of
> replication entries
> java.io.FileNotFoundException: File does not exist:
> hdfs://namenode/hbase/oldWALs/hostname%2C60020%2C1636802867348.1636944748704
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1612)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1605)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1620)
> at
> org.apache.hadoop.hbase.regionserver.wal.ReaderBase.init(ReaderBase.java:64)
> at
> org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.init(ProtobufLogReader.java:168)
> at
> org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:321)
> at
> org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:303)
> at
> org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:291)
> at
> org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:427)
> at
> org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.openReader(WALEntryStream.java:355)
> at
> org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.openNextLog(WALEntryStream.java:303)
> at
> org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.checkReader(WALEntryStream.java:294)
> at
> org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.tryAdvanceEntry(WALEntryStream.java:175)
> at
> org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.hasNext(WALEntryStream.java:101)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.readWALEntries(ReplicationSourceWALReader.java:192)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.run(ReplicationSourceWALReader.java:138)
> {code}
>
>
--
This message was sent by Atlassian Jir
[jira] [Updated] (HBASE-26482) HMaster may clean wals that is replicating in rare cases
[
https://issues.apache.org/jira/browse/HBASE-26482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duo Zhang updated HBASE-26482:
--
Fix Version/s: 2.5.0
3.0.0-alpha-2
2.4.9
> HMaster may clean wals that is replicating in rare cases
>
>
> Key: HBASE-26482
> URL: https://issues.apache.org/jira/browse/HBASE-26482
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Reporter: zhuobin zheng
>Assignee: zhuobin zheng
>Priority: Critical
> Fix For: 2.5.0, 3.0.0-alpha-2, 2.4.9
>
>
> In our cluster, i can found some FileNotFoundException when
> ReplicationSourceWALReader running for replication recovery queue.
> I guss the wal most likely removed by hmaste. And i found something to
> support it.
> The method getAllWALs:
> [https://github.com/apache/hbase/blob/master/hbase-replication/src/main/java/org/apache/hadoop/hbase/replication/ZKReplicationQueueStorage.java#L509
>
> |https://github.com/apache/hbase/blob/master/hbase-replication/src/main/java/org/apache/hadoop/hbase/replication/ZKReplicationQueueStorage.java#L509]Use
> zk cversion of /hbase/replication/rs as an optimistic lock to control
> concurrent ops.
> But, zk cversion *only can only reflect the changes of child nodes, but not
> the changes of grandchildren.*
> So, HMaster may loss some wal from this method in follow situation.
> # HMaster do log clean , and invoke getAllWALs to filter log which should
> not be deleted.
> # HMaster cache current cversion of /hbase/replication/rs as *v0*
> # HMaster cache all RS server name, and traverse them, get the WAL in each
> Queue
> # *RS2* dead after HMaster traverse {*}RS1{*}, and before traverse *RS2*
> # *RS1* claim one queue of *RS2,* which named *peerid-RS2* now
> # By the way , the cversion of /hbase/replication/rs not changed before all
> of *RS2* queue is removed, because the children of /hbase/replication/rs not
> change.
> # So, Hmaster will lost the wals in *peerid-RS2,* because we have already
> traversed *RS1 ,* and ** this queue not exists in *RS2*
> The above expression is currently only speculation, not confirmed
> Flie Not Found Log.
>
> {code:java}
> // code placeholder
> 2021-11-22 15:18:39,593 ERROR
> [ReplicationExecutor-0.replicationSource,peer_id-hostname,60020,1636802867348.replicationSource.wal-reader.hostname%2C60020%2C1636802867348,peer_id-hostname,60020,1636802867348]
> regionserver.WALEntryStream: Couldn't locate log:
> hdfs://namenode/hbase/oldWALs/hostname%2C60020%2C1636802867348.1636944748704
> 2021-11-22 15:18:39,593 ERROR
> [ReplicationExecutor-0.replicationSource,peer_id-hostname,60020,1636802867348.replicationSource.wal-reader.hostname%2C60020%2C1636802867348,peer_id-hostname,60020,1636802867348]
> regionserver.ReplicationSourceWALReader: Failed to read stream of
> replication entries
> java.io.FileNotFoundException: File does not exist:
> hdfs://namenode/hbase/oldWALs/hostname%2C60020%2C1636802867348.1636944748704
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1612)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1605)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1620)
> at
> org.apache.hadoop.hbase.regionserver.wal.ReaderBase.init(ReaderBase.java:64)
> at
> org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.init(ProtobufLogReader.java:168)
> at
> org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:321)
> at
> org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:303)
> at
> org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:291)
> at
> org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:427)
> at
> org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.openReader(WALEntryStream.java:355)
> at
> org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.openNextLog(WALEntryStream.java:303)
> at
> org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.checkReader(WALEntryStream.java:294)
> at
> org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.tryAdvanceEntry(WALEntryStream.java:175)
> at
> org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.hasNext(WALEntryStream.java:101)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.readWALEntries(ReplicationSourceWALReader.java:192)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.ru
[jira] [Created] (HBASE-26484) Update vote.tmpl in our create-release scripts to link KEYS from downloads.a.o instead of dist.a.o
Duo Zhang created HBASE-26484: - Summary: Update vote.tmpl in our create-release scripts to link KEYS from downloads.a.o instead of dist.a.o Key: HBASE-26484 URL: https://issues.apache.org/jira/browse/HBASE-26484 Project: HBase Issue Type: Task Reporter: Duo Zhang This is required by the ASF policy. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [hbase] Apache9 commented on a change in pull request #3876: HBASE-26482 HMaster may clean wals that is replicating in rare cases
Apache9 commented on a change in pull request #3876:
URL: https://github.com/apache/hbase/pull/3876#discussion_r755633516
##
File path:
hbase-replication/src/test/java/org/apache/hadoop/hbase/replication/TestZKReplicationQueueStorage.java
##
@@ -206,18 +206,22 @@ public void testAddRemoveLog() throws
ReplicationException {
}
}
- // For HBASE-12865
+ // For HBASE-12865, HBASE-26482
Review comment:
Better introduce another test? I think this test is to confirm that when
there is only a single queue than we will delete the server znode so the
cversion of the queuesZNode will be changed.
We should add another test where even we do not delete the server znode, the
cversion of the queuesZNode will still be changed.
##
File path:
hbase-replication/src/main/java/org/apache/hadoop/hbase/replication/ZKReplicationQueueStorage.java
##
@@ -417,6 +417,11 @@ public long getWALPosition(ServerName serverName, String
queueId, String fileNam
}
// add delete op for peer
listOfOps.add(ZKUtilOp.deleteNodeFailSilent(oldQueueNode));
+ // Append new queue id for prevent lock competition in zookeeper server.
+ String claimLockZNode = ZNodePaths.joinZNode(queuesZNode, "cversion_" +
newQueueId);
+ // update cversion for queuesZNode
+ listOfOps.add(ZKUtilOp.createAndFailSilent(claimLockZNode,
HConstants.EMPTY_BYTE_ARRAY));
Review comment:
So this is the trick. Better speak more clearly that this is for
changing the cversion of the root queueZNode, and add a link to the getAllWALs
method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] zhengzhuobinzzb commented on pull request #3876: HBASE-26482 HMaster may clean wals that is replicating in rare cases
zhengzhuobinzzb commented on pull request #3876: URL: https://github.com/apache/hbase/pull/3876#issuecomment-977381822 @Apache9 @apurtell @sunhelly @clarax @comnetwork Please help review in your free time, Thanks~ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] zhengzhuobinzzb commented on pull request #3876: HBASE-26482 HMaster may clean wals that is replicating in rare cases
zhengzhuobinzzb commented on pull request #3876: URL: https://github.com/apache/hbase/pull/3876#issuecomment-977378637 Precommit check failed: ' /home/jenkins/jenkins-home/workspace/HBase: Read-only file system ' Should i force push to re-trigger checks? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-24749) Direct insert HFiles and Persist in-memory HFile tracking
[
https://issues.apache.org/jira/browse/HBASE-24749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17448305#comment-17448305
]
Tak-Lon (Stephen) Wu commented on HBASE-24749:
--
no issue here, thanks for working on HBASE-26067
> Direct insert HFiles and Persist in-memory HFile tracking
> -
>
> Key: HBASE-24749
> URL: https://issues.apache.org/jira/browse/HBASE-24749
> Project: HBase
> Issue Type: Umbrella
> Components: Compaction, HFile
>Affects Versions: 3.0.0-alpha-1
>Reporter: Tak-Lon (Stephen) Wu
>Assignee: Tak-Lon (Stephen) Wu
>Priority: Major
> Labels: design, discussion, objectstore, storeFile, storeengine
> Attachments: 1B100m-25m25m-performance.pdf, Apache HBase - Direct
> insert HFiles and Persist in-memory HFile tracking.pdf
>
>
> We propose a new feature (a new store engine) to remove the {{.tmp}}
> directory used in the commit stage for common HFile operations such as flush
> and compaction to improve the write throughput and latency on object stores.
> Specifically for S3 filesystems, this will also mitigate read-after-write
> inconsistencies caused by immediate HFiles validation after moving the
> HFile(s) to data directory.
> Please see attached for this proposal and the initial result captured with
> 25m (25m operations) and 1B (100m operations) YCSB workload A LOAD and RUN,
> and workload C RUN result.
> The goal of this JIRA is to discuss with the community if the proposed
> improvement on the object stores use case makes senses and if we miss
> anything should be included.
> Improvement Highlights
> 1. Lower write latency, especially the p99+
> 2. Higher write throughput on flush and compaction
> 3. Lower MTTR on region (re)open or assignment
> 4. Remove consistent check dependencies (e.g. DynamoDB) supported by file
> system implementation
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [hbase] Apache-HBase commented on pull request #3857: HBASE-26458 Add UNSET_SNAPSHOT_PROP and fix TTL defaulting
Apache-HBase commented on pull request #3857: URL: https://github.com/apache/hbase/pull/3857#issuecomment-977270156 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 7s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | -0 :warning: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ branch-1 Compile Tests _ | | +0 :ok: | mvndep | 2m 31s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 8m 20s | branch-1 passed | | +1 :green_heart: | compile | 2m 5s | branch-1 passed with JDK Azul Systems, Inc.-1.8.0_262-b19 | | +1 :green_heart: | compile | 2m 0s | branch-1 passed with JDK Azul Systems, Inc.-1.7.0_272-b10 | | +1 :green_heart: | checkstyle | 7m 42s | branch-1 passed | | +0 :ok: | refguide | 4m 1s | branch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect. | | +1 :green_heart: | shadedjars | 3m 10s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 3m 28s | branch-1 passed with JDK Azul Systems, Inc.-1.8.0_262-b19 | | +1 :green_heart: | javadoc | 4m 14s | branch-1 passed with JDK Azul Systems, Inc.-1.7.0_272-b10 | | +0 :ok: | spotbugs | 2m 50s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 15m 29s | branch-1 passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 8s | the patch passed | | +1 :green_heart: | compile | 2m 2s | the patch passed with JDK Azul Systems, Inc.-1.8.0_262-b19 | | +1 :green_heart: | javac | 2m 2s | the patch passed | | +1 :green_heart: | compile | 2m 0s | the patch passed with JDK Azul Systems, Inc.-1.7.0_272-b10 | | +1 :green_heart: | javac | 2m 0s | the patch passed | | +1 :green_heart: | checkstyle | 7m 39s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +0 :ok: | refguide | 3m 0s | patch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect. | | +1 :green_heart: | shadedjars | 3m 4s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | hadoopcheck | 5m 4s | Patch does not cause any errors with Hadoop 2.8.5 2.9.2. | | +1 :green_heart: | javadoc | 3m 33s | the patch passed with JDK Azul Systems, Inc.-1.8.0_262-b19 | | +1 :green_heart: | javadoc | 4m 10s | the patch passed with JDK Azul Systems, Inc.-1.7.0_272-b10 | | +1 :green_heart: | findbugs | 16m 3s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 180m 34s | root in the patch failed. | | +1 :green_heart: | asflicense | 2m 13s | The patch does not generate ASF License warnings. | | | | 290m 47s | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.hbase.mapreduce.TestSecureLoadIncrementalHFiles | | | hadoop.hbase.replication.TestReplicationSource | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3857/4/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3857 | | Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile refguide | | uname | Linux b2f09c5a2894 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-home/workspace/Base-PreCommit-GitHub-PR_PR-3857/out/precommit/personality/provided.sh | | git revision | branch-1 / 1701f0de0a | | Default Java | Azul Systems, Inc.-1.7.0_272-b10 | | Multi-JDK versions | /usr/lib/jvm/zulu-8-amd64:Azul Systems, Inc.-1.8.0_262-b19 /usr/lib/jvm/zulu-7-amd64:Azul Systems, Inc.-1.7.0_272-b10 | | refguide | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3857/4/artifact/out/branch-site/book.html | | refguide | https://ci-hadoop.apache.org/job/HBa
[jira] [Commented] (HBASE-26067) Change the way on how we track store file list
[ https://issues.apache.org/jira/browse/HBASE-26067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17448302#comment-17448302 ] Zach York commented on HBASE-26067: --- [~wchevreuil][~zhangduo] Do you have any initial perf numbers on this? I'm curious how it compares to Stephen's original results. > Change the way on how we track store file list > -- > > Key: HBASE-26067 > URL: https://issues.apache.org/jira/browse/HBASE-26067 > Project: HBase > Issue Type: Umbrella > Components: HFile >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > > Open a separated jira to track the work since it can not be fully included in > HBASE-24749. > I think this could be a landed prior to HBASE-24749, as if this works, we > could have different implementations for tracking store file list. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (HBASE-24749) Direct insert HFiles and Persist in-memory HFile tracking
[
https://issues.apache.org/jira/browse/HBASE-24749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17448301#comment-17448301
]
Zach York commented on HBASE-24749:
---
I think it's fine to close this issue. If we later determine this approach has
better characteristics than the implemented approach, we can always implement
it within the new interfaces and can reopen/create a new issue.
> Direct insert HFiles and Persist in-memory HFile tracking
> -
>
> Key: HBASE-24749
> URL: https://issues.apache.org/jira/browse/HBASE-24749
> Project: HBase
> Issue Type: Umbrella
> Components: Compaction, HFile
>Affects Versions: 3.0.0-alpha-1
>Reporter: Tak-Lon (Stephen) Wu
>Assignee: Tak-Lon (Stephen) Wu
>Priority: Major
> Labels: design, discussion, objectstore, storeFile, storeengine
> Attachments: 1B100m-25m25m-performance.pdf, Apache HBase - Direct
> insert HFiles and Persist in-memory HFile tracking.pdf
>
>
> We propose a new feature (a new store engine) to remove the {{.tmp}}
> directory used in the commit stage for common HFile operations such as flush
> and compaction to improve the write throughput and latency on object stores.
> Specifically for S3 filesystems, this will also mitigate read-after-write
> inconsistencies caused by immediate HFiles validation after moving the
> HFile(s) to data directory.
> Please see attached for this proposal and the initial result captured with
> 25m (25m operations) and 1B (100m operations) YCSB workload A LOAD and RUN,
> and workload C RUN result.
> The goal of this JIRA is to discuss with the community if the proposed
> improvement on the object stores use case makes senses and if we miss
> anything should be included.
> Improvement Highlights
> 1. Lower write latency, especially the p99+
> 2. Higher write throughput on flush and compaction
> 3. Lower MTTR on region (re)open or assignment
> 4. Remove consistent check dependencies (e.g. DynamoDB) supported by file
> system implementation
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [hbase-filesystem] wchevreuil commented on a change in pull request #30: HBASE-26483. [HBOSS] add support for openFile(path)
wchevreuil commented on a change in pull request #30:
URL: https://github.com/apache/hbase-filesystem/pull/30#discussion_r78431
##
File path:
hbase-oss/src/main/java/org/apache/hadoop/hbase/oss/HBaseObjectStoreSemantics.java
##
@@ -170,6 +173,26 @@ public FSDataInputStream open(Path f) throws IOException {
}
}
+ @Override
+ public FutureDataInputStreamBuilder openFile(final Path path)
+ throws IOException, UnsupportedOperationException {
+return new LockedFutureDataInputStreamBuilder(sync, path,
+fs.openFile(path));
+ }
+
+ /**
+ * This is mostly unsupported, and as there's no way to
+ * get the path from a pathHandle, impossible to lock.
+ * @param pathHandle path
+ * @return never returns successfully.
+ * @throws UnsupportedOperationException always
+ */
+ @Override
+ public FutureDataInputStreamBuilder openFile(final PathHandle pathHandle)
+ throws IOException, UnsupportedOperationException {
+throw new UnsupportedOperationException("openFile(PathHandle)
unsupported");
+ }
+
Review comment:
It seems we also missed wrapping `open(PathHandle fd, int bufferSize)`,
on HBaseObjectStoreSemantics. Can we do something similar?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (HBASE-26249) Ameliorate compaction made by bulk-loading files
[
https://issues.apache.org/jira/browse/HBASE-26249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17448209#comment-17448209
]
Hudson commented on HBASE-26249:
Results for branch branch-2
[build #399 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/399/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/399/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/399/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/399/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/399/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Ameliorate compaction made by bulk-loading files
>
>
> Key: HBASE-26249
> URL: https://issues.apache.org/jira/browse/HBASE-26249
> Project: HBase
> Issue Type: Improvement
>Reporter: Xiaolin Ha
>Assignee: Xiaolin Ha
>Priority: Major
> Fix For: 2.5.0, 3.0.0-alpha-2
>
>
> HBASE-25213 introduced to trigger compact after bulkload, but if a store
> needs compaction, each file loaded to this store will add a compaction runner
> of this store to the compaction queue. Then repeatedly compute for
> needCompaction and selectFiles for this store, and the compaction of the
> store may be continuous.
> We can mark the queued compaction runners for the stores, and avoid
> repeatedly adding same store compactions to the queue after bulk load files.
> This method can reduce the pressure during the period just after bulk loading
> and spread the pressure to other times by the periodic CompactionChecker.
> For the need of splitting mentioned before, after the completion of
> compaction, it will trigger split of the region if it needs. So we do not
> need to trigger split after bulk loading.
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (HBASE-26470) Use openlabtesting protoc on linux arm64 in HBASE 2.x
[
https://issues.apache.org/jira/browse/HBASE-26470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17448208#comment-17448208
]
Hudson commented on HBASE-26470:
Results for branch branch-2
[build #399 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/399/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/399/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/399/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/399/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/399/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Use openlabtesting protoc on linux arm64 in HBASE 2.x
> -
>
> Key: HBASE-26470
> URL: https://issues.apache.org/jira/browse/HBASE-26470
> Project: HBase
> Issue Type: Sub-task
> Components: build
>Reporter: Mark Jens
>Assignee: Mark Jens
>Priority: Major
> Fix For: 2.5.0, 2.4.9
>
>
> This is the same issue as HBASE-23612 but for HBase 2.x.
> Please backport the fix to HBase 2.x because at the moment it is not possible
> to use Apache Phoenix on Linux ARM64.
> I will send a new Pull Request!
> -
> Currently, there is no protoc 2.5.0 for release [1]. So we can make a new one
> for ARM specific. For make sure that could work on ARM.
> We will introduce a new ARM artifact for protoc, group_id is
> org.openlabtesting.protobuf .. This is just used for protobuf-maven-plugin to
> compile .proto files. As the 3.X version of protoc support ARM already. So
> this won't affect the internal protoc usage, which is 3.5.1-1 now.
>
> [1][https://github.com/protocolbuffers/protobuf/issues/3844#issuecomment-343355946]
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [hbase-filesystem] steveloughran opened a new pull request #30: HBASE-26483. [HBOSS] add support for openFile(path)
steveloughran opened a new pull request #30: URL: https://github.com/apache/hbase-filesystem/pull/30 Adds support for the openFile(path) method which returns a builder to open a file. Currently this falls through to the wrapped class. With this patch a builder is returned which acquires a lock in the final build() call, not when the openFile(path) method is invoked -as that does not trigger any file IO. openFile(PathHandle) is rejected as unsupported. This doesn't build with the hadoop-3.2 profile, but that didn't seem to build anyway. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (HBASE-26483) [HBOSS] add lock around openFile operation
Steve Loughran created HBASE-26483: -- Summary: [HBOSS] add lock around openFile operation Key: HBASE-26483 URL: https://issues.apache.org/jira/browse/HBASE-26483 Project: HBase Issue Type: Improvement Components: hboss Reporter: Steve Loughran The HBoss FS wrapper doesn't wrap the openFile(path) call with a lock, which means anything using that builder isn't going to have access synchronized. adding a wrapper for this method will allow hbase to use the api call and so request different read policies on different files, or other options -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] (HBASE-26250) Automatic and near real-time healing of locality
[ https://issues.apache.org/jira/browse/HBASE-26250 ] Bryan Beaudreault deleted comment on HBASE-26250: --- was (Author: bbeaudreault): While I'm looking for overall feedback on the approach so that I can contribute this tool to the community, I'd also like to specifically request feedback/guidance on the design described in the "Refreshing HFiles" section. As some of the devs know, I'm working on a major version upgrade at HubSpot and need to update our internal tool to work with HBase2. The internals in this area have changed a bit and I don't want to assume that a straight port of our cdh5-based implementation will be the best approach for HBase2. Thanks! > Automatic and near real-time healing of locality > > > Key: HBASE-26250 > URL: https://issues.apache.org/jira/browse/HBASE-26250 > Project: HBase > Issue Type: New Feature >Reporter: Bryan Beaudreault >Assignee: Bryan Beaudreault >Priority: Major > > I’m proposing a somewhat major new tool for quickly and efficiently > alleviating latency pains due to locality. This is especially useful in cloud > environments, and has been highly impactful at HubSpot, where we run > thousands of RegionServers across 40+ multi-zone clusters. Please see the > attached design doc for details on the problem, why compactions are not > enough to solve the problem, and an overview (with diagram) of the components > that make up this new tool. > As spec'd, this new feature would require submission of a new tool in the > HDFS project. Once we reach consensus on the approach I can create the > relevant upstream HDFS JIRA. > See the design doc here: > [https://docs.google.com/document/d/1GLGzrF1QLyhyOCr2fFw0LCymnyFPT0ktShTaaXn-75A/edit#heading=h.aswo7shg76b6] > Note: This issue is an attempt to upstream a tool that has been fully > deployed for all clusters in production at HubSpot for about 6 months. It's > been very effective for us as currently implemented, but will need to be > re-organized and re-designed a bit to fit into the HBase/HDFS projects. As > such I'd like feedback on the design before putting in too much effort on > porting multiple components into PRs. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [hbase] Apache-HBase commented on pull request #3875: HBASE-26459 HMaster should move non-meta region only if meta is ONLINE
Apache-HBase commented on pull request #3875: URL: https://github.com/apache/hbase/pull/3875#issuecomment-976788982 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 6m 12s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 1s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | -0 :warning: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ branch-1 Compile Tests _ | | +0 :ok: | mvndep | 2m 36s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 9m 27s | branch-1 passed | | +1 :green_heart: | compile | 1m 20s | branch-1 passed with JDK Azul Systems, Inc.-1.8.0_262-b19 | | +1 :green_heart: | compile | 1m 28s | branch-1 passed with JDK Azul Systems, Inc.-1.7.0_272-b10 | | +1 :green_heart: | checkstyle | 2m 47s | branch-1 passed | | +1 :green_heart: | shadedjars | 4m 42s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 18s | branch-1 passed with JDK Azul Systems, Inc.-1.8.0_262-b19 | | +1 :green_heart: | javadoc | 1m 26s | branch-1 passed with JDK Azul Systems, Inc.-1.7.0_272-b10 | | +0 :ok: | spotbugs | 3m 33s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 5m 12s | branch-1 passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 20s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 53s | the patch passed | | +1 :green_heart: | compile | 1m 28s | the patch passed with JDK Azul Systems, Inc.-1.8.0_262-b19 | | +1 :green_heart: | javac | 1m 28s | the patch passed | | +1 :green_heart: | compile | 1m 36s | the patch passed with JDK Azul Systems, Inc.-1.7.0_272-b10 | | +1 :green_heart: | javac | 1m 36s | the patch passed | | -1 :x: | checkstyle | 1m 58s | hbase-server: The patch generated 1 new + 212 unchanged - 0 fixed = 213 total (was 212) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedjars | 4m 8s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | hadoopcheck | 6m 30s | Patch does not cause any errors with Hadoop 2.8.5 2.9.2. | | +1 :green_heart: | javadoc | 1m 4s | the patch passed with JDK Azul Systems, Inc.-1.8.0_262-b19 | | +1 :green_heart: | javadoc | 1m 21s | the patch passed with JDK Azul Systems, Inc.-1.7.0_272-b10 | | +1 :green_heart: | findbugs | 5m 27s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 3m 2s | hbase-common in the patch passed. | | -1 :x: | unit | 167m 13s | hbase-server in the patch failed. | | +1 :green_heart: | asflicense | 0m 50s | The patch does not generate ASF License warnings. | | | | 239m 51s | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.hbase.master.cleaner.TestSnapshotFromMaster | | | hadoop.hbase.replication.TestMasterReplication | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3875/2/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3875 | | Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile | | uname | Linux db9637011fd5 4.15.0-143-generic #147-Ubuntu SMP Wed Apr 14 16:10:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-agent/workspace/Base-PreCommit-GitHub-PR_PR-3875/out/precommit/personality/provided.sh | | git revision | branch-1 / 1701f0de0a | | Default Java | Azul Systems, Inc.-1.7.0_272-b10 | | Multi-JDK versions | /usr/lib/jvm/zulu-8-amd64:Azul Systems, Inc.-1.8.0_262-b19 /usr/lib/jvm/zulu-7-amd64:Azul Systems, Inc.-1.7.0_272-b10 | | checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3875/2/artifact/out/diff-checkstyle-hbase-server.txt | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3875/2/artifact/out/patch-unit-hbase-server.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR
[GitHub] [hbase] Apache-HBase commented on pull request #3876: HBASE-26482 HMaster may clean wals that is replicating in rare cases
Apache-HBase commented on pull request #3876: URL: https://github.com/apache/hbase/pull/3876#issuecomment-976776768 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 31s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 6m 6s | master passed | | +1 :green_heart: | compile | 0m 21s | master passed | | +1 :green_heart: | shadedjars | 10m 4s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 23s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 5m 58s | the patch passed | | +1 :green_heart: | compile | 0m 21s | the patch passed | | +1 :green_heart: | javac | 0m 21s | the patch passed | | +1 :green_heart: | shadedjars | 11m 7s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 19s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 44s | hbase-replication in the patch passed. | | | | 37m 59s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3876/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3876 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 9478596e36e6 4.15.0-142-generic #146-Ubuntu SMP Tue Apr 13 01:11:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 96d699a8f0 | | Default Java | AdoptOpenJDK-11.0.10+9 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3876/1/testReport/ | | Max. process+thread count | 285 (vs. ulimit of 3) | | modules | C: hbase-replication U: hbase-replication | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3876/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3876: HBASE-26482 HMaster may clean wals that is replicating in rare cases
Apache-HBase commented on pull request #3876: URL: https://github.com/apache/hbase/pull/3876#issuecomment-976767307 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 9s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 47s | master passed | | +1 :green_heart: | compile | 0m 17s | master passed | | +1 :green_heart: | shadedjars | 9m 3s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 17s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 25s | the patch passed | | +1 :green_heart: | compile | 0m 18s | the patch passed | | +1 :green_heart: | javac | 0m 18s | the patch passed | | +1 :green_heart: | shadedjars | 9m 19s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 14s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 42s | hbase-replication in the patch passed. | | | | 31m 27s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3876/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3876 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 1e3339c1b921 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18 19:21:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 96d699a8f0 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3876/1/testReport/ | | Max. process+thread count | 310 (vs. ulimit of 3) | | modules | C: hbase-replication U: hbase-replication | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3876/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-26482) HMaster may clean wals that is replicating in rare cases
[
https://issues.apache.org/jira/browse/HBASE-26482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17448075#comment-17448075
]
zhuobin zheng commented on HBASE-26482:
---
JIRA: HBASE-12865 Has fixed this problem.
HBASE-12865 delete *rsZnode* after all queue claimed. Old method will claim all
queue in rs. So we can delete rs Znode after all queue claimed
But now, In this method, we only claim one queue, so we can't delete *rsZnode.*
I submit a simple patch to fix it by add ops(create znode, delete znode) to
multiOp for zk(for update cversion).
> HMaster may clean wals that is replicating in rare cases
>
>
> Key: HBASE-26482
> URL: https://issues.apache.org/jira/browse/HBASE-26482
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Reporter: zhuobin zheng
>Priority: Critical
>
> In our cluster, i can found some FileNotFoundException when
> ReplicationSourceWALReader running for replication recovery queue.
> I guss the wal most likely removed by hmaste. And i found something to
> support it.
> The method getAllWALs:
> [https://github.com/apache/hbase/blob/master/hbase-replication/src/main/java/org/apache/hadoop/hbase/replication/ZKReplicationQueueStorage.java#L509
>
> |https://github.com/apache/hbase/blob/master/hbase-replication/src/main/java/org/apache/hadoop/hbase/replication/ZKReplicationQueueStorage.java#L509]Use
> zk cversion of /hbase/replication/rs as an optimistic lock to control
> concurrent ops.
> But, zk cversion *only can only reflect the changes of child nodes, but not
> the changes of grandchildren.*
> So, HMaster may loss some wal from this method in follow situation.
> # HMaster do log clean , and invoke getAllWALs to filter log which should
> not be deleted.
> # HMaster cache current cversion of /hbase/replication/rs as *v0*
> # HMaster cache all RS server name, and traverse them, get the WAL in each
> Queue
> # *RS2* dead after HMaster traverse {*}RS1{*}, and before traverse *RS2*
> # *RS1* claim one queue of *RS2,* which named *peerid-RS2* now
> # By the way , the cversion of /hbase/replication/rs not changed before all
> of *RS2* queue is removed, because the children of /hbase/replication/rs not
> change.
> # So, Hmaster will lost the wals in *peerid-RS2,* because we have already
> traversed *RS1 ,* and ** this queue not exists in *RS2*
> The above expression is currently only speculation, not confirmed
> Flie Not Found Log.
>
> {code:java}
> // code placeholder
> 2021-11-22 15:18:39,593 ERROR
> [ReplicationExecutor-0.replicationSource,peer_id-hostname,60020,1636802867348.replicationSource.wal-reader.hostname%2C60020%2C1636802867348,peer_id-hostname,60020,1636802867348]
> regionserver.WALEntryStream: Couldn't locate log:
> hdfs://namenode/hbase/oldWALs/hostname%2C60020%2C1636802867348.1636944748704
> 2021-11-22 15:18:39,593 ERROR
> [ReplicationExecutor-0.replicationSource,peer_id-hostname,60020,1636802867348.replicationSource.wal-reader.hostname%2C60020%2C1636802867348,peer_id-hostname,60020,1636802867348]
> regionserver.ReplicationSourceWALReader: Failed to read stream of
> replication entries
> java.io.FileNotFoundException: File does not exist:
> hdfs://namenode/hbase/oldWALs/hostname%2C60020%2C1636802867348.1636944748704
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1612)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1605)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1620)
> at
> org.apache.hadoop.hbase.regionserver.wal.ReaderBase.init(ReaderBase.java:64)
> at
> org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.init(ProtobufLogReader.java:168)
> at
> org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:321)
> at
> org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:303)
> at
> org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:291)
> at
> org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:427)
> at
> org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.openReader(WALEntryStream.java:355)
> at
> org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.openNextLog(WALEntryStream.java:303)
> at
> org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.checkReader(WALEntryStream.java:294)
> at
> org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.tryAdvanceEntry(WALEntryStream.java:175)
> at
> org.apache.hadoop.hbase.replication.regionserver.WALEntryS
[jira] [Assigned] (HBASE-26481) Consider rolling upgrading from old region replication framework
[ https://issues.apache.org/jira/browse/HBASE-26481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang reassigned HBASE-26481: - Assignee: Duo Zhang > Consider rolling upgrading from old region replication framework > > > Key: HBASE-26481 > URL: https://issues.apache.org/jira/browse/HBASE-26481 > Project: HBase > Issue Type: Sub-task > Components: read replicas >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > > In the old implementation, we will cerate a replication peer for region > replication. And in the new framework, we do not need the peer any more, so > we have deleted the RegionReplicaReplicationEndpoint class. > But when rolling upgrading, the peer is still there so we will try to load it > and it will lead to a class not found exception. We need to handle it. > And during the upgrading, some of the region servers may not have the correct > replicateToReplica method yet. Maybe we could try to catch the no such method > error and try to fallback to use replay method. Although it is not perfect, > but finally a flush can fix everything. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work started] (HBASE-26481) Consider rolling upgrading from old region replication framework
[ https://issues.apache.org/jira/browse/HBASE-26481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HBASE-26481 started by Duo Zhang. - > Consider rolling upgrading from old region replication framework > > > Key: HBASE-26481 > URL: https://issues.apache.org/jira/browse/HBASE-26481 > Project: HBase > Issue Type: Sub-task > Components: read replicas >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > > In the old implementation, we will cerate a replication peer for region > replication. And in the new framework, we do not need the peer any more, so > we have deleted the RegionReplicaReplicationEndpoint class. > But when rolling upgrading, the peer is still there so we will try to load it > and it will lead to a class not found exception. We need to handle it. > And during the upgrading, some of the region servers may not have the correct > replicateToReplica method yet. Maybe we could try to catch the no such method > error and try to fallback to use replay method. Although it is not perfect, > but finally a flush can fix everything. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [hbase] Apache-HBase commented on pull request #3870: HBASE-26479: Print too slow/big scan's operation_id in region server log
Apache-HBase commented on pull request #3870: URL: https://github.com/apache/hbase/pull/3870#issuecomment-976636409 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 9s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 35s | master passed | | +1 :green_heart: | compile | 1m 7s | master passed | | +1 :green_heart: | shadedjars | 9m 30s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 45s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 19s | the patch passed | | +1 :green_heart: | compile | 1m 6s | the patch passed | | +1 :green_heart: | javac | 1m 6s | the patch passed | | +1 :green_heart: | shadedjars | 9m 6s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 38s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 220m 41s | hbase-server in the patch passed. | | | | 254m 47s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3870/3/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3870 | | JIRA Issue | HBASE-26479 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux a607f20de5a7 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18 19:21:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / b870b6d9ac | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3870/3/testReport/ | | Max. process+thread count | 3540 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3870/3/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-24749) Direct insert HFiles and Persist in-memory HFile tracking
[
https://issues.apache.org/jira/browse/HBASE-24749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17448051#comment-17448051
]
Duo Zhang commented on HBASE-24749:
---
For me I think we could close this issue after we merge HBASE-26067 back to
master. And there is an issue for implementing region based SFT. HBASE-26262
But let's see what [~zyork] and [~taklwu] think? They are the author of this
issue.
Thanks.
> Direct insert HFiles and Persist in-memory HFile tracking
> -
>
> Key: HBASE-24749
> URL: https://issues.apache.org/jira/browse/HBASE-24749
> Project: HBase
> Issue Type: Umbrella
> Components: Compaction, HFile
>Affects Versions: 3.0.0-alpha-1
>Reporter: Tak-Lon (Stephen) Wu
>Assignee: Tak-Lon (Stephen) Wu
>Priority: Major
> Labels: design, discussion, objectstore, storeFile, storeengine
> Attachments: 1B100m-25m25m-performance.pdf, Apache HBase - Direct
> insert HFiles and Persist in-memory HFile tracking.pdf
>
>
> We propose a new feature (a new store engine) to remove the {{.tmp}}
> directory used in the commit stage for common HFile operations such as flush
> and compaction to improve the write throughput and latency on object stores.
> Specifically for S3 filesystems, this will also mitigate read-after-write
> inconsistencies caused by immediate HFiles validation after moving the
> HFile(s) to data directory.
> Please see attached for this proposal and the initial result captured with
> 25m (25m operations) and 1B (100m operations) YCSB workload A LOAD and RUN,
> and workload C RUN result.
> The goal of this JIRA is to discuss with the community if the proposed
> improvement on the object stores use case makes senses and if we miss
> anything should be included.
> Improvement Highlights
> 1. Lower write latency, especially the p99+
> 2. Higher write throughput on flush and compaction
> 3. Lower MTTR on region (re)open or assignment
> 4. Remove consistent check dependencies (e.g. DynamoDB) supported by file
> system implementation
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Resolved] (HBASE-26456) Limit the size for one replicating
[ https://issues.apache.org/jira/browse/HBASE-26456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang resolved HBASE-26456. --- Fix Version/s: HBASE-26233 Hadoop Flags: Reviewed Resolution: Fixed Merged to branch HBASE-26233. Thanks [~Xiaolin Ha] for reviewing. > Limit the size for one replicating > -- > > Key: HBASE-26456 > URL: https://issues.apache.org/jira/browse/HBASE-26456 > Project: HBase > Issue Type: Sub-task > Components: read replicas >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: HBASE-26233 > > > Now we will always try to replicate all the pending entries, which could be > too large. We should have a size/count limit, to avoid putting too much > pressure to the target region server for some extreme cases. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (HBASE-24749) Direct insert HFiles and Persist in-memory HFile tracking
[
https://issues.apache.org/jira/browse/HBASE-24749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17448029#comment-17448029
]
Wellington Chevreuil commented on HBASE-24749:
--
We had favoured the design approach from HBASE-26067, and have been doing all
necessary refactorings as subtasks to allow for a pluggable way of creating and
tracking storefiles, so I think this Jira (and all its subtasks), is not
relevant anymore. Can we close it as duplicate or abandoned, to avoid any
confusions on what's still being worked? [~zhangduo] [~elserj] [~zyork]
[~stack]
> Direct insert HFiles and Persist in-memory HFile tracking
> -
>
> Key: HBASE-24749
> URL: https://issues.apache.org/jira/browse/HBASE-24749
> Project: HBase
> Issue Type: Umbrella
> Components: Compaction, HFile
>Affects Versions: 3.0.0-alpha-1
>Reporter: Tak-Lon (Stephen) Wu
>Assignee: Tak-Lon (Stephen) Wu
>Priority: Major
> Labels: design, discussion, objectstore, storeFile, storeengine
> Attachments: 1B100m-25m25m-performance.pdf, Apache HBase - Direct
> insert HFiles and Persist in-memory HFile tracking.pdf
>
>
> We propose a new feature (a new store engine) to remove the {{.tmp}}
> directory used in the commit stage for common HFile operations such as flush
> and compaction to improve the write throughput and latency on object stores.
> Specifically for S3 filesystems, this will also mitigate read-after-write
> inconsistencies caused by immediate HFiles validation after moving the
> HFile(s) to data directory.
> Please see attached for this proposal and the initial result captured with
> 25m (25m operations) and 1B (100m operations) YCSB workload A LOAD and RUN,
> and workload C RUN result.
> The goal of this JIRA is to discuss with the community if the proposed
> improvement on the object stores use case makes senses and if we miss
> anything should be included.
> Improvement Highlights
> 1. Lower write latency, especially the p99+
> 2. Higher write throughput on flush and compaction
> 3. Lower MTTR on region (re)open or assignment
> 4. Remove consistent check dependencies (e.g. DynamoDB) supported by file
> system implementation
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [hbase] shahrs87 commented on pull request #3871: HBASE-26480 Close NamedQueueRecorder to allow HMaster/RS to shutdown gracefully
shahrs87 commented on pull request #3871: URL: https://github.com/apache/hbase/pull/3871#issuecomment-976602170 The failed tests are unrelated to my code. All of them passes locally. ``` [INFO] --- maven-surefire-plugin:2.22.2:test (default-test) @ hbase-server --- [INFO] [INFO] --- [INFO] T E S T S [INFO] --- [INFO] Running org.apache.hadoop.hbase.master.TestMasterBalanceThrottling [INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 34.132 s - in org.apache.hadoop.hbase.master.TestMasterBalanceThrottling [INFO] Running org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesUseSecurityEndPoint [INFO] Tests run: 22, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 356.012 s - in org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesUseSecurityEndPoint [INFO] Running org.apache.hadoop.hbase.mapreduce.TestSecureLoadIncrementalHFiles [INFO] Tests run: 22, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 397.67 s - in org.apache.hadoop.hbase.mapreduce.TestSecureLoadIncrementalHFiles [INFO] Running org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles [INFO] Tests run: 22, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 369.027 s - in org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles [INFO] [INFO] Results: [INFO] [INFO] Tests run: 68, Failures: 0, Errors: 0, Skipped: 0 [INFO] ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-24989) [HBOSS] Some code cleanup
[ https://issues.apache.org/jira/browse/HBASE-24989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17448023#comment-17448023 ] Steve Loughran commented on HBASE-24989: HADOOP-17409 is going to break EmbeddedS3 as we cut all of the s3guard classes. hboss is going to need to remove the bit where s3guard is set up...not needed anyway > [HBOSS] Some code cleanup > - > > Key: HBASE-24989 > URL: https://issues.apache.org/jira/browse/HBASE-24989 > Project: HBase > Issue Type: Improvement >Affects Versions: hbase-filesystem-1.0.0-alpha1 >Reporter: Wellington Chevreuil >Assignee: Wellington Chevreuil >Priority: Trivial > Fix For: hbase-filesystem-1.0.0-alpha2 > > > This is a cleanup of unused methods/imported classes around several classes > of HBOSS project. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [hbase] YutSean commented on pull request #3875: HBASE-26459 HMaster should move non-meta region only if meta is ONLINE
YutSean commented on pull request #3875: URL: https://github.com/apache/hbase/pull/3875#issuecomment-976577529 OK, let me try. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3803: HBASE-26304: Reflect out of band locality improvements in metrics and balancer
Apache-HBase commented on pull request #3803: URL: https://github.com/apache/hbase/pull/3803#issuecomment-976572918 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 27s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 29s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 55s | master passed | | +1 :green_heart: | compile | 1m 52s | master passed | | +1 :green_heart: | shadedjars | 8m 15s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 19s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 18s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 54s | the patch passed | | +1 :green_heart: | compile | 1m 54s | the patch passed | | +1 :green_heart: | javac | 1m 54s | the patch passed | | +1 :green_heart: | shadedjars | 9m 1s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 31s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 59s | hbase-common in the patch passed. | | +1 :green_heart: | unit | 7m 4s | hbase-balancer in the patch passed. | | +1 :green_heart: | unit | 159m 25s | hbase-server in the patch passed. | | | | 203m 54s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3803/9/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3803 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 32e789e8e6ae 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 1c48248ef8 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3803/9/testReport/ | | Max. process+thread count | 3673 (vs. ulimit of 3) | | modules | C: hbase-common hbase-balancer hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3803/9/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-26468) Region Server doesn't exit cleanly incase it crashes.
[
https://issues.apache.org/jira/browse/HBASE-26468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17448017#comment-17448017
]
Rushabh Shah commented on HBASE-26468:
--
> Maybe we could add a delay? For example, if the process does not exit for 30
> seconds, we call System.exit to force quit, and the return value should be
> something other than 0 to indicate that this is a force terminate.
Sounds like a good idea. Thank you [~zhangduo] !
> Region Server doesn't exit cleanly incase it crashes.
> -
>
> Key: HBASE-26468
> URL: https://issues.apache.org/jira/browse/HBASE-26468
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 1.6.0
>Reporter: Rushabh Shah
>Assignee: Rushabh Shah
>Priority: Major
> Fix For: 2.5.0, 3.0.0-alpha-2, 1.7.2, 2.3.8, 2.4.9
>
>
> Observed this in our production cluster running 1.6 version.
> RS crashed due to some reason but the process was still running. On debugging
> more, found out there was 1 non-daemon thread running and that was not
> allowing RS to exit cleanly. Our clusters are managed by Ambari and have auto
> restart capability within them. But since the process was running and pid
> file was present, Ambari also couldn't do much. There will be some bug where
> we will miss to stop some non daemon thread. Shutdown hook will not be called
> unless one of the following 2 conditions are met:
> # The Java virtual machine shuts down in response to two kinds of events:
> The program exits normally, when the last non-daemon thread exits or when the
> exit (equivalently, System.exit) method is invoked, or
> # The virtual machine is terminated in response to a user interrupt, such as
> typing ^C, or a system-wide event, such as user logoff or system shutdown.
> Considering the first condition, when the last non-daemon thread exits or
> when the exit method is invoked.
> Below is the code snippet from
> [HRegionServerCommandLine.java|https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServerCommandLine.java#L51]
> {code:java}
> private int start() throws Exception {
> try {
> if (LocalHBaseCluster.isLocal(conf)) {
> // Ignore this.
> } else {
> HRegionServer hrs =
> HRegionServer.constructRegionServer(regionServerClass, conf);
> hrs.start();
> hrs.join();
> if (hrs.isAborted()) {
> throw new RuntimeException("HRegionServer Aborted");
> }
> }
> } catch (Throwable t) {
> LOG.error("Region server exiting", t);
> return 1;
> }
> return 0;
> }
> {code}
> Within HRegionServer, there is a subtle difference between when a server is
> aborted v/s when it is stopped. If it is stopped, then isAborted will return
> false and it will exit with return code 0.
> Below is the code from
> [ServerCommandLine.java|https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/util/ServerCommandLine.java#L147]
> {code:java}
> public void doMain(String args[]) {
> try {
> int ret = ToolRunner.run(HBaseConfiguration.create(), this, args);
> if (ret != 0) {
> System.exit(ret);
> }
> } catch (Exception e) {
> LOG.error("Failed to run", e);
> System.exit(-1);
> }
> }
> {code}
> If return code is 0, then it won't call System.exit. This means JVM will wait
> to call ShutdownHook until all non daemon threads are stopped which means
> infinite wait if we don't close all non-daemon threads cleanly.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [hbase] shahrs87 commented on pull request #3875: HBASE-26459 HMaster should move non-meta region only if meta is ONLINE
shahrs87 commented on pull request #3875: URL: https://github.com/apache/hbase/pull/3875#issuecomment-976546783 @YutSean Can we add some tests ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-26249) Ameliorate compaction made by bulk-loading files
[
https://issues.apache.org/jira/browse/HBASE-26249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17448013#comment-17448013
]
Hudson commented on HBASE-26249:
Results for branch master
[build #449 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/449/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/449/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/449/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/449/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Ameliorate compaction made by bulk-loading files
>
>
> Key: HBASE-26249
> URL: https://issues.apache.org/jira/browse/HBASE-26249
> Project: HBase
> Issue Type: Improvement
>Reporter: Xiaolin Ha
>Assignee: Xiaolin Ha
>Priority: Major
> Fix For: 2.5.0, 3.0.0-alpha-2
>
>
> HBASE-25213 introduced to trigger compact after bulkload, but if a store
> needs compaction, each file loaded to this store will add a compaction runner
> of this store to the compaction queue. Then repeatedly compute for
> needCompaction and selectFiles for this store, and the compaction of the
> store may be continuous.
> We can mark the queued compaction runners for the stores, and avoid
> repeatedly adding same store compactions to the queue after bulk load files.
> This method can reduce the pressure during the period just after bulk loading
> and spread the pressure to other times by the periodic CompactionChecker.
> For the need of splitting mentioned before, after the completion of
> compaction, it will trigger split of the region if it needs. So we do not
> need to trigger split after bulk loading.
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (HBASE-25869) WAL value compression
[ https://issues.apache.org/jira/browse/HBASE-25869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17447984#comment-17447984 ] Robin Roy commented on HBASE-25869: --- Hello, I have been trying to test this feature in my local environment using ycsb tool, but I am unable to attain the performance improvement as mentioned here. The WAL size isn't much different from the normal scenario. Can you please guide me on how to properly test this feature. > WAL value compression > - > > Key: HBASE-25869 > URL: https://issues.apache.org/jira/browse/HBASE-25869 > Project: HBase > Issue Type: Bug > Components: Operability, wal >Reporter: Andrew Kyle Purtell >Assignee: Andrew Kyle Purtell >Priority: Major > Fix For: 3.0.0-alpha-1, 2.5.0 > > > WAL storage can be expensive, especially if the cell values represented in > the edits are large, consisting of blobs or significant lengths of text. Such > WALs might need to be kept around for a fairly long time to satisfy > replication constraints on a space limited (or space-contended) filesystem. > We have a custom dictionary compression scheme for cell metadata that is > engaged when WAL compression is enabled in site configuration. This is fine > for that application, where we can expect the universe of values and their > lengths in the custom dictionaries to be constrained. For arbitrary cell > values it is better to use one of the available compression codecs, which are > suitable for arbitrary albeit compressible data. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [hbase] Apache-HBase commented on pull request #3875: HBASE-26459 HMaster should move non-meta region only if meta is ONLINE
Apache-HBase commented on pull request #3875: URL: https://github.com/apache/hbase/pull/3875#issuecomment-976449863 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 0s | Docker mode activated. | | -1 :x: | docker | 10m 20s | Docker failed to build yetus/hbase:57dbad11da. | | Subsystem | Report/Notes | |--:|:-| | GITHUB PR | https://github.com/apache/hbase/pull/3875 | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3875/1/console | | versions | git=2.17.1 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] YutSean opened a new pull request #3875: HBASE-26459 HMaster should move non-meta region only if meta is ONLINE
YutSean opened a new pull request #3875: URL: https://github.com/apache/hbase/pull/3875 https://issues.apache.org/jira/browse/HBASE-26459 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3803: HBASE-26304: Reflect out of band locality improvements in metrics and balancer
Apache-HBase commented on pull request #3803: URL: https://github.com/apache/hbase/pull/3803#issuecomment-976438625 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 28s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 31s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 20s | master passed | | +1 :green_heart: | compile | 5m 3s | master passed | | +1 :green_heart: | checkstyle | 1m 58s | master passed | | +0 :ok: | refguide | 5m 1s | branch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect. | | +1 :green_heart: | spotbugs | 4m 44s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 17s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 38s | the patch passed | | +1 :green_heart: | compile | 5m 29s | the patch passed | | +1 :green_heart: | javac | 5m 29s | the patch passed | | -0 :warning: | checkstyle | 1m 13s | hbase-server: The patch generated 3 new + 72 unchanged - 0 fixed = 75 total (was 72) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. | | +0 :ok: | refguide | 4m 51s | patch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect. | | +1 :green_heart: | hadoopcheck | 23m 47s | Patch does not cause any errors with Hadoop 3.1.2 3.2.2 3.3.1. | | +1 :green_heart: | spotbugs | 4m 46s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 39s | The patch does not generate ASF License warnings. | | | | 78m 41s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3803/9/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3803 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile refguide xml | | uname | Linux fe11a3c5a015 4.15.0-156-generic #163-Ubuntu SMP Thu Aug 19 23:31:58 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 1c48248ef8 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | refguide | https://nightlies.apache.org/hbase/HBase/HBase-PreCommit-GitHub-PR/PR-3803/9/yetus-general-check/output/branch-site/book.html | | checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3803/9/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | refguide | https://nightlies.apache.org/hbase/HBase/HBase-PreCommit-GitHub-PR/PR-3803/9/yetus-general-check/output/patch-site/book.html | | Max. process+thread count | 96 (vs. ulimit of 3) | | modules | C: hbase-common hbase-balancer hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3803/9/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3870: HBASE-26479: Print too slow/big scan's operation_id in region server log
Apache-HBase commented on pull request #3870: URL: https://github.com/apache/hbase/pull/3870#issuecomment-976410852 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 26s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 12s | master passed | | +1 :green_heart: | compile | 3m 11s | master passed | | +1 :green_heart: | checkstyle | 1m 5s | master passed | | +1 :green_heart: | spotbugs | 2m 6s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 49s | the patch passed | | +1 :green_heart: | compile | 3m 11s | the patch passed | | +1 :green_heart: | javac | 3m 11s | the patch passed | | +1 :green_heart: | checkstyle | 1m 3s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 19m 27s | Patch does not cause any errors with Hadoop 3.1.2 3.2.2 3.3.1. | | +1 :green_heart: | spotbugs | 2m 17s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 16s | The patch does not generate ASF License warnings. | | | | 49m 8s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3870/3/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3870 | | JIRA Issue | HBASE-26479 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux 7d3e7d1ed60d 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / b870b6d9ac | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Max. process+thread count | 95 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3870/3/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (HBASE-26475) The flush and compact methods in HTU should skip processing secondary replicas
[ https://issues.apache.org/jira/browse/HBASE-26475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang resolved HBASE-26475. --- Fix Version/s: 2.5.0 3.0.0-alpha-2 2.4.9 Hadoop Flags: Reviewed Resolution: Fixed Pushed to branch-2.4+. Thanks [~Xiaolin Ha] for reviewing. > The flush and compact methods in HTU should skip processing secondary replicas > -- > > Key: HBASE-26475 > URL: https://issues.apache.org/jira/browse/HBASE-26475 > Project: HBase > Issue Type: Improvement > Components: test >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 2.5.0, 3.0.0-alpha-2, 2.4.9 > > > To avoid generating bunch of "NOT flushing because not writable" warning logs. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (HBASE-26477) Fix MiniMapReduceCluster failure under Java17
[
https://issues.apache.org/jira/browse/HBASE-26477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17447923#comment-17447923
]
Duo Zhang commented on HBASE-26477:
---
Mind opening a PR for branch-2? The patch for master can not apply to branch-2
cleanly...
Thanks.
> Fix MiniMapReduceCluster failure under Java17
> -
>
> Key: HBASE-26477
> URL: https://issues.apache.org/jira/browse/HBASE-26477
> Project: HBase
> Issue Type: Task
>Reporter: Yutong Xiao
>Assignee: Yutong Xiao
>Priority: Major
>
> MiniMapReduceCluster in HBaseTestingUtil.java will not work. Find the error
> message in the container log:
> {code:java}
> Caused by: java.lang.reflect.InaccessibleObjectException: Unable to make
> protected final java.lang.Class
> java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain)
> throws java.lang.ClassFormatError accessible: module java.base does not
> "opens java.lang" to unnamed module @755c9148
> at
> java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:354)
> ~[?:?]
> at
> java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:297)
> ~[?:?]
> at java.lang.reflect.Method.checkCanSetAccessible(Method.java:199)
> ~[?:?]
> at java.lang.reflect.Method.setAccessible(Method.java:193) ~[?:?]
> at
> com.google.inject.internal.cglib.core.$ReflectUtils$2.run(ReflectUtils.java:56)
> ~[guice-4.0.jar:?]
> at
> java.security.AccessController.doPrivileged(AccessController.java:318) ~[?:?]
> at
> com.google.inject.internal.cglib.core.$ReflectUtils.(ReflectUtils.java:46)
> ~[guice-4.0.jar:?]
> {code}
> The container of MiniMapReduceCluster is running in an independent JVM. So
> that we need to pass the JVM opts to the container JVM.
> Impacted UTs:
> TestSecureExportSnapshot
> TestMobSecureExportSnapshot
> TestVerifyReplicationCrossDiffHdfs
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [hbase] virajjasani commented on pull request #3852: HBASE-26458 Update Snapshot TTL doc
virajjasani commented on pull request #3852: URL: https://github.com/apache/hbase/pull/3852#issuecomment-976398282 Thanks @joswiatek. Left one comment on #3857 for branch-1 backport. Once done, will merge both PRs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] virajjasani commented on a change in pull request #3857: HBASE-26458 Add UNSET_SNAPSHOT_PROP and fix TTL defaulting
virajjasani commented on a change in pull request #3857:
URL: https://github.com/apache/hbase/pull/3857#discussion_r755006739
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
##
@@ -3712,7 +3712,10 @@ public void snapshot(final String snapshotName, final
TableName tableName,
builder.setTable(tableName.getNameAsString());
builder.setName(snapshotName);
builder.setType(type);
-builder.setTtl(getTtlFromSnapshotProps(snapshotProps));
+long ttl = getTtlFromSnapshotProps(snapshotProps);
+if (ttl != 1L && ttl < TimeUnit.MILLISECONDS.toSeconds(Long.MAX_VALUE)) {
Review comment:
This would be `ttl != -1`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] Apache9 merged pull request #3873: HBASE-26456 Limit the size for one replicating
Apache9 merged pull request #3873: URL: https://github.com/apache/hbase/pull/3873 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache9 merged pull request #3867: HBASE-26477 Fix MiniMapReduceCluster failure under Java17
Apache9 merged pull request #3867: URL: https://github.com/apache/hbase/pull/3867 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache9 merged pull request #3868: HBASE-26475 The flush and compact methods in HTU should skip processi…
Apache9 merged pull request #3868: URL: https://github.com/apache/hbase/pull/3868 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (HBASE-26414) Tracing INSTRUMENTATION_NAME is incorrect
[
https://issues.apache.org/jira/browse/HBASE-26414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhuobin zheng reassigned HBASE-26414:
-
Assignee: Nick Dimiduk (was: zhuobin zheng)
> Tracing INSTRUMENTATION_NAME is incorrect
> -
>
> Key: HBASE-26414
> URL: https://issues.apache.org/jira/browse/HBASE-26414
> Project: HBase
> Issue Type: Bug
> Components: tracing
>Affects Versions: 2.5.0, 3.0.0-alpha-2
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
>Priority: Blocker
> Fix For: 2.5.0, 3.0.0-alpha-2
>
>
> I believe the value we use for {{TraceUtil#INSTRUMENTATION_NAME}},
> {{"io.opentelemetry.contrib.hbase"}}, is incorrect. According to the java
> docs,
> {noformat}
>* @param instrumentationName The name of the instrumentation library, not
> the name of the
>* instrument*ed* library (e.g., "io.opentelemetry.contrib.mongodb").
> Must not be null.
> {noformat}
> This namespace appears to be reserved for implementations shipped by the otel
> project, found under
> https://github.com/open-telemetry/opentelemetry-java-instrumentation/tree/main/instrumentation
> I don't have a suggestion for a suitable name at this time. Will report back.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [hbase] jojochuang commented on pull request #3867: HBASE-26477 Fix MiniMapReduceCluster failure under Java17
jojochuang commented on pull request #3867: URL: https://github.com/apache/hbase/pull/3867#issuecomment-976351859 Ok. I guess it would be a good idea to update the doc. For example, in the Java version here: https://hbase.apache.org/book.html#basic.prerequisites where we can add the required java options for a production MR cluster. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] YutSean edited a comment on pull request #3867: HBASE-26477 Fix MiniMapReduceCluster failure under Java17
YutSean edited a comment on pull request #3867: URL: https://github.com/apache/hbase/pull/3867#issuecomment-976302375 Adding this option will have problem under Java version 8 (not sure if works for java 11) and will crash the JVM. Maybe we should not set it as default in hadoop MR if we still support java8. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] YutSean commented on pull request #3867: HBASE-26477 Fix MiniMapReduceCluster failure under Java17
YutSean commented on pull request #3867: URL: https://github.com/apache/hbase/pull/3867#issuecomment-976302375 Adding this option will have problem under Java version 8 (not sure if is works for java 11) and will crash the JVM. Maybe we should not set it as default in hadoop MR if we still support java8. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3869: HBASE-26476 Make DefaultMemStore extensible for HStore.memstore
Apache-HBase commented on pull request #3869: URL: https://github.com/apache/hbase/pull/3869#issuecomment-976302156 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 46s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 5m 39s | master passed | | +1 :green_heart: | compile | 1m 45s | master passed | | +1 :green_heart: | shadedjars | 10m 53s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 52s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 6m 25s | the patch passed | | +1 :green_heart: | compile | 1m 39s | the patch passed | | +1 :green_heart: | javac | 1m 39s | the patch passed | | +1 :green_heart: | shadedjars | 10m 16s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 49s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 229m 54s | hbase-server in the patch passed. | | | | 271m 51s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3869/3/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3869 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 16169946fecb 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18 19:21:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / b870b6d9ac | | Default Java | AdoptOpenJDK-11.0.10+9 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3869/3/testReport/ | | Max. process+thread count | 3093 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3869/3/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] YutSean commented on pull request #3867: HBASE-26477 Fix MiniMapReduceCluster failure under Java17
YutSean commented on pull request #3867: URL: https://github.com/apache/hbase/pull/3867#issuecomment-976299437 Here is the only JVM options for this case. After adding this option, the UT passed locally. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (HBASE-26249) Ameliorate compaction made by bulk-loading files
[ https://issues.apache.org/jira/browse/HBASE-26249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaolin Ha resolved HBASE-26249. Fix Version/s: 2.5.0 3.0.0-alpha-2 Resolution: Fixed Merged to branch-2 and master, thanks [~zhangduo] for reviewing. > Ameliorate compaction made by bulk-loading files > > > Key: HBASE-26249 > URL: https://issues.apache.org/jira/browse/HBASE-26249 > Project: HBase > Issue Type: Improvement >Reporter: Xiaolin Ha >Assignee: Xiaolin Ha >Priority: Major > Fix For: 2.5.0, 3.0.0-alpha-2 > > > HBASE-25213 introduced to trigger compact after bulkload, but if a store > needs compaction, each file loaded to this store will add a compaction runner > of this store to the compaction queue. Then repeatedly compute for > needCompaction and selectFiles for this store, and the compaction of the > store may be continuous. > We can mark the queued compaction runners for the stores, and avoid > repeatedly adding same store compactions to the queue after bulk load files. > This method can reduce the pressure during the period just after bulk loading > and spread the pressure to other times by the periodic CompactionChecker. > For the need of splitting mentioned before, after the completion of > compaction, it will trigger split of the region if it needs. So we do not > need to trigger split after bulk loading. > -- This message was sent by Atlassian Jira (v8.20.1#820001)
