[GitHub] [hbase] Apache-HBase commented on pull request #3496: HBASE-24734 RegionInfo#containsRange should support check meta table
Apache-HBase commented on pull request #3496: URL: https://github.com/apache/hbase/pull/3496#issuecomment-884673086 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 26s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 56s | master passed | | +1 :green_heart: | compile | 4m 53s | master passed | | +1 :green_heart: | checkstyle | 1m 53s | master passed | | +1 :green_heart: | spotbugs | 3m 42s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 43s | the patch passed | | +1 :green_heart: | compile | 4m 55s | the patch passed | | +1 :green_heart: | javac | 4m 55s | the patch passed | | +1 :green_heart: | checkstyle | 1m 56s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 18m 15s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | spotbugs | 4m 19s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 40s | The patch does not generate ASF License warnings. | | | | 57m 37s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3496/3/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3496 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux fc9523a19839 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 21c4578f22 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Max. process+thread count | 96 (vs. ulimit of 3) | | modules | C: hbase-common hbase-client hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3496/3/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (HBASE-26062) SIGSEGV in AsyncFSWAL consume
[
https://issues.apache.org/jira/browse/HBASE-26062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Stack resolved HBASE-26062.
---
Resolution: Duplicate
Thanks [~anoop.hbase] There is ASYNC_WAL on this cluster afterall (when I
wrote the above, thought there was none). Resolving as duplicate of what we see
over on HBASE-24984
> SIGSEGV in AsyncFSWAL consume
> -
>
> Key: HBASE-26062
> URL: https://issues.apache.org/jira/browse/HBASE-26062
> Project: HBase
> Issue Type: Bug
>Reporter: Michael Stack
>Priority: Major
>
> Seems related to the parent issue. Its happened a few times on one of our
> clusters here. Below are two examples. Need more detail but perhaps the call
> has timed out, the buffer has thus been freed, but the late consume on the
> other side of the ringbuffer doesn't know that and goes ahead (Just
> speculation).
>
> {code:java}
> # SIGSEGV (0xb) at pc=0x7f8b3ef5b77c, pid=37631, tid=0x7f61560ed700
> RAX=0xdf6e is an unknown valueRBX=0x7f8a38d7b6f8 is an
> oopjava.nio.DirectByteBuffer - klass:
> 'java/nio/DirectByteBuffer'RCX=0x7f60e2767898 is pointing into
> metadataRDX=0x0de7 is an unknown valueRSP=0x7f61560ec6f0 is
> pointing into the stack for thread: 0x7f8b3017b800RBP=[error occurred
> during error reporting (printing register info), id 0xb]
> Stack: [0x7f6155fed000,0x7f61560ee000], sp=0x7f61560ec6f0, free
> space=1021kNative frames: (J=compiled Java code, j=interpreted, Vv=VM code,
> C=native code)J 23901 C2
> java.util.stream.MatchOps$1MatchSink.accept(Ljava/lang/Object;)V (44 bytes) @
> 0x7f8b3ef5b77c [0x7f8b3ef5b640+0x13c]J 16165 C2
> java.util.ArrayList$ArrayListSpliterator.tryAdvance(Ljava/util/function/Consumer;)Z
> (79 bytes) @ 0x7f8b3d67b344 [0x7f8b3d67b2c0+0x84]J 16160 C2
> java.util.stream.MatchOps$MatchOp.evaluateSequential(Ljava/util/stream/PipelineHelper;Ljava/util/Spliterator;)Ljava/lang/Object;
> (7 bytes) @ 0x7f8b3d67bc9c [0x7f8b3d67b900+0x39c]J 17729 C2
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALActionListener.visitLogEntryBeforeWrite(Lorg/apache/hadoop/hbase/wal/WALKey;Lorg/apache/hadoop/hbase/wal/WALEdit;)V
> (10 bytes) @ 0x7f8b3fc39010 [0x7f8b3fc388a0+0x770]J 29991 C2
> org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.appendAndSync()V (261
> bytes) @ 0x7f8b3fd03d90 [0x7f8b3fd039e0+0x3b0]J 20773 C2
> org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.consume()V (474 bytes) @
> 0x7f8b40283728 [0x7f8b40283480+0x2a8]J 15191 C2
> org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL$$Lambda$76.run()V (8
> bytes) @ 0x7f8b3ed69ecc [0x7f8b3ed69ea0+0x2c]J 17383% C2
> java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V
> (225 bytes) @ 0x7f8b3d9423f8 [0x7f8b3d942260+0x198]j
> java.util.concurrent.ThreadPoolExecutor$Worker.run()V+5j
> java.lang.Thread.run()V+11v ~StubRoutines::call_stubV [libjvm.so+0x66b9ba]
> JavaCalls::call_helper(JavaValue*, methodHandle*, JavaCallArguments*,
> Thread*)+0xe1aV [libjvm.so+0x669073] JavaCalls::call_virtual(JavaValue*,
> KlassHandle, Symbol*, Symbol*, JavaCallArguments*, Thread*)+0x263V
> [libjvm.so+0x669647] JavaCalls::call_virtual(JavaValue*, Handle,
> KlassHandle, Symbol*, Symbol*, Thread*)+0x57V [libjvm.so+0x6aaa4c]
> thread_entry(JavaThread*, Thread*)+0x6cV [libjvm.so+0xa224cb]
> JavaThread::thread_main_inner()+0xdbV [libjvm.so+0xa22816]
> JavaThread::run()+0x316V [libjvm.so+0x8c4202] java_start(Thread*)+0x102C
> [libpthread.so.0+0x76ba] start_thread+0xca {code}
>
> This one is from a month previous and has a deeper stack... we're trying to
> read a Cell...
>
> {code:java}
> Stack: [0x7fa1d5fb8000,0x7fa1d60b9000], sp=0x7fa1d60b7660, free
> space=1021kNative frames: (J=compiled Java code, j=interpreted, Vv=VM code,
> C=native code)J 30665 C2
> org.apache.hadoop.hbase.PrivateCellUtil.matchingFamily(Lorg/apache/hadoop/hbase/Cell;[BII)Z
> (59 bytes) @ 0x7fcc2d29eeb2 [0x7fcc2d29e7c0+0x6f2]J 25816 C2
> org.apache.hadoop.hbase.CellUtil.matchingFamily(Lorg/apache/hadoop/hbase/Cell;[B)Z
> (28 bytes) @ 0x7fcc2a0430f8 [0x7fcc2a0430e0+0x18]J 17236 C2
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALActionListener$$Lambda$254.test(Ljava/lang/Object;)Z
> (8 bytes) @ 0x7fcc2b40bc68 [0x7fcc2b40bc20+0x48]J 13735 C2
> java.util.ArrayList$ArrayListSpliterator.tryAdvance(Ljava/util/function/Consumer;)Z
> (79 bytes) @ 0x7fcc2b7d936c [0x7fcc2b7d92c0+0xac]J 17162 C2
> java.util.stream.MatchOps$MatchOp.evaluateSequential(Ljava/util/stream/PipelineHelper;Ljava/util/Spliterator;)Ljava/lang/Object;
> (7 bytes) @ 0x7fcc29bc05e8
[jira] [Updated] (HBASE-24734) RegionInfo#containsRange should support check meta table
[
https://issues.apache.org/jira/browse/HBASE-24734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yi Mei updated HBASE-24734:
---
Summary: RegionInfo#containsRange should support check meta table (was:
Wrong comparator opening Region when 'split-to-WAL' enabled.)
> RegionInfo#containsRange should support check meta table
>
>
> Key: HBASE-24734
> URL: https://issues.apache.org/jira/browse/HBASE-24734
> Project: HBase
> Issue Type: Sub-task
> Components: HFile, MTTR
>Reporter: Michael Stack
>Priority: Major
>
> Came across this when we were testing the 'split-to-hfile' feature running
> ITBLL:
>
> {code:java}
> 2020-07-10 10:16:49,983 INFO org.apache.hadoop.hbase.regionserver.HRegion:
> Closing region hbase:meta,,1.15882307402020-07-10 10:16:49,997 INFO
> org.apache.hadoop.hbase.regionserver.HRegion: Closed
> hbase:meta,,1.15882307402020-07-10 10:16:49,998 WARN
> org.apache.hadoop.hbase.regionserver.handler.AssignRegionHandler: Fatal error
> occurred while opening region hbase:meta,,1.1588230740,
> aborting...java.lang.IllegalArgumentException: Invalid range:
> IntegrationTestBigLinkedList,,1594350463222.8f89e01a5245e79946e22d8a8ab4698b.
> >
> IntegrationTestBigLinkedList,\x10\x02J\xA1,1594349535271.be24dc276f686e6dcc7fb9d3f91c8387.
> at
> org.apache.hadoop.hbase.client.RegionInfoBuilder$MutableRegionInfo.containsRange(RegionInfoBuilder.java:300)
> at
> org.apache.hadoop.hbase.regionserver.HStore.tryCommitRecoveredHFile(HStore.java:)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.loadRecoveredHFilesIfAny(HRegion.java:5442)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals(HRegion.java:1010)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:950)
>at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7490)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegionFromTableDir(HRegion.java:7448)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7424)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7382)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7333)
> at
> org.apache.hadoop.hbase.regionserver.handler.AssignRegionHandler.process(AssignRegionHandler.java:135)
> at
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)2020-07-10
> 10:16:50,005 ERROR org.apache.hadoop.hbase.regionserver.HRegionServer: *
> ABORTING region server hbasedn149.example.org,16020,1594375563853: Failed to
> open region hbase:meta,,1.1588230740 and can not recover
> *java.lang.IllegalArgumentException: Invalid range:
> IntegrationTestBigLinkedList,,1594350463222.8f89e01a5245e79946e22d8a8ab4698b.
> >
> IntegrationTestBigLinkedList,\x10\x02J\xA1,1594349535271.be24dc276f686e6dcc7fb9d3f91c8387.
> {code}
> Seems basic case of wrong comparator. Below passes if I use the meta
> comparator
> {code:java}
> @Test
> public void testBinaryKeys() throws Exception {
> Set set = new TreeSet<>(CellComparatorImpl.COMPARATOR);
> final byte [] fam = Bytes.toBytes("col");
> final byte [] qf = Bytes.toBytes("umn");
> final byte [] nb = new byte[0];
> Cell [] keys = {
> createByteBufferKeyValueFromKeyValue(
> new KeyValue(Bytes.toBytes("a,\u\u,2"), fam, qf, 2,
> nb)),
> createByteBufferKeyValueFromKeyValue(
> new KeyValue(Bytes.toBytes("a,\u0001,3"), fam, qf, 3, nb)),
> createByteBufferKeyValueFromKeyValue(
> new KeyValue(Bytes.toBytes("a,,1"), fam, qf, 1, nb)),
> createByteBufferKeyValueFromKeyValue(
> new KeyValue(Bytes.toBytes("a,\u1000,5"), fam, qf, 5, nb)),
> createByteBufferKeyValueFromKeyValue(
> new KeyValue(Bytes.toBytes("a,a,4"), fam, qf, 4, nb)),
> createByteBufferKeyValueFromKeyValue(
> new KeyValue(Bytes.toBytes("a,a,0"), fam, qf, 0, nb)),
> };
> // Add to set with bad comparator
> Collections.addAll(set, keys);
> // This will output the keys incorrectly.
> boolean assertion = false;
> int count = 0;
> try {
> for (Cell k: set) {
> assertTrue("count=" + count + ", " + k.toString(), count++ ==
> k.getTimestamp());
> }
> } catch (AssertionError e) {
> // Expected
> assertion = true;
> }
> assertTrue(assertion);
> // Make set with good comparator
> set = new
[GitHub] [hbase] mymeiyi commented on pull request #3496: HBASE-24734 Wrong comparator opening Region when 'split-to-WAL' enabled.
mymeiyi commented on pull request #3496: URL: https://github.com/apache/hbase/pull/3496#issuecomment-884654516 > Better change the title of the issue? Ok, let me do it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3515: Hbase 26097
Apache-HBase commented on pull request #3515: URL: https://github.com/apache/hbase/pull/3515#issuecomment-884648151 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 6m 3s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ branch-2 Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 4s | branch-2 passed | | +1 :green_heart: | compile | 0m 31s | branch-2 passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 18s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 43s | the patch passed | | +1 :green_heart: | compile | 0m 44s | the patch passed | | +1 :green_heart: | javac | 0m 44s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | xml | 0m 2s | The patch has no ill-formed XML file. | | -1 :x: | hadoopcheck | 11m 40s | The patch causes 10 errors with Hadoop v3.2.1. | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 23s | The patch does not generate ASF License warnings. | | | | 29m 12s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3515/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3515 | | Optional Tests | dupname asflicense javac hadoopcheck xml compile | | uname | Linux 9030680aff75 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18 19:21:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / 8bc180c773 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | hadoopcheck | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3515/1/artifact/yetus-general-check/output/patch-javac-3.2.1.txt | | Max. process+thread count | 87 (vs. ulimit of 12500) | | modules | C: hbase-shaded hbase-shaded/hbase-shaded-endpoint U: hbase-shaded | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3515/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3515: Hbase 26097
Apache-HBase commented on pull request #3515: URL: https://github.com/apache/hbase/pull/3515#issuecomment-884647665 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 34s | Docker mode activated. | | -0 :warning: | yetus | 0m 5s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2 Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 22s | branch-2 passed | | +1 :green_heart: | compile | 0m 30s | branch-2 passed | | +1 :green_heart: | shadedjars | 6m 50s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 19s | branch-2 passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 26s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 4s | the patch passed | | +1 :green_heart: | compile | 0m 49s | the patch passed | | +1 :green_heart: | javac | 0m 49s | the patch passed | | +1 :green_heart: | shadedjars | 6m 44s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 31s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 58s | hbase-shaded in the patch passed. | | +1 :green_heart: | unit | 0m 17s | hbase-shaded-endpoint in the patch passed. | | | | 27m 43s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3515/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3515 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 2bdc40313239 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / 8bc180c773 | | Default Java | AdoptOpenJDK-11.0.10+9 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3515/1/testReport/ | | Max. process+thread count | 498 (vs. ulimit of 12500) | | modules | C: hbase-shaded hbase-shaded/hbase-shaded-endpoint U: hbase-shaded | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3515/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3515: Hbase 26097
Apache-HBase commented on pull request #3515: URL: https://github.com/apache/hbase/pull/3515#issuecomment-884647260 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 34s | Docker mode activated. | | -0 :warning: | yetus | 0m 8s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2 Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 59s | branch-2 passed | | +1 :green_heart: | compile | 0m 30s | branch-2 passed | | +1 :green_heart: | shadedjars | 6m 19s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 19s | branch-2 passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 24s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 39s | the patch passed | | +1 :green_heart: | compile | 0m 46s | the patch passed | | +1 :green_heart: | javac | 0m 46s | the patch passed | | +1 :green_heart: | shadedjars | 6m 23s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 32s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 0m 49s | hbase-shaded in the patch failed. | | +1 :green_heart: | unit | 0m 15s | hbase-shaded-endpoint in the patch passed. | | | | 25m 53s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3515/1/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3515 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 826be5e0bdba 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / 8bc180c773 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3515/1/artifact/yetus-jdk8-hadoop2-check/output/patch-unit-hbase-shaded.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3515/1/testReport/ | | Max. process+thread count | 195 (vs. ulimit of 12500) | | modules | C: hbase-shaded hbase-shaded/hbase-shaded-endpoint U: hbase-shaded | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3515/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] zyxxoo commented on pull request #3502: HBASE-26097: to resolve hbase version 2.x protobuf version conflict
zyxxoo commented on pull request #3502: URL: https://github.com/apache/hbase/pull/3502#issuecomment-884641747 > HBase 2.0.x is alread EOL. If this is for 2.x only, please open a PR against branch-2? > > Thanks. Hi, i have opened a new PR https://github.com/apache/hbase/pull/3515, please review again -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] zyxxoo opened a new pull request #3515: Hbase 26097
zyxxoo opened a new pull request #3515: URL: https://github.com/apache/hbase/pull/3515 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-26062) SIGSEGV in AsyncFSWAL consume
[
https://issues.apache.org/jira/browse/HBASE-26062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17385232#comment-17385232
]
Anoop Sam John commented on HBASE-26062:
[~stack] no ASYNC_WAL durability? If so we have other issues !!!
> SIGSEGV in AsyncFSWAL consume
> -
>
> Key: HBASE-26062
> URL: https://issues.apache.org/jira/browse/HBASE-26062
> Project: HBase
> Issue Type: Bug
>Reporter: Michael Stack
>Priority: Major
>
> Seems related to the parent issue. Its happened a few times on one of our
> clusters here. Below are two examples. Need more detail but perhaps the call
> has timed out, the buffer has thus been freed, but the late consume on the
> other side of the ringbuffer doesn't know that and goes ahead (Just
> speculation).
>
> {code:java}
> # SIGSEGV (0xb) at pc=0x7f8b3ef5b77c, pid=37631, tid=0x7f61560ed700
> RAX=0xdf6e is an unknown valueRBX=0x7f8a38d7b6f8 is an
> oopjava.nio.DirectByteBuffer - klass:
> 'java/nio/DirectByteBuffer'RCX=0x7f60e2767898 is pointing into
> metadataRDX=0x0de7 is an unknown valueRSP=0x7f61560ec6f0 is
> pointing into the stack for thread: 0x7f8b3017b800RBP=[error occurred
> during error reporting (printing register info), id 0xb]
> Stack: [0x7f6155fed000,0x7f61560ee000], sp=0x7f61560ec6f0, free
> space=1021kNative frames: (J=compiled Java code, j=interpreted, Vv=VM code,
> C=native code)J 23901 C2
> java.util.stream.MatchOps$1MatchSink.accept(Ljava/lang/Object;)V (44 bytes) @
> 0x7f8b3ef5b77c [0x7f8b3ef5b640+0x13c]J 16165 C2
> java.util.ArrayList$ArrayListSpliterator.tryAdvance(Ljava/util/function/Consumer;)Z
> (79 bytes) @ 0x7f8b3d67b344 [0x7f8b3d67b2c0+0x84]J 16160 C2
> java.util.stream.MatchOps$MatchOp.evaluateSequential(Ljava/util/stream/PipelineHelper;Ljava/util/Spliterator;)Ljava/lang/Object;
> (7 bytes) @ 0x7f8b3d67bc9c [0x7f8b3d67b900+0x39c]J 17729 C2
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALActionListener.visitLogEntryBeforeWrite(Lorg/apache/hadoop/hbase/wal/WALKey;Lorg/apache/hadoop/hbase/wal/WALEdit;)V
> (10 bytes) @ 0x7f8b3fc39010 [0x7f8b3fc388a0+0x770]J 29991 C2
> org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.appendAndSync()V (261
> bytes) @ 0x7f8b3fd03d90 [0x7f8b3fd039e0+0x3b0]J 20773 C2
> org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.consume()V (474 bytes) @
> 0x7f8b40283728 [0x7f8b40283480+0x2a8]J 15191 C2
> org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL$$Lambda$76.run()V (8
> bytes) @ 0x7f8b3ed69ecc [0x7f8b3ed69ea0+0x2c]J 17383% C2
> java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V
> (225 bytes) @ 0x7f8b3d9423f8 [0x7f8b3d942260+0x198]j
> java.util.concurrent.ThreadPoolExecutor$Worker.run()V+5j
> java.lang.Thread.run()V+11v ~StubRoutines::call_stubV [libjvm.so+0x66b9ba]
> JavaCalls::call_helper(JavaValue*, methodHandle*, JavaCallArguments*,
> Thread*)+0xe1aV [libjvm.so+0x669073] JavaCalls::call_virtual(JavaValue*,
> KlassHandle, Symbol*, Symbol*, JavaCallArguments*, Thread*)+0x263V
> [libjvm.so+0x669647] JavaCalls::call_virtual(JavaValue*, Handle,
> KlassHandle, Symbol*, Symbol*, Thread*)+0x57V [libjvm.so+0x6aaa4c]
> thread_entry(JavaThread*, Thread*)+0x6cV [libjvm.so+0xa224cb]
> JavaThread::thread_main_inner()+0xdbV [libjvm.so+0xa22816]
> JavaThread::run()+0x316V [libjvm.so+0x8c4202] java_start(Thread*)+0x102C
> [libpthread.so.0+0x76ba] start_thread+0xca {code}
>
> This one is from a month previous and has a deeper stack... we're trying to
> read a Cell...
>
> {code:java}
> Stack: [0x7fa1d5fb8000,0x7fa1d60b9000], sp=0x7fa1d60b7660, free
> space=1021kNative frames: (J=compiled Java code, j=interpreted, Vv=VM code,
> C=native code)J 30665 C2
> org.apache.hadoop.hbase.PrivateCellUtil.matchingFamily(Lorg/apache/hadoop/hbase/Cell;[BII)Z
> (59 bytes) @ 0x7fcc2d29eeb2 [0x7fcc2d29e7c0+0x6f2]J 25816 C2
> org.apache.hadoop.hbase.CellUtil.matchingFamily(Lorg/apache/hadoop/hbase/Cell;[B)Z
> (28 bytes) @ 0x7fcc2a0430f8 [0x7fcc2a0430e0+0x18]J 17236 C2
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALActionListener$$Lambda$254.test(Ljava/lang/Object;)Z
> (8 bytes) @ 0x7fcc2b40bc68 [0x7fcc2b40bc20+0x48]J 13735 C2
> java.util.ArrayList$ArrayListSpliterator.tryAdvance(Ljava/util/function/Consumer;)Z
> (79 bytes) @ 0x7fcc2b7d936c [0x7fcc2b7d92c0+0xac]J 17162 C2
> java.util.stream.MatchOps$MatchOp.evaluateSequential(Ljava/util/stream/PipelineHelper;Ljava/util/Spliterator;)Ljava/lang/Object;
> (7 bytes) @ 0x7fcc29bc05e8 [0x7fcc29bbfe80+0x768]J 16934 C2
>
[jira] [Commented] (HBASE-25701) RegionServer JVM crash when append wal entry
[ https://issues.apache.org/jira/browse/HBASE-25701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17385229#comment-17385229 ] Anoop Sam John commented on HBASE-25701: [~jutia] Looks like HBASE-24984 fix will fix ur case. Can you pls take a look. We can close this issue also. > RegionServer JVM crash when append wal entry > > > Key: HBASE-25701 > URL: https://issues.apache.org/jira/browse/HBASE-25701 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.2.6 >Reporter: Juanjuan Tian >Priority: Major > > Region Server JVM crash when append wal entry, JVM crash log: > # > # A fatal error has been detected by the Java Runtime Environment: > # > # EXCEPTION_ACCESS_VIOLATION (0xc005) at pc=0x027af93f, > pid=17992, tid=0x2d54 > # > # JRE version: Java(TM) SE Runtime Environment (8.0_202-b08) (build > 1.8.0_202-b08) > # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.202-b08 mixed mode > windows-amd64 compressed oops) > # Problematic frame: > # J 10214 C2 org.apache.hadoop.hbase.ByteBufferKeyValue.getFamilyLength()B (9 > bytes) @ 0x027af93f [0x027af860+0xdf] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-24984) WAL corruption due to early DBBs re-use when Durability.ASYNC_WAL is used with multi operation
[ https://issues.apache.org/jira/browse/HBASE-24984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-24984: --- Hadoop Flags: Reviewed Resolution: Fixed Status: Resolved (was: Patch Available) Pushed to branch-2.3+ branches. Thanks for the fix [~gouravk]. Thanks all for the reviews. > WAL corruption due to early DBBs re-use when Durability.ASYNC_WAL is used > with multi operation > -- > > Key: HBASE-24984 > URL: https://issues.apache.org/jira/browse/HBASE-24984 > Project: HBase > Issue Type: Bug > Components: rpc, wal >Affects Versions: 2.1.6 >Reporter: Liu Junhong >Assignee: Gaurav Kanade >Priority: Critical > Fix For: 2.5.0, 2.3.6, 3.0.0-alpha-2, 2.4.5 > > Attachments: > 0001-HBASE-24984-WAL-corruption-due-to-early-DBBs-re-use-.patch > > > After bugfix HBASE-22539, When client use BufferedMutator or multiple > mutation , there will be one RpcCall and mutliple FSWALEntry . At the time > RpcCall finish and one FSWALEntry call release() , the remain FSWALEntries > may trigger RuntimeException or segmentation fault . > We should use RefCnt instead of AtomicInteger for > org.apache.hadoop.hbase.ipc.ServerCall.reference? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-24984) WAL corruption due to early DBBs re-use when Durability.ASYNC_WAL is used with multi operation
[ https://issues.apache.org/jira/browse/HBASE-24984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-24984: --- Affects Version/s: (was: 2.1.6) 2.0.0 > WAL corruption due to early DBBs re-use when Durability.ASYNC_WAL is used > with multi operation > -- > > Key: HBASE-24984 > URL: https://issues.apache.org/jira/browse/HBASE-24984 > Project: HBase > Issue Type: Bug > Components: rpc, wal >Affects Versions: 2.0.0 >Reporter: Liu Junhong >Assignee: Gaurav Kanade >Priority: Critical > Fix For: 2.5.0, 2.3.6, 3.0.0-alpha-2, 2.4.5 > > Attachments: > 0001-HBASE-24984-WAL-corruption-due-to-early-DBBs-re-use-.patch > > > After bugfix HBASE-22539, When client use BufferedMutator or multiple > mutation , there will be one RpcCall and mutliple FSWALEntry . At the time > RpcCall finish and one FSWALEntry call release() , the remain FSWALEntries > may trigger RuntimeException or segmentation fault . > We should use RefCnt instead of AtomicInteger for > org.apache.hadoop.hbase.ipc.ServerCall.reference? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache9 commented on pull request #3502: HBASE-26097: to resolve hbase version 2.x protobuf version conflict
Apache9 commented on pull request #3502: URL: https://github.com/apache/hbase/pull/3502#issuecomment-884626473 HBase 2.0.x is alread EOL. If this is for 2.x only, please open a PR against branch-2? Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] anoopsjohn merged pull request #3512: HBASE-24984 WAL corruption due to early DBBs re-use when Durability.ASYNC_WAL is used with multi operation
anoopsjohn merged pull request #3512: URL: https://github.com/apache/hbase/pull/3512 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-26088) conn.getBufferedMutator(tableName) leaks thread executors and other problems
[
https://issues.apache.org/jira/browse/HBASE-26088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17385208#comment-17385208
]
Hudson commented on HBASE-26088:
Results for branch branch-2.4
[build #165 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/165/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/165/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/165/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/165/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/165/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> conn.getBufferedMutator(tableName) leaks thread executors and other problems
>
>
> Key: HBASE-26088
> URL: https://issues.apache.org/jira/browse/HBASE-26088
> Project: HBase
> Issue Type: Bug
> Components: Client
>Affects Versions: 2.0.0
>Reporter: Whitney Jackson
>Assignee: Rushabh Shah
>Priority: Critical
> Fix For: 2.5.0, 2.3.6, 2.4.5
>
>
> TL;DR: {{conn.getBufferedMutator(tableName)}} is dangerous in hbase client
> 2.4.4 and doesn't match documented behavior in 1.4.13.
> To work around the problems until fixed do this:
> {code:java}
> var mySingletonPool = HTable.getDefaultExecutor(hbaseConf);
> var params = new BufferedMutatorParams(tableName);
> params.pool(mySingletonPool);
> var myMutator = conn.getBufferedMutator(params);
> {code}
> And avoid code like this:
> {code:java}
> var myMutator = conn.getBufferedMutator(tableName);
> {code}
> The full story:
> My application started leaking threads after upgrading from hbase client
> 1.4.13 to 2.4.4. So much so that after less than a minute of runtime more
> that 30k threads are leaked and all available virtual memory on the box (> 50
> GB) is consumed. Other processes on the box start crashing with memory
> allocation errors. Even running {{ls}} at the shell fails with OS resource
> allocation failures.
> A thread dump after just a few seconds of runtime shows thousands of threads
> like this:
> {code:java}
> "htable-pool-0" #8841 prio=5 os_prio=0 cpu=0.15ms elapsed=7.49s
> tid=0x7efb6d2a1000 nid=0x57d2 waiting on condition [0x7ef8a6c38000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at jdk.internal.misc.Unsafe.park(java.base@11.0.6/Native Method)
> - parking to wait for <0x0007e7cd6188> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
> at
> java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.6/LockSupport.java:234)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(java.base@11.0.6/SynchronousQueue.java:462)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(java.base@11.0.6/SynchronousQueue.java:361)
> at
> java.util.concurrent.SynchronousQueue.poll(java.base@11.0.6/SynchronousQueue.java:937)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.6/ThreadPoolExecutor.java:1053)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.6/ThreadPoolExecutor.java:1114)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.6/ThreadPoolExecutor.java:628)
> at java.lang.Thread.run(java.base@11.0.6/Thread.java:834)
> {code}
>
> Note: All the threads are labeled {{htable-pool-0}}. That suggests we're
> leaking thread executors not just threads. The {{htable-pool}} part indicates
> the problem is to do with {{HTable.getDefaultExecutor(conf)}} and the only
> part of my code that interacts with that is a call to
> {{conn.getBufferedMutator(tableName)}}.
>
> Looking at the hbase client code shows a few problems:
> 1) Neither 1.4.13 nor 2.4.4's behavior matches the documentation for
> {{conn.getBufferedMutator(tableName)}} which says:
> {quote}This BufferedMutator will use the Connection's ExecutorService.
> {quote}
> That suggests some singleton thread executor is being used which is not the
> case.
>
> 2) Under 1.4.13 you get a new {{ThreadPoolExecutor}} for every
>
[jira] [Created] (HBASE-26111) Release 3.0.0-alpha-2
Duo Zhang created HBASE-26111: - Summary: Release 3.0.0-alpha-2 Key: HBASE-26111 URL: https://issues.apache.org/jira/browse/HBASE-26111 Project: HBase Issue Type: Umbrella Reporter: Duo Zhang -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-26052) Release 3.0.0-alpha-1
[ https://issues.apache.org/jira/browse/HBASE-26052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang resolved HBASE-26052. --- Resolution: Fixed > Release 3.0.0-alpha-1 > - > > Key: HBASE-26052 > URL: https://issues.apache.org/jira/browse/HBASE-26052 > Project: HBase > Issue Type: Umbrella >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-26052) Release 3.0.0-alpha-1
[ https://issues.apache.org/jira/browse/HBASE-26052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17385203#comment-17385203 ] Duo Zhang commented on HBASE-26052: --- Ah, forgot to push the rel/3.0.0-alpha-1 tag... Anyway, all things are done. Let me resolve. > Release 3.0.0-alpha-1 > - > > Key: HBASE-26052 > URL: https://issues.apache.org/jira/browse/HBASE-26052 > Project: HBase > Issue Type: Umbrella >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #3504: HBASE-26093 Replication is stuck due to zero length wal file in oldWALs directory
Apache-HBase commented on pull request #3504: URL: https://github.com/apache/hbase/pull/3504#issuecomment-884608765 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 28s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 29s | master passed | | +1 :green_heart: | compile | 1m 21s | master passed | | +1 :green_heart: | shadedjars | 8m 55s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 41s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 21s | the patch passed | | +1 :green_heart: | compile | 1m 18s | the patch passed | | +1 :green_heart: | javac | 1m 18s | the patch passed | | +1 :green_heart: | shadedjars | 8m 45s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 40s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 141m 6s | hbase-server in the patch failed. | | | | 174m 21s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3504/6/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3504 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 174e307de7b5 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 21c4578f22 | | Default Java | AdoptOpenJDK-11.0.10+9 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3504/6/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3504/6/testReport/ | | Max. process+thread count | 4097 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3504/6/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3513: HBASE-26109: Update version to 1.7.2-SNAPSHOT
Apache-HBase commented on pull request #3513: URL: https://github.com/apache/hbase/pull/3513#issuecomment-884606989 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 11m 39s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | -0 :warning: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ branch-1 Compile Tests _ | | +0 :ok: | mvndep | 2m 52s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 7m 58s | branch-1 passed | | +1 :green_heart: | compile | 1m 47s | branch-1 passed with JDK Azul Systems, Inc.-1.8.0_262-b19 | | +1 :green_heart: | compile | 1m 42s | branch-1 passed with JDK Azul Systems, Inc.-1.7.0_272-b10 | | +1 :green_heart: | checkstyle | 5m 2s | branch-1 passed | | -1 :x: | shadedjars | 0m 19s | branch has 7 errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 2m 29s | branch-1 passed with JDK Azul Systems, Inc.-1.8.0_262-b19 | | +1 :green_heart: | javadoc | 2m 41s | branch-1 passed with JDK Azul Systems, Inc.-1.7.0_272-b10 | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 1m 48s | branch-1 passed | | +0 :ok: | mvndep | 2m 16s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 1m 56s | the patch passed | | +1 :green_heart: | compile | 1m 41s | the patch passed with JDK Azul Systems, Inc.-1.8.0_262-b19 | | +1 :green_heart: | javac | 1m 41s | the patch passed | | +1 :green_heart: | compile | 1m 44s | the patch passed with JDK Azul Systems, Inc.-1.7.0_272-b10 | | +1 :green_heart: | javac | 1m 44s | the patch passed | | +1 :green_heart: | checkstyle | 4m 58s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | -1 :x: | xml | 0m 6s | The patch has 35 ill-formed XML file(s). | | -1 :x: | shadedjars | 0m 14s | patch has 7 errors when building our shaded downstream artifacts. | | +1 :green_heart: | hadoopcheck | 4m 34s | Patch does not cause any errors with Hadoop 2.8.5 2.9.2. | | +1 :green_heart: | javadoc | 2m 18s | the patch passed with JDK Azul Systems, Inc.-1.8.0_262-b19 | | +1 :green_heart: | javadoc | 2m 44s | the patch passed with JDK Azul Systems, Inc.-1.7.0_272-b10 | ||| _ Other Tests _ | | -1 :x: | unit | 186m 30s | root in the patch failed. | | -1 :x: | asflicense | 0m 38s | The patch generated 6 ASF License warnings. | | | | 249m 37s | | | Reason | Tests | |---:|:--| | XML | Parsing Error(s): | | | hbase-annotations/pom.xml | | | hbase-archetypes/hbase-archetype-builder/pom.xml | | | hbase-archetypes/hbase-client-project/pom.xml | | | hbase-archetypes/hbase-shaded-client-project/pom.xml | | | hbase-archetypes/pom.xml | | | hbase-assembly/pom.xml | | | hbase-checkstyle/pom.xml | | | hbase-client/pom.xml | | | hbase-common/pom.xml | | | hbase-error-prone/pom.xml | | | hbase-examples/pom.xml | | | hbase-external-blockcache/pom.xml | | | hbase-hadoop-compat/pom.xml | | | hbase-hadoop2-compat/pom.xml | | | hbase-hbtop/pom.xml | | | hbase-it/pom.xml | | | hbase-metrics-api/pom.xml | | | hbase-metrics/pom.xml | | | hbase-prefix-tree/pom.xml | | | hbase-procedure/pom.xml | | | hbase-protocol/pom.xml | | | hbase-resource-bundle/pom.xml | | | hbase-rest/pom.xml | | | hbase-rsgroup/pom.xml | | | hbase-server/pom.xml | | | hbase-shaded/hbase-shaded-check-invariants/pom.xml | | | hbase-shaded/hbase-shaded-client/pom.xml | | | hbase-shaded/hbase-shaded-server/pom.xml | | | hbase-shaded/hbase-shaded-testing-util-tester/pom.xml | | | hbase-shaded/hbase-shaded-testing-util/pom.xml | | | hbase-shaded/pom.xml | | | hbase-shell/pom.xml | | | hbase-testing-util/pom.xml | | | hbase-thrift/pom.xml | | | pom.xml | | Failed junit tests | hadoop.hbase.client.TestAdmin1 | | | hadoop.hbase.mapreduce.TestLoadIncrementalHFiles | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3513/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3513 | | JIRA Issue |
[GitHub] [hbase] Apache-HBase commented on pull request #3460: HBASE-26064 Introduce a StoreFileTracker to abstract the stor…
Apache-HBase commented on pull request #3460: URL: https://github.com/apache/hbase/pull/3460#issuecomment-884603122 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 37s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 6m 46s | master passed | | +1 :green_heart: | compile | 1m 52s | master passed | | +1 :green_heart: | shadedjars | 11m 13s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 58s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 6m 13s | the patch passed | | +1 :green_heart: | compile | 1m 51s | the patch passed | | +1 :green_heart: | javac | 1m 51s | the patch passed | | +1 :green_heart: | shadedjars | 13m 23s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 59s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 238m 0s | hbase-server in the patch passed. | | | | 285m 10s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3460/10/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3460 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux e01bfad0 4.15.0-142-generic #146-Ubuntu SMP Tue Apr 13 01:11:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 21c4578f22 | | Default Java | AdoptOpenJDK-11.0.10+9 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3460/10/testReport/ | | Max. process+thread count | 2928 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3460/10/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3514: HBASE-26110: Add download links for 1.7.1
Apache-HBase commented on pull request #3514: URL: https://github.com/apache/hbase/pull/3514#issuecomment-884577331 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 29s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 3s | master passed | | +1 :green_heart: | mvnsite | 9m 15s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 36s | the patch passed | | +1 :green_heart: | mvnsite | 8m 59s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | xml | 0m 2s | The patch has no ill-formed XML file. | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 19s | The patch does not generate ASF License warnings. | | | | 28m 36s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3514/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3514 | | JIRA Issue | HBASE-26110 | | Optional Tests | dupname asflicense mvnsite xml | | uname | Linux 0cac9c6c9a2f 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 21c4578f22 | | Max. process+thread count | 89 (vs. ulimit of 3) | | modules | C: . U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3514/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3504: HBASE-26093 Replication is stuck due to zero length wal file in oldWALs directory
Apache-HBase commented on pull request #3504: URL: https://github.com/apache/hbase/pull/3504#issuecomment-884575246 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 38s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 12s | master passed | | +1 :green_heart: | compile | 4m 4s | master passed | | +1 :green_heart: | checkstyle | 1m 16s | master passed | | +1 :green_heart: | spotbugs | 2m 44s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 42s | the patch passed | | +1 :green_heart: | compile | 4m 6s | the patch passed | | +1 :green_heart: | javac | 4m 6s | the patch passed | | +1 :green_heart: | checkstyle | 1m 12s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 23m 50s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | spotbugs | 3m 23s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 16s | The patch does not generate ASF License warnings. | | | | 60m 17s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3504/6/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3504 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux 00c3c6839e41 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 21c4578f22 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Max. process+thread count | 96 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3504/6/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3504: HBASE-26093 Replication is stuck due to zero length wal file in oldWALs directory
Apache-HBase commented on pull request #3504: URL: https://github.com/apache/hbase/pull/3504#issuecomment-884575029 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 39s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 5m 21s | master passed | | +1 :green_heart: | compile | 1m 28s | master passed | | +1 :green_heart: | shadedjars | 12m 47s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 55s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 6m 8s | the patch passed | | +1 :green_heart: | compile | 1m 31s | the patch passed | | +1 :green_heart: | javac | 1m 31s | the patch passed | | +1 :green_heart: | shadedjars | 10m 56s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 49s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 16m 48s | hbase-server in the patch failed. | | | | 60m 1s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3504/6/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3504 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux ee15aa0a8fd5 4.15.0-142-generic #146-Ubuntu SMP Tue Apr 13 01:11:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 21c4578f22 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3504/6/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3504/6/testReport/ | | Max. process+thread count | 638 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3504/6/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-24984) WAL corruption due to early DBBs re-use when Durability.ASYNC_WAL is used with multi operation
[ https://issues.apache.org/jira/browse/HBASE-24984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17385162#comment-17385162 ] Huaxiang Sun commented on HBASE-24984: -- Thank you, [~gouravk] > WAL corruption due to early DBBs re-use when Durability.ASYNC_WAL is used > with multi operation > -- > > Key: HBASE-24984 > URL: https://issues.apache.org/jira/browse/HBASE-24984 > Project: HBase > Issue Type: Bug > Components: rpc, wal >Affects Versions: 2.1.6 >Reporter: Liu Junhong >Assignee: Gaurav Kanade >Priority: Critical > Fix For: 2.5.0, 2.3.6, 3.0.0-alpha-2, 2.4.5 > > Attachments: > 0001-HBASE-24984-WAL-corruption-due-to-early-DBBs-re-use-.patch > > > After bugfix HBASE-22539, When client use BufferedMutator or multiple > mutation , there will be one RpcCall and mutliple FSWALEntry . At the time > RpcCall finish and one FSWALEntry call release() , the remain FSWALEntries > may trigger RuntimeException or segmentation fault . > We should use RefCnt instead of AtomicInteger for > org.apache.hadoop.hbase.ipc.ServerCall.reference? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #3460: HBASE-26064 Introduce a StoreFileTracker to abstract the stor…
Apache-HBase commented on pull request #3460: URL: https://github.com/apache/hbase/pull/3460#issuecomment-884568585 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 26s | Docker mode activated. | | -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 1s | master passed | | +1 :green_heart: | compile | 1m 0s | master passed | | +1 :green_heart: | shadedjars | 8m 7s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 40s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 40s | the patch passed | | +1 :green_heart: | compile | 1m 2s | the patch passed | | +1 :green_heart: | javac | 1m 2s | the patch passed | | +1 :green_heart: | shadedjars | 8m 11s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 37s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 144m 16s | hbase-server in the patch passed. | | | | 174m 28s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3460/10/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3460 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 8d3eabded48e 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 21c4578f22 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3460/10/testReport/ | | Max. process+thread count | 4228 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3460/10/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3514: HBASE-26110: Add download links for 1.7.1
Apache-HBase commented on pull request #3514: URL: https://github.com/apache/hbase/pull/3514#issuecomment-884568172 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 39s | Docker mode activated. | | -0 :warning: | yetus | 0m 5s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | ||| _ Patch Compile Tests _ | ||| _ Other Tests _ | | | | 2m 0s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3514/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3514 | | JIRA Issue | HBASE-26110 | | Optional Tests | | | uname | Linux c9eef5afbc4b 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 21c4578f22 | | Max. process+thread count | 47 (vs. ulimit of 3) | | modules | C: . U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3514/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3514: HBASE-26110: Add download links for 1.7.1
Apache-HBase commented on pull request #3514: URL: https://github.com/apache/hbase/pull/3514#issuecomment-884568038 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 29s | Docker mode activated. | | -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | ||| _ Patch Compile Tests _ | ||| _ Other Tests _ | | | | 1m 42s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3514/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3514 | | JIRA Issue | HBASE-26110 | | Optional Tests | | | uname | Linux 008c0474dcd5 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 21c4578f22 | | Max. process+thread count | 53 (vs. ulimit of 3) | | modules | C: . U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3514/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-26110) Add download links for 1.7.1
[ https://issues.apache.org/jira/browse/HBASE-26110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharath Vissapragada updated HBASE-26110: - Status: Patch Available (was: Open) > Add download links for 1.7.1 > > > Key: HBASE-26110 > URL: https://issues.apache.org/jira/browse/HBASE-26110 > Project: HBase > Issue Type: Sub-task > Components: website >Affects Versions: 1.7.1 >Reporter: Bharath Vissapragada >Assignee: Bharath Vissapragada >Priority: Major > Fix For: 1.7.1 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] bharathv opened a new pull request #3514: HBASE-26110: Add download links for 1.7.1
bharathv opened a new pull request #3514: URL: https://github.com/apache/hbase/pull/3514 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (HBASE-26110) Add download links for 1.7.1
Bharath Vissapragada created HBASE-26110: Summary: Add download links for 1.7.1 Key: HBASE-26110 URL: https://issues.apache.org/jira/browse/HBASE-26110 Project: HBase Issue Type: Sub-task Components: website Affects Versions: 1.7.1 Reporter: Bharath Vissapragada Assignee: Bharath Vissapragada Fix For: 1.7.1 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #3504: HBASE-26093 Replication is stuck due to zero length wal file in oldWALs directory
Apache-HBase commented on pull request #3504: URL: https://github.com/apache/hbase/pull/3504#issuecomment-884552280 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 34s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 5m 24s | master passed | | +1 :green_heart: | compile | 1m 31s | master passed | | +1 :green_heart: | shadedjars | 11m 16s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 51s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 5m 6s | the patch passed | | +1 :green_heart: | compile | 1m 36s | the patch passed | | +1 :green_heart: | javac | 1m 36s | the patch passed | | +1 :green_heart: | shadedjars | 12m 9s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 52s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 271m 2s | hbase-server in the patch failed. | | | | 313m 44s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3504/5/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3504 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 965643601b8e 4.15.0-142-generic #146-Ubuntu SMP Tue Apr 13 01:11:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 21c4578f22 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3504/5/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3504/5/testReport/ | | Max. process+thread count | 2886 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3504/5/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] joshelser commented on a change in pull request #3488: HBASE-25393 Support split and merge region with direct insert into CF…
joshelser commented on a change in pull request #3488:
URL: https://github.com/apache/hbase/pull/3488#discussion_r674299072
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/MergeTableRegionsProcedure.java
##
@@ -107,6 +113,21 @@ public MergeTableRegionsProcedure(final MasterProcedureEnv
env,
// Preflight depends on mergedRegion being set (at least).
preflightChecks(env, true);
this.force = force;
+createMergeStrategy(env.getMasterConfiguration());
+ }
+
+ private void createMergeStrategy(Configuration conf) {
+String className = conf.get(MERGE_REGION_STRATEGY,
DefaultMergeStrategy.class.getName());
+createMergeStrategy(className);
+ }
+
+ private void createMergeStrategy(String className) {
Review comment:
Can replace this with `o.a.hadoop.util.ReflectionUtils#newInstance()`.
We also have a `ReflectionUtils` in HBase that lets you do the same with a
custom constructor (if we need one for some reason)
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/MergeRegionsStrategy.java
##
@@ -0,0 +1,99 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.master.assignment;
+
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.RegionInfo;
+import org.apache.hadoop.hbase.master.MasterFileSystem;
+import org.apache.hadoop.hbase.master.RegionState;
+import org.apache.hadoop.hbase.master.procedure.MasterProcedureEnv;
+import org.apache.hadoop.hbase.regionserver.HRegionFileSystem;
+import org.apache.hadoop.hbase.util.CommonFSUtils;
+import org.apache.yetus.audience.InterfaceAudience;
+
+import java.io.IOException;
+
+/**
+ * Region merge directory creation strategy to decouple create dir logic from
+ * MergeTableRegionsProcedure and allow for plugable
behaviour.
+ */
+@InterfaceAudience.Private
+public abstract class MergeRegionsStrategy {
+
+ /**
+ * Creates the resulting merging region dir and files in the file system,
then updates
+ * meta table information for the given region. Specific logic on where in
the files system to
+ * create the region structure is delegated to
innerMergeRegions and the
+ * actual HRegionFileSystemWriteStrategy implementation.
+ * @param env the MasterProcedureEnv wrapping several meta information
required.
+ * @param regionsToMerge array of RegionInfo representing the regions being
merged.
+ * @param mergedRegion the resulting merging region.
+ * @throws IOException if any error occurs while creating the region dir.
+ */
+ public void createMergedRegion(MasterProcedureEnv env, RegionInfo[]
regionsToMerge,
+RegionInfo mergedRegion) throws IOException {
+final MasterFileSystem mfs = env.getMasterServices().getMasterFileSystem();
+final Path tabledir = CommonFSUtils.getTableDir(mfs.getRootDir(),
regionsToMerge[0].getTable());
+final FileSystem fs = mfs.getFileSystem();
+HRegionFileSystem mergeRegionFs = innerMergeRegions(env, fs,
regionsToMerge,
+ tabledir, mergedRegion);
+assert mergeRegionFs != null;
+mergeRegionFs.commitMergedRegion(mergedRegion);
+// Prepare to create merged regions
+env.getAssignmentManager().getRegionStates().
+
getOrCreateRegionStateNode(mergedRegion).setState(RegionState.State.MERGING_NEW);
+ }
+
+ /**
+ * Should define specific logic about where in the file system the region
structure should be
+ * created.
+ * @param env the MasterProcedureEnv wrapping several meta information
required.
+ * @param fs the FileSystem instance to write the region directory.
+ * @param regionsToMerge array of RegionInfo representing the regions being
merged.
+ * @param tableDir Path instance for the table dir.
+ * @param mergedRegion the resulting merging region.
+ * @return HRegionFileSystem for the resulting merging region.
+ * @throws IOException if any error occurs while creating the region dir.
+ */
+ abstract protected HRegionFileSystem innerMergeRegions(MasterProcedureEnv
env, FileSystem fs,
+RegionInfo[] regionsToMerge, Path
[GitHub] [hbase] Apache-HBase commented on pull request #3512: hbase24984 branch-2
Apache-HBase commented on pull request #3512: URL: https://github.com/apache/hbase/pull/3512#issuecomment-884528401 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 4m 34s | Docker mode activated. | | -0 :warning: | yetus | 0m 7s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2 Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 33s | branch-2 passed | | +1 :green_heart: | compile | 1m 9s | branch-2 passed | | +1 :green_heart: | shadedjars | 7m 8s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 46s | branch-2 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 10s | the patch passed | | +1 :green_heart: | compile | 1m 8s | the patch passed | | +1 :green_heart: | javac | 1m 8s | the patch passed | | +1 :green_heart: | shadedjars | 7m 8s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 40s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 138m 40s | hbase-server in the patch passed. | | | | 172m 16s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3512/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3512 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux c62a2867936b 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / 253b83db57 | | Default Java | AdoptOpenJDK-11.0.10+9 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3512/2/testReport/ | | Max. process+thread count | 3973 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3512/2/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-26109) Wrap up 1.7.1 release
[ https://issues.apache.org/jira/browse/HBASE-26109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17385117#comment-17385117 ] Bharath Vissapragada commented on HBASE-26109: -- Waiting for the mirrors to catch up so that I can make the doc change. > Wrap up 1.7.1 release > - > > Key: HBASE-26109 > URL: https://issues.apache.org/jira/browse/HBASE-26109 > Project: HBase > Issue Type: Task > Components: documentation >Affects Versions: 1.7.1 >Reporter: Bharath Vissapragada >Assignee: Bharath Vissapragada >Priority: Major > Labels: release > Fix For: 1.7.1 > > > - Set branch-1 version to 1.7.2-SNAPSHOT > - Update SVN release bits, add download links to 1.7.1 > - Remove 1.7.0 from the mirrors > - Commit HBASE-26071 > in that order. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] bharathv opened a new pull request #3513: HBASE-26109: Update version to 1.7.2-SNAPSHOT
bharathv opened a new pull request #3513: URL: https://github.com/apache/hbase/pull/3513 Merge 1.7.1 CHANGES.txt (Website change in master branch docs) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3460: HBASE-26064 Introduce a StoreFileTracker to abstract the stor…
Apache-HBase commented on pull request #3460: URL: https://github.com/apache/hbase/pull/3460#issuecomment-884519335 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 31s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 54s | master passed | | +1 :green_heart: | compile | 3m 27s | master passed | | +1 :green_heart: | checkstyle | 1m 10s | master passed | | +1 :green_heart: | spotbugs | 2m 17s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 50s | the patch passed | | +1 :green_heart: | compile | 3m 21s | the patch passed | | +1 :green_heart: | javac | 3m 21s | the patch passed | | +1 :green_heart: | checkstyle | 1m 9s | hbase-server: The patch generated 0 new + 136 unchanged - 12 fixed = 136 total (was 148) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 18m 36s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | spotbugs | 2m 15s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 12s | The patch does not generate ASF License warnings. | | | | 48m 42s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3460/10/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3460 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux 0c789804ab52 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 21c4578f22 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Max. process+thread count | 96 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3460/10/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] shahrs87 commented on pull request #3504: HBASE-26093 Replication is stuck due to zero length wal file in oldWALs directory
shahrs87 commented on pull request #3504:
URL: https://github.com/apache/hbase/pull/3504#issuecomment-884494797
```
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
132.699 s <<< FAILURE! - in
org.apache.hadoop.hbase.replication.TestReplicationKillMasterRSWithSeparateOldWALs
[ERROR]
org.apache.hadoop.hbase.replication.TestReplicationKillMasterRSWithSeparateOldWALs.killOneMasterRS
Time elapsed: 68.343 s <<< ERROR!
org.apache.hadoop.hbase.client.RetriesExhaustedException:
Failed after attempts=2, exceptions:
2021-07-21T19:02:25.047Z, java.net.ConnectException: Call to
address=rushabh-ltmflld.internal.salesforce.com:58846 failed on connection
exception:
org.apache.hbase.thirdparty.io.netty.channel.ConnectTimeoutException:
connection timed out:
rushabh-ltmflld.internal.salesforce.com/10.33.225.220:58846
2021-07-21T19:03:15.052Z,
org.apache.hadoop.hbase.exceptions.TimeoutIOException: Timeout(49891ms) waiting
for region location for test, row='ddd', replicaId=0
Caused by: org.apache.hadoop.hbase.exceptions.TimeoutIOException:
Timeout(49891ms) waiting for region location for test, row='ddd', replicaId=0
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
214.664 s - in org.apache.hadoop.hbase.replication.TestReplicationDroppedTables
[INFO] Tests run: 23, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
355.69 s - in
org.apache.hadoop.hbase.security.access.TestSnapshotScannerHDFSAclController
[INFO]
[INFO] Results:
[INFO]
[ERROR] Errors:
[ERROR]
TestReplicationKillMasterRSWithSeparateOldWALs>TestReplicationKillMasterRS.killOneMasterRS:47->TestReplicationKillRS.loadTableAndKillRS:61
» RetriesExhausted
[INFO]
[ERROR] Tests run: 28, Failures: 0, Errors: 1, Skipped: 0
[INFO]
```
TestReplicationKillMasterRSWithSeparateOldWALs is a valid failure.
It is failing here with the following stack trace
```
RSS path:
hdfs://localhost:50931/user/rushabh.shah/test-data/764d7a00-1268-227d-46e0-bbc4d930588d/oldWALs/rushabh-ltmflld.internal.salesforce.com%2C50948%2C1626900149374.1626900153398
RSS serverName: null
2021-07-21T16:42:56,624 ERROR
[RS_CLAIM_REPLICATION_QUEUE-regionserver/rushabh-ltmflld:0-0.replicationSource,2-rushabh-ltmflld.internal.salesforce.com,50948,1626900149374.replicationSource.wal-reader.rushabh-ltmflld.internal.salesforce.com%2C50948%2C1626900149374,2-rushabh-ltmflld.internal.salesforce.com,50948,1626900149374]
regionserver.ReplicationSource(428): Unexpected exception in
RS_CLAIM_REPLICATION_QUEUE-regionserver/rushabh-ltmflld:0-0.replicationSource,2-rushabh-ltmflld.internal.salesforce.com,50948,1626900149374.replicationSource.wal-reader.rushabh-ltmflld.internal.salesforce.com%2C50948%2C1626900149374,2-rushabh-ltmflld.internal.salesforce.com,50948,1626900149374
currentPath=hdfs://localhost:50931/user/rushabh.shah/test-data/764d7a00-1268-227d-46e0-bbc4d930588d/oldWALs/rushabh-ltm%2C50948%2C1626900149374.1626900153398
java.lang.NullPointerException: null
at
org.apache.hadoop.hbase.wal.AbstractFSWALProvider.findArchivedLog(AbstractFSWALProvider.java:526)
~[classes/:?]
at
org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.handleFileNotFound(WALEntryStream.java:321)
~[classes/:?]
at
org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.openReader(WALEntryStream.java:342)
~[classes/:?]
at
org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.openNextLog(WALEntryStream.java:307)
~[classes/:?]
at
org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.checkReader(WALEntryStream.java:297)
~[classes/:?]
at
org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.tryAdvanceEntry(WALEntryStream.java:177)
~[classes/:?]
at
org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.hasNext(WALEntryStream.java:103)
~[classes/:?]
at
org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.tryAdvanceStreamAndCreateWALBatch(ReplicationSourceWALReader.java:243)
~[classes/:?]
at
org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.run(ReplicationSourceWALReader.java:141)
~[classes/:?]
```
I printed the wal file name and it failed with NPE above. Interestingly the
path contains oldWALs directory and it is trying to find archive log for
oldWALs directory.
Below is the method of findArchivedLog method
```
public static Path findArchivedLog(Path path, Configuration conf) throws
IOException {
Path walRootDir = CommonFSUtils.getWALRootDir(conf);
FileSystem fs = path.getFileSystem(conf);
// Try finding the log in old dir
Path oldLogDir = new Path(walRootDir, HConstants.HREGION_OLDLOGDIR_NAME);
Path archivedLogLocation = new Path(oldLogDir, path.getName());
if (fs.exists(archivedLogLocation)) {
[GitHub] [hbase] wchevreuil commented on pull request #3460: HBASE-26064 Introduce a StoreFileTracker to abstract the stor…
wchevreuil commented on pull request #3460: URL: https://github.com/apache/hbase/pull/3460#issuecomment-884492959 Sorry, hit close button by mistake thinking I was closing a comment of mine. Had just reopened it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] wchevreuil removed a comment on pull request #3460: HBASE-26064 Introduce a StoreFileTracker to abstract the stor…
wchevreuil removed a comment on pull request #3460: URL: https://github.com/apache/hbase/pull/3460#issuecomment-884492012 > @wchevreuil PTAL. > > I will commit this to a feature branch first as there are still lots of other place to integrate, such as MergeTableRegionsProcedure/SplitTableRegionProcedure, snapshot, major compaction tool and so on(just search the usage of HRegionFileSystem.getStorefiles). > > This is another direction on how to make it possible to write directly to data directory for compaction and flush. And I think this is a more general way to support different persistency ways, either with a file or with a region, and easier to migrate betewwn different persisent ways(or even without persistency) > > Feel free to comment anything here. I will put up a design doc soon. > > Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] wchevreuil closed pull request #3460: HBASE-26064 Introduce a StoreFileTracker to abstract the stor…
wchevreuil closed pull request #3460: URL: https://github.com/apache/hbase/pull/3460 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] wchevreuil commented on pull request #3460: HBASE-26064 Introduce a StoreFileTracker to abstract the stor…
wchevreuil commented on pull request #3460: URL: https://github.com/apache/hbase/pull/3460#issuecomment-884492012 > @wchevreuil PTAL. > > I will commit this to a feature branch first as there are still lots of other place to integrate, such as MergeTableRegionsProcedure/SplitTableRegionProcedure, snapshot, major compaction tool and so on(just search the usage of HRegionFileSystem.getStorefiles). > > This is another direction on how to make it possible to write directly to data directory for compaction and flush. And I think this is a more general way to support different persistency ways, either with a file or with a region, and easier to migrate betewwn different persisent ways(or even without persistency) > > Feel free to comment anything here. I will put up a design doc soon. > > Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] wchevreuil commented on a change in pull request #3460: HBASE-26064 Introduce a StoreFileTracker to abstract the stor…
wchevreuil commented on a change in pull request #3460:
URL: https://github.com/apache/hbase/pull/3460#discussion_r674328976
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java
##
@@ -1088,9 +1086,19 @@ public HStoreFile tryCommitRecoveredHFile(Path path)
throws IOException {
* @return store file created.
*/
private HStoreFile commitFile(Path path, long logCacheFlushId, MonitoredTask
status)
- throws IOException {
-// Write-out finished successfully, move into the right spot
-Path dstPath =
getRegionFileSystem().commitStoreFile(getColumnFamilyName(), path);
+throws IOException {
+HRegionFileSystem hfs = getRegionFileSystem();
+Path storeDir = hfs.getStoreDir(getColumnFamilyName());
+Path dstPath;
+// As we want to support writing to data directory directly, here we need
to check whether the
+// store file is already in the right place
+if (path.getParent() != null && path.getParent().equals(storeDir)) {
+ // already in the right place, skip renmaing
+ dstPath = path;
+} else {
+ // Write-out finished successfully, move into the right spot
+ dstPath = hfs.commitStoreFile(getColumnFamilyName(), path);
+}
Review comment:
nit: As mentioned in the design doc, this could be delegated to SFT
implementations (similarly to how createWriter was moved from
Compactor/StoreFlusher to SFT).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3511: HBASE-26107 MOB compaction with missing files catches incorrect exception
Apache-HBase commented on pull request #3511: URL: https://github.com/apache/hbase/pull/3511#issuecomment-884490575 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 38s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 5m 35s | master passed | | +1 :green_heart: | compile | 1m 24s | master passed | | +1 :green_heart: | shadedjars | 11m 4s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 51s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 5m 21s | the patch passed | | +1 :green_heart: | compile | 1m 40s | the patch passed | | +1 :green_heart: | javac | 1m 40s | the patch passed | | +1 :green_heart: | shadedjars | 11m 25s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 52s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 260m 6s | hbase-server in the patch passed. | | | | 302m 17s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3511/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3511 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 4c8540f700e6 4.15.0-142-generic #146-Ubuntu SMP Tue Apr 13 01:11:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 21c4578f22 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3511/1/testReport/ | | Max. process+thread count | 2999 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3511/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3504: HBASE-26093 Replication is stuck due to zero length wal file in oldWALs directory
Apache-HBase commented on pull request #3504: URL: https://github.com/apache/hbase/pull/3504#issuecomment-884487858 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 30s | Docker mode activated. | | -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 15s | master passed | | +1 :green_heart: | compile | 1m 15s | master passed | | +1 :green_heart: | shadedjars | 8m 44s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 43s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 31s | the patch passed | | +1 :green_heart: | compile | 1m 16s | the patch passed | | +1 :green_heart: | javac | 1m 16s | the patch passed | | +1 :green_heart: | shadedjars | 8m 36s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 42s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 143m 17s | hbase-server in the patch failed. | | | | 176m 1s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3504/5/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3504 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 082ec6ae49b4 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 21c4578f22 | | Default Java | AdoptOpenJDK-11.0.10+9 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3504/5/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3504/5/testReport/ | | Max. process+thread count | 3863 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3504/5/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-26109) Wrap up 1.7.1 release
[
https://issues.apache.org/jira/browse/HBASE-26109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17385089#comment-17385089
]
Bharath Vissapragada commented on HBASE-26109:
--
Looks like I don't have access to the svn repo to move content around. Working
with Andrew offline to move the RC bits to release.
{noformat}
svn mv https://dist.apache.org/repos/dist/dev/hbase/1.7.1RC0
https://dist.apache.org/repos/dist/release/hbase/1.7.1 -m"Release Apache HBase
1.7.1"
svn: E175013: Access to '/repos/dist/!svn/rvr/48933/dev/hbase/1.7.1RC0'
forbidden
{noformat}
> Wrap up 1.7.1 release
> -
>
> Key: HBASE-26109
> URL: https://issues.apache.org/jira/browse/HBASE-26109
> Project: HBase
> Issue Type: Task
> Components: documentation
>Affects Versions: 1.7.1
>Reporter: Bharath Vissapragada
>Assignee: Bharath Vissapragada
>Priority: Major
> Labels: release
> Fix For: 1.7.1
>
>
> - Set branch-1 version to 1.7.2-SNAPSHOT
> - Update SVN release bits, add download links to 1.7.1
> - Remove 1.7.0 from the mirrors
> - Commit HBASE-26071
> in that order.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #3512: hbase24984 branch-2
Apache-HBase commented on pull request #3512: URL: https://github.com/apache/hbase/pull/3512#issuecomment-884454903 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 36s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ branch-2 Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 58s | branch-2 passed | | +1 :green_heart: | compile | 3m 21s | branch-2 passed | | +1 :green_heart: | checkstyle | 1m 10s | branch-2 passed | | +1 :green_heart: | spotbugs | 2m 15s | branch-2 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 25s | the patch passed | | +1 :green_heart: | compile | 3m 27s | the patch passed | | +1 :green_heart: | javac | 3m 27s | the patch passed | | +1 :green_heart: | checkstyle | 1m 5s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 11m 59s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. | | +1 :green_heart: | spotbugs | 2m 23s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 14s | The patch does not generate ASF License warnings. | | | | 41m 49s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3512/2/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3512 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux d241c20e4d2a 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / 253b83db57 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Max. process+thread count | 96 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3512/2/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] joshelser commented on pull request #3488: HBASE-25393 Support split and merge region with direct insert into CF…
joshelser commented on pull request #3488: URL: https://github.com/apache/hbase/pull/3488#issuecomment-884452513 > I guess why we create it in tmp directory first is because the CatalogJanitor? Not sure. But theoretically, since we do not need to scan the filesystem to get all the regions for a table when opening it, I do not think we need to introduce a Strategy to handle both cases, just change the code to write it to the final place. This could be done on master and branch-2. > I guess hbck recovery could be a problem for the default tracking, if the resulting merging/splitting region dir got already created in the FS, fails before completing the operation then vanishes from meta That's a fun thought. Things like rebuilding meta from the filesystem suddenly become dubious. I think we also know that building and HBCK which is capable of identifying a "split-in-progress" is hard. Like Duo say, we can put the Region in the "correct" place and it will just be used once meta is updated. What if we had some special marker in the Region which we cleaned up after it was opened the first time? HBCK could look for that marker and know that if we find a Region on the FS with this marker that isn't in meta, we should be able to just ignore/delete it (as a split/merge which was in-flight). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3512: hbase24984 branch-2
Apache-HBase commented on pull request #3512: URL: https://github.com/apache/hbase/pull/3512#issuecomment-884449760 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 10s | Docker mode activated. | | -0 :warning: | yetus | 0m 7s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2 Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 14s | branch-2 passed | | +1 :green_heart: | compile | 0m 57s | branch-2 passed | | +1 :green_heart: | shadedjars | 6m 0s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 36s | branch-2 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 19s | the patch passed | | +1 :green_heart: | compile | 0m 55s | the patch passed | | +1 :green_heart: | javac | 0m 55s | the patch passed | | +1 :green_heart: | shadedjars | 7m 5s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 39s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 8m 14s | hbase-server in the patch failed. | | | | 33m 49s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3512/2/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3512 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux f8322efc15f6 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / 253b83db57 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3512/2/artifact/yetus-jdk8-hadoop2-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3512/2/testReport/ | | Max. process+thread count | 849 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3512/2/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] joshelser commented on a change in pull request #3389: HBASE-25392 Direct insert compacted HFiles into data directory.
joshelser commented on a change in pull request #3389:
URL: https://github.com/apache/hbase/pull/3389#discussion_r674267085
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/Compactor.java
##
@@ -558,18 +559,19 @@ protected InternalScanner createScanner(HStore store,
ScanInfo scanInfo,
* @param cr the compaction request.
* @param newFiles the new files created by this compaction under a temp dir.
* @param user the running user.
+ * @param fileAcessor a lambda expression with logic for loading a
HStoreFile given a Path.
* @return A list of the resulting store files already placed in the store
dir and loaded into the
* store cache.
* @throws IOException if the commit fails.
*/
- public List commitCompaction(CompactionRequestImpl cr,
List newFiles, User user)
- throws IOException {
+ public List commitCompaction(CompactionRequestImpl cr,
List newFiles,
+ User user, Function fileAcessor) throws IOException {
Review comment:
I like the idea of giving the callback -- do you think it would make a
nicer API to define our own FunctionalInterface that has some javadoc
associated with it? I think it will leave your calling code in HStore as
simple, but give the casual observer some more understanding what this function
is supposed to do.
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java
##
@@ -1505,7 +1505,14 @@ public void
deleteChangedReaderObserver(ChangedReadersObserver o) {
List newFiles) throws IOException {
// Do the steps necessary to complete the compaction.
setStoragePolicyFromFileName(newFiles);
-List sfs = this.storeEngine.compactor.commitCompaction(cr,
newFiles, user);
+List sfs = this.storeEngine.compactor.commitCompaction(cr,
newFiles, user,
+ p -> {
+try {
+ return this.createStoreFileAndReader((Path) p);
+}catch(IOException e){
Review comment:
nit whitespace/checkstyle
but I like the solution!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] joshelser commented on a change in pull request #3389: HBASE-25392 Direct insert compacted HFiles into data directory.
joshelser commented on a change in pull request #3389:
URL: https://github.com/apache/hbase/pull/3389#discussion_r674264765
##
File path:
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/compactions/TestDefaultCompactor.java
##
@@ -0,0 +1,123 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.regionserver.compactions;
+
+import static junit.framework.TestCase.assertEquals;
+import static org.mockito.Mockito.mock;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.HBaseClassTestRule;
+import org.apache.hadoop.hbase.HBaseTestingUtility;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.client.Table;
+import org.apache.hadoop.hbase.regionserver.HStore;
+import org.apache.hadoop.hbase.regionserver.HStoreFile;
+import org.apache.hadoop.hbase.regionserver.StoreFileWriter;
+import org.apache.hadoop.hbase.testclassification.MediumTests;
+import org.apache.hadoop.hbase.testclassification.RegionServerTests;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.ClassRule;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.rules.TestName;
+
+/**
+ * Test class for DefaultCompactor.
+ */
+@Category({ RegionServerTests.class, MediumTests.class })
+public class TestDefaultCompactor {
+
+ @ClassRule
+ public static final HBaseClassTestRule CLASS_RULE =
+HBaseClassTestRule.forClass(TestDefaultCompactor.class);
+
+ @Rule
+ public TestName name = new TestName();
+
+ private final Configuration config = new Configuration();
+ private HStore store;
+ private final String cfName = "cf";
+ private Compactor.FileDetails mockFileDetails;
+
+ private HBaseTestingUtility UTIL = new HBaseTestingUtility();
+ private TableName table;
+
+ @Before
+ public void setup() throws Exception {
Review comment:
exactly
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] joshelser commented on a change in pull request #3389: HBASE-25392 Direct insert compacted HFiles into data directory.
joshelser commented on a change in pull request #3389:
URL: https://github.com/apache/hbase/pull/3389#discussion_r674264495
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/Compactor.java
##
@@ -533,4 +550,46 @@ protected InternalScanner createScanner(HStore store,
ScanInfo scanInfo,
return new StoreScanner(store, scanInfo, scanners, smallestReadPoint,
earliestPutTs,
dropDeletesFromRow, dropDeletesToRow);
}
+
+ /**
+ * Default implementation for committing store files created after a
compaction. Assumes new files
+ * had been created on a temp directory, so it renames those files into the
actual store dir,
+ * then create a reader and cache it into the store.
+ * @param cr the compaction request.
+ * @param newFiles the new files created by this compaction under a temp dir.
+ * @param user the running user.
+ * @return A list of the resulting store files already placed in the store
dir and loaded into the
+ * store cache.
+ * @throws IOException if the commit fails.
+ */
+ public List commitCompaction(CompactionRequestImpl cr,
List newFiles, User user)
+ throws IOException {
+List sfs = new ArrayList<>(newFiles.size());
+for (Path newFile : newFiles) {
+ assert newFile != null;
+ this.store.validateStoreFile(newFile);
+ // Move the file into the right spot
+ HStoreFile sf = createFileInStoreDir(newFile);
+ if (this.store.getCoprocessorHost() != null) {
+this.store.getCoprocessorHost().postCompact(this.store, sf,
cr.getTracker(), cr, user);
+ }
+ assert sf != null;
+ sfs.add(sf);
+}
+return sfs;
+ }
+
+ /**
+ * Assumes new file was created initially on a temp folder.
Review comment:
> it immediately broke StripeCompactor and DateTieredCompactor, which
are relying in this logic, but not extending DefaultCompactor.
Devil's advocate: do you think that's correct for StripeC and DTC to do?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] joshelser commented on a change in pull request #3389: HBASE-25392 Direct insert compacted HFiles into data directory.
joshelser commented on a change in pull request #3389: URL: https://github.com/apache/hbase/pull/3389#discussion_r674263971 ## File path: hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/Compactor.java ## @@ -278,6 +280,21 @@ protected final StoreFileWriter createTmpWriter(FileDetails fd, boolean shouldDr HConstants.EMPTY_STRING); } + /** + * Default method for initializing a StoreFileWriter in the compaction process, this creates the Review comment: > What I meant is that other compactors may overwrite the method with a different logic. I'm just thinking about what other potential compaction algorithms may choose to do. I would expect that any future compactions would still want to leverage the "commit logic" here to avoid that rename. Just keeping tabs on "pre-optimization" without a specific use-case in mind. Although, as long as other compaction algorithms can extend DirectStoreCompactor and get the new shiny solution, it's not a concern. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] joshelser commented on a change in pull request #3389: HBASE-25392 Direct insert compacted HFiles into data directory.
joshelser commented on a change in pull request #3389:
URL: https://github.com/apache/hbase/pull/3389#discussion_r674262341
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java
##
@@ -694,7 +700,7 @@ private void
refreshStoreFilesInternal(Collection newFiles) throw
refreshStoreSizeAndTotalBytes();
}
- protected HStoreFile createStoreFileAndReader(final Path p) throws
IOException {
+ public HStoreFile createStoreFileAndReader(final Path p) throws IOException {
Review comment:
> Maybe a better solution is to define a lambda function in
Compactor.commitCompaction, so that HStore can wrap up createStoreFileAndReader
when calling Compactor.commitCompaction().
Sounds reasonable!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] joshelser commented on a change in pull request #3389: HBASE-25392 Direct insert compacted HFiles into data directory.
joshelser commented on a change in pull request #3389:
URL: https://github.com/apache/hbase/pull/3389#discussion_r674262341
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java
##
@@ -694,7 +700,7 @@ private void
refreshStoreFilesInternal(Collection newFiles) throw
refreshStoreSizeAndTotalBytes();
}
- protected HStoreFile createStoreFileAndReader(final Path p) throws
IOException {
+ public HStoreFile createStoreFileAndReader(final Path p) throws IOException {
Review comment:
> Maybe a better solution is to define a lambda function in
Compactor.commitCompaction, so that HStore can wrap up createStoreFileAndReader
when calling Compactor.commitCompaction().
Sounds resonable!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] joshelser commented on a change in pull request #3389: HBASE-25392 Direct insert compacted HFiles into data directory.
joshelser commented on a change in pull request #3389:
URL: https://github.com/apache/hbase/pull/3389#discussion_r674261906
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/CacheConfig.java
##
@@ -291,12 +291,36 @@ public void setCacheDataOnWrite(boolean cacheDataOnWrite)
{
* cacheIndexesOnWrite
* cacheBloomsOnWrite
*/
- public void enableCacheOnWrite() {
+ public void enableCacheOnWriteForCompactions() {
this.cacheDataOnWrite = true;
this.cacheIndexesOnWrite = true;
this.cacheBloomsOnWrite = true;
}
+ /**
+ * If hbase.rs.cachecompactedblocksonwrite configuration is set to true and
+ * 'totalCompactedFilesSize' is lower than
'cacheCompactedDataOnWriteThreshold',
+ * enables cache on write for below properties:
+ * - cacheDataOnWrite
+ * - cacheIndexesOnWrite
+ * - cacheBloomsOnWrite
+ *
+ * Otherwise, sets 'cacheDataOnWrite' only to false.
+ *
+ * @param totalCompactedFilesSize the total size of compacted files.
+ * @return true if the checks mentioned above pass and the cache is enabled,
false otherwise.
+ */
+ public boolean enableCacheOnWriteForCompactions(long
totalCompactedFilesSize) {
Review comment:
Cool, can resolve this then? (unless Stack has more to add)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3512: hbase24984 branch-2
Apache-HBase commented on pull request #3512: URL: https://github.com/apache/hbase/pull/3512#issuecomment-884426729 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 15s | Docker mode activated. | | -0 :warning: | yetus | 0m 6s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2 Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 46s | branch-2 passed | | +1 :green_heart: | compile | 0m 57s | branch-2 passed | | +1 :green_heart: | shadedjars | 6m 4s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 37s | branch-2 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 21s | the patch passed | | +1 :green_heart: | compile | 0m 59s | the patch passed | | +1 :green_heart: | javac | 0m 59s | the patch passed | | -1 :x: | shadedjars | 0m 13s | patch has 7 errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 37s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 151m 17s | hbase-server in the patch passed. | | | | 171m 34s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3512/1/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3512 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 3b02b24d785f 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / 253b83db57 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | shadedjars | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3512/1/artifact/yetus-jdk8-hadoop2-check/output/patch-shadedjars.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3512/1/testReport/ | | Max. process+thread count | 3585 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3512/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3512: hbase24984 branch-2
Apache-HBase commented on pull request #3512: URL: https://github.com/apache/hbase/pull/3512#issuecomment-884420019 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 38s | Docker mode activated. | | -0 :warning: | yetus | 0m 7s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2 Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 34s | branch-2 passed | | +1 :green_heart: | compile | 1m 12s | branch-2 passed | | +1 :green_heart: | shadedjars | 7m 17s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 44s | branch-2 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 24s | the patch passed | | +1 :green_heart: | compile | 1m 8s | the patch passed | | +1 :green_heart: | javac | 1m 8s | the patch passed | | -1 :x: | shadedjars | 0m 13s | patch has 7 errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 45s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 138m 10s | hbase-server in the patch passed. | | | | 161m 22s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3512/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3512 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 455b2af780f5 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / 253b83db57 | | Default Java | AdoptOpenJDK-11.0.10+9 | | shadedjars | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3512/1/artifact/yetus-jdk11-hadoop3-check/output/patch-shadedjars.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3512/1/testReport/ | | Max. process+thread count | 3820 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3512/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3504: HBASE-26093 Replication is stuck due to zero length wal file in oldWALs directory
Apache-HBase commented on pull request #3504: URL: https://github.com/apache/hbase/pull/3504#issuecomment-884412703 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 34s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 5m 11s | master passed | | +1 :green_heart: | compile | 3m 47s | master passed | | +1 :green_heart: | checkstyle | 1m 5s | master passed | | +1 :green_heart: | spotbugs | 2m 56s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 23s | the patch passed | | +1 :green_heart: | compile | 4m 2s | the patch passed | | +1 :green_heart: | javac | 4m 2s | the patch passed | | +1 :green_heart: | checkstyle | 1m 13s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 21m 35s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | spotbugs | 2m 52s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 15s | The patch does not generate ASF License warnings. | | | | 56m 54s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3504/5/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3504 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux f853d28a6f6a 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 21c4578f22 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Max. process+thread count | 96 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3504/5/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3511: HBASE-26107 MOB compaction with missing files catches incorrect exception
Apache-HBase commented on pull request #3511: URL: https://github.com/apache/hbase/pull/3511#issuecomment-884407451 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 3m 51s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 32s | master passed | | +1 :green_heart: | compile | 1m 11s | master passed | | +1 :green_heart: | shadedjars | 8m 16s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 43s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 20s | the patch passed | | +1 :green_heart: | compile | 1m 13s | the patch passed | | +1 :green_heart: | javac | 1m 13s | the patch passed | | +1 :green_heart: | shadedjars | 8m 10s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 41s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 135m 9s | hbase-server in the patch passed. | | | | 170m 26s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3511/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3511 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 50a310da73f3 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 21c4578f22 | | Default Java | AdoptOpenJDK-11.0.10+9 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3511/1/testReport/ | | Max. process+thread count | 3813 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3511/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3504: HBASE-26093 Replication is stuck due to zero length wal file in oldWALs directory
Apache-HBase commented on pull request #3504: URL: https://github.com/apache/hbase/pull/3504#issuecomment-884373196 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 56s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 55s | master passed | | +1 :green_heart: | compile | 1m 15s | master passed | | +1 :green_heart: | shadedjars | 10m 1s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 46s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 42s | the patch passed | | +1 :green_heart: | compile | 1m 13s | the patch passed | | +1 :green_heart: | javac | 1m 13s | the patch passed | | +1 :green_heart: | shadedjars | 10m 59s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 45s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 242m 56s | hbase-server in the patch failed. | | | | 281m 46s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3504/4/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3504 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux d2d94bc3cf85 4.15.0-142-generic #146-Ubuntu SMP Tue Apr 13 01:11:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 21c4578f22 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3504/4/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3504/4/testReport/ | | Max. process+thread count | 2991 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3504/4/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-26109) Wrap up 1.7.1 release
[ https://issues.apache.org/jira/browse/HBASE-26109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharath Vissapragada updated HBASE-26109: - Affects Version/s: 1.7.1 > Wrap up 1.7.1 release > - > > Key: HBASE-26109 > URL: https://issues.apache.org/jira/browse/HBASE-26109 > Project: HBase > Issue Type: Task > Components: documentation >Affects Versions: 1.7.1 >Reporter: Bharath Vissapragada >Assignee: Bharath Vissapragada >Priority: Major > Labels: release > Fix For: 1.7.1 > > > - Set branch-1 version to 1.7.2-SNAPSHOT > - Update SVN release bits, add download links to 1.7.1 > - Remove 1.7.0 from the mirrors > - Commit HBASE-26071 > in that order. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-26109) Wrap up 1.7.1 release
[ https://issues.apache.org/jira/browse/HBASE-26109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharath Vissapragada updated HBASE-26109: - Labels: release (was: ) > Wrap up 1.7.1 release > - > > Key: HBASE-26109 > URL: https://issues.apache.org/jira/browse/HBASE-26109 > Project: HBase > Issue Type: Task > Components: documentation >Reporter: Bharath Vissapragada >Assignee: Bharath Vissapragada >Priority: Major > Labels: release > Fix For: 1.7.1 > > > - Set branch-1 version to 1.7.2-SNAPSHOT > - Update SVN release bits, add download links to 1.7.1 > - Remove 1.7.0 from the mirrors > - Commit HBASE-26071 > in that order. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24984) WAL corruption due to early DBBs re-use when Durability.ASYNC_WAL is used with multi operation
[ https://issues.apache.org/jira/browse/HBASE-24984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17385011#comment-17385011 ] Gaurav Kanade commented on HBASE-24984: --- [~huaxiangsun] added PR for branch-2, [~anoop.hbase] will merge and cherry pick to branch-2.3 and branch-2.4 > WAL corruption due to early DBBs re-use when Durability.ASYNC_WAL is used > with multi operation > -- > > Key: HBASE-24984 > URL: https://issues.apache.org/jira/browse/HBASE-24984 > Project: HBase > Issue Type: Bug > Components: rpc, wal >Affects Versions: 2.1.6 >Reporter: Liu Junhong >Assignee: Gaurav Kanade >Priority: Critical > Fix For: 2.5.0, 2.3.6, 3.0.0-alpha-2, 2.4.5 > > Attachments: > 0001-HBASE-24984-WAL-corruption-due-to-early-DBBs-re-use-.patch > > > After bugfix HBASE-22539, When client use BufferedMutator or multiple > mutation , there will be one RpcCall and mutliple FSWALEntry . At the time > RpcCall finish and one FSWALEntry call release() , the remain FSWALEntries > may trigger RuntimeException or segmentation fault . > We should use RefCnt instead of AtomicInteger for > org.apache.hadoop.hbase.ipc.ServerCall.reference? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-26109) Wrap up 1.7.1 release
Bharath Vissapragada created HBASE-26109: Summary: Wrap up 1.7.1 release Key: HBASE-26109 URL: https://issues.apache.org/jira/browse/HBASE-26109 Project: HBase Issue Type: Task Components: documentation Reporter: Bharath Vissapragada Assignee: Bharath Vissapragada Fix For: 1.7.1 - Set branch-1 version to 1.7.2-SNAPSHOT - Update SVN release bits, add download links to 1.7.1 - Remove 1.7.0 from the mirrors - Commit HBASE-26071 in that order. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #3512: hbase24984 branch-2
Apache-HBase commented on pull request #3512: URL: https://github.com/apache/hbase/pull/3512#issuecomment-884343298 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 39s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +0 :ok: | shelldocs | 0m 0s | Shelldocs was not available. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ branch-2 Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 55s | branch-2 passed | | +1 :green_heart: | compile | 3m 20s | branch-2 passed | | +1 :green_heart: | checkstyle | 1m 5s | branch-2 passed | | +1 :green_heart: | spotbugs | 2m 11s | branch-2 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 23s | the patch passed | | +1 :green_heart: | compile | 3m 22s | the patch passed | | +1 :green_heart: | javac | 3m 22s | the patch passed | | +1 :green_heart: | checkstyle | 1m 6s | the patch passed | | +1 :green_heart: | shellcheck | 0m 0s | There were no new shellcheck issues. | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 12m 16s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. | | +1 :green_heart: | spotbugs | 2m 16s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 16s | The patch does not generate ASF License warnings. | | | | 41m 50s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3512/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3512 | | Optional Tests | dupname asflicense shellcheck shelldocs javac spotbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux 6041c0d73631 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / 253b83db57 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Max. process+thread count | 96 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3512/1/console | | versions | git=2.17.1 maven=3.6.3 shellcheck=0.4.6 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-24984) WAL corruption due to early DBBs re-use when Durability.ASYNC_WAL is used with multi operation
[ https://issues.apache.org/jira/browse/HBASE-24984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17384991#comment-17384991 ] Huaxiang Sun commented on HBASE-24984: -- [~gouravk], the fix is merged. Can you work on backports to branch-2's? Thanks. > WAL corruption due to early DBBs re-use when Durability.ASYNC_WAL is used > with multi operation > -- > > Key: HBASE-24984 > URL: https://issues.apache.org/jira/browse/HBASE-24984 > Project: HBase > Issue Type: Bug > Components: rpc, wal >Affects Versions: 2.1.6 >Reporter: Liu Junhong >Assignee: Gaurav Kanade >Priority: Critical > Fix For: 2.5.0, 2.3.6, 3.0.0-alpha-2, 2.4.5 > > Attachments: > 0001-HBASE-24984-WAL-corruption-due-to-early-DBBs-re-use-.patch > > > After bugfix HBASE-22539, When client use BufferedMutator or multiple > mutation , there will be one RpcCall and mutliple FSWALEntry . At the time > RpcCall finish and one FSWALEntry call release() , the remain FSWALEntries > may trigger RuntimeException or segmentation fault . > We should use RefCnt instead of AtomicInteger for > org.apache.hadoop.hbase.ipc.ServerCall.reference? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #3511: HBASE-26107 MOB compaction with missing files catches incorrect exception
Apache-HBase commented on pull request #3511: URL: https://github.com/apache/hbase/pull/3511#issuecomment-884333128 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 3m 59s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 19s | master passed | | +1 :green_heart: | compile | 3m 25s | master passed | | +1 :green_heart: | checkstyle | 1m 8s | master passed | | +1 :green_heart: | spotbugs | 2m 30s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 53s | the patch passed | | +1 :green_heart: | compile | 3m 26s | the patch passed | | +1 :green_heart: | javac | 3m 26s | the patch passed | | +1 :green_heart: | checkstyle | 1m 7s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 19m 23s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | spotbugs | 2m 28s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 16s | The patch does not generate ASF License warnings. | | | | 54m 9s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3511/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3511 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux 1f0caf9c1c8d 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 21c4578f22 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Max. process+thread count | 96 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3511/1/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] gkanade opened a new pull request #3512: hbase24984 branch-2
gkanade opened a new pull request #3512: URL: https://github.com/apache/hbase/pull/3512 Creating patch for branch-2, HBASE-24984 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-25343) Avoid the failed meta replica region temporarily in Load Balance mode
[ https://issues.apache.org/jira/browse/HBASE-25343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huaxiang Sun updated HBASE-25343: - Fix Version/s: (was: 2.4.6) > Avoid the failed meta replica region temporarily in Load Balance mode > - > > Key: HBASE-25343 > URL: https://issues.apache.org/jira/browse/HBASE-25343 > Project: HBase > Issue Type: Sub-task > Components: meta replicas >Affects Versions: 2.4.0 >Reporter: Huaxiang Sun >Assignee: Huaxiang Sun >Priority: Major > Fix For: 2.5.0, 3.0.0-alpha-2 > > > This is a follow-up enhancement with Stack, Duo. With the newly introduced > meta replica LoadBalance mode, if there is something wrong with one of meta > replica regions, the current logic is that it keeps trying until the meta > replica region is onlined again or it reports error, i.e, there is no HA at > LoadBalance mode. HA can be implemented if it reports timeout with one meta > replica region and tries another meta replica region. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #3504: HBASE-26093 Replication is stuck due to zero length wal file in oldWALs directory
Apache-HBase commented on pull request #3504: URL: https://github.com/apache/hbase/pull/3504#issuecomment-884306460 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 27s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 30s | master passed | | +1 :green_heart: | compile | 1m 11s | master passed | | +1 :green_heart: | shadedjars | 8m 19s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 44s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 16s | the patch passed | | +1 :green_heart: | compile | 1m 12s | the patch passed | | +1 :green_heart: | javac | 1m 12s | the patch passed | | +1 :green_heart: | shadedjars | 8m 27s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 15s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 146m 25s | hbase-server in the patch failed. | | | | 178m 59s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3504/4/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3504 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux e089d0be6bba 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 21c4578f22 | | Default Java | AdoptOpenJDK-11.0.10+9 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3504/4/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3504/4/testReport/ | | Max. process+thread count | 4441 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3504/4/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (HBASE-26108) add option to disable scanMetrics in TableSnapshotInputFormat
Huaxiang Sun created HBASE-26108: Summary: add option to disable scanMetrics in TableSnapshotInputFormat Key: HBASE-26108 URL: https://issues.apache.org/jira/browse/HBASE-26108 Project: HBase Issue Type: Improvement Affects Versions: 2.3.5 Reporter: Huaxiang Sun Assignee: Huaxiang Sun When running spark job with TableSnapshotInputFormat, we found that scan is very slower. We found that scanMetrics is hardcoded as enabled, spark's newAPIHadoopRDD uses DummyReporter in hadoop, which causes the following exception and 80% cpu time is spent on this exception handling. Need to provide an option to disable scanMetrics. java.base@11.0.5/java.lang.Throwable.fillInStackTrace(Native Method) java.base@11.0.5/java.lang.Throwable.fillInStackTrace(Throwable.java:787) => holding Monitor(java.util.MissingResourceException@258206255}) java.base@11.0.5/java.lang.Throwable.(Throwable.java:292) java.base@11.0.5/java.lang.Exception.(Exception.java:84) java.base@11.0.5/java.lang.RuntimeException.(RuntimeException.java:80) java.base@11.0.5/java.util.MissingResourceException.(MissingResourceException.java:85) java.base@11.0.5/java.util.ResourceBundle.throwMissingResourceException(ResourceBundle.java:2055) java.base@11.0.5/java.util.ResourceBundle.getBundleImpl(ResourceBundle.java:1689) java.base@11.0.5/java.util.ResourceBundle.getBundleImpl(ResourceBundle.java:1593) java.base@11.0.5/java.util.ResourceBundle.getBundle(ResourceBundle.java:1284) app//org.apache.hadoop.mapreduce.util.ResourceBundles.getBundle(ResourceBundles.java:37) app//org.apache.hadoop.mapreduce.util.ResourceBundles.getValue(ResourceBundles.java:56) => holding Monitor(java.lang.Class@545605549}) app//org.apache.hadoop.mapreduce.util.ResourceBundles.getCounterGroupName(ResourceBundles.java:77) app//org.apache.hadoop.mapreduce.counters.CounterGroupFactory.newGroup(CounterGroupFactory.java:94) app//org.apache.hadoop.mapreduce.counters.AbstractCounters.getGroup(AbstractCounters.java:227) app//org.apache.hadoop.mapreduce.counters.AbstractCounters.findCounter(AbstractCounters.java:154) app//org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl$DummyReporter.getCounter(TaskAttemptContextImpl.java:110) app//org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl.getCounter(TaskAttemptContextImpl.java:76) org.apache.hadoop.hbase.mapreduce.TableRecordReaderImpl.updateCounters(TableRecordReaderImpl.java:311) org.apache.hadoop.hbase.mapreduce.TableSnapshotInputFormat$TableSnapshotRegionRecordReader.nextKeyValue(TableSnapshotInputFormat.java:167) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] petersomogyi opened a new pull request #3511: HBASE-26107 MOB compaction with missing files catches incorrect exception
petersomogyi opened a new pull request #3511: URL: https://github.com/apache/hbase/pull/3511 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3488: HBASE-25393 Support split and merge region with direct insert into CF…
Apache-HBase commented on pull request #3488: URL: https://github.com/apache/hbase/pull/3488#issuecomment-884291192 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 9s | Docker mode activated. | | -0 :warning: | yetus | 0m 6s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ HBASE-24749 Compile Tests _ | | +0 :ok: | mvndep | 0m 28s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 36s | HBASE-24749 passed | | +1 :green_heart: | compile | 1m 50s | HBASE-24749 passed | | +1 :green_heart: | shadedjars | 8m 11s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 51s | HBASE-24749 passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 17s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 38s | the patch passed | | +1 :green_heart: | compile | 1m 47s | the patch passed | | +1 :green_heart: | javac | 1m 47s | the patch passed | | +1 :green_heart: | shadedjars | 8m 15s | patch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 37s | hbase-server generated 5 new + 20 unchanged - 0 fixed = 25 total (was 20) | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 47s | hbase-protocol-shaded in the patch passed. | | +1 :green_heart: | unit | 155m 51s | hbase-server in the patch passed. | | | | 189m 53s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3488/5/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3488 | | Optional Tests | unit javac javadoc shadedjars compile | | uname | Linux 79fb26aa9cb4 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | HBASE-24749 / 49b68b0e00 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3488/5/artifact/yetus-jdk8-hadoop3-check/output/diff-javadoc-javadoc-hbase-server.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3488/5/testReport/ | | Max. process+thread count | 4642 (vs. ulimit of 3) | | modules | C: hbase-protocol-shaded hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3488/5/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (HBASE-26107) MOB compaction with missing files catches incorrect exception
Peter Somogyi created HBASE-26107:
-
Summary: MOB compaction with missing files catches incorrect
exception
Key: HBASE-26107
URL: https://issues.apache.org/jira/browse/HBASE-26107
Project: HBase
Issue Type: Bug
Components: mob
Affects Versions: 3.0.0-alpha-1
Reporter: Peter Somogyi
Assignee: Peter Somogyi
The MOB compaction catches FileNotFoundException when
{{hbase.unsafe.mob.discard.miss}} is true to handle missing MOB cells. The FNFE
is wrapped in DoNotRetryIOException so the compaction fails for the given
region.
{noformat}
2021-07-21 13:51:05,880 WARN
org.apache.hadoop.hbase.mob.DefaultMobStoreCompactor:
hbase.unsafe.mob.discard.miss=true. This is unsafe setting recommended only
when first upgrading to a version with the distributed mob compaction feature
on a cluster that has experienced MOB data corruption.
2021-07-21 13:51:05,880 WARN
org.apache.hadoop.hbase.mob.DefaultMobStoreCompactor:
hbase.unsafe.mob.discard.miss=true. This is unsafe setting recommended only
when first upgrading to a version with the distributed mob compaction feature
on a cluster that has experienced MOB data corruption.
2021-07-21 13:51:05,880 INFO
org.apache.hadoop.hbase.mob.DefaultMobStoreCompactor: Compact MOB=true
optimized configured=false optimized enabled=false maximum MOB file
size=1073741824 major=true store=[table=IntegrationTestIngestWithMOB
family=test_cf region=3a2ee81f9244c39ba61d694e616c1a89]
2021-07-21 13:51:05,880 INFO
org.apache.hadoop.hbase.mob.DefaultMobStoreCompactor: Compact MOB=true
optimized configured=false optimized enabled=false maximum MOB file
size=1073741824 major=true store=[table=IntegrationTestIngestWithMOB
family=test_cf region=7a96f55bb9ae04500a06cbaef02da6a3]
2021-07-21 13:51:05,888 INFO
org.apache.hadoop.hbase.regionserver.RSRpcServices: Compacting
IntegrationTestIngestWithMOB,,1626787996628.c71cad04514b17ee86a407490bd27424.
2021-07-21 13:51:05,891 INFO
org.apache.hadoop.hbase.regionserver.RSRpcServices: Compacting
IntegrationTestIngestWithMOB,,1626787996628.8fd002bda07755decda67b7084d1e0f6.
2021-07-21 13:51:05,895 ERROR org.apache.hadoop.hbase.regionserver.HMobStore:
The mob file
1bbd886460827015e5d605ed44252251202107200e5065290b424e38992f5556d9943b6a_7a96f55bb9ae04500a06cbaef02da6a3
could not be found in the locations
[hdfs://example.com:8020/hbase/mobdir/data/default/IntegrationTestIngestWithMOB/e9b5d936e7f55a4f1c3246a8d5ce5
3c2/test_cf,
hdfs://example.com:8020/hbase/archive/data/default/IntegrationTestIngestWithMOB/e9b5d936e7f55a4f1c3246a8d5ce53c2/test_cf]
or it is corrupt
2021-07-21 13:51:05,895 INFO
org.apache.hadoop.hbase.regionserver.throttle.PressureAwareThroughputController:
7a96f55bb9ae04500a06cbaef02da6a3#test_cf#compaction#1 average throughput is
0.07 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active
operations remaining, total limit is 10.00 MB/second
2021-07-21 13:51:05,908 INFO
org.apache.hadoop.hbase.regionserver.RSRpcServices: Compacting
IntegrationTestIngestWithMOB,,1626787996628.53186ca5008e3a964eee5f96ee3f1b26.
2021-07-21 13:51:05,997 ERROR
org.apache.hadoop.hbase.regionserver.CompactSplit: Compaction failed
Request=regionName=IntegrationTestIngestWithMOB,,1626787996628.7a96f55bb9ae04500a06cbaef02da6a3.,
storeName=test_cf, fileCount=1, fileSize=110.6 M (110.6 M), priority=1,
time=1626875465819
java.io.IOException: Mob compaction failed for region:
7a96f55bb9ae04500a06cbaef02da6a3
at
org.apache.hadoop.hbase.mob.DefaultMobStoreCompactor.performCompaction(DefaultMobStoreCompactor.java:575)
at
org.apache.hadoop.hbase.regionserver.compactions.Compactor.compact(Compactor.java:327)
at
org.apache.hadoop.hbase.mob.DefaultMobStoreCompactor.compact(DefaultMobStoreCompactor.java:227)
at
org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.java:126)
at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1407)
at
org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:2183)
at
org.apache.hadoop.hbase.regionserver.CompactSplit$CompactionRunner.doCompaction(CompactSplit.java:633)
at
org.apache.hadoop.hbase.regionserver.CompactSplit$CompactionRunner.run(CompactSplit.java:675)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.hadoop.hbase.DoNotRetryIOException:
java.io.FileNotFoundException: File does not exist:
[jira] [Created] (HBASE-26106) AbstractFSWALProvider#getArchivedLogPath doesn't look for wal file in all oldWALs directory.
Rushabh Shah created HBASE-26106:
Summary: AbstractFSWALProvider#getArchivedLogPath doesn't look for
wal file in all oldWALs directory.
Key: HBASE-26106
URL: https://issues.apache.org/jira/browse/HBASE-26106
Project: HBase
Issue Type: Bug
Components: wal
Affects Versions: 2.4.4, 3.0.0-alpha-1, 2.5.0
Reporter: Rushabh Shah
Assignee: Rushabh Shah
Below is the code for AbstractFSWALProvider#getArchivedLogPath
{code:java}
public static Path getArchivedLogPath(Path path, Configuration conf) throws
IOException {
Path rootDir = CommonFSUtils.getWALRootDir(conf);
Path oldLogDir = new Path(rootDir, HConstants.HREGION_OLDLOGDIR_NAME);
if (conf.getBoolean(SEPARATE_OLDLOGDIR, DEFAULT_SEPARATE_OLDLOGDIR)) {
ServerName serverName = getServerNameFromWALDirectoryName(path);
if (serverName == null) {
LOG.error("Couldn't locate log: " + path);
return path;
}
oldLogDir = new Path(oldLogDir, serverName.getServerName());
}
Path archivedLogLocation = new Path(oldLogDir, path.getName());
final FileSystem fs = CommonFSUtils.getWALFileSystem(conf);
if (fs.exists(archivedLogLocation)) {
LOG.info("Log " + path + " was moved to " + archivedLogLocation);
return archivedLogLocation;
} else {
LOG.error("Couldn't locate log: " + path);
return path;
}
}
{code}
This method is called from the following places.
[AbstractFSWALProvider#openReader|https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/wal/AbstractFSWALProvider.java#L524]
[ReplicationSource#getFileSize|https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java#L399]
[WALInputFormat.WALRecordReader#nextKeyValue|https://github.com/apache/hbase/blob/master/hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/WALInputFormat.java#L220]
All of the above calls are trying to find the log in archive path after they
couldn't locate the wal in walsDir and they are not used for moving a log file
to archive directory.
But we will look for archive path within serverName directory only if conf key
is true.
Cc [~zhangduo]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] shahrs87 commented on a change in pull request #3504: HBASE-26093 Replication is stuck due to zero length wal file in oldWALs directory
shahrs87 commented on a change in pull request #3504:
URL: https://github.com/apache/hbase/pull/3504#discussion_r674044050
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/wal/AbstractFSWALProvider.java
##
@@ -500,6 +499,39 @@ public static Path getArchivedLogPath(Path path,
Configuration conf) throws IOEx
}
}
+ /**
+ * Get the archived WAL file path
+ * @param path - active WAL file path
+ * @param conf - configuration
+ * @return archived path if exists, path - otherwise
+ * @throws IOException exception
+ */
+ public static Path getArchivedLog(Path path, Configuration conf) throws
IOException {
Review comment:
> Suggest opening a new PR to address the incorrect usages of
getArchivedLogPath.
Created https://issues.apache.org/jira/browse/HBASE-26106 for this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Resolved] (HBASE-22184) [security] Support get|set LogLevel in HTTPS mode
[ https://issues.apache.org/jira/browse/HBASE-22184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang resolved HBASE-22184. --- Resolution: Fixed > [security] Support get|set LogLevel in HTTPS mode > - > > Key: HBASE-22184 > URL: https://issues.apache.org/jira/browse/HBASE-22184 > Project: HBase > Issue Type: New Feature > Components: logging, security, website >Reporter: Reid Chan >Assignee: Wei-Chiu Chuang >Priority: Major > Labels: security > Fix For: 2.3.0, 1.4.10, 3.0.0-alpha-1 > > Attachments: HBASE-22184.branch-1.001.patch, > HBASE-22184.branch-1.002.patch, HBASE-22184.master.001.patch, > HBASE-22184.master.002.patch, HBASE-22184.master.003.patch > > > As title read. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Reopened] (HBASE-22184) [security] Support get|set LogLevel in HTTPS mode
[ https://issues.apache.org/jira/browse/HBASE-22184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang reopened HBASE-22184: --- > [security] Support get|set LogLevel in HTTPS mode > - > > Key: HBASE-22184 > URL: https://issues.apache.org/jira/browse/HBASE-22184 > Project: HBase > Issue Type: New Feature > Components: logging, security, website >Reporter: Reid Chan >Assignee: Wei-Chiu Chuang >Priority: Major > Labels: security > Fix For: 3.0.0-alpha-1, 1.4.10, 2.3.0 > > Attachments: HBASE-22184.branch-1.001.patch, > HBASE-22184.branch-1.002.patch, HBASE-22184.master.001.patch, > HBASE-22184.master.002.patch, HBASE-22184.master.003.patch > > > As title read. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-25807) Move method reportProcedureDone from RegionServerStatus.proto to Master.proto
[ https://issues.apache.org/jira/browse/HBASE-25807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang resolved HBASE-25807. --- Fix Version/s: (was: 3.0.0-alpha-1) Resolution: Fixed > Move method reportProcedureDone from RegionServerStatus.proto to Master.proto > - > > Key: HBASE-25807 > URL: https://issues.apache.org/jira/browse/HBASE-25807 > Project: HBase > Issue Type: Sub-task >Reporter: Sun Xin >Assignee: Sun Xin >Priority: Major > > We next need use the procedure mechanism to implement enable/disable/refresh > peer, and ReplicationServer also needs reportProcedureDone to master, so I > hope to move method reportProcedureDone to Master.proto from > RegionServerStatus.proto. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Reopened] (HBASE-25807) Move method reportProcedureDone from RegionServerStatus.proto to Master.proto
[ https://issues.apache.org/jira/browse/HBASE-25807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang reopened HBASE-25807: --- > Move method reportProcedureDone from RegionServerStatus.proto to Master.proto > - > > Key: HBASE-25807 > URL: https://issues.apache.org/jira/browse/HBASE-25807 > Project: HBase > Issue Type: Sub-task >Reporter: Sun Xin >Assignee: Sun Xin >Priority: Major > Fix For: 3.0.0-alpha-1 > > > We next need use the procedure mechanism to implement enable/disable/refresh > peer, and ReplicationServer also needs reportProcedureDone to master, so I > hope to move method reportProcedureDone to Master.proto from > RegionServerStatus.proto. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Reopened] (HBASE-26025) Add a flag to mark if the IOError can be solved by retry in thrift IOError
[ https://issues.apache.org/jira/browse/HBASE-26025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang reopened HBASE-26025: --- > Add a flag to mark if the IOError can be solved by retry in thrift IOError > -- > > Key: HBASE-26025 > URL: https://issues.apache.org/jira/browse/HBASE-26025 > Project: HBase > Issue Type: Improvement > Components: Thrift >Reporter: Yutong Xiao >Assignee: Yutong Xiao >Priority: Major > Fix For: 3.0.0-alpha-1, 2.5.0, 2.3.6, 1.7.1, 2.4.5 > > > Currently, if an HBaseIOException occurs, the thrift client can only get the > error message. This is inconvenient for the client constructing a retry > mechanism to handle the exception. So I added a canRetry mark in IOError to > make the client side exception handling smarter. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-26025) Add a flag to mark if the IOError can be solved by retry in thrift IOError
[ https://issues.apache.org/jira/browse/HBASE-26025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang resolved HBASE-26025. --- Resolution: Fixed > Add a flag to mark if the IOError can be solved by retry in thrift IOError > -- > > Key: HBASE-26025 > URL: https://issues.apache.org/jira/browse/HBASE-26025 > Project: HBase > Issue Type: Improvement > Components: Thrift >Reporter: Yutong Xiao >Assignee: Yutong Xiao >Priority: Major > Fix For: 2.5.0, 2.3.6, 1.7.1, 2.4.5, 3.0.0-alpha-1 > > > Currently, if an HBaseIOException occurs, the thrift client can only get the > error message. This is inconvenient for the client constructing a retry > mechanism to handle the exception. So I added a canRetry mark in IOError to > make the client side exception handling smarter. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-20876) Improve docs style in HConstants
[
https://issues.apache.org/jira/browse/HBASE-20876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duo Zhang resolved HBASE-20876.
---
Resolution: Fixed
> Improve docs style in HConstants
>
>
> Key: HBASE-20876
> URL: https://issues.apache.org/jira/browse/HBASE-20876
> Project: HBase
> Issue Type: Improvement
>Reporter: Reid Chan
>Assignee: Wei-Chiu Chuang
>Priority: Minor
> Labels: beginner, beginners, newbie
> Fix For: 3.0.0-alpha-1
>
> Attachments: HBASE-20876.master.001.patch
>
>
> In {{HConstants}}, there's a docs snippet:
> {code}
> /** Don't use it! This'll get you the wrong path in a secure cluster.
> * Use FileSystem.getHomeDirectory() or
> * "/user/" + UserGroupInformation.getCurrentUser().getShortUserName() */
> {code}
> It's ugly style.
> Let's improve this docs with following
> {code}
> /**
> * Description
> */
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Reopened] (HBASE-20876) Improve docs style in HConstants
[
https://issues.apache.org/jira/browse/HBASE-20876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duo Zhang reopened HBASE-20876:
---
> Improve docs style in HConstants
>
>
> Key: HBASE-20876
> URL: https://issues.apache.org/jira/browse/HBASE-20876
> Project: HBase
> Issue Type: Improvement
>Reporter: Reid Chan
>Assignee: Wei-Chiu Chuang
>Priority: Minor
> Labels: beginner, beginners, newbie
> Fix For: 3.0.0-alpha-1
>
> Attachments: HBASE-20876.master.001.patch
>
>
> In {{HConstants}}, there's a docs snippet:
> {code}
> /** Don't use it! This'll get you the wrong path in a secure cluster.
> * Use FileSystem.getHomeDirectory() or
> * "/user/" + UserGroupInformation.getCurrentUser().getShortUserName() */
> {code}
> It's ugly style.
> Let's improve this docs with following
> {code}
> /**
> * Description
> */
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (HBASE-21404) Master/RS navbar active state does not work
[ https://issues.apache.org/jira/browse/HBASE-21404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang resolved HBASE-21404. --- Resolution: Fixed > Master/RS navbar active state does not work > --- > > Key: HBASE-21404 > URL: https://issues.apache.org/jira/browse/HBASE-21404 > Project: HBase > Issue Type: Bug > Components: UI >Affects Versions: 3.0.0-alpha-1, 2.0.0 >Reporter: Nihal Jain >Assignee: Nihal Jain >Priority: Minor > Fix For: 3.0.0-alpha-1 > > Attachments: HBASE-21404.master.001.patch, > HBASE-21404.master.001.patch, HBASE-21404.master.002.patch, > master_after_patch.png, master_before.png, rs_after_patch.png, rs_before.png > > > In master/rs web UI, the current active tab is not updated when user switches > to any tab other than "Home" tab. > For example: even though say if we are on "tabledetailed.jsp", the navbar > does not update the active state of that tab. See master_before.png -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Reopened] (HBASE-21404) Master/RS navbar active state does not work
[ https://issues.apache.org/jira/browse/HBASE-21404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang reopened HBASE-21404: --- > Master/RS navbar active state does not work > --- > > Key: HBASE-21404 > URL: https://issues.apache.org/jira/browse/HBASE-21404 > Project: HBase > Issue Type: Bug > Components: UI >Affects Versions: 3.0.0-alpha-1, 2.0.0 >Reporter: Nihal Jain >Assignee: Nihal Jain >Priority: Minor > Fix For: 3.0.0-alpha-1 > > Attachments: HBASE-21404.master.001.patch, > HBASE-21404.master.001.patch, HBASE-21404.master.002.patch, > master_after_patch.png, master_before.png, rs_after_patch.png, rs_before.png > > > In master/rs web UI, the current active tab is not updated when user switches > to any tab other than "Home" tab. > For example: even though say if we are on "tabledetailed.jsp", the navbar > does not update the active state of that tab. See master_before.png -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] shahrs87 commented on a change in pull request #3504: HBASE-26093 Replication is stuck due to zero length wal file in oldWALs directory
shahrs87 commented on a change in pull request #3504:
URL: https://github.com/apache/hbase/pull/3504#discussion_r674010737
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/wal/AbstractFSWALProvider.java
##
@@ -500,6 +499,39 @@ public static Path getArchivedLogPath(Path path,
Configuration conf) throws IOEx
}
}
+ /**
+ * Get the archived WAL file path
+ * @param path - active WAL file path
+ * @param conf - configuration
+ * @return archived path if exists, path - otherwise
+ * @throws IOException exception
+ */
+ public static Path getArchivedLog(Path path, Configuration conf) throws
IOException {
Review comment:
Cool. Thank you !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3504: HBASE-26093 Replication is stuck due to zero length wal file in oldWALs directory
Apache-HBase commented on pull request #3504: URL: https://github.com/apache/hbase/pull/3504#issuecomment-884219192 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 57s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 5m 14s | master passed | | +1 :green_heart: | compile | 4m 17s | master passed | | +1 :green_heart: | checkstyle | 1m 23s | master passed | | +1 :green_heart: | spotbugs | 2m 32s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 48s | the patch passed | | +1 :green_heart: | compile | 4m 13s | the patch passed | | +1 :green_heart: | javac | 4m 13s | the patch passed | | -0 :warning: | checkstyle | 1m 16s | hbase-server: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 23m 35s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | spotbugs | 3m 39s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 16s | The patch does not generate ASF License warnings. | | | | 63m 21s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3504/4/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3504 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux e95a5884eb72 4.15.0-142-generic #146-Ubuntu SMP Tue Apr 13 01:11:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 21c4578f22 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3504/4/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | Max. process+thread count | 86 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3504/4/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache9 commented on a change in pull request #3504: HBASE-26093 Replication is stuck due to zero length wal file in oldWALs directory
Apache9 commented on a change in pull request #3504:
URL: https://github.com/apache/hbase/pull/3504#discussion_r673995889
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/wal/AbstractFSWALProvider.java
##
@@ -500,6 +499,39 @@ public static Path getArchivedLogPath(Path path,
Configuration conf) throws IOEx
}
}
+ /**
+ * Get the archived WAL file path
+ * @param path - active WAL file path
+ * @param conf - configuration
+ * @return archived path if exists, path - otherwise
+ * @throws IOException exception
+ */
+ public static Path getArchivedLog(Path path, Configuration conf) throws
IOException {
Review comment:
Good. I think you have found potential bugs in our code base. We should
change them all to make use of the new method you introduced in this PR.
Suggest opening a new PR to address the incorrect usages of getArchivedLogPath.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (HBASE-26099) Add 3.0.0-alpha-1 to download page
[
https://issues.apache.org/jira/browse/HBASE-26099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17384911#comment-17384911
]
Hudson commented on HBASE-26099:
Results for branch master
[build #348 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/348/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/348/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/348/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/348/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Add 3.0.0-alpha-1 to download page
> --
>
> Key: HBASE-26099
> URL: https://issues.apache.org/jira/browse/HBASE-26099
> Project: HBase
> Issue Type: Sub-task
> Components: website
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 3.0.0-alpha-2
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-26100) Set version as 3.0.0-alpha-2-SNAPSHOT in master
[
https://issues.apache.org/jira/browse/HBASE-26100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17384912#comment-17384912
]
Hudson commented on HBASE-26100:
Results for branch master
[build #348 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/348/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/348/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/348/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/348/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Set version as 3.0.0-alpha-2-SNAPSHOT in master
> ---
>
> Key: HBASE-26100
> URL: https://issues.apache.org/jira/browse/HBASE-26100
> Project: HBase
> Issue Type: Sub-task
> Components: build, pom
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 3.0.0-alpha-2
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] shahrs87 commented on a change in pull request #3504: HBASE-26093 Replication is stuck due to zero length wal file in oldWALs directory
shahrs87 commented on a change in pull request #3504:
URL: https://github.com/apache/hbase/pull/3504#discussion_r673983493
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/wal/AbstractFSWALProvider.java
##
@@ -500,6 +499,39 @@ public static Path getArchivedLogPath(Path path,
Configuration conf) throws IOEx
}
}
+ /**
+ * Get the archived WAL file path
+ * @param path - active WAL file path
+ * @param conf - configuration
+ * @return archived path if exists, path - otherwise
+ * @throws IOException exception
+ */
+ public static Path getArchivedLog(Path path, Configuration conf) throws
IOException {
Review comment:
> The above getArchivedLogPath is used when we want to move a file into
the archived directory, we need to check the configuration to see if we want to
put the log into the separate directory or not.
In production code, these 3 places are where we are calling
`getArchivedLogPath` util method.
1.
[AbstractFSWALProvider#openReader](https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/wal/AbstractFSWALProvider.java#L524)
2.
[ReplicationSource#getFileSize](https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java#L399)
3.
[WALInputFormat.WALRecordReader#nextKeyValue](https://github.com/apache/hbase/blob/master/hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/WALInputFormat.java#L220)
@Apache9 All of the above calls are trying to find the log in archive path
after they couldn't locate the wal in walsDir and they are not used for moving
a log file to archive directory. Correct me if I am wrong.
[AbstractFSWALProvider#getWALArchiveDirectoryName](https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/wal/AbstractFSWALProvider.java#L334)
is the mehod we use to locate the wal archive directory to move the wal.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3488: HBASE-25393 Support split and merge region with direct insert into CF…
Apache-HBase commented on pull request #3488: URL: https://github.com/apache/hbase/pull/3488#issuecomment-884198411 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 31s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +0 :ok: | prototool | 0m 0s | prototool was not available. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ HBASE-24749 Compile Tests _ | | +0 :ok: | mvndep | 0m 33s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 58s | HBASE-24749 passed | | +1 :green_heart: | compile | 5m 1s | HBASE-24749 passed | | +1 :green_heart: | checkstyle | 1m 20s | HBASE-24749 passed | | +1 :green_heart: | spotbugs | 6m 33s | HBASE-24749 passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 9s | the patch passed | | -1 :x: | compile | 1m 51s | hbase-server in the patch failed. | | -0 :warning: | cc | 1m 51s | hbase-server in the patch failed. | | -0 :warning: | javac | 1m 51s | hbase-server in the patch failed. | | -0 :warning: | checkstyle | 1m 19s | hbase-server: The patch generated 30 new + 40 unchanged - 3 fixed = 70 total (was 43) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 20m 23s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | hbaseprotoc | 1m 59s | the patch passed | | -1 :x: | spotbugs | 2m 51s | hbase-server generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 24s | The patch does not generate ASF License warnings. | | | | 67m 28s | | | Reason | Tests | |---:|:--| | FindBugs | module:hbase-server | | | org.apache.hadoop.hbase.regionserver.DirectStoreFSWriteStrategy.commitDaughterRegion(RegionInfo) makes inefficient use of keySet iterator instead of entrySet iterator At DirectStoreFSWriteStrategy.java:keySet iterator instead of entrySet iterator At DirectStoreFSWriteStrategy.java:[line 234] | | | org.apache.hadoop.hbase.regionserver.DirectStoreFSWriteStrategy.commitMergedRegion(RegionInfo) makes inefficient use of keySet iterator instead of entrySet iterator At DirectStoreFSWriteStrategy.java:keySet iterator instead of entrySet iterator At DirectStoreFSWriteStrategy.java:[line 255] | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3488/5/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3488 | | Optional Tests | dupname asflicense cc hbaseprotoc prototool javac spotbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux ea97ea83e84b 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | HBASE-24749 / 49b68b0e00 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | compile | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3488/5/artifact/yetus-general-check/output/patch-compile-hbase-server.txt | | cc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3488/5/artifact/yetus-general-check/output/patch-compile-hbase-server.txt | | javac | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3488/5/artifact/yetus-general-check/output/patch-compile-hbase-server.txt | | checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3488/5/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | spotbugs | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3488/5/artifact/yetus-general-check/output/new-spotbugs-hbase-server.html | | Max. process+thread count | 96 (vs. ulimit of 3) | | modules | C: hbase-protocol-shaded hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3488/5/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to
[jira] [Updated] (HBASE-26077) Add heartbeat for ReplicationServer
[ https://issues.apache.org/jira/browse/HBASE-26077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-26077: -- Fix Version/s: (was: 3.0.0-alpha-1) > Add heartbeat for ReplicationServer > --- > > Key: HBASE-26077 > URL: https://issues.apache.org/jira/browse/HBASE-26077 > Project: HBase > Issue Type: Sub-task >Affects Versions: 3.0.0-alpha-1 >Reporter: Sun Xin >Assignee: Sun Xin >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-26084) Add owner of replication queue for ReplicationQueueInfo
[ https://issues.apache.org/jira/browse/HBASE-26084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-26084: -- Fix Version/s: (was: 3.0.0-alpha-1) > Add owner of replication queue for ReplicationQueueInfo > --- > > Key: HBASE-26084 > URL: https://issues.apache.org/jira/browse/HBASE-26084 > Project: HBase > Issue Type: Sub-task >Affects Versions: 3.0.0-alpha-1 >Reporter: Sun Xin >Assignee: Sun Xin >Priority: Major > > The current ReplicationQueueInfo only has queueId, which is not enough to > distinguish queues in ReplicationServer, so we need to add the RS holding > the queue for ReplicationQueueInfo. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-26078) Dispatch replication sources to ReplicationServer
[ https://issues.apache.org/jira/browse/HBASE-26078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-26078: -- Fix Version/s: (was: 3.0.0-alpha-1) > Dispatch replication sources to ReplicationServer > - > > Key: HBASE-26078 > URL: https://issues.apache.org/jira/browse/HBASE-26078 > Project: HBase > Issue Type: Sub-task > Components: Replication >Affects Versions: 3.0.0-alpha-1 >Reporter: Sun Xin >Assignee: Sun Xin >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache9 commented on a change in pull request #3504: HBASE-26093 Replication is stuck due to zero length wal file in oldWALs directory
Apache9 commented on a change in pull request #3504:
URL: https://github.com/apache/hbase/pull/3504#discussion_r673963594
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceWALReader.java
##
@@ -274,11 +275,14 @@ private boolean handleEofException(Exception e,
WALEntryBatch batch) {
// since we don't add current log to recovered source queue so it is safe
to remove.
if ((e instanceof EOFException || e.getCause() instanceof EOFException) &&
(source.isRecovered() || queue.size() > 1) && this.eofAutoRecovery) {
- Path head = queue.peek();
+ Path path = queue.peek();
try {
-if (fs.getFileStatus(head).getLen() == 0) {
- // head of the queue is an empty log file
- LOG.warn("Forcing removal of 0 length log in queue: {}", head);
+if (!fs.exists(path)) {
+ // There is a chance that wal has moved to oldWALs directory, so
look there also.
+ path = AbstractFSWALProvider.getArchivedLogPath(path, conf);
Review comment:
Oh, just saw this comment. Please see my new comment below, you need to
keep getArchivedLogPath, as it is used when we want to archive a wal file.
It is OK to remove the getArchivedLog in WALEntryStream.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] Apache9 commented on a change in pull request #3504: HBASE-26093 Replication is stuck due to zero length wal file in oldWALs directory
Apache9 commented on a change in pull request #3504:
URL: https://github.com/apache/hbase/pull/3504#discussion_r673961819
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/wal/AbstractFSWALProvider.java
##
@@ -500,6 +499,39 @@ public static Path getArchivedLogPath(Path path,
Configuration conf) throws IOEx
}
}
+ /**
+ * Get the archived WAL file path
+ * @param path - active WAL file path
+ * @param conf - configuration
+ * @return archived path if exists, path - otherwise
+ * @throws IOException exception
+ */
+ public static Path getArchivedLog(Path path, Configuration conf) throws
IOException {
Review comment:
Better name it findArchivedLog.
The above getArchivedLogPath is used when we want to move a file into the
archived directory, we need to check the configuration to see if we want to put
the log into the separate directory or not.
And for this method, it is used when we want to read a wal file, so we need
to try all the places to see if we can find it, as we may change the way on how
to archive the log on the fly(not sure but in general we want to support
onConfigurationChange for more and more configurations).
So name it findArchivedLog will be better, and maybe we could just return
null, or return an Optional object when we can not find the wal file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
