[jira] [Comment Edited] (HBASE-25701) RegionServer JVM crash when append wal entry
[ https://issues.apache.org/jira/browse/HBASE-25701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17312243#comment-17312243 ] Juanjuan Tian edited comment on HBASE-25701 at 3/31/21, 9:50 AM: -- [~anoop.hbase] I don't think they are same case, since I'm using habse 2.2.6 version which is already with fix from HBASE-22539, but issue still happen when append wal Entry was (Author: jutia): [~anoop.hbase] I don't think they are same case, since I'm using habse 2.2.6 version which is already with fix from HBASE-22539, but issue still happen when appendEntry > RegionServer JVM crash when append wal entry > > > Key: HBASE-25701 > URL: https://issues.apache.org/jira/browse/HBASE-25701 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.2.6 >Reporter: Juanjuan Tian >Priority: Major > > Region Server JVM crash when append wal entry, JVM crash log: > # > # A fatal error has been detected by the Java Runtime Environment: > # > # EXCEPTION_ACCESS_VIOLATION (0xc005) at pc=0x027af93f, > pid=17992, tid=0x2d54 > # > # JRE version: Java(TM) SE Runtime Environment (8.0_202-b08) (build > 1.8.0_202-b08) > # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.202-b08 mixed mode > windows-amd64 compressed oops) > # Problematic frame: > # J 10214 C2 org.apache.hadoop.hbase.ByteBufferKeyValue.getFamilyLength()B (9 > bytes) @ 0x027af93f [0x027af860+0xdf] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25701) RegionServer JVM crash when append wal entry
[ https://issues.apache.org/jira/browse/HBASE-25701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17312243#comment-17312243 ] Juanjuan Tian commented on HBASE-25701: [~anoop.hbase] I don't think they are same case, since I'm using habse 2.2.6 version which is already with fix from HBASE-22539, but issue still happen when appendEntry > RegionServer JVM crash when append wal entry > > > Key: HBASE-25701 > URL: https://issues.apache.org/jira/browse/HBASE-25701 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.2.6 >Reporter: Juanjuan Tian >Priority: Major > > Region Server JVM crash when append wal entry, JVM crash log: > # > # A fatal error has been detected by the Java Runtime Environment: > # > # EXCEPTION_ACCESS_VIOLATION (0xc005) at pc=0x027af93f, > pid=17992, tid=0x2d54 > # > # JRE version: Java(TM) SE Runtime Environment (8.0_202-b08) (build > 1.8.0_202-b08) > # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.202-b08 mixed mode > windows-amd64 compressed oops) > # Problematic frame: > # J 10214 C2 org.apache.hadoop.hbase.ByteBufferKeyValue.getFamilyLength()B (9 > bytes) @ 0x027af93f [0x027af860+0xdf] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HBASE-25701) RegionServer JVM crash when append wal entry
[ https://issues.apache.org/jira/browse/HBASE-25701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17311591#comment-17311591 ] Juanjuan Tian edited comment on HBASE-25701 at 3/30/21, 4:00 PM: -- [~anoop.hbase] do you mean fixed in Jira https://issues.apache.org/jira/browse/HBASE-22539, I don't think they are same issue, since I'm using habse 2.2.6 with fix from HBASE-22539, but issue still happen when appendEntry, not when replay wal. if you are not meaning HBASE-22539, could you help provide the Jira id? was (Author: jutia): [~anoop.hbase] do you mean fixed in Jira https://issues.apache.org/jira/browse/HBASE-22539, I don't think they are same isssue, since I'm using habse 2.2.6 with fix from HBASE-22539, but issue still happen when appendEntry, not replay. if you are not meaning HBASE-22539, could you help provide the Jira id? > RegionServer JVM crash when append wal entry > > > Key: HBASE-25701 > URL: https://issues.apache.org/jira/browse/HBASE-25701 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.2.6 >Reporter: Juanjuan Tian >Priority: Major > > Region Server JVM crash when append wal entry, JVM crash log: > # > # A fatal error has been detected by the Java Runtime Environment: > # > # EXCEPTION_ACCESS_VIOLATION (0xc005) at pc=0x027af93f, > pid=17992, tid=0x2d54 > # > # JRE version: Java(TM) SE Runtime Environment (8.0_202-b08) (build > 1.8.0_202-b08) > # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.202-b08 mixed mode > windows-amd64 compressed oops) > # Problematic frame: > # J 10214 C2 org.apache.hadoop.hbase.ByteBufferKeyValue.getFamilyLength()B (9 > bytes) @ 0x027af93f [0x027af860+0xdf] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25701) RegionServer JVM crash when append wal entry
[ https://issues.apache.org/jira/browse/HBASE-25701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17311591#comment-17311591 ] Juanjuan Tian commented on HBASE-25701: [~anoop.hbase] do you mean fixed in Jira https://issues.apache.org/jira/browse/HBASE-22539, I don't think they are same isssue, since I'm using habse 2.2.6 with fix from HBASE-22539, but issue still happen when appendEntry, not replay. if you are not meaning HBASE-22539, could you help provide the Jira id? > RegionServer JVM crash when append wal entry > > > Key: HBASE-25701 > URL: https://issues.apache.org/jira/browse/HBASE-25701 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.2.6 >Reporter: Juanjuan Tian >Priority: Major > > Region Server JVM crash when append wal entry, JVM crash log: > # > # A fatal error has been detected by the Java Runtime Environment: > # > # EXCEPTION_ACCESS_VIOLATION (0xc005) at pc=0x027af93f, > pid=17992, tid=0x2d54 > # > # JRE version: Java(TM) SE Runtime Environment (8.0_202-b08) (build > 1.8.0_202-b08) > # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.202-b08 mixed mode > windows-amd64 compressed oops) > # Problematic frame: > # J 10214 C2 org.apache.hadoop.hbase.ByteBufferKeyValue.getFamilyLength()B (9 > bytes) @ 0x027af93f [0x027af860+0xdf] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24984) WAL corruption due to early DBBs re-use when Durability.ASYNC_WAL is used with multi operation
[ https://issues.apache.org/jira/browse/HBASE-24984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17309887#comment-17309887 ] Juanjuan Tian commented on HBASE-24984: Do you still observe this issue? any fix available? > WAL corruption due to early DBBs re-use when Durability.ASYNC_WAL is used > with multi operation > -- > > Key: HBASE-24984 > URL: https://issues.apache.org/jira/browse/HBASE-24984 > Project: HBase > Issue Type: Bug > Components: rpc, wal >Affects Versions: 2.1.6 >Reporter: Liu Junhong >Priority: Major > > After bugfix HBASE-22539, When client use BufferedMutator or multiple > mutation , there will be one RpcCall and mutliple FSWALEntry . At the time > RpcCall finish and one FSWALEntry call release() , the remain FSWALEntries > may trigger RuntimeException or segmentation fault . > We should use RefCnt instead of AtomicInteger for > org.apache.hadoop.hbase.ipc.ServerCall.reference? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HBASE-22539) WAL corruption due to early DBBs re-use when Durability.ASYNC_WAL is used
[ https://issues.apache.org/jira/browse/HBASE-22539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17309885#comment-17309885 ] Juanjuan Tian edited comment on HBASE-22539 at 3/27/21, 7:02 AM: -- encountered silimar issue, but happen when append wal entry, file a new issue here, like the issue as [~mopishv0] descripbe, not sure if any fix ready HBASE-25701 RegionServer JVM crash when append wal entry - ASF JIRA (apache.org) was (Author: jutia): encountered silimar issue, but happen when append wal, file a new issue here, like the issue as [~mopishv0] descripbe, not sure if any fix ready HBASE-25701 RegionServer JVM crash when append wal entry - ASF JIRA (apache.org) > WAL corruption due to early DBBs re-use when Durability.ASYNC_WAL is used > - > > Key: HBASE-22539 > URL: https://issues.apache.org/jira/browse/HBASE-22539 > Project: HBase > Issue Type: Bug > Components: rpc, wal >Affects Versions: 2.2.0, 2.0.5, 2.1.5 >Reporter: Wellington Chevreuil >Assignee: Duo Zhang >Priority: Blocker > Fix For: 3.0.0-alpha-1, 2.3.0, 2.0.6, 2.2.1, 2.1.6 > > Attachments: HBASE-22539-UT.patch, HBASE-22539.branch-2.001.patch > > > Summary > We had been chasing a WAL corruption issue reported on one of our customers > deployments running release 2.1.1 (CDH 6.1.0). After providing a custom > modified jar with the extra sanity checks implemented by HBASE-21401 applied > on some code points, plus additional debugging messages, we believe it is > related to DirectByteBuffer usage, and Unsafe copy from offheap memory to > on-heap array triggered > [here|https://github.com/apache/hbase/blob/branch-2.1/hbase-common/src/main/java/org/apache/hadoop/hbase/util/ByteBufferUtils.java#L1157], > such as when writing into a non ByteBufferWriter type, as done > [here|https://github.com/apache/hbase/blob/branch-2.1/hbase-common/src/main/java/org/apache/hadoop/hbase/io/ByteBufferWriterOutputStream.java#L84]. > More details on the following comment. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24984) WAL corruption due to early DBBs re-use when Durability.ASYNC_WAL is used with multi operation
[ https://issues.apache.org/jira/browse/HBASE-24984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17309886#comment-17309886 ] Juanjuan Tian commented on HBASE-24984: [~mopishv0] any fix available? > WAL corruption due to early DBBs re-use when Durability.ASYNC_WAL is used > with multi operation > -- > > Key: HBASE-24984 > URL: https://issues.apache.org/jira/browse/HBASE-24984 > Project: HBase > Issue Type: Bug > Components: rpc, wal >Affects Versions: 2.1.6 >Reporter: Liu Junhong >Priority: Major > > After bugfix HBASE-22539, When client use BufferedMutator or multiple > mutation , there will be one RpcCall and mutliple FSWALEntry . At the time > RpcCall finish and one FSWALEntry call release() , the remain FSWALEntries > may trigger RuntimeException or segmentation fault . > We should use RefCnt instead of AtomicInteger for > org.apache.hadoop.hbase.ipc.ServerCall.reference? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HBASE-22539) WAL corruption due to early DBBs re-use when Durability.ASYNC_WAL is used
[ https://issues.apache.org/jira/browse/HBASE-22539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17309885#comment-17309885 ] Juanjuan Tian edited comment on HBASE-22539 at 3/27/21, 6:52 AM: -- encountered silimar issue, but happen when append wal, file a new issue here, like the issue as [~mopishv0] descripbe, not sure if any fix ready HBASE-25701 RegionServer JVM crash when append wal entry - ASF JIRA (apache.org) was (Author: jutia): encountered silimar issue, but happen when append wal, file a new issue here, like the issue as [~zhangduo] descripbe, not sure if any fix ready [HBASE-25701] RegionServer JVM crash when append wal entry - ASF JIRA (apache.org) > WAL corruption due to early DBBs re-use when Durability.ASYNC_WAL is used > - > > Key: HBASE-22539 > URL: https://issues.apache.org/jira/browse/HBASE-22539 > Project: HBase > Issue Type: Bug > Components: rpc, wal >Affects Versions: 2.2.0, 2.0.5, 2.1.5 >Reporter: Wellington Chevreuil >Assignee: Duo Zhang >Priority: Blocker > Fix For: 3.0.0-alpha-1, 2.3.0, 2.0.6, 2.2.1, 2.1.6 > > Attachments: HBASE-22539-UT.patch, HBASE-22539.branch-2.001.patch > > > Summary > We had been chasing a WAL corruption issue reported on one of our customers > deployments running release 2.1.1 (CDH 6.1.0). After providing a custom > modified jar with the extra sanity checks implemented by HBASE-21401 applied > on some code points, plus additional debugging messages, we believe it is > related to DirectByteBuffer usage, and Unsafe copy from offheap memory to > on-heap array triggered > [here|https://github.com/apache/hbase/blob/branch-2.1/hbase-common/src/main/java/org/apache/hadoop/hbase/util/ByteBufferUtils.java#L1157], > such as when writing into a non ByteBufferWriter type, as done > [here|https://github.com/apache/hbase/blob/branch-2.1/hbase-common/src/main/java/org/apache/hadoop/hbase/io/ByteBufferWriterOutputStream.java#L84]. > More details on the following comment. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-22539) WAL corruption due to early DBBs re-use when Durability.ASYNC_WAL is used
[ https://issues.apache.org/jira/browse/HBASE-22539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17309885#comment-17309885 ] Juanjuan Tian commented on HBASE-22539: encountered silimar issue, but happen when append wal, file a new issue here, like the issue as [~zhangduo] descripbe, not sure if any fix ready [HBASE-25701] RegionServer JVM crash when append wal entry - ASF JIRA (apache.org) > WAL corruption due to early DBBs re-use when Durability.ASYNC_WAL is used > - > > Key: HBASE-22539 > URL: https://issues.apache.org/jira/browse/HBASE-22539 > Project: HBase > Issue Type: Bug > Components: rpc, wal >Affects Versions: 2.2.0, 2.0.5, 2.1.5 >Reporter: Wellington Chevreuil >Assignee: Duo Zhang >Priority: Blocker > Fix For: 3.0.0-alpha-1, 2.3.0, 2.0.6, 2.2.1, 2.1.6 > > Attachments: HBASE-22539-UT.patch, HBASE-22539.branch-2.001.patch > > > Summary > We had been chasing a WAL corruption issue reported on one of our customers > deployments running release 2.1.1 (CDH 6.1.0). After providing a custom > modified jar with the extra sanity checks implemented by HBASE-21401 applied > on some code points, plus additional debugging messages, we believe it is > related to DirectByteBuffer usage, and Unsafe copy from offheap memory to > on-heap array triggered > [here|https://github.com/apache/hbase/blob/branch-2.1/hbase-common/src/main/java/org/apache/hadoop/hbase/util/ByteBufferUtils.java#L1157], > such as when writing into a non ByteBufferWriter type, as done > [here|https://github.com/apache/hbase/blob/branch-2.1/hbase-common/src/main/java/org/apache/hadoop/hbase/io/ByteBufferWriterOutputStream.java#L84]. > More details on the following comment. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HBASE-25701) RegionServer JVM crash when append wal entry
[ https://issues.apache.org/jira/browse/HBASE-25701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17309857#comment-17309857 ] Juanjuan Tian edited comment on HBASE-25701 at 3/27/21, 4:06 AM: -- [~wchevreuil] [~zhangduo] [~openinx] any idea why issue happen? ASYNC_WAL' is used in this case was (Author: jutia): [~wchevreuil] [~zhangduo] [~openinx] any idea why issue happen? > RegionServer JVM crash when append wal entry > > > Key: HBASE-25701 > URL: https://issues.apache.org/jira/browse/HBASE-25701 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.2.6 >Reporter: Juanjuan Tian >Priority: Major > > Region Server JVM crash when append wal entry, JVM crash log: > # > # A fatal error has been detected by the Java Runtime Environment: > # > # EXCEPTION_ACCESS_VIOLATION (0xc005) at pc=0x027af93f, > pid=17992, tid=0x2d54 > # > # JRE version: Java(TM) SE Runtime Environment (8.0_202-b08) (build > 1.8.0_202-b08) > # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.202-b08 mixed mode > windows-amd64 compressed oops) > # Problematic frame: > # J 10214 C2 org.apache.hadoop.hbase.ByteBufferKeyValue.getFamilyLength()B (9 > bytes) @ 0x027af93f [0x027af860+0xdf] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25701) RegionServer JVM crash when append wal entry
[ https://issues.apache.org/jira/browse/HBASE-25701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17309859#comment-17309859 ] Juanjuan Tian commented on HBASE-25701: seems bytebuffer is relased before append wal entry > RegionServer JVM crash when append wal entry > > > Key: HBASE-25701 > URL: https://issues.apache.org/jira/browse/HBASE-25701 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.2.6 >Reporter: Juanjuan Tian >Priority: Major > > Region Server JVM crash when append wal entry, JVM crash log: > # > # A fatal error has been detected by the Java Runtime Environment: > # > # EXCEPTION_ACCESS_VIOLATION (0xc005) at pc=0x027af93f, > pid=17992, tid=0x2d54 > # > # JRE version: Java(TM) SE Runtime Environment (8.0_202-b08) (build > 1.8.0_202-b08) > # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.202-b08 mixed mode > windows-amd64 compressed oops) > # Problematic frame: > # J 10214 C2 org.apache.hadoop.hbase.ByteBufferKeyValue.getFamilyLength()B (9 > bytes) @ 0x027af93f [0x027af860+0xdf] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25701) RegionServer JVM crash when append wal entry
[ https://issues.apache.org/jira/browse/HBASE-25701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17309857#comment-17309857 ] Juanjuan Tian commented on HBASE-25701: [~wchevreuil] [~zhangduo] [~openinx] any idea why issue happen? > RegionServer JVM crash when append wal entry > > > Key: HBASE-25701 > URL: https://issues.apache.org/jira/browse/HBASE-25701 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.2.6 >Reporter: Juanjuan Tian >Priority: Major > > Region Server JVM crash when append wal entry, JVM crash log: > # > # A fatal error has been detected by the Java Runtime Environment: > # > # EXCEPTION_ACCESS_VIOLATION (0xc005) at pc=0x027af93f, > pid=17992, tid=0x2d54 > # > # JRE version: Java(TM) SE Runtime Environment (8.0_202-b08) (build > 1.8.0_202-b08) > # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.202-b08 mixed mode > windows-amd64 compressed oops) > # Problematic frame: > # J 10214 C2 org.apache.hadoop.hbase.ByteBufferKeyValue.getFamilyLength()B (9 > bytes) @ 0x027af93f [0x027af860+0xdf] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HBASE-25701) RegionServer JVM crash when append wal entry
[ https://issues.apache.org/jira/browse/HBASE-25701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17309856#comment-17309856 ] Juanjuan Tian edited comment on HBASE-25701 at 3/27/21, 4:04 AM: -- sometimes JVM crash when append wal entry, I add some callstack logs, at org.apache.hadoop.hbase.util.ByteBufferUtils.toByte(ByteBufferUtils.java:418) at org.apache.hadoop.hbase.ByteBufferKeyValue.getFamilyLength(ByteBufferKeyValue.java:120) at org.apache.hadoop.hbase.ByteBufferKeyValue.getFamilyLength(ByteBufferKeyValue.java:102) at org.apache.hadoop.hbase.PrivateCellUtil.matchingFamily(PrivateCellUtil.java:731) at org.apache.hadoop.hbase.CellUtil.matchingFamily(CellUtil.java:816) at org.apache.hadoop.hbase.wal.WALEdit.isMetaEditFamily(WALEdit.java:209) at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALActionListener.lambda$scopeWALEdits$0(ReplicationSourceWALActionListener.java:76) at java.util.stream.MatchOps$1MatchSink.accept(MatchOps.java:90) at java.util.ArrayList$ArrayListSpliterator.tryAdvance(ArrayList.java:1359) at java.util.stream.ReferencePipeline.forEachWithCancel(ReferencePipeline.java:126) at java.util.stream.AbstractPipeline.copyIntoWithCancel(AbstractPipeline.java:498) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:485) at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) at java.util.stream.MatchOps$MatchOp.evaluateSequential(MatchOps.java:230) at java.util.stream.MatchOps$MatchOp.evaluateSequential(MatchOps.java:196) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.util.stream.ReferencePipeline.allMatch(ReferencePipeline.java:454) at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALActionListener.scopeWALEdits(ReplicationSourceWALActionListener.java:76) at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALActionListener.visitLogEntryBeforeWrite(ReplicationSourceWALActionListener.java:59) at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.appendEntry(AbstractFSWAL.java:930) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.appendAndSync(AsyncFSWAL.java:497) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.consume(AsyncFSWAL.java:599) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) was (Author: jutia): I add some callstack logs, sometimes JVM crash when append wal entry at org.apache.hadoop.hbase.util.ByteBufferUtils.toByte(ByteBufferUtils.java:418) at org.apache.hadoop.hbase.ByteBufferKeyValue.getFamilyLength(ByteBufferKeyValue.java:120) at org.apache.hadoop.hbase.ByteBufferKeyValue.getFamilyLength(ByteBufferKeyValue.java:102) at org.apache.hadoop.hbase.PrivateCellUtil.matchingFamily(PrivateCellUtil.java:731) at org.apache.hadoop.hbase.CellUtil.matchingFamily(CellUtil.java:816) at org.apache.hadoop.hbase.wal.WALEdit.isMetaEditFamily(WALEdit.java:209) at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALActionListener.lambda$scopeWALEdits$0(ReplicationSourceWALActionListener.java:76) at java.util.stream.MatchOps$1MatchSink.accept(MatchOps.java:90) at java.util.ArrayList$ArrayListSpliterator.tryAdvance(ArrayList.java:1359) at java.util.stream.ReferencePipeline.forEachWithCancel(ReferencePipeline.java:126) at java.util.stream.AbstractPipeline.copyIntoWithCancel(AbstractPipeline.java:498) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:485) at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) at java.util.stream.MatchOps$MatchOp.evaluateSequential(MatchOps.java:230) at java.util.stream.MatchOps$MatchOp.evaluateSequential(MatchOps.java:196) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.util.stream.ReferencePipeline.allMatch(ReferencePipeline.java:454) at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALActionListener.scopeWALEdits(ReplicationSourceWALActionListener.java:76) at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALActionListener.visitLogEntryBeforeWrite(ReplicationSourceWALActionListener.java:59) at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.appendEntry(AbstractFSWAL.java:930) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.appendAndSync(AsyncFSWAL.java:497) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.consume(AsyncFSWAL.java:599) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) > RegionServer JVM crash when append wal entry > > >
[jira] [Updated] (HBASE-25701) RegionServer JVM crash when append wal entry
[ https://issues.apache.org/jira/browse/HBASE-25701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Juanjuan Tian updated HBASE-25701: --- Affects Version/s: (was: 2.2.5) 2.2.6 > RegionServer JVM crash when append wal entry > > > Key: HBASE-25701 > URL: https://issues.apache.org/jira/browse/HBASE-25701 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.2.6 >Reporter: Juanjuan Tian >Priority: Major > > Region Server JVM crash when append wal entry, JVM crash log: > # > # A fatal error has been detected by the Java Runtime Environment: > # > # EXCEPTION_ACCESS_VIOLATION (0xc005) at pc=0x027af93f, > pid=17992, tid=0x2d54 > # > # JRE version: Java(TM) SE Runtime Environment (8.0_202-b08) (build > 1.8.0_202-b08) > # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.202-b08 mixed mode > windows-amd64 compressed oops) > # Problematic frame: > # J 10214 C2 org.apache.hadoop.hbase.ByteBufferKeyValue.getFamilyLength()B (9 > bytes) @ 0x027af93f [0x027af860+0xdf] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25701) RegionServer JVM crash when append wal entry
[ https://issues.apache.org/jira/browse/HBASE-25701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17309856#comment-17309856 ] Juanjuan Tian commented on HBASE-25701: I add some callstack logs, sometimes JVM crash when append wal entry at org.apache.hadoop.hbase.util.ByteBufferUtils.toByte(ByteBufferUtils.java:418) at org.apache.hadoop.hbase.ByteBufferKeyValue.getFamilyLength(ByteBufferKeyValue.java:120) at org.apache.hadoop.hbase.ByteBufferKeyValue.getFamilyLength(ByteBufferKeyValue.java:102) at org.apache.hadoop.hbase.PrivateCellUtil.matchingFamily(PrivateCellUtil.java:731) at org.apache.hadoop.hbase.CellUtil.matchingFamily(CellUtil.java:816) at org.apache.hadoop.hbase.wal.WALEdit.isMetaEditFamily(WALEdit.java:209) at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALActionListener.lambda$scopeWALEdits$0(ReplicationSourceWALActionListener.java:76) at java.util.stream.MatchOps$1MatchSink.accept(MatchOps.java:90) at java.util.ArrayList$ArrayListSpliterator.tryAdvance(ArrayList.java:1359) at java.util.stream.ReferencePipeline.forEachWithCancel(ReferencePipeline.java:126) at java.util.stream.AbstractPipeline.copyIntoWithCancel(AbstractPipeline.java:498) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:485) at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) at java.util.stream.MatchOps$MatchOp.evaluateSequential(MatchOps.java:230) at java.util.stream.MatchOps$MatchOp.evaluateSequential(MatchOps.java:196) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.util.stream.ReferencePipeline.allMatch(ReferencePipeline.java:454) at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALActionListener.scopeWALEdits(ReplicationSourceWALActionListener.java:76) at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALActionListener.visitLogEntryBeforeWrite(ReplicationSourceWALActionListener.java:59) at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.appendEntry(AbstractFSWAL.java:930) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.appendAndSync(AsyncFSWAL.java:497) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.consume(AsyncFSWAL.java:599) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) > RegionServer JVM crash when append wal entry > > > Key: HBASE-25701 > URL: https://issues.apache.org/jira/browse/HBASE-25701 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.2.5 >Reporter: Juanjuan Tian >Priority: Major > > Region Server JVM crash when append wal entry, JVM crash log: > # > # A fatal error has been detected by the Java Runtime Environment: > # > # EXCEPTION_ACCESS_VIOLATION (0xc005) at pc=0x027af93f, > pid=17992, tid=0x2d54 > # > # JRE version: Java(TM) SE Runtime Environment (8.0_202-b08) (build > 1.8.0_202-b08) > # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.202-b08 mixed mode > windows-amd64 compressed oops) > # Problematic frame: > # J 10214 C2 org.apache.hadoop.hbase.ByteBufferKeyValue.getFamilyLength()B (9 > bytes) @ 0x027af93f [0x027af860+0xdf] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-25701) RegionServer JVM crash when append wal entry
Juanjuan Tian created HBASE-25701: -- Summary: RegionServer JVM crash when append wal entry Key: HBASE-25701 URL: https://issues.apache.org/jira/browse/HBASE-25701 Project: HBase Issue Type: Improvement Affects Versions: 2.2.5 Reporter: Juanjuan Tian Region Server JVM crash when append wal entry, JVM crash log: # # A fatal error has been detected by the Java Runtime Environment: # # EXCEPTION_ACCESS_VIOLATION (0xc005) at pc=0x027af93f, pid=17992, tid=0x2d54 # # JRE version: Java(TM) SE Runtime Environment (8.0_202-b08) (build 1.8.0_202-b08) # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.202-b08 mixed mode windows-amd64 compressed oops) # Problematic frame: # J 10214 C2 org.apache.hadoop.hbase.ByteBufferKeyValue.getFamilyLength()B (9 bytes) @ 0x027af93f [0x027af860+0xdf] -- This message was sent by Atlassian Jira (v8.3.4#803005)
