[jira] [Commented] (HBASE-25701) RegionServer JVM crash when append wal entry
[ https://issues.apache.org/jira/browse/HBASE-25701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17409604#comment-17409604 ] Huaxiang Sun commented on HBASE-25701: -- [~jutia] Did you get chance to try 2.3.6 or 2.4.5 which has the fix for HBASE-24984? Please update if coredump can still be reproduced or not. > RegionServer JVM crash when append wal entry > > > Key: HBASE-25701 > URL: https://issues.apache.org/jira/browse/HBASE-25701 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.2.6 >Reporter: Juanjuan Tian >Priority: Major > > Region Server JVM crash when append wal entry, JVM crash log: > # > # A fatal error has been detected by the Java Runtime Environment: > # > # EXCEPTION_ACCESS_VIOLATION (0xc005) at pc=0x027af93f, > pid=17992, tid=0x2d54 > # > # JRE version: Java(TM) SE Runtime Environment (8.0_202-b08) (build > 1.8.0_202-b08) > # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.202-b08 mixed mode > windows-amd64 compressed oops) > # Problematic frame: > # J 10214 C2 org.apache.hadoop.hbase.ByteBufferKeyValue.getFamilyLength()B (9 > bytes) @ 0x027af93f [0x027af860+0xdf] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25701) RegionServer JVM crash when append wal entry
[ https://issues.apache.org/jira/browse/HBASE-25701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17385229#comment-17385229 ] Anoop Sam John commented on HBASE-25701: [~jutia] Looks like HBASE-24984 fix will fix ur case. Can you pls take a look. We can close this issue also. > RegionServer JVM crash when append wal entry > > > Key: HBASE-25701 > URL: https://issues.apache.org/jira/browse/HBASE-25701 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.2.6 >Reporter: Juanjuan Tian >Priority: Major > > Region Server JVM crash when append wal entry, JVM crash log: > # > # A fatal error has been detected by the Java Runtime Environment: > # > # EXCEPTION_ACCESS_VIOLATION (0xc005) at pc=0x027af93f, > pid=17992, tid=0x2d54 > # > # JRE version: Java(TM) SE Runtime Environment (8.0_202-b08) (build > 1.8.0_202-b08) > # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.202-b08 mixed mode > windows-amd64 compressed oops) > # Problematic frame: > # J 10214 C2 org.apache.hadoop.hbase.ByteBufferKeyValue.getFamilyLength()B (9 > bytes) @ 0x027af93f [0x027af860+0xdf] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25701) RegionServer JVM crash when append wal entry
[ https://issues.apache.org/jira/browse/HBASE-25701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17376223#comment-17376223 ] Michael Stack commented on HBASE-25701: --- Linking HBASE-26062 > RegionServer JVM crash when append wal entry > > > Key: HBASE-25701 > URL: https://issues.apache.org/jira/browse/HBASE-25701 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.2.6 >Reporter: Juanjuan Tian >Priority: Major > > Region Server JVM crash when append wal entry, JVM crash log: > # > # A fatal error has been detected by the Java Runtime Environment: > # > # EXCEPTION_ACCESS_VIOLATION (0xc005) at pc=0x027af93f, > pid=17992, tid=0x2d54 > # > # JRE version: Java(TM) SE Runtime Environment (8.0_202-b08) (build > 1.8.0_202-b08) > # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.202-b08 mixed mode > windows-amd64 compressed oops) > # Problematic frame: > # J 10214 C2 org.apache.hadoop.hbase.ByteBufferKeyValue.getFamilyLength()B (9 > bytes) @ 0x027af93f [0x027af860+0xdf] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25701) RegionServer JVM crash when append wal entry
[ https://issues.apache.org/jira/browse/HBASE-25701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17376142#comment-17376142 ] Huaxiang Sun commented on HBASE-25701: -- We run into a similar core dump at production cluster (2.3.5), I think the issue is HBASE-24984. I link these issues together and try to come up a unitest to reproduce the issue. > RegionServer JVM crash when append wal entry > > > Key: HBASE-25701 > URL: https://issues.apache.org/jira/browse/HBASE-25701 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.2.6 >Reporter: Juanjuan Tian >Priority: Major > > Region Server JVM crash when append wal entry, JVM crash log: > # > # A fatal error has been detected by the Java Runtime Environment: > # > # EXCEPTION_ACCESS_VIOLATION (0xc005) at pc=0x027af93f, > pid=17992, tid=0x2d54 > # > # JRE version: Java(TM) SE Runtime Environment (8.0_202-b08) (build > 1.8.0_202-b08) > # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.202-b08 mixed mode > windows-amd64 compressed oops) > # Problematic frame: > # J 10214 C2 org.apache.hadoop.hbase.ByteBufferKeyValue.getFamilyLength()B (9 > bytes) @ 0x027af93f [0x027af860+0xdf] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25701) RegionServer JVM crash when append wal entry
[ https://issues.apache.org/jira/browse/HBASE-25701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17312243#comment-17312243 ] Juanjuan Tian commented on HBASE-25701: [~anoop.hbase] I don't think they are same case, since I'm using habse 2.2.6 version which is already with fix from HBASE-22539, but issue still happen when appendEntry > RegionServer JVM crash when append wal entry > > > Key: HBASE-25701 > URL: https://issues.apache.org/jira/browse/HBASE-25701 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.2.6 >Reporter: Juanjuan Tian >Priority: Major > > Region Server JVM crash when append wal entry, JVM crash log: > # > # A fatal error has been detected by the Java Runtime Environment: > # > # EXCEPTION_ACCESS_VIOLATION (0xc005) at pc=0x027af93f, > pid=17992, tid=0x2d54 > # > # JRE version: Java(TM) SE Runtime Environment (8.0_202-b08) (build > 1.8.0_202-b08) > # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.202-b08 mixed mode > windows-amd64 compressed oops) > # Problematic frame: > # J 10214 C2 org.apache.hadoop.hbase.ByteBufferKeyValue.getFamilyLength()B (9 > bytes) @ 0x027af93f [0x027af860+0xdf] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25701) RegionServer JVM crash when append wal entry
[ https://issues.apache.org/jira/browse/HBASE-25701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17312063#comment-17312063 ] Anoop Sam John commented on HBASE-25701: Oh ya I mean that only. Issue was with Durability.ASYNC_WAL for puts. So in ur case that is not the scenario? > RegionServer JVM crash when append wal entry > > > Key: HBASE-25701 > URL: https://issues.apache.org/jira/browse/HBASE-25701 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.2.6 >Reporter: Juanjuan Tian >Priority: Major > > Region Server JVM crash when append wal entry, JVM crash log: > # > # A fatal error has been detected by the Java Runtime Environment: > # > # EXCEPTION_ACCESS_VIOLATION (0xc005) at pc=0x027af93f, > pid=17992, tid=0x2d54 > # > # JRE version: Java(TM) SE Runtime Environment (8.0_202-b08) (build > 1.8.0_202-b08) > # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.202-b08 mixed mode > windows-amd64 compressed oops) > # Problematic frame: > # J 10214 C2 org.apache.hadoop.hbase.ByteBufferKeyValue.getFamilyLength()B (9 > bytes) @ 0x027af93f [0x027af860+0xdf] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25701) RegionServer JVM crash when append wal entry
[ https://issues.apache.org/jira/browse/HBASE-25701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17311591#comment-17311591 ] Juanjuan Tian commented on HBASE-25701: [~anoop.hbase] do you mean fixed in Jira https://issues.apache.org/jira/browse/HBASE-22539, I don't think they are same isssue, since I'm using habse 2.2.6 with fix from HBASE-22539, but issue still happen when appendEntry, not replay. if you are not meaning HBASE-22539, could you help provide the Jira id? > RegionServer JVM crash when append wal entry > > > Key: HBASE-25701 > URL: https://issues.apache.org/jira/browse/HBASE-25701 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.2.6 >Reporter: Juanjuan Tian >Priority: Major > > Region Server JVM crash when append wal entry, JVM crash log: > # > # A fatal error has been detected by the Java Runtime Environment: > # > # EXCEPTION_ACCESS_VIOLATION (0xc005) at pc=0x027af93f, > pid=17992, tid=0x2d54 > # > # JRE version: Java(TM) SE Runtime Environment (8.0_202-b08) (build > 1.8.0_202-b08) > # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.202-b08 mixed mode > windows-amd64 compressed oops) > # Problematic frame: > # J 10214 C2 org.apache.hadoop.hbase.ByteBufferKeyValue.getFamilyLength()B (9 > bytes) @ 0x027af93f [0x027af860+0xdf] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25701) RegionServer JVM crash when append wal entry
[ https://issues.apache.org/jira/browse/HBASE-25701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17311587#comment-17311587 ] Anoop Sam John commented on HBASE-25701: There was an issue already been fixed. Ya the issue was release of BB before the cells are been written to WAL in case of Aync write. > RegionServer JVM crash when append wal entry > > > Key: HBASE-25701 > URL: https://issues.apache.org/jira/browse/HBASE-25701 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.2.6 >Reporter: Juanjuan Tian >Priority: Major > > Region Server JVM crash when append wal entry, JVM crash log: > # > # A fatal error has been detected by the Java Runtime Environment: > # > # EXCEPTION_ACCESS_VIOLATION (0xc005) at pc=0x027af93f, > pid=17992, tid=0x2d54 > # > # JRE version: Java(TM) SE Runtime Environment (8.0_202-b08) (build > 1.8.0_202-b08) > # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.202-b08 mixed mode > windows-amd64 compressed oops) > # Problematic frame: > # J 10214 C2 org.apache.hadoop.hbase.ByteBufferKeyValue.getFamilyLength()B (9 > bytes) @ 0x027af93f [0x027af860+0xdf] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25701) RegionServer JVM crash when append wal entry
[ https://issues.apache.org/jira/browse/HBASE-25701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17309859#comment-17309859 ] Juanjuan Tian commented on HBASE-25701: seems bytebuffer is relased before append wal entry > RegionServer JVM crash when append wal entry > > > Key: HBASE-25701 > URL: https://issues.apache.org/jira/browse/HBASE-25701 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.2.6 >Reporter: Juanjuan Tian >Priority: Major > > Region Server JVM crash when append wal entry, JVM crash log: > # > # A fatal error has been detected by the Java Runtime Environment: > # > # EXCEPTION_ACCESS_VIOLATION (0xc005) at pc=0x027af93f, > pid=17992, tid=0x2d54 > # > # JRE version: Java(TM) SE Runtime Environment (8.0_202-b08) (build > 1.8.0_202-b08) > # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.202-b08 mixed mode > windows-amd64 compressed oops) > # Problematic frame: > # J 10214 C2 org.apache.hadoop.hbase.ByteBufferKeyValue.getFamilyLength()B (9 > bytes) @ 0x027af93f [0x027af860+0xdf] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25701) RegionServer JVM crash when append wal entry
[ https://issues.apache.org/jira/browse/HBASE-25701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17309857#comment-17309857 ] Juanjuan Tian commented on HBASE-25701: [~wchevreuil] [~zhangduo] [~openinx] any idea why issue happen? > RegionServer JVM crash when append wal entry > > > Key: HBASE-25701 > URL: https://issues.apache.org/jira/browse/HBASE-25701 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.2.6 >Reporter: Juanjuan Tian >Priority: Major > > Region Server JVM crash when append wal entry, JVM crash log: > # > # A fatal error has been detected by the Java Runtime Environment: > # > # EXCEPTION_ACCESS_VIOLATION (0xc005) at pc=0x027af93f, > pid=17992, tid=0x2d54 > # > # JRE version: Java(TM) SE Runtime Environment (8.0_202-b08) (build > 1.8.0_202-b08) > # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.202-b08 mixed mode > windows-amd64 compressed oops) > # Problematic frame: > # J 10214 C2 org.apache.hadoop.hbase.ByteBufferKeyValue.getFamilyLength()B (9 > bytes) @ 0x027af93f [0x027af860+0xdf] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25701) RegionServer JVM crash when append wal entry
[ https://issues.apache.org/jira/browse/HBASE-25701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17309856#comment-17309856 ] Juanjuan Tian commented on HBASE-25701: I add some callstack logs, sometimes JVM crash when append wal entry at org.apache.hadoop.hbase.util.ByteBufferUtils.toByte(ByteBufferUtils.java:418) at org.apache.hadoop.hbase.ByteBufferKeyValue.getFamilyLength(ByteBufferKeyValue.java:120) at org.apache.hadoop.hbase.ByteBufferKeyValue.getFamilyLength(ByteBufferKeyValue.java:102) at org.apache.hadoop.hbase.PrivateCellUtil.matchingFamily(PrivateCellUtil.java:731) at org.apache.hadoop.hbase.CellUtil.matchingFamily(CellUtil.java:816) at org.apache.hadoop.hbase.wal.WALEdit.isMetaEditFamily(WALEdit.java:209) at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALActionListener.lambda$scopeWALEdits$0(ReplicationSourceWALActionListener.java:76) at java.util.stream.MatchOps$1MatchSink.accept(MatchOps.java:90) at java.util.ArrayList$ArrayListSpliterator.tryAdvance(ArrayList.java:1359) at java.util.stream.ReferencePipeline.forEachWithCancel(ReferencePipeline.java:126) at java.util.stream.AbstractPipeline.copyIntoWithCancel(AbstractPipeline.java:498) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:485) at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) at java.util.stream.MatchOps$MatchOp.evaluateSequential(MatchOps.java:230) at java.util.stream.MatchOps$MatchOp.evaluateSequential(MatchOps.java:196) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.util.stream.ReferencePipeline.allMatch(ReferencePipeline.java:454) at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALActionListener.scopeWALEdits(ReplicationSourceWALActionListener.java:76) at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALActionListener.visitLogEntryBeforeWrite(ReplicationSourceWALActionListener.java:59) at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.appendEntry(AbstractFSWAL.java:930) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.appendAndSync(AsyncFSWAL.java:497) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.consume(AsyncFSWAL.java:599) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) > RegionServer JVM crash when append wal entry > > > Key: HBASE-25701 > URL: https://issues.apache.org/jira/browse/HBASE-25701 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.2.5 >Reporter: Juanjuan Tian >Priority: Major > > Region Server JVM crash when append wal entry, JVM crash log: > # > # A fatal error has been detected by the Java Runtime Environment: > # > # EXCEPTION_ACCESS_VIOLATION (0xc005) at pc=0x027af93f, > pid=17992, tid=0x2d54 > # > # JRE version: Java(TM) SE Runtime Environment (8.0_202-b08) (build > 1.8.0_202-b08) > # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.202-b08 mixed mode > windows-amd64 compressed oops) > # Problematic frame: > # J 10214 C2 org.apache.hadoop.hbase.ByteBufferKeyValue.getFamilyLength()B (9 > bytes) @ 0x027af93f [0x027af860+0xdf] -- This message was sent by Atlassian Jira (v8.3.4#803005)
