[jira] [Commented] (HBASE-26198) RegionServer dead on hadoop 3.3.1: NoSuchMethodError LocatedBlocks.getLocations()
[ https://issues.apache.org/jira/browse/HBASE-26198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17409262#comment-17409262 ] Michael Stack commented on HBASE-26198: --- Thanks [~mengqi] ... The change in the LocatedBlock.getLocations() signature looks to have happened here, committed for hadoop-2.7, way back when. {code:java} commit ab934e85947dcf2092050023909dd81ae274ff45 Author: Arpit Agarwal Date: Mon Feb 9 12:17:40 2015 -0800HDFS-7647. DatanodeManager.sortLocatedBlocks sorts DatanodeInfos but not StorageIDs. (Contributed by Milan Desai){code} I'm bit baffled as to why I did not run into this testing – or why I don't see it in the 3.x clusters we usually run on. Let me try 2.4.5 on 3.3.1 again (it'll be a few days I think...). Thanks. > RegionServer dead on hadoop 3.3.1: NoSuchMethodError > LocatedBlocks.getLocations() > - > > Key: HBASE-26198 > URL: https://issues.apache.org/jira/browse/HBASE-26198 > Project: HBase > Issue Type: Bug > Components: regionserver >Reporter: mengqi >Priority: Major > Attachments: 4ad46153842c29898189b90fc986925c87966ce6.diff, a.diff, > image-2021-08-16-16-24-32-418.png > > > !image-2021-08-16-16-24-32-418.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-26198) RegionServer dead on hadoop 3.3.1: NoSuchMethodError LocatedBlocks.getLocations()
[ https://issues.apache.org/jira/browse/HBASE-26198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17405062#comment-17405062 ] mengqi commented on HBASE-26198: master log {code:java} 2021-08-26 14:45:24,871 ERROR org.apache.hadoop.hbase.master.HMaster: Failed to become active master java.lang.NoSuchMethodError: org.apache.hadoop.hdfs.protocol.LocatedBlock.getLocations()[Lorg/apache/hadoop/hdfs/protocol/DatanodeInfo; at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.connectToDataNodes(FanOutOneBlockAsyncDFSOutputHelper.java:439) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.createOutput(FanOutOneBlockAsyncDFSOutputHelper.java:545) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$400(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$8.doCall(FanOutOneBlockAsyncDFSOutputHelper.java:616) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$8.doCall(FanOutOneBlockAsyncDFSOutputHelper.java:611) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.createOutput(FanOutOneBlockAsyncDFSOutputHelper.java:624) at org.apache.hadoop.hbase.io.asyncfs.AsyncFSOutputHelper.createOutput(AsyncFSOutputHelper.java:53) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.initOutput(AsyncProtobufLogWriter.java:180) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.init(AbstractProtobufLogWriter.java:166) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createAsyncWriter(AsyncFSWALProvider.java:113) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:669) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:130) at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:841) at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:548) at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.init(AbstractFSWAL.java:489) at org.apache.hadoop.hbase.wal.AbstractFSWALProvider.getWAL(AbstractFSWALProvider.java:161) at org.apache.hadoop.hbase.wal.AbstractFSWALProvider.getWAL(AbstractFSWALProvider.java:63) at org.apache.hadoop.hbase.wal.WALFactory.getWAL(WALFactory.java:296) at org.apache.hadoop.hbase.master.region.MasterRegion.createWAL(MasterRegion.java:187) at org.apache.hadoop.hbase.master.region.MasterRegion.open(MasterRegion.java:260) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:303) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:837) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2241) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:510) at java.lang.Thread.run(Thread.java:748) {code} regionserver log {code:java} java.lang.NoSuchMethodError: org.apache.hadoop.hdfs.protocol.LocatedBlock.getLocations()[Lorg/apache/hadoop/hdfs/protocol/DatanodeInfo; at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.connectToDataNodes(FanOutOneBlockAsyncDFSOutputHelper.java:439) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.createOutput(FanOutOneBlockAsyncDFSOutputHelper.java:545) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$400(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$8.doCall(FanOutOneBlockAsyncDFSOutputHelper.java:616) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$8.doCall(FanOutOneBlockAsyncDFSOutputHelper.java:611) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.createOutput(FanOutOneBlockAsyncDFSOutputHelper.java:624) at org.apache.hadoop.hbase.io.asyncfs.AsyncFSOutputHelper.createOutput(AsyncFSOutputHelper.java:53) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.initOutput(AsyncProtobufLogWriter.java:180) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.init(AbstractProtobufLogWriter.java:166) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createAsyncWriter(AsyncFSWALProvider.java:113) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:669) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:130) at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:841) at
[jira] [Commented] (HBASE-26198) RegionServer dead on hadoop 3.3.1: NoSuchMethodError LocatedBlocks.getLocations()
[ https://issues.apache.org/jira/browse/HBASE-26198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17400115#comment-17400115 ] Michael Stack commented on HBASE-26198: --- We cannot vouch for vendor version of HBase > RegionServer dead on hadoop 3.3.1: NoSuchMethodError > LocatedBlocks.getLocations() > - > > Key: HBASE-26198 > URL: https://issues.apache.org/jira/browse/HBASE-26198 > Project: HBase > Issue Type: Bug > Components: regionserver >Reporter: mengqi >Priority: Major > Attachments: 4ad46153842c29898189b90fc986925c87966ce6.diff, > image-2021-08-16-16-24-32-418.png > > > !image-2021-08-16-16-24-32-418.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-26198) RegionServer dead on hadoop 3.3.1: NoSuchMethodError LocatedBlocks.getLocations()
[ https://issues.apache.org/jira/browse/HBASE-26198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17400070#comment-17400070 ] Wei-Chiu Chuang commented on HBASE-26198: - I am pretty sure CDH6.2 HBase doesn't compile against Hadoop 3.3.1. Did you simply replace the CDH Hadoop jars with Apache Hadoop 3.3.1? If so, I can believe it doesn't work out of box due to HDFS-15255. There are a number of HBase changes you would need to apply on top of CDH HBase 6.2.0. If you can apply them and recompile against Apache Hadoop 3.3.1, you should be able to run. > RegionServer dead on hadoop 3.3.1: NoSuchMethodError > LocatedBlocks.getLocations() > - > > Key: HBASE-26198 > URL: https://issues.apache.org/jira/browse/HBASE-26198 > Project: HBase > Issue Type: Bug > Components: regionserver >Reporter: mengqi >Priority: Major > Attachments: 4ad46153842c29898189b90fc986925c87966ce6.diff, > image-2021-08-16-16-24-32-418.png > > > !image-2021-08-16-16-24-32-418.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-26198) RegionServer dead on hadoop 3.3.1: NoSuchMethodError LocatedBlocks.getLocations()
[ https://issues.apache.org/jira/browse/HBASE-26198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17400062#comment-17400062 ] mengqi commented on HBASE-26198: Our hbase version is 2.1.0-cdh6.2.0 > RegionServer dead on hadoop 3.3.1: NoSuchMethodError > LocatedBlocks.getLocations() > - > > Key: HBASE-26198 > URL: https://issues.apache.org/jira/browse/HBASE-26198 > Project: HBase > Issue Type: Bug > Components: regionserver >Reporter: mengqi >Priority: Major > Attachments: 4ad46153842c29898189b90fc986925c87966ce6.diff, > image-2021-08-16-16-24-32-418.png > > > !image-2021-08-16-16-24-32-418.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-26198) RegionServer dead on hadoop 3.3.1: NoSuchMethodError LocatedBlocks.getLocations()
[ https://issues.apache.org/jira/browse/HBASE-26198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17399824#comment-17399824 ] Michael Stack commented on HBASE-26198: --- Interesting. I tested 3.3.1. Which hbase version is this? Thanks. > RegionServer dead on hadoop 3.3.1: NoSuchMethodError > LocatedBlocks.getLocations() > - > > Key: HBASE-26198 > URL: https://issues.apache.org/jira/browse/HBASE-26198 > Project: HBase > Issue Type: Bug > Components: regionserver >Reporter: mengqi >Priority: Major > Attachments: 4ad46153842c29898189b90fc986925c87966ce6.diff, > image-2021-08-16-16-24-32-418.png > > > !image-2021-08-16-16-24-32-418.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-26198) regionserver dead on hadoop 3.3.1
[ https://issues.apache.org/jira/browse/HBASE-26198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17399680#comment-17399680 ] mengqi commented on HBASE-26198: I only have hadoop 3.3.1 in my classpath , and this happened when reply WALs > regionserver dead on hadoop 3.3.1 > - > > Key: HBASE-26198 > URL: https://issues.apache.org/jira/browse/HBASE-26198 > Project: HBase > Issue Type: Bug > Components: regionserver >Reporter: mengqi >Priority: Major > Attachments: 4ad46153842c29898189b90fc986925c87966ce6.diff, > image-2021-08-16-16-24-32-418.png > > > !image-2021-08-16-16-24-32-418.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-26198) regionserver dead on hadoop 3.3.1
[ https://issues.apache.org/jira/browse/HBASE-26198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17399662#comment-17399662 ] Wei-Chiu Chuang commented on HBASE-26198: - I suspect it was because the method signature of the private HDFS API LocatedBlock.getLocations() was changed by HDFS-15255. But even in that case it shouldn't fail for that exception, especially since I ran the UTs which should cover this code path. Can you check if you have multiple Hadoop libraries in the classpath? It's likely you have both hadoop 3.3.1 and 3.1.2 (default) in the classpath. > regionserver dead on hadoop 3.3.1 > - > > Key: HBASE-26198 > URL: https://issues.apache.org/jira/browse/HBASE-26198 > Project: HBase > Issue Type: Bug > Components: regionserver >Reporter: mengqi >Priority: Major > Attachments: 4ad46153842c29898189b90fc986925c87966ce6.diff, > image-2021-08-16-16-24-32-418.png > > > !image-2021-08-16-16-24-32-418.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)