> On May 24, 2019, 3:18 p.m., Velmurugan Periasamy wrote: > > Ship It! > > Abhay Kulkarni wrote: > About description of this JIRA: In my testing for hbase and hadoop up I > found the following to be the cause of the failure. > > 2019-05-23 16:23:56,805 WARN [main] util.NativeCodeLoader: Unable to > load native-hadoop library for your platform... using builtin-java classes > where applicable > Formatting using clusterid: testClusterID > 2019-05-23 16:23:58,033 WARN [main] impl.MetricsConfig: Cannot locate > configuration: tried > hadoop-metrics2-namenode.properties,hadoop-metrics2.properties > 2019-05-23 16:23:58,186 WARN [main] http.HttpRequestLog: Jetty request > log can only be enabled using Log4j > 2019-05-23 16:23:58,863 WARN [main] common.MetricsLoggerTask: Metrics > logging will not be async since the logger is not log4j > 2019-05-23 16:23:58,947 WARN [main] http.HttpRequestLog: Jetty request > log can only be enabled using Log4j > 2019-05-23 16:23:59,224 WARN [main] common.MetricsLoggerTask: Metrics > logging will not be async since the logger is not log4j > 2019-05-23 16:23:59,722 ERROR [main] server.ZooKeeperServer: > ZKShutdownHandler is not registered, so ZooKeeper server won't take any > action on ERROR or SHUTDOWN server state changes > 2019-05-23 16:24:01,269 WARN [Thread-107] hbase.ZNodeClearer: > Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on > crash by start scripts (Longer MTTR!) > 2019-05-23 16:24:02,216 ERROR [Thread-107] config.RangerConfiguration: > addResourceIfReadable(ranger-hbase-audit.xml): couldn't find resource file > location > 2019-05-23 16:24:02,302 WARN [RS:0;10.22.9.38:52417] > regionserver.HRegionServer: reportForDuty failed; sleeping and then retrying. > 2019-05-23 16:24:02,916 WARN [master/10.22.9.38:52415] > assignment.AssignmentManager: No servers available; cannot place 1 unassigned > regions. > 2019-05-23 16:24:03,933 WARN [master/10.22.9.38:52415] > assignment.AssignmentManager: No servers available; cannot place 1 unassigned > regions. > 2019-05-23 16:24:04,946 WARN [master/10.22.9.38:52415] > assignment.AssignmentManager: No servers available; cannot place 1 unassigned > regions. > 2019-05-23 16:24:05,324 WARN [RS:0;10.22.9.38:52417] hbase.ZNodeClearer: > Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on > crash by start scripts (Longer MTTR!) > 2019-05-23 16:24:08,521 ERROR [RS:0;10.22.9.38:52417] > regionserver.HRegionServer: ***** ABORTING region server > 10.22.9.38,52417,1558653841129: Unhandled: Found interface > org.apache.hadoop.hdfs.protocol.HdfsFileStatus, but class was expected ***** > java.lang.IncompatibleClassChangeError: Found interface > org.apache.hadoop.hdfs.protocol.HdfsFileStatus, but class was expected > at > org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.createOutput(FanOutOneBlockAsyncDFSOutputHelper.java:768) > at > org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$400(FanOutOneBlockAsyncDFSOutputHelper.java:118) > at > org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$16.doCall(FanOutOneBlockAsyncDFSOutputHelper.java:848) > at > org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$16.doCall(FanOutOneBlockAsyncDFSOutputHelper.java:843) > at > org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) > at > org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.createOutput(FanOutOneBlockAsyncDFSOutputHelper.java:856) > at > org.apache.hadoop.hbase.io.asyncfs.AsyncFSOutputHelper.createOutput(AsyncFSOutputHelper.java:51) > at > org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.initOutput(AsyncProtobufLogWriter.java:168) > at > org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.init(AbstractProtobufLogWriter.java:165) > at > org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createAsyncWriter(AsyncFSWALProvider.java:113) > at > org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:612) > at > org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:124) > at > org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:764) > at > org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:494) > at > org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.<init>(AsyncFSWAL.java:251) > at > org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createWAL(AsyncFSWALProvider.java:73) > at > org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createWAL(AsyncFSWALProvider.java:48) > at > org.apache.hadoop.hbase.wal.AbstractFSWALProvider.getWAL(AbstractFSWALProvider.java:138) > at > org.apache.hadoop.hbase.wal.AbstractFSWALProvider.getWAL(AbstractFSWALProvider.java:57) > at org.apache.hadoop.hbase.wal.WALFactory.getWAL(WALFactory.java:276) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.getWAL(HRegionServer.java:2064) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.buildServerLoad(HRegionServer.java:1290) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.tryRegionServerReport(HRegionServer.java:1172) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:989) > at > org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.runRegionServer(MiniHBaseCluster.java:183) > at > org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$000(MiniHBaseCluster.java:129) > at > org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$1.run(MiniHBaseCluster.java:167) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:360) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1709) > at > org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:307) > at > org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.run(MiniHBaseCluster.java:164) > at java.lang.Thread.run(Thread.java:748) > 2019-05-23 16:24:08,524 ERROR [RS:0;10.22.9.38:52417] > regionserver.HRegionServer: RegionServer abort: loaded coprocessors are: > [org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor] > > > Also, Hbase JIRA which addresses this is : > https://issues.apache.org/jira/browse/HBASE-22394
Complete stacktrace after setting test cases log level(ranger.root.logger=WARN,console) in the log4j.properties of ranger hbase plugins : https://gist.github.com/pradeepagrawal8184/798bd00df71797ae74fe7aa4d53f3017 - Pradeep ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/70677/#review215499 ----------------------------------------------------------- On May 19, 2019, 12:56 p.m., Pradeep Agrawal wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/70677/ > ----------------------------------------------------------- > > (Updated May 19, 2019, 12:56 p.m.) > > > Review request for ranger, Ankita Sinha, bhavik patel, Gautam Borad, Abhay > Kulkarni, Madhan Neethiraj, Mehul Parikh, Nikhil P, Nitin Galave, Ramesh > Mani, Sailaja Polavarapu, and Velmurugan Periasamy. > > > Bugs: RANGER-2432 > https://issues.apache.org/jira/browse/RANGER-2432 > > > Repository: ranger > > > Description > ------- > > **Following libraries can be upgraded:** > > Hadoop version from 3.0.0 to 3.1.1 > HBase version from 2.0.0 to 2.0.2 > Hive version from 3.0.0 to 3.1.0 > Atlas version from 1.1.0 to 2.0.0 > > > **Note:** Disabled Hbase test cases as setup() in > HBaseRangerAuthorizationTest is failing. It could be due to any of the issues > mentioned in https://issues.apache.org/jira/browse/HBASE-21688. If its so, > then probably fix is not available in the released hbase versions as I have > tried all released Hbase > 2.0.0. Also have tried with all the combination of > hadoop > 3.0.0 > It appears this issue is due to changes in hadoop after 3.0.0 release but not > fixed from hbase end in released version. > > I could not find any work around to get them working as earlier. If you have > any approach to fix these testcases please leave your comment here or in the > Jira. > > **Stacktrace after the testcase failure:** > > Thread 1 (main): > State: RUNNABLE > Blocked count: 50 > Waited count: 499 > Stack: > sun.management.ThreadImpl.getThreadInfo1(Native Method) > sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:178) > sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:139) > > org.apache.hadoop.util.ReflectionUtils.printThreadInfo(ReflectionUtils.java:169) > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > java.lang.reflect.Method.invoke(Method.java:498) > > org.apache.hadoop.hbase.util.Threads$PrintThreadInfoLazyHolder$1.printThreadInfo(Threads.java:294) > org.apache.hadoop.hbase.util.Threads.printThreadInfo(Threads.java:341) > > org.apache.hadoop.hbase.util.JVMClusterUtil.startup(JVMClusterUtil.java:221) > > org.apache.hadoop.hbase.LocalHBaseCluster.startup(LocalHBaseCluster.java:413) > org.apache.hadoop.hbase.MiniHBaseCluster.init(MiniHBaseCluster.java:258) > org.apache.hadoop.hbase.MiniHBaseCluster.<init>(MiniHBaseCluster.java:115) > > org.apache.hadoop.hbase.HBaseTestingUtility.startMiniHBaseCluster(HBaseTestingUtility.java:1038) > > org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:984) > > org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:855) > > org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:849) > > org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:778) > > org.apache.ranger.authorization.hbase.HBaseRangerAuthorizationTest.setup(HBaseRangerAuthorizationTest.java:100) > [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: > 207.69 s <<< FAILURE! - in > org.apache.ranger.authorization.hbase.HBaseRangerAuthorizationTest > [ERROR] org.apache.ranger.authorization.hbase.HBaseRangerAuthorizationTest > Time elapsed: 207.69 s <<< ERROR! > java.io.IOException: Shutting down > at > org.apache.ranger.authorization.hbase.HBaseRangerAuthorizationTest.setup(HBaseRangerAuthorizationTest.java:100) > Caused by: java.lang.RuntimeException: Master not initialized after 200000ms > seconds > at > org.apache.ranger.authorization.hbase.HBaseRangerAuthorizationTest.setup(HBaseRangerAuthorizationTest.java:100) > > > Diffs > ----- > > > hbase-agent/src/test/java/org/apache/ranger/authorization/hbase/HBaseRangerAuthorizationTest.java > 172b310f1 > > hive-agent/src/test/java/org/apache/ranger/services/hive/HIVERangerAuthorizerTest.java > 62381743f > pom.xml 40b053bd2 > security-admin/pom.xml 76bb317e5 > src/main/assembly/admin-web.xml c862155f3 > > > Diff: https://reviews.apache.org/r/70677/diff/3/ > > > Testing > ------- > > Tested Ranger Build, Ranger installation and Ranger admin service, zone, > policy, user and group crud operation. > > > Thanks, > > Pradeep Agrawal > >
