[
https://issues.apache.org/jira/browse/RANGER-1412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Qiang Zhang updated RANGER-1412:
--------------------------------
Description:
Start hadoop failed after enabling ranger HDFS plugins. The error information
is as following:
2017-02-27 02:34:58,885 ERROR
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: FSNamesystem
initialization failed.
java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:134)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:843)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:673)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:585)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:645)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:812)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:796)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1493)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1559)
...............
at
org.apache.ranger.authorization.hadoop.RangerHdfsAuthorizer.init(RangerHdfsAuthorizer.java:64)
at
org.apache.ranger.authorization.hadoop.RangerHdfsAuthorizer.<init>(RangerHdfsAuthorizer.java:43)
at sun.reflect.GeneratedConstructorAccessor7.newInstance(Unknown Source)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at java.lang.Class.newInstance(Class.java:383)
at
org.apache.ranger.authorization.hadoop.RangerHdfsAuthorizer.init(RangerHdfsAuthorizer.java:64)
at
org.apache.ranger.authorization.hadoop.RangerHdfsAuthorizer.<init>(RangerHdfsAuthorizer.java:43)
at sun.reflect.GeneratedConstructorAccessor7.newInstance(Unknown Source)
2017-02-27 02:34:58,922 INFO org.apache.hadoop.util.ExitUtil: Exiting with
status 1
The cause of the issue is that the enable-hdfs-plugin.sh was done as following.
1. Link ranger-1.0.0-SNAPSHOT-hdfs-plugin/lib/ranger-hdfs-plugin-impl to
/var/local/hadoop/hadoop-2.7.3/lib/ranger-hdfs-plugin-impl
2. Link
ranger-1.0.0-SNAPSHOT-hdfs-plugin/lib/ranger-hdfs-plugin-shim-1.0.0-SNAPSHOT.jar
to
/var/local/hadoop/hadoop-2.7.3/lib/ranger-hdfs-plugin-shim-1.0.0-SNAPSHOT.jar
3. Link
ranger-1.0.0-SNAPSHOT-hdfs-plugin/lib/ranger-plugin-classloader-1.0.0-SNAPSHOT.jar
to
/var/local/hadoop/hadoop-2.7.3/lib/ranger-plugin-classloader-1.0.0-SNAPSHOT.jar
lrwxrwxrwx 1 root root 72 Feb 27 02:33 ranger-hdfs-plugin-impl ->
/usr/local/ranger-1.0.0-SNAPSHOT-hdfs-plugin/lib/ranger-hdfs-plugin-impl/
lrwxrwxrwx 1 root root 91 Feb 27 02:33
ranger-hdfs-plugin-shim-1.0.0-SNAPSHOT.jar ->
/usr/local/ranger-1.0.0-SNAPSHOT-hdfs-plugin/lib/ranger-hdfs-plugin-shim-1.0.0-SNAPSHOT.jar
lrwxrwxrwx 1 root root 93 Feb 27 02:33
ranger-plugin-classloader-1.0.0-SNAPSHOT.jar ->
/usr/local/ranger-1.0.0-SNAPSHOT-hdfs-plugin/lib/ranger-plugin-classloader-1.0.0-SNAPSHOT.jar
The above link are error. The hadoop can not find dependent packages when
starting. They should link as following.
1. Link ranger-1.0.0-SNAPSHOT-hdfs-plugin/lib/ranger-hdfs-plugin-impl to
/var/local/hadoop/hadoop-2.7.3/share/hadoop/hdfs/lib/ranger-hdfs-plugin-impl
2. Link
ranger-1.0.0-SNAPSHOT-hdfs-plugin/lib/ranger-hdfs-plugin-shim-1.0.0-SNAPSHOT.jar
to
/var/local/hadoop/hadoop-2.7.3/share/hadoop/hdfs/lib/ranger-hdfs-plugin-shim-1.0.0-SNAPSHOT.jar
3. Link
ranger-1.0.0-SNAPSHOT-hdfs-plugin/lib/ranger-plugin-classloader-1.0.0-SNAPSHOT.jar
to
/var/local/hadoop/hadoop-2.7.3/share/hadoop/hdfs/lib/ranger-plugin-classloader-1.0.0-SNAPSHOT.jar
The program install path is as following:
1. Install pseudo distributions hadoop. It's path is
/var/local/hadoop/hadoop-2.7.3.
2. ranger hdfs plugin path is /usr/local/ranger-1.0.0-SNAPSHOT-hdfs-plugin.
Test and verify:
I carefully tested and verified the patch before commit the issue.
was:
Start hadoop failed after enabling ranger HDFS plugins. The error information
is as following:
2017-02-27 02:34:58,885 ERROR
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: FSNamesystem
initialization failed.
java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:134)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:843)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:673)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:585)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:645)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:812)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:796)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1493)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1559)
...............
at
org.apache.ranger.authorization.hadoop.RangerHdfsAuthorizer.init(RangerHdfsAuthorizer.java:64)
at
org.apache.ranger.authorization.hadoop.RangerHdfsAuthorizer.<init>(RangerHdfsAuthorizer.java:43)
at sun.reflect.GeneratedConstructorAccessor7.newInstance(Unknown Source)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at java.lang.Class.newInstance(Class.java:383)
at
org.apache.ranger.authorization.hadoop.RangerHdfsAuthorizer.init(RangerHdfsAuthorizer.java:64)
at
org.apache.ranger.authorization.hadoop.RangerHdfsAuthorizer.<init>(RangerHdfsAuthorizer.java:43)
at sun.reflect.GeneratedConstructorAccessor7.newInstance(Unknown Source)
2017-02-27 02:34:58,922 INFO org.apache.hadoop.util.ExitUtil: Exiting with
status 1
The cause of the issue is that the enable-hdfs-plugin.sh was done as following.
1. Link ranger-1.0.0-SNAPSHOT-hdfs-plugin/lib/ranger-hdfs-plugin-impl to
/var/local/hadoop/hadoop-2.7.3/lib/ranger-hdfs-plugin-impl
2. Link
ranger-1.0.0-SNAPSHOT-hdfs-plugin/lib/ranger-hdfs-plugin-shim-1.0.0-SNAPSHOT.jar
to
/var/local/hadoop/hadoop-2.7.3/lib/ranger-hdfs-plugin-shim-1.0.0-SNAPSHOT.jar
3. Link
ranger-1.0.0-SNAPSHOT-hdfs-plugin/lib/ranger-plugin-classloader-1.0.0-SNAPSHOT.jar
to
/var/local/hadoop/hadoop-2.7.3/lib/ranger-plugin-classloader-1.0.0-SNAPSHOT.jar
lrwxrwxrwx 1 root root 72 Feb 27 02:33 ranger-hdfs-plugin-impl ->
/usr/local/ranger-1.0.0-SNAPSHOT-hdfs-plugin/lib/ranger-hdfs-plugin-impl/
lrwxrwxrwx 1 root root 91 Feb 27 02:33
ranger-hdfs-plugin-shim-1.0.0-SNAPSHOT.jar ->
/usr/local/ranger-1.0.0-SNAPSHOT-hdfs-plugin/lib/ranger-hdfs-plugin-shim-1.0.0-SNAPSHOT.jar
lrwxrwxrwx 1 root root 93 Feb 27 02:33
ranger-plugin-classloader-1.0.0-SNAPSHOT.jar ->
/usr/local/ranger-1.0.0-SNAPSHOT-hdfs-plugin/lib/ranger-plugin-classloader-1.0.0-SNAPSHOT.jar
The above link are error. The hadoop can not find dependent packages when
starting. They should link as following.
1. Link ranger-1.0.0-SNAPSHOT-hdfs-plugin/lib/ranger-hdfs-plugin-impl to
/var/local/hadoop/hadoop-2.7.3/share/hadoop/hdfs/lib/ranger-hdfs-plugin-impl
2. Link
ranger-1.0.0-SNAPSHOT-hdfs-plugin/lib/ranger-hdfs-plugin-shim-1.0.0-SNAPSHOT.jar
to
/var/local/hadoop/hadoop-2.7.3/share/hadoop/hdfs/lib/ranger-hdfs-plugin-shim-1.0.0-SNAPSHOT.jar
3. Link
ranger-1.0.0-SNAPSHOT-hdfs-plugin/lib/ranger-plugin-classloader-1.0.0-SNAPSHOT.jar
to
/var/local/hadoop/hadoop-2.7.3/share/hadoop/hdfs/lib/ranger-plugin-classloader-1.0.0-SNAPSHOT.jar
The program install path is as following:
1. Install pseudo distributions hadoop. It's path is
/var/local/hadoop/hadoop-2.7.3.
2. ranger hdfs plugin path is /usr/local/ranger-1.0.0-SNAPSHOT-hdfs-plugin.
> Start hadoop failed after enabling ranger HDFS plugins
> ------------------------------------------------------
>
> Key: RANGER-1412
> URL: https://issues.apache.org/jira/browse/RANGER-1412
> Project: Ranger
> Issue Type: Bug
> Components: plugins
> Affects Versions: 1.0.0
> Reporter: Qiang Zhang
> Assignee: Qiang Zhang
> Priority: Critical
> Labels: patch
> Attachments:
> 0001-RANGER-1412-Start-hadoop-failed-after-enabling-range.patch
>
>
> Start hadoop failed after enabling ranger HDFS plugins. The error information
> is as following:
> 2017-02-27 02:34:58,885 ERROR
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: FSNamesystem
> initialization failed.
> java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
> at
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:134)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:843)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:673)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:585)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:645)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:812)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:796)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1493)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1559)
> ...............
> at
> org.apache.ranger.authorization.hadoop.RangerHdfsAuthorizer.init(RangerHdfsAuthorizer.java:64)
> at
> org.apache.ranger.authorization.hadoop.RangerHdfsAuthorizer.<init>(RangerHdfsAuthorizer.java:43)
> at sun.reflect.GeneratedConstructorAccessor7.newInstance(Unknown Source)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> at java.lang.Class.newInstance(Class.java:383)
> at
> org.apache.ranger.authorization.hadoop.RangerHdfsAuthorizer.init(RangerHdfsAuthorizer.java:64)
> at
> org.apache.ranger.authorization.hadoop.RangerHdfsAuthorizer.<init>(RangerHdfsAuthorizer.java:43)
> at sun.reflect.GeneratedConstructorAccessor7.newInstance(Unknown Source)
> 2017-02-27 02:34:58,922 INFO org.apache.hadoop.util.ExitUtil: Exiting with
> status 1
> The cause of the issue is that the enable-hdfs-plugin.sh was done as
> following.
> 1. Link ranger-1.0.0-SNAPSHOT-hdfs-plugin/lib/ranger-hdfs-plugin-impl to
> /var/local/hadoop/hadoop-2.7.3/lib/ranger-hdfs-plugin-impl
> 2. Link
> ranger-1.0.0-SNAPSHOT-hdfs-plugin/lib/ranger-hdfs-plugin-shim-1.0.0-SNAPSHOT.jar
> to
> /var/local/hadoop/hadoop-2.7.3/lib/ranger-hdfs-plugin-shim-1.0.0-SNAPSHOT.jar
> 3. Link
> ranger-1.0.0-SNAPSHOT-hdfs-plugin/lib/ranger-plugin-classloader-1.0.0-SNAPSHOT.jar
> to
> /var/local/hadoop/hadoop-2.7.3/lib/ranger-plugin-classloader-1.0.0-SNAPSHOT.jar
> lrwxrwxrwx 1 root root 72 Feb 27 02:33 ranger-hdfs-plugin-impl ->
> /usr/local/ranger-1.0.0-SNAPSHOT-hdfs-plugin/lib/ranger-hdfs-plugin-impl/
> lrwxrwxrwx 1 root root 91 Feb 27 02:33
> ranger-hdfs-plugin-shim-1.0.0-SNAPSHOT.jar ->
> /usr/local/ranger-1.0.0-SNAPSHOT-hdfs-plugin/lib/ranger-hdfs-plugin-shim-1.0.0-SNAPSHOT.jar
> lrwxrwxrwx 1 root root 93 Feb 27 02:33
> ranger-plugin-classloader-1.0.0-SNAPSHOT.jar ->
> /usr/local/ranger-1.0.0-SNAPSHOT-hdfs-plugin/lib/ranger-plugin-classloader-1.0.0-SNAPSHOT.jar
> The above link are error. The hadoop can not find dependent packages when
> starting. They should link as following.
> 1. Link ranger-1.0.0-SNAPSHOT-hdfs-plugin/lib/ranger-hdfs-plugin-impl to
> /var/local/hadoop/hadoop-2.7.3/share/hadoop/hdfs/lib/ranger-hdfs-plugin-impl
> 2. Link
> ranger-1.0.0-SNAPSHOT-hdfs-plugin/lib/ranger-hdfs-plugin-shim-1.0.0-SNAPSHOT.jar
> to
> /var/local/hadoop/hadoop-2.7.3/share/hadoop/hdfs/lib/ranger-hdfs-plugin-shim-1.0.0-SNAPSHOT.jar
> 3. Link
> ranger-1.0.0-SNAPSHOT-hdfs-plugin/lib/ranger-plugin-classloader-1.0.0-SNAPSHOT.jar
> to
> /var/local/hadoop/hadoop-2.7.3/share/hadoop/hdfs/lib/ranger-plugin-classloader-1.0.0-SNAPSHOT.jar
> The program install path is as following:
> 1. Install pseudo distributions hadoop. It's path is
> /var/local/hadoop/hadoop-2.7.3.
> 2. ranger hdfs plugin path is /usr/local/ranger-1.0.0-SNAPSHOT-hdfs-plugin.
> Test and verify:
> I carefully tested and verified the patch before commit the issue.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
