[
https://issues.apache.org/jira/browse/RANGER-4228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
caijialiang updated RANGER-4228:
--------------------------------
Description:
1.Missing hadoop shade package
A temporary solution is to copy the required JAR files from Hadoop
cp /usr/bigtop/current/hadoop-hdfs-client/lib/hadoop-shaded-guava-1.1.1.jar
/usr/bigtop/current/kafka-broker/libs/
{code:java}
[2023-05-06 14:33:22,254] ERROR [KafkaServer id=1001] Fatal error during
KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
java.lang.NoClassDefFoundError:
org/apache/hadoop/thirdparty/com/google/common/base/Preconditions
at
org.apache.hadoop.conf.Configuration$DeprecationDelta.<init>(Configuration.java:430)
at
org.apache.hadoop.conf.Configuration$DeprecationDelta.<init>(Configuration.java:443)
at org.apache.hadoop.conf.Configuration.<clinit>(Configuration.java:525)
at
org.apache.ranger.plugin.service.RangerBasePlugin.<init>(RangerBasePlugin.java:79)
at
org.apache.ranger.authorization.kafka.authorizer.RangerKafkaAuthorizer.configure(RangerKafkaAuthorizer.java:100)
at
org.apache.ranger.authorization.kafka.authorizer.RangerKafkaAuthorizer.configure(RangerKafkaAuthorizer.java:95)
at
kafka.security.authorizer.AuthorizerWrapper.configure(AuthorizerWrapper.scala:86)
at kafka.server.KafkaServer.$anonfun$startup$11(KafkaServer.scala:352)
at kafka.server.KafkaServer.$anonfun$startup$11$adapted(KafkaServer.scala:352)
at scala.Option.foreach(Option.scala:407)
at kafka.server.KafkaServer.startup(KafkaServer.scala:352)
at kafka.Kafka$.main(Kafka.scala:109)
at kafka.Kafka.main(Kafka.scala)
Caused by: java.lang.ClassNotFoundException:
org.apache.hadoop.thirdparty.com.google.common.base.Preconditions
at java.lang.ClassLoader.findClass(ClassLoader.java:530)
at
org.apache.ranger.plugin.classloader.RangerPluginClassLoader$MyClassLoader.findClass(RangerPluginClassLoader.java:316)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at
org.apache.ranger.plugin.classloader.RangerPluginClassLoader.loadClass(RangerPluginClassLoader.java:160)
... 13 more {code}
https://issues.apache.org/jira/browse/HADOOP-14284
Starting from Hadoop 3, Hadoop uses the shaded Guava library located in the
hadoop-shade jar instead of the original Guava library. However, unlike HBase
or Hive, Kafka does not add the Hadoop lib to its class path during startup.
Therefore, in a big data cluster using Hadoop 3, the HBase and Hive plugins can
start without errors, but Kafka may report errors due to the lack of the
hadoop-shade jar in its class path. As a temporary solution, the necessary jar
files can be copied from Hadoop to Kafka.
2.Missing common-compress package
A temporary solution is to copy the required JAR files from Hadoop."
{code:java}
[2023-05-08 10:15:18,025] ERROR [KafkaServer id=1001] Fatal error during
KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
java.lang.NoClassDefFoundError:
org/apache/commons/compress/archivers/tar/TarArchiveInputStream
at
org.apache.hadoop.fs.RawLocalFileSystem$DeprecatedRawLocalFileStatus.loadPermissionInfoByNonNativeIO(RawLocalFileSystem.java:864)
at
org.apache.hadoop.fs.RawLocalFileSystem$DeprecatedRawLocalFileStatus.loadPermissionInfo(RawLocalFileSystem.java:855)
at
org.apache.hadoop.fs.RawLocalFileSystem$DeprecatedRawLocalFileStatus.getPermission(RawLocalFileSystem.java:816)
at
org.apache.hadoop.security.alias.KeyStoreProvider.stashOriginalFilePermissions(KeyStoreProvider.java:78)
at
org.apache.hadoop.security.alias.AbstractJavaKeyStoreProvider.locateKeystore(AbstractJavaKeyStoreProvider.java:324)
at
org.apache.hadoop.security.alias.AbstractJavaKeyStoreProvider.<init>(AbstractJavaKeyStoreProvider.java:86)
at
org.apache.hadoop.security.alias.KeyStoreProvider.<init>(KeyStoreProvider.java:49)
at
org.apache.hadoop.security.alias.JavaKeyStoreProvider.<init>(JavaKeyStoreProvider.java:42)
at
org.apache.hadoop.security.alias.JavaKeyStoreProvider.<init>(JavaKeyStoreProvider.java:35)
at
org.apache.hadoop.security.alias.JavaKeyStoreProvider$Factory.createProvider(JavaKeyStoreProvider.java:68)
at
org.apache.hadoop.security.alias.CredentialProviderFactory.getProviders(CredentialProviderFactory.java:91)
at
org.apache.ranger.authorization.hadoop.utils.RangerCredentialProvider.getCredentialProviders(RangerCredentialProvider.java:69)
at
org.apache.ranger.authorization.hadoop.utils.RangerCredentialProvider.getCredentialString(RangerCredentialProvider.java:47)
at
org.apache.ranger.audit.provider.MiscUtil.getCredentialString(MiscUtil.java:455)
at
org.apache.ranger.audit.destination.SolrAuditDestination.getKeyManagers(SolrAuditDestination.java:348)
at
org.apache.ranger.audit.destination.SolrAuditDestination.connect(SolrAuditDestination.java:131)
at
org.apache.ranger.audit.destination.SolrAuditDestination.init(SolrAuditDestination.java:96)
at
org.apache.ranger.audit.provider.AuditProviderFactory.init(AuditProviderFactory.java:183)
at
org.apache.ranger.plugin.service.RangerBasePlugin.init(RangerBasePlugin.java:234)
at
org.apache.ranger.authorization.kafka.authorizer.RangerKafkaAuthorizer.configure(RangerKafkaAuthorizer.java:105)
at
org.apache.ranger.authorization.kafka.authorizer.RangerKafkaAuthorizer.configure(RangerKafkaAuthorizer.java:95)
at
kafka.security.authorizer.AuthorizerWrapper.configure(AuthorizerWrapper.scala:86)
at kafka.server.KafkaServer.$anonfun$startup$11(KafkaServer.scala:352)
at kafka.server.KafkaServer.$anonfun$startup$11$adapted(KafkaServer.scala:352)
at scala.Option.foreach(Option.scala:407)
at kafka.server.KafkaServer.startup(KafkaServer.scala:352)
at kafka.Kafka$.main(Kafka.scala:109)
at kafka.Kafka.main(Kafka.scala)
Caused by: java.lang.ClassNotFoundException:
org.apache.commons.compress.archivers.tar.TarArchiveInputStream
at java.lang.ClassLoader.findClass(ClassLoader.java:530)
at
org.apache.ranger.plugin.classloader.RangerPluginClassLoader$MyClassLoader.findClass(RangerPluginClassLoader.java:316)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at
org.apache.ranger.plugin.classloader.RangerPluginClassLoader.loadClass(RangerPluginClassLoader.java:160)
... 28 more
[2023-05-08 10:15:18,028] INFO [KafkaServer id=1001] shutting down
(kafka.server.KafkaServer) {code}
cp /usr/bigtop/current/hadoop-hdfs-client/lib/commons-compress-1.21.jar
/usr/bigtop/current/kafka-broker/libs
was:
1.Missing hadoop shade package
A temporary solution is to copy the required JAR files from Hadoop
cp /usr/bigtop/current/hadoop-hdfs-client/lib/hadoop-shaded-guava-1.1.1.jar
/usr/bigtop/current/kafka-broker/libs/
[2023-05-06 14:33:22,254] ERROR [KafkaServer id=1001] Fatal error during
KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
java.lang.NoClassDefFoundError:
org/apache/hadoop/thirdparty/com/google/common/base/Preconditions
at
org.apache.hadoop.conf.Configuration$DeprecationDelta.<init>(Configuration.java:430)
at
org.apache.hadoop.conf.Configuration$DeprecationDelta.<init>(Configuration.java:443)
at org.apache.hadoop.conf.Configuration.<clinit>(Configuration.java:525)
at
org.apache.ranger.plugin.service.RangerBasePlugin.<init>(RangerBasePlugin.java:79)
at
org.apache.ranger.authorization.kafka.authorizer.RangerKafkaAuthorizer.configure(RangerKafkaAuthorizer.java:100)
at
org.apache.ranger.authorization.kafka.authorizer.RangerKafkaAuthorizer.configure(RangerKafkaAuthorizer.java:95)
at
kafka.security.authorizer.AuthorizerWrapper.configure(AuthorizerWrapper.scala:86)
at kafka.server.KafkaServer.$anonfun$startup$11(KafkaServer.scala:352)
at
kafka.server.KafkaServer.$anonfun$startup$11$adapted(KafkaServer.scala:352)
at scala.Option.foreach(Option.scala:407)
at kafka.server.KafkaServer.startup(KafkaServer.scala:352)
at kafka.Kafka$.main(Kafka.scala:109)
at kafka.Kafka.main(Kafka.scala)
Caused by: java.lang.ClassNotFoundException:
org.apache.hadoop.thirdparty.com.google.common.base.Preconditions
at java.lang.ClassLoader.findClass(ClassLoader.java:530)
at
org.apache.ranger.plugin.classloader.RangerPluginClassLoader$MyClassLoader.findClass(RangerPluginClassLoader.java:316)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at
org.apache.ranger.plugin.classloader.RangerPluginClassLoader.loadClass(RangerPluginClassLoader.java:160)
... 13 more
https://issues.apache.org/jira/browse/HADOOP-14284
Starting from Hadoop 3, Hadoop uses the shaded Guava library located in the
hadoop-shade jar instead of the original Guava library. However, unlike HBase
or Hive, Kafka does not add the Hadoop lib to its class path during startup.
Therefore, in a big data cluster using Hadoop 3, the HBase and Hive plugins can
start without errors, but Kafka may report errors due to the lack of the
hadoop-shade jar in its class path. As a temporary solution, the necessary jar
files can be copied from Hadoop to Kafka.
2.Missing common-compress package
A temporary solution is to copy the required JAR files from Hadoop."
cp /usr/bigtop/current/hadoop-hdfs-client/lib/commons-compress-1.21.jar
/usr/bigtop/current/kafka-broker/libs
[2023-05-08 10:15:18,025] ERROR [KafkaServer id=1001] Fatal error during
KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
java.lang.NoClassDefFoundError:
org/apache/commons/compress/archivers/tar/TarArchiveInputStream
at
org.apache.hadoop.fs.RawLocalFileSystem$DeprecatedRawLocalFileStatus.loadPermissionInfoByNonNativeIO(RawLocalFileSystem.java:864)
at
org.apache.hadoop.fs.RawLocalFileSystem$DeprecatedRawLocalFileStatus.loadPermissionInfo(RawLocalFileSystem.java:855)
at
org.apache.hadoop.fs.RawLocalFileSystem$DeprecatedRawLocalFileStatus.getPermission(RawLocalFileSystem.java:816)
at
org.apache.hadoop.security.alias.KeyStoreProvider.stashOriginalFilePermissions(KeyStoreProvider.java:78)
at
org.apache.hadoop.security.alias.AbstractJavaKeyStoreProvider.locateKeystore(AbstractJavaKeyStoreProvider.java:324)
at
org.apache.hadoop.security.alias.AbstractJavaKeyStoreProvider.<init>(AbstractJavaKeyStoreProvider.java:86)
at
org.apache.hadoop.security.alias.KeyStoreProvider.<init>(KeyStoreProvider.java:49)
at
org.apache.hadoop.security.alias.JavaKeyStoreProvider.<init>(JavaKeyStoreProvider.java:42)
at
org.apache.hadoop.security.alias.JavaKeyStoreProvider.<init>(JavaKeyStoreProvider.java:35)
at
org.apache.hadoop.security.alias.JavaKeyStoreProvider$Factory.createProvider(JavaKeyStoreProvider.java:68)
at
org.apache.hadoop.security.alias.CredentialProviderFactory.getProviders(CredentialProviderFactory.java:91)
at
org.apache.ranger.authorization.hadoop.utils.RangerCredentialProvider.getCredentialProviders(RangerCredentialProvider.java:69)
at
org.apache.ranger.authorization.hadoop.utils.RangerCredentialProvider.getCredentialString(RangerCredentialProvider.java:47)
at
org.apache.ranger.audit.provider.MiscUtil.getCredentialString(MiscUtil.java:455)
at
org.apache.ranger.audit.destination.SolrAuditDestination.getKeyManagers(SolrAuditDestination.java:348)
at
org.apache.ranger.audit.destination.SolrAuditDestination.connect(SolrAuditDestination.java:131)
at
org.apache.ranger.audit.destination.SolrAuditDestination.init(SolrAuditDestination.java:96)
at
org.apache.ranger.audit.provider.AuditProviderFactory.init(AuditProviderFactory.java:183)
at
org.apache.ranger.plugin.service.RangerBasePlugin.init(RangerBasePlugin.java:234)
at
org.apache.ranger.authorization.kafka.authorizer.RangerKafkaAuthorizer.configure(RangerKafkaAuthorizer.java:105)
at
org.apache.ranger.authorization.kafka.authorizer.RangerKafkaAuthorizer.configure(RangerKafkaAuthorizer.java:95)
at
kafka.security.authorizer.AuthorizerWrapper.configure(AuthorizerWrapper.scala:86)
at kafka.server.KafkaServer.$anonfun$startup$11(KafkaServer.scala:352)
at
kafka.server.KafkaServer.$anonfun$startup$11$adapted(KafkaServer.scala:352)
at scala.Option.foreach(Option.scala:407)
at kafka.server.KafkaServer.startup(KafkaServer.scala:352)
at kafka.Kafka$.main(Kafka.scala:109)
at kafka.Kafka.main(Kafka.scala)
Caused by: java.lang.ClassNotFoundException:
org.apache.commons.compress.archivers.tar.TarArchiveInputStream
at java.lang.ClassLoader.findClass(ClassLoader.java:530)
at
org.apache.ranger.plugin.classloader.RangerPluginClassLoader$MyClassLoader.findClass(RangerPluginClassLoader.java:316)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at
org.apache.ranger.plugin.classloader.RangerPluginClassLoader.loadClass(RangerPluginClassLoader.java:160)
... 28 more
[2023-05-08 10:15:18,028] INFO [KafkaServer id=1001] shutting down
(kafka.server.KafkaServer)
> Kafka2.8 can't restart after enable ranger plugin
> -------------------------------------------------
>
> Key: RANGER-4228
> URL: https://issues.apache.org/jira/browse/RANGER-4228
> Project: Ranger
> Issue Type: Bug
> Components: Ranger
> Affects Versions: 2.3.0, 2.4.0
> Reporter: caijialiang
> Priority: Major
>
> 1.Missing hadoop shade package
> A temporary solution is to copy the required JAR files from Hadoop
>
> cp /usr/bigtop/current/hadoop-hdfs-client/lib/hadoop-shaded-guava-1.1.1.jar
> /usr/bigtop/current/kafka-broker/libs/
> {code:java}
> [2023-05-06 14:33:22,254] ERROR [KafkaServer id=1001] Fatal error during
> KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
> java.lang.NoClassDefFoundError:
> org/apache/hadoop/thirdparty/com/google/common/base/Preconditions
> at
> org.apache.hadoop.conf.Configuration$DeprecationDelta.<init>(Configuration.java:430)
> at
> org.apache.hadoop.conf.Configuration$DeprecationDelta.<init>(Configuration.java:443)
> at org.apache.hadoop.conf.Configuration.<clinit>(Configuration.java:525)
> at
> org.apache.ranger.plugin.service.RangerBasePlugin.<init>(RangerBasePlugin.java:79)
> at
> org.apache.ranger.authorization.kafka.authorizer.RangerKafkaAuthorizer.configure(RangerKafkaAuthorizer.java:100)
> at
> org.apache.ranger.authorization.kafka.authorizer.RangerKafkaAuthorizer.configure(RangerKafkaAuthorizer.java:95)
> at
> kafka.security.authorizer.AuthorizerWrapper.configure(AuthorizerWrapper.scala:86)
> at kafka.server.KafkaServer.$anonfun$startup$11(KafkaServer.scala:352)
> at kafka.server.KafkaServer.$anonfun$startup$11$adapted(KafkaServer.scala:352)
> at scala.Option.foreach(Option.scala:407)
> at kafka.server.KafkaServer.startup(KafkaServer.scala:352)
> at kafka.Kafka$.main(Kafka.scala:109)
> at kafka.Kafka.main(Kafka.scala)
> Caused by: java.lang.ClassNotFoundException:
> org.apache.hadoop.thirdparty.com.google.common.base.Preconditions
> at java.lang.ClassLoader.findClass(ClassLoader.java:530)
> at
> org.apache.ranger.plugin.classloader.RangerPluginClassLoader$MyClassLoader.findClass(RangerPluginClassLoader.java:316)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at
> org.apache.ranger.plugin.classloader.RangerPluginClassLoader.loadClass(RangerPluginClassLoader.java:160)
> ... 13 more {code}
> https://issues.apache.org/jira/browse/HADOOP-14284
> Starting from Hadoop 3, Hadoop uses the shaded Guava library located in the
> hadoop-shade jar instead of the original Guava library. However, unlike HBase
> or Hive, Kafka does not add the Hadoop lib to its class path during startup.
> Therefore, in a big data cluster using Hadoop 3, the HBase and Hive plugins
> can start without errors, but Kafka may report errors due to the lack of the
> hadoop-shade jar in its class path. As a temporary solution, the necessary
> jar files can be copied from Hadoop to Kafka.
>
> 2.Missing common-compress package
> A temporary solution is to copy the required JAR files from Hadoop."
> {code:java}
> [2023-05-08 10:15:18,025] ERROR [KafkaServer id=1001] Fatal error during
> KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
> java.lang.NoClassDefFoundError:
> org/apache/commons/compress/archivers/tar/TarArchiveInputStream
> at
> org.apache.hadoop.fs.RawLocalFileSystem$DeprecatedRawLocalFileStatus.loadPermissionInfoByNonNativeIO(RawLocalFileSystem.java:864)
> at
> org.apache.hadoop.fs.RawLocalFileSystem$DeprecatedRawLocalFileStatus.loadPermissionInfo(RawLocalFileSystem.java:855)
> at
> org.apache.hadoop.fs.RawLocalFileSystem$DeprecatedRawLocalFileStatus.getPermission(RawLocalFileSystem.java:816)
> at
> org.apache.hadoop.security.alias.KeyStoreProvider.stashOriginalFilePermissions(KeyStoreProvider.java:78)
> at
> org.apache.hadoop.security.alias.AbstractJavaKeyStoreProvider.locateKeystore(AbstractJavaKeyStoreProvider.java:324)
> at
> org.apache.hadoop.security.alias.AbstractJavaKeyStoreProvider.<init>(AbstractJavaKeyStoreProvider.java:86)
> at
> org.apache.hadoop.security.alias.KeyStoreProvider.<init>(KeyStoreProvider.java:49)
> at
> org.apache.hadoop.security.alias.JavaKeyStoreProvider.<init>(JavaKeyStoreProvider.java:42)
> at
> org.apache.hadoop.security.alias.JavaKeyStoreProvider.<init>(JavaKeyStoreProvider.java:35)
> at
> org.apache.hadoop.security.alias.JavaKeyStoreProvider$Factory.createProvider(JavaKeyStoreProvider.java:68)
> at
> org.apache.hadoop.security.alias.CredentialProviderFactory.getProviders(CredentialProviderFactory.java:91)
> at
> org.apache.ranger.authorization.hadoop.utils.RangerCredentialProvider.getCredentialProviders(RangerCredentialProvider.java:69)
> at
> org.apache.ranger.authorization.hadoop.utils.RangerCredentialProvider.getCredentialString(RangerCredentialProvider.java:47)
> at
> org.apache.ranger.audit.provider.MiscUtil.getCredentialString(MiscUtil.java:455)
> at
> org.apache.ranger.audit.destination.SolrAuditDestination.getKeyManagers(SolrAuditDestination.java:348)
> at
> org.apache.ranger.audit.destination.SolrAuditDestination.connect(SolrAuditDestination.java:131)
> at
> org.apache.ranger.audit.destination.SolrAuditDestination.init(SolrAuditDestination.java:96)
> at
> org.apache.ranger.audit.provider.AuditProviderFactory.init(AuditProviderFactory.java:183)
> at
> org.apache.ranger.plugin.service.RangerBasePlugin.init(RangerBasePlugin.java:234)
> at
> org.apache.ranger.authorization.kafka.authorizer.RangerKafkaAuthorizer.configure(RangerKafkaAuthorizer.java:105)
> at
> org.apache.ranger.authorization.kafka.authorizer.RangerKafkaAuthorizer.configure(RangerKafkaAuthorizer.java:95)
> at
> kafka.security.authorizer.AuthorizerWrapper.configure(AuthorizerWrapper.scala:86)
> at kafka.server.KafkaServer.$anonfun$startup$11(KafkaServer.scala:352)
> at kafka.server.KafkaServer.$anonfun$startup$11$adapted(KafkaServer.scala:352)
> at scala.Option.foreach(Option.scala:407)
> at kafka.server.KafkaServer.startup(KafkaServer.scala:352)
> at kafka.Kafka$.main(Kafka.scala:109)
> at kafka.Kafka.main(Kafka.scala)
> Caused by: java.lang.ClassNotFoundException:
> org.apache.commons.compress.archivers.tar.TarArchiveInputStream
> at java.lang.ClassLoader.findClass(ClassLoader.java:530)
> at
> org.apache.ranger.plugin.classloader.RangerPluginClassLoader$MyClassLoader.findClass(RangerPluginClassLoader.java:316)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at
> org.apache.ranger.plugin.classloader.RangerPluginClassLoader.loadClass(RangerPluginClassLoader.java:160)
> ... 28 more
> [2023-05-08 10:15:18,028] INFO [KafkaServer id=1001] shutting down
> (kafka.server.KafkaServer) {code}
> cp /usr/bigtop/current/hadoop-hdfs-client/lib/commons-compress-1.21.jar
> /usr/bigtop/current/kafka-broker/libs
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
