[
https://issues.apache.org/jira/browse/RANGER-3595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17480801#comment-17480801
]
kirby zhou edited comment on RANGER-3595 at 1/24/22, 3:27 AM:
--------------------------------------------------------------
[~pradeep]
The output of security-admin is a .WAR, it have the correct struct of a webapp.
{code:java}
WEB-INF/classes/org/apache/...
WEB-INF/web.xml
WEB-INF/lib{code}
But the output of kms is .JAR, its struct looks like:
{code:java}
org/apache/... # it wont be loaded by any.
WEB-INF/web.xml # which is a dirty workaround by somebody {code}
Both kms.jar and admin.war are unpacked under 'ews/webapp', but the unpacked
kms classes are meaningless.
Actually, kms load its own class by './ews/webapp/lib/ranger-kms-*.jar',
[https://github.com/apache/ranger/blob/master/distro/src/main/assembly/kms.xml#L30]
{code:java}
<moduleSet>
<useAllReactorProjects>true</useAllReactorProjects>
<includes>
<include>org.apache.ranger:ranger-kms</include>
</includes>
<binaries>
<outputDirectory>ews/webapp/lib</outputDirectory>
<includeDependencies>true</includeDependencies>
<unpack>false</unpack>
{code}
I have tried to convert kms to WAR, so the unpacked classes can be meaningful.
But it is difficult to fix the relation of ClassLoader. I have tried putting
kms classes under ews/webapp/WEB-INF/classes, startup failure happens.
The classes under ( ews/lib/{*}, ews/webapp/lib/{*},
ews/webapp/WEB-INF/classes/lib/* ) and ( webapp/WEB-INF/lib/* ,
webapp/WEB-INF/classes ) are loaded by different ClassLoader. the first group
is bootstrap and loaded by system ClassLoader, the followings are webapp loaded
by Tomcat WebApp ClassLoader.
We place too many jars under ews/webapp/lib which is a mixture of bootstrap and
webapp. It makes something like ServiceLoader works strange. kms.xml rewrite
the whole dependencies of KMS to reduce the package size and mix bootstrap and
webapp jars together. It is hard to classifying which jars should be bootstrap
and which are webapp now.
The situation of security-admin is much better, bootstrap jars are placed under
ews/lib, the jars of admin webapp are placed under ews/webapp/WEB-INF/lib.
There are no jars under ews/webapp/lib.
So I have a simple patch to workaround, just put ranger-kms.jar under
webapp/lib and remove webapp/org/* . It works and pretty than current master.
was (Author: kirbyzhou):
[~pradeep]
The output of security-admin is a .WAR, it have the correct struct of a webapp.
{code:java}
WEB-INF/classes/org/apache/...
WEB-INF/web.xml
WEB-INF/lib{code}
But the output of kms is .JAR, its struct looks like:
{code:java}
org/apache/... # it wont be loaded by any.
WEB-INF/web.xml # which is a dirty workaround by somebody {code}
Both kms.jar and admin.war are unpacked under 'ews/webapp', but the unpacked
kms classes are meaningless.
Actually, kms load its own class by './ews/webapp/lib/ranger-kms-*.jar',
[https://github.com/apache/ranger/blob/master/distro/src/main/assembly/kms.xml#L30]
{code:java}
<moduleSet>
<useAllReactorProjects>true</useAllReactorProjects>
<includes>
<include>org.apache.ranger:ranger-kms</include>
</includes>
<binaries>
<outputDirectory>ews/webapp/lib</outputDirectory>
<includeDependencies>true</includeDependencies>
<unpack>false</unpack>
{code}
I have tried to convert kms to WAR, so the unpacked classes can be meaningful.
But it is difficult to fix the relation of ClassLoader.
The classes under ( ews/lib/{*}, ews/webapp/lib/{*},
ews/webapp/WEB-INF/classes/lib/* ) and ( webapp/WEB-INF/lib/* ,
webapp/WEB-INF/classes ) are loaded by different ClassLoader. the first group
is system ClassLoader, the followings are Tomcat WebApp ClassLoader.
We place too many jars out of webapp/WEB-INF/lib, it makes something like
ServiceLoader works strange. So I have a simple patch to workaround, put all
classes / jar out of webapp/WEB-INF/lib. It works and pretty than current
master.
> Tar of KMS contains rubbish files
> ---------------------------------
>
> Key: RANGER-3595
> URL: https://issues.apache.org/jira/browse/RANGER-3595
> Project: Ranger
> Issue Type: Improvement
> Components: kms
> Reporter: kirby zhou
> Priority: Major
>
> There are lots of .class files under ews/webapp/. They wont be loaded by any
> classpath. And they are duplicated against files inside
> ews/webapp/lib/ranger-kms-3.0.0-SNAPSHOT.jar.
> It seems dirty and may cause some security problem.
> {code:bash}
> #] tar tf target/ranger-3.0.0-SNAPSHOT-kms.tar.gz ranger-3.0.0-SNAPSHOT-kms/
> | egrep 'ews/webapp/org' | head
> ranger-3.0.0-SNAPSHOT-kms/ews/webapp/org/
> ranger-3.0.0-SNAPSHOT-kms/ews/webapp/org/apache/
> ranger-3.0.0-SNAPSHOT-kms/ews/webapp/org/apache/ranger/
> ranger-3.0.0-SNAPSHOT-kms/ews/webapp/org/apache/ranger/kms/
> ranger-3.0.0-SNAPSHOT-kms/ews/webapp/org/apache/ranger/kms/biz/
> ranger-3.0.0-SNAPSHOT-kms/ews/webapp/org/apache/ranger/kms/dao/
> ranger-3.0.0-SNAPSHOT-kms/ews/webapp/org/apache/ranger/entity/
> ranger-3.0.0-SNAPSHOT-kms/ews/webapp/org/apache/hadoop/
> ranger-3.0.0-SNAPSHOT-kms/ews/webapp/org/apache/hadoop/crypto/
> ranger-3.0.0-SNAPSHOT-kms/ews/webapp/org/apache/hadoop/crypto/key/
> //代码占位符
> {code}
> * The reason is that:
> distro/src/main/assembly/kms.xml
>
> {code:java}
> <moduleSet>
> <useAllReactorProjects>true</useAllReactorProjects>
> <includes>
> <include>org.apache.ranger:ranger-kms</include>
> </includes>
> <binaries>
> <outputDirectory>ews/webapp</outputDirectory>
> <includeDependencies>false</includeDependencies>
> <unpack>true</unpack>
> </binaries>
> </moduleSet> {code}
> Why ?
>
> The secret is in kms/scripts/setup.sh:
>
> {code:java}
> setup_kms(){
> #copying ranger kms provider
> oldP=${PWD}
> cd $PWD/ews/webapp
> log "[I] Adding ranger kms provider as services in hadoop-common jar"
> for f in lib/hadoop-common*.jar
> do
> ${JAVA_HOME}/bin/jar -uf ${f}
> META-INF/services/org.apache.hadoop.crypto.key.KeyProviderFactory
> chown ${unix_user}:${unix_group} ${f}
> done
> cd ${oldP}
> }
> {code}
>
>
> The code above is VERY VERY DIRTY!
> It hacks into hadoop-common.jar., Overwrite resource
> "META-INF/services/org.apache.hadoop.crypto.key.KeyProviderFactory". Ensure
> the following code can load
> 'org.apache.hadoop.crypto.key.RangerKeyStoreProvider$Factory' by
> 'META-INF/.../KeyProviderFactory'.
>
>
> {code:java}
> // org.apache.hadoop.crypto.key: KeyProviderFactory.java
> private static final ServiceLoader<KeyProviderFactory> serviceLoader =
> ServiceLoader.load(KeyProviderFactory.class,
> KeyProviderFactory.class.getClassLoader());
> {code}
>
>
> But this is unnecessary.
> ServiceLoader will read all resources with the same name using the
> ClassLoader of KeyProviderFactory. We just need to put a jar contains that
> property side by side of hadoop-common.jar ( ews/webapp/lib/ ). And
> ranger-kms-3.0.0-SNAPSHOT.jar already here.
> {code:java}
> % tar tf ../target/ranger-*-kms.tar.gz | egrep 'kms[^/]*\.jar|hadoop-common'
> ranger-3.0.0-SNAPSHOT-kms/ews/webapp/lib/ranger-kms-3.0.0-SNAPSHOT.jar
> ranger-3.0.0-SNAPSHOT-kms/ews/webapp/lib/hadoop-common-3.3.0.jar
> ...
> % tar tf target/ranger-kms-3.0.0-SNAPSHOT.jar | fgrep ProviderFactory
> META-INF/services/org.apache.hadoop.crypto.key.KeyProviderFactory
> {code}
>
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
