gumartinm commented on pull request #319: URL: https://github.com/apache/submarine/pull/319#issuecomment-647705165
> I have checked the spark 2.4 maven dependencies > > https://github.com/apache/spark/blob/47a73b2baac38ff2fa4e0aad63a79891bbce7cf0/dev/deps/spark-deps-hadoop-2.6#L94 > > Anything I missed? Its package is shaded because of this configuration: https://github.com/apache/submarine/blob/master/submarine-security/spark-security/pom.xml#L389 If it is not included in the uber jar with its shaed package, submarine throws this exception (because it is trying to load the shaded package): ` java.lang.NoClassDefFoundError: submarine_spark_ranger_project/org/codehaus/jackson/jaxrs/JacksonJsonProvider at org.apache.ranger.plugin.util.RangerRESTClient.buildClient(RangerRESTClient.java:209) at org.apache.ranger.plugin.util.RangerRESTClient.getClient(RangerRESTClient.java:176) at org.apache.ranger.plugin.util.RangerRESTClient.getResource(RangerRESTClient.java:156) at org.apache.ranger.admin.client.RangerAdminRESTClient.createWebResource(RangerAdminRESTClient.java:635) at org.apache.ranger.admin.client.RangerAdminRESTClient.getServicePoliciesIfUpdated(RangerAdminRESTClient.java:142) at org.apache.ranger.plugin.util.PolicyRefresher.loadPolicyfromPolicyAdmin(PolicyRefresher.java:251) at org.apache.ranger.plugin.util.PolicyRefresher.loadPolicy(PolicyRefresher.java:191) at org.apache.ranger.plugin.util.PolicyRefresher.startRefresher(PolicyRefresher.java:137) at org.apache.ranger.plugin.service.RangerBasePlugin.init(RangerBasePlugin.java:229) at org.apache.submarine.spark.security.RangerSparkPlugin$.init(RangerSparkPlugin.scala:43) at org.apache.submarine.spark.security.RangerSparkPlugin$.<init>(RangerSparkPlugin.scala:64) at org.apache.submarine.spark.security.RangerSparkPlugin$.<clinit>(RangerSparkPlugin.scala) at org.apache.submarine.spark.security.RangerSparkAuthorizer$.org$apache$submarine$spark$security$RangerSparkAuthorizer$$getSparkResource(RangerSparkAuthorizer.scala:257) at org.apache.submarine.spark.security.RangerSparkAuthorizer$$anonfun$addAccessRequest$1$1.apply(RangerSparkAuthorizer.scala:75) at org.apache.submarine.spark.security.RangerSparkAuthorizer$$anonfun$addAccessRequest$1$1.apply(RangerSparkAuthorizer.scala:74) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48) at org.apache.submarine.spark.security.RangerSparkAuthorizer$.addAccessRequest$1(RangerSparkAuthorizer.scala:74) at org.apache.submarine.spark.security.RangerSparkAuthorizer$.checkPrivileges(RangerSparkAuthorizer.scala:98) at org.apache.spark.sql.catalyst.optimizer.SubmarineSparkRangerAuthorizationExtension.apply(SubmarineSparkRangerAuthorizationExtension.scala:64) at org.apache.spark.sql.catalyst.optimizer.SubmarineSparkRangerAuthorizationExtension.apply(SubmarineSparkRangerAuthorizationExtension.scala:39) at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1$$anonfun$apply$1.apply(RuleExecutor.scala:87) at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1$$anonfun$apply$1.apply(RuleExecutor.scala:84) at scala.collection.LinearSeqOptimized$class.foldLeft(LinearSeqOptimized.scala:124) at scala.collection.immutable.List.foldLeft(List.scala:84) at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1.apply(RuleExecutor.scala:84) at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1.apply(RuleExecutor.scala:76) at scala.collection.immutable.List.foreach(List.scala:392) at org.apache.spark.sql.catalyst.rules.RuleExecutor.execute(RuleExecutor.scala:76) at org.apache.spark.sql.execution.QueryExecution.optimizedPlan$lzycompute(QueryExecution.scala:67) at org.apache.spark.sql.execution.QueryExecution.optimizedPlan(QueryExecution.scala:67) at org.apache.spark.sql.execution.QueryExecution.sparkPlan$lzycompute(QueryExecution.scala:73) at org.apache.spark.sql.execution.QueryExecution.sparkPlan(QueryExecution.scala:69) at org.apache.spark.sql.execution.QueryExecution.executedPlan$lzycompute(QueryExecution.scala:78) at org.apache.spark.sql.execution.QueryExecution.executedPlan(QueryExecution.scala:78) at org.apache.spark.sql.Dataset.withAction(Dataset.scala:3365) at org.apache.spark.sql.Dataset.<init>(Dataset.scala:194) at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:79) at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:642) at org.apache.spark.sql.SQLContext.sql(SQLContext.scala:694) Caused by: java.lang.ClassNotFoundException: submarine_spark_ranger_project.org.codehaus.jackson.jaxrs.JacksonJsonProvider at java.net.URLClassLoader.findClass(URLClassLoader.java:382) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ` I guess, we have two options (not sure if there could be some other solution): **a)** We could remove this line and package would not be shaded: https://github.com/apache/submarine/blob/master/submarine-security/spark-security/pom.xml#L389 **b)** Apply this PR and make `org.codehaus.jackson:jackson-jaxrs` be included in the uber jar. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
