HeartSaVioR edited a comment on issue #21339: [SPARK-24287][Core] Spark -packages option should support classifier, no-transitive, and custom conf URL: https://github.com/apache/spark/pull/21339#issuecomment-476107562 I've pulled this patch and played with functionalities, and works well. I've used hive-exec which has default classifier as well as `core` classifier: hive-exec with default classifier includes some dependencies into jar (`org.json:json` is included) which `core` classifier contains it as dependency (not included in jar). > ./bin/spark-shell ``` scala> import org.json.JSON <console>:24: error: object json is not a member of package org import org.json.JSON ``` Not able to load since it's not in default Spark classpath. (expected) > ./bin/spark-shell --packages "org.apache.hive:hive-exec:3.1.1?classifier=core" ``` scala> import org.json.JSON import org.json.JSON ``` With classifier it properly loads it as transitive dependencies. (works well) > ./bin/spark-shell --packages "org.apache.hive:hive-exec:3.1.1?classifier=core&transitive=false" ``` scala> import org.json.JSON <console>:23: error: object json is not a member of package org import org.json.JSON ``` It ignores whole transitive dependencies, hence `org.json:json` is not pulled. (works well) > ./bin/spark-shell --packages "org.apache.hive:hive-exec:3.1.1?classifier=core&exclude=json" ``` scala> import org.json.JSON <console>:23: error: object json is not a member of package org import org.json.JSON ``` It excludes `org.json:json` from transitive dependencies. (works well) > ./bin/spark-shell --packages "org.apache.hive:hive-exec:3.1.1" ``` scala> import org.json.JSON import org.json.JSON ``` The jar itself contains `org.json:json` so should be able to load. (This represents classifier works as expected.) > ./bin/spark-shell --packages "org.apache.hive:hive-exec:3.1.1?exclude=json" ``` scala> import org.json.JSON import org.json.JSON ``` Even we exclude `org.json:json` it's still able to load, since the jar contains it. (This represents classifier works as expected.) FYI I had to fix scalastyle before building. I'll start reviewing the code and leave comments. cc. @srowen @gaborgsomogyi You might be interested on this PR.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
