HeartSaVioR edited a comment on issue #21339: [SPARK-24287][Core] Spark -packages option should support classifier, no-transitive, and custom conf URL: https://github.com/apache/spark/pull/21339#issuecomment-476462142 Have been thinking this more, and while I still feel supporting `exclude` would not hurt much (for complexity, maintenance cost), it may not be necessary on the simple cases cause Spark libs will be added to the classpath prior to transitive dependencies being pulled by `--packages`. Suppose the case there's conflict between Spark libs and transitive dependencies which would make some problems: 1) Spark's point of view: Spark will not be bothered by transitive dependencies if it works as expected 2) dependency's point of view: dependency will be running into problem, but excluding from dependency artifact side will not help since the problematic dependency will be retained in Spark side. So `exclude` (and others) more suitable when pulling multiple artifacts with `--packages` (and also `--jars`) which transitive dependencies of them should be arranged. If we would want to let `--packages` only handle simple cases it may be OK, but as we know it wouldn't work for cases which fine-grained control of transitive dependencies is required (even this case `exclude` only came in my experience, others weren't).
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
