I believe the main issue here is that spark.jars is a bit "too late" to actually prepend things to the class path. For most use cases this value is not read until after the JVM has already started and the system classloader has already loaded.
The jar argument gets added via the dynamic class loader so it necessarily has to come after wards :/ Driver extra classpath and it's friends, modify the actual launch command of the driver (or executors) so they can prepend whenever they want. In general you do not want to have conflicting jars at all if possible and I would recommend looking into shading if it's really important for your application to use a specific incompatible version of a library. Jar (and extraClasspath) are really just for adding additional jars and I personally would try not to rely on classpath ordering to get the right libraries recognized. On Thu, Jul 16, 2020 at 1:55 PM Nupur Shukla <nupur14shu...@gmail.com> wrote: > Hello, > > How can we use *spark.jars* to to specify conflicting jars (that is, jars > that are already present in the spark's default classpath)? Jars specified > in this conf gets "appended" to the classpath, and thus gets looked at > after the default classpath. Is it not intended to be used to specify > conflicting jars? > Meanwhile when *spark.driver.extraClassPath* conf is specified, this path > is "prepended" to the classpath and thus takes precedence over the default > classpath. > > How can I use both to specify different jars and paths but achieve a > precedence of spark.jars path > spark.driver.extraClassPath > spark default > classpath (left to right precedence order)? > > Experiment conducted: > > I am using sample-project.jar which has one class in it SampleProject. > This has a method which prints the version number of the jar. For this > experiment I am using 3 versions of this sample-project.jar > Sample-project-1.0.0.jar is present in the spark default classpath in my > test cluster > Sample-project-2.0.0.jar is present in folder /home/<user>/ClassPathConf > on driver > Sample-project-3.0.0.jar is present in folder /home/<user>/JarsConf on > driver > > (Empty cell in img below means that conf was not specified) > > [image: image.png] > > > Thank you, > Nupur > > >
