No, these are different major versions of these components, each of which gets used by something in the transitive dependency graph. They are not redundant because they're not actually presenting roughly the same component in the same namespace.
However the parquet-hadoop bit looks wrong, in that it should be harmonized to one 1.x version. It's not that Spark uses inconsistent versions but that transitive deps do. We can still harmonize them in the build if it causes problems. On Sat, Sep 17, 2016 at 8:14 PM, Jacek Laskowski <ja...@japila.pl> wrote: > Hi, > > Just noticed in assembly/target/scala-2.11/jars that similar libraries > have different versions: > > -rw-r--r-- 1 jacek staff 1230201 17 wrz 09:51 netty-3.8.0.Final.jar > -rw-r--r-- 1 jacek staff 2305335 17 wrz 09:51 netty-all-4.0.41.Final.jar > > and > > -rw-r--r-- 1 jacek staff 218076 17 wrz 09:51 parquet-hadoop-1.8.1.jar > -rw-r--r-- 1 jacek staff 2796935 17 wrz 09:51 > parquet-hadoop-bundle-1.6.0.jar > > and > > -rw-r--r-- 1 jacek staff 46983 17 wrz 09:51 > jackson-annotations-2.6.5.jar > -rw-r--r-- 1 jacek staff 258876 17 wrz 09:51 jackson-core-2.6.5.jar > -rw-r--r-- 1 jacek staff 232248 17 wrz 09:51 jackson-core-asl-1.9.13.jar > -rw-r--r-- 1 jacek staff 1171380 17 wrz 09:51 jackson-databind-2.6.5.jar > -rw-r--r-- 1 jacek staff 18336 17 wrz 09:51 jackson-jaxrs-1.9.13.jar > -rw-r--r-- 1 jacek staff 780664 17 wrz 09:51 > jackson-mapper-asl-1.9.13.jar > -rw-r--r-- 1 jacek staff 41263 17 wrz 09:51 > jackson-module-paranamer-2.6.5.jar > -rw-r--r-- 1 jacek staff 515604 17 wrz 09:51 > jackson-module-scala_2.11-2.6.5.jar > -rw-r--r-- 1 jacek staff 27084 17 wrz 09:51 jackson-xc-1.9.13.jar > > and > > -rw-r--r-- 1 jacek staff 188671 17 wrz 09:51 commons-beanutils-1.7.0.jar > -rw-r--r-- 1 jacek staff 206035 17 wrz 09:51 > commons-beanutils-core-1.8.0.jar > > and > > -rw-r--r-- 1 jacek staff 445288 17 wrz 09:51 antlr-2.7.7.jar > -rw-r--r-- 1 jacek staff 164368 17 wrz 09:51 antlr-runtime-3.4.jar > -rw-r--r-- 1 jacek staff 302248 17 wrz 09:51 antlr4-runtime-4.5.3.jar > > Even if that does not cause any class mismatches, it might be worth to > exclude them to minimize the size of the Spark distro. > > What do you think? > > Pozdrawiam, > Jacek Laskowski > ---- > https://medium.com/@jaceklaskowski/ > Mastering Apache Spark 2.0 http://bit.ly/mastering-apache-spark > Follow me at https://twitter.com/jaceklaskowski > > --------------------------------------------------------------------- > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org > --------------------------------------------------------------------- To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
