Fat jar with shading as the application not as an additional jar package
> Am 18.03.2019 um 14:08 schrieb Jörn Franke <jornfra...@gmail.com>: > > Maybe that class is already loaded as part of a core library of Spark? > > Do you have concrete class names? > > In doubt create a fat jar and shade the dependencies in question > >> Am 18.03.2019 um 12:34 schrieb Federico D'Ambrosio <fedex...@gmail.com>: >> >> Hello everyone, >> >> We're having a serious issue, where we get ClassNotFoundException because, >> apparently the class is not found within the classpath of Spark, in both the >> Driver and Executors. >> >> First, I checked whether the class was actually within the jar with jar tf, >> and there actually is. Then, I activated the following options to see which >> classes are actually loaded: >> >> --conf 'spark.driver.extraJavaOptions=-verbose:class' --conf >> 'spark.executor.extraJavaOptions=-verbose:class' >> >> and I can see from the YARN stdout logs that some classes, just like the one >> throwing the exception, are not actually being loaded while other classes >> are. >> I tried, then, using --jars to pass the jar containing the missing classes, >> and also using addJar() from the spark context, to no avail. >> >> This looks like an issue with Spark class loader. >> >> Any idea about what's happenig here? I'm using Spark 2.3.1.3.0.0.0-1634 (HDP >> 3.0). >> >> Thank you for your help, >> Federico
