i have found no way around this. basically this makes SPARK_CLASSPATH unusable. and the alternative for enabling lzo on a cluster is not reasonable.
one has to set in spark-defaults.conf: spark.executor.extraClassPath /usr/lib/hadoop/lib/hadoop-lzo-cdh4-0.4.15-gplextras.jar spark.executor.extraLibraryPath /usr/lib/hadoop/lib/native/Linux-amd64-64:/usr/lib/hadoop/lib/native and then for every job submitted one has to do using spark-submit (i get errors if i leave this out): --driver-class-path /usr/lib/hadoop/lib/hadoop-lzo-cdh4-0.4.15-gplextras.jar --driver-library-path /usr/lib/hadoop/lib/native/Linux-amd64-64:/usr/lib/hadoop/lib/native i think the settings in spark-defaults.conf are fine, but expecting the user to set the driver-class-path and driver-library-path like that is cumbersome and requires knowledge of locations of certain files on the cluster. thats not workable to me. On Mon, Sep 15, 2014 at 11:16 AM, Koert Kuipers <ko...@tresata.com> wrote: > in spark 1.1.0 i get this error: > > 2014-09-14 23:17:01 ERROR actor.OneForOneStrategy: Found both > spark.driver.extraClassPath and SPARK_CLASSPATH. Use only the former. > > i checked my application. i do not set spark.driver.extraClassPath or > SPARK_CLASSPATH. > > SPARK_CLASSPATH is set in spark-env.sh since the machine is a worker and > this is how lzo is added to classpath. as a user i cannot modify this. > > looking at the logs i see the value of SPARK_CLASSPATH ends up in > spark.driver.extraClassPath, but thats not done by me. i presume some logic > in spark-submit does this. > >
