does your spark version come with batteries (hadoop included) or is it build with hadoop provided and you are adding hadoop binaries to classpath
On Wed, Feb 24, 2016 at 3:08 PM, <ross.cramb...@thomsonreuters.com> wrote: > I’m trying to save a data frame in Avro format but am getting the > following error: > > > java.lang.NoSuchMethodError: > org.apache.avro.generic.GenericData.createDatumWriter(Lorg/apache/avro/Schema;)Lorg/apache/avro/io/DatumWriter; > > I found the following workaround > https://github.com/databricks/spark-avro/issues/91 - which seems to say > that this is from a mismatch in Avro versions. I have tried following both > solutions detailed to no avail: > - Manually downloading avro-1.7.7.jar and including it in > /usr/lib/hadoop-mapreduce/ > - Adding avro-1.7.7.jar to spark.driver.extraClassPath and > spark.executor.extraClassPath > - The same with avro-1.6.6 > > I am still getting the same error, and now I am just stabbing in the dark. > Anyone else still running into this issue? > > > I am using Pyspark 1.5.2 on EMR. >
