Dagang Wei created SPARK-26093:
----------------------------------
Summary: Read Avro: ClassNotFoundException:
org.apache.spark.sql.avro.AvroFileFormat.DefaultSource
Key: SPARK-26093
URL: https://issues.apache.org/jira/browse/SPARK-26093
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 2.4.0
Environment: Spark 2.4.0
Scala 2.11.12
Java 1.8.0_181
Reporter: Dagang Wei
I downloaded and unpacked spark-2.4.0-bin-hadoop2.7.tgz to my Linux, then I
followed [Read Avro
files|https://docs.databricks.com/spark/latest/data-sources/read-avro.html] to
read a local Avro file in spark-shell:
$ bin/spark-shell --packages com.databricks:spark-avro_2.11:4.0.0
...
version 2.4.0
Using Scala version 2.11.12 (OpenJDK 64-Bit Server VM, Java 1.8.0_181)
...
scala>
import com.databricks.spark.avro._
scala>
val df = spark.read.avro("file:///.../foo.avro")
java.lang.ClassNotFoundException: Failed to find data source:
org.apache.spark.sql.avro.AvroFileFormat. Please find packages at
http://spark.apache.org/third-party-projects.html
at
org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSource(DataSource.scala:657)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:194)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:178)
at
com.databricks.spark.avro.package$AvroDataFrameReader$$anonfun$avro$2.apply(package.scala:34)
at
com.databricks.spark.avro.package$AvroDataFrameReader$$anonfun$avro$2.apply(package.scala:34)
... 51 elided
Caused by: java.lang.ClassNotFoundException:
org.apache.spark.sql.avro.AvroFileFormat.DefaultSource
at
scala.reflect.internal.util.AbstractFileClassLoader.findClass(AbstractFileClassLoader.scala:62)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at
org.apache.spark.sql.execution.datasources.DataSource$$anonfun$20$$anonfun$apply$12.apply(DataSource.scala:634)
at
org.apache.spark.sql.execution.datasources.DataSource$$anonfun$20$$anonfun$apply$12.apply(DataSource.scala:634)
at scala.util.Try$.apply(Try.scala:192)
at
org.apache.spark.sql.execution.datasources.DataSource$$anonfun$20.apply(DataSource.scala:634)
at
org.apache.spark.sql.execution.datasources.DataSource$$anonfun$20.apply(DataSource.scala:634)
at scala.util.Try.orElse(Try.scala:84)
at
org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSource(DataSource.scala:634)
... 55 more
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
