For more details on my question
http://apache-spark-user-list.1001560.n3.nabble.com/How-to-generate-Java-bean-class-for-avro-files-using-spark-avro-project-tp22413.html
Thanks,
Yamini
On Tue, Apr 7, 2015 at 2:23 PM, Yamini Maddirala yamini.m...@gmail.com
wrote:
Hi Michael,
Yes, I did try spark-avro 0.2.0 databricks project. I am using CHD5.3
which is based on spark 1.2. Hence I'm bound to use spark-avro 0.2.0
instead of the latest.
I'm not sure how spark-avro project can help me in this scenario.
1. I have JavaDStream of type avro generic record
:JavaDStreamGenericRecord [This is the data being read from kafka topics]
2. I'm able to get JavaSchemaRDD using the avro file like this
final JavaSchemaRDD schemaRDD2 = AvroUtils.avroFile(sqlContext,
/xyz-Project/trunk/src/main/resources/xyz.avro);
3. I don't know how I can apply schema in step 2 to data in step 1.
I chose to do something like this
JavaSchemaRDD schemaRDD = sqlContext.applySchema(genericRecordJavaRDD,
xyz.class);
Used avro maven plugin to generate xyz class in Java. But this is not
good because avro maven plugin creates a field SCHEMA which is not
supported in applySchema method.
Please let me know how to deal with this.
Appreciate your help
Thanks,
Yamini
On Tue, Apr 7, 2015 at 1:57 PM, Michael Armbrust mich...@databricks.com
wrote:
Have you looked at spark-avro?
https://github.com/databricks/spark-avro
On Tue, Apr 7, 2015 at 3:57 AM, Yamini yamini.m...@gmail.com wrote:
Using spark(1.2) streaming to read avro schema based topics flowing in
kafka
and then using spark sql context to register data as temp table. Avro
maven
plugin(1.7.7 version) generates the java bean class for the avro file but
includes a field named SCHEMA$ of type org.apache.avro.Schema which is
not
supported in the JavaSQLContext class[Method : applySchema].
How to auto generate java bean class for the avro file and over come the
above mentioned problem.
Thanks.
-
Thanks,
Yamini
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/scala-MatchError-class-org-apache-avro-Schema-of-class-java-lang-Class-tp22402.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org
