Hi there,
I have saved my records in to parquet format and am using Spark1.5. But when
I try to fetch the columns it throws exception*
java.lang.ClassCastException: java.lang.Long cannot be cast to
org.apache.spark.unsafe.types.UTF8String*.
This filed is saved as String while writing parquet. so here is the sample
code and output for the same..
logger.info("troubling thing is ::" +
sqlContext.sql(fileSelectQuery).schema().toString());
DataFrame df= sqlContext.sql(fileSelectQuery);
JavaRDD<Row> rdd2 = df.toJavaRDD();
First Line in the code (Logger) prints this:
troubling thing is ::StructType(StructField(batch_id,StringType,true))
But the moment after it the execption comes up.
Any idea why it is treating the filed as Long? (yeah one unique thing about
column is it is always a number e.g. Time-stamp).
Any help is appreciated.
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/SQLcontext-changing-String-field-to-Long-tp25005.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org
