Please help me on the below error & give me different approach on the below
data manipulation.
Error:Unable to find encoder for type stored in a Dataset. Primitive types
(Int, String, etc) and Product types (case classes) are supported by
importing spark.implicits._ Support for serializing other types will be
added in future releases.
not enough arguments for method flatmap: (implicit evidence$8:
org.apache.spark.sql.Encoder[org.apache.spark.sql.Row])org.apache.spark.sql.Datset[org.apache.spark.sql.Row]
unspecified value parameter evidence$8.
Code:
def
getFlattenDataframe(someDataframe:DataFrame,spark:SparkSession):DataFrame =
{
val mymanualschema = new StructType(Array(
StructField("field1",StringType,true),
StructField("field2",StringType,true),
StructField("field3",StringType,true),
StructField("field4",IntegerType,true),
StructField("field5",DoubleType,true)))
val flattenRDD= someDataFrame.flatMap{curRow:Row=>getRows(curRow)} ---->
error showing in this line
spark.createDataFrame(flattenRdd,mymanualschema)
def getRows(CurRow:Row):Array[Row]={
val somefield =curRow.getAs[String]("field1")
--- saome manipulation happening here and finally return a array of rows
return res[Row]
}
Could you please someone help me what causing the issues here.i have tested
the import spark.implicits not working. how to fix this error or else help
me in different approach here.
--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscr...@spark.apache.org
