You can try to create new columns with the nested value,
> On Jan 29, 2018, at 15:26, Arnav kumar <akns2...@gmail.com> wrote:
>
> Hello Experts,
>
> I would need your advice in resolving the below issue when I am trying to
> retrieving the data from a dataframe.
>
> Can you please let me know where I am going wrong.
>
> code :
>
>
> // create the dataframe by parsing the json
> // Message Helper describes the JSON Struct
> //data out is the json string received from Streaming Engine.
>
> val dataDF = sparkSession.createDataFrame(dataOut, MessageHelper.sqlMapping)
> dataDF.printSchema()
> /* -- out put of dataDF.printSchema
>
> root
> |-- messageID: string (nullable = true)
> |-- messageType: string (nullable = true)
> |-- meta: array (nullable = true)
> | |-- element: struct (containsNull = true)
> | | |-- messageParsedTimestamp: string (nullable = true)
> | | |-- ipaddress: string (nullable = true)
> |-- messageData: array (nullable = true)
> | |-- element: struct (containsNull = true)
> | | |-- packetID: string (nullable = true)
> | | |-- messageID: string (nullable = true)
> | | |-- unixTime: string (nullable = true)
>
>
>
> */
>
>
> dataDF.createOrReplaceTempView("message")
> val routeEventDF=sparkSession.sql("select messageId
> ,messageData.unixTime,messageData.packetID, messageData.messageID from
> message")
> routeEventDF.show
>
>
> Error on routeEventDF.show
> Caused by: java.lang.RuntimeException:
> org.apache.spark.sql.catalyst.expressions.GenericRow is not a valid external
> type for schema of
> array<struct<messageParsedTimestamp:string,ipaddress:string,port:string,message:string>>>>
> at
> org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.evalIfFalseExpr14$(Unknown
> Source)
>
>
> Appreciate your help
>
> Best Regards
> Arnav Kumar.
>
>
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscr...@spark.apache.org
