Hello Sir,
I have a scenario to flatten the different combinations of map type(key
value) in a column called eve_data like below:
How do we flatten the map type into proper columns using pyspark
1) Source Dataframe having 2 columns(event id,data)
eve_id,eve_data
001, "k1":"abc",
"k2":"xyz"
"k3":"10091"
eve_id,eve_data
002, "k1":"12",
"k2":"jack",
"k3":"0.01",
"k4":"0998"
eve_id,eve_data
003, "k1":"aaa",
"k2":"xxxx",
"k3":"device",
"k4":"endpoint",
"k5":"-"
Final output:
(flatten the output of each event ids key values).The number of key values
will be different for each event id.so i want to flatten the records for all
the map type(key values) as below
eve_id k1 k2 k3
001 abc xyz 10091
eve_id, k1 k2 k3 k4
002, 12 jack 0.01 0998
eve_id, k1 k2 k3 k4 k5
003, aaa xxxx device endpoint -
Thanks
Anbu
--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/
---------------------------------------------------------------------
To unsubscribe e-mail: [email protected]
