Hi,
Imagine you have a structure like this:
val events = sqlContext.createDataFrame(
Seq(
("a", Map("a"->1,"b"->1)),
("b", Map("b"->1,"c"->1)),
("c", Map("a"->1,"c"->1))
)
).toDF("id","map")
What I want to achieve is have the map values as a separate columns.
Basically I want to achieve this:
+---+----+----+----+
| id| a| b| c|
+---+----+----+----+
| a| 1| 1|null|
| b|null| 1| 1|
| c| 1|null| 1|
+---+----+----+----+
I managed to create it with an explode-pivot combo, but for large dataset,
and a list of map keys around 1000 I imagine this will
be prohibitively expensive. I reckon there must be a much easier way to
achieve that, than:
val exploded =
events.select(col("id"),explode(col("map"))).groupBy("id").pivot("key").sum("value")
Any help would be appreciated. :)
