Tanjin Panna created SPARK-27037:
------------------------------------
Summary: Pyspark Row .asDict() cannot handle MapType with a Struct
as the key or value
Key: SPARK-27037
URL: https://issues.apache.org/jira/browse/SPARK-27037
Project: Spark
Issue Type: Bug
Components: PySpark
Affects Versions: 2.4.0
Reporter: Tanjin Panna
When we have a tuple as the key or value in a `MapType` and call the
`asDict()`, we still have a `Row` in the output for the key and value:
{code}
>>> from pyspark.sql import Row
>>>> df = spark.createDataFrame(
[
Row(tuple_key_map={('hello', True): (1234, 111)}),
Row(tuple_key_map={('there', False): (5678, 343)})
]
)
>>> df.schema
StructType(List(StructField(tuple_key_map,MapType(StructType(List(StructField(_1,StringType,true),StructField(_2,BooleanType,true))),StructType(List(StructField(_1,LongType,true),StructField(_2,LongType,true))),true),true)))
>>> df.show(truncate=False)
+-------------------------------+
|tuple_key_map |
+-------------------------------+
|[[hello, true] -> [1234, 111]] |
|[[there, false] -> [5678, 343]]|
+-------------------------------+
>>> df.head().asDict()
{'tuple_key_map': {Row(_1=u'hello', _2=True): Row(_1=1234, _2=111)}}
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
