Tanjin Panna created SPARK-27037: ------------------------------------ Summary: Pyspark Row .asDict() cannot handle MapType with a Struct as the key or value Key: SPARK-27037 URL: https://issues.apache.org/jira/browse/SPARK-27037 Project: Spark Issue Type: Bug Components: PySpark Affects Versions: 2.4.0 Reporter: Tanjin Panna
When we have a tuple as the key or value in a `MapType` and call the `asDict()`, we still have a `Row` in the output for the key and value: {code} >>> from pyspark.sql import Row >>>> df = spark.createDataFrame( [ Row(tuple_key_map={('hello', True): (1234, 111)}), Row(tuple_key_map={('there', False): (5678, 343)}) ] ) >>> df.schema StructType(List(StructField(tuple_key_map,MapType(StructType(List(StructField(_1,StringType,true),StructField(_2,BooleanType,true))),StructType(List(StructField(_1,LongType,true),StructField(_2,LongType,true))),true),true))) >>> df.show(truncate=False) +-------------------------------+ |tuple_key_map | +-------------------------------+ |[[hello, true] -> [1234, 111]] | |[[there, false] -> [5678, 343]]| +-------------------------------+ >>> df.head().asDict() {'tuple_key_map': {Row(_1=u'hello', _2=True): Row(_1=1234, _2=111)}} {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org