[
https://issues.apache.org/jira/browse/SPARK-7507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14555482#comment-14555482
]
Nicholas Chammas commented on SPARK-7507:
-----------------------------------------
Since {{Row}} seems most analogous to a {{namedtuple}} in Python, here is an
interesting parallel that suggests we should perhaps instead support
{{vars(Row)}} and not {{dict(Row)}}.
http://stackoverflow.com/q/26180528/877069
https://docs.python.org/3/library/functions.html#vars
https://docs.python.org/3/library/collections.html#collections.somenamedtuple._asdict
{quote}
somenamedtuple._asdict()
Return a new OrderedDict which maps field names to their corresponding values.
Note, this method is no longer needed now that the same effect can be achieved
by using the built-in vars() function:
{quote}
> pyspark.sql.types.StructType and Row should implement __iter__()
> ----------------------------------------------------------------
>
> Key: SPARK-7507
> URL: https://issues.apache.org/jira/browse/SPARK-7507
> Project: Spark
> Issue Type: Sub-task
> Components: PySpark, SQL
> Reporter: Nicholas Chammas
> Priority: Minor
>
> {{StructType}} looks an awful lot like a Python dictionary.
> However, it doesn't implement {{\_\_iter\_\_()}}, so doing a quick conversion
> like this doesn't work:
> {code}
> >>> df = sqlContext.jsonRDD(sc.parallelize(['{"name": "El Magnifico"}']))
> >>> df.schema
> StructType(List(StructField(name,StringType,true)))
> >>> dict(df.schema)
> Traceback (most recent call last):
> File "<stdin>", line 1, in <module>
> TypeError: 'StructType' object is not iterable
> {code}
> This would be super helpful for doing any custom schema manipulations without
> having to go through the whole {{.json() -> json.loads() -> manipulate() ->
> json.dumps() -> .fromJson()}} charade.
> Same goes for {{Row}}, which offers an
> [{{asDict()}}|https://spark.apache.org/docs/1.3.1/api/python/pyspark.sql.html#pyspark.sql.Row.asDict]
> method but doesn't support the more Pythonic {{dict(Row)}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
