[
https://issues.apache.org/jira/browse/SPARK-7506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15280606#comment-15280606
]
Nicholas Chammas edited comment on SPARK-7506 at 5/11/16 6:51 PM:
------------------------------------------------------------------
[~davies] - Would you be interested in a PR that adds a new method called
{{fromDict()}} and makes {{fromJson()}} an alias of that method? It would
preserve compatibility, while (kinda) correcting the naming problem here.
was (Author: nchammas):
[~davies] - Would you be interested in a PR that adds new methods called
{{toDict()}} and {{fromDict()}}, and made {{json()}} and {{fromJson()}} aliases
of those methods? It would preserve compatibility, while correcting the naming
problem here.
> pyspark.sql.types.StructType.fromJson() is incorrectly named
> ------------------------------------------------------------
>
> Key: SPARK-7506
> URL: https://issues.apache.org/jira/browse/SPARK-7506
> Project: Spark
> Issue Type: Sub-task
> Components: PySpark, SQL
> Affects Versions: 1.3.0, 1.3.1
> Reporter: Nicholas Chammas
> Priority: Minor
>
> {code}
> >>> json_rdd = sqlContext.jsonRDD(sc.parallelize(['{"name": "Nick"}']))
> >>> json_rdd.schema
> StructType(List(StructField(name,StringType,true)))
> >>> type(json_rdd.schema)
> <class 'pyspark.sql.types.StructType'>
> >>> json_rdd.schema.json()
> '{"fields":[{"metadata":{},"name":"name","nullable":true,"type":"string"}],"type":"struct"}'
> >>> pyspark.sql.types.StructType.fromJson(json_rdd.schema.json())
> Traceback (most recent call last):
> File "<stdin>", line 1, in <module>
> File
> "/Applications/apache-spark/spark-1.3.1-bin-hadoop2.4/python/pyspark/sql/types.py",
> line 346, in fromJson
> return StructType([StructField.fromJson(f) for f in json["fields"]])
> TypeError: string indices must be integers, not str
> >>> import json
> >>> pyspark.sql.types.StructType.fromJson(json.loads(json_rdd.schema.json()))
> StructType(List(StructField(name,StringType,true)))
> >>>
> {code}
> So {{fromJson()}} doesn't actually expect JSON, which is a string. It expects
> a dictionary.
> This method should probably be renamed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
