What's the use case?
I'm wondering if we should even expose fromJSON. I think it's more a bug
than feature.
On Thu, May 7, 2015 at 1:55 PM, Nicholas Chammas <nicholas.cham...@gmail.com
> wrote:
> Observe, my fellow Sparkophiles (Spark 1.3.1):
>
> >>> json_rdd = sqlContext.jsonRDD(sc.parallelize(['{"name": "Nick"}']))
> >>> json_rdd.schema
> StructType(List(StructField(name,StringType,true)))
> >>> type(json_rdd.schema)
> <class 'pyspark.sql.types.StructType'>
> >>> json_rdd.schema.json()
>
> '{"fields":[{"metadata":{},"name":"name","nullable":true,"type":"string"}],"type":"struct"}'
> >>> pyspark.sql.types.StructType.fromJson(json_rdd.schema.json())
> Traceback (most recent call last):
> File "<stdin>", line 1, in <module>
> File
> "/Applications/apache-spark/spark-1.3.1-bin-hadoop2.4/python/pyspark/sql/types.py",
> line 346, in fromJson
> return StructType([StructField.fromJson(f) for f in json["fields"]])
> TypeError: string indices must be integers, not str
> >>> import json
> >>>
> pyspark.sql.types.StructType.fromJson(json.loads(json_rdd.schema.json()))
> StructType(List(StructField(name,StringType,true)))
> >>>
>
> So fromJson() doesn’t actually expect JSON, which is a string. It expects a
> dictionary.
>
> Did I misunderstand something, or is this method poorly named?
>
> My impression was that the combination of toJson() and fromJson() would
> return the original input as-is, but it looks like we needed the added step
> of json.loads() to make things work.
>
> Nick
>
>
