Renaming fields to get around SPARK-2775
<https://issues.apache.org/jira/browse/SPARK-2775>.
I’m doing this clunky thing:
1. Convert a DataFrame’s schema to JSON, and then a Python dictionary.
2. Replace the problematic characters in the schema field names.
3.
Convert the resulting dictionary back into JSON and then back into a
DataFrame schema.
4.
Apply the new schema with the fixed field names.
Nick
On Thu, May 7, 2015 at 6:51 PM Reynold Xin <r...@databricks.com> wrote:
> What's the use case?
>
> I'm wondering if we should even expose fromJSON. I think it's more a bug
> than feature.
>
>
> On Thu, May 7, 2015 at 1:55 PM, Nicholas Chammas <
> nicholas.cham...@gmail.com> wrote:
>
>> Observe, my fellow Sparkophiles (Spark 1.3.1):
>>
>> >>> json_rdd = sqlContext.jsonRDD(sc.parallelize(['{"name": "Nick"}']))
>> >>> json_rdd.schema
>> StructType(List(StructField(name,StringType,true)))
>> >>> type(json_rdd.schema)
>> <class 'pyspark.sql.types.StructType'>
>> >>> json_rdd.schema.json()
>>
>> '{"fields":[{"metadata":{},"name":"name","nullable":true,"type":"string"}],"type":"struct"}'
>> >>> pyspark.sql.types.StructType.fromJson(json_rdd.schema.json())
>> Traceback (most recent call last):
>> File "<stdin>", line 1, in <module>
>> File
>> "/Applications/apache-spark/spark-1.3.1-bin-hadoop2.4/python/pyspark/sql/types.py",
>> line 346, in fromJson
>> return StructType([StructField.fromJson(f) for f in json["fields"]])
>> TypeError: string indices must be integers, not str
>> >>> import json
>> >>>
>> pyspark.sql.types.StructType.fromJson(json.loads(json_rdd.schema.json()))
>> StructType(List(StructField(name,StringType,true)))
>> >>>
>>
>> So fromJson() doesn’t actually expect JSON, which is a string. It expects
>> a
>> dictionary.
>>
>> Did I misunderstand something, or is this method poorly named?
>>
>> My impression was that the combination of toJson() and fromJson() would
>> return the original input as-is, but it looks like we needed the added
>> step
>> of json.loads() to make things work.
>>
>> Nick
>>
>>
>
>
