jroof88 edited a comment on pull request #29720:
URL: https://github.com/apache/spark/pull/29720#issuecomment-730689336
@HyukjinKwon I thought I would throw an example here for good measure:
```python
import json
from pyspark.sql.types import StructType
df = spark.sql("SELECT * FROM XXX.YYY")
schema_dict = df.schema.jsonValue()
schema_dict['fields'].append({'metadata': {'comment': 'Jacks Test Comment'},
'name': 'jacks_test_column', 'type': 'string'})
print("New Column: ", schema_dict['fields'][len(schema_dict['fields'])-1])
schema = StructType.fromJson(schema_dict)
'''
/databricks/spark/python/pyspark/sql/types.py in fromJson(cls, json)
435 return StructField(json["name"],
436 _parse_datatype_json_value(json["type"]),
--> 437 json["nullable"],
438 json["metadata"])
439
KeyError: 'nullable'
'''
```
Small Example Here where I am trying to add some metadata to a table. The
key `nullable` should have a default value so I only have to worry about it
when I want to explicitly set it.
I just don't really understand the statement `JSON isn't supposed to be
constructed by users` if in fact it can easily be constructed and manipulated
by the user. I would bet users manually create schemas from a variety of
sources and run into this error.
In addition, I'm happy to implement it in Scala but I would like the OK in
Python before spending time there 😄. Are there are other contributors to
PySpark that might have an opinion here?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
