jroof88 opened a new pull request #29720: URL: https://github.com/apache/spark/pull/29720
### What changes were proposed in this pull request? <!-- When create a StructType from a Python dictionary you utilize the `[StructType.fromJson](https://github.com/apache/spark/blob/master/python/pyspark/sql/types.py#L569-L571)` function. This function then internally calls `[StructField.fromJson](https://github.com/apache/spark/blob/328d81a2d1131742bcfba5117896c093db39e721/python/pyspark/sql/types.py#L425-L430)` for each field in the list. This function requires that you must have all keys populated. The constructor for `[StructField](https://github.com/apache/spark/blob/328d81a2d1131742bcfba5117896c093db39e721/python/pyspark/sql/types.py#L386-L395)` has default arguments: ``` class StructField(DataType): ... def __init__(self, name, dataType, nullable=True, metadata=None): ... self.name = name self.dataType = dataType self.nullable = nullable self.metadata = metadata or {} ``` It seems logical to mirror this when creating it from a Python dict. In addition, I added similar logic to `MapType` and `ArrayType` that follow the same pattern of requiring keys in the JSON that have default arguments in the constructor. --> ### Why are the changes needed? <!-- Please clarify why the changes are needed. For instance, 1. If you propose a new API, clarify the use case for a new API. 2. If you fix a bug, you can clarify why it is a bug. --> ### Does this PR introduce _any_ user-facing change? <!-- No. There are no user facing changes. --> ### How was this patch tested? <!-- Test are currently a work in progress. --> ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
