anthonywainer commented on code in PR #38285:
URL: https://github.com/apache/spark/pull/38285#discussion_r999251797
##########
python/pyspark/sql/types.py:
##########
@@ -653,8 +653,8 @@ def fromJson(cls, json: Dict[str, Any]) -> "StructField":
return StructField(
json["name"],
_parse_datatype_json_value(json["type"]),
- json["nullable"],
- json["metadata"],
+ json.get("nullable", True),
+ json.get("metadata"),
Review Comment:
the way to avoid this issue users would be to declare the nullable and
metadata attributes mandatory:
Example:
{
"name": "auditname",
"type": "string",
"nullable": true, -> mandatory
"metadata": {} -> mandatory
}
I have avoided skipping this by not using the fromJson method, I mean
passing the json directly.
json = {
"name": "auditname",
"type: StringType()
}
StructField(**json)
this gets tricky because it is necessary to add some previous logics to do
the above, the solution would be to use .get in the fromJson method.
def fromJson(cls, json: Dict[str, Any]) -> "StructField":
return StructField(
json["name"],
_parse_datatatype_json_value(json["type"]),
json.get("nullable", True), -> Use get
json.get("metadata"), -> Use get
)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
