Anthony Wainer Cachay Guivin created SPARK-40820: ----------------------------------------------------
Summary: creating StructType from Json Key: SPARK-40820 URL: https://issues.apache.org/jira/browse/SPARK-40820 Project: Spark Issue Type: Bug Components: PySpark Affects Versions: 3.3.0 Reporter: Anthony Wainer Cachay Guivin When create a StructType from a Python dictionary you utilize the [StructType.fromJson|https://github.com/apache/spark/blob/master/python/pyspark/sql/types.py#L569-L571] method. To create a schema can be created as follows from the code below, but it requires to put inside the json: Nullable and Metadata, this is inconsistent because within the DataType class this by default. {code:python} json = { "name": "name", "type": "string" } StructField.fromJson(json) {code} Error: !image-2022-10-17-15-52-21-787.png! Proposed coding solution: Instead use indexes for getting from a dictionary, it would be better to use .get {code:python} def fromJson(cls, json: Dict[str, Any]) -> "StructField": return StructField( json["name"], _parse_datatype_json_value(json["type"]), json.get("nullable"), json.get("metadata"), ) {code} -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org