[
https://issues.apache.org/jira/browse/SPARK-9101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mateusz Buśkiewicz updated SPARK-9101:
--------------------------------------
Affects Version/s: 1.4.1
> Can't use null in selectExpr
> ----------------------------
>
> Key: SPARK-9101
> URL: https://issues.apache.org/jira/browse/SPARK-9101
> Project: Spark
> Issue Type: Bug
> Components: PySpark
> Affects Versions: 1.4.0, 1.4.1
> Reporter: Mateusz Buśkiewicz
>
> In 1.3.1 this worked:
> {code:python}
> df = sqlContext.createDataFrame([[1]], schema=['col'])
> df.selectExpr('null as newCol').collect()
> {code}
> In 1.4.0 it fails with the following stacktrace:
> {code}
> Traceback (most recent call last):
> File "<input>", line 1, in <module>
> File
> "/opt/boxen/homebrew/opt/apache-spark/libexec/python/pyspark/sql/dataframe.py",
> line 316, in collect
> cls = _create_cls(self.schema)
> File
> "/opt/boxen/homebrew/opt/apache-spark/libexec/python/pyspark/sql/dataframe.py",
> line 229, in schema
> self._schema = _parse_datatype_json_string(self._jdf.schema().json())
> File
> "/opt/boxen/homebrew/opt/apache-spark/libexec/python/pyspark/sql/types.py",
> line 519, in _parse_datatype_json_string
> return _parse_datatype_json_value(json.loads(json_string))
> File
> "/opt/boxen/homebrew/opt/apache-spark/libexec/python/pyspark/sql/types.py",
> line 539, in _parse_datatype_json_value
> return _all_complex_types[tpe].fromJson(json_value)
> File
> "/opt/boxen/homebrew/opt/apache-spark/libexec/python/pyspark/sql/types.py",
> line 386, in fromJson
> return StructType([StructField.fromJson(f) for f in json["fields"]])
> File
> "/opt/boxen/homebrew/opt/apache-spark/libexec/python/pyspark/sql/types.py",
> line 347, in fromJson
> _parse_datatype_json_value(json["type"]),
> File
> "/opt/boxen/homebrew/opt/apache-spark/libexec/python/pyspark/sql/types.py",
> line 535, in _parse_datatype_json_value
> raise ValueError("Could not parse datatype: %s" % json_value)
> ValueError: Could not parse datatype: null
> {code}
> https://github.com/apache/spark/blob/v1.4.0/python/pyspark/sql/types.py#L461
> The cause:_atomic_types doesn't contain NullType
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
