HyukjinKwon commented on code in PR #36545:
URL: https://github.com/apache/spark/pull/36545#discussion_r875352636
##########
python/pyspark/sql/session.py:
##########
@@ -570,10 +570,20 @@ def _inferSchemaFromList(
if not data:
raise ValueError("can not infer schema from empty dataset")
infer_dict_as_struct = self._jconf.inferDictAsStruct()
+ infer_array_from_first_element =
self._jconf.legacyInferArrayTypeFromFirstElement()
Review Comment:
Yeah, that's right that it causes a behaviour change. However, the
(previous) string type coercion behaviour in an element of an array is actually
a mistake I believe. For example, such type coercion is not supported in
regular type inference:
```python
>>> spark.createDataFrame([{"a": "1"}, {"a" :2}])
Traceback (most recent call last):
...
TypeError: field a: Can not merge type <class
'pyspark.sql.types.StringType'> and <class 'pyspark.sql.types.LongType'>
```
So, what this PR actually does is to match the behaviour with non-nested
type inference. The switch was added for users dependent on the previous
behaviour.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
