Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/19792#discussion_r156279744
--- Diff: python/pyspark/sql/types.py ---
@@ -1083,7 +1083,8 @@ def _infer_schema(row):
elif hasattr(row, "_fields"): # namedtuple
items = zip(row._fields, tuple(row))
else:
- names = ['_%d' % i for i in range(1, len(row) + 1)]
+ if names is None:
+ names = ['_%d' % i for i in range(1, len(row) + 1)]
--- End diff --
@gberger, Let's revert this change too. Seems it's going to introduce a
behaviour change:
**Before**
```
>>> spark.createDataFrame([["a", "b"]], ["col1"]).show()
17/12/12 15:29:45 WARN ObjectStore: Failed to get database global_temp,
returning NoSuchObjectException
+----+---+
|col1| _2|
+----+---+
| a| b|
+----+---+
```
**After**
```
>>> spark.createDataFrame([["a", "b"]], ["col1"]).show()
...
java.lang.IllegalStateException: Input row doesn't have expected number of
values required by the schema. 1 fields are required while 2 values are
provided.
at
org.apache.spark.sql.execution.python.EvaluatePython$.fromJava(EvaluatePython.scala:148)
...
```
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
