This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 8f895e9 [SPARK-36779][PYTHON] Fix when list of data type tuples has
len = 1
8f895e9 is described below
commit 8f895e9e9699d20c40dc5f351ee6007f28b4befd
Author: dgd-contributor <[email protected]>
AuthorDate: Fri Sep 17 09:27:46 2021 +0900
[SPARK-36779][PYTHON] Fix when list of data type tuples has len = 1
### What changes were proposed in this pull request?
Fix when list of data type tuples has len = 1
### Why are the changes needed?
Fix when list of data type tuples has len = 1
``` python
>>> ps.DataFrame[("a", int), [int]]
typing.Tuple[pyspark.pandas.typedef.typehints.IndexNameType, int]
>>> ps.DataFrame[("a", int), [("b", int)]]
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/dgd/spark/python/pyspark/pandas/frame.py", line 11998, in
__class_getitem__
return create_tuple_for_frame_type(params)
File "/Users/dgd/spark/python/pyspark/pandas/typedef/typehints.py", line
685, in create_tuple_for_frame_type
return Tuple[extract_types(params)]
File "/Users/dgd/spark/python/pyspark/pandas/typedef/typehints.py", line
755, in extract_types
return (index_type,) + extract_types(data_types)
File "/Users/dgd/spark/python/pyspark/pandas/typedef/typehints.py", line
770, in extract_types
raise TypeError(
TypeError: Type hints should be specified as one of:
- DataFrame[type, type, ...]
- DataFrame[name: type, name: type, ...]
- DataFrame[dtypes instance]
- DataFrame[zip(names, types)]
- DataFrame[index_type, [type, ...]]
- DataFrame[(index_name, index_type), [(name, type), ...]]
- DataFrame[dtype instance, dtypes instance]
- DataFrame[(index_name, index_type), zip(names, types)]
However, got [('b', <class 'int'>)].
```
### Does this PR introduce _any_ user-facing change?
After:
``` python
>>> ps.DataFrame[("a", int), [("b", int)]]
typing.Tuple[pyspark.pandas.typedef.typehints.IndexNameType,
pyspark.pandas.typedef.typehints.NameType]
```
### How was this patch tested?
exist test
Closes #34019 from
dgd-contributor/fix_when_list_of_tuple_data_type_have_len=1.
Authored-by: dgd-contributor <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
python/pyspark/pandas/typedef/typehints.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/python/pyspark/pandas/typedef/typehints.py
b/python/pyspark/pandas/typedef/typehints.py
index c198e55..ecdc532 100644
--- a/python/pyspark/pandas/typedef/typehints.py
+++ b/python/pyspark/pandas/typedef/typehints.py
@@ -746,7 +746,7 @@ def extract_types(params: Any) -> Tuple:
data_types = params[1]
if (
isinstance(data_types, list)
- and len(data_types) > 1
+ and len(data_types) >= 1
and isinstance(data_types[0], tuple)
): # type: ignore
# Example:
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
