itholic commented on a change in pull request #34179:
URL: https://github.com/apache/spark/pull/34179#discussion_r722847893
##########
File path: python/pyspark/pandas/indexes/multi.py
##########
@@ -375,6 +375,15 @@ def name(self) -> Name:
def name(self, name: Name) -> None:
raise PandasNotImplementedError(class_name="pd.MultiIndex",
property_name="name")
+ @property
+ def dtypes(self) -> Series:
+ return pd.Series(
+ [field.dtype for field in self._internal.index_fields],
+ index=pd.Index(
+ [name if len(name) > 1 else name[0] for name in
self._internal.index_names]
+ ),
+ )
Review comment:
> hmm, if we return `ps.Series`, we can not use index type hint for now.
Can we use `pd.Series` ?
>
> ```python
> >>> ps.DataFrame[psdf.index.dtypes, psdf.dtypes]
> Traceback (most recent call last):
> File "<stdin>", line 1, in <module>
> File "/u02/spark/python/pyspark/pandas/frame.py", line 12097, in
__class_getitem__
> return create_tuple_for_frame_type(params)
> File "/u02/spark/python/pyspark/pandas/typedef/typehints.py", line 729,
in create_tuple_for_frame_type
> return Tuple[_extract_types(params)]
> File "/u02/spark/python/pyspark/pandas/typedef/typehints.py", line 760,
in _extract_types
> index_params = _to_tuple_of_params(index_params)
> File "/u02/spark/python/pyspark/pandas/typedef/typehints.py", line 833,
in _to_tuple_of_params
> params = tuple(params)
> File "/u02/spark/python/pyspark/pandas/series.py", line 6366, in __iter__
> return MissingPandasLikeSeries.__iter__(self)
> File "/u02/spark/python/pyspark/pandas/missing/__init__.py", line 23, in
unsupported_function
> raise PandasNotImplementedError(
> pyspark.pandas.exceptions.PandasNotImplementedError: The method
`pd.Series.__iter__()` is not implemented. If you want to collect your data as
an NumPy array, use 'to_numpy()' instead.
> ```
Yeah, I think it's normal because we don't support the iteration for
`Series` yet.
I wonder if we can say the nlevels of MultiIndex is always expected to be
small.
If the nlevels of MultiIndex is always expected to be small, it's okay to
just return the pandas Series.
Otherwise, we should return pandas-on-Spark Series.
I'm pretty sure most of them expected to be small,
but could you double check if it's okay, @HyukjinKwon @ueshin
@xinrong-databricks ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
