[
https://issues.apache.org/jira/browse/SPARK-43158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17712879#comment-17712879
]
Snoot.io commented on SPARK-43158:
----------------------------------
User 'HyukjinKwon' has created a pull request for this issue:
https://github.com/apache/spark/pull/40814
> Set upperbound of pandas version in binder integrations
> -------------------------------------------------------
>
> Key: SPARK-43158
> URL: https://issues.apache.org/jira/browse/SPARK-43158
> Project: Spark
> Issue Type: Bug
> Components: Documentation
> Affects Versions: 3.4.0
> Reporter: Hyukjin Kwon
> Assignee: Hyukjin Kwon
> Priority: Major
> Fix For: 3.3.3, 3.4.0, 3.5.0
>
>
> {code}
> df.toPandas
> {code}
> fails with
> {code}
> ---------------------------------------------------------------------------
> TypeError Traceback (most recent call last)
> Cell In[14], line 1
> ----> 1 df.toPandas()
> File
> /srv/conda/envs/notebook/lib/python3.10/site-packages/pyspark/sql/pandas/conversion.py:251,
> in PandasConversionMixin.toPandas(self)
> 248 should_check_timedelta = is_timedelta64_dtype(t) and len(pdf) == 0
> 250 if (t is not None and not is_timedelta64_dtype(t)) or
> should_check_timedelta:
> --> 251 series = series.astype(t, copy=False)
> 253 with catch_warnings():
> 254 from pandas.errors import PerformanceWarning
> File
> /srv/conda/envs/notebook/lib/python3.10/site-packages/pandas/core/generic.py:6324,
> in NDFrame.astype(self, dtype, copy, errors)
> 6317 results = [
> 6318 self.iloc[:, i].astype(dtype, copy=copy)
> 6319 for i in range(len(self.columns))
> 6320 ]
> 6322 else:
> 6323 # else, only a single dtype is given
> -> 6324 new_data = self._mgr.astype(dtype=dtype, copy=copy, errors=errors)
> 6325 return self._constructor(new_data).__finalize__(self,
> method="astype")
> 6327 # GH 33113: handle empty frame or series
> File
> /srv/conda/envs/notebook/lib/python3.10/site-packages/pandas/core/internals/managers.py:451,
> in BaseBlockManager.astype(self, dtype, copy, errors)
> 448 elif using_copy_on_write():
> 449 copy = False
> --> 451 return self.apply(
> 452 "astype",
> 453 dtype=dtype,
> 454 copy=copy,
> 455 errors=errors,
> 456 using_cow=using_copy_on_write(),
> 457 )
> File
> /srv/conda/envs/notebook/lib/python3.10/site-packages/pandas/core/internals/managers.py:352,
> in BaseBlockManager.apply(self, f, align_keys, **kwargs)
> 350 applied = b.apply(f, **kwargs)
> 351 else:
> --> 352 applied = getattr(b, f)(**kwargs)
> 353 result_blocks = extend_blocks(applied, result_blocks)
> 355 out = type(self).from_blocks(result_blocks, self.axes)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
