zero323 commented on a change in pull request #35410:
URL: https://github.com/apache/spark/pull/35410#discussion_r805165758
##########
File path: python/pyspark/sql/pandas/conversion.py
##########
@@ -88,9 +88,10 @@ def toPandas(self) -> "PandasDataFrameLike":
import pandas as pd
from pandas.core.dtypes.common import is_timedelta64_dtype
- timezone = self.sql_ctx._conf.sessionLocalTimeZone() # type:
ignore[attr-defined]
+ jconf = self.sparkSession._jconf # type: ignore[attr-defined]
Review comment:
This `ignore` shouldn't be necessary.
##########
File path: python/pyspark/sql/session.py
##########
@@ -348,6 +336,11 @@ def _repr_html_(self) -> str:
sc_HTML=self.sparkContext._repr_html_(), # type:
ignore[attr-defined]
)
+ @property
+ def _jconf(self) -> SparkConf:
Review comment:
This should be `-> "JavaObject"`, shouldn't it?
##########
File path: python/pyspark/sql/pandas/conversion.py
##########
@@ -88,9 +88,10 @@ def toPandas(self) -> "PandasDataFrameLike":
import pandas as pd
from pandas.core.dtypes.common import is_timedelta64_dtype
- timezone = self.sql_ctx._conf.sessionLocalTimeZone() # type:
ignore[attr-defined]
+ jconf = self.sparkSession._jconf # type: ignore[attr-defined]
+ timezone = jconf.sessionLocalTimeZone() # type: ignore[attr-defined]
Review comment:
This one, and the following, also once `SparkSession._jconf` return type
is fixed.
##########
File path: python/pyspark/sql/pandas/conversion.py
##########
@@ -603,25 +601,25 @@ def _create_from_pandas_with_arrow(
for pdf_slice in pdf_slices
]
- jsqlContext = self._wrapped._jsqlContext # type: ignore[attr-defined]
+ jsparkSession = self._jsparkSession # type: ignore[attr-defined]
Review comment:
We can add a hint
```python
_jsparkSession: "JavaObject"
```
at the top of the class for this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
