Igor Dvorzhak created SPARK-55452:
-------------------------------------
Summary: PySpark Connect MLWriter does not support overwrite()
Key: SPARK-55452
URL: https://issues.apache.org/jira/browse/SPARK-55452
Project: Spark
Issue Type: Bug
Components: Connect, ML, PySpark
Affects Versions: 4.1.1, 4.0.2, 4.2.0
Reporter: Igor Dvorzhak
`MLWriter.overwrite()` call fails for PySpark Connect Sessions:
```
self = <xgboost.spark.core.SparkXGBModelWriter object at 0x7f81b0eaf620>, path
= 'file:/tmp/tmphb6f4hc7'
def _handleOverwrite(self, path: str) -> None:
from pyspark.ml.wrapper import JavaWrapper
_java_obj =
JavaWrapper._new_java_obj("org.apache.spark.ml.util.FileSystemOverwrite")
wrapper = JavaWrapper(_java_obj)
> wrapper._call_java("handleOverwrite", path, True,
> self.sparkSession._jsparkSession)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.venv/lib/python3.13/site-packages/pyspark/ml/util.py:579:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <pyspark.sql.connect.session.SparkSession object at 0x7f81b1163cb0>,
name = '_jsparkSession'
def _{_}getattr{_}_(self, name: str) -> Any:
if name in ["_jsc", "_jconf", "_jvm", "_jsparkSession", "sparkContext",
"newSession"]:
> raise PySparkAttributeError(
errorClass="JVM_ATTRIBUTE_NOT_SUPPORTED", messageParameters=
{"attr_name": name}
)
E pyspark.errors.exceptions.base.PySparkAttributeError:
[JVM_ATTRIBUTE_NOT_SUPPORTED] Attribute `_jsparkSession` is not supported in
Spark Connect as it depends on the JVM. If you need to use this attribute, do
not use Spark Connect when creating your session. Visit
[https://spark.apache.org/docs/latest/sql-getting-started.html#starting-point-sparksession]
for creating regular Spark Session in detail.
.venv/lib/python3.13/site-packages/pyspark/sql/connect/session.py:1000:
PySparkAttributeError
```
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
