Hyukjin Kwon created SPARK-61884:
------------------------------------
Summary: Log the server's warning of an ML fit as PySpark does,
and give it to the Rust API's fitted model
Key: SPARK-61884
URL: https://issues.apache.org/jira/browse/SPARK-61884
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.4.0
Reporter: Hyukjin Kwon
PySpark 4.3.0's try_remote_fit logs the fit result's warning_message with
logging.getLogger("pyspark.ml.util").warning (ml/util.py:192-194). The server
sets it when a tree's training stops early under the ML cache's memory control
(MLHandler.scala:234-247, from RandomForest.scala and
GradientBoostedTrees.scala), so a random forest, a GBT or a decision tree whose
model is truncated warns, e.g. "The random forest training stops early because
the model size (35016 bytes) exceeds threshold (24000 bytes).". The drop-in's
native fits, and the Rust API's, read only the model's reference and uid of the
fit result, so nothing tells the user that the model was truncated.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]