Hyukjin Kwon created SPARK-61884:
------------------------------------

             Summary: Log the server's warning of an ML fit as PySpark does, 
and give it to the Rust API's fitted model
                 Key: SPARK-61884
                 URL: https://issues.apache.org/jira/browse/SPARK-61884
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: connect-rust-4.4.0
            Reporter: Hyukjin Kwon


PySpark 4.3.0's try_remote_fit logs the fit result's warning_message with 
logging.getLogger("pyspark.ml.util").warning (ml/util.py:192-194). The server 
sets it when a tree's training stops early under the ML cache's memory control 
(MLHandler.scala:234-247, from RandomForest.scala and 
GradientBoostedTrees.scala), so a random forest, a GBT or a decision tree whose 
model is truncated warns, e.g. "The random forest training stops early because 
the model size (35016 bytes) exceeds threshold (24000 bytes).". The drop-in's 
native fits, and the Rust API's, read only the model's reference and uid of the 
fit result, so nothing tells the user that the model was truncated.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to