Hyukjin Kwon created SPARK-61731:
------------------------------------

             Summary: Fetch StopWordsRemover's server defaults, and send the 
datasets of PrefixSpan and the ALS subsets and fetch the ALS rank as PySpark 
does
                 Key: SPARK-61731
                 URL: https://issues.apache.org/jira/browse/SPARK-61731
             Project: Spark
          Issue Type: Improvement
          Components: Connect
    Affects Versions: connect-rust-4.3.0
            Reporter: Hyukjin Kwon


Follow-up of the native feature, fpm and recommendation classes of Spark 
Connect Rust, to match pyspark.ml 4.3.0:
- StopWordsRemover's constructor holds the server defaults (stopWords, locale) 
that PySpark's constructor fetches of the ML helper; without a session it 
raises PySpark's bare AssertionError.
- ALSModel: rank is fetched on each access, as PySpark does, and 
recommendForAllUsers, recommendForAllItems, recommendForUserSubset and 
recommendForItemSubset are members of the native model.
- PrefixSpan.findFrequentSequentialPatterns and 
ALSModel.recommendForUserSubset/recommendForItemSubset send their dataset as 
pyspark.ml.connect.serialize does, in a lazy DataFrame that the server rejects 
of a value that is not a DataFrame, as with official PySpark.
- The Rust API's ALSModel::rank() fetches the rank on each call, and a load 
sends its Read alone.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to