[
https://issues.apache.org/jira/browse/SPARK-61731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon resolved SPARK-61731.
----------------------------------
Resolution: Fixed
Issue resolved by pull request 710
[https://github.com/apache/spark-connect-rust/pull/710]
> Fetch StopWordsRemover's server defaults, and send the datasets of PrefixSpan
> and the ALS subsets and fetch the ALS rank as PySpark does
> ----------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61731
> URL: https://issues.apache.org/jira/browse/SPARK-61731
> Project: Spark
> Issue Type: Improvement
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Assignee: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> Follow-up of the native feature, fpm and recommendation classes of Spark
> Connect Rust, to match pyspark.ml 4.3.0:
> - StopWordsRemover's constructor holds the server defaults (stopWords,
> locale) that PySpark's constructor fetches of the ML helper; without a
> session it raises PySpark's bare AssertionError.
> - ALSModel: rank is fetched on each access, as PySpark does, and
> recommendForAllUsers, recommendForAllItems, recommendForUserSubset and
> recommendForItemSubset are members of the native model.
> - PrefixSpan.findFrequentSequentialPatterns and
> ALSModel.recommendForUserSubset/recommendForItemSubset send their dataset as
> pyspark.ml.connect.serialize does, in a lazy DataFrame that the server
> rejects of a value that is not a DataFrame, as with official PySpark.
> - The Rust API's ALSModel::rank() fetches the rank on each call, and a load
> sends its Read alone.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]