[
https://issues.apache.org/jira/browse/SPARK-18781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15787786#comment-15787786
]
Liang-Chi Hsieh commented on SPARK-18781:
-----------------------------------------
[~eyal] Do you have estimated time cost of the approximate count? Since it is
just an approximation, I think it should not take big portion of the whole time
of predict.
If the batches are large, as [~sowen] said, this overhead is trivial compared
with the predication. And I agreed that this is the predicate method is meant
for.
Even if the batches are small, I don't think the approximation will largely
increase the time of running predict. I can't image that the given
usersProducts will cost much more time in approximating count than the joining
+ ddot in predication.
> Allow MatrixFactorizationModel.predict to skip user/product approximation
> count
> -------------------------------------------------------------------------------
>
> Key: SPARK-18781
> URL: https://issues.apache.org/jira/browse/SPARK-18781
> Project: Spark
> Issue Type: Improvement
> Components: MLlib
> Reporter: Eyal Allweil
> Priority: Minor
>
> When
> [MatrixFactorizationModel.predict|https://spark.apache.org/docs/1.6.1/api/java/org/apache/spark/mllib/recommendation/MatrixFactorizationModel.html#predict(org.apache.spark.rdd.RDD)]
> is used, it first calculates an approximation count of the users and
> products in order to determine the most efficient way to proceed. In many
> cases, the answer to this question is fixed (typically there are more users
> than products by an order of magnitude) and this check is unnecessary. Adding
> a parameter to this predict method to allow choosing the implementation (and
> skipping the check) would be nice.
> It would be especially nice in development cycles when you are repeatedly
> tweaking your model and which pairs you're predicting for and this
> approximate count represents a meaningful portion of the time you wait for
> results.
> I can provide a pull request with this ability added that preserves the
> existing behavior.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
