[
https://issues.apache.org/jira/browse/FLINK-4613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15516644#comment-15516644
]
ASF GitHub Bot commented on FLINK-4613:
---------------------------------------
Github user gaborhermann commented on the issue:
https://github.com/apache/flink/pull/2542
We did not measure performance against Spark or other implementations yet.
Those would reflect the performance of Flink ALS implementation, as there is
not much difference between the implicit and explicit implementations.
Instead, we compared the implicit case with the explicit case in the Flink
implementation on the same datasets, to make sure the implicit case does not
decrease the performance significantly. (Of course, we expected the implicit
case to be slower due to the extra precomputation and broadcasting of `Xt * X`.)
```
expl impl
100 8885 9196
1000 7879 11282
10000 8839 9220
100000 7102 10998
1000000 7543 10680
```
The numbers in the left column indicate the size of the training set (I'm
not sure about the measure, but @jfeher can tell about it). The numbers are the
training time in milliseconds in the explicit and implicit case respectively.
We did the measurements on a small cluster of 3 nodes.
It seems, there is a large constant overhead, but it's not significantly
slower in the implicit case.
We could do further, more thorough measurements if needed, but maybe that
would be another issue. Benchmarking more and optimizing both the original ALS
algorithm and the specific `Xt * X` computation in the implicit case could be a
separate PR.
What are your thoughts on this?
> Extend ALS to handle implicit feedback datasets
> -----------------------------------------------
>
> Key: FLINK-4613
> URL: https://issues.apache.org/jira/browse/FLINK-4613
> Project: Flink
> Issue Type: New Feature
> Components: Machine Learning Library
> Reporter: Gábor Hermann
> Assignee: Gábor Hermann
>
> The Alternating Least Squares implementation should be extended to handle
> _implicit feedback_ datasets. These datasets do not contain explicit ratings
> by users, they are rather built by collecting user behavior (e.g. user
> listened to artist X for Y minutes), and they require a slightly different
> optimization objective. See details by [Hu et
> al|http://dx.doi.org/10.1109/ICDM.2008.22].
> We do not need to modify much in the original ALS algorithm. See [Spark ALS
> implementation|https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/ml/recommendation/ALS.scala],
> which could be a basis for this extension. Only the updating factor part is
> modified, and most of the changes are in the local parts of the algorithm
> (i.e. UDFs). In fact, the only modification that is not local, is
> precomputing a matrix product Y^T * Y and broadcasting it to all the nodes,
> which we can do with broadcast DataSets.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
