Hi all - I've spent a while playing with this. Two significant sources of speed up that I've achieved are
1) Manually multiplying the feature vectors and caching either the user or product vector 2) By doing so, if one of the RDDs is a global it becomes possible to parallelize this step by running it in a thread and submitting multiple threads to yarn engine. Doing so I've achieved an over 75x speed up compared with the packaged versio inside ml lib. Sent with Good (www.good.com) -----Original Message----- From: Sean Owen [so...@cloudera.com<mailto:so...@cloudera.com>] Sent: Thursday, February 12, 2015 05:47 PM Eastern Standard Time To: Crystal Xing Cc: user@spark.apache.org Subject: Re: Is there a fast way to do fast top N product recommendations for all users Not now, but see https://issues.apache.org/jira/browse/SPARK-3066 As an aside, it's quite expensive to make recommendations for all users. IMHO this is not something to do, if you can avoid it architecturally. For example, consider precomputing recommendations only for users whose probability of needing recommendations soon is not very small. Usually, only a small number of users are active. On Thu, Feb 12, 2015 at 10:26 PM, Crystal Xing <crystalxin...@gmail.com> wrote: > Hi, > > > I wonder if there is a way to do fast top N product recommendations for all > users in training using mllib's ALS algorithm. > > I am currently calling > > public Rating[] recommendProducts(int user, > int num) > > method in MatrixFactorizatoinModel for users one by one > and it is quite slow since it does not operate on RDD input? > > I also tried to generate all possible > user-product pairs and use > public JavaRDD<Rating> predict(JavaPairRDD<Integer,Integer> usersProducts) > > to fill out the matrix. Since I have a large number of user and products, > > the job stucks and transforming all pairs. > > > I wonder if there is a better way to do this. > > Thanks, > > Crystal. --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org ________________________________________________________ The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer.
