So based on the 1M dataset, the time spent in getAllOtherItems(userID) is among the 2 and 10 seconds. for example, If one user rates 200 items and for each item, the time spent in calculating the neighbors is expected to 30ms. So that makes 6 seconds. It is generally okay. But if the dataset is expanded to 100M dataset, I think 30ms may grow up to 30 * 100 ms and that will be a long time.
>It still seems strange to observe such a bottleneck, I'm not sure >what's going on. >You are using an in-memory model like GenericDataModel? >We could look at ways to optimize that method, though it looks reasonably >tight. >Where within that method do you see time spent? > >2010/7/20 Young <[email protected]>: >> Hi again, >> When I do the itembased recommendation, I find there are some latency in >> getAllOtherItems(long userID). Because it is calculating the items' >> neighbors and merge these neighbors together. So I am thinking if I >> precompute each item's neighbors and store in the database, then when I >> getAllOtherItems(), I could merge these neighbors directly. Is this useful >> for reducing the latency? >> Or is there other way to make the online-recommendation much faster? >> Thank you. >> >> >> >> >>>Yes you probably want a new, separate table. You have an extra step of >>>computing some notion of similarity anyway, and you probably want to >>>separate this table from your main data table anyhow for reasons of >>>performance and business logic separation. >>> >>>2010/7/19 Young <[email protected]>: >>>> So my prpblem is that I want to build datamodel based on what user has >>>> bought or added to their favorite or rated. >>>> You mean I need a table describe all these user behavior. For example, if >>>> user buys one item, I guess the user preference is 4 and add into this >>>> table? >>>> >>>> >>>> >>>> >>>>>No, you need one table (or view if you like) containing all data. If >>>>>you can't do this, you could write your own copy of a JDBCDataModel >>>>>that can query multiple tables, or, that changes its SQL queries to >>>>>use UNION statements. I imagine it will slow down a lot. >>>>> >>>>>If you mean, can you use a table with preferences with a model that >>>>>ignores preferences, sure you can. The extra column is ignored. >>>>> >>>>>2010/7/19 Young <[email protected]>: >>>>>> Hi, >>>>>> I have three tables, one is with preference and another two are without >>>>>> preference. Does mahout have some algorithm to integret these tables >>>>>> into one datamodel? >>>>>> >>>>>> Thank you >>>> >>
