Hi sebastian Thank you. My observation is based on in-memory datamodel. ----Young
>Hi Young, > >I guess you are using a jdbc-based DataModel. > >If I understand getAllOtherItemIDs(...) correctly, it's doing the following: > >for each item i preferred by the current user > for each user u preferring i > fetch other items preferred by u > >This method could issue a lot of database queries if you encounter items >that are preferred by a lot of users. You could check whether you have >set the correct indexes on the database tables and generally check your >database connectivity (do the calls have to go over the network?), but I >think the best thing would be to use an in-memory data model, which >should not be a problem with 10M preferences. > >--sebastian > > >Am 20.07.2010 20:04, schrieb Sean Owen: >> It still seems strange to observe such a bottleneck, I'm not sure >> what's going on. >> You are using an in-memory model like GenericDataModel? >> We could look at ways to optimize that method, though it looks reasonably >> tight. >> Where within that method do you see time spent? >> >> 2010/7/20 Young <[email protected]>: >> >>> Hi again, >>> When I do the itembased recommendation, I find there are some latency in >>> getAllOtherItems(long userID). Because it is calculating the items' >>> neighbors and merge these neighbors together. So I am thinking if I >>> precompute each item's neighbors and store in the database, then when I >>> getAllOtherItems(), I could merge these neighbors directly. Is this useful >>> for reducing the latency? >>> Or is there other way to make the online-recommendation much faster? >>> Thank you. >>> >>> >>> >>> >>> >>>> Yes you probably want a new, separate table. You have an extra step of >>>> computing some notion of similarity anyway, and you probably want to >>>> separate this table from your main data table anyhow for reasons of >>>> performance and business logic separation. >>>> >>>> 2010/7/19 Young <[email protected]>: >>>> >>>>> So my prpblem is that I want to build datamodel based on what user has >>>>> bought or added to their favorite or rated. >>>>> You mean I need a table describe all these user behavior. For example, if >>>>> user buys one item, I guess the user preference is 4 and add into this >>>>> table? >>>>> >>>>> >>>>> >>>>> >>>>> >>>>>> No, you need one table (or view if you like) containing all data. If >>>>>> you can't do this, you could write your own copy of a JDBCDataModel >>>>>> that can query multiple tables, or, that changes its SQL queries to >>>>>> use UNION statements. I imagine it will slow down a lot. >>>>>> >>>>>> If you mean, can you use a table with preferences with a model that >>>>>> ignores preferences, sure you can. The extra column is ignored. >>>>>> >>>>>> 2010/7/19 Young <[email protected]>: >>>>>> >>>>>>> Hi, >>>>>>> I have three tables, one is with preference and another two are without >>>>>>> preference. Does mahout have some algorithm to integret these tables >>>>>>> into one datamodel? >>>>>>> >>>>>>> Thank you >>>>>>> >>>>> >>> >
