Hi Young,
I guess you are using a jdbc-based DataModel.
If I understand getAllOtherItemIDs(...) correctly, it's doing the following:
for each item i preferred by the current user
for each user u preferring i
fetch other items preferred by u
This method could issue a lot of database queries if you encounter items
that are preferred by a lot of users. You could check whether you have
set the correct indexes on the database tables and generally check your
database connectivity (do the calls have to go over the network?), but I
think the best thing would be to use an in-memory data model, which
should not be a problem with 10M preferences.
--sebastian
Am 20.07.2010 20:04, schrieb Sean Owen:
> It still seems strange to observe such a bottleneck, I'm not sure
> what's going on.
> You are using an in-memory model like GenericDataModel?
> We could look at ways to optimize that method, though it looks reasonably
> tight.
> Where within that method do you see time spent?
>
> 2010/7/20 Young <[email protected]>:
>
>> Hi again,
>> When I do the itembased recommendation, I find there are some latency in
>> getAllOtherItems(long userID). Because it is calculating the items'
>> neighbors and merge these neighbors together. So I am thinking if I
>> precompute each item's neighbors and store in the database, then when I
>> getAllOtherItems(), I could merge these neighbors directly. Is this useful
>> for reducing the latency?
>> Or is there other way to make the online-recommendation much faster?
>> Thank you.
>>
>>
>>
>>
>>
>>> Yes you probably want a new, separate table. You have an extra step of
>>> computing some notion of similarity anyway, and you probably want to
>>> separate this table from your main data table anyhow for reasons of
>>> performance and business logic separation.
>>>
>>> 2010/7/19 Young <[email protected]>:
>>>
>>>> So my prpblem is that I want to build datamodel based on what user has
>>>> bought or added to their favorite or rated.
>>>> You mean I need a table describe all these user behavior. For example, if
>>>> user buys one item, I guess the user preference is 4 and add into this
>>>> table?
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>> No, you need one table (or view if you like) containing all data. If
>>>>> you can't do this, you could write your own copy of a JDBCDataModel
>>>>> that can query multiple tables, or, that changes its SQL queries to
>>>>> use UNION statements. I imagine it will slow down a lot.
>>>>>
>>>>> If you mean, can you use a table with preferences with a model that
>>>>> ignores preferences, sure you can. The extra column is ignored.
>>>>>
>>>>> 2010/7/19 Young <[email protected]>:
>>>>>
>>>>>> Hi,
>>>>>> I have three tables, one is with preference and another two are without
>>>>>> preference. Does mahout have some algorithm to integret these tables
>>>>>> into one datamodel?
>>>>>>
>>>>>> Thank you
>>>>>>
>>>>
>>
