Hi again,
I use java profiler to find out the latency comes from the FastIDSet.addAll()
and FastIDSet.add()..
Blows are source code.
protected FastIDSet getAllOtherItems(long theUserID) throws TasteException {
......
for (int j = 0; j < size2; j++) {
possibleItemsIDs.addAll(dataModel.getItemIDsFromUser(prefs2.getUserID(j)));
}
......
}
public FastIDSet getItemIDsFromUser(long userID) throws TasteException {
PreferenceArray prefs = getPreferencesFromUser(userID);
int size = prefs.length();
FastIDSet result = new FastIDSet(size);
for (int i = 0; i < size; i++) {
result.add(prefs.getItemID(i));
}
return result;
}
>Yes, I use the genericdatamodel which is in-memory.
>
>
>
>
>
>>Is only the similarity matrix in-memory? The crucial thing here is the
>>data model not the similarity matrix, are you using an in-memory data model?
>>
>>Am 21.07.2010 08:33, schrieb Young:
>>> Yes, I am pretty sure. I have stored the similarity matrix in-memory and I
>>> print out the time spent in getAllOtherItems() and this is the only one
>>> time-consuming method in the recommendation. My laptop CPU is Intel P8600
>>> 2.4G, and the memory used for JVM is 1GB.
>>>
>>>
>>>
>>>
>>>
>>>
>>>> Hi Young,
>>>>
>>>> I would disagree that a response time of 6 seconds is OK for online
>>>> recommendations, the time should be something like < 100ms.
>>>> I'm really surprised that you would see such response times with an
>>>> in-memory data model, I have experience with in-memory models of roughly
>>>> the same size
>>>> and usually the computations are blazingly fast.
>>>>
>>>> Are you absolutely sure that the time is spent in this method and not
>>>> later in the similarity computation?
>>>>
>>>> --sebastian
>>>>
>>>> Am 21.07.2010 07:54, schrieb Young:
>>>>
>>>>> So based on the 1M dataset, the time spent in getAllOtherItems(userID) is
>>>>> among the 2 and 10 seconds.
>>>>> for example,
>>>>> If one user rates 200 items and for each item, the time spent in
>>>>> calculating the neighbors is expected to 30ms.
>>>>> So that makes 6 seconds. It is generally okay. But if the dataset is
>>>>> expanded to 100M dataset, I think 30ms may grow up to 30 * 100 ms and
>>>>> that will be a long time.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>> It still seems strange to observe such a bottleneck, I'm not sure
>>>>>> what's going on.
>>>>>> You are using an in-memory model like GenericDataModel?
>>>>>> We could look at ways to optimize that method, though it looks
>>>>>> reasonably tight.
>>>>>> Where within that method do you see time spent?
>>>>>>
>>>>>> 2010/7/20 Young <[email protected]>:
>>>>>>
>>>>>>
>>>>>>> Hi again,
>>>>>>> When I do the itembased recommendation, I find there are some latency
>>>>>>> in getAllOtherItems(long userID). Because it is calculating the items'
>>>>>>> neighbors and merge these neighbors together. So I am thinking if I
>>>>>>> precompute each item's neighbors and store in the database, then when I
>>>>>>> getAllOtherItems(), I could merge these neighbors directly. Is this
>>>>>>> useful for reducing the latency?
>>>>>>> Or is there other way to make the online-recommendation much faster?
>>>>>>> Thank you.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> Yes you probably want a new, separate table. You have an extra step of
>>>>>>>> computing some notion of similarity anyway, and you probably want to
>>>>>>>> separate this table from your main data table anyhow for reasons of
>>>>>>>> performance and business logic separation.
>>>>>>>>
>>>>>>>> 2010/7/19 Young <[email protected]>:
>>>>>>>>
>>>>>>>>
>>>>>>>>> So my prpblem is that I want to build datamodel based on what user
>>>>>>>>> has bought or added to their favorite or rated.
>>>>>>>>> You mean I need a table describe all these user behavior. For
>>>>>>>>> example, if user buys one item, I guess the user preference is 4 and
>>>>>>>>> add into this table?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> No, you need one table (or view if you like) containing all data. If
>>>>>>>>>> you can't do this, you could write your own copy of a JDBCDataModel
>>>>>>>>>> that can query multiple tables, or, that changes its SQL queries to
>>>>>>>>>> use UNION statements. I imagine it will slow down a lot.
>>>>>>>>>>
>>>>>>>>>> If you mean, can you use a table with preferences with a model that
>>>>>>>>>> ignores preferences, sure you can. The extra column is ignored.
>>>>>>>>>>
>>>>>>>>>> 2010/7/19 Young <[email protected]>:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> Hi,
>>>>>>>>>>> I have three tables, one is with preference and another two are
>>>>>>>>>>> without preference. Does mahout have some algorithm to integret
>>>>>>>>>>> these tables into one datamodel?
>>>>>>>>>>>
>>>>>>>>>>> Thank you
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>>>
>>>>
>>
