Well,
there must be something wrong here, I've seen systems in production with
similar sized data that responded clearly below 1 second all the time. I
really can't imagine
that FastIDSet would be the cause for this.
Are you sure the JVM has all the machine for itself? No email
application in the background checking mails, no memory swapping of the OS?
If you want you can make your data and test code available and I can
check it on my notebook.
--sebastian
Am 21.07.2010 11:23, schrieb Young:
> Hi again,
> I use java profiler to find out the latency comes from the FastIDSet.addAll()
> and FastIDSet.add()..
>
> Blows are source code.
>
> protected FastIDSet getAllOtherItems(long theUserID) throws TasteException {
> ......
> for (int j = 0; j < size2; j++) {
>
> possibleItemsIDs.addAll(dataModel.getItemIDsFromUser(prefs2.getUserID(j)));
> }
> ......
> }
> public FastIDSet getItemIDsFromUser(long userID) throws TasteException {
> PreferenceArray prefs = getPreferencesFromUser(userID);
> int size = prefs.length();
> FastIDSet result = new FastIDSet(size);
> for (int i = 0; i < size; i++) {
> result.add(prefs.getItemID(i));
> }
> return result;
> }
>
>
>
>
>
>
>
>
>
>> Yes, I use the genericdatamodel which is in-memory.
>>
>>
>>
>>
>>
>>
>>> Is only the similarity matrix in-memory? The crucial thing here is the
>>> data model not the similarity matrix, are you using an in-memory data model?
>>>
>>> Am 21.07.2010 08:33, schrieb Young:
>>>
>>>> Yes, I am pretty sure. I have stored the similarity matrix in-memory and I
>>>> print out the time spent in getAllOtherItems() and this is the only one
>>>> time-consuming method in the recommendation. My laptop CPU is Intel P8600
>>>> 2.4G, and the memory used for JVM is 1GB.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>> Hi Young,
>>>>>
>>>>> I would disagree that a response time of 6 seconds is OK for online
>>>>> recommendations, the time should be something like < 100ms.
>>>>> I'm really surprised that you would see such response times with an
>>>>> in-memory data model, I have experience with in-memory models of roughly
>>>>> the same size
>>>>> and usually the computations are blazingly fast.
>>>>>
>>>>> Are you absolutely sure that the time is spent in this method and not
>>>>> later in the similarity computation?
>>>>>
>>>>> --sebastian
>>>>>
>>>>> Am 21.07.2010 07:54, schrieb Young:
>>>>>
>>>>>
>>>>>> So based on the 1M dataset, the time spent in getAllOtherItems(userID)
>>>>>> is among the 2 and 10 seconds.
>>>>>> for example,
>>>>>> If one user rates 200 items and for each item, the time spent in
>>>>>> calculating the neighbors is expected to 30ms.
>>>>>> So that makes 6 seconds. It is generally okay. But if the dataset is
>>>>>> expanded to 100M dataset, I think 30ms may grow up to 30 * 100 ms and
>>>>>> that will be a long time.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>> It still seems strange to observe such a bottleneck, I'm not sure
>>>>>>> what's going on.
>>>>>>> You are using an in-memory model like GenericDataModel?
>>>>>>> We could look at ways to optimize that method, though it looks
>>>>>>> reasonably tight.
>>>>>>> Where within that method do you see time spent?
>>>>>>>
>>>>>>> 2010/7/20 Young <[email protected]>:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> Hi again,
>>>>>>>> When I do the itembased recommendation, I find there are some latency
>>>>>>>> in getAllOtherItems(long userID). Because it is calculating the items'
>>>>>>>> neighbors and merge these neighbors together. So I am thinking if I
>>>>>>>> precompute each item's neighbors and store in the database, then when
>>>>>>>> I getAllOtherItems(), I could merge these neighbors directly. Is this
>>>>>>>> useful for reducing the latency?
>>>>>>>> Or is there other way to make the online-recommendation much faster?
>>>>>>>> Thank you.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> Yes you probably want a new, separate table. You have an extra step of
>>>>>>>>> computing some notion of similarity anyway, and you probably want to
>>>>>>>>> separate this table from your main data table anyhow for reasons of
>>>>>>>>> performance and business logic separation.
>>>>>>>>>
>>>>>>>>> 2010/7/19 Young <[email protected]>:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> So my prpblem is that I want to build datamodel based on what user
>>>>>>>>>> has bought or added to their favorite or rated.
>>>>>>>>>> You mean I need a table describe all these user behavior. For
>>>>>>>>>> example, if user buys one item, I guess the user preference is 4 and
>>>>>>>>>> add into this table?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> No, you need one table (or view if you like) containing all data. If
>>>>>>>>>>> you can't do this, you could write your own copy of a JDBCDataModel
>>>>>>>>>>> that can query multiple tables, or, that changes its SQL queries to
>>>>>>>>>>> use UNION statements. I imagine it will slow down a lot.
>>>>>>>>>>>
>>>>>>>>>>> If you mean, can you use a table with preferences with a model that
>>>>>>>>>>> ignores preferences, sure you can. The extra column is ignored.
>>>>>>>>>>>
>>>>>>>>>>> 2010/7/19 Young <[email protected]>:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> Hi,
>>>>>>>>>>>> I have three tables, one is with preference and another two are
>>>>>>>>>>>> without preference. Does mahout have some algorithm to integret
>>>>>>>>>>>> these tables into one datamodel?
>>>>>>>>>>>>
>>>>>>>>>>>> Thank you
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>
>>>>>
>>>
