Hi Zoran,
it was not the many emails. just a hole in how we handle the list in the moment. For the ordering of pages in KIM - yes we did not promote this so far as it has not been the focus of the default setup. As mentioned in some other situations it is quite important. We will surely change the focus more into this direction and also try to emphasize on the full power of the possibilities to maintain hybrid indices in KIM on text, structure of documents, entity references and ontology & an instance base at the same time. and respectively the possibility to query by restricting over all of these modalities in the same time.

on different factors which might influence the order of pages:
- one factor is popularity of course, but it is very different in various setups: it might be pages linking to each other, papers citing each other and so on - actuality is another one - old / new or most recently updated and so on; it is interesting to mention that both these factors are interesting in both directions: sometimes old is very appealing (the oldest doc on OWLIM?) or the least popular pub in my town - so i do not step upon the guys i had a fight with. So especially for research it is nice to put emphasis on giving more power to the human to decide what is important of these factors, how much and in which direction.
other factors could be:
- association with the profile of the user;
- association with a particular context: e.g. previous things the user liked - which captures usually more long term associations; -- this one is also interesting for modeling how association with things you like changes over time; say: i like this song, three years later - i have expressed opinion for over 1000 songs. i probably still like the first one, but maybe my association with the latest songs is stronger. so if you build a recommendation system (which is very similar retrieval problem) you will like to include a certain emphasis on things one liked more recently as this increases your potential to be useful. - behaviour of all people using the system: statistics on top of that for example: after searching for "intellectual rock" (weird uh?) a big percentage of people clicked on the first song you suggest, but 90 percent of them did not listen to the end - so maybe it is not a good suggestion; - on the other hand ... you might be completely off-the-main-stream a \ because you like a rare type of disharmony in the music ... b\ you name stuff differently (e.g. for you rap means only the old-school run d.m.c. stuff and hip hop you associate with the new girlie-face gangsta wannabes. ... just checking if you read still) ; so this means this approximate-on-all-other-citizens approach is not good for you then: then, you blend. you blend what the citizens tend to like with what you tend to like - but again give more power to the person to say : hey - i do not need any recommendations based on community metrics - just follow my behaviour, current context (of course i am at work - how could you imagine i look for paris hilton when i type in just paris ?!)

some other stuff is related to influence of some external state of affairs (possibly linked to the profile of the person): like when it is full moon people's searches for "moon" and "full moon" increase dramatically so based on that you can gather context, learn the tendency and blend it into the your scoring function.

so you have characteristics which are query and user independent ( how linked are the documents; what is the frequency of terms in the corpus)
query dependent ( similarity between the query and the doc )
dependent on the particular person or group or approximated across all "citizens" of your system and so on;

blending these together (e.g. through a weighed sum) allows you to give different components of the score different weights; or allow the person to somehow influence this (not a simple usability issue actually - although there are good ideas).

another very important part is that for a very broad field of search (like general web search engines) you might benefit from completely different weights for the separate score components based on the profile of the user, the context or the type of the query (when you try also to judge that looking for java examples - the student is actually looking for source code and not companies trying to sell smth. but when he looks for airplane ticket to hannover - it is a different story.

some more characteristics which are interesting are - how authoritative is the source - finding something from the yellow press and something from Herald Tribune is completely different situation resulting from what the system suggests. you can play with trying to model mood-altering-potential let's say - i would prefer to find results which are considered more funny;

the text above should be a bit messy (a bit tired already), but i hope you got some ideas.

btw i did not understand if you still do this on top of KIM or outside of it ?

kind regards
borislav


On Oct 30, 2008, at 7:27 PM, Zoran Jeremic wrote:

Hi Borislav,

Thank you for your answer. I realized that you missed my email, because there were many other emails at the same time I sent my. However, I have solved the problem. Actually, my problem was that I didn't found the literature that explains the cosine similarity in details. All examples I found did described how to compare two pages (doc-doc), and finally I found a book that describes how to compare term with page (query - doc), so it was easy to implement it.

>So if you are having an annotation of an entity reference in the document and you have >indexed it in KIM >then when you search (no matter if keyword-based or semantic search based on what you >have in the knowledge base) the doc store will return the documents ordered based on >this similarity and you will not need to worry for tf-idf / cosine.

I didn't know that KIM order docs somehow, and even now I'm not sure if it is possible in my approach. I don't search all pages that contain some term. Rather, I annotate a set of web pages for a set of terms (design patterns), I use term frequency to find a dominant design pattern in each page, and for each design pattern I order all pages where dominant pattern is the one I check using a cosine similarity. Maybe it would be easier to use KIM to make this instead, but as you told, this way it is more exciting and more desirable as my work is research rather then pure software development. >there are few other tricks on how you can model additional factors to influence the >score/order of docs in these stores - but let's first solve this.

Could you please indicate some other factors that could affect the order of pages? Maybe it could be usefull in my research.

All the best,
Zoran

--- On Thu, 10/30/08, borislav popov <[EMAIL PROTECTED]> wrote:
From: borislav popov <[EMAIL PROTECTED]>
Subject: Re: [KIM-discussion] Cosine similarity to check web page relevance
To: [EMAIL PROTECTED]
Cc: [email protected]
Date: Thursday, October 30, 2008, 3:49 AM

Zoran,
this mail somehow slipped through the support watchdogs.
let's think together on this one:
in the default version of KIM we do not expose anything on ranking / sorting until now. On the other hand we order results in various ways in systems based on KIM and your question is very valid. the DIY approach you apply is actually very nice - one doesn't see very often people trying to dive deeper in how things are being ordered.

anyway i will mention a potential shortcut (although it is much more exciting to continue the way you do it!) if you use the default document store of KIM - it is actually an IR engine - Lucene - and it operates in the way you are trying to model (on various steroids actually :) ) So if you are having an annotation of an entity reference in the document and you have indexed it in KIM then when you search (no matter if keyword-based or semantic search based on what you have in the knowledge base) the doc store will return the documents ordered based on this similarity and you will not need to worry for tf-idf / cosine.

but ! as i say your approach is exciting and also the proposed shortcut is doc store specific - so have it in mind. there are few other tricks on how you can model additional factors to influence the score/order of docs in these stores - but let's first solve this.

regarding the cosine similarity - it is symmetrical - so it doesn't matter in which order you compare: query to doc or doc to query.

maybe i did not answer everything - but let me know how it goes !!!

kind regards
borislav

On Oct 3, 2008, at 9:01 PM, Zoran Jeremic wrote:

Hi everybody,

I'm interesting if anybody of you could help me with suggestions about how to use cosine similarity (http://en.wikipedia.org/wiki/Cosine_similarity ) to get the page relevance for some term. I suppose that some of you have already used it.

Here is the short description of what I have done:
I analize (annotate) a set of web pages to get an instances of design patterns described in my ontology. I got a term frequency inverse document frequency (http://en.wikipedia.org/wiki/Tf-idf) for each design pattern I found on the page. Now I want to sort all pages by the relevance for some specific pattern (i.e. Abstract Factory), but I'm not sure how to use cosine similarity formula here. What should be A vector, and what should be B vector? Should I compare the one page with the instance of pattern I want to check, and how can I do it?

Thanks
Zoran

_______________________________________________
NOTE: Please REPLY TO ALL to ensure that your reply reaches all members of this mailing list.

KIM-discussion mailing list
[email protected]
http://ontotext.com/mailman/listinfo/kim-discussion_ontotext.com



_______________________________________________
NOTE: Please REPLY TO ALL to ensure that your reply reaches all members of this 
mailing list.

KIM-discussion mailing list
[email protected]
http://ontotext.com/mailman/listinfo/kim-discussion_ontotext.com

Reply via email to