On Tue, 11 Dec 2001, Stefano Mazzocchi wrote: > <context> > In this message I talk about indexing, searching and retrieving large > quantities of semi-structured XML data (mostly documents). > </context> > > XML documents can be seen as text and indexed as such. Unfortunately, > this would totally break usefulness since <b>Hello</b> would not be > indexed as 'hello' between 'b' but simply as that textual token. > > Such a model would result in a very poor searching experience since the > user must know in advance how the markup is used. Add namespaces to the > picture, and this becomes clearly useless. > > - o - > > An improved model indexes the results of the parsing stage, separating > the 'textual content' (element contents and attributes value) from the > 'contextual content' (element names and attribute names). > > This is, of course, a better indexing, but still has a few defects: > > 1) markup information is lost > 2) summary information is hard to recreate because of #1 > > One solution is to avoid indexing attributes, hoping that element > content is more structured in itself, thus the summary gives better > results. Unfortunately, attributes convey content as much as elements > do. One cannot assume or force the opposite. > > - o - > > I propose a solution which I call 'Semantic Relevance Rating' (shortly > SRR, that you can pronounce as 'SoRRy' :)
:) > SRR adds the ability to merge semantic information from the markup > structure into the index, thus avoiding to loose the contextual > information embedded in them. > > Let me show you an example of what this is. Suppose we have to index > this document: > > <document> > <metadata> > <title>How <keyword>SRR</keyword> works</title> > <authors> > <author name="Stefano Mazzocchi" email="[EMAIL PROTECTED]/> > </authors> > <legal>Copyright © Stefano Mazzocchi. All rights > reserved.</legal> > <abstract> > <para> > This article explains how <keyword>semantic relevance > rating</keyword> > works and outlines possible usage cases of the technology > </para> > </abstract> > <related-concepts> > <concept>semantic search</concept> > <concept>indexing</concept> > <concept>data mining</concept> > </related-concepts> > </metadata> > <content> > <section title="Introduction"> > <para>Here I explain how <strong>Semantic Relevance Rating</strong> > works.</para> > </section> > ... > <section title="A usage case"> > <para>This is a usage case:</para> > <figure href="schema.jpg" caption="Cocoon-based indexing system"/> > </section> > ... > </content> > </document> > > just like a browser wouldn't be able to present the above document > without style information, an indexer is not able to 'understand' or > interpret the semantic relevance of each textual piece without some > information that matches their contextual part. > > Following the stylesheet concept, I propose the creation of a > 'relevance-sheet' which contains this information for the indexer that > allows it to index the structured content in the way it was intended > from the document authors. > > Here is an example relevance-sheet for the above document: > > document.srr: > <srr:relevances xmlns:srr="..." extends="metadata.srr"> > <srr:context xpath="section/@title" relevance="2"/> > <srr:context xpath="para" relevance="1"/> > <srr:context xpath="strong" relevance="1.5"/> > <srr:context xpath="figure/@href" relevance="0"/> > <srr:context xpath="figure/@caption" relevance="1.5"/> > </srr:relevances> > > metadata.srr: > <srr:relevances xmlns:srr="..."> > <srr:context xpath="title" relevance="2"/> > <srr:context xpath="author/@name" relevance="2"/> > <srr:context xpath="author/@email" relevance="0"/> > <srr:context xpath="keyword" relevance="5"/> > <srr:context xpath="abstract" relevance="2"/> > <srr:context xpath="related-concepts/concept" relevance="3.5"/> > </srr:relevances> I think you missed to tell us that an SRR rates a schema not a document (I know this because we've talked about it privately). How do you think overriding releveance values should be handled? Suppose the following: document.srr: <srr:relevances xmlns:srr="..." extends="metadata.srr"> <srr:context xpath="section/@title" relevance="2"/> <srr:context xpath="para" relevance="1"/> <srr:context xpath="strong" relevance="1.5"/> <srr:context xpath="figure/@href" relevance="0"/> <srr:context xpath="figure/@caption" relevance="1.5"/> <srr:context xpath="keyword" relevance="15"/> </srr:relevances> metadata.srr: <srr:relevances xmlns:srr="..."> <srr:context xpath="title" relevance="2"/> <srr:context xpath="author/@name" relevance="2"/> <srr:context xpath="author/@email" relevance="0"/> <srr:context xpath="keyword" relevance="5"/> <srr:context xpath="abstract" relevance="2"/> <srr:context xpath="related-concepts/concept" relevance="3.5"/> </srr:relevances> Note the xpath to "keyword" in both SRRs. > The way the SRR sheet is associated with the document is not defined > here since it is another concern. Another concern is how SRR are obtained to rate schemas. And also how SRR (obtained from different sources) for the exact same schema are handled. I think all relevance rating values in an SRR should sum up to a fixed value to yeald equal ratings among different sources. But then all SRR for different schemas are equally rated. Are there SRRs (read schemas) which are more important than others? > > - o - > > The SRR solution yields a few interesting results: > > 1) the cost of 'semantizing' the information is proportional to the > number of schemas included in the data corpus to index, unlike RDF-like > solutions which costs are proportional to the entire information > included in the corpus. > For example, in a system where there are 10 different schemas and a > milion documents, the cost will be associated in creating SRR > relevance-sheets for those 10 schemas, compared to the cost of adding > semantic RDF information in each and every file. > > This is the exact same concept of SoC between content and style, here > associated to the separation between content and its semantic relevance > interpretation. This is an economic and realistic approach because without automated RDFizability of documents nobody will pay such a prize for semantic searching capabilities. And talking about automated RDFizability will raise the question how relevant can this be made. > 2) the user experience is no different from the one he/she's used to: he > doesn't need to know the schema of the documents nor any information > about metadata or metadata fields in order to obtain the information. If she/he is guided by some tools which handles the validation according to the underlying schema of the document. > 3) the SRR drives the indexing behavior and indicates whether or not > some information should be indexed or not. > > Since the relevance factors are multiplied by the indexer, text that is > associated to a context of 'zero' semantic relevance is skipped and > avoids polluting the index with information that might 'get in the way' > unwanted. > > - o - > > I'm currently working with the Lucene guys to add the ability to 'rate' > textual input in the indexer. When this is done, adding SRR capabilities > with Cocoon is a matter of writing the relevance sheets for our DTDs. > > Comments? You got it now :) Giacomo --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, email: [EMAIL PROTECTED]
