<context>
In this message I talk about indexing, searching and retrieving large
quantities of semi-structured XML data (mostly documents).
</context>
XML documents can be seen as text and indexed as such. Unfortunately,
this would totally break usefulness since <b>Hello</b> would not be
indexed as 'hello' between 'b' but simply as that textual token.
Such a model would result in a very poor searching experience since the
user must know in advance how the markup is used. Add namespaces to the
picture, and this becomes clearly useless.
- o -
An improved model indexes the results of the parsing stage, separating
the 'textual content' (element contents and attributes value) from the
'contextual content' (element names and attribute names).
This is, of course, a better indexing, but still has a few defects:
1) markup information is lost
2) summary information is hard to recreate because of #1
One solution is to avoid indexing attributes, hoping that element
content is more structured in itself, thus the summary gives better
results. Unfortunately, attributes convey content as much as elements
do. One cannot assume or force the opposite.
- o -
I propose a solution which I call 'Semantic Relevance Rating' (shortly
SRR, that you can pronounce as 'SoRRy' :)
SRR adds the ability to merge semantic information from the markup
structure into the index, thus avoiding to loose the contextual
information embedded in them.
Let me show you an example of what this is. Suppose we have to index
this document:
<document>
<metadata>
<title>How <keyword>SRR</keyword> works</title>
<authors>
<author name="Stefano Mazzocchi" email="[EMAIL PROTECTED]/>
</authors>
<legal>Copyright © Stefano Mazzocchi. All rights
reserved.</legal>
<abstract>
<para>
This article explains how <keyword>semantic relevance
rating</keyword>
works and outlines possible usage cases of the technology
</para>
</abstract>
<related-concepts>
<concept>semantic search</concept>
<concept>indexing</concept>
<concept>data mining</concept>
</related-concepts>
</metadata>
<content>
<section title="Introduction">
<para>Here I explain how <strong>Semantic Relevance Rating</strong>
works.</para>
</section>
...
<section title="A usage case">
<para>This is a usage case:</para>
<figure href="schema.jpg" caption="Cocoon-based indexing system"/>
</section>
...
</content>
</document>
just like a browser wouldn't be able to present the above document
without style information, an indexer is not able to 'understand' or
interpret the semantic relevance of each textual piece without some
information that matches their contextual part.
Following the stylesheet concept, I propose the creation of a
'relevance-sheet' which contains this information for the indexer that
allows it to index the structured content in the way it was intended
from the document authors.
Here is an example relevance-sheet for the above document:
document.srr:
<srr:relevances xmlns:srr="..." extends="metadata.srr">
<srr:context xpath="section/@title" relevance="2"/>
<srr:context xpath="para" relevance="1"/>
<srr:context xpath="strong" relevance="1.5"/>
<srr:context xpath="figure/@href" relevance="0"/>
<srr:context xpath="figure/@caption" relevance="1.5"/>
</srr:relevances>
metadata.srr:
<srr:relevances xmlns:srr="...">
<srr:context xpath="title" relevance="2"/>
<srr:context xpath="author/@name" relevance="2"/>
<srr:context xpath="author/@email" relevance="0"/>
<srr:context xpath="keyword" relevance="5"/>
<srr:context xpath="abstract" relevance="2"/>
<srr:context xpath="related-concepts/concept" relevance="3.5"/>
</srr:relevances>
The way the SRR sheet is associated with the document is not defined
here since it is another concern.
- o -
The SRR solution yields a few interesting results:
1) the cost of 'semantizing' the information is proportional to the
number of schemas included in the data corpus to index, unlike RDF-like
solutions which costs are proportional to the entire information
included in the corpus.
For example, in a system where there are 10 different schemas and a
milion documents, the cost will be associated in creating SRR
relevance-sheets for those 10 schemas, compared to the cost of adding
semantic RDF information in each and every file.
This is the exact same concept of SoC between content and style, here
associated to the separation between content and its semantic relevance
interpretation.
2) the user experience is no different from the one he/she's used to: he
doesn't need to know the schema of the documents nor any information
about metadata or metadata fields in order to obtain the information.
3) the SRR drives the indexing behavior and indicates whether or not
some information should be indexed or not.
Since the relevance factors are multiplied by the indexer, text that is
associated to a context of 'zero' semantic relevance is skipped and
avoids polluting the index with information that might 'get in the way'
unwanted.
- o -
I'm currently working with the Lucene guys to add the ability to 'rate'
textual input in the indexer. When this is done, adding SRR capabilities
with Cocoon is a matter of writing the relevance sheets for our DTDs.
Comments?
--
Stefano Mazzocchi One must still have chaos in oneself to be
able to give birth to a dancing star.
<[EMAIL PROTECTED]> Friedrich Nietzsche
--------------------------------------------------------------------
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, email: [EMAIL PROTECTED]
