Paul and Wolfang,
Thank you very much for your input. I think there are two distinct
problems that have emerged from this thread:
1) The ability to create efficient structures to index and query XML
documents (element, attributes and corresponding values) with a
full-text query language and perforators. After all XML is text. As
Paul pointed out people have already tried this with Lucene.
2) The need for a standard query language like XQuery aiming at
system interoperability in the now XMLized world that has the same
effect that SQL had in the relational world.
While I can see how in the SQL case extension functions can be used
to implement full-text capabilities, in the XML case full-text is
required to query and retrieve XML (sub-document) elements and
attributes based on the free text (natural language) values AND
also to query the strings that represent the structure itself. For
example, in simple SQL queries the names of the tables and columns
need to be known to project corresponding values and are not part of
the search conditions (in WHERE clauses only values corresponding to
table/columns are evaluated).
In XQuery both the structure and the content are searchable, thus
requiring full-text operators. That is why XQuery Full-Text requires
the unification and standardization both XQuery and Full- Text
"languages". Needless is to say that the implementation will differ
from system to system.
I do agree though that the abstraction of full-text capabilities
through functional extensions is a great first step. Check out
Oracle's XML Query Service (http://www.oracle.com/technology/tech/
xml/xds/index.html and , http://www.oracle.com/technology/oramag/
oracle/05-mar/o25xml.html) a Java based XQuery engine that has
abstracted "data sources" such as Web Services, RDBMS, etc. as
functions that while returning XML can receive parameters and supply
full-text capabilities. If Mark's implementation of Lucene query and
output in XML comes to fruition a Lucene data source will become yet
another stream of XML that can be queried, processed and rendered by
the mid-tier XQuery engine.
-- Joaquin
While maintaining my bookmarks I ran into this:
"Case Study: Enabling Low-Cost XML-Aware Searching
Capable of Complex Querying":
http://www.idealliance.org/papers/xmle02/dx_xmle02/papers/
03-02-08/03-02-08.html
Some loose thoughts:
In the system described there a Lucene document is used for each
low level xml construct, even when it contains very few characters
of text.
The resulting Lucene indexes are at least 2.5 times the size of the
original document, which is not a surprise given this document
structure.
Normal index size is about one third of the indexed text.
I don't know about the XQuery standard, but I was wondering
whether this unusual document structure and the non straightforward
fit between Lucene queries and XQuery queries are related.
As for the joines and iterations over items from the stream of XML
results: iteration over matching XML constructs should be no problem
in Lucene. Joins in Lucene are normally done via boolean filters,
so I was wondering how XQuery joins fit these.
The case study above has a note a the end of par 5.3:
"The Search Result list that comes back could then be organized
by document id to group together all the results for a single XML
document. This is not provided by default, but has been done with
extension to this code."
Regards,
Paul Elschot
On Friday 16 December 2005 03:45, Wolfgang Hoschek wrote:
I think implementing an XQuery Full-Text engine is far beyond the
scope of Lucene.
Implementing a building block for the fulltext aspect of it would be
more manageable. Unfortunately The W3C fulltext drafts
indiscriminately mix and mingle two completely different languages
into a single language, without clear boundaries. That's why most
practical folks implement XQuery fulltext search via extension
functions rather than within XQuery itself. This also allows for much
more detailed tokenization, configuration and extensibility than what
would be possible with the W3C draft.
Wolfgang.
On Dec 15, 2005, at 4:20 PM, [EMAIL PROTECTED] wrote:
Mark,
This is very cool. When I was at TripleHop we did something very
similar where both query and results conformed to an XML Schema and
we used XML over HTTP as our main vehicle to do remote/federated
searches with quick rendering with stylesheets.
That however is the first piece of the puzzle. If you really want
to go beyond search (in the traditional sense) and be able to
perform more complex operations such as joines and iterations over
items from the stream of XML results you are getting you should
consider implementing an XQuery Full-Text engine with Lucene
adopting the now standard XQuery language.
Here is the pointer to the working draft on the W3C working draft
on XQuery 1.0 and XPath 2.0 Full-Text:
http://www.w3.org/TR/xquery-full-text/
Now I'm part of the task force editing this draft so your comments
are very much welcomed.
-- J.D.
http://www.inperspective.com/lucene/LXQueryV0_1.zip
I've implemented just a few queries (Boolean, Term, FilteredQuery,
BoostingQuery ...) but other queries are fairly trivial to add.
At this stage I am more interested in feedback on parser design/
approach
rather than trying to achieve complete coverage of all the Lucene
Query
types or debating the choice of tag names.
Please see the readme.txt in the package for more details.
Cheers
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
While maintaining my bookmarks I ran into this:
"Case Study: Enabling Low-Cost XML-Aware Searching
Capable of Complex Querying":
http://www.idealliance.org/papers/xmle02/dx_xmle02/papers/
03-02-08/03-02-08.html
Some loose thoughts:
In the system described there a Lucene document is used for each
low level xml construct, even when it contains very few characters
of text.
The resulting Lucene indexes are at least 2.5 times the size of the
original document, which is not a surprise given this document
structure.
Normal index size is about one third of the indexed text.
I don't know about the XQuery standard, but I was wondering
whether this unusual document structure and the non straightforward
fit between Lucene queries and XQuery queries are related.
As for the joines and iterations over items from the stream of XML
results: iteration over matching XML constructs should be no problem
in Lucene. Joins in Lucene are normally done via boolean filters,
so I was wondering how XQuery joins fit these.
The case study above has a note a the end of par 5.3:
"The Search Result list that comes back could then be organized
by document id to group together all the results for a single XML
document. This is not provided by default, but has been done with
extension to this code."
Regards,
Paul Elschot
On Friday 16 December 2005 03:45, Wolfgang Hoschek wrote:
I think implementing an XQuery Full-Text engine is far beyond the
scope of Lucene.
Implementing a building block for the fulltext aspect of it would be
more manageable. Unfortunately The W3C fulltext drafts
indiscriminately mix and mingle two completely different languages
into a single language, without clear boundaries. That's why most
practical folks implement XQuery fulltext search via extension
functions rather than within XQuery itself. This also allows for much
more detailed tokenization, configuration and extensibility than what
would be possible with the W3C draft.
Wolfgang.
On Dec 15, 2005, at 4:20 PM, [EMAIL PROTECTED] wrote:
Mark,
This is very cool. When I was at TripleHop we did something very
similar where both query and results conformed to an XML Schema and
we used XML over HTTP as our main vehicle to do remote/federated
searches with quick rendering with stylesheets.
That however is the first piece of the puzzle. If you really want
to go beyond search (in the traditional sense) and be able to
perform more complex operations such as joines and iterations over
items from the stream of XML results you are getting you should
consider implementing an XQuery Full-Text engine with Lucene
adopting the now standard XQuery language.
Here is the pointer to the working draft on the W3C working draft
on XQuery 1.0 and XPath 2.0 Full-Text:
http://www.w3.org/TR/xquery-full-text/
Now I'm part of the task force editing this draft so your comments
are very much welcomed.
-- J.D.
http://www.inperspective.com/lucene/LXQueryV0_1.zip
I've implemented just a few queries (Boolean, Term, FilteredQuery,
BoostingQuery ...) but other queries are fairly trivial to add.
At this stage I am more interested in feedback on parser design/
approach
rather than trying to achieve complete coverage of all the Lucene
Query
types or debating the choice of tag names.
Please see the readme.txt in the package for more details.
Cheers
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
