Hi everyone
I have developed an indexing store (subclassing j2ee/rdbms
adapter)(which indexes properties, NOT content), at the moment just for
MySQL 4.1+ but presumably easily transferable to any othe RDMS that
supports subselects. But...the code is horrible and undocumented,
because as soon as I got it working I then moved on to a "virtual" store
(i.e. a relational filesystem, which will be the sweetest thing as soon
as I've sorted out some sort of leak which means the connections don't
go back to the pool).
The reason I did some of the strange things I did, which I've marked
below with (!), is that I've made a refreshing Netbeans webdav client,
which for the sake of completeness needs to know if absolutely any
modifications have happened to a given uri/object/file, particularly
lock changes. My idea was to have a group of authors working on the same
set of files, only allowing offline content changes if the file's
checked out, rather than the local/remote model that's standard. I'll
explain why if anyone's interested, but in another mail.
So> a few notes/queries on the implementation. I can upload my sources,
but I'd need a little time to make it slightly less embarassing, and any
input/critiques I can get on the solution below (architecturally I find
Slide a completely impossible act to follow. Which makes me a little
shy).
My last update from CVS was early December, and I modified some
org.apache.slide files as needed - I'd imagine similar modifications
would be needed for any indexing implementation:
Search languages:
-----------------
There is API support for different search languages, but no actual
implementation.
In Domain.xml, I added a new parameter for search languages and patched
SearchImpl.java as per attachment
<parameter
name="search_language_classes">org.apache.slide.search.basic.BasicSearch
Language,
com.ella.alexis.chakriya.slide.indexing.MultipleSearchLanguage</paramete
r>
</configuration>
Property indexing
-----------------
Two separate issues - creation of the indexing table (s), and index
updating
Index table creation/rebuilding.
- the better way to do this would be through Slide Admin. But
1. is slide admin maintained? (no). do you want someone to maintain it?
2. is there any way in Tomcat/servlet containers to define webapp
dependencies, and hence classloader access across webapps, rather than
"copy it all into common", which sort of defeats the purpose of webapps?
My actual implementation uses a parameter in the nodestore element which
tells the adapter whether or not to rebuild the index table on adapter
initialisation.
The ideal way to support index building would, I think, be to define a
"resource" path in the filesystem itself, which the indexing services
use to determine the data types and tables for the various properties,
i.e. a folder
/files/_resources
containing
dav.xsd
myXMLNS1.xsd
...
which maps from xml data types to the various db types....and, of
course, allows validation checks if desired. (jakarta-db-ojb??)
I haven't done it this way, although on my client I already do, so I
will definitely move it across. Instead, I extended the old QName table
with [table_name] and [data_type] columns. Table (re)building goes:
1. update qname so it contains all defined properties
2. add any columns to the table(s) for which a qname record has a
defined data_type value.
3. index all files.
(!) Rather than use interceptors, every method in the adapter that
modifies the underlying RDBMS also reindexes the file.
Principally because my client is mostly interested in the "meta-meta"
property of whether anything (acl, lock, properties) have changed, so as
to cache successfully. Change propogation to children I simply ignored
(because for a relational system, which is what I was after, there's no
such thing as a definitive parent/child relationship).
SEARCH method
-------------
? How does the BasicQuery implementation access the datasouce? I re
protected-ed the "getConnection" method in a subclass of j2eeStore, in
the query implementation's package. Seems reasonable to me.
? How to search for meta-meta properties (i.e. change/delete
notification).
I defined some new properties which are only evaluated for
infinite depth searches on the filesystem root ("/")
It's wrong, and inelegant, among other things it means SEARCH
gives you info which PROPFIND doesn't, because in Slide the store has
much greater control over SEARCH than PROPFIND.
Ideally, I think, the real solution would be a PropertyProvider that
plugs into the store directly. But whole slabs of
src/webdav/server/org/apache/slide/webdav/util
use statements like the following:
AbstractResourceKind.isComputedProperty(propName)
which means any subclassing of the property provision section of slide
would mandate refactoring of these classes.
Performance
-----------
On my machine (a battered laptop with specs blah), the following on a
folder containing 500 files
getting the following, PROPFINDs equivalent:
<d:searchrequest xmlns:d="DAV:"
xmlns:alx="http://www.ella-associates.org/alexis/";>
<alx:multiplesearch xmlns:alx="http://www.ella-associates.org/alexis/";>
<d:basicsearch xmlns:d="DAV:">
<d:select>
<d:prop>
<d:displayname />
<d:getcontentlength />
<d:getlastmodified />
</d:prop>
</d:select>
<d:from>
<d:scope>
<d:href>/slide/files/documents/bigf</d:href>
<d:depth>1</d:depth>
</d:scope>
</d:from>
<d:where>
<d:gt>
<d:prop>
<d:resource-id />
</d:prop>
<d:literal>0</d:literal>
</d:gt>
</d:where>
</d:basicsearch>
</alx:multiplesearch></d:searchrequest>
***alx:multiplesearch bundles a lot of search requests, which means a
client wanting change info on a bunch of folders only has to send one
SEARCH
PROPFIND (1st time, revisiondescriptor cache building)
16294 ms
PROPFIND (2nd time, from cache)
9003 ms
SEARCH
772 ms
(and some of that's just the pipes waiting on each other)
Which sort of begs the question, why not allow PROPFIND to just call a
SEARCH, if appropriate? (And yes, I have got classes to generate a
SEARCH body).
"KNOWN ISSUES"
these aren't truly critical, because anything this search implementation
can't handle it just throws back to the default implementation
- doesn't allow ACL/Lockdiscovery in the select
- only supports depth 1 (or infinity for meta-meta)
- (Critical) doesn't resolve links/binding (but doable completely in
SQL, I just need to check out BindingStore)
///////////////////////
Any feedback on the above would be truly appreciated. Particularly from
Stefano Mazzocchi.
Nick Reddel
/*
* $Header: /home/cvspublic/jakarta-slide/src/share/org/apache/slide/search/SearchImpl.java,v 1.13 2002/06/19 11:00:13 juergen Exp $
* $Revision: 1.13 $
* $Date: 2002/06/19 11:00:13 $
*
* ====================================================================
*
* The Apache Software License, Version 1.1
*
* Copyright (c) 1999-2002 The Apache Software Foundation. All rights
* reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
*
* 1. Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
*
* 2. Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in
* the documentation and/or other materials provided with the
* distribution.
*
* 3. The end-user documentation included with the redistribution, if
* any, must include the following acknowlegement:
* "This product includes software developed by the
* Apache Software Foundation (http://www.apache.org/)."
* Alternately, this acknowlegement may appear in the software itself,
* if and wherever such third-party acknowlegements normally appear.
*
* 4. The names "The Jakarta Project", "Slide", and "Apache Software
* Foundation" must not be used to endorse or promote products derived
* from this software without prior written permission. For written
* permission, please contact [EMAIL PROTECTED]
*
* 5. Products derived from this software may not be called "Apache"
* nor may "Apache" appear in their names without prior written
* permission of the Apache Group.
*
* THIS SOFTWARE IS PROVIDED ``AS IS'' AND ANY EXPRESSED OR IMPLIED
* WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
* OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
* DISCLAIMED. IN NO EVENT SHALL THE APACHE SOFTWARE FOUNDATION OR
* ITS CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
* SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
* LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
* USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
* ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
* OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
* OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
* SUCH DAMAGE.
* ====================================================================
*
* This software consists of voluntary contributions made by many
* individuals on behalf of the Apache Software Foundation. For more
* information on the Apache Software Foundation, please see
* <http://www.apache.org/>.
*
* [Additional notices, if required by prior licensing conditions]
*
*/
package org.apache.slide.search;
import java.util.HashMap;
import java.util.Map;
import java.lang.reflect.Constructor;
import java.lang.reflect.InvocationTargetException;
import org.apache.slide.common.Namespace;
import org.apache.slide.common.NamespaceConfig;
import org.apache.slide.common.ServiceAccessException;
import org.apache.slide.common.SlideToken;
import org.apache.slide.structure.Structure;
import org.apache.slide.content.Content;
import org.apache.slide.authenticate.CredentialsToken;
import org.apache.slide.util.Configuration;
import org.apache.slide.util.logger.Logger;
import org.apache.slide.search.basic.BasicSearchLanguage;
import org.jdom.Element;
/**
* Search helper.
*
* @author <a href="mailto:[EMAIL PROTECTED]">Remy Maucherat</a>
* @version $Revision: 1.13 $
*/
public final class SearchImpl implements Search {
// ----------------------------------------------------------- Constructors
/**
* Constructor.
*
* @param namespace Namespace
* @param namespaceConfig Namespace configuration
*/
public SearchImpl(Namespace namespace,
NamespaceConfig namespaceConfig,
Structure structureHelper,
Content contentHelper) {
this.namespace = namespace;
this.namespaceConfig = namespaceConfig;
this.structureHelper = structureHelper;
this.contentHelper = contentHelper;
initializeLanguages();
}
private void initializeLanguages() {
String languageClassNames = namespaceConfig.getParameter("search_language_classes");
if (languageClassNames == null){
languageClassNames = "org.apache.slide.search.basic.BasicSearchLanguage";
}
String[] classNames = languageClassNames.split("[^a-zA-z0-9_.]+");
try{
searchLanguages = new SearchLanguage[classNames.length];
for (int i = 0; i < searchLanguages.length; i++){
searchLanguages [i] = (SearchLanguage) Class.forName(classNames[i]).newInstance();
}
}
catch (Exception e){
e.printStackTrace();
searchLanguages = new SearchLanguage[] {
new BasicSearchLanguage()
};
}
for (int i = 0; i < searchLanguages.length; i++){
grammarByUri.put(searchLanguages [i].getGrammarUri(),
searchLanguages [i]);
}
}
// ----------------------------------------------------- Instance Variables
// TODO: must be configurable in domain.xml (Namespace.java)
private SearchLanguage [] searchLanguages ;
private Map grammarByUri = new HashMap();
/**
* Log channel for logger
*/
private final static String LOG_CHANNEL = SearchImpl.class.getName();
/**
* Namespace.
*/
private Namespace namespace;
/**
* Structure.
*/
private Structure structureHelper;
/**
* Namespace configuration.
*/
private NamespaceConfig namespaceConfig;
/**
* Namespace configuration.
*/
private Content contentHelper;
// ------------------------------------------------------- Security Methods
/**
* Search.
*
* @param token Credentials token
* @param object Object on which permission is granted
* @exception ServiceAccessException DataSource access error
*/
public SearchQueryResult search(SlideToken token, SearchQuery query)
throws ServiceAccessException {
return query.execute();
}
/**
* Return the allowed query languages.
*/
public SearchLanguage[] getSupportedLanguages() {
return searchLanguages;
}
/**
* Retrieve a SearchLanguage identified by the grammar uri (namespace)
*
* @param grammarUri identifier for the SearchLanguage
*
* @return the SearchLanguage
*
* @throws BadQueryException
*
*/
public SearchLanguage getLanguage(String grammarUri)
throws BadQueryException {
SearchLanguage result =(SearchLanguage) grammarByUri.get(grammarUri);
if (result == null)
throw new BadQueryException("grammar not found: " + grammarUri);
return result;
}
/**
* Creates a SearchQuery.
*
* @param grammarUri identifier for the SearchLanguage
* @param queryElement the JDOM element containing the query
* @param token the SlideToken
* @param maxDepth may be 0, 1 or INFINIT
*
* @return the SearchQuery
*
* @throws BadQueryException
*
*/
public SearchQuery createSearchQuery(String grammarUri,
Element queryElement,
SlideToken token,
int maxDepth)
throws BadQueryException {
return createSearchQuery(grammarUri, queryElement, token, maxDepth, (PropertyProvider)null);
}
/**
* Creates a SearchQuery.
*
* @param grammarUri identifier for the SearchLanguage.
* @param searchRequestElement the JDOM element containing the query
* @param token the SlideToken.
* @param maxDepth may be 0, 1 or INFINITY.
* @param propertyProvider the PropertyProvider to use (may be
* <code>null</code>).
*
* @return the SearchQuery
*
* @throws BadQueryException
*/
public SearchQuery createSearchQuery(String grammarUri,
Element searchRequestElement,
SlideToken token,
int maxDepth,
PropertyProvider propertyProvider)
throws BadQueryException {
return createSearchQuery(grammarUri, searchRequestElement, token, maxDepth, propertyProvider, null);
}
/**
* Creates a SearchQuery.
*
* @param grammarUri identifier for the SearchLanguage
* @param queryElement the JDOM element containing the query
* @param token the SlideToken
* @param maxDepth may be 0, 1 or INFINIT
*
* @return the SearchQuery
*
* @throws BadQueryException
*
*/
public SearchQuery createSearchQuery(String grammarUri,
Element queryElement,
SlideToken token,
int maxDepth,
String requestUri)
throws BadQueryException {
return createSearchQuery(grammarUri, queryElement, token, maxDepth, null, requestUri);
}
/**
* Creates a SearchQuery.
*
* @param grammarUri identifier for the SearchLanguage.
* @param searchRequestElement the JDOM element containing the query
* @param token the SlideToken.
* @param maxDepth may be 0, 1 or INFINITY.
* @param propertyProvider the PropertyProvider to use (may be
* <code>null</code>).
* @param requestUri the URI of the request.
*
* @return the SearchQuery
*
* @throws BadQueryException
*/
public SearchQuery createSearchQuery(String grammarUri,
Element searchRequestElement,
SlideToken token,
int maxDepth,
PropertyProvider propertyProvider,
String requestUri)
throws BadQueryException {
SearchQuery result = null;
// create search token
SearchToken searchToken =
SearchToken.createSearchToken(token, contentHelper,
structureHelper, maxDepth,
requestUri, namespace);
SearchLanguage language = getLanguage(grammarUri);
result = language.parseQuery(searchRequestElement, searchToken, propertyProvider);
return result;
}
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
