Author: ahorincar
Date: Sun Aug 17 20:22:10 2014
New Revision: 1618520
URL: http://svn.apache.org/r1618520
Log:
Added docstrings, refactored code, added unit test for checking if the schema
is valid, added README file containing the installation guide.
Added:
bloodhound/branches/bep_0014_solr/bloodhound_solr/README.wiki
bloodhound/branches/bep_0014_solr/bloodhound_solr/bhsolr/tests/data/
bloodhound/branches/bep_0014_solr/bloodhound_solr/bhsolr/tests/data/schema_test_file.xml
Removed:
bloodhound/branches/bep_0014_solr/bloodhound_solr/README
bloodhound/branches/bep_0014_solr/bloodhound_solr/bhsolr/tests/backend.py
bloodhound/branches/bep_0014_solr/bloodhound_solr/bhsolr/tests/search_resources/
Modified:
bloodhound/branches/bep_0014_solr/bloodhound_solr/bhsolr/admin.py
bloodhound/branches/bep_0014_solr/bloodhound_solr/bhsolr/schema.py
bloodhound/branches/bep_0014_solr/bloodhound_solr/bhsolr/solr_backend.py
bloodhound/branches/bep_0014_solr/bloodhound_solr/bhsolr/tests/__init__.py
bloodhound/branches/bep_0014_solr/bloodhound_solr/bhsolr/tests/schema.py
bloodhound/branches/bep_0014_solr/bloodhound_solr/setup.py
bloodhound/branches/bep_0014_solr/bloodhound_theme/bhtheme/templates/bh_more_like_this.html
Added: bloodhound/branches/bep_0014_solr/bloodhound_solr/README.wiki
URL:
http://svn.apache.org/viewvc/bloodhound/branches/bep_0014_solr/bloodhound_solr/README.wiki?rev=1618520&view=auto
==============================================================================
--- bloodhound/branches/bep_0014_solr/bloodhound_solr/README.wiki (added)
+++ bloodhound/branches/bep_0014_solr/bloodhound_solr/README.wiki Sun Aug 17
20:22:10 2014
@@ -0,0 +1,149 @@
+= Plugin to add Apache Solr as a search alternative for Apache Bloodhound
+
+== Description
+
+The plugin enhances Apache Bloodhound with Apache Solr functionality, namely
+it allows using Apache Solr as a search alternative.
+
+== Dependencies
+
+This plugin depends on the following components to be installed:
+
+ - [http://trac.edgewall.org Trac] ,,Since version ''' 1.0 ''',, .
+ - [http://lucene.apache.org/solr/index.html Apache Solr] ,,Version '''
+ 4.7.2 ''',, .
+ - [http://lxml.de/ Lxml]
+ - [https://github.com/tow/sunburnt Sunburnt] ,,Since version '''0.6''',, .
+ - [https://code.google.com/p/httplib2/ Httplib2]
+
+
+== Latest Version
+The latest version of the source code for the plugin can be found
+[https://svn.apache.org/repos/asf/bloodhound/branches/bep_0014_solr/ here].
+
+== Installation
+
+== Apache Solr
+This plugin has been tested with Apache Solr, '''version 4.7.2.'''[[BR]]
+
+Before proceeding to install Apache Solr, you need to have '''Java 1.7''' or
+greater installed.
+
+Apache Solr can be downloaded from
+[http://www.apache.org/dyn/closer.cgi/lucene/solr here].[[BR]]
+
+Once you've downloaded Apache Solr, unzip it in your chosen directory and
+navigate (in a terminal) to the
+
+{{{
+<path_to_solr_dir>/example directory
+}}}
+
+The Solr Server can be started by issuing the following command:
+[[BR]]
+
+'''On UNIX''':
+{{{
+java -jar start.jar
+}}}
+
+
+'''On Windows''':
+{{{
+java -Dsolr.solr.home=<path_to_solr_dir> -jar start.jar
+}}}
+
+
+You can test whether this worked by loading
+{{{
+http://localhost:8983/solr/
+}}}
+
+in your browser.
+
+== Bloodhound Solr plugin
+Once downloaded, the plugin should be placed in the directory where you have
+installed your Bloodhound instance (e.g. /home/bloodhound).
+[[BR]]
+
+You should then make the following changes in the base.ini file:
+
+Under the '''[bhsearch]''' section, add the following:
+
+{{{
+search_backend = SolrBackend
+solr_server_url = http://localhost:8983/solr/
+}}}
+
+
+Under the '''[components]''' section, add the following:
+
+{{{
+bhsolr.* = enabled
+}}}
+
+
+'''Note:''' For compatibility with Apache Solr, you should choose a default
+field to be used when
+no query is specified. This can be updated in the '''solrconfig.xml''' file
+found at:
+
+{{{
+<your_solr_directory>/libexec/example/solr/collection1/conf
+}}}
+
+Search for the request handler having the name set to ''''/select'''', and
+modify the line
+
+{{{
+ <str name="df">text</str>
+}}}
+to
+{{{
+ <str name="df">your_chosen_default_field</str>
+}}}
+
+Make sure your chosen default field exists in the Bloodhound schema (see below
+the existing fields in the Bloodhound schema).[[BR]]
+Current Bloodhound schema: '''unique_id''', '''id''', '''type''',
+'''product''', '''milestone''', '''time''', '''due''', '''completed''',
+'''author''', '''component''', '''status''', '''resolution''', '''keywords''',
+'''summary''', '''content''', '''changes''', '''owner''', '''repository''',
+'''revision''', '''message''', '''required_permission''', '''name''',
+'''query_suggestion_basket''', '''relations'''.
+
+Once all these steps have been accomplished, install the Bloodhound Solr
+plugin as described
+[http://trac.edgewall.org/wiki/TracPlugins#InstallingaTracplugin here].
+
+The next and final step is to generate the '''schema.xml''' file to be used
+with your Solr instance. From the '''bloodhound/installer''' directory,
+issue the following '''trac-admin''' command:
+
+{{{
+trac-admin <path_to_your_bloodhound_environment> bhsolr generate_schema
+<path_to_your_solr_directory>/libexec/example/solr/collection1/conf
+}}}
+
+'''Note:''' Make sure your Bloodhound environment is activated and your Solr
+server is running. Also, if you don't provide the path to where the schema
+should be generated, the program will generate in your current working
+directory.
+
+In order for Solr to use the newly generated schema, you need to restart your
+Solr server.
+
+At this point you should be able to use Apache Solr as a search platform for
+Bloodhound.
+
+== Licensing
+
+The plugin is licensed under the [http://www.apache.org/licenses/LICENSE-2.0
+Apache License] ,,Version '''2.0''',, .
+
+== Bug / feature requests
+
+Existing bugs and feature requests for the Bloodhound Solr plugin are
+[query:status=new|assigned|reopened&keywords=~bep-0014 here].
+If you have any issues, please create a
+[/newticket?keywords=bep-0014 new ticket].
Modified: bloodhound/branches/bep_0014_solr/bloodhound_solr/bhsolr/admin.py
URL:
http://svn.apache.org/viewvc/bloodhound/branches/bep_0014_solr/bloodhound_solr/bhsolr/admin.py?rev=1618520&r1=1618519&r2=1618520&view=diff
==============================================================================
--- bloodhound/branches/bep_0014_solr/bloodhound_solr/bhsolr/admin.py (original)
+++ bloodhound/branches/bep_0014_solr/bloodhound_solr/bhsolr/admin.py Sun Aug
17 20:22:10 2014
@@ -17,16 +17,35 @@
# specific language governing permissions and limitations
# under the License.
+
+"""
+Module holding the implementation of the IAdminCommandProvider
+interface, so that a new trac-admin command is available to be used.
+"""
+
from trac.core import Component, implements
from trac.admin import IAdminCommandProvider
from bhsolr.schema import SolrSchema
class BloodhoundSolrAdmin(Component):
-
implements(IAdminCommandProvider)
- # IAdminCommandProvider methods
+ """Class implementing the IAdminCommandProvider.
+
+ Provide a new trac-admin command, suitable for use with the
+ Bloodhound Solr plugin.
+ """
+
def get_admin_commands(self):
+ """Generate a new trac-admin command.
+
+ Create a trac-admin command for generating a schema.xml file.
+ Yields a touple containing the command name for generating a
+ Solr schema, the argument description (i.e. the path to where
+ the Solr schema should be generated, a help text and the
+ callback function that generates the Solr schema.
+ """
+
yield ('bhsolr generate_schema', '<path>',
'Generate Solr schema',
None, SolrSchema(self.env).generate_schema)
Modified: bloodhound/branches/bep_0014_solr/bloodhound_solr/bhsolr/schema.py
URL:
http://svn.apache.org/viewvc/bloodhound/branches/bep_0014_solr/bloodhound_solr/bhsolr/schema.py?rev=1618520&r1=1618519&r2=1618520&view=diff
==============================================================================
--- bloodhound/branches/bep_0014_solr/bloodhound_solr/bhsolr/schema.py
(original)
+++ bloodhound/branches/bep_0014_solr/bloodhound_solr/bhsolr/schema.py Sun Aug
17 20:22:10 2014
@@ -17,6 +17,11 @@
# specific language governing permissions and limitations
# under the License.
+"""
+Module providing a feature to generate a schema.xml file containing
+the definition of a Solr schema that can be used for Bloodhound.
+"""
+
import os
from lxml import etree
@@ -27,12 +32,30 @@ from bhsolr.solr_backend import SolrBack
class SolrSchema(Component):
+ """Class for creating a schema.xml file.
+
+ Define the fields to be written to the schema.xml file, and write
+ it at a specified path.
+
+ Class Attributes:
+ REQUIRED_FIELDS -- the fields required for each searchable object
+ FIELDS_TYPE_DICT -- the types each field can have, along with its
+ equivalent name to be used in the Solr schema.xml file
+
+ Instance Attributes:
+ path -- the path to where tha schema should be saved
+ schema -- the schema definition from bhsearch.WhooshBackend
+ schema_element -- the main etree.Element for the schema.xml file
+ fields_element -- the etree.SubElement for defining the fields
+ unique_key_element -- the etree.SubElement for defining the unique
+ key
+ """
+
REQUIRED_FIELDS = {
"id": True,
"unique_id": True,
"type": True
}
-
FIELDS_TYPE_DICT = {
"ID": "string",
"DATETIME": "date",
@@ -41,8 +64,12 @@ class SolrSchema(Component):
}
def __init__(self):
+ """Initialise main XML elements and set their values."""
self.path = None
+
+ # The main parent elements.
self.schema = WhooshBackend.SCHEMA
+
self.schema_element = etree.Element("schema")
self.schema_element.set("name", "Bloodhound Solr Schema")
self.schema_element.set("version", "1")
@@ -53,6 +80,9 @@ class SolrSchema(Component):
"uniqueKey")
self.unique_key_element.text = SolrBackend.UNIQUE_ID
+ # Children of the element containing the definitions of fields;
+ # these are required so that Solr is able to validate the
+ # schema.
version_field = etree.SubElement(self.fields_element, "field")
version_field.set("name", "_version_")
version_field.set("type", "long")
@@ -75,13 +105,29 @@ class SolrSchema(Component):
stored_name.set("multiValued", "false")
def generate_schema(self, path=None):
+ """Write a schema.xml file.
+
+ Create all XML elements needed in the schema.xml file, along
+ with their values and write the file.
+
+ Keyword Arguments:
+ path -- the path to where the schema.xml file should be saved
+ (default None)
+ """
+
+ # If the user hasn't provided a path to where the schema.xml
+ # file should be saved, then the schema.xml file will be saved
+ # in the current directory.
if not path:
path = os.getcwd()
self.path = os.path.join(path, 'schema.xml')
- self.add_all_fields()
+ # Adds all fields and all type definitions to the main
+ # etree.Element
+ self.prepare_all_fields()
self.add_type_definitions()
+ # Writes the schema file.
doc = etree.ElementTree(self.schema_element)
out_file = open(os.path.join(path, 'schema.xml'), 'w')
doc.write(out_file, xml_declaration=True, encoding='UTF-8',
@@ -91,6 +137,25 @@ class SolrSchema(Component):
def add_field(
self, field_name, name_attr, type_attr, indexed_attr,
stored_attr, required_attr, multivalued_attr):
+ """Add field to the etree.Sublement object.
+
+ Add a child to the 'fields' SubElement, and set the
+ attributes for this child.
+
+ Keyword Arguments:
+ field_name -- the name of the XML element
+ name_attr -- the name of the schema field
+ type_attr -- the type of the schema field
+ indexed_attr -- boolean for keeping track whether the schema
+ field should be indexed or not
+ stored_attr -- boolean for keeping track whether the schema
+ field should be stored or not
+ required_attr -- boolean for keeping track whether the schema
+ field is required or not
+ multivalued_attr -- boolean for keeping track whether the
+ schema field is multivalued or not
+ """
+
field = etree.SubElement(self.fields_element, field_name)
field.set("name", name_attr)
field.set("type", type_attr)
@@ -99,7 +164,13 @@ class SolrSchema(Component):
field.set("required", required_attr)
field.set("multiValued", multivalued_attr)
- def add_all_fields(self):
+ def prepare_all_fields(self):
+ """Prepare the attributes for each field in the schema.
+
+ Iterate through all the schema fields, and get the values for
+ each attribute needed.
+ """
+
for (field_name, field_attrs) in self.schema.items():
class_name = str(field_attrs.__class__.__name__)
type_attr = self.FIELDS_TYPE_DICT[class_name]
@@ -111,10 +182,12 @@ class SolrSchema(Component):
else:
required_attr = "false"
+ # Add field to the etree.Subelement holding all fields.
self.add_field("field", field_name, type_attr, indexed_attr,
stored_attr, required_attr, "false")
def add_type_definitions(self):
+ """Add definitions of all types used for the schema fields."""
self.types_element = etree.SubElement(self.schema_element, "types")
self._add_string_type_definition()
self._add_text_general_type_definition()
@@ -123,12 +196,14 @@ class SolrSchema(Component):
self._add_lowercase_type_definition()
def _add_string_type_definition(self):
+ """Create the XML definition of the 'string' type."""
field_type = etree.SubElement(self.types_element, "fieldType")
field_type.set("name", "string")
field_type.set("class", "solr.StrField")
field_type.set("sortMissingLast", "true")
def _add_text_general_type_definition(self):
+ """Create the XML definition of the 'text_general' type."""
field_type = etree.SubElement(self.types_element, "fieldType")
field_type.set("name", "text_general")
field_type.set("class", "solr.TextField")
@@ -169,6 +244,7 @@ class SolrSchema(Component):
analyzer_query_filter_l.set("class", "solr.LowerCaseFilterFactory")
def _add_date_type_definition(self):
+ """Create the XML definition of the 'date' type."""
field_type = etree.SubElement(self.types_element, "fieldType")
field_type.set("name", "date")
field_type.set("class", "solr.TrieDateField")
@@ -176,6 +252,7 @@ class SolrSchema(Component):
field_type.set("positionIncrementGap", "0")
def _add_long_type_definition(self):
+ """Create the XML definition of the 'long' type."""
field_type = etree.SubElement(self.types_element, "fieldType")
field_type.set("name", "long")
field_type.set("class", "solr.TrieLongField")
@@ -183,6 +260,7 @@ class SolrSchema(Component):
field_type.set("positionIncrementGap", "0")
def _add_lowercase_type_definition(self):
+ """Create the XML definition of the 'lowercase' type."""
field_type = etree.SubElement(self.types_element, "fieldType")
field_type.set("name", "lowercase")
field_type.set("class", "solr.TextField")
Modified:
bloodhound/branches/bep_0014_solr/bloodhound_solr/bhsolr/solr_backend.py
URL:
http://svn.apache.org/viewvc/bloodhound/branches/bep_0014_solr/bloodhound_solr/bhsolr/solr_backend.py?rev=1618520&r1=1618519&r2=1618520&view=diff
==============================================================================
--- bloodhound/branches/bep_0014_solr/bloodhound_solr/bhsolr/solr_backend.py
(original)
+++ bloodhound/branches/bep_0014_solr/bloodhound_solr/bhsolr/solr_backend.py
Sun Aug 17 20:22:10 2014
@@ -17,6 +17,11 @@
# specific language governing permissions and limitations
# under the License.
+"""
+Module providing an implementation of the bhsearch.api.ISearchBackend
+interface, suitable for use with Apache Solr.
+"""
+
import re
import hashlib
@@ -40,8 +45,27 @@ from trac.util.datefmt import utc
class SolrBackend(Component):
implements(ISearchBackend)
- UNIQUE_ID = "unique_id"
+ """Class for implementing the ISearchBackend interface.
+
+ Define the ISearchBackend methods, so that these methods can be
+ appropriately used for communicating with the Solr server (via
+ Sunburnt).
+
+ Class Attributes:
+ UNIQUE_ID -- the field to be used for distinguishing between
+ searchable objects
+ HIGHLIGHTABLE_FIELDS -- the fields that can be highlighted in
+ search results
+ server_url -- Option object for adding a new configuration option
+ (that lets the user set the Solr server URL) to the trac.ini
+ configuration file
+
+ Instance Attributes:
+ solr_interface -- a SolrInterface object for communicating with the
+ Solr server through Sunburnt
+ """
+ UNIQUE_ID = "unique_id"
HIGHLIGHTABLE_FIELDS = {
"unique_id" : True,
"id" : True,
@@ -62,7 +86,6 @@ class SolrBackend(Component):
"message" : True,
"name" : True
}
-
server_url = Option(
BHSEARCH_CONFIG_SECTION,
'solr_server_url',
@@ -70,31 +93,86 @@ class SolrBackend(Component):
doc_domain='bhsearch')
def __init__(self):
+ """Initialise a SolrInterface object.
+
+ Initialise the SolrInterface object with the Solr server
+ URL provided in trac.ini.
+ """
+
self.solr_interface = SolrInterface(str(self.server_url))
def add_doc(self, doc, operation_context=None):
+ """Add a new document to the Solr index.
+
+ The contents should be a dict with fields matching the search
+ schema. The only required fields are type and id, everything
+ else is optional.
+
+ Keyword Arguments:
+ doc -- the document to be added
+ operation_context -- required by the ISearchBackend API
+ (default None)
+ """
+
self._reformat_doc(doc)
+ # Create a unique ID for distinguishing between the searchable
+ # objects.
doc[self.UNIQUE_ID] = self._create_unique_id(doc.get("product", ''),
- doc["type"], doc["id"])
+ doc["type"], doc["id"])
self.solr_interface.add(doc)
self.solr_interface.commit()
- def delete_doc(product, doc_type, doc_id, operation_context=None):
+ def delete_doc(self, product, doc_type, doc_id, operation_context=None):
+ """Delete a document from the Solr index.
+
+ The product, type and ID of the document must be provided.
+
+ Keyword Arguments:
+ product -- the product associated with the searchable object
+ doc_type -- the type of the searchable object
+ doc_id -- the ID of the searchable object
+ operation_context -- required by the ISearchBackend API
+ (default None)
+ """
+
unique_id = self._create_unique_id(product, doc_type, doc_id)
self.solr_interface.delete(unique_id)
def optimize(self):
+ """Optimise the Solr index."""
self.solr_interface.optimize()
def query(
self, query, query_string, sort = None, fields = None,
filter = None, facets = None, pagenum = 1, pagelen = 20,
highlight = False, highlight_fields = None, context = None):
+ """Process the query to be made to the Solr server.
+
+ Create a query chain, and execute the query to the Solr server.
+ Return the results, the More Like This results and their
+ associated hexdigests.
+
+ Keyword Arguments:
+ query -- a whoosh.Query object holding the parsed query
+ query_string -- the original query string
+ sort -- a list of SortInstruction objects (default None)
+ fields -- a list of fields to select (default None)
+ filter -- a filter query object (default None)
+ facets -- a list of facet fields (default None)
+ pagenum -- the number of pages (default 1)
+ pagelen -- the maximum page length (default 20)
+ highlight -- highlight matched terms in field (default False)
+ highlight_fields -- a list of fields to highlight
+ (default None)
+ context -- required by the ISearchBackend API
+ (default None)
+ """
if not query_string:
query_string = "*.*"
- final_query_chain = self._create_query_chain(query, query_string)
+ # Create the query chain to be queried against Solr.
+ final_query_chain = self._create_query_chain(query_string)
solr_query = self.solr_interface.query(final_query_chain)
faceted_solr_query = solr_query.facet_by(facets)
highlighted_solr_query = faceted_solr_query.highlight(
@@ -105,16 +183,32 @@ class SolrBackend(Component):
start=start, rows=pagelen)
results = paginated_solr_query.execute()
+ # Process the More Like This results for the original query.
mlt, hexdigests = self.query_more_like_this(paginated_solr_query,
fields="type", mindf=1,
mintf=1)
+ # Process the query result so that it is compatible with the
+ # bhsearch.web_ui internal methods that display the results.
query_result = self._create_query_result(highlighted_solr_query,
results, fields, pagenum,
pagelen)
return query_result, mlt, hexdigests
def query_more_like_this(self, query_chain, **kwargs):
+ """Retrieve and process More Like These results.
+
+ Execute the query_chain against the Solr server, create a
+ hexdigest for each document retrieved (for interface purposes),
+ process the result accordingly and return a dictionary
+ containing the More Like This results and a dictionary holding
+ the hexdigests for each document.
+
+ Keyword Arguments:
+ query_chain -- the query chain to be executed to the Solr server
+ **kwargs -- remaining keyword arguments
+ """
+
mlt_results = query_chain.mlt(**kwargs).execute().more_like_these
mlt_dict = {}
hexdigests = {}
@@ -125,13 +219,22 @@ class SolrBackend(Component):
for mlt_doc in results.docs:
if doc not in mlt_dict:
- mlt_dict[doc] = [self._process_doc(mlt_doc)]
+ mlt_dict[doc] = [self._process_mlt_doc(mlt_doc)]
else:
- mlt_dict[doc].append(self._process_doc(mlt_doc))
+ mlt_dict[doc].append(self._process_mlt_doc(mlt_doc))
return mlt_dict, hexdigests
- def _process_doc(self, doc):
+ def _process_mlt_doc(self, doc):
+ """Build a dictionary containing required details for the doc.
+
+ Store the required field values for the doc and return this
+ dictionary.
+
+ Keyword Arguments:
+ doc -- a SolrResult object holding the document's details
+ """
+
ui_doc = dict(doc)
if doc.get('product'):
@@ -147,9 +250,19 @@ class SolrBackend(Component):
def _create_query_result(
self, query, results, fields, pagenum, pagelen):
+ """Create a QueryResult object with the retrieved results.
+
+ Keyword Arguments:
+ query -- a whoosh.Query object holding the parsed query
+ results -- a SolrResponse object holding the retrieve results
+ fields -- a list of fields to select
+ pagenum -- the number of pages
+ pagelen -- the maximum page length
+ """
+
total_num, total_page_count, page_num, offset = \
- self._prepare_query_result_attributes(query, results,
- pagenum, pagelen)
+ self._prepare_query_result_attributes(query, pagenum,
+ pagelen)
query_results = QueryResult()
query_results.hits = total_num
@@ -172,7 +285,19 @@ class SolrBackend(Component):
return query_results
- def _create_query_chain(self, query, query_string):
+ def _create_query_chain(self, query_string):
+ """Create the final query chain.
+
+ For each token in the query string, create a query with the
+ token against each field. Concatenate all created queries, and
+ return the final query chain.
+
+ Keyword Arguments:
+ query_string -- the original query string
+ """
+
+ # Match the regex against the query string, in order to find
+ # all the words in the query.
matches = re.findall(re.compile(r'([\w\*]+)'), query_string)
tokens = set([match for match in matches])
@@ -187,6 +312,17 @@ class SolrBackend(Component):
return final_query_chain
def _process_record(self, fields, retrieved_record):
+ """Process attributes for the retrieved document or record.
+
+ Create a dictionary holding all fields with their retrieved
+ value from the record.
+
+ Keyword Arguments:
+ retrieved_record -- a dictionary holding the retrieved record's
+ attributes
+ fields -- a list of fields to select
+ """
+
result_doc = dict()
if fields:
for field in fields:
@@ -202,12 +338,22 @@ class SolrBackend(Component):
return result_doc
def _from_whoosh_format(self, value):
+ """Find the timezone for the datetime instance."""
if isinstance(value, datetime):
value = utc.localize(value)
return value
- def _prepare_query_result_attributes(
- self, query, results, pagenum, pagelen):
+ def _prepare_query_result_attributes(self, query, pagenum, pagelen):
+ """Prepare the QueryResult attributes.
+
+ Prepare the attributes needed for creating a QueryResult object.
+
+ Keyword Arguments:
+ query -- the query to be executed against Solr
+ pagenum -- the number of pages
+ pagelen -- the maximum number of results per page
+ """
+
results_total_num = query.execute().result.numFound
total_page_count = int(ceil(results_total_num / pagelen))
pagenum = min(total_page_count, pagenum)
@@ -219,9 +365,12 @@ class SolrBackend(Component):
return results_total_num, total_page_count, pagenum, offset
def is_index_outdated(self):
+ # Not applicable to SolrBackend.
return False
def recreate_index(self):
+ # This method is replaced by the trac-admin command to generate
+ # a Solr schema.
return True
@contextmanager
@@ -229,20 +378,28 @@ class SolrBackend(Component):
yield
def _search_fields_for_token(self, token):
+ """Create a query chain for each field.
+
+ For each field, create a query chain querying for the token.
+
+ Keyword Arguments:
+ token -- a String holding the current token
+ """
+
q_chain = None
field_boosts = DefaultQueryParser(self.env).field_boosts
for field, boost in field_boosts.iteritems():
- if field != 'query_suggestion_basket' and field != 'relations':
- field_token_dict = {field: token}
- if q_chain is None:
- q_chain = self.solr_interface.Q(**field_token_dict)**boost
- else:
- q_chain |= self.solr_interface.Q(**field_token_dict)**boost
+ field_token_dict = {field: token}
+ if q_chain is None:
+ q_chain = self.solr_interface.Q(**field_token_dict)**boost
+ else:
+ q_chain |= self.solr_interface.Q(**field_token_dict)**boost
return q_chain
def _reformat_doc(self, doc):
+ # Needed for compatibility with bhsearch
for key, value in doc.items():
if key is None:
del doc[None]
@@ -254,6 +411,7 @@ class SolrBackend(Component):
doc[key] = self._to_whoosh_format(value)
def _to_whoosh_format(self, value):
+ # Needed for compatibility with bhsearch
if isinstance(value, basestring):
value = unicode(value)
elif isinstance(value, datetime):
@@ -261,12 +419,14 @@ class SolrBackend(Component):
return value
def _convert_date_to_tz_naive_utc(self, value):
+ # Needed for compatibility with bhsearch
if value.tzinfo:
utc_time = value.astimezone(utc)
value = utc_time.replace(tzinfo=None)
return value
def _create_unique_id(self, product, doc_type, doc_id):
+ """Create a unique ID for the doc."""
if product:
return u"%s:%s:%s" % (product, doc_type, doc_id)
else:
Modified:
bloodhound/branches/bep_0014_solr/bloodhound_solr/bhsolr/tests/__init__.py
URL:
http://svn.apache.org/viewvc/bloodhound/branches/bep_0014_solr/bloodhound_solr/bhsolr/tests/__init__.py?rev=1618520&r1=1618519&r2=1618520&view=diff
==============================================================================
--- bloodhound/branches/bep_0014_solr/bloodhound_solr/bhsolr/tests/__init__.py
(original)
+++ bloodhound/branches/bep_0014_solr/bloodhound_solr/bhsolr/tests/__init__.py
Sun Aug 17 20:22:10 2014
@@ -1,15 +1,36 @@
+# -*- coding: UTF-8 -*-
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
try:
import unittest2 as unittest
except ImportError:
import unittest
from bhsolr.tests import (
- backend
+ solr_backend, schema
)
def suite():
test_suite = unittest.TestSuite()
+ test_suite.addTest(solr_backend.suite())
+ test_suite.addTest(schema.suite())
return test_suite
if __name__ == '__main__':
Added:
bloodhound/branches/bep_0014_solr/bloodhound_solr/bhsolr/tests/data/schema_test_file.xml
URL:
http://svn.apache.org/viewvc/bloodhound/branches/bep_0014_solr/bloodhound_solr/bhsolr/tests/data/schema_test_file.xml?rev=1618520&view=auto
==============================================================================
---
bloodhound/branches/bep_0014_solr/bloodhound_solr/bhsolr/tests/data/schema_test_file.xml
(added)
+++
bloodhound/branches/bep_0014_solr/bloodhound_solr/bhsolr/tests/data/schema_test_file.xml
Sun Aug 17 20:22:10 2014
@@ -0,0 +1,57 @@
+<?xml version='1.0' encoding='UTF-8'?>
+<schema name="Bloodhound Solr Schema" version="1">
+ <fields>
+ <field name="_version_" type="long" indexed="true" stored="true"
multiValued="false"/>
+ <field name="_root_" type="string" indexed="true" stored="false"/>
+ <field name="_stored_name" type="string" indexed="true" stored="true"
required="false" multiValued="false"/>
+ <field name="author" type="string" indexed="true" stored="true"
required="false" multiValued="false"/>
+ <field name="changes" type="text_general" indexed="true" stored="false"
required="false" multiValued="false"/>
+ <field name="completed" type="date" indexed="true" stored="true"
required="false" multiValued="false"/>
+ <field name="component" type="string" indexed="true" stored="true"
required="false" multiValued="false"/>
+ <field name="content" type="text_general" indexed="true" stored="true"
required="false" multiValued="false"/>
+ <field name="due" type="date" indexed="true" stored="true"
required="false" multiValued="false"/>
+ <field name="id" type="string" indexed="true" stored="true"
required="true" multiValued="false"/>
+ <field name="keywords" type="string" indexed="true" stored="false"
required="false" multiValued="false"/>
+ <field name="message" type="text_general" indexed="true" stored="true"
required="false" multiValued="false"/>
+ <field name="milestone" type="string" indexed="true" stored="true"
required="false" multiValued="false"/>
+ <field name="name" type="text_general" indexed="true" stored="true"
required="false" multiValued="false"/>
+ <field name="owner" type="text_general" indexed="true" stored="true"
required="false" multiValued="false"/>
+ <field name="product" type="string" indexed="true" stored="true"
required="false" multiValued="false"/>
+ <field name="query_suggestion_basket" type="text_general" indexed="true"
stored="false" required="false" multiValued="false"/>
+ <field name="relations" type="string" indexed="true" stored="false"
required="false" multiValued="false"/>
+ <field name="repository" type="text_general" indexed="true" stored="true"
required="false" multiValued="false"/>
+ <field name="required_permission" type="string" indexed="true"
stored="false" required="false" multiValued="false"/>
+ <field name="resolution" type="string" indexed="true" stored="true"
required="false" multiValued="false"/>
+ <field name="revision" type="text_general" indexed="true" stored="true"
required="false" multiValued="false"/>
+ <field name="status" type="string" indexed="true" stored="true"
required="false" multiValued="false"/>
+ <field name="summary" type="text_general" indexed="true" stored="true"
required="false" multiValued="false"/>
+ <field name="time" type="date" indexed="true" stored="true"
required="false" multiValued="false"/>
+ <field name="type" type="string" indexed="true" stored="true"
required="true" multiValued="false"/>
+ <field name="unique_id" type="string" indexed="true" stored="true"
required="true" multiValued="false"/>
+ </fields>
+ <uniqueKey>unique_id</uniqueKey>
+ <types>
+ <fieldType name="string" class="solr.StrField" sortMissingLast="true"/>
+ <fieldType name="text_general" class="solr.TextField"
positionIncrementGap="100">
+ <analyzer type="index">
+ <tokenizer class="solr.StandardTokenizerFactory"/>
+ <filter class="solr.StopFilterFactory" ignoreCase="true"
words="stopwords.txt"/>
+ <filter class="solr.LowerCaseFilterFactory"/>
+ </analyzer>
+ <analyzer type="query">
+ <tokenizer class="solr.StandardTokenizerFactory"/>
+ <filter class="solr.StopFilterFactory" ignoreCase="true"
words="stopwords.txt"/>
+ <filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt"
ignoreCase="true" expand="true"/>
+ <filter class="solr.LowerCaseFilterFactory"/>
+ </analyzer>
+ </fieldType>
+ <fieldType name="date" class="solr.TrieDateField" precisionStep="0"
positionIncrementGap="0"/>
+ <fieldType name="long" class="solr.TrieLongField" precisionStep="0"
positionIncrementGap="0"/>
+ <fieldType name="lowercase" class="solr.TextField"
positionIncrementGap="100">
+ <analyzer>
+ <tokenizer class="solr.KeywordTokenizerFactory"/>
+ <filter class="solr.LowerCaseFilterFactory"/>
+ </analyzer>
+ </fieldType>
+ </types>
+</schema>
Modified:
bloodhound/branches/bep_0014_solr/bloodhound_solr/bhsolr/tests/schema.py
URL:
http://svn.apache.org/viewvc/bloodhound/branches/bep_0014_solr/bloodhound_solr/bhsolr/tests/schema.py?rev=1618520&r1=1618519&r2=1618520&view=diff
==============================================================================
--- bloodhound/branches/bep_0014_solr/bloodhound_solr/bhsolr/tests/schema.py
(original)
+++ bloodhound/branches/bep_0014_solr/bloodhound_solr/bhsolr/tests/schema.py
Sun Aug 17 20:22:10 2014
@@ -0,0 +1,62 @@
+# -*- coding: UTF-8 -*-
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import unittest
+import difflib
+import pkg_resources
+
+from trac.test import EnvironmentStub
+from bhsolr.schema import SolrSchema
+
+
+class SolrSchemaTestCase(unittest.TestCase):
+ def setUp(self):
+
+ self.env = EnvironmentStub(enable=('trac.*', 'bhsolr.*'))
+ resource_filename = pkg_resources.resource_filename
+ self.data_path = resource_filename(__name__,
+ "/data/schema_test_file.xml")
+ self.test_path = "/Users/antonia/Desktop"
+ self.schema = SolrSchema(self.env)
+
+ def test_generates_schema_correctly(self):
+ self.schema.generate_schema(path=self.test_path)
+
+ print self.data_path
+ with open(self.data_path, 'r') as hosts0:
+ with open(self.test_path + "/schema.xml", 'r') as hosts1:
+ diff = difflib.unified_diff(
+ hosts0.readlines(),
+ hosts1.readlines(),
+ fromfile='schema_test_file.xml',
+ tofile='schema.xml',
+ )
+
+ for line in diff:
+ self.assertEqual(line, "",
+ "The schema.xml file is not valid.")
+
+def suite():
+ suite = unittest.TestSuite()
+ suite.addTest(unittest.makeSuite(SolrSchemaTestCase))
+ return suite
+
+
+if __name__ == '__main__':
+ unittest.main(defaultTest='suite')
Modified: bloodhound/branches/bep_0014_solr/bloodhound_solr/setup.py
URL:
http://svn.apache.org/viewvc/bloodhound/branches/bep_0014_solr/bloodhound_solr/setup.py?rev=1618520&r1=1618519&r2=1618520&view=diff
==============================================================================
--- bloodhound/branches/bep_0014_solr/bloodhound_solr/setup.py (original)
+++ bloodhound/branches/bep_0014_solr/bloodhound_solr/setup.py Sun Aug 17
20:22:10 2014
@@ -21,10 +21,9 @@ from setuptools import setup, find_packa
PKG_INFO = {
- 'bhsolr': ['../README', '../TESTING_README'],
+ 'bhsolr': ['../README.wiki', '../TESTING_README'],
'bhsolr.search_resources' : [],
- 'bhsolr.tests': ['*.*'],
- 'bhsolr.tests.search_resources': ['*.*'],
+ 'bhsolr.tests': ['*.*', 'data/*.*'],
}
Modified:
bloodhound/branches/bep_0014_solr/bloodhound_theme/bhtheme/templates/bh_more_like_this.html
URL:
http://svn.apache.org/viewvc/bloodhound/branches/bep_0014_solr/bloodhound_theme/bhtheme/templates/bh_more_like_this.html?rev=1618520&r1=1618519&r2=1618520&view=diff
==============================================================================
---
bloodhound/branches/bep_0014_solr/bloodhound_theme/bhtheme/templates/bh_more_like_this.html
(original)
+++
bloodhound/branches/bep_0014_solr/bloodhound_theme/bhtheme/templates/bh_more_like_this.html
Sun Aug 17 20:22:10 2014
@@ -1,3 +1,22 @@
+<!--!
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
<!DOCTYPE html
PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd";>
