[
https://issues.apache.org/jira/browse/SOLR-42?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Grant Ingersoll updated SOLR-42:
--------------------------------
Attachment: SOLR-42.patch
Here's a new version, with some more testing and the ability to preserve
certain tags via a passed in set. This is useful for text where some tags are
meaningful or at least are useful further down the chain. I am not sure why
the html test file is not included. svn stat shows:
{quote}
A + src/test/test-files/htmlStripReaderTest.html
A src/test/org/apache/solr/analysis/HTMLStripReaderTest.java
M src/java/org/apache/solr/analysis/HTMLStripReader.java
{quote}
> Highlighting problems with HTMLStripWhitespaceTokenizerFactory
> --------------------------------------------------------------
>
> Key: SOLR-42
> URL: https://issues.apache.org/jira/browse/SOLR-42
> Project: Solr
> Issue Type: Bug
> Components: highlighter
> Reporter: Andrew May
> Assignee: Grant Ingersoll
> Priority: Minor
> Attachments: htmlStripReaderTest.html, HTMLStripReaderTest.java,
> SOLR-42.patch, SOLR-42.patch
>
>
> Indexing content that contains HTML markup, causes problems with highlighting
> if the HTMLStripWhitespaceTokenizerFactory is used (to prevent the tag names
> from being searchable).
> Example title field:
> <SUP>40</SUP>Ar/<SUP>39</SUP>Ar laserprobe dating of mylonitic fabrics in a
> polyorogenic terrane of NW Iberia
> Searching for title:fabrics with highlighting on, the highlighted version has
> the <em> tags in the wrong place - 22 characters to the left of where they
> should be (i.e. the sum of the lengths of the tags).
> Response from Yonik on the solr-user mailing-list:
> HTMLStripWhitespaceTokenizerFactory works in two phases...
> HTMLStripReader removes the HTML and passes the result to
> WhitespaceTokenizer... at that point, Tokens are generated, but the
> offsets will correspond to the text after HTML removal, not before.
> I did it this way so that HTMLStripReader could go before any
> tokenizer (like StandardTokenizer).
> Can you open a JIRA bug for this? The fix would be a special version
> of HTMLStripReader integrated with a WhitespaceTokenizer to keep
> offsets correct.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
