Highlighting and InvalidTokenOffsetsException in Lucene 4.0

[Scott Smith]

I'm migrating code from Lucene 3.5 to 4.0.  I have the following code which is 
supposed to highlight text.  I get the exception InvalidTokenOffsetsException.  
I have no idea what that means.  I am using a custom analyzer which seems to 
work for searching/indexing, so I assume it will work here (even though it took 
a couple of "minor" changes to get it to compile in 4.0  This code used to work 
in 3.5.

Anyone have any ideas?

Scott

Code fragment:

        try
        {
            ctf = new CachingTokenFilter(myCustomAnalyzer
                    .tokenStream(MyFieldName, new StringReader(myText)));
        }
        catch (IOException e1)
        {
            s_oLog.error("Search:markCommon: Exception creating 
CachingTokenFilter: " +
                    e1.getMessage());
            return null;

        }
        String markedString;
        SimpleHTMLFormatter formatter;
        try
        {
            formatter = new SimpleHTMLFormatter(_zBeginHighlight,
                    _zEndHighlight);
            Scorer score = new QueryScorer(q);
            ht = new Highlighter(formatter, score);
            ht.setTextFragmenter(new NullFragmenter());
            markedString = ht.getBestFragment(ctf, myText);
        }
        catch (IOException e)
        {
            s_oLog.error("Search:markCommon: Unable to highlight string: "
                    + e.getMessage());
            return null;
        }
        catch(InvalidTokenOffsetsException e2)
        {
            s_oLog.error("Search:markCommon: Unable to highlight string2: "
                    + e2.getMessage());
            return null;
        }