Hey Grant,
Fair point on the next(). In this case I'm iterating through the terms returned
from a PrefixTermEnum so I know they're in the index.
The analyser I'm using looks like this:
public class TypeSavingAnalyzer extends StandardAnalyzer {
public TypeSavingAnalyzer(Version version) {
super(version);
}
@Override
public TokenStream tokenStream(String fieldName, Reader reader) {
TokenStream result = super.tokenStream(fieldName, reader);
return new TypeAsPayloadTokenFilter(result);
}
}
I'm creating the field with Field.TermVector.WITH_POSITIONS and see the little
"p" indicator in luke however see no indication of a payload (though am not
100% sure where to look) being present and I still get false from
isPayloadAvailable.
Basically what I'm trying to do is store in the payload some meta data so I
know what type the term is as at the moment when I get a list of terms back
from the reader some are numeric. It seems NumericField doesn't allow payloads
so it seems like I may have to find an alternative method. Any suggestions
would be greatly appreciated!
Regards,
Derek
-----Original Message-----
From: Grant Ingersoll [mailto:[email protected]]
Sent: 14 October 2010 15:22
To: [email protected]
Subject: Re: determining the type of a term - retrieving a payload
On Oct 13, 2010, at 11:37 AM, Sykes, Derek wrote:
> Hi there,
>
> I'm currently trying to work out how I can determine the type
> (string/number/date/etc)of a term. I've not seen any off the shelf way to do
> it so am trying to store a payload against each term that records the type.
>
> I'm having a little trouble retrieving a payload I'd stored onto the term.
> At the moment I'm using the TypeAsPayloadTokenFilter, however I'll change
> this soon as it's not really what I'm after.
>
> I'm then getting a TermEnum from either reader.terms() or using a
> PrefixTermEnum.
>
> For each term I want the payload from I'm running this function:
>
> private String getPayload(Term term) {
> String payload = null;
> try {
> TermPositions termPositions = reader.termPositions(term);
> termPositions.next();
next() returns a boolean as to whether there is a valid entry, so you need to
check that. You may not actually have a match for that term.
> if (termPositions.isPayloadAvailable()) {
> byte[] payloadBytes = new
> byte[termPositions.getPayloadLength()];
> payloadBytes = termPositions.getPayload(payloadBytes, 0);
> payload = new String(payloadBytes);
> LOG.debug(payload);
> }
> }
> catch( IOException e) {
> ...
What does your Analysis process look like? Many of Lucene's analysis pieces
don't bother setting type. Have you looked at the index with Luke? That
should show you the payloads. Also, have a look at the SpanTermQuery. You can
use the Spans object to step directly through the position matches and get the
payloads, if they are available.
--------------------------
Grant Ingersoll
http://www.lucidimagination.com
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
--------------------------------------------------------------------------
NOTICE: Morgan Stanley is not acting as a municipal advisor and the opinions or
views contained herein are not intended to be, and do not constitute, advice
within the meaning of Section 975 of the Dodd-Frank Wall Street Reform and
Consumer Protection Act. If you have received this communication in error,
please destroy all electronic and paper copies and notify the sender
immediately. Mistransmission is not intended to waive confidentiality or
privilege. Morgan Stanley reserves the right, to the extent permitted under
applicable law, to monitor electronic communications. This message is subject
to terms available at the following link:
http://www.morganstanley.com/disclaimers. If you cannot access these links,
please notify us by reply message and we will send the contents to you. By
messaging with Morgan Stanley you consent to the foregoing.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
