I designed the API with this limitation intentionally to prevent users from thinking that they can call TermPositions.getPayload() more than once with no costs.
If we allow to call it more often than once then we have to seek back in the posting stream. Even if this is just a seek in the underlying IndexInput buffer, we still have to perform an arraycopy from that buffer to the array that getPayload() returns. If the beginning of the payload is already outside the current buffer, then a seek on the HD will happen in addition, which is even more expensive. So I'd like to keep the API as is. An application should always be able to buffer a payload byte[] array if it needs to access it more than once. For convenience, user could also create a very simple Termpositions decorator that caches the most recently loaded payload and allows calling getPayload() more than once. However, I hesitate to add such a payload caching to SegmentTermPositions, because the size of the payloads is application-specific and so should the policy be that grows/shrinks a caching byte[] array. -Michael Grant Ingersoll wrote: > In working on LUCENE-1001, things are getting a bit complicated with > loading payloads in overlapping spans (which causes the dreaded Can't > load payload more than once error). > > This got me thinking about why we need the rule that payloads can only > be loaded once. I forget the reasoning behind this. Can we just store > where the current position before we load the payload and then seek back > to that point if we need to load the payload again? I suppose in the > case of really large payloads the seek on the IndexInput could be > expensive, but in reality, most payloads aren't likely to be more than a > few bytes, right? There also seems to be some interactions with the > lazy skipping that I haven't quite pinned down yet. What else am I > forgetting? > > The other alternative I can think of is I could cache the payloads, but > that seems unwieldy too. > > -Grant > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] > > --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
