Michael McCandless wrote:
> OK thanks for the responses. This is indeed tricky stuff!
>
> On Sat, Sep 12, 2009 at 12:28 AM, Mark Miller <markrmil...@gmail.com> wrote:
>
>
>> They start at the left and march right - each Span always starting
>> after the last started,
>>
>
> That's not quite always true -- eg I got span 1-8, twice, once I added
> "b" as a clause to the SNQ.
>
Mmm - right - depends on how you look at it I think - it is less simple
with terms at multiple positions, in that now each Span doesn't start
in the *position* after the last - but if you line up the terms like you
did, its still the same - the first 1 - 8 starts at the first term at
pos 1, and
the next 1 to 8 starts at the seconds term at pos 1. One starts after
the other (though if you think Lucene positions, I realize they virtually
start at the same spot).
>
>> You might want exhaustive for highlighting as well - but its
>> different algorithms ...
>>
>
> Yeah, how we would represent spans for highlighting is tricky... we
> had discussed this ("how to represent spans for aggregate queries")
> recently, I think under LUCENE-1522.
>
> I think we'd have to return a tree structure, that mirrors the query's
> tree structure, to hold the spans, rather than try to enumerate
> ("denormalize") all possible expansions. Each leaf node would hold
> actual data (position, term, payload, etc.), and then the tree nodes
> would express how they are and/ord/near'd together. My app could then
> walk the tree to compute any combination I wanted.
>
>
>> In the end, I accepted my definition of works as - when I ask for
>> the payloads back, will I end up with a bag of all the payloads that
>> the Spans touched. I think you do.
>>
>
> Yeah I think you do, except each payload is only returned once. So
> it's only the first span that hits a payload that will return it.
>
> So it sounds like SNQ just isn't guaranteed to be exhaustive in how it
> enumerates the spans, eg I'll never see that 2nd occurrence of "k",
> nor its associated payload.
>
Not only not guaranteed, but its just not going to happen - its not
how spans match. If I say find n within 300 of m with the following:
n m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m
m m m m m m m m m m m m
Only the first m will match. It will start at the left, find the n, then
say great, an m within 300, this doc matches, we are done. There is
not another n to start on or finish on to the right. It doesn't then
touch the next 300 m's - just they way Doug implemented them from what I
can tell. Its only exhaustive from the
left - find m within 300 of n, order matters (m first)
m m m m m m m m m m m m m m m m m m n
This will be a bunch of spans - start at the left - the first m to n
matches, then the second m - n matches, then the third m to n matches,
and so on as we move right.
> For now I'll just match this behavior ("can only load payload once")
> in all codecs in LUCENE-1458... the test passes again once I do that.
>
>
>> I meant, all those Spans came from one query - so you got your bag
>> of payloads right? If each Span was a separate entity, it would
>> obviously be way wrong - but from a single SpanQuery, at least you
>> got all the payloads in some form :)
>>
>
> Right, this is all one query... but the payload for the 2nd
> occurrence of "k" was never included in any span so I didn't get "all"
> payloads.
>
You got all the payloads the query matched - I think you need a
different query (or
we change the Spans algorithm completely)
> Maybe if/once we incorporate spans into Lucene's normal queries
> (optionally, so there's no performance hit if you don't ask for them)
> we can re-visit these issues.
>
> Mike
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: java-dev-h...@lucene.apache.org
>
>
--
- Mark
http://www.lucidimagination.com
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-dev-h...@lucene.apache.org
