Re: Joining on multi valued fields
On Thu, Aug 4, 2011 at 11:21 AM, wrote: > Hi Yonik > > So I tested the join using the sample data below and the latest trunk. I > still got the same behaviour. > > HOWEVER! In this case it was nothing to do with the patch or solr version. It > was the tokeniser splitting G1 into G and 1. Ah, glad you figured it out! > So thank you for a nice patch and your suggestions. > > I do have a couple of questions for you: At what level does the join happen > and what do you expect the performance penalty to be. We might use this > extensively if the performance penalty isn't great. With the current implementation, the performance is proportional to the number of unique terms in the fields being joined. -Yonik http://www.lucidimagination.com
RE: Joining on multi valued fields
Hi Yonik
So I tested the join using the sample data below and the latest trunk. I still
got the same behaviour.
HOWEVER! In this case it was nothing to do with the patch or solr version. It
was the tokeniser splitting G1 into G and 1.
So thank you for a nice patch and your suggestions.
I do have a couple of questions for you: At what level does the join happen and
what do you expect the performance penalty to be. We might use this extensively
if the performance penalty isn't great.
Thanks again,
Matt
-Original Message-
From: Fowler, Matthew (Markets Eikon)
Sent: 03 August 2011 15:04
To: yo...@lucidimagination.com
Cc: solr-user@lucene.apache.org
Subject: RE: Joining on multi valued fields
No I haven't. I will get the latest out of the trunk and report back.
Cheers again,
Matt
-Original Message-
From: ysee...@gmail.com [mailto:ysee...@gmail.com] On Behalf Of Yonik Seeley
Sent: 03 August 2011 14:51
To: Fowler, Matthew (Markets Eikon)
Cc: solr-user@lucene.apache.org
Subject: Re: Joining on multi valued fields
Hmmm, if these are real responses from a solr server "at rest" (i.e.
documents not being changed between queries) then what you show
definitely looks like a bug.
That's interesting, since TestJoin implements a random test that
should cover cases like this pretty well.
I assume you are using a version of trunk (4.0-dev) and not just the
actual attached to the JIRA issue (which IIRC had at least one bug...
SOLR-2521).
Have you tried a more recent version of trunk?
-Yonik
http://www.lucidimagination.com
On Wed, Aug 3, 2011 at 7:00 AM, wrote:
> Hi Yonik
>
> Sorry for my late reply. I have been trying to get to the bottom of this
> but I'm getting inconsistent behaviour. Here's an example:
>
> Query = "pi:rcs100" - Here going to use "pid_rcs" as join
> value
>
>
>
> rcs100
> rcs
> G1
> Emerging Market Countries
> All business events relating to companies
> and other issuers of securities.
>
>
>
>
> Query = "code:G1" - See how many docs have "G1" in their
> code field. Notice that "code" is multi valued
>
> -
> -
> cat
> 2011-04-22T05:48:57Z
> nCIF3wGpXk+1029782
> -
> G1
> G7U
> GK
> ME7
> ME8
> MN
> MR
>
>
> -
> cat
> 2011-04-22T05:48:57Z
> nCIF7YcLP+1029782
> -
> G1
> G7U
> GK
> ME7
> ME8
> MN
> MR
>
>
>
>
>
> Now for the join: http://10.15.39.137:8983/solr/file/select?q={!join
> from=pid_rcs to=code}pi:rcs100
>
> -
> -
> cat
> 2011-04-22T05:48:57Z
> nCIF3wGpXk+1029782
> -
> G1
> G7U
> GK
> ME7
> ME8
> MN
> MR
>
>
> -
> cat
> 2011-04-22T05:48:57Z
> nCIF7YcLP+1029782
> -
> G1
> G7U
> GK
> ME7
> ME8
> MN
> MR
>
>
> -
> cat
> 2011-04-22T05:48:58Z
> nCN1763203+1029782
> -
> A2
> A5
> A9
> AN
> B125
> B126
> B130
> BL63
> G41
> GK
> MZ
>
>
>
>
>
> So as you can see I get back 3 results when only 2 match the criteria.
> i.e. docs where G1 is present in multi valued "code" field. Why should
> the last document be included in the result of the join?
>
> Thank you,
>
> Matt
>
>
> -Original Message-
> From: ysee...@gmail.com [mailto:ysee...@gmail.com] On Behalf Of Yonik
> Seeley
> Sent: 01 August 2011 18:28
> To: solr-user@lucene.apache.org
> Subject: Re: Joining on multi valued fields
>
> On Mon, Aug 1, 2011 at 12:58 PM,
> wrote:
>> I have been using the JOIN patch
>> https://issues.apache.org/jira/browse/SOLR-2272 with great success.
>>
>> However I have hit a case where it doesn't seem to be working. It
>> doesn't seem to work when joining to a multi-valued field.
>
> That should work (and the unit tests do test with multi-valued fields).
> Can you come up with a simple example where you are not getting the
> expected results?
>
> -Yonik
> http://www.lucidimagination.com
>
> This email was sent to you by Thomson Reuters, the global news and
> information company. Any views expressed in this message are those of the
> individual sender, except where the sender specifically states them to be the
> views of Thomson Reuters.
>
This email was sent to you by Thomson Reuters, the global news and information
company. Any views expressed in this message are those of the individual
sender, except where the sender specifically states them to be the views of
Thomson Reuters.
This email was sent to you by Thomson Reuters, the global news and information
company. Any views expressed in this message are those of the individual
sender, except where the sender specifically states them to be the views of
Thomson Reuters.
RE: Joining on multi valued fields
No I haven't. I will get the latest out of the trunk and report back.
Cheers again,
Matt
-Original Message-
From: ysee...@gmail.com [mailto:ysee...@gmail.com] On Behalf Of Yonik Seeley
Sent: 03 August 2011 14:51
To: Fowler, Matthew (Markets Eikon)
Cc: solr-user@lucene.apache.org
Subject: Re: Joining on multi valued fields
Hmmm, if these are real responses from a solr server "at rest" (i.e.
documents not being changed between queries) then what you show
definitely looks like a bug.
That's interesting, since TestJoin implements a random test that
should cover cases like this pretty well.
I assume you are using a version of trunk (4.0-dev) and not just the
actual attached to the JIRA issue (which IIRC had at least one bug...
SOLR-2521).
Have you tried a more recent version of trunk?
-Yonik
http://www.lucidimagination.com
On Wed, Aug 3, 2011 at 7:00 AM, wrote:
> Hi Yonik
>
> Sorry for my late reply. I have been trying to get to the bottom of this
> but I'm getting inconsistent behaviour. Here's an example:
>
> Query = "pi:rcs100" - Here going to use "pid_rcs" as join
> value
>
>
>
> rcs100
> rcs
> G1
> Emerging Market Countries
> All business events relating to companies
> and other issuers of securities.
>
>
>
>
> Query = "code:G1" - See how many docs have "G1" in their
> code field. Notice that "code" is multi valued
>
> -
> -
> cat
> 2011-04-22T05:48:57Z
> nCIF3wGpXk+1029782
> -
> G1
> G7U
> GK
> ME7
> ME8
> MN
> MR
>
>
> -
> cat
> 2011-04-22T05:48:57Z
> nCIF7YcLP+1029782
> -
> G1
> G7U
> GK
> ME7
> ME8
> MN
> MR
>
>
>
>
>
> Now for the join: http://10.15.39.137:8983/solr/file/select?q={!join
> from=pid_rcs to=code}pi:rcs100
>
> -
> -
> cat
> 2011-04-22T05:48:57Z
> nCIF3wGpXk+1029782
> -
> G1
> G7U
> GK
> ME7
> ME8
> MN
> MR
>
>
> -
> cat
> 2011-04-22T05:48:57Z
> nCIF7YcLP+1029782
> -
> G1
> G7U
> GK
> ME7
> ME8
> MN
> MR
>
>
> -
> cat
> 2011-04-22T05:48:58Z
> nCN1763203+1029782
> -
> A2
> A5
> A9
> AN
> B125
> B126
> B130
> BL63
> G41
> GK
> MZ
>
>
>
>
>
> So as you can see I get back 3 results when only 2 match the criteria.
> i.e. docs where G1 is present in multi valued "code" field. Why should
> the last document be included in the result of the join?
>
> Thank you,
>
> Matt
>
>
> -Original Message-
> From: ysee...@gmail.com [mailto:ysee...@gmail.com] On Behalf Of Yonik
> Seeley
> Sent: 01 August 2011 18:28
> To: solr-user@lucene.apache.org
> Subject: Re: Joining on multi valued fields
>
> On Mon, Aug 1, 2011 at 12:58 PM,
> wrote:
>> I have been using the JOIN patch
>> https://issues.apache.org/jira/browse/SOLR-2272 with great success.
>>
>> However I have hit a case where it doesn't seem to be working. It
>> doesn't seem to work when joining to a multi-valued field.
>
> That should work (and the unit tests do test with multi-valued fields).
> Can you come up with a simple example where you are not getting the
> expected results?
>
> -Yonik
> http://www.lucidimagination.com
>
> This email was sent to you by Thomson Reuters, the global news and
> information company. Any views expressed in this message are those of the
> individual sender, except where the sender specifically states them to be the
> views of Thomson Reuters.
>
This email was sent to you by Thomson Reuters, the global news and information
company. Any views expressed in this message are those of the individual
sender, except where the sender specifically states them to be the views of
Thomson Reuters.
Re: Joining on multi valued fields
Hmmm, if these are real responses from a solr server "at rest" (i.e.
documents not being changed between queries) then what you show
definitely looks like a bug.
That's interesting, since TestJoin implements a random test that
should cover cases like this pretty well.
I assume you are using a version of trunk (4.0-dev) and not just the
actual attached to the JIRA issue (which IIRC had at least one bug...
SOLR-2521).
Have you tried a more recent version of trunk?
-Yonik
http://www.lucidimagination.com
On Wed, Aug 3, 2011 at 7:00 AM, wrote:
> Hi Yonik
>
> Sorry for my late reply. I have been trying to get to the bottom of this
> but I'm getting inconsistent behaviour. Here's an example:
>
> Query = "pi:rcs100" - Here going to use "pid_rcs" as join
> value
>
>
>
> rcs100
> rcs
> G1
> Emerging Market Countries
> All business events relating to companies
> and other issuers of securities.
>
>
>
>
> Query = "code:G1" - See how many docs have "G1" in their
> code field. Notice that "code" is multi valued
>
> -
> -
> cat
> 2011-04-22T05:48:57Z
> nCIF3wGpXk+1029782
> -
> G1
> G7U
> GK
> ME7
> ME8
> MN
> MR
>
>
> -
> cat
> 2011-04-22T05:48:57Z
> nCIF7YcLP+1029782
> -
> G1
> G7U
> GK
> ME7
> ME8
> MN
> MR
>
>
>
>
>
> Now for the join: http://10.15.39.137:8983/solr/file/select?q={!join
> from=pid_rcs to=code}pi:rcs100
>
> -
> -
> cat
> 2011-04-22T05:48:57Z
> nCIF3wGpXk+1029782
> -
> G1
> G7U
> GK
> ME7
> ME8
> MN
> MR
>
>
> -
> cat
> 2011-04-22T05:48:57Z
> nCIF7YcLP+1029782
> -
> G1
> G7U
> GK
> ME7
> ME8
> MN
> MR
>
>
> -
> cat
> 2011-04-22T05:48:58Z
> nCN1763203+1029782
> -
> A2
> A5
> A9
> AN
> B125
> B126
> B130
> BL63
> G41
> GK
> MZ
>
>
>
>
>
> So as you can see I get back 3 results when only 2 match the criteria.
> i.e. docs where G1 is present in multi valued "code" field. Why should
> the last document be included in the result of the join?
>
> Thank you,
>
> Matt
>
>
> -Original Message-
> From: ysee...@gmail.com [mailto:ysee...@gmail.com] On Behalf Of Yonik
> Seeley
> Sent: 01 August 2011 18:28
> To: solr-user@lucene.apache.org
> Subject: Re: Joining on multi valued fields
>
> On Mon, Aug 1, 2011 at 12:58 PM,
> wrote:
>> I have been using the JOIN patch
>> https://issues.apache.org/jira/browse/SOLR-2272 with great success.
>>
>> However I have hit a case where it doesn't seem to be working. It
>> doesn't seem to work when joining to a multi-valued field.
>
> That should work (and the unit tests do test with multi-valued fields).
> Can you come up with a simple example where you are not getting the
> expected results?
>
> -Yonik
> http://www.lucidimagination.com
>
> This email was sent to you by Thomson Reuters, the global news and
> information company. Any views expressed in this message are those of the
> individual sender, except where the sender specifically states them to be the
> views of Thomson Reuters.
>
RE: Joining on multi valued fields
Hi Yonik
Sorry for my late reply. I have been trying to get to the bottom of this
but I'm getting inconsistent behaviour. Here's an example:
Query = "pi:rcs100" - Here going to use "pid_rcs" as join
value
rcs100
rcs
G1
Emerging Market Countries
All business events relating to companies
and other issuers of securities.
Query = "code:G1" - See how many docs have "G1" in their
code field. Notice that "code" is multi valued
-
-
cat
2011-04-22T05:48:57Z
nCIF3wGpXk+1029782
-
G1
G7U
GK
ME7
ME8
MN
MR
-
cat
2011-04-22T05:48:57Z
nCIF7YcLP+1029782
-
G1
G7U
GK
ME7
ME8
MN
MR
Now for the join: http://10.15.39.137:8983/solr/file/select?q={!join
from=pid_rcs to=code}pi:rcs100
-
-
cat
2011-04-22T05:48:57Z
nCIF3wGpXk+1029782
-
G1
G7U
GK
ME7
ME8
MN
MR
-
cat
2011-04-22T05:48:57Z
nCIF7YcLP+1029782
-
G1
G7U
GK
ME7
ME8
MN
MR
-
cat
2011-04-22T05:48:58Z
nCN1763203+1029782
-
A2
A5
A9
AN
B125
B126
B130
BL63
G41
GK
MZ
So as you can see I get back 3 results when only 2 match the criteria.
i.e. docs where G1 is present in multi valued "code" field. Why should
the last document be included in the result of the join?
Thank you,
Matt
-Original Message-
From: ysee...@gmail.com [mailto:ysee...@gmail.com] On Behalf Of Yonik
Seeley
Sent: 01 August 2011 18:28
To: solr-user@lucene.apache.org
Subject: Re: Joining on multi valued fields
On Mon, Aug 1, 2011 at 12:58 PM,
wrote:
> I have been using the JOIN patch
> https://issues.apache.org/jira/browse/SOLR-2272 with great success.
>
> However I have hit a case where it doesn't seem to be working. It
> doesn't seem to work when joining to a multi-valued field.
That should work (and the unit tests do test with multi-valued fields).
Can you come up with a simple example where you are not getting the
expected results?
-Yonik
http://www.lucidimagination.com
This email was sent to you by Thomson Reuters, the global news and information
company. Any views expressed in this message are those of the individual
sender, except where the sender specifically states them to be the views of
Thomson Reuters.
Re: Joining on multi valued fields
On Mon, Aug 1, 2011 at 12:58 PM, wrote: > I have been using the JOIN patch > https://issues.apache.org/jira/browse/SOLR-2272 with great success. > > However I have hit a case where it doesn't seem to be working. It > doesn't seem to work when joining to a multi-valued field. That should work (and the unit tests do test with multi-valued fields). Can you come up with a simple example where you are not getting the expected results? -Yonik http://www.lucidimagination.com
Joining on multi valued fields
Hi List I have been using the JOIN patch https://issues.apache.org/jira/browse/SOLR-2272 with great success. However I have hit a case where it doesn't seem to be working. It doesn't seem to work when joining to a multi-valued field. Has anyone any experience using the patch to join on multi-valued fields? Thanks, Matt This email was sent to you by Thomson Reuters, the global news and information company. Any views expressed in this message are those of the individual sender, except where the sender specifically states them to be the views of Thomson Reuters.
