Solr Composite Unique key from existing fields in schema
Hi, I'm using DIH to index my entities. I'm facing an issue while delta-import. I've declared multiple entities in one data-config.xml. The entities will have different primary key. Now if I want to delta-import how should I mention the UniqueKey in schema.xml. My data-config structure is like this document entity/entity entity/entity entity/entity /document Thanks Rajesh [Aspire Systems] This e-mail message and any attachments are for the sole use of the intended recipient(s) and may contain proprietary, confidential, trade secret or privileged information. Any unauthorized review, use, disclosure or distribution is prohibited and may be a violation of law. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message.
Re: Solr Composite Unique key from existing fields in schema
Hi, Once I used template transformer to generate unique id across entities. http://wiki.apache.org/solr/DataImportHandler#TemplateTransformer On Wednesday, December 10, 2014 8:51 AM, Rajesh Panneerselvam rajesh.panneersel...@aspiresys.com wrote: Hi, I'm using DIH to index my entities. I'm facing an issue while delta-import. I've declared multiple entities in one data-config.xml. The entities will have different primary key. Now if I want to delta-import how should I mention the UniqueKey in schema.xml. My data-config structure is like this document entity/entity entity/entity entity/entity /document Thanks Rajesh [Aspire Systems] This e-mail message and any attachments are for the sole use of the intended recipient(s) and may contain proprietary, confidential, trade secret or privileged information. Any unauthorized review, use, disclosure or distribution is prohibited and may be a violation of law. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message.
Re: Solr Composite Unique key from existing fields in schema
You could use a combination of all your composite key columns and put them in a field in solr which can then be used as the unique key. As in if you have two columns c1 and c2, you could have a field in solr which have the value as c1_c2 or something on those lines. -- View this message in context: http://lucene.472066.n3.nabble.com/Solr-Composite-Unique-key-from-existing-fields-in-schema-tp4116036p4116195.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr Composite Unique key from existing fields in schema
Also take a look at http://wiki.apache.org/solr/UniqueKey#Use_cases_which_require_a_unique_key_generated_from_data_in_the_document http://wiki.apache.org/solr/UniqueKey#Use_cases_which_require_a_unique_key_generated_from_data_in_the_document -- View this message in context: http://lucene.472066.n3.nabble.com/Solr-Composite-Unique-key-from-existing-fields-in-schema-tp4116036p4116198.html Sent from the Solr - User mailing list archive at Nabble.com.
Solr Composite Unique key from existing fields in schema
Hi, I am developing a search application using SOLR. I don't have primary key in any table. Composite key is being used in my application. How do i implement composite key as unique key in this case. please help. i am struggling. -- Thanks Regards Anurag Verma Arise! Awake! And stop not till the goal is reached!
Solr Composite Unique key from existing fields in schema
Hi All, Historically we have used a single field in our schema as a uniqueKey. field name=docidtype=string indexed=true stored=true multiValued=false required=true/ field name=userid type=string indexed=true stored=true multiValued=false required=true/ uniqueKeydocid/uniqueKey Wanted to change this to a composite key something like uniqueKeyuserid-docid/uniqueKey. I know I can auto generate compositekey at document insert time, using custom code to generate a new field, but wanted to know if there was an inbuilt SOLR mechanism of doing this. That would prevent us from creating and storing an extra field. Thanks, Rishi.
Re: Solr Composite Unique key from existing fields in schema
The cleanest is to do this from the outside. Alternatively, it will perhaps work to populate your uniqueKey in a custom UpdateProcessor. You can try. -- Jan Høydahl, search solution architect Cominvent AS - www.cominvent.com 28. mai 2013 kl. 17:12 skrev Rishi Easwaran rishi.easwa...@aol.com: Hi All, Historically we have used a single field in our schema as a uniqueKey. field name=docidtype=string indexed=true stored=true multiValued=false required=true/ field name=userid type=string indexed=true stored=true multiValued=false required=true/ uniqueKeydocid/uniqueKey Wanted to change this to a composite key something like uniqueKeyuserid-docid/uniqueKey. I know I can auto generate compositekey at document insert time, using custom code to generate a new field, but wanted to know if there was an inbuilt SOLR mechanism of doing this. That would prevent us from creating and storing an extra field. Thanks, Rishi.
Re: Solr Composite Unique key from existing fields in schema
You can do this by combining the builtin update processors.
Add this to your solrconfig:
updateRequestProcessorChain name=composite-id
processor class=solr.CloneFieldUpdateProcessorFactory
str name=sourcedocid_s/str
str name=sourceuserid_s/str
str name=destid/str
/processor
processor class=solr.ConcatFieldUpdateProcessorFactory
str name=fieldNameid/str
str name=delimiter--/str
/processor
processor class=solr.LogUpdateProcessorFactory /
processor class=solr.RunUpdateProcessorFactory /
/updateRequestProcessorChain
Add documents such as:
curl
http://localhost:8983/solr/update?commit=trueupdate.chain=composite-id; \
-H 'Content-type:application/json' -d '
[{title: Hello World,
docid_s: doc-1,
userid_s: user-1,
comments_ss: [Easy, Fast]}]'
And get results like:
title:[Hello World],
docid_s:doc-1,
userid_s:user-1,
comments_ss:[Easy,
Fast],
id:doc-1--user-1,
Add as many fields in whatever order you want using source in the clone
update processor, and pick your composite key field name as well. And set
the delimiter string as well in the concat update processor.
I managed to reverse the field order from what you requested (userid,
docid).
I used the standard Solr example schema, so I used dynamic fields for the
two ids, but use your own field names.
-- Jack Krupansky
-Original Message-
From: Rishi Easwaran
Sent: Tuesday, May 28, 2013 11:12 AM
To: solr-user@lucene.apache.org
Subject: Solr Composite Unique key from existing fields in schema
Hi All,
Historically we have used a single field in our schema as a uniqueKey.
field name=docidtype=string indexed=true stored=true
multiValued=false required=true/
field name=userid type=string indexed=true stored=true
multiValued=false required=true/
uniqueKeydocid/uniqueKey
Wanted to change this to a composite key something like
uniqueKeyuserid-docid/uniqueKey.
I know I can auto generate compositekey at document insert time, using
custom code to generate a new field, but wanted to know if there was an
inbuilt SOLR mechanism of doing this. That would prevent us from creating
and storing an extra field.
Thanks,
Rishi.
Re: Solr Composite Unique key from existing fields in schema
Thanks Jack, looks like that will do the trick from me. I will try it out.
-Original Message-
From: Jack Krupansky j...@basetechnology.com
To: solr-user solr-user@lucene.apache.org
Sent: Tue, May 28, 2013 12:07 pm
Subject: Re: Solr Composite Unique key from existing fields in schema
You can do this by combining the builtin update processors.
Add this to your solrconfig:
updateRequestProcessorChain name=composite-id
processor class=solr.CloneFieldUpdateProcessorFactory
str name=sourcedocid_s/str
str name=sourceuserid_s/str
str name=destid/str
/processor
processor class=solr.ConcatFieldUpdateProcessorFactory
str name=fieldNameid/str
str name=delimiter--/str
/processor
processor class=solr.LogUpdateProcessorFactory /
processor class=solr.RunUpdateProcessorFactory /
/updateRequestProcessorChain
Add documents such as:
curl
http://localhost:8983/solr/update?commit=trueupdate.chain=composite-id; \
-H 'Content-type:application/json' -d '
[{title: Hello World,
docid_s: doc-1,
userid_s: user-1,
comments_ss: [Easy, Fast]}]'
And get results like:
title:[Hello World],
docid_s:doc-1,
userid_s:user-1,
comments_ss:[Easy,
Fast],
id:doc-1--user-1,
Add as many fields in whatever order you want using source in the clone
update processor, and pick your composite key field name as well. And set
the delimiter string as well in the concat update processor.
I managed to reverse the field order from what you requested (userid,
docid).
I used the standard Solr example schema, so I used dynamic fields for the
two ids, but use your own field names.
-- Jack Krupansky
-Original Message-
From: Rishi Easwaran
Sent: Tuesday, May 28, 2013 11:12 AM
To: solr-user@lucene.apache.org
Subject: Solr Composite Unique key from existing fields in schema
Hi All,
Historically we have used a single field in our schema as a uniqueKey.
field name=docidtype=string indexed=true stored=true
multiValued=false required=true/
field name=userid type=string indexed=true stored=true
multiValued=false required=true/
uniqueKeydocid/uniqueKey
Wanted to change this to a composite key something like
uniqueKeyuserid-docid/uniqueKey.
I know I can auto generate compositekey at document insert time, using
custom code to generate a new field, but wanted to know if there was an
inbuilt SOLR mechanism of doing this. That would prevent us from creating
and storing an extra field.
Thanks,
Rishi.
Re: Solr Composite Unique key from existing fields in schema
Jack,
No sure if this is the correct behaviour.
I set up updateRequestorPorcess chain as mentioned below, but looks like the
compositeId that is generated is based on input order.
For example:
If my input comes in as
field name=docid1/field
field name=userid12345/field
I get the following compositeId1-12345.
If I reverse the input
field name=userid12345/field
field name=docid1/field
I get the following compositeId 12345-1 .
In this case the compositeId is not unique and I am getting duplicates.
Thanks,
Rishi.
-Original Message-
From: Jack Krupansky j...@basetechnology.com
To: solr-user solr-user@lucene.apache.org
Sent: Tue, May 28, 2013 12:07 pm
Subject: Re: Solr Composite Unique key from existing fields in schema
You can do this by combining the builtin update processors.
Add this to your solrconfig:
updateRequestProcessorChain name=composite-id
processor class=solr.CloneFieldUpdateProcessorFactory
str name=sourcedocid_s/str
str name=sourceuserid_s/str
str name=destid/str
/processor
processor class=solr.ConcatFieldUpdateProcessorFactory
str name=fieldNameid/str
str name=delimiter--/str
/processor
processor class=solr.LogUpdateProcessorFactory /
processor class=solr.RunUpdateProcessorFactory /
/updateRequestProcessorChain
Add documents such as:
curl
http://localhost:8983/solr/update?commit=trueupdate.chain=composite-id; \
-H 'Content-type:application/json' -d '
[{title: Hello World,
docid_s: doc-1,
userid_s: user-1,
comments_ss: [Easy, Fast]}]'
And get results like:
title:[Hello World],
docid_s:doc-1,
userid_s:user-1,
comments_ss:[Easy,
Fast],
id:doc-1--user-1,
Add as many fields in whatever order you want using source in the clone
update processor, and pick your composite key field name as well. And set
the delimiter string as well in the concat update processor.
I managed to reverse the field order from what you requested (userid,
docid).
I used the standard Solr example schema, so I used dynamic fields for the
two ids, but use your own field names.
-- Jack Krupansky
-Original Message-
From: Rishi Easwaran
Sent: Tuesday, May 28, 2013 11:12 AM
To: solr-user@lucene.apache.org
Subject: Solr Composite Unique key from existing fields in schema
Hi All,
Historically we have used a single field in our schema as a uniqueKey.
field name=docidtype=string indexed=true stored=true
multiValued=false required=true/
field name=userid type=string indexed=true stored=true
multiValued=false required=true/
uniqueKeydocid/uniqueKey
Wanted to change this to a composite key something like
uniqueKeyuserid-docid/uniqueKey.
I know I can auto generate compositekey at document insert time, using
custom code to generate a new field, but wanted to know if there was an
inbuilt SOLR mechanism of doing this. That would prevent us from creating
and storing an extra field.
Thanks,
Rishi.
Re: Solr Composite Unique key from existing fields in schema
The order in the ID should be purely dependent on the order of the field
names in the processor configuration:
str name=sourcedocid_s/str
str name=sourceuserid_s/str
-- Jack Krupansky
-Original Message-
From: Rishi Easwaran
Sent: Tuesday, May 28, 2013 2:54 PM
To: solr-user@lucene.apache.org
Subject: Re: Solr Composite Unique key from existing fields in schema
Jack,
No sure if this is the correct behaviour.
I set up updateRequestorPorcess chain as mentioned below, but looks like the
compositeId that is generated is based on input order.
For example:
If my input comes in as
field name=docid1/field
field name=userid12345/field
I get the following compositeId1-12345.
If I reverse the input
field name=userid12345/field
field name=docid1/field
I get the following compositeId 12345-1 .
In this case the compositeId is not unique and I am getting duplicates.
Thanks,
Rishi.
-Original Message-
From: Jack Krupansky j...@basetechnology.com
To: solr-user solr-user@lucene.apache.org
Sent: Tue, May 28, 2013 12:07 pm
Subject: Re: Solr Composite Unique key from existing fields in schema
You can do this by combining the builtin update processors.
Add this to your solrconfig:
updateRequestProcessorChain name=composite-id
processor class=solr.CloneFieldUpdateProcessorFactory
str name=sourcedocid_s/str
str name=sourceuserid_s/str
str name=destid/str
/processor
processor class=solr.ConcatFieldUpdateProcessorFactory
str name=fieldNameid/str
str name=delimiter--/str
/processor
processor class=solr.LogUpdateProcessorFactory /
processor class=solr.RunUpdateProcessorFactory /
/updateRequestProcessorChain
Add documents such as:
curl
http://localhost:8983/solr/update?commit=trueupdate.chain=composite-id; \
-H 'Content-type:application/json' -d '
[{title: Hello World,
docid_s: doc-1,
userid_s: user-1,
comments_ss: [Easy, Fast]}]'
And get results like:
title:[Hello World],
docid_s:doc-1,
userid_s:user-1,
comments_ss:[Easy,
Fast],
id:doc-1--user-1,
Add as many fields in whatever order you want using source in the clone
update processor, and pick your composite key field name as well. And set
the delimiter string as well in the concat update processor.
I managed to reverse the field order from what you requested (userid,
docid).
I used the standard Solr example schema, so I used dynamic fields for the
two ids, but use your own field names.
-- Jack Krupansky
-Original Message-
From: Rishi Easwaran
Sent: Tuesday, May 28, 2013 11:12 AM
To: solr-user@lucene.apache.org
Subject: Solr Composite Unique key from existing fields in schema
Hi All,
Historically we have used a single field in our schema as a uniqueKey.
field name=docidtype=string indexed=true stored=true
multiValued=false required=true/
field name=userid type=string indexed=true stored=true
multiValued=false required=true/
uniqueKeydocid/uniqueKey
Wanted to change this to a composite key something like
uniqueKeyuserid-docid/uniqueKey.
I know I can auto generate compositekey at document insert time, using
custom code to generate a new field, but wanted to know if there was an
inbuilt SOLR mechanism of doing this. That would prevent us from creating
and storing an extra field.
Thanks,
Rishi.
Re: Solr Composite Unique key from existing fields in schema
I thought the same, but that doesn't seem to be the case.
-Original Message-
From: Jack Krupansky j...@basetechnology.com
To: solr-user solr-user@lucene.apache.org
Sent: Tue, May 28, 2013 3:32 pm
Subject: Re: Solr Composite Unique key from existing fields in schema
The order in the ID should be purely dependent on the order of the field
names in the processor configuration:
str name=sourcedocid_s/str
str name=sourceuserid_s/str
-- Jack Krupansky
-Original Message-
From: Rishi Easwaran
Sent: Tuesday, May 28, 2013 2:54 PM
To: solr-user@lucene.apache.org
Subject: Re: Solr Composite Unique key from existing fields in schema
Jack,
No sure if this is the correct behaviour.
I set up updateRequestorPorcess chain as mentioned below, but looks like the
compositeId that is generated is based on input order.
For example:
If my input comes in as
field name=docid1/field
field name=userid12345/field
I get the following compositeId1-12345.
If I reverse the input
field name=userid12345/field
field name=docid1/field
I get the following compositeId 12345-1 .
In this case the compositeId is not unique and I am getting duplicates.
Thanks,
Rishi.
-Original Message-
From: Jack Krupansky j...@basetechnology.com
To: solr-user solr-user@lucene.apache.org
Sent: Tue, May 28, 2013 12:07 pm
Subject: Re: Solr Composite Unique key from existing fields in schema
You can do this by combining the builtin update processors.
Add this to your solrconfig:
updateRequestProcessorChain name=composite-id
processor class=solr.CloneFieldUpdateProcessorFactory
str name=sourcedocid_s/str
str name=sourceuserid_s/str
str name=destid/str
/processor
processor class=solr.ConcatFieldUpdateProcessorFactory
str name=fieldNameid/str
str name=delimiter--/str
/processor
processor class=solr.LogUpdateProcessorFactory /
processor class=solr.RunUpdateProcessorFactory /
/updateRequestProcessorChain
Add documents such as:
curl
http://localhost:8983/solr/update?commit=trueupdate.chain=composite-id; \
-H 'Content-type:application/json' -d '
[{title: Hello World,
docid_s: doc-1,
userid_s: user-1,
comments_ss: [Easy, Fast]}]'
And get results like:
title:[Hello World],
docid_s:doc-1,
userid_s:user-1,
comments_ss:[Easy,
Fast],
id:doc-1--user-1,
Add as many fields in whatever order you want using source in the clone
update processor, and pick your composite key field name as well. And set
the delimiter string as well in the concat update processor.
I managed to reverse the field order from what you requested (userid,
docid).
I used the standard Solr example schema, so I used dynamic fields for the
two ids, but use your own field names.
-- Jack Krupansky
-Original Message-
From: Rishi Easwaran
Sent: Tuesday, May 28, 2013 11:12 AM
To: solr-user@lucene.apache.org
Subject: Solr Composite Unique key from existing fields in schema
Hi All,
Historically we have used a single field in our schema as a uniqueKey.
field name=docidtype=string indexed=true stored=true
multiValued=false required=true/
field name=userid type=string indexed=true stored=true
multiValued=false required=true/
uniqueKeydocid/uniqueKey
Wanted to change this to a composite key something like
uniqueKeyuserid-docid/uniqueKey.
I know I can auto generate compositekey at document insert time, using
custom code to generate a new field, but wanted to know if there was an
inbuilt SOLR mechanism of doing this. That would prevent us from creating
and storing an extra field.
Thanks,
Rishi.
Re: Solr Composite Unique key from existing fields in schema
The TL;DR response: Try this:
updateRequestProcessorChain name=composite-id
processor class=solr.CloneFieldUpdateProcessorFactory
str name=sourceuserid_s/str
str name=destid/str
/processor
processor class=solr.CloneFieldUpdateProcessorFactory
str name=sourcedocid_s/str
str name=destid/str
/processor
processor class=solr.ConcatFieldUpdateProcessorFactory
str name=fieldNameid/str
str name=delimiter--/str
/processor
processor class=solr.LogUpdateProcessorFactory /
processor class=solr.RunUpdateProcessorFactory /
/updateRequestProcessorChain
That will assure that the userid gets processed before the docid.
I'll have to review the contract for CloneFieldUpdateProcessorFactory to see
what is or ain't guaranteed when there are multiple input fields - whether
this is a bug or a feature or simply undefined.
-- Jack Krupansky
-Original Message-
From: Rishi Easwaran
Sent: Tuesday, May 28, 2013 3:54 PM
To: solr-user@lucene.apache.org
Subject: Re: Solr Composite Unique key from existing fields in schema
I thought the same, but that doesn't seem to be the case.
-Original Message-
From: Jack Krupansky j...@basetechnology.com
To: solr-user solr-user@lucene.apache.org
Sent: Tue, May 28, 2013 3:32 pm
Subject: Re: Solr Composite Unique key from existing fields in schema
The order in the ID should be purely dependent on the order of the field
names in the processor configuration:
str name=sourcedocid_s/str
str name=sourceuserid_s/str
-- Jack Krupansky
-Original Message-
From: Rishi Easwaran
Sent: Tuesday, May 28, 2013 2:54 PM
To: solr-user@lucene.apache.org
Subject: Re: Solr Composite Unique key from existing fields in schema
Jack,
No sure if this is the correct behaviour.
I set up updateRequestorPorcess chain as mentioned below, but looks like the
compositeId that is generated is based on input order.
For example:
If my input comes in as
field name=docid1/field
field name=userid12345/field
I get the following compositeId1-12345.
If I reverse the input
field name=userid12345/field
field name=docid1/field
I get the following compositeId 12345-1 .
In this case the compositeId is not unique and I am getting duplicates.
Thanks,
Rishi.
-Original Message-
From: Jack Krupansky j...@basetechnology.com
To: solr-user solr-user@lucene.apache.org
Sent: Tue, May 28, 2013 12:07 pm
Subject: Re: Solr Composite Unique key from existing fields in schema
You can do this by combining the builtin update processors.
Add this to your solrconfig:
updateRequestProcessorChain name=composite-id
processor class=solr.CloneFieldUpdateProcessorFactory
str name=sourcedocid_s/str
str name=sourceuserid_s/str
str name=destid/str
/processor
processor class=solr.ConcatFieldUpdateProcessorFactory
str name=fieldNameid/str
str name=delimiter--/str
/processor
processor class=solr.LogUpdateProcessorFactory /
processor class=solr.RunUpdateProcessorFactory /
/updateRequestProcessorChain
Add documents such as:
curl
http://localhost:8983/solr/update?commit=trueupdate.chain=composite-id; \
-H 'Content-type:application/json' -d '
[{title: Hello World,
docid_s: doc-1,
userid_s: user-1,
comments_ss: [Easy, Fast]}]'
And get results like:
title:[Hello World],
docid_s:doc-1,
userid_s:user-1,
comments_ss:[Easy,
Fast],
id:doc-1--user-1,
Add as many fields in whatever order you want using source in the clone
update processor, and pick your composite key field name as well. And set
the delimiter string as well in the concat update processor.
I managed to reverse the field order from what you requested (userid,
docid).
I used the standard Solr example schema, so I used dynamic fields for the
two ids, but use your own field names.
-- Jack Krupansky
-Original Message-
From: Rishi Easwaran
Sent: Tuesday, May 28, 2013 11:12 AM
To: solr-user@lucene.apache.org
Subject: Solr Composite Unique key from existing fields in schema
Hi All,
Historically we have used a single field in our schema as a uniqueKey.
field name=docidtype=string indexed=true stored=true
multiValued=false required=true/
field name=userid type=string indexed=true stored=true
multiValued=false required=true/
uniqueKeydocid/uniqueKey
Wanted to change this to a composite key something like
uniqueKeyuserid-docid/uniqueKey.
I know I can auto generate compositekey at document insert time, using
custom code to generate a new field, but wanted to know if there was an
inbuilt SOLR mechanism of doing this. That would prevent us from creating
and storing an extra field.
Thanks,
Rishi.
Re: Solr Composite Unique key from existing fields in schema
Thanks Jack, That fixed it and guarantees the order.
As far as I can tell SOLR cloud 4.2.1 needs a uniquekey defined in its schema,
or I get an exception.
SolrCore Initialization Failures
* testCloud2_shard1_replica1:
org.apache.solr.common.SolrException:org.apache.solr.common.SolrException:
QueryElevationComponent requires the schema to have a uniqueKeyField.
Now that I have an autogenerated composite-id, it has to become a part of my
schema as uniquekey for SOLR cloud to work.
field name=docidtype=string indexed=true stored=true
multiValued=false required=true/
field name=userid type=string indexed=true stored=true
multiValued=false required=true/
field name=compositeId type=string indexed=true stored=true
multiValued=false required=true/
uniqueKeycompositeId/uniqueKey
Is there a way to avoid compositeId field being defined in my schema.xml, would
like to avoid the overhead of storing this field in my index.
Thanks,
Rishi.
-Original Message-
From: Jack Krupansky j...@basetechnology.com
To: solr-user solr-user@lucene.apache.org
Sent: Tue, May 28, 2013 4:33 pm
Subject: Re: Solr Composite Unique key from existing fields in schema
The TL;DR response: Try this:
updateRequestProcessorChain name=composite-id
processor class=solr.CloneFieldUpdateProcessorFactory
str name=sourceuserid_s/str
str name=destid/str
/processor
processor class=solr.CloneFieldUpdateProcessorFactory
str name=sourcedocid_s/str
str name=destid/str
/processor
processor class=solr.ConcatFieldUpdateProcessorFactory
str name=fieldNameid/str
str name=delimiter--/str
/processor
processor class=solr.LogUpdateProcessorFactory /
processor class=solr.RunUpdateProcessorFactory /
/updateRequestProcessorChain
That will assure that the userid gets processed before the docid.
I'll have to review the contract for CloneFieldUpdateProcessorFactory to see
what is or ain't guaranteed when there are multiple input fields - whether
this is a bug or a feature or simply undefined.
-- Jack Krupansky
-Original Message-
From: Rishi Easwaran
Sent: Tuesday, May 28, 2013 3:54 PM
To: solr-user@lucene.apache.org
Subject: Re: Solr Composite Unique key from existing fields in schema
I thought the same, but that doesn't seem to be the case.
-Original Message-
From: Jack Krupansky j...@basetechnology.com
To: solr-user solr-user@lucene.apache.org
Sent: Tue, May 28, 2013 3:32 pm
Subject: Re: Solr Composite Unique key from existing fields in schema
The order in the ID should be purely dependent on the order of the field
names in the processor configuration:
str name=sourcedocid_s/str
str name=sourceuserid_s/str
-- Jack Krupansky
-Original Message-
From: Rishi Easwaran
Sent: Tuesday, May 28, 2013 2:54 PM
To: solr-user@lucene.apache.org
Subject: Re: Solr Composite Unique key from existing fields in schema
Jack,
No sure if this is the correct behaviour.
I set up updateRequestorPorcess chain as mentioned below, but looks like the
compositeId that is generated is based on input order.
For example:
If my input comes in as
field name=docid1/field
field name=userid12345/field
I get the following compositeId1-12345.
If I reverse the input
field name=userid12345/field
field name=docid1/field
I get the following compositeId 12345-1 .
In this case the compositeId is not unique and I am getting duplicates.
Thanks,
Rishi.
-Original Message-
From: Jack Krupansky j...@basetechnology.com
To: solr-user solr-user@lucene.apache.org
Sent: Tue, May 28, 2013 12:07 pm
Subject: Re: Solr Composite Unique key from existing fields in schema
You can do this by combining the builtin update processors.
Add this to your solrconfig:
updateRequestProcessorChain name=composite-id
processor class=solr.CloneFieldUpdateProcessorFactory
str name=sourcedocid_s/str
str name=sourceuserid_s/str
str name=destid/str
/processor
processor class=solr.ConcatFieldUpdateProcessorFactory
str name=fieldNameid/str
str name=delimiter--/str
/processor
processor class=solr.LogUpdateProcessorFactory /
processor class=solr.RunUpdateProcessorFactory /
/updateRequestProcessorChain
Add documents such as:
curl
http://localhost:8983/solr/update?commit=trueupdate.chain=composite-id; \
-H 'Content-type:application/json' -d '
[{title: Hello World,
docid_s: doc-1,
userid_s: user-1,
comments_ss: [Easy, Fast]}]'
And get results like:
title:[Hello World],
docid_s:doc-1,
userid_s:user-1,
comments_ss:[Easy,
Fast],
id:doc-1--user-1,
Add as many fields in whatever order you want using source in the clone
update processor, and pick your composite key field name as well. And set
the delimiter string as well in the concat update processor.
I managed to reverse the field order from what you requested (userid,
docid).
I used the standard Solr example schema, so I used dynamic fields for the
two ids, but use your own field names
Re: Solr Composite Unique key from existing fields in schema
Great. And I did verify that the field order cannot be guaranteed by a
single CloneFieldUpdateProcessorFactory with multiple field names - the
underlying code iterates over the input values, checks the field selector
for membership and then immediately adds to the output, so changing the
input order will change the output order. Also, field names are stored in a
HashSet anyway, which would tend to scramble their order.
-- Jack Krupansky
-Original Message-
From: Rishi Easwaran
Sent: Tuesday, May 28, 2013 6:01 PM
To: solr-user@lucene.apache.org
Subject: Re: Solr Composite Unique key from existing fields in schema
Thanks Jack, That fixed it and guarantees the order.
As far as I can tell SOLR cloud 4.2.1 needs a uniquekey defined in its
schema, or I get an exception.
SolrCore Initialization Failures
* testCloud2_shard1_replica1:
org.apache.solr.common.SolrException:org.apache.solr.common.SolrException:
QueryElevationComponent requires the schema to have a uniqueKeyField.
Now that I have an autogenerated composite-id, it has to become a part of my
schema as uniquekey for SOLR cloud to work.
field name=docidtype=string indexed=true stored=true
multiValued=false required=true/
field name=userid type=string indexed=true stored=true
multiValued=false required=true/
field name=compositeId type=string indexed=true stored=true
multiValued=false required=true/
uniqueKeycompositeId/uniqueKey
Is there a way to avoid compositeId field being defined in my schema.xml,
would like to avoid the overhead of storing this field in my index.
Thanks,
Rishi.
-Original Message-
From: Jack Krupansky j...@basetechnology.com
To: solr-user solr-user@lucene.apache.org
Sent: Tue, May 28, 2013 4:33 pm
Subject: Re: Solr Composite Unique key from existing fields in schema
The TL;DR response: Try this:
updateRequestProcessorChain name=composite-id
processor class=solr.CloneFieldUpdateProcessorFactory
str name=sourceuserid_s/str
str name=destid/str
/processor
processor class=solr.CloneFieldUpdateProcessorFactory
str name=sourcedocid_s/str
str name=destid/str
/processor
processor class=solr.ConcatFieldUpdateProcessorFactory
str name=fieldNameid/str
str name=delimiter--/str
/processor
processor class=solr.LogUpdateProcessorFactory /
processor class=solr.RunUpdateProcessorFactory /
/updateRequestProcessorChain
That will assure that the userid gets processed before the docid.
I'll have to review the contract for CloneFieldUpdateProcessorFactory to see
what is or ain't guaranteed when there are multiple input fields - whether
this is a bug or a feature or simply undefined.
-- Jack Krupansky
-Original Message-
From: Rishi Easwaran
Sent: Tuesday, May 28, 2013 3:54 PM
To: solr-user@lucene.apache.org
Subject: Re: Solr Composite Unique key from existing fields in schema
I thought the same, but that doesn't seem to be the case.
-Original Message-
From: Jack Krupansky j...@basetechnology.com
To: solr-user solr-user@lucene.apache.org
Sent: Tue, May 28, 2013 3:32 pm
Subject: Re: Solr Composite Unique key from existing fields in schema
The order in the ID should be purely dependent on the order of the field
names in the processor configuration:
str name=sourcedocid_s/str
str name=sourceuserid_s/str
-- Jack Krupansky
-Original Message-
From: Rishi Easwaran
Sent: Tuesday, May 28, 2013 2:54 PM
To: solr-user@lucene.apache.org
Subject: Re: Solr Composite Unique key from existing fields in schema
Jack,
No sure if this is the correct behaviour.
I set up updateRequestorPorcess chain as mentioned below, but looks like the
compositeId that is generated is based on input order.
For example:
If my input comes in as
field name=docid1/field
field name=userid12345/field
I get the following compositeId1-12345.
If I reverse the input
field name=userid12345/field
field name=docid1/field
I get the following compositeId 12345-1 .
In this case the compositeId is not unique and I am getting duplicates.
Thanks,
Rishi.
-Original Message-
From: Jack Krupansky j...@basetechnology.com
To: solr-user solr-user@lucene.apache.org
Sent: Tue, May 28, 2013 12:07 pm
Subject: Re: Solr Composite Unique key from existing fields in schema
You can do this by combining the builtin update processors.
Add this to your solrconfig:
updateRequestProcessorChain name=composite-id
processor class=solr.CloneFieldUpdateProcessorFactory
str name=sourcedocid_s/str
str name=sourceuserid_s/str
str name=destid/str
/processor
processor class=solr.ConcatFieldUpdateProcessorFactory
str name=fieldNameid/str
str name=delimiter--/str
/processor
processor class=solr.LogUpdateProcessorFactory /
processor class=solr.RunUpdateProcessorFactory /
/updateRequestProcessorChain
Add documents such as:
curl
http://localhost:8983/solr/update?commit=trueupdate.chain=composite-id; \
-H 'Content-type:application/json' -d '
[{title
