Re: _version_ / Versioning using timespan
I see. You can create a JIRA and submit patch and see if committers agree
or have different opinion/suggestion.
Thanks,
Susheel
On Fri, Jun 2, 2017 at 10:01 AM, Sergio García Maroto <marot...@gmail.com>
wrote:
> You are right about that but in some cases I may need to reindex my data
> and wanted to avoid deleting the full index so
> I can still server queries. I thought reindexing same version would be
> handy or at least to have the flexibility.
>
> On 2 June 2017 at 14:53, Susheel Kumar <susheel2...@gmail.com> wrote:
>
> > I see the difference now between using _version_ vs custom versionField.
> > Both seems to behave differently. The _version_ field if used allows
> same
> > version to be updated and that's the perception I had in mind for custom
> > versionField.
> >
> > My question is why do you want to update the document if same version.
> > Shouldn't you pass higher version if the doc has changed and that makes
> the
> > update to be accepted ?
> >
> > On Fri, Jun 2, 2017 at 8:13 AM, Susheel Kumar <susheel2...@gmail.com>
> > wrote:
> >
> > > Just to confirm again before go too far, are you able to execute these
> > > examples and see same output given under "Optimistic Concurrency".
> > > https://cwiki.apache.org/confluence/display/solr/
> > > Updating+Parts+of+Documents#UpdatingPartsofDocuments-In-PlaceUpdates
> > >
> > > Let me know which example you fail to get same output as described in.
> > >
> > > On Fri, Jun 2, 2017 at 5:11 AM, Sergio García Maroto <
> marot...@gmail.com
> > >
> > > wrote:
> > >
> > >> I had a look to the source code and I see
> > >> DocBasedVersionConstraintsProcessorFactory
> > >>
> > >> if (0 < ((Comparable)newUserVersion).compareTo((Comparable)
> > >> oldUserVersion)) {
> > >> // log.info("VERSION returning true (proceed with update)"
> );
> > >> return true;
> > >> }
> > >>
> > >> I can't find a way of overwriting same version without changing that
> > piece
> > >> of code.
> > >> Would be possible to add a parameter to the
> > >> "DocBasedVersionConstraintsProcessorFactory" something like
> > >> "overwrite.same.version=true"
> > >> so the new code would look like.
> > >>
> > >>
> > >> int compareTo = ((Comparable)newUserVersion).compareTo((Comparable)
> > >> oldUserVersion);
> > >> if ( ((overwritesameversion) && 0 <= compareTo) || (0 < compareTo)) {
> > >> // log.info("VERSION returning true (proceed with update)"
> );
> > >> return true;
> > >> }
> > >>
> > >>
> > >> Is that thing going to break anyhting? Can i do that change?
> > >>
> > >> Thanks
> > >> Sergio
> > >>
> > >>
> > >> On 2 June 2017 at 10:10, Sergio García Maroto <marot...@gmail.com>
> > wrote:
> > >>
> > >> > I am using 6.1.0.
> > >> > I tried with two different field types, long and date.
> > >> > > />
> > >> > > stored="true"/>
> > >> >
> > >> > I am using this configuration on the solrconfig.xml
> > >> >
> > >> >
> > >> >
> > >> > false
> > >> > UpdatedDateSD
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> > i had a look to the wiki page and it says https://cwiki.apache.org/
> > >> > confluence/display/solr/Updating+Parts+of+Documents
> > >> >
> > >> > *Once configured, this update processor will reject (HTTP error code
> > >> 409)
> > >> > any attempt to update an existing document where the value of
> > >> > the my_version_l field in the "new" document is not greater then the
> > >> value
> > >> > of that field in the existing document.*
> > >> >
> > >> > Do you have any tip on how to get same versions not getting
> rejected.
> > >> >
> > >> > Thanks a lot.
> > >> >
> > >> >
> > >> > On 1 June 2017 at 19:04, Susheel Kumar <susheel2...@gmail.com>
> wrote:
> > >> >
> > >> >> Which version of solr are you using? I tested in 6.0 and if I
> supply
> > >> same
> > >> >> version, it overwrite/update the document exactly as per the wiki
> > >> >> documentation.
> > >> >>
> > >> >> Thanks,
> > >> >> Susheel
> > >> >>
> > >> >> On Thu, Jun 1, 2017 at 7:57 AM, marotosg <marot...@gmail.com>
> wrote:
> > >> >>
> > >> >> > Thanks a lot Susheel.
> > >> >> > I see this is actually what I need. I have been testing it and
> > >> notice
> > >> >> the
> > >> >> > value of the field has to be always greater for a new document to
> > get
> > >> >> > indexed. if you send the same version number it doesn't work.
> > >> >> >
> > >> >> > Is it possible somehow to overwrite documents with the same
> > version?
> > >> >> >
> > >> >> > Thanks
> > >> >> >
> > >> >> >
> > >> >> >
> > >> >> > --
> > >> >> > View this message in context: http://lucene.472066.n3.
> > >> >> > nabble.com/version-Versioning-using-timespan-
> > tp4338171p4338475.html
> > >> >> > Sent from the Solr - User mailing list archive at Nabble.com.
> > >> >> >
> > >> >>
> > >> >
> > >> >
> > >>
> > >
> > >
> >
>
Re: _version_ / Versioning using timespan
You are right about that but in some cases I may need to reindex my data
and wanted to avoid deleting the full index so
I can still server queries. I thought reindexing same version would be
handy or at least to have the flexibility.
On 2 June 2017 at 14:53, Susheel Kumar <susheel2...@gmail.com> wrote:
> I see the difference now between using _version_ vs custom versionField.
> Both seems to behave differently. The _version_ field if used allows same
> version to be updated and that's the perception I had in mind for custom
> versionField.
>
> My question is why do you want to update the document if same version.
> Shouldn't you pass higher version if the doc has changed and that makes the
> update to be accepted ?
>
> On Fri, Jun 2, 2017 at 8:13 AM, Susheel Kumar <susheel2...@gmail.com>
> wrote:
>
> > Just to confirm again before go too far, are you able to execute these
> > examples and see same output given under "Optimistic Concurrency".
> > https://cwiki.apache.org/confluence/display/solr/
> > Updating+Parts+of+Documents#UpdatingPartsofDocuments-In-PlaceUpdates
> >
> > Let me know which example you fail to get same output as described in.
> >
> > On Fri, Jun 2, 2017 at 5:11 AM, Sergio García Maroto <marot...@gmail.com
> >
> > wrote:
> >
> >> I had a look to the source code and I see
> >> DocBasedVersionConstraintsProcessorFactory
> >>
> >> if (0 < ((Comparable)newUserVersion).compareTo((Comparable)
> >> oldUserVersion)) {
> >> // log.info("VERSION returning true (proceed with update)" );
> >> return true;
> >> }
> >>
> >> I can't find a way of overwriting same version without changing that
> piece
> >> of code.
> >> Would be possible to add a parameter to the
> >> "DocBasedVersionConstraintsProcessorFactory" something like
> >> "overwrite.same.version=true"
> >> so the new code would look like.
> >>
> >>
> >> int compareTo = ((Comparable)newUserVersion).compareTo((Comparable)
> >> oldUserVersion);
> >> if ( ((overwritesameversion) && 0 <= compareTo) || (0 < compareTo)) {
> >> // log.info("VERSION returning true (proceed with update)" );
> >> return true;
> >> }
> >>
> >>
> >> Is that thing going to break anyhting? Can i do that change?
> >>
> >> Thanks
> >> Sergio
> >>
> >>
> >> On 2 June 2017 at 10:10, Sergio García Maroto <marot...@gmail.com>
> wrote:
> >>
> >> > I am using 6.1.0.
> >> > I tried with two different field types, long and date.
> >> > />
> >> > stored="true"/>
> >> >
> >> > I am using this configuration on the solrconfig.xml
> >> >
> >> >
> >> >
> >> > false
> >> > UpdatedDateSD
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> > i had a look to the wiki page and it says https://cwiki.apache.org/
> >> > confluence/display/solr/Updating+Parts+of+Documents
> >> >
> >> > *Once configured, this update processor will reject (HTTP error code
> >> 409)
> >> > any attempt to update an existing document where the value of
> >> > the my_version_l field in the "new" document is not greater then the
> >> value
> >> > of that field in the existing document.*
> >> >
> >> > Do you have any tip on how to get same versions not getting rejected.
> >> >
> >> > Thanks a lot.
> >> >
> >> >
> >> > On 1 June 2017 at 19:04, Susheel Kumar <susheel2...@gmail.com> wrote:
> >> >
> >> >> Which version of solr are you using? I tested in 6.0 and if I supply
> >> same
> >> >> version, it overwrite/update the document exactly as per the wiki
> >> >> documentation.
> >> >>
> >> >> Thanks,
> >> >> Susheel
> >> >>
> >> >> On Thu, Jun 1, 2017 at 7:57 AM, marotosg <marot...@gmail.com> wrote:
> >> >>
> >> >> > Thanks a lot Susheel.
> >> >> > I see this is actually what I need. I have been testing it and
> >> notice
> >> >> the
> >> >> > value of the field has to be always greater for a new document to
> get
> >> >> > indexed. if you send the same version number it doesn't work.
> >> >> >
> >> >> > Is it possible somehow to overwrite documents with the same
> version?
> >> >> >
> >> >> > Thanks
> >> >> >
> >> >> >
> >> >> >
> >> >> > --
> >> >> > View this message in context: http://lucene.472066.n3.
> >> >> > nabble.com/version-Versioning-using-timespan-
> tp4338171p4338475.html
> >> >> > Sent from the Solr - User mailing list archive at Nabble.com.
> >> >> >
> >> >>
> >> >
> >> >
> >>
> >
> >
>
Re: _version_ / Versioning using timespan
I see the difference now between using _version_ vs custom versionField.
Both seems to behave differently. The _version_ field if used allows same
version to be updated and that's the perception I had in mind for custom
versionField.
My question is why do you want to update the document if same version.
Shouldn't you pass higher version if the doc has changed and that makes the
update to be accepted ?
On Fri, Jun 2, 2017 at 8:13 AM, Susheel Kumar <susheel2...@gmail.com> wrote:
> Just to confirm again before go too far, are you able to execute these
> examples and see same output given under "Optimistic Concurrency".
> https://cwiki.apache.org/confluence/display/solr/
> Updating+Parts+of+Documents#UpdatingPartsofDocuments-In-PlaceUpdates
>
> Let me know which example you fail to get same output as described in.
>
> On Fri, Jun 2, 2017 at 5:11 AM, Sergio García Maroto <marot...@gmail.com>
> wrote:
>
>> I had a look to the source code and I see
>> DocBasedVersionConstraintsProcessorFactory
>>
>> if (0 < ((Comparable)newUserVersion).compareTo((Comparable)
>> oldUserVersion)) {
>> // log.info("VERSION returning true (proceed with update)" );
>> return true;
>> }
>>
>> I can't find a way of overwriting same version without changing that piece
>> of code.
>> Would be possible to add a parameter to the
>> "DocBasedVersionConstraintsProcessorFactory" something like
>> "overwrite.same.version=true"
>> so the new code would look like.
>>
>>
>> int compareTo = ((Comparable)newUserVersion).compareTo((Comparable)
>> oldUserVersion);
>> if ( ((overwritesameversion) && 0 <= compareTo) || (0 < compareTo)) {
>> // log.info("VERSION returning true (proceed with update)" );
>> return true;
>> }
>>
>>
>> Is that thing going to break anyhting? Can i do that change?
>>
>> Thanks
>> Sergio
>>
>>
>> On 2 June 2017 at 10:10, Sergio García Maroto <marot...@gmail.com> wrote:
>>
>> > I am using 6.1.0.
>> > I tried with two different field types, long and date.
>> >
>> >
>> >
>> > I am using this configuration on the solrconfig.xml
>> >
>> >
>> >
>> > false
>> > UpdatedDateSD
>> >
>> >
>> >
>> >
>> >
>> >
>> > i had a look to the wiki page and it says https://cwiki.apache.org/
>> > confluence/display/solr/Updating+Parts+of+Documents
>> >
>> > *Once configured, this update processor will reject (HTTP error code
>> 409)
>> > any attempt to update an existing document where the value of
>> > the my_version_l field in the "new" document is not greater then the
>> value
>> > of that field in the existing document.*
>> >
>> > Do you have any tip on how to get same versions not getting rejected.
>> >
>> > Thanks a lot.
>> >
>> >
>> > On 1 June 2017 at 19:04, Susheel Kumar <susheel2...@gmail.com> wrote:
>> >
>> >> Which version of solr are you using? I tested in 6.0 and if I supply
>> same
>> >> version, it overwrite/update the document exactly as per the wiki
>> >> documentation.
>> >>
>> >> Thanks,
>> >> Susheel
>> >>
>> >> On Thu, Jun 1, 2017 at 7:57 AM, marotosg <marot...@gmail.com> wrote:
>> >>
>> >> > Thanks a lot Susheel.
>> >> > I see this is actually what I need. I have been testing it and
>> notice
>> >> the
>> >> > value of the field has to be always greater for a new document to get
>> >> > indexed. if you send the same version number it doesn't work.
>> >> >
>> >> > Is it possible somehow to overwrite documents with the same version?
>> >> >
>> >> > Thanks
>> >> >
>> >> >
>> >> >
>> >> > --
>> >> > View this message in context: http://lucene.472066.n3.
>> >> > nabble.com/version-Versioning-using-timespan-tp4338171p4338475.html
>> >> > Sent from the Solr - User mailing list archive at Nabble.com.
>> >> >
>> >>
>> >
>> >
>>
>
>
Re: _version_ / Versioning using timespan
Just to confirm again before go too far, are you able to execute these
examples and see same output given under "Optimistic Concurrency".
https://cwiki.apache.org/confluence/display/solr/Updating+Parts+of+Documents#UpdatingPartsofDocuments-In-PlaceUpdates
Let me know which example you fail to get same output as described in.
On Fri, Jun 2, 2017 at 5:11 AM, Sergio García Maroto <marot...@gmail.com>
wrote:
> I had a look to the source code and I see
> DocBasedVersionConstraintsProcessorFactory
>
> if (0 < ((Comparable)newUserVersion).compareTo((Comparable)
> oldUserVersion)) {
> // log.info("VERSION returning true (proceed with update)" );
> return true;
> }
>
> I can't find a way of overwriting same version without changing that piece
> of code.
> Would be possible to add a parameter to the
> "DocBasedVersionConstraintsProcessorFactory" something like
> "overwrite.same.version=true"
> so the new code would look like.
>
>
> int compareTo = ((Comparable)newUserVersion).compareTo((Comparable)
> oldUserVersion);
> if ( ((overwritesameversion) && 0 <= compareTo) || (0 < compareTo)) {
> // log.info("VERSION returning true (proceed with update)" );
> return true;
> }
>
>
> Is that thing going to break anyhting? Can i do that change?
>
> Thanks
> Sergio
>
>
> On 2 June 2017 at 10:10, Sergio García Maroto <marot...@gmail.com> wrote:
>
> > I am using 6.1.0.
> > I tried with two different field types, long and date.
> >
> >
> >
> > I am using this configuration on the solrconfig.xml
> >
> >
> >
> > false
> > UpdatedDateSD
> >
> >
> >
> >
> >
> >
> > i had a look to the wiki page and it says https://cwiki.apache.org/
> > confluence/display/solr/Updating+Parts+of+Documents
> >
> > *Once configured, this update processor will reject (HTTP error code 409)
> > any attempt to update an existing document where the value of
> > the my_version_l field in the "new" document is not greater then the
> value
> > of that field in the existing document.*
> >
> > Do you have any tip on how to get same versions not getting rejected.
> >
> > Thanks a lot.
> >
> >
> > On 1 June 2017 at 19:04, Susheel Kumar <susheel2...@gmail.com> wrote:
> >
> >> Which version of solr are you using? I tested in 6.0 and if I supply
> same
> >> version, it overwrite/update the document exactly as per the wiki
> >> documentation.
> >>
> >> Thanks,
> >> Susheel
> >>
> >> On Thu, Jun 1, 2017 at 7:57 AM, marotosg <marot...@gmail.com> wrote:
> >>
> >> > Thanks a lot Susheel.
> >> > I see this is actually what I need. I have been testing it and
> notice
> >> the
> >> > value of the field has to be always greater for a new document to get
> >> > indexed. if you send the same version number it doesn't work.
> >> >
> >> > Is it possible somehow to overwrite documents with the same version?
> >> >
> >> > Thanks
> >> >
> >> >
> >> >
> >> > --
> >> > View this message in context: http://lucene.472066.n3.
> >> > nabble.com/version-Versioning-using-timespan-tp4338171p4338475.html
> >> > Sent from the Solr - User mailing list archive at Nabble.com.
> >> >
> >>
> >
> >
>
Re: _version_ / Versioning using timespan
I had a look to the source code and I see
DocBasedVersionConstraintsProcessorFactory
if (0 < ((Comparable)newUserVersion).compareTo((Comparable)
oldUserVersion)) {
// log.info("VERSION returning true (proceed with update)" );
return true;
}
I can't find a way of overwriting same version without changing that piece
of code.
Would be possible to add a parameter to the
"DocBasedVersionConstraintsProcessorFactory" something like
"overwrite.same.version=true"
so the new code would look like.
int compareTo = ((Comparable)newUserVersion).compareTo((Comparable)
oldUserVersion);
if ( ((overwritesameversion) && 0 <= compareTo) || (0 < compareTo)) {
// log.info("VERSION returning true (proceed with update)" );
return true;
}
Is that thing going to break anyhting? Can i do that change?
Thanks
Sergio
On 2 June 2017 at 10:10, Sergio García Maroto <marot...@gmail.com> wrote:
> I am using 6.1.0.
> I tried with two different field types, long and date.
>
>
>
> I am using this configuration on the solrconfig.xml
>
>
>
> false
> UpdatedDateSD
>
>
>
>
>
>
> i had a look to the wiki page and it says https://cwiki.apache.org/
> confluence/display/solr/Updating+Parts+of+Documents
>
> *Once configured, this update processor will reject (HTTP error code 409)
> any attempt to update an existing document where the value of
> the my_version_l field in the "new" document is not greater then the value
> of that field in the existing document.*
>
> Do you have any tip on how to get same versions not getting rejected.
>
> Thanks a lot.
>
>
> On 1 June 2017 at 19:04, Susheel Kumar <susheel2...@gmail.com> wrote:
>
>> Which version of solr are you using? I tested in 6.0 and if I supply same
>> version, it overwrite/update the document exactly as per the wiki
>> documentation.
>>
>> Thanks,
>> Susheel
>>
>> On Thu, Jun 1, 2017 at 7:57 AM, marotosg <marot...@gmail.com> wrote:
>>
>> > Thanks a lot Susheel.
>> > I see this is actually what I need. I have been testing it and notice
>> the
>> > value of the field has to be always greater for a new document to get
>> > indexed. if you send the same version number it doesn't work.
>> >
>> > Is it possible somehow to overwrite documents with the same version?
>> >
>> > Thanks
>> >
>> >
>> >
>> > --
>> > View this message in context: http://lucene.472066.n3.
>> > nabble.com/version-Versioning-using-timespan-tp4338171p4338475.html
>> > Sent from the Solr - User mailing list archive at Nabble.com.
>> >
>>
>
>
Re: _version_ / Versioning using timespan
I am using 6.1.0. I tried with two different field types, long and date. I am using this configuration on the solrconfig.xml false UpdatedDateSD i had a look to the wiki page and it says https://cwiki.apache.org/confluence/display/solr/Updating+Parts+of+Documents *Once configured, this update processor will reject (HTTP error code 409) any attempt to update an existing document where the value of the my_version_l field in the "new" document is not greater then the value of that field in the existing document.* Do you have any tip on how to get same versions not getting rejected. Thanks a lot. On 1 June 2017 at 19:04, Susheel Kumar <susheel2...@gmail.com> wrote: > Which version of solr are you using? I tested in 6.0 and if I supply same > version, it overwrite/update the document exactly as per the wiki > documentation. > > Thanks, > Susheel > > On Thu, Jun 1, 2017 at 7:57 AM, marotosg <marot...@gmail.com> wrote: > > > Thanks a lot Susheel. > > I see this is actually what I need. I have been testing it and notice > the > > value of the field has to be always greater for a new document to get > > indexed. if you send the same version number it doesn't work. > > > > Is it possible somehow to overwrite documents with the same version? > > > > Thanks > > > > > > > > -- > > View this message in context: http://lucene.472066.n3. > > nabble.com/version-Versioning-using-timespan-tp4338171p4338475.html > > Sent from the Solr - User mailing list archive at Nabble.com. > > >
Re: _version_ / Versioning using timespan
Which version of solr are you using? I tested in 6.0 and if I supply same version, it overwrite/update the document exactly as per the wiki documentation. Thanks, Susheel On Thu, Jun 1, 2017 at 7:57 AM, marotosg <marot...@gmail.com> wrote: > Thanks a lot Susheel. > I see this is actually what I need. I have been testing it and notice the > value of the field has to be always greater for a new document to get > indexed. if you send the same version number it doesn't work. > > Is it possible somehow to overwrite documents with the same version? > > Thanks > > > > -- > View this message in context: http://lucene.472066.n3. > nabble.com/version-Versioning-using-timespan-tp4338171p4338475.html > Sent from the Solr - User mailing list archive at Nabble.com. >
Re: _version_ / Versioning using timespan
Thanks a lot Susheel. I see this is actually what I need. I have been testing it and notice the value of the field has to be always greater for a new document to get indexed. if you send the same version number it doesn't work. Is it possible somehow to overwrite documents with the same version? Thanks -- View this message in context: http://lucene.472066.n3.nabble.com/version-Versioning-using-timespan-tp4338171p4338475.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: _version_ / Versioning using timespan
"Document Centric Versioning Constraints" is what you are looking for if you want this to handled in Solr https://cwiki.apache.org/confluence/display/solr/Updating+Parts+of+Documents -- Susheel On Wed, May 31, 2017 at 6:46 AM, marotosg <marot...@gmail.com> wrote: > Hi all. > > I need to implement an indexation solution where my Solr index doesn't get > a > wrong version. > Due to the fact I have many version for the same entity In some cases my > client may end up indexing an earlier version of my entity after a newer > one. > > I was wondering if I can use the _version_ field where I send always the > timespam of last update to the entity. > > Is this going to reject earlier versions if a newer is already indexed? > > Thanks, > Sergio > > > > > > -- > View this message in context: http://lucene.472066.n3. > nabble.com/version-Versioning-using-timespan-tp4338171.html > Sent from the Solr - User mailing list archive at Nabble.com. >
_version_ / Versioning using timespan
Hi all. I need to implement an indexation solution where my Solr index doesn't get a wrong version. Due to the fact I have many version for the same entity In some cases my client may end up indexing an earlier version of my entity after a newer one. I was wondering if I can use the _version_ field where I send always the timespam of last update to the entity. Is this going to reject earlier versions if a newer is already indexed? Thanks, Sergio -- View this message in context: http://lucene.472066.n3.nabble.com/version-Versioning-using-timespan-tp4338171.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: is API versioning supported in rolr?
On 5/26/2016 2:37 AM, Nuhaa All Bakry wrote: > Wondering if versioning is built-in in Solr? Say I have deployed a working > SolrCloud (v1.0) and there are applications consuming the REST APIs. Is there > a way to deploy the next v1.1 without removing v1.0? The reason I ask is > because we dont want the deployment of Solr to be tightly dependent on the > deployment of the applications, or vice versa. > > I cant find a documentation on this (yet). Please share if you know where I > can read more about this. In general, the Solr HTTP API does *not* change very much. Quite a lot of the HTTP API is dictated by solrconfig.xml, with recommendations in the examples, so it is frequently possible to make a new *major* version behave like the previous major version. Upgrading through two major versions might reveal differences that cannot be adjusted by configuration. The response formats (json, xml, javabin, etc) can accept a version parameter which can control aspects of that response format, and using an old version number can ensure compatibility with older code. New versions of the response formats are very rare, and many people use them without a version number and don't have any problems. If you are only using the HTTP API, then I would not be concerned too much about this when upgrading within a major version ... and your code may also work perfectly fine through a major version upgrade. There is one exception to what I said above: If you are using CloudSolrClient from SolrJ (or another cloud-aware client) to talk to your SolrCloud, then this is much more tightly coupled to the version, because CloudSolrClient talks to *zookeeper* (as well as using the HTTP API) and is strongly aware of SolrCloud internals. Mixing different SolrJ and Solr versions is discouraged when using CloudSolrClient, unless the version difference is very small and SolrJ is newer. The reason this can be a problem is that SolrCloud internals have been evolving *very* rapidly with each new release. Thanks, Shawn
is API versioning supported in rolr?
Hello, Wondering if versioning is built-in in Solr? Say I have deployed a working SolrCloud (v1.0) and there are applications consuming the REST APIs. Is there a way to deploy the next v1.1 without removing v1.0? The reason I ask is because we dont want the deployment of Solr to be tightly dependent on the deployment of the applications, or vice versa. I cant find a documentation on this (yet). Please share if you know where I can read more about this. regards, nuhaa
Re: Validate data Indexed and versioning
On 3/2/2015 4:32 AM, marotosg wrote: Is there any general approach to check if your indexed document matches the database row?. No, there's nothing out of the box that will do this. You'll need to write such an application yourself. You might want to leverage the /export handler in your application: https://cwiki.apache.org/confluence/display/solr/Exporting+Result+Sets Note that if you are not specifying stored=true on all of your fields in the Solr schema, you will not be able to fully validate all of the data in your index. Thanks, Shawn
Validate data Indexed and versioning
Hi, I am trying to define a way of validating if my index has the same content than my database. I am indexing a very complex denormalized version of the database with many items and nested documents. I have an indexation service which pulls records from a staging table(created based on a ETL process), transforms this table into xml which will be posted to Solr. Is there any general approach to check if your indexed document matches the database row?. One option I see is to create an additional service to run against solr and database and validate if has the same data but this is going to be very intensive. I was more on the opinion of solr telling the record indexed and content like number of nested docs of type A,B etc., Any suggestions would help. Thanks Regards -- View this message in context: http://lucene.472066.n3.nabble.com/Validate-data-Indexed-and-versioning-tp4190304.html Sent from the Solr - User mailing list archive at Nabble.com.
RE: Validate data Indexed and versioning
First, I would invest the largest effort towards developing good test cases and a good test harness for your ETL software itself. If validation in production does encounter errors, it should be considered a bug in your code! So be sure to always add these cases to your test harness. Also, the row level validation can and should be driven by metadata. I'm assuming you have a mapping between RDBMS table names and Solr entity types? And, for any given entity type, a table that maps solr field names and datatypes to their RDBMS equivalents? My assumption would be that the ETL process itself uses such metadata. The same data could be used for production data validation. My inclination would be to integrate granular / row-level validation into the ETL job itself. For summary validation, if re-indexing from scratch every time, just run some facet queries and compare to the equivalent summaries for the SQL input data (assuming you are familiar with SQL group by and having clauses).If using incremental loads, make sure you can associate the loaded data with the ETL job that loaded it (timestamp, batch ID, etc.). Then simply scope the facet queries by the batch in question and compare to the SQL summary. -Original Message- From: marotosg [mailto:marot...@gmail.com] Sent: Monday, March 02, 2015 6:32 AM To: solr-user@lucene.apache.org Subject: Validate data Indexed and versioning Hi, I am trying to define a way of validating if my index has the same content than my database. I am indexing a very complex denormalized version of the database with many items and nested documents. I have an indexation service which pulls records from a staging table(created based on a ETL process), transforms this table into xml which will be posted to Solr. Is there any general approach to check if your indexed document matches the database row?. One option I see is to create an additional service to run against solr and database and validate if has the same data but this is going to be very intensive. I was more on the opinion of solr telling the record indexed and content like number of nested docs of type A,B etc., Any suggestions would help. Thanks Regards -- View this message in context: http://lucene.472066.n3.nabble.com/Validate-data-Indexed-and-versioning-tp4190304.html Sent from the Solr - User mailing list archive at Nabble.com. * This e-mail may contain confidential or privileged information. If you are not the intended recipient, please notify the sender immediately and then delete it. TIAA-CREF *
Document versioning support
Suppose I have a text field called myTextField . Sometimes the field content may change . I would like to have all versions of this field be indexed in Solr . What I want to do , is to make 'myTextField' contain the latest version of the content and create additional multivalued field called 'myTextField_history' which will contain all previous versions . In this way I can make a boost on 'myTextField' during a search and also have all versions be indexed and also to know what version is the latest . Does Solr has some build in mechanism which can do the same ? Or my solution is good enough ? -- View this message in context: http://lucene.472066.n3.nabble.com/Document-versioning-support-tp4168417.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Versioning
Depends on exactly what you mean by versioning. But if you mean that every document in Solr gets a version-number which is increased every time the document is updated, all you need to do is to add a _version_ field in you schema: http://wiki.apache.org/solr/SolrCloud#Required_Config Believe you will get optimistic locking out-of-the-box if you do this (you will also need the updateLog configured in solrconfig.xml). Or else you can take my patch for SOLR-3178 and have optimistic locking work as described on: http://wiki.apache.org/solr/Per%20Steffensen/Update%20semantics Regards, Per Steffensen Sushil jain skrev: Hello Everyone, I am a Solr beginner. I just want to know if versioning of data is possible in Solr, if yes then please share the procedure. Thanks Regards, Sushil Jain
Re: Unique key constraint and optimistic locking (versioning)
Created SOLR-3178 covering the versioning/optimistic-locking part. In combination SOLR-3173 and SOLR-3178 should provide the features I am missing, and that I believe lots of other SOLR users will be able to benefit from. Please help shape by commenting on the Jira issues. Thanks. Per Steffensen skrev: Created SOLR-3173 on the part about making insert fail if document (with same uniqueKey) already exists. SOLR-3173 also includes to make update not insert document if not already exists - just for consistency with normal RDBMS behaviour. So basically the feature allowes you to turn on this behaviour of having database (RDBMS) semantics, and when you do you get both. Tomorrrow will create another Jira issue on the versioning/optimistic locking part. Per Steffensen skrev: Hi Does solr/lucene provide any mechanism for unique key constraint and optimistic locking (versioning)? Unique key constraint: That a client will not succeed creating a new document in solr/lucene if a document already exists having the same value in some field (e.g. an id field). Of course implemented right, so that even though two or more threads are concurrently trying to create a new document with the same value in this field, only one of them will succeed. Optimistic locking (versioning): That a client will only succeed updating a document if this updated document is based on the version of the document currently stored in solr/lucene. Implemented in the optimistic way that clients during an update have to tell which version of the document they fetched from Solr and that they therefore have used as a starting-point for their updated document. So basically having a version field on the document that clients increase by one before sending to solr for update, and some code in Solr that only makes the update succeed if the version number of the updated document is exactly one higher than the version number of the document already stored. Of course again implemented right, so that even though two or more thrads are concurrently trying to update a document, and they all have their updated document based on the current version in solr/lucene, only one of them will succeed. Or do I have to do stuff like this myself outside solr/lucene - e.g. in the client using solr. Regards, Per Steffensen
Re: Unique key constraint and optimistic locking (versioning)
Created SOLR-3173 on the part about making insert fail if document (with same uniqueKey) already exists. SOLR-3173 also includes to make update not insert document if not already exists - just for consistency with normal RDBMS behaviour. So basically the feature allowes you to turn on this behaviour of having database (RDBMS) semantics, and when you do you get both. Tomorrrow will create another Jira issue on the versioning/optimistic locking part. Per Steffensen skrev: Hi Does solr/lucene provide any mechanism for unique key constraint and optimistic locking (versioning)? Unique key constraint: That a client will not succeed creating a new document in solr/lucene if a document already exists having the same value in some field (e.g. an id field). Of course implemented right, so that even though two or more threads are concurrently trying to create a new document with the same value in this field, only one of them will succeed. Optimistic locking (versioning): That a client will only succeed updating a document if this updated document is based on the version of the document currently stored in solr/lucene. Implemented in the optimistic way that clients during an update have to tell which version of the document they fetched from Solr and that they therefore have used as a starting-point for their updated document. So basically having a version field on the document that clients increase by one before sending to solr for update, and some code in Solr that only makes the update succeed if the version number of the updated document is exactly one higher than the version number of the document already stored. Of course again implemented right, so that even though two or more thrads are concurrently trying to update a document, and they all have their updated document based on the current version in solr/lucene, only one of them will succeed. Or do I have to do stuff like this myself outside solr/lucene - e.g. in the client using solr. Regards, Per Steffensen
Re: Unique key constraint and optimistic locking (versioning)
Em skrev: Hi Per, I want an error to occur if a document with the same id already exists, when my intent is to INSERT a new document. When my intent is to UPDATE a document in solr/lucene I want the old document already in solr/lucene deleted and the new version of this document added (exactly as you explain). It will not be possible for solr/lucene to decide what to do unless I give it some information about my intent - whether it is INSERT or UPDATE semantics I want. I guess solr/lucene always give me INSERT sematics when a document with the same id does not already exist, and that it always give me UPDATE semantics when a document with the same id does exist? I cannot decide? Given that you've set a uniqueKey-field and there already exists a document with that uniqueKey, it will delete the old one and insert the new one. There is really no difference between the semantics - updates do not exist. To create a UNIQUE-constraint as you know it from a database you have to check whether a document is already in the index *or* whether it is already pending (waiting for getting flushed to the index). Fortunately Solr manages a so called pending-set with all those documents waiting for beeing flushed to disk (Solr 3.5). We are using latest greates 4.0-SNAPSHOT code, because we want to take advantage of SolrCloud stuff. Can you give a code-pointer to where I can find the pending-set stuff? Does solr use this pending-set for query responses, so that solr deliver 100% real-time search results? I think you have to write your own DirectUpdateHandler to achieve what you want on the Solr-level or to extend Lucenes IndexWriter to do it on the Lucene-Level. While doing so, keep track of what is going on in the trunk and how Near-Real-Time-Search will change the current way of handling updates. Will do. We already use auto soft commits. There is not built-in way to make solr/lucene give me an error if I try to insert a new document with an id equal to a document already in the index/core/shard. The existing document will always be updated (implemented as old deleted and new added). Correct? Exactly. If you really want to get your hands on that topic I suggest you to learn more about Lucene's IndexWriter: http://lucene.apache.org/core/old_versioned_docs/versions/3_5_0/api/all/index.html?org/apache/lucene/index/IndexWriter.html Kind Regards, Em
Re: Unique key constraint and optimistic locking (versioning)
Given that you've set a uniqueKey-field and there already exists a document with that uniqueKey, it will delete the old one and insert the new one. There is really no difference between the semantics - updates do not exist. To create a UNIQUE-constraint as you know it from a database you have to check whether a document is already in the index *or* whether it is already pending (waiting for getting flushed to the index). Fortunately Solr manages a so called pending-set with all those documents waiting for beeing flushed to disk (Solr 3.5). We are using latest greates 4.0-SNAPSHOT code, because we want to take advantage of SolrCloud stuff. Can you give a code-pointer to where I can find the pending-set stuff? I am not sure if this is what you're asking but you should be able to get the latest data from Solr by using realtime get http://wiki.apache.org/solr/RealTimeGet -- Sami Siren
Re: Unique key constraint and optimistic locking (versioning)
Sami Siren skrev: Given that you've set a uniqueKey-field and there already exists a document with that uniqueKey, it will delete the old one and insert the new one. There is really no difference between the semantics - updates do not exist. To create a UNIQUE-constraint as you know it from a database you have to check whether a document is already in the index *or* whether it is already pending (waiting for getting flushed to the index). Fortunately Solr manages a so called pending-set with all those documents waiting for beeing flushed to disk (Solr 3.5). We are using latest greates 4.0-SNAPSHOT code, because we want to take advantage of SolrCloud stuff. Can you give a code-pointer to where I can find the pending-set stuff? I am not sure if this is what you're asking but you should be able to get the latest data from Solr by using realtime get http://wiki.apache.org/solr/RealTimeGet Thanks a lot! I might be very usefull, if this provide 100% real time get - that is, if it gets the latest version of the document, also when neither a soft-commit nor a hard-commit has been performed since the lastest version of the document was indexed. Does it do that, or does it need a soft commit (then I believe it is only a near real time get operation)? -- Sami Siren
Re: Unique key constraint and optimistic locking (versioning)
On Fri, Feb 24, 2012 at 12:06 PM, Per Steffensen st...@designware.dk wrote: Sami Siren skrev: Given that you've set a uniqueKey-field and there already exists a document with that uniqueKey, it will delete the old one and insert the new one. There is really no difference between the semantics - updates do not exist. To create a UNIQUE-constraint as you know it from a database you have to check whether a document is already in the index *or* whether it is already pending (waiting for getting flushed to the index). Fortunately Solr manages a so called pending-set with all those documents waiting for beeing flushed to disk (Solr 3.5). We are using latest greates 4.0-SNAPSHOT code, because we want to take advantage of SolrCloud stuff. Can you give a code-pointer to where I can find the pending-set stuff? I am not sure if this is what you're asking but you should be able to get the latest data from Solr by using realtime get http://wiki.apache.org/solr/RealTimeGet Thanks a lot! I might be very usefull, if this provide 100% real time get - that is, if it gets the latest version of the document, also when neither a soft-commit nor a hard-commit has been performed since the lastest version of the document was indexed. Does it do that, or does it need a soft commit (then I believe it is only a near real time get operation)? I believe it does not require any kind of commit to happen so it should really be a real time get as the name suggests. -- Sami Siren
Re: Unique key constraint and optimistic locking (versioning)
Hi Per, Can you give a code-pointer to where I can find the pending-set stuff? Does solr use this pending-set for query responses, so that solr deliver 100% real-time search results? As of Solr 3.5 it can be found within the DirectUpdateHandler and DirectUpdateHandler2-classes. I am currently unaware of how things change in 4.0. Kind regards, Em
Re: Unique key constraint and optimistic locking (versioning)
This is a really cool feature! Thanks for pointing us in that direction! As the Quick Start says, a document does not need a commit nor a soft-commit or anything else to be available via RealTimeGet. However, regarding a versioning-system, one always has to keep in mind that an uncommited document is not guaranteed to be persisted in the index. So if you give a Duplicate-Key-Error, because there is a pending document with that key and afterwards the server goes down for any reason, you might end up without that document inside of Solr. You need a log for failover. Kind regards, Em Am 24.02.2012 11:06, schrieb Per Steffensen: Sami Siren skrev: Given that you've set a uniqueKey-field and there already exists a document with that uniqueKey, it will delete the old one and insert the new one. There is really no difference between the semantics - updates do not exist. To create a UNIQUE-constraint as you know it from a database you have to check whether a document is already in the index *or* whether it is already pending (waiting for getting flushed to the index). Fortunately Solr manages a so called pending-set with all those documents waiting for beeing flushed to disk (Solr 3.5). We are using latest greates 4.0-SNAPSHOT code, because we want to take advantage of SolrCloud stuff. Can you give a code-pointer to where I can find the pending-set stuff? I am not sure if this is what you're asking but you should be able to get the latest data from Solr by using realtime get http://wiki.apache.org/solr/RealTimeGet Thanks a lot! I might be very usefull, if this provide 100% real time get - that is, if it gets the latest version of the document, also when neither a soft-commit nor a hard-commit has been performed since the lastest version of the document was indexed. Does it do that, or does it need a soft commit (then I believe it is only a near real time get operation)? -- Sami Siren
Re: Unique key constraint and optimistic locking (versioning)
On Fri, Feb 24, 2012 at 6:55 AM, Em mailformailingli...@yahoo.de wrote: However, regarding a versioning-system, one always has to keep in mind that an uncommited document is not guaranteed to be persisted in the index. We now have durability via an update log. With a recent nightly trunk build, you can send a document to solr w/o committing, then kill -9 the JVM, then restart it and the log will be used to recover that document (and you should be able to see it in the index) -Yonik lucidimagination.com
Re: Unique key constraint and optimistic locking (versioning)
Em skrev: This is a really cool feature! Thanks for pointing us in that direction! A feature where you can flag your index operation to provide create sematics would be cool. When setting the create-semantics flag, an index operation will fail if a document with simular id (or whatever you use for uniqueKey) already exist. When the flag is not set index semantics will be just as it is today. ElasticSearch has this, except that they call it OpType which has the possible values create and index (index is default). Most other alternatives to Solr(Cloud) provide this as well. We need it in my current project. We might make it outside Solr/Lucene but I hope to be able to convince my ProductOwner to make it as a Solr-feature contributing it back - especiallly if the Solr community agrees that it would be a nice and commonly usable feature. Believe it is a commonly usable feature - especially when using Solr as a NoSQL data store and not just a search index (as http://wiki.apache.org/solr/RealTimeGet says) As the Quick Start says, a document does not need a commit nor a soft-commit or anything else to be available via RealTimeGet. However, regarding a versioning-system, one always has to keep in mind that an uncommited document is not guaranteed to be persisted in the index. So if you give a Duplicate-Key-Error, because there is a pending document with that key and afterwards the server goes down for any reason, you might end up without that document inside of Solr. You need a log for failover. Yes I know. Or you might just not consider a datarecord inserted into Solr before it has been indexed AND a hard-commit has happened. You can have many threads working on indexing datarecords into Solr but not deleting/acknowledging the source for those datarecords before next hard-commit has happend after index. But I believe it is another issue - one we also have plans about dealing with. Thanks everybody! Kind regards, Em Am 24.02.2012 11:06, schrieb Per Steffensen: Sami Siren skrev: Given that you've set a uniqueKey-field and there already exists a document with that uniqueKey, it will delete the old one and insert the new one. There is really no difference between the semantics - updates do not exist. To create a UNIQUE-constraint as you know it from a database you have to check whether a document is already in the index *or* whether it is already pending (waiting for getting flushed to the index). Fortunately Solr manages a so called pending-set with all those documents waiting for beeing flushed to disk (Solr 3.5). We are using latest greates 4.0-SNAPSHOT code, because we want to take advantage of SolrCloud stuff. Can you give a code-pointer to where I can find the pending-set stuff? I am not sure if this is what you're asking but you should be able to get the latest data from Solr by using realtime get http://wiki.apache.org/solr/RealTimeGet Thanks a lot! I might be very usefull, if this provide 100% real time get - that is, if it gets the latest version of the document, also when neither a soft-commit nor a hard-commit has been performed since the lastest version of the document was indexed. Does it do that, or does it need a soft commit (then I believe it is only a near real time get operation)? -- Sami Siren
Re: Unique key constraint and optimistic locking (versioning)
Yonik Seeley skrev: On Fri, Feb 24, 2012 at 6:55 AM, Em mailformailingli...@yahoo.de wrote: However, regarding a versioning-system, one always has to keep in mind that an uncommited document is not guaranteed to be persisted in the index. We now have durability via an update log. With a recent nightly trunk build, you can send a document to solr w/o committing, then kill -9 the JVM, then restart it and the log will be used to recover that document (and you should be able to see it in the index) Cool. We have a test doing exactly that - indexing 2000 documents into Solr, kill-9'ing Solr in the middle of the process, starting Solr again and checking that 2000 documents will eventually be searchable. It lights red as it is right now, but we are using a 4.0-SNAPSHOT from late december. We will try to update to newest code and see if it lights green :-) Do you have to do something to enable this log-and-recover feature, or does it just run out-of-the-box? Any documentation? -Yonik lucidimagination.com
Re: Unique key constraint and optimistic locking (versioning)
On Fri, Feb 24, 2012 at 9:04 AM, Per Steffensen st...@designware.dk wrote: Cool. We have a test doing exactly that - indexing 2000 documents into Solr, kill-9'ing Solr in the middle of the process, starting Solr again and checking that 2000 documents will eventually be searchable. It lights red as it is right now, but we are using a 4.0-SNAPSHOT from late december. We will try to update to newest code and see if it lights green :-) Do you have to do something to enable this log-and-recover feature, or does it just run out-of-the-box? Any documentation? Same as realtime-get, you need the update log configured. It's currently tested in TestRecovery. -Yonik lucidimagination.com
Re: Unique key constraint and optimistic locking (versioning)
Yonik Seeley skrev: On Fri, Feb 24, 2012 at 9:04 AM, Per Steffensen st...@designware.dk wrote: Cool. We have a test doing exactly that - indexing 2000 documents into Solr, kill-9'ing Solr in the middle of the process, starting Solr again and checking that 2000 documents will eventually be searchable. It lights red as it is right now, but we are using a 4.0-SNAPSHOT from late december. We will try to update to newest code and see if it lights green :-) Do you have to do something to enable this log-and-recover feature, or does it just run out-of-the-box? Any documentation? Same as realtime-get, you need the update log configured. It's currently tested in TestRecovery. Thanks! Any performance measurements comparing with and without update log? -Yonik lucidimagination.com
Re: Unique key constraint and optimistic locking (versioning)
Per Steffensen skrev: Em skrev: This is a really cool feature! Thanks for pointing us in that direction! A feature where you can flag your index operation to provide create sematics would be cool. When setting the create-semantics flag, an index operation will fail if a document with simular id (or whatever you use for uniqueKey) already exist. When the flag is not set index semantics will be just as it is today. ElasticSearch has this, except that they call it OpType which has the possible values create and index (index is default). Most other alternatives to Solr(Cloud) provide this as well. We need it in my current project. We might make it outside Solr/Lucene but I hope to be able to convince my ProductOwner to make it as a Solr-feature contributing it back - especiallly if the Solr community agrees that it would be a nice and commonly usable feature. Believe it is a commonly usable feature - especially when using Solr as a NoSQL data store and not just a search index (as http://wiki.apache.org/solr/RealTimeGet says) But of course if you guys who know Solr/Lucene source will make if for us, it will be greatly apprechiated :-)
Re: Unique key constraint and optimistic locking (versioning)
On Fri, Feb 24, 2012 at 8:59 AM, Per Steffensen st...@designware.dk wrote: We might make it outside Solr/Lucene but I hope to be able to convince my ProductOwner to make it as a Solr-feature contributing it back - especiallly if the Solr community agrees that it would be a nice and commonly usable feature. Our current distributed indexing (solr cloud) design explicitly took optimistic concurrency into account, and it's been on my todo list. It should actually be pretty easy - we have all the plumbing in place now that we already use for distributed indexing. For example, versioning is already used to handle reorders of updates when a leader forwards updates to replicas (and an older update to a document is simply dropped). If you'd like to help out, the starting point where versioning is handled is in DistributedUpdateProcessor (which is now a default processor and also works in non-distrib mode). -Yonik lucidimagination.com
Re: Unique key constraint and optimistic locking (versioning)
Hi Per, if you are evaluating with your ProductOwner whether he/she wants to contribute back: Try to not see it only as a gift to the community for a highly usefull product, but also see it as a protection of your investment. What you are going to customize will be deeply integrated in Solr - in code where changes might occur more often than in others, since they are critical to Solr's overall performance. When you are contributing back and committers rate your contribution as valuable and safe enough to commit it to the trunk, almost every enhancement of those code-regions is likely to be compatible with your changes. This highly reduces maintenance-costs, while it leaves more ressources for new innovations and features that make your product great. Kind regards, Em We might make it outside Solr/Lucene but I hope to be able to convince my ProductOwner to make it as a Solr-feature contributing it back - especiallly if the Solr community agrees that it would be a nice and commonly usable feature. Believe it is a commonly usable feature - especially when using Solr as a NoSQL data store and not just a search index (as http://wiki.apache.org/solr/RealTimeGet says)
Re: Unique key constraint and optimistic locking (versioning)
Yonik, thanks for sharing deeper details about how SolrCloud is going to work. Do you plan to release any wiki-updates about the small details, so that other developers are able to get in touch with what you've already done there? I think small guides and the mentioning of class-names and their relationships and/or lifecycles are good startpoints that do not need much time to write them down (take How to write distributed SearchComponents as an example). Kind regards, Em Am 24.02.2012 15:18, schrieb Yonik Seeley: On Fri, Feb 24, 2012 at 8:59 AM, Per Steffensen st...@designware.dk wrote: We might make it outside Solr/Lucene but I hope to be able to convince my ProductOwner to make it as a Solr-feature contributing it back - especiallly if the Solr community agrees that it would be a nice and commonly usable feature. Our current distributed indexing (solr cloud) design explicitly took optimistic concurrency into account, and it's been on my todo list. It should actually be pretty easy - we have all the plumbing in place now that we already use for distributed indexing. For example, versioning is already used to handle reorders of updates when a leader forwards updates to replicas (and an older update to a document is simply dropped). If you'd like to help out, the starting point where versioning is handled is in DistributedUpdateProcessor (which is now a default processor and also works in non-distrib mode). -Yonik lucidimagination.com
Re: Unique key constraint and optimistic locking (versioning)
On Feb 24, 2012, at 6:55 AM, Em wrote: You need a log for failover. There is a transaction log. - Mark Miller lucidimagination.com
Re: Unique key constraint and optimistic locking (versioning)
Em skrev: Hi Per, Solr provides the so called UniqueKey-field. Refer to the Wiki to learn more: http://wiki.apache.org/solr/UniqueKey Belive the uniqueKey does not enforce a unique key constraint, so that you are not allowed to create a document with an id's when an document with the same id already exists. So it is not the whole solution. Optimistic locking (versioning) ... is not provided by Solr out of the box. If you add a new document with the same UniqueKey it replaces the old one. You have to do the versioning on your own (and keep in mind concurrent updates). Kind regards, Em Am 21.02.2012 13:50, schrieb Per Steffensen: Hi Does solr/lucene provide any mechanism for unique key constraint and optimistic locking (versioning)? Unique key constraint: That a client will not succeed creating a new document in solr/lucene if a document already exists having the same value in some field (e.g. an id field). Of course implemented right, so that even though two or more threads are concurrently trying to create a new document with the same value in this field, only one of them will succeed. Optimistic locking (versioning): That a client will only succeed updating a document if this updated document is based on the version of the document currently stored in solr/lucene. Implemented in the optimistic way that clients during an update have to tell which version of the document they fetched from Solr and that they therefore have used as a starting-point for their updated document. So basically having a version field on the document that clients increase by one before sending to solr for update, and some code in Solr that only makes the update succeed if the version number of the updated document is exactly one higher than the version number of the document already stored. Of course again implemented right, so that even though two or more thrads are concurrently trying to update a document, and they all have their updated document based on the current version in solr/lucene, only one of them will succeed. Or do I have to do stuff like this myself outside solr/lucene - e.g. in the client using solr. Regards, Per Steffensen
Re: Unique key constraint and optimistic locking (versioning)
Hi Per, well, Solr has no Update-Method like a RDBMS. It is a re-insert of the whole document. Therefore a document with an existing UniqueKey marks the old document as deleted and inserts the new one. However this is not the whole story, since this constraint only works per index/SolrCore/Shard (depending on your use-case). Does this help you? Kind regards, Em Am 23.02.2012 15:34, schrieb Per Steffensen: Em skrev: Hi Per, Solr provides the so called UniqueKey-field. Refer to the Wiki to learn more: http://wiki.apache.org/solr/UniqueKey Belive the uniqueKey does not enforce a unique key constraint, so that you are not allowed to create a document with an id's when an document with the same id already exists. So it is not the whole solution. Optimistic locking (versioning) ... is not provided by Solr out of the box. If you add a new document with the same UniqueKey it replaces the old one. You have to do the versioning on your own (and keep in mind concurrent updates). Kind regards, Em Am 21.02.2012 13:50, schrieb Per Steffensen: Hi Does solr/lucene provide any mechanism for unique key constraint and optimistic locking (versioning)? Unique key constraint: That a client will not succeed creating a new document in solr/lucene if a document already exists having the same value in some field (e.g. an id field). Of course implemented right, so that even though two or more threads are concurrently trying to create a new document with the same value in this field, only one of them will succeed. Optimistic locking (versioning): That a client will only succeed updating a document if this updated document is based on the version of the document currently stored in solr/lucene. Implemented in the optimistic way that clients during an update have to tell which version of the document they fetched from Solr and that they therefore have used as a starting-point for their updated document. So basically having a version field on the document that clients increase by one before sending to solr for update, and some code in Solr that only makes the update succeed if the version number of the updated document is exactly one higher than the version number of the document already stored. Of course again implemented right, so that even though two or more thrads are concurrently trying to update a document, and they all have their updated document based on the current version in solr/lucene, only one of them will succeed. Or do I have to do stuff like this myself outside solr/lucene - e.g. in the client using solr. Regards, Per Steffensen
Re: Unique key constraint and optimistic locking (versioning)
Em skrev: Hi Per, well, Solr has no Update-Method like a RDBMS. It is a re-insert of the whole document. Therefore a document with an existing UniqueKey marks the old document as deleted and inserts the new one. Yes I understand. But it is not always what I want to acheive. I want an error to occur if a document with the same id already exists, when my intent is to INSERT a new document. When my intent is to UPDATE a document in solr/lucene I want the old document already in solr/lucene deleted and the new version of this document added (exactly as you explain). It will not be possible for solr/lucene to decide what to do unless I give it some information about my intent - whether it is INSERT or UPDATE semantics I want. I guess solr/lucene always give me INSERT sematics when a document with the same id does not already exist, and that it always give me UPDATE semantics when a document with the same id does exist? I cannot decide? However this is not the whole story, since this constraint only works per index/SolrCore/Shard (depending on your use-case). Yes I know. But with the right routing strategy based on id's I will be able to acheive what I want if the feature was just there per index/core/shard. Does this help you? Yes it helps me getting sure, that what I am looking for is not there. There is not built-in way to make solr/lucene give me an error if I try to insert a new document with an id equal to a document already in the index/core/shard. The existing document will always be updated (implemented as old deleted and new added). Correct? Kind regards, Em Regards, Per Steffensen
Re: Unique key constraint and optimistic locking (versioning)
Per: Yep, you've got it. You could write a custom update handler that queried (via TermDocs or something) for the ID when your intent was to INSERT, but it'll have to be custom work. I suppose you could query with a divide-and-conquer approach, that is query for id:(1 2 58 90... all your insert IDs) and go/no-go based on whether your return had any hits, but that supposed you have some idea whether pre-existing documents are likely. But Solr doesn't have anything like you're looking for. Best Erick On Thu, Feb 23, 2012 at 10:32 AM, Per Steffensen st...@designware.dk wrote: Em skrev: Hi Per, well, Solr has no Update-Method like a RDBMS. It is a re-insert of the whole document. Therefore a document with an existing UniqueKey marks the old document as deleted and inserts the new one. Yes I understand. But it is not always what I want to acheive. I want an error to occur if a document with the same id already exists, when my intent is to INSERT a new document. When my intent is to UPDATE a document in solr/lucene I want the old document already in solr/lucene deleted and the new version of this document added (exactly as you explain). It will not be possible for solr/lucene to decide what to do unless I give it some information about my intent - whether it is INSERT or UPDATE semantics I want. I guess solr/lucene always give me INSERT sematics when a document with the same id does not already exist, and that it always give me UPDATE semantics when a document with the same id does exist? I cannot decide? However this is not the whole story, since this constraint only works per index/SolrCore/Shard (depending on your use-case). Yes I know. But with the right routing strategy based on id's I will be able to acheive what I want if the feature was just there per index/core/shard. Does this help you? Yes it helps me getting sure, that what I am looking for is not there. There is not built-in way to make solr/lucene give me an error if I try to insert a new document with an id equal to a document already in the index/core/shard. The existing document will always be updated (implemented as old deleted and new added). Correct? Kind regards, Em Regards, Per Steffensen
Re: Unique key constraint and optimistic locking (versioning)
Hi Per, I want an error to occur if a document with the same id already exists, when my intent is to INSERT a new document. When my intent is to UPDATE a document in solr/lucene I want the old document already in solr/lucene deleted and the new version of this document added (exactly as you explain). It will not be possible for solr/lucene to decide what to do unless I give it some information about my intent - whether it is INSERT or UPDATE semantics I want. I guess solr/lucene always give me INSERT sematics when a document with the same id does not already exist, and that it always give me UPDATE semantics when a document with the same id does exist? I cannot decide? Given that you've set a uniqueKey-field and there already exists a document with that uniqueKey, it will delete the old one and insert the new one. There is really no difference between the semantics - updates do not exist. To create a UNIQUE-constraint as you know it from a database you have to check whether a document is already in the index *or* whether it is already pending (waiting for getting flushed to the index). Fortunately Solr manages a so called pending-set with all those documents waiting for beeing flushed to disk (Solr 3.5). I think you have to write your own DirectUpdateHandler to achieve what you want on the Solr-level or to extend Lucenes IndexWriter to do it on the Lucene-Level. While doing so, keep track of what is going on in the trunk and how Near-Real-Time-Search will change the current way of handling updates. There is not built-in way to make solr/lucene give me an error if I try to insert a new document with an id equal to a document already in the index/core/shard. The existing document will always be updated (implemented as old deleted and new added). Correct? Exactly. If you really want to get your hands on that topic I suggest you to learn more about Lucene's IndexWriter: http://lucene.apache.org/core/old_versioned_docs/versions/3_5_0/api/all/index.html?org/apache/lucene/index/IndexWriter.html Kind Regards, Em
Re: Unique key constraint and optimistic locking (versioning)
Thanks a lot. We will use the UniqueKey feature and build versioning ourselves. Do you think it would be a good idea if we built a versioning feature into Solr/Lucene instead of doing it outside, so that others can benefit from the feature as well? Guess contributions will be made according to http://wiki.apache.org/solr/HowToContribute. It is possible for outsiders (like us) to get a SVN branch at svn.apache.org to prepare contributions, or do we have to use our own SVN? Are there any plans migrating lucene/solr codebase to Git, which will make it easier getting a separate area to work on the code (making a Git fork), and suggest the contribution back to core lucene/solr (doing a Git pull request)? Thanks! Per Steffensen Em skrev: Hi Per, Solr provides the so called UniqueKey-field. Refer to the Wiki to learn more: http://wiki.apache.org/solr/UniqueKey Optimistic locking (versioning) ... is not provided by Solr out of the box. If you add a new document with the same UniqueKey it replaces the old one. You have to do the versioning on your own (and keep in mind concurrent updates). Kind regards, Em Am 21.02.2012 13:50, schrieb Per Steffensen: Hi Does solr/lucene provide any mechanism for unique key constraint and optimistic locking (versioning)? Unique key constraint: That a client will not succeed creating a new document in solr/lucene if a document already exists having the same value in some field (e.g. an id field). Of course implemented right, so that even though two or more threads are concurrently trying to create a new document with the same value in this field, only one of them will succeed. Optimistic locking (versioning): That a client will only succeed updating a document if this updated document is based on the version of the document currently stored in solr/lucene. Implemented in the optimistic way that clients during an update have to tell which version of the document they fetched from Solr and that they therefore have used as a starting-point for their updated document. So basically having a version field on the document that clients increase by one before sending to solr for update, and some code in Solr that only makes the update succeed if the version number of the updated document is exactly one higher than the version number of the document already stored. Of course again implemented right, so that even though two or more thrads are concurrently trying to update a document, and they all have their updated document based on the current version in solr/lucene, only one of them will succeed. Or do I have to do stuff like this myself outside solr/lucene - e.g. in the client using solr. Regards, Per Steffensen
Re: Unique key constraint and optimistic locking (versioning)
Per Steffensen skrev: Thanks a lot. We will use the UniqueKey feature and build versioning ourselves. Do you think it would be a good idea if we built a versioning feature into Solr/Lucene instead of doing it outside, so that others can benefit from the feature as well? Guess contributions will be made according to http://wiki.apache.org/solr/HowToContribute. It is possible for outsiders (like us) to get a SVN branch at svn.apache.org to prepare contributions, or do we have to use our own SVN? Are there any plans migrating lucene/solr codebase to Git, which will make it easier getting a separate area to work on the code (making a Git fork), and suggest the contribution back to core lucene/solr (doing a Git pull request)? Sorry - didnt see the Eclipse (using Git) chapter on http://wiki.apache.org/solr/HowToContribute. We might contribute in that area. Thanks! Per Steffensen
Unique key constraint and optimistic locking (versioning)
Hi Does solr/lucene provide any mechanism for unique key constraint and optimistic locking (versioning)? Unique key constraint: That a client will not succeed creating a new document in solr/lucene if a document already exists having the same value in some field (e.g. an id field). Of course implemented right, so that even though two or more threads are concurrently trying to create a new document with the same value in this field, only one of them will succeed. Optimistic locking (versioning): That a client will only succeed updating a document if this updated document is based on the version of the document currently stored in solr/lucene. Implemented in the optimistic way that clients during an update have to tell which version of the document they fetched from Solr and that they therefore have used as a starting-point for their updated document. So basically having a version field on the document that clients increase by one before sending to solr for update, and some code in Solr that only makes the update succeed if the version number of the updated document is exactly one higher than the version number of the document already stored. Of course again implemented right, so that even though two or more thrads are concurrently trying to update a document, and they all have their updated document based on the current version in solr/lucene, only one of them will succeed. Or do I have to do stuff like this myself outside solr/lucene - e.g. in the client using solr. Regards, Per Steffensen
Re: Unique key constraint and optimistic locking (versioning)
Hi Per, Solr provides the so called UniqueKey-field. Refer to the Wiki to learn more: http://wiki.apache.org/solr/UniqueKey Optimistic locking (versioning) ... is not provided by Solr out of the box. If you add a new document with the same UniqueKey it replaces the old one. You have to do the versioning on your own (and keep in mind concurrent updates). Kind regards, Em Am 21.02.2012 13:50, schrieb Per Steffensen: Hi Does solr/lucene provide any mechanism for unique key constraint and optimistic locking (versioning)? Unique key constraint: That a client will not succeed creating a new document in solr/lucene if a document already exists having the same value in some field (e.g. an id field). Of course implemented right, so that even though two or more threads are concurrently trying to create a new document with the same value in this field, only one of them will succeed. Optimistic locking (versioning): That a client will only succeed updating a document if this updated document is based on the version of the document currently stored in solr/lucene. Implemented in the optimistic way that clients during an update have to tell which version of the document they fetched from Solr and that they therefore have used as a starting-point for their updated document. So basically having a version field on the document that clients increase by one before sending to solr for update, and some code in Solr that only makes the update succeed if the version number of the updated document is exactly one higher than the version number of the document already stored. Of course again implemented right, so that even though two or more thrads are concurrently trying to update a document, and they all have their updated document based on the current version in solr/lucene, only one of them will succeed. Or do I have to do stuff like this myself outside solr/lucene - e.g. in the client using solr. Regards, Per Steffensen
ideas for versioning query?
A customer has an interesting problem: some documents will have multiple versions. In search results, only the most recent version of a given document should be shown. The trick is that each user has access to a different set of document versions, and each user should see only the most recent version of a document that they have access to. Is this something that can reasonably be solved with grouping? In 3.x? I haven't followed the grouping discussions closely: would someone point me in the right direction please? -- Michael Sokolov Engineering Director www.ifactory.com
Re: ideas for versioning query?
Hi Michael, I guess this could be solved using grouping as you said. Documents inside a group can be sorted on a field (in your case, the version field, see parameter group.sort), and you can show only the first one. It will be more complex to show facets (post grouping faceting is work in progress but still not committed to the trunk). I would be easier from the Solr side if you could do something at index time, like indicating which document is the current one and which one is an old one (you would need to update the old document whenever a new version is indexed). Regards, Tomás On Mon, Aug 1, 2011 at 10:47 AM, Mike Sokolov soko...@ifactory.com wrote: A customer has an interesting problem: some documents will have multiple versions. In search results, only the most recent version of a given document should be shown. The trick is that each user has access to a different set of document versions, and each user should see only the most recent version of a document that they have access to. Is this something that can reasonably be solved with grouping? In 3.x? I haven't followed the grouping discussions closely: would someone point me in the right direction please? -- Michael Sokolov Engineering Director www.ifactory.com
Re: ideas for versioning query?
Thanks, Tomas. Yes we are planning to keep a current flag in the most current document. But there are cases where, for a given user, the most current document is not that one, because they only have access to some older documents. I took a look at http://wiki.apache.org/solr/FieldCollapsing and it seems as if it will do what we need here. My one concern is that it might not be efficient at computing group.ngroups for a very large number of groups, which we would ideally want. Is that something I should be worried about? -Mike On 08/01/2011 10:08 AM, Tomás Fernández Löbbe wrote: Hi Michael, I guess this could be solved using grouping as you said. Documents inside a group can be sorted on a field (in your case, the version field, see parameter group.sort), and you can show only the first one. It will be more complex to show facets (post grouping faceting is work in progress but still not committed to the trunk). I would be easier from the Solr side if you could do something at index time, like indicating which document is the current one and which one is an old one (you would need to update the old document whenever a new version is indexed). Regards, Tomás On Mon, Aug 1, 2011 at 10:47 AM, Mike Sokolovsoko...@ifactory.com wrote: A customer has an interesting problem: some documents will have multiple versions. In search results, only the most recent version of a given document should be shown. The trick is that each user has access to a different set of document versions, and each user should see only the most recent version of a document that they have access to. Is this something that can reasonably be solved with grouping? In 3.x? I haven't followed the grouping discussions closely: would someone point me in the right direction please? -- Michael Sokolov Engineering Director www.ifactory.com
Re: ideas for versioning query?
Hi Mike, how many docs and groups do you have in your index? I think the group.sort option fits your requirements. If I remember correctly group.ngroup=true adds something like 30% extra time on top of the search request with grouping, but that was on my local test dataset (~30M docs, ~8000 groups) and my machine. You might encounter different search times when setting group.ngroup=true. Martijn 2011/8/1 Mike Sokolov soko...@ifactory.com Thanks, Tomas. Yes we are planning to keep a current flag in the most current document. But there are cases where, for a given user, the most current document is not that one, because they only have access to some older documents. I took a look at http://wiki.apache.org/solr/**FieldCollapsinghttp://wiki.apache.org/solr/FieldCollapsingand it seems as if it will do what we need here. My one concern is that it might not be efficient at computing group.ngroups for a very large number of groups, which we would ideally want. Is that something I should be worried about? -Mike On 08/01/2011 10:08 AM, Tomás Fernández Löbbe wrote: Hi Michael, I guess this could be solved using grouping as you said. Documents inside a group can be sorted on a field (in your case, the version field, see parameter group.sort), and you can show only the first one. It will be more complex to show facets (post grouping faceting is work in progress but still not committed to the trunk). I would be easier from the Solr side if you could do something at index time, like indicating which document is the current one and which one is an old one (you would need to update the old document whenever a new version is indexed). Regards, Tomás On Mon, Aug 1, 2011 at 10:47 AM, Mike Sokolovsoko...@ifactory.com wrote: A customer has an interesting problem: some documents will have multiple versions. In search results, only the most recent version of a given document should be shown. The trick is that each user has access to a different set of document versions, and each user should see only the most recent version of a document that they have access to. Is this something that can reasonably be solved with grouping? In 3.x? I haven't followed the grouping discussions closely: would someone point me in the right direction please? -- Michael Sokolov Engineering Director www.ifactory.com -- Met vriendelijke groet, Martijn van Groningen
Re: ideas for versioning query?
I think a 30% increase is acceptable. Yes, I think we'll try it. Although our case is more like # groups ~ # documents / N, where N is a smallish number (~1-5?). We are planning for a variety of different index sizes, but aiming for a sweet spot around a few M docs. -Mike On 08/01/2011 11:00 AM, Martijn v Groningen wrote: Hi Mike, how many docs and groups do you have in your index? I think the group.sort option fits your requirements. If I remember correctly group.ngroup=true adds something like 30% extra time on top of the search request with grouping, but that was on my local test dataset (~30M docs, ~8000 groups) and my machine. You might encounter different search times when setting group.ngroup=true. Martijn 2011/8/1 Mike Sokolovsoko...@ifactory.com Thanks, Tomas. Yes we are planning to keep a current flag in the most current document. But there are cases where, for a given user, the most current document is not that one, because they only have access to some older documents. I took a look at http://wiki.apache.org/solr/**FieldCollapsinghttp://wiki.apache.org/solr/FieldCollapsingand it seems as if it will do what we need here. My one concern is that it might not be efficient at computing group.ngroups for a very large number of groups, which we would ideally want. Is that something I should be worried about? -Mike On 08/01/2011 10:08 AM, Tomás Fernández Löbbe wrote: Hi Michael, I guess this could be solved using grouping as you said. Documents inside a group can be sorted on a field (in your case, the version field, see parameter group.sort), and you can show only the first one. It will be more complex to show facets (post grouping faceting is work in progress but still not committed to the trunk). I would be easier from the Solr side if you could do something at index time, like indicating which document is the current one and which one is an old one (you would need to update the old document whenever a new version is indexed). Regards, Tomás On Mon, Aug 1, 2011 at 10:47 AM, Mike Sokolovsoko...@ifactory.com wrote: A customer has an interesting problem: some documents will have multiple versions. In search results, only the most recent version of a given document should be shown. The trick is that each user has access to a different set of document versions, and each user should see only the most recent version of a document that they have access to. Is this something that can reasonably be solved with grouping? In 3.x? I haven't followed the grouping discussions closely: would someone point me in the right direction please? -- Michael Sokolov Engineering Director www.ifactory.com
Re: Solr versioning policy
: 1. Is this the plan moving forward (to aim for a new minor release : approximately every couple of months)? The goal is to release minor versions more frequently as features and low priority bug fixes are available. If there is a high priority bug fix available, and and no likelihood of a near-term minor release, then bug fixes releases (ie: 3.4.1) will be done (as has always been the case) This new accelerated minor-feature release approach is possible because of the parallel development branches approach that was instituted a while back, but once those branches were created it took some time to get the test/build/release processes automated enough that devs felt formortable releasing more frequently. There's no hard and fast rule about often releases will happen. Anyone can step up and push for a release if they feel the features are ready. : 2. Will minor version increases always be backwards compatible (so I could : upgrade from 3.x to 3.y where y x without having to update the : schema/config or rebuild the indexes)? That has always been the goal, yes. Sometimes the mechanism for dealing with new bugs/features requires making changes to config files and when known this is noted in the Upgrading section of CHANGES.txt for the affected release. -Hoss
Solr versioning policy
Hi, I've noticed that since the 3.1 release new minor version releases have been happening about every two months. I have a couple of questions: 1. Is this the plan moving forward (to aim for a new minor release approximately every couple of months)? 2. Will minor version increases always be backwards compatible (so I could upgrade from 3.x to 3.y where y x without having to update the schema/config or rebuild the indexes)? It might be worth sticking something up on the wiki which gives an overview of the versioning policy just to clarify things. (I had a look and couldn't find anything.) Cheers, Mike.
RE: Lucene versioning policy
Any update on possible graduation from the incubator? Any chance it could coincide with Hoss's presentation at ApacheCon? -D -Original Message- From: Chris Hostetter [mailto:[EMAIL PROTECTED] Sent: Thursday, June 08, 2006 11:48 PM To: solr-user@lucene.apache.org Subject: Re: Lucene versioning policy : Also, are there any plans to split solr into a release/development mode? : : I'd really like to use solr in a commercial setting, but having nothing but : nightly builds available makes me uneasy. I believe that as long as Solr is in the incubator, nightly builds are the only releases we are allowed to have. This is a side note in the incubation policy about exiting incubation... Note: incubator projects are not permitted to issue an official Release. Test snapshots (however good the quality) and Release plans are OK. ...of course, there is some conflicting info higher up in the same doc that suggests they are allowed, but they require jumping through some hoops... http://incubator.apache.org/incubation/Incubation_Policy.html#Releases -Hoss
Re: Lucene versioning policy
: http://svn.apache.org/viewvc/incubator/solr/trunk/CHANGES.txt : : But it is imperfect... Perhaps an entry should be added when updating : the Lucene version too. +1 ... definitely. -Hoss
Re: Lucene versioning policy
: Also, are there any plans to split solr into a release/development mode? : : I'd really like to use solr in a commercial setting, but having nothing but : nightly builds available makes me uneasy. I believe that as long as Solr is in the incubator, nightly builds are the only releases we are allowed to have. This is a side note in the incubation policy about exiting incubation... Note: incubator projects are not permitted to issue an official Release. Test snapshots (however good the quality) and Release plans are OK. ...of course, there is some conflicting info higher up in the same doc that suggests they are allowed, but they require jumping through some hoops... http://incubator.apache.org/incubation/Incubation_Policy.html#Releases -Hoss
