Re: Unique key constraint and optimistic locking (versioning)

2012-02-29 Thread Per Steffensen
Created SOLR-3178 covering the versioning/optimistic-locking part. In combination SOLR-3173 and SOLR-3178 should provide the features I am missing, and that I believe lots of other SOLR users will be able to benefit from. Please help shape by commenting on the Jira issues. Thanks. Per

Re: Unique key constraint and optimistic locking (versioning)

2012-02-28 Thread Per Steffensen
Created SOLR-3173 on the part about making insert fail if document (with same uniqueKey) already exists. SOLR-3173 also includes to make update not insert document if not already exists - just for consistency with normal RDBMS behaviour. So basically the feature allowes you to turn on this

Re: Unique key constraint and optimistic locking (versioning)

2012-02-24 Thread Per Steffensen
Em skrev: Hi Per, I want an error to occur if a document with the same id already exists, when my intent is to INSERT a new document. When my intent is to UPDATE a document in solr/lucene I want the old document already in solr/lucene deleted and the new version of this document added

Re: Unique key constraint and optimistic locking (versioning)

2012-02-24 Thread Sami Siren
Given that you've set a uniqueKey-field and there already exists a document with that uniqueKey, it will delete the old one and insert the new one. There is really no difference between the semantics - updates do not exist. To create a UNIQUE-constraint as you know it from a database you have

Re: Unique key constraint and optimistic locking (versioning)

2012-02-24 Thread Per Steffensen
Sami Siren skrev: Given that you've set a uniqueKey-field and there already exists a document with that uniqueKey, it will delete the old one and insert the new one. There is really no difference between the semantics - updates do not exist. To create a UNIQUE-constraint as you know it from a

Re: Unique key constraint and optimistic locking (versioning)

2012-02-24 Thread Sami Siren
On Fri, Feb 24, 2012 at 12:06 PM, Per Steffensen st...@designware.dk wrote: Sami Siren skrev: Given that you've set a uniqueKey-field and there already exists a document with that uniqueKey, it will delete the old one and insert the new one. There is really no difference between the semantics

Re: Unique key constraint and optimistic locking (versioning)

2012-02-24 Thread Em
Hi Per, Can you give a code-pointer to where I can find the pending-set stuff? Does solr use this pending-set for query responses, so that solr deliver 100% real-time search results? As of Solr 3.5 it can be found within the DirectUpdateHandler and DirectUpdateHandler2-classes. I am currently

Re: Unique key constraint and optimistic locking (versioning)

2012-02-24 Thread Em
This is a really cool feature! Thanks for pointing us in that direction! As the Quick Start says, a document does not need a commit nor a soft-commit or anything else to be available via RealTimeGet. However, regarding a versioning-system, one always has to keep in mind that an uncommited

Re: Unique key constraint and optimistic locking (versioning)

2012-02-24 Thread Yonik Seeley
On Fri, Feb 24, 2012 at 6:55 AM, Em mailformailingli...@yahoo.de wrote: However, regarding a versioning-system, one always has to keep in mind that an uncommited document is not guaranteed to be persisted in the index. We now have durability via an update log. With a recent nightly trunk build,

Re: Unique key constraint and optimistic locking (versioning)

2012-02-24 Thread Per Steffensen
Em skrev: This is a really cool feature! Thanks for pointing us in that direction! A feature where you can flag your index operation to provide create sematics would be cool. When setting the create-semantics flag, an index operation will fail if a document with simular id (or whatever you

Re: Unique key constraint and optimistic locking (versioning)

2012-02-24 Thread Per Steffensen
Yonik Seeley skrev: On Fri, Feb 24, 2012 at 6:55 AM, Em mailformailingli...@yahoo.de wrote: However, regarding a versioning-system, one always has to keep in mind that an uncommited document is not guaranteed to be persisted in the index. We now have durability via an update log. With

Re: Unique key constraint and optimistic locking (versioning)

2012-02-24 Thread Yonik Seeley
On Fri, Feb 24, 2012 at 9:04 AM, Per Steffensen st...@designware.dk wrote: Cool. We have a test doing exactly that - indexing 2000 documents into Solr, kill-9'ing Solr in the middle of the process, starting Solr again and checking that 2000 documents will eventually be searchable. It lights red

Re: Unique key constraint and optimistic locking (versioning)

2012-02-24 Thread Per Steffensen
Yonik Seeley skrev: On Fri, Feb 24, 2012 at 9:04 AM, Per Steffensen st...@designware.dk wrote: Cool. We have a test doing exactly that - indexing 2000 documents into Solr, kill-9'ing Solr in the middle of the process, starting Solr again and checking that 2000 documents will eventually be

Re: Unique key constraint and optimistic locking (versioning)

2012-02-24 Thread Per Steffensen
Per Steffensen skrev: Em skrev: This is a really cool feature! Thanks for pointing us in that direction! A feature where you can flag your index operation to provide create sematics would be cool. When setting the create-semantics flag, an index operation will fail if a document with

Re: Unique key constraint and optimistic locking (versioning)

2012-02-24 Thread Yonik Seeley
On Fri, Feb 24, 2012 at 8:59 AM, Per Steffensen st...@designware.dk wrote: We might make it outside Solr/Lucene but I hope to be able to convince my ProductOwner to make it as a Solr-feature contributing it back - especiallly if the Solr community agrees that it would be a nice and commonly

Re: Unique key constraint and optimistic locking (versioning)

2012-02-24 Thread Em
Hi Per, if you are evaluating with your ProductOwner whether he/she wants to contribute back: Try to not see it only as a gift to the community for a highly usefull product, but also see it as a protection of your investment. What you are going to customize will be deeply integrated in Solr - in

Re: Unique key constraint and optimistic locking (versioning)

2012-02-24 Thread Em
Yonik, thanks for sharing deeper details about how SolrCloud is going to work. Do you plan to release any wiki-updates about the small details, so that other developers are able to get in touch with what you've already done there? I think small guides and the mentioning of class-names and their

Re: Unique key constraint and optimistic locking (versioning)

2012-02-24 Thread Mark Miller
On Feb 24, 2012, at 6:55 AM, Em wrote: You need a log for failover. There is a transaction log. - Mark Miller lucidimagination.com

Re: Unique key constraint and optimistic locking (versioning)

2012-02-23 Thread Per Steffensen
Em skrev: Hi Per, Solr provides the so called UniqueKey-field. Refer to the Wiki to learn more: http://wiki.apache.org/solr/UniqueKey Belive the uniqueKey does not enforce a unique key constraint, so that you are not allowed to create a document with an id's when an document with the same

Re: Unique key constraint and optimistic locking (versioning)

2012-02-23 Thread Em
Hi Per, well, Solr has no Update-Method like a RDBMS. It is a re-insert of the whole document. Therefore a document with an existing UniqueKey marks the old document as deleted and inserts the new one. However this is not the whole story, since this constraint only works per index/SolrCore/Shard

Re: Unique key constraint and optimistic locking (versioning)

2012-02-23 Thread Per Steffensen
Em skrev: Hi Per, well, Solr has no Update-Method like a RDBMS. It is a re-insert of the whole document. Therefore a document with an existing UniqueKey marks the old document as deleted and inserts the new one. Yes I understand. But it is not always what I want to acheive. I want an error

Re: Unique key constraint and optimistic locking (versioning)

2012-02-23 Thread Erick Erickson
Per: Yep, you've got it. You could write a custom update handler that queried (via TermDocs or something) for the ID when your intent was to INSERT, but it'll have to be custom work. I suppose you could query with a divide-and-conquer approach, that is query for id:(1 2 58 90... all your insert

Re: Unique key constraint and optimistic locking (versioning)

2012-02-23 Thread Em
Hi Per, I want an error to occur if a document with the same id already exists, when my intent is to INSERT a new document. When my intent is to UPDATE a document in solr/lucene I want the old document already in solr/lucene deleted and the new version of this document added (exactly as you

Re: Unique key constraint and optimistic locking (versioning)

2012-02-22 Thread Per Steffensen
Thanks a lot. We will use the UniqueKey feature and build versioning ourselves. Do you think it would be a good idea if we built a versioning feature into Solr/Lucene instead of doing it outside, so that others can benefit from the feature as well? Guess contributions will be made according to

Re: Unique key constraint and optimistic locking (versioning)

2012-02-22 Thread Per Steffensen
Per Steffensen skrev: Thanks a lot. We will use the UniqueKey feature and build versioning ourselves. Do you think it would be a good idea if we built a versioning feature into Solr/Lucene instead of doing it outside, so that others can benefit from the feature as well? Guess contributions

Re: Unique key constraint and optimistic locking (versioning)

2012-02-21 Thread Em
Hi Per, Solr provides the so called UniqueKey-field. Refer to the Wiki to learn more: http://wiki.apache.org/solr/UniqueKey Optimistic locking (versioning) ... is not provided by Solr out of the box. If you add a new document with the same UniqueKey it replaces the old one. You have to do the