[jira] [Updated] (SOLR-3178) Versioning - optimistic locking
[ https://issues.apache.org/jira/browse/SOLR-3178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Miller updated SOLR-3178: -- Fix Version/s: (was: 4.1) Versioning - optimistic locking --- Key: SOLR-3178 URL: https://issues.apache.org/jira/browse/SOLR-3178 Project: Solr Issue Type: New Feature Components: update Affects Versions: 3.5 Environment: All Reporter: Per Steffensen Assignee: Per Steffensen Labels: RDBMS, insert, locking, nosql, optimistic, uniqueKey, update, versioning Attachments: SOLR-3173_3178_3382_3428_plus.patch, SOLR-3173_3178_3382_3428_plus.patch, SOLR_3173_3178_3382_plus.patch, SOLR-3178.patch Original Estimate: 168h Remaining Estimate: 168h In order increase the ability of Solr to be used as a NoSql database (lots of concurrent inserts, updates, deletes and queries in the entire lifetime of the index) instead of just a search index (first: everything indexed (in one thread), after: only queries), I would like Solr to support versioning to be used for optimistic locking. When my intent (see SOLR-3173) is to update an existing document, I will need to provide a version-number equal to the version number I got when I fetched the existing document for update plus one. If this provided version-number does not correspond to the newest version-number of that document at the time of update plus one, I will get a VersionConflict error. If it does correspond the document will be updated with the new one, so that the newest version-number of that document is NOW one higher than before the update. Correct but efficient concurrency handling. When my intent (see SOLR-3173) is to insert a new document, the version number provided will not be used - instead a version-number 0 will be used. According to SOLR-3173 insert will only succeed if a document with the same value on uniqueKey-field does not already exist. In general when talking about different versions of the same document, of course we need to be able to identify when a document is the same - that, per definition, is when the values of the uniqueKey-fields are equal. The functionality provided by this issue is only really meaningfull when you run with updateLog activated. This issue might be solved more or less at the same time as SOLR-3173, and only one single SVN patch might be given to cover both issues. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-3178) Versioning - optimistic locking
[ https://issues.apache.org/jira/browse/SOLR-3178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Per Steffensen updated SOLR-3178: - Attachment: SOLR-3173_3178_3382_3428_plus.patch Updated patch SOLR-3173_3178_3382_3428_plus.patch to fit on top of Solr 4.0.0. Actually updated to fit on top of revision 1394844 of branch lucene_solr_4_0, but I believe this is the same as Solr 4.0.0. Versioning - optimistic locking --- Key: SOLR-3178 URL: https://issues.apache.org/jira/browse/SOLR-3178 Project: Solr Issue Type: New Feature Components: update Affects Versions: 3.5 Environment: All Reporter: Per Steffensen Assignee: Per Steffensen Labels: RDBMS, insert, locking, nosql, optimistic, uniqueKey, update, versioning Fix For: 4.1 Attachments: SOLR-3173_3178_3382_3428_plus.patch, SOLR-3173_3178_3382_3428_plus.patch, SOLR_3173_3178_3382_plus.patch, SOLR-3178.patch Original Estimate: 168h Remaining Estimate: 168h In order increase the ability of Solr to be used as a NoSql database (lots of concurrent inserts, updates, deletes and queries in the entire lifetime of the index) instead of just a search index (first: everything indexed (in one thread), after: only queries), I would like Solr to support versioning to be used for optimistic locking. When my intent (see SOLR-3173) is to update an existing document, I will need to provide a version-number equal to the version number I got when I fetched the existing document for update plus one. If this provided version-number does not correspond to the newest version-number of that document at the time of update plus one, I will get a VersionConflict error. If it does correspond the document will be updated with the new one, so that the newest version-number of that document is NOW one higher than before the update. Correct but efficient concurrency handling. When my intent (see SOLR-3173) is to insert a new document, the version number provided will not be used - instead a version-number 0 will be used. According to SOLR-3173 insert will only succeed if a document with the same value on uniqueKey-field does not already exist. In general when talking about different versions of the same document, of course we need to be able to identify when a document is the same - that, per definition, is when the values of the uniqueKey-fields are equal. The functionality provided by this issue is only really meaningfull when you run with updateLog activated. This issue might be solved more or less at the same time as SOLR-3173, and only one single SVN patch might be given to cover both issues. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-3178) Versioning - optimistic locking
[ https://issues.apache.org/jira/browse/SOLR-3178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Muir updated SOLR-3178: -- Fix Version/s: (was: 4.0) 4.1 Versioning - optimistic locking --- Key: SOLR-3178 URL: https://issues.apache.org/jira/browse/SOLR-3178 Project: Solr Issue Type: New Feature Components: update Affects Versions: 3.5 Environment: All Reporter: Per Steffensen Assignee: Per Steffensen Labels: RDBMS, insert, locking, nosql, optimistic, uniqueKey, update, versioning Fix For: 4.1 Attachments: SOLR-3173_3178_3382_3428_plus.patch, SOLR_3173_3178_3382_plus.patch, SOLR-3178.patch Original Estimate: 168h Remaining Estimate: 168h In order increase the ability of Solr to be used as a NoSql database (lots of concurrent inserts, updates, deletes and queries in the entire lifetime of the index) instead of just a search index (first: everything indexed (in one thread), after: only queries), I would like Solr to support versioning to be used for optimistic locking. When my intent (see SOLR-3173) is to update an existing document, I will need to provide a version-number equal to the version number I got when I fetched the existing document for update plus one. If this provided version-number does not correspond to the newest version-number of that document at the time of update plus one, I will get a VersionConflict error. If it does correspond the document will be updated with the new one, so that the newest version-number of that document is NOW one higher than before the update. Correct but efficient concurrency handling. When my intent (see SOLR-3173) is to insert a new document, the version number provided will not be used - instead a version-number 0 will be used. According to SOLR-3173 insert will only succeed if a document with the same value on uniqueKey-field does not already exist. In general when talking about different versions of the same document, of course we need to be able to identify when a document is the same - that, per definition, is when the values of the uniqueKey-fields are equal. The functionality provided by this issue is only really meaningfull when you run with updateLog activated. This issue might be solved more or less at the same time as SOLR-3173, and only one single SVN patch might be given to cover both issues. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-3178) Versioning - optimistic locking
[ https://issues.apache.org/jira/browse/SOLR-3178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Per Steffensen updated SOLR-3178: - Attachment: SOLR-3173_3178_3382_3428_plus.patch SOLR-3173_3178_3382_3428_plus.patch updated to fit on top of revision 1355667 on 4.x branch. I have tried to describe for (almost) all files in the patch, which problem the changes in them solve. Please see below. A lot of files has changed, but it is very small changes in most of the files basically just making sure SolrRequestInfo-threadlocal is set in tests and other stuff like that. The solutions to SOLR-3173 and SOLR-3178 are not very big. Actually much more code has been done to solve SOLR-3382. With those descriptions I hope to convince you to do the commit. Trust you test-suite - it has not been changed, except that additional asserts has been added. I am really not religious about whether the version-control code is in DistributedUpdateProcessor or DirectUpdateHandler2, but I am religious about being backward compatible wrt such fundamental changes in semantics, and I am religious about getting typed errors/exceptions back in responses, so that I can react properly. I do believe version-control belong in DirectUpdateHandler2, but if you want it in DistributedUpdateProcessor, basically just refactor (after commit) and move the changes in DirectUpdateHandler2 to DistributedUpdateHandler. * Solved SOLR-3173 and SOLR-3178 (implementation of fail-on-unique-key-violation and version-check), by introducing 3 modes Solr server can run in (controlled by semanticsMode in solrconfig.xml | updateHandler) - one for backward compatibility, one for a very strict version control, and a hybrid (default). ** solr/core/src/java/org/apache/solr/update/DirectUpdateHandler2.java (the new features are controlled from here, but basically using the rules in UpdateSemanticsMode. Got rid of duplicate code in constructors - g) ** solr/core/src/java/org/apache/solr/update/UpdateSemanticsMode.java (enum that encapsulates the essence in the 3 possible semanticsModes. Used by DirectUpdateHandler2. Nice separation of concern) ** solr/core/src/java/org/apache/solr/update/UpdateCommand.java (added leaderLogic calculated (as isLeader) in DistributedUpdateProcesser to UpdateCommand so that it is available in DirectUpdateHandler2, where it is used to know if the new checks must be performed or not. UpdateCommand also carrying requestVersion, the version sent in the request) ** solr/core/src/java/org/apache/solr/core/SolrConfig.java (control via configuration which semantics-mode to use) ** solr/core/src/java/org/apache/solr/update/processor/DistributedUpdateProcessor.java (maintains leaderLogic on UpdateCommand) ** solr/core/src/java/org/apache/solr/update/SolrCmdDistributor.java (maintains requestVersion on UpdateCommand) ** solr/core/src/java/org/apache/solr/update/UpdateLog.java (maintains requestVersion on UpdateCommand) ** solr/core/src/java/org/apache/solr/update/PeerSync.java (maintains leaderLogic and requestVersion on UpdateCommand) ** solr/core/src/test-files/solr/collection1/conf/solrconfig-classic-semantics.xml (default test solrconfig.xml but with classic semantics-mode, used to test that the new features fail-on-unique-key-violation and version-check are not activated when running classic semantics - the backward compatible mode) ** solr/core/src/test-files/solr/collection1/conf/solrconfig-classic-consistency-hybrid-semantics.xml (default test solrconfig.xml but with classic-consistency-hybrid semantics-mode. Actually default solrconfig.xml could have been used since classic-consistency-hybrid semantics is default, but used to test that it (also) works when this mode is selected explicitly) ** solr/test-framework/src/java/org/apache/solr/SolrTestCaseJ4.java and solr/test-framework/src/java/org/apache/solr/util/AbstractSolrTestCase.java and solr/test-framework/src/java/org/apache/solr/util/TestHarness.java (basically just a lot of changes making testing easier) * Removed the (temporary and poor IMHO) implementation of fail-on-unique-key-violation and version-check, but kept the introduced tests, so you can see that the same functionality is still provided (on default semantics classic-consistency-hybrid) ** solr/core/src/java/org/apache/solr/update/processor/DistributedUpdateProcessor.java ** solr/core/src/test/org/apache/solr/search/TestRealTimeGet.java (now we do not only get exceptions in bad situations, we get typed exceptions telling us what is wrong) * Solved SOLR-3382 - partRef and PartialError(s) introduced in multi-document-update-requests so that they can be paired up with errors in responses, so that the client will know that document-updates failed with what errors and therefore also which documents where updated successfully ** solr/solrj/src/java/org/apache/solr/common/RequestPart.java (new generic
[jira] [Updated] (SOLR-3178) Versioning - optimistic locking
[ https://issues.apache.org/jira/browse/SOLR-3178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Per Steffensen updated SOLR-3178: - Attachment: (was: SOLR-3173_3178_3382_3428_plus.patch) Versioning - optimistic locking --- Key: SOLR-3178 URL: https://issues.apache.org/jira/browse/SOLR-3178 Project: Solr Issue Type: New Feature Components: update Affects Versions: 3.5 Environment: All Reporter: Per Steffensen Assignee: Per Steffensen Labels: RDBMS, insert, locking, nosql, optimistic, uniqueKey, update, versioning Fix For: 4.0 Attachments: SOLR-3173_3178_3382_3428_plus.patch, SOLR-3178.patch, SOLR_3173_3178_3382_plus.patch Original Estimate: 168h Remaining Estimate: 168h In order increase the ability of Solr to be used as a NoSql database (lots of concurrent inserts, updates, deletes and queries in the entire lifetime of the index) instead of just a search index (first: everything indexed (in one thread), after: only queries), I would like Solr to support versioning to be used for optimistic locking. When my intent (see SOLR-3173) is to update an existing document, I will need to provide a version-number equal to the version number I got when I fetched the existing document for update plus one. If this provided version-number does not correspond to the newest version-number of that document at the time of update plus one, I will get a VersionConflict error. If it does correspond the document will be updated with the new one, so that the newest version-number of that document is NOW one higher than before the update. Correct but efficient concurrency handling. When my intent (see SOLR-3173) is to insert a new document, the version number provided will not be used - instead a version-number 0 will be used. According to SOLR-3173 insert will only succeed if a document with the same value on uniqueKey-field does not already exist. In general when talking about different versions of the same document, of course we need to be able to identify when a document is the same - that, per definition, is when the values of the uniqueKey-fields are equal. The functionality provided by this issue is only really meaningfull when you run with updateLog activated. This issue might be solved more or less at the same time as SOLR-3173, and only one single SVN patch might be given to cover both issues. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-3178) Versioning - optimistic locking
[ https://issues.apache.org/jira/browse/SOLR-3178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Per Steffensen updated SOLR-3178: - Attachment: SOLR-3173_3178_3382_3428_plus.patch Find attached SOLR-3173_3178_3382_3428_plus.patch The patch fits on top of trunk revision 1332666 and is ready for commit. Since last patch (SOLR_3173_3178_3382_plus.patch) I have made the following: * Implemented test ClassicConsistencyHybridUpdateSemanticsSolrCloudTest verifying that partial errors are propagated correctly in a cloud (ZK) setup, when an update document is not sent directly to the Solr instance running the leader of the slice where the document is to be stored, and the Solr instance therefore has to forward the document to the leader, and when the leader forwards to replica. This made me discover the problems discribed in SOLR-3428, which I had to fix in order to make the test green. * Implemented tests (in JsonLoaderTest and CSVRequestHandlerTest) verifying that you can send part references as claimed on http://wiki.apache.org/solr/Per%20Steffensen/Update%20semantics when you are sending JSON or CSV requests * Documents for which a partial error occur on leader will not be forwarded to replica. * Partial errors are now catched and collected for the entire request in DistributedUpdateProcessor, but the actual version-check etc is still performed in DirectUpdateHandler2 (because that is an per-document-level task) * Added Apache License 2.0 text to all new files * Added class-level-JavaDoc to all new classes * Removed a few unimportant changes I will be leaving for an extended weekend now, and will not be available again before wedensday next week. I will probably read and answer comments from time to time until saturday night, though. I really really really hope to return to the great news about the patch having been committed. I believe it is now really ready - well worked through features, well covered by tests and all existing plus new tests are green. Great progress, maybe not perfection, but we can shape the last edges later if we find any. Hope you will take the opportunity to commit before too long, now that you have a patch based on a fairly new revision, to avoid having to many conflicts to solve. Just to avoid any mistakes - the patch covers SORL-3173, SORL-3178, SOLR-3382 and SOLR-3428. Regards, Per Steffensen Versioning - optimistic locking --- Key: SOLR-3178 URL: https://issues.apache.org/jira/browse/SOLR-3178 Project: Solr Issue Type: New Feature Components: update Affects Versions: 3.5 Environment: All Reporter: Per Steffensen Assignee: Per Steffensen Labels: RDBMS, insert, locking, nosql, optimistic, uniqueKey, update, versioning Fix For: 4.0 Attachments: SOLR-3173_3178_3382_3428_plus.patch, SOLR-3178.patch, SOLR_3173_3178_3382_plus.patch Original Estimate: 168h Remaining Estimate: 168h In order increase the ability of Solr to be used as a NoSql database (lots of concurrent inserts, updates, deletes and queries in the entire lifetime of the index) instead of just a search index (first: everything indexed (in one thread), after: only queries), I would like Solr to support versioning to be used for optimistic locking. When my intent (see SOLR-3173) is to update an existing document, I will need to provide a version-number equal to the version number I got when I fetched the existing document for update plus one. If this provided version-number does not correspond to the newest version-number of that document at the time of update plus one, I will get a VersionConflict error. If it does correspond the document will be updated with the new one, so that the newest version-number of that document is NOW one higher than before the update. Correct but efficient concurrency handling. When my intent (see SOLR-3173) is to insert a new document, the version number provided will not be used - instead a version-number 0 will be used. According to SOLR-3173 insert will only succeed if a document with the same value on uniqueKey-field does not already exist. In general when talking about different versions of the same document, of course we need to be able to identify when a document is the same - that, per definition, is when the values of the uniqueKey-fields are equal. The functionality provided by this issue is only really meaningfull when you run with updateLog activated. This issue might be solved more or less at the same time as SOLR-3173, and only one single SVN patch might be given to cover both issues. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators:
[jira] [Updated] (SOLR-3178) Versioning - optimistic locking
[ https://issues.apache.org/jira/browse/SOLR-3178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Per Steffensen updated SOLR-3178: - Attachment: SOLR_3173_3178_3382_plus.patch Find attached SOLR_3173_3178_3382_plus.patch which should fit on top of revision 1327417. Patch includes: - All our code-changes done during the work with SOLR-3173, SOLR-3178 and SOLR-3382. How it is supposed to work in more detail, is described on http://wiki.apache.org/solr/Per%20Steffensen/Update%20semantics and is of course also expressed in included tests. - Also includes -- Misc clean-up --- E.g. removing constants redundant OVERWRITE = overwrite so that only UpdateParams.OVERWRITE is left. You need very good arguments to ever use different names for the same thing (overwrite) among XML, JSON, HTTP request params etc, so it is kind of wrong to have the constant defined for each of those. -- Unimportant changes - e.g. --- Corrected spelling errors --- Removed unnecessary imports Not implemented yet - Error propagation for redistributed updates. That is, the responses from the redistributed updates in DistributedUpdateProcessor using SolrCmdDistributor, have to be collected an merged into a combined response from this DistributedUpdateProcessor. -- Both implementation and tests are missing. - Tests verifying that updates using JSON and CSV requests as described on http://wiki.apache.org/solr/Per%20Steffensen/Update%20semantics also works as described. Especially transfer of partRefs in JSON and CSV requests. - Tests verifying that semanticsMode consistency works as it is supposed to - just like ClassicConsistencyHybridXXXTests tests this for classic-consistency-hybrid semanticsMode and like ClassicUpdateSemanticsTest shortly tests it when semanticsMode is explicitly set to classic (classic is the default so this mode is already tested a lot using all the others tests in the project) Other stuff not done yet - Add Apache 2.0 Licence to all new files - Check correct indenting Versioning - optimistic locking --- Key: SOLR-3178 URL: https://issues.apache.org/jira/browse/SOLR-3178 Project: Solr Issue Type: New Feature Components: update Affects Versions: 3.5 Environment: All Reporter: Per Steffensen Assignee: Per Steffensen Labels: RDBMS, insert, locking, nosql, optimistic, uniqueKey, update, versioning Fix For: 4.0 Attachments: SOLR-3178.patch, SOLR_3173_3178_3382_plus.patch Original Estimate: 168h Remaining Estimate: 168h In order increase the ability of Solr to be used as a NoSql database (lots of concurrent inserts, updates, deletes and queries in the entire lifetime of the index) instead of just a search index (first: everything indexed (in one thread), after: only queries), I would like Solr to support versioning to be used for optimistic locking. When my intent (see SOLR-3173) is to update an existing document, I will need to provide a version-number equal to the version number I got when I fetched the existing document for update plus one. If this provided version-number does not correspond to the newest version-number of that document at the time of update plus one, I will get a VersionConflict error. If it does correspond the document will be updated with the new one, so that the newest version-number of that document is NOW one higher than before the update. Correct but efficient concurrency handling. When my intent (see SOLR-3173) is to insert a new document, the version number provided will not be used - instead a version-number 0 will be used. According to SOLR-3173 insert will only succeed if a document with the same value on uniqueKey-field does not already exist. In general when talking about different versions of the same document, of course we need to be able to identify when a document is the same - that, per definition, is when the values of the uniqueKey-fields are equal. The functionality provided by this issue is only really meaningfull when you run with updateLog activated. This issue might be solved more or less at the same time as SOLR-3173, and only one single SVN patch might be given to cover both issues. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-3178) Versioning - optimistic locking
[ https://issues.apache.org/jira/browse/SOLR-3178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yonik Seeley updated SOLR-3178: --- Attachment: SOLR-3178.patch Here's a patch that seems to work fine. There's really no extra input-API - all you do is supply a _version_ in the document (or as a parameter on the URL for a delete) and you get back a 409 if it's incorrect. I'll probably commit this soon and we can incrementally improve by adding better support to SolrJ (I'm not sure there's an easy way to specify the version for a delete from solrJ). Versioning - optimistic locking --- Key: SOLR-3178 URL: https://issues.apache.org/jira/browse/SOLR-3178 Project: Solr Issue Type: New Feature Components: update Affects Versions: 3.5 Environment: All Reporter: Per Steffensen Assignee: Per Steffensen Labels: RDBMS, insert, locking, nosql, optimistic, uniqueKey, update, versioning Fix For: 4.0 Attachments: SOLR-3178.patch Original Estimate: 168h Remaining Estimate: 168h In order increase the ability of Solr to be used as a NoSql database (lots of concurrent inserts, updates, deletes and queries in the entire lifetime of the index) instead of just a search index (first: everything indexed (in one thread), after: only queries), I would like Solr to support versioning to be used for optimistic locking. When my intent (see SOLR-3173) is to update an existing document, I will need to provide a version-number equal to the version number I got when I fetched the existing document for update plus one. If this provided version-number does not correspond to the newest version-number of that document at the time of update plus one, I will get a VersionConflict error. If it does correspond the document will be updated with the new one, so that the newest version-number of that document is NOW one higher than before the update. Correct but efficient concurrency handling. When my intent (see SOLR-3173) is to insert a new document, the version number provided will not be used - instead a version-number 0 will be used. According to SOLR-3173 insert will only succeed if a document with the same value on uniqueKey-field does not already exist. In general when talking about different versions of the same document, of course we need to be able to identify when a document is the same - that, per definition, is when the values of the uniqueKey-fields are equal. The functionality provided by this issue is only really meaningfull when you run with updateLog activated. This issue might be solved more or less at the same time as SOLR-3173, and only one single SVN patch might be given to cover both issues. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
