[jira] [Commented] (SOLR-11925) Auto delete oldest collections in a time routed alias
[ https://issues.apache.org/jira/browse/SOLR-11925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357905#comment-16357905 ] ASF subversion and git services commented on SOLR-11925: Commit ec8d22a1ce23d03699315f9aa64da2a54441deae in lucene-solr's branch refs/heads/branch_7x from [~dsmiley] [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=ec8d22a ] SOLR-11925: Rename RoutedAliasCreateCollectionCmd as MaintainRoutedAliasCmd (internal Cmd) (cherry picked from commit 1527ce5) > Auto delete oldest collections in a time routed alias > - > > Key: SOLR-11925 > URL: https://issues.apache.org/jira/browse/SOLR-11925 > Project: Solr > Issue Type: Sub-task > Security Level: Public(Default Security Level. Issues are Public) > Components: SolrCloud >Reporter: David Smiley >Assignee: David Smiley >Priority: Major > Fix For: 7.3 > > Attachments: SOLR-11925.patch, SOLR-11925.patch > > > The oldest collections in a Time Routed Alias should be automatically > deleted, according to a new alias metadata that establishes how long. It can > be checked as new data flows in at TimeRoutedAliasUpdateProcessor and thus it > won't occur if new data isn't coming in. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-11925) Auto delete oldest collections in a time routed alias
[ https://issues.apache.org/jira/browse/SOLR-11925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357904#comment-16357904 ] ASF subversion and git services commented on SOLR-11925: Commit 5ce83237e804ac1130eaf5cf793955667793fee0 in lucene-solr's branch refs/heads/branch_7x from [~dsmiley] [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=5ce8323 ] SOLR-11925: Time Routed Aliases: router.autoDeleteAge feature (cherry picked from commit 02b5172) > Auto delete oldest collections in a time routed alias > - > > Key: SOLR-11925 > URL: https://issues.apache.org/jira/browse/SOLR-11925 > Project: Solr > Issue Type: Sub-task > Security Level: Public(Default Security Level. Issues are Public) > Components: SolrCloud >Reporter: David Smiley >Assignee: David Smiley >Priority: Major > Fix For: 7.3 > > Attachments: SOLR-11925.patch, SOLR-11925.patch > > > The oldest collections in a Time Routed Alias should be automatically > deleted, according to a new alias metadata that establishes how long. It can > be checked as new data flows in at TimeRoutedAliasUpdateProcessor and thus it > won't occur if new data isn't coming in. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-11925) Auto delete oldest collections in a time routed alias
[ https://issues.apache.org/jira/browse/SOLR-11925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357902#comment-16357902 ] ASF subversion and git services commented on SOLR-11925: Commit 1527ce57d49721923ae43a81a10fe872ce94a2d8 in lucene-solr's branch refs/heads/master from [~dsmiley] [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=1527ce5 ] SOLR-11925: Rename RoutedAliasCreateCollectionCmd as MaintainRoutedAliasCmd (internal Cmd) > Auto delete oldest collections in a time routed alias > - > > Key: SOLR-11925 > URL: https://issues.apache.org/jira/browse/SOLR-11925 > Project: Solr > Issue Type: Sub-task > Security Level: Public(Default Security Level. Issues are Public) > Components: SolrCloud >Reporter: David Smiley >Assignee: David Smiley >Priority: Major > Fix For: 7.3 > > Attachments: SOLR-11925.patch, SOLR-11925.patch > > > The oldest collections in a Time Routed Alias should be automatically > deleted, according to a new alias metadata that establishes how long. It can > be checked as new data flows in at TimeRoutedAliasUpdateProcessor and thus it > won't occur if new data isn't coming in. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-11925) Auto delete oldest collections in a time routed alias
[ https://issues.apache.org/jira/browse/SOLR-11925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357901#comment-16357901 ] ASF subversion and git services commented on SOLR-11925: Commit 02b5172ea2e677e137b1d8563b335434433e048f in lucene-solr's branch refs/heads/master from [~dsmiley] [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=02b5172 ] SOLR-11925: Time Routed Aliases: router.autoDeleteAge feature > Auto delete oldest collections in a time routed alias > - > > Key: SOLR-11925 > URL: https://issues.apache.org/jira/browse/SOLR-11925 > Project: Solr > Issue Type: Sub-task > Security Level: Public(Default Security Level. Issues are Public) > Components: SolrCloud >Reporter: David Smiley >Assignee: David Smiley >Priority: Major > Fix For: 7.3 > > Attachments: SOLR-11925.patch, SOLR-11925.patch > > > The oldest collections in a Time Routed Alias should be automatically > deleted, according to a new alias metadata that establishes how long. It can > be checked as new data flows in at TimeRoutedAliasUpdateProcessor and thus it > won't occur if new data isn't coming in. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-11925) Auto delete oldest collections in a time routed alias
[ https://issues.apache.org/jira/browse/SOLR-11925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357636#comment-16357636 ] David Smiley commented on SOLR-11925: - I updated the patch with your feedback; mostly adding some docs and rewording this setting to "autoDeleteAge". If tests check out I'll commit tonight. Thanks for your feedback Gus! > Auto delete oldest collections in a time routed alias > - > > Key: SOLR-11925 > URL: https://issues.apache.org/jira/browse/SOLR-11925 > Project: Solr > Issue Type: Sub-task > Security Level: Public(Default Security Level. Issues are Public) > Components: SolrCloud >Reporter: David Smiley >Assignee: David Smiley >Priority: Major > Fix For: 7.3 > > Attachments: SOLR-11925.patch, SOLR-11925.patch > > > The oldest collections in a Time Routed Alias should be automatically > deleted, according to a new alias metadata that establishes how long. It can > be checked as new data flows in at TimeRoutedAliasUpdateProcessor and thus it > won't occur if new data isn't coming in. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-11925) Auto delete oldest collections in a time routed alias
[ https://issues.apache.org/jira/browse/SOLR-11925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357477#comment-16357477 ] David Smiley commented on SOLR-11925: - bq. Why is remoteInvoke on RouedAliasCreateCollectionCommand when it's only used from TimeRoutedAliasUpdateProcessor? I thought it might be nice to have a static method on the command with typed arguments so that it's (a) easy to invoke remotely, and (b) keeps details of composing & writing the message in one class. Do you think this is worse? bq. Rename field on zkstateReader aliasesHolder to aliasesManager? +1 will do bq. You added 2 parseSolrDateToInstant() methods with different params, but neither seems to be used? Whoops. I'll delete them; they are one-liners any way. bq. you seem to have dropped some conditionals looking for null or -1 error codes That moved to a new method on SolrResponse: getException(). I appreciate it's complicated to follow this in the diff; I found the original code being refactored difficult to understand. bq. Thinking about the validation in TimeRoutedAlias... Good commentary... yeah something to be mindful of in ModifyAlias in the future. bq. We seem to rely on applyModificationandExportToZk to ensure that the alias is updated before deleting collections, but I think that's a little risky... The code here is running on the node that has the Overseer, and thus issuing an admin command to delete the collection from the same node "sees" the same state (I think). If I'm wrong, what you describe doesn't sound serious? deleteOldestCollectionsAndUpdateAlias() will not throw an exception (thus won't prevent creating next collection), it will return errors in the overseer message response if there are any (results.add(...)). If this comes to pass, we could modify DeleteCollectionCmd to ensure it's aliases is up to date first. bq. Javadoc/doc +1 will do. > Auto delete oldest collections in a time routed alias > - > > Key: SOLR-11925 > URL: https://issues.apache.org/jira/browse/SOLR-11925 > Project: Solr > Issue Type: Sub-task > Security Level: Public(Default Security Level. Issues are Public) > Components: SolrCloud >Reporter: David Smiley >Assignee: David Smiley >Priority: Major > Fix For: 7.3 > > Attachments: SOLR-11925.patch > > > The oldest collections in a Time Routed Alias should be automatically > deleted, according to a new alias metadata that establishes how long. It can > be checked as new data flows in at TimeRoutedAliasUpdateProcessor and thus it > won't occur if new data isn't coming in. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-11925) Auto delete oldest collections in a time routed alias
[ https://issues.apache.org/jira/browse/SOLR-11925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16356438#comment-16356438 ] Gus Heck commented on SOLR-11925: - Took a look at the patch, here are my thoughts: Like: * rename of handle response * fluent style for addMetadata (returning the command) * create and delete sections of results :) * Nice cleanup in invokeAction, especially collecting and propagating the exception earlier. Question: * Why is remoteInvoke on RouedAliasCreateCollectionCommand when it's only used from TimeRoutedAliasUpdateProcessor? * Rename field on zkstateReader aliasesHolder to aliasesManager? * You added 2 parseSolrDateToInstant() methods with different params, but neither seems to be used? * in your cleanup of invokeAction etc, you seem to have dropped some conditionals looking for null or -1 error codes, which was added by MarkMiller in 2013... (bottom of handleResponse, which is now sentToOCPQueue) Any idea what that was trapping? Why is it not needed now? Thought about: * Thinking about the validation in TimeRoutedAlias ... Does seem that if an alias metadata gets messed up causing validation issue, all operations sending data to that alias will start failing. This worried me initially, but is probably ok. Must not get used in ModifyAlias in the future, since that would make it impossible to repair broken metadata. Not something we are doing now, but a potential pitfall. In constructor validation of values often worries me due to the inability to model busted state if state becomes busted. * We seem to rely on applyModificationandExportToZk to ensure that the alias is updated before deleting collections, but I think that's a little risky given the need for zookeeper watches to update. If we delete collections quickly after that nodes that have not processed watch notifications may send to the deleted shard? Do we need another (ugly, but magic) 100ms wait * I think we can worry about making deletion async if that becomes a pain point later. Simple to start as you have it now is good. Suggest: * Javadoc for deleteCollectionsAndUpdateAlias() - especially explanation of the correct notion for the "from" parameter. * Similarly doc for autoDeleteBeforeTime property on TimeRoutedAlias... time stuff is always fiddly, helps to have some doc. Not sure why it's not autoDeleteAfterInterval? Maybe you mean autoDeleteIfBefore? > Auto delete oldest collections in a time routed alias > - > > Key: SOLR-11925 > URL: https://issues.apache.org/jira/browse/SOLR-11925 > Project: Solr > Issue Type: Sub-task > Security Level: Public(Default Security Level. Issues are Public) > Components: SolrCloud >Reporter: David Smiley >Assignee: David Smiley >Priority: Major > Fix For: 7.3 > > Attachments: SOLR-11925.patch > > > The oldest collections in a Time Routed Alias should be automatically > deleted, according to a new alias metadata that establishes how long. It can > be checked as new data flows in at TimeRoutedAliasUpdateProcessor and thus it > won't occur if new data isn't coming in. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-11925) Auto delete oldest collections in a time routed alias
[ https://issues.apache.org/jira/browse/SOLR-11925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16349809#comment-16349809 ] David Smiley commented on SOLR-11925: - The attached patch deletes old collections immediately before creating a new collection. Includes test. The only nocommit is to rename RoutedAliasCreateCollectionCmd to MaintainRoutedAliasCmd since it doesn't just create now. Misc Time Routed Alias changes: * org.apache.solr.client.solrj.request.CollectionAdminRequest.ModifyAlias#addMetadata now returns "this" to support chained method calls Included with this patch are some internal refactorings/improvements: * CollectionsHandler.handleResponse is now named sendToOCPQueue (I chose the same name as operation.sendToOCPQueue boolean field). The API contract of this method is simpler to understand as well; it does not take in the SolrQueryResponse. * Added SolrResponse.getException that parses out the "exception" with response code & message to create a SolrException. This logic used to be in handleResponse/sendToOCPQueue but it's generally useful and helped enable the improved method signature. Two of sendToOCPQueue()'s callers use this method. * DateMathParser now has a constructor that takes *both* "now" and the timezone; formerly it only had one for the timezone and it required that you call setNow if you wanted to do that. I felt that was a weird asymmetry; both can be done through the constructor now. > Auto delete oldest collections in a time routed alias > - > > Key: SOLR-11925 > URL: https://issues.apache.org/jira/browse/SOLR-11925 > Project: Solr > Issue Type: Sub-task > Security Level: Public(Default Security Level. Issues are Public) > Components: SolrCloud >Reporter: David Smiley >Assignee: David Smiley >Priority: Major > Fix For: 7.3 > > Attachments: SOLR-11925.patch > > > The oldest collections in a Time Routed Alias should be automatically > deleted, according to a new alias metadata that establishes how long. It can > be checked as new data flows in at TimeRoutedAliasUpdateProcessor and thus it > won't occur if new data isn't coming in. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
