[jira] [Commented] (SOLR-11670) Implement a periodic house-keeping task
[ https://issues.apache.org/jira/browse/SOLR-11670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16404885#comment-16404885 ] ASF subversion and git services commented on SOLR-11670: Commit ed2d3583300263fa6aff4ad41b262bb2c32ae01c in lucene-solr's branch refs/heads/master from [~ab] [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=ed2d358 ] SOLR-11670: Make sure defaults are applied in simulated cluster. > Implement a periodic house-keeping task > --- > > Key: SOLR-11670 > URL: https://issues.apache.org/jira/browse/SOLR-11670 > Project: Solr > Issue Type: Sub-task > Security Level: Public(Default Security Level. Issues are Public) > Components: AutoScaling >Reporter: Andrzej Bialecki >Assignee: Andrzej Bialecki >Priority: Major > Fix For: 7.4, master (8.0) > > Attachments: SOLR-11670.patch, SOLR-11670.patch, SOLR-11670.patch > > > Some high-impact cluster changes (such as split shard) leave the original > data and original state that is no longer actively used. This makes sense due > to safety reasons and to make it easier to roll-back the changes. > However, this unused data will accumulate over time, especially when actions > like split shard are invoked automatically by the autoscaling framework. We > need a periodic task that would clean up this kind of data after a certain > period. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-11670) Implement a periodic house-keeping task
[ https://issues.apache.org/jira/browse/SOLR-11670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16404888#comment-16404888 ] ASF subversion and git services commented on SOLR-11670: Commit f6319d6d0a80e5f82b26f6b340ad250618f6b565 in lucene-solr's branch refs/heads/branch_7x from [~ab] [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=f6319d6 ] SOLR-11670: Make sure defaults are applied in simulated cluster. > Implement a periodic house-keeping task > --- > > Key: SOLR-11670 > URL: https://issues.apache.org/jira/browse/SOLR-11670 > Project: Solr > Issue Type: Sub-task > Security Level: Public(Default Security Level. Issues are Public) > Components: AutoScaling >Reporter: Andrzej Bialecki >Assignee: Andrzej Bialecki >Priority: Major > Fix For: 7.4, master (8.0) > > Attachments: SOLR-11670.patch, SOLR-11670.patch, SOLR-11670.patch > > > Some high-impact cluster changes (such as split shard) leave the original > data and original state that is no longer actively used. This makes sense due > to safety reasons and to make it easier to roll-back the changes. > However, this unused data will accumulate over time, especially when actions > like split shard are invoked automatically by the autoscaling framework. We > need a periodic task that would clean up this kind of data after a certain > period. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-11670) Implement a periodic house-keeping task
[ https://issues.apache.org/jira/browse/SOLR-11670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16404607#comment-16404607 ] Andrzej Bialecki commented on SOLR-11670: -- You're right [~dsmiley] - fixed, thanks. > Implement a periodic house-keeping task > --- > > Key: SOLR-11670 > URL: https://issues.apache.org/jira/browse/SOLR-11670 > Project: Solr > Issue Type: Sub-task > Security Level: Public(Default Security Level. Issues are Public) > Components: AutoScaling >Reporter: Andrzej Bialecki >Assignee: Andrzej Bialecki >Priority: Major > Fix For: 7.4, master (8.0) > > Attachments: SOLR-11670.patch, SOLR-11670.patch, SOLR-11670.patch > > > Some high-impact cluster changes (such as split shard) leave the original > data and original state that is no longer actively used. This makes sense due > to safety reasons and to make it easier to roll-back the changes. > However, this unused data will accumulate over time, especially when actions > like split shard are invoked automatically by the autoscaling framework. We > need a periodic task that would clean up this kind of data after a certain > period. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-11670) Implement a periodic house-keeping task
[ https://issues.apache.org/jira/browse/SOLR-11670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16404605#comment-16404605 ] ASF subversion and git services commented on SOLR-11670: Commit 80d7b2ada369d3caac8f6c6d94ccf8b2683ab5d6 in lucene-solr's branch refs/heads/branch_7x from [~ab] [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=80d7b2a ] SOLR-11670 This functionality was added in 7.4 and not in 7.3.0. > Implement a periodic house-keeping task > --- > > Key: SOLR-11670 > URL: https://issues.apache.org/jira/browse/SOLR-11670 > Project: Solr > Issue Type: Sub-task > Security Level: Public(Default Security Level. Issues are Public) > Components: AutoScaling >Reporter: Andrzej Bialecki >Assignee: Andrzej Bialecki >Priority: Major > Fix For: 7.4, master (8.0) > > Attachments: SOLR-11670.patch, SOLR-11670.patch, SOLR-11670.patch > > > Some high-impact cluster changes (such as split shard) leave the original > data and original state that is no longer actively used. This makes sense due > to safety reasons and to make it easier to roll-back the changes. > However, this unused data will accumulate over time, especially when actions > like split shard are invoked automatically by the autoscaling framework. We > need a periodic task that would clean up this kind of data after a certain > period. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-11670) Implement a periodic house-keeping task
[ https://issues.apache.org/jira/browse/SOLR-11670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16404602#comment-16404602 ] ASF subversion and git services commented on SOLR-11670: Commit fa03a3843c7e81046398d03bb5d4f1eb78e43fcb in lucene-solr's branch refs/heads/master from [~ab] [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=fa03a38 ] SOLR-11670 This functionality was added in 7.4 and not in 7.3.0. > Implement a periodic house-keeping task > --- > > Key: SOLR-11670 > URL: https://issues.apache.org/jira/browse/SOLR-11670 > Project: Solr > Issue Type: Sub-task > Security Level: Public(Default Security Level. Issues are Public) > Components: AutoScaling >Reporter: Andrzej Bialecki >Assignee: Andrzej Bialecki >Priority: Major > Fix For: 7.4, master (8.0) > > Attachments: SOLR-11670.patch, SOLR-11670.patch, SOLR-11670.patch > > > Some high-impact cluster changes (such as split shard) leave the original > data and original state that is no longer actively used. This makes sense due > to safety reasons and to make it easier to roll-back the changes. > However, this unused data will accumulate over time, especially when actions > like split shard are invoked automatically by the autoscaling framework. We > need a periodic task that would clean up this kind of data after a certain > period. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-11670) Implement a periodic house-keeping task
[ https://issues.apache.org/jira/browse/SOLR-11670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16404292#comment-16404292 ] David Smiley commented on SOLR-11670: - [~ab] you put this into the 7.3 section of CHANGES.txt but I think it belongs in 7.4? > Implement a periodic house-keeping task > --- > > Key: SOLR-11670 > URL: https://issues.apache.org/jira/browse/SOLR-11670 > Project: Solr > Issue Type: Sub-task > Security Level: Public(Default Security Level. Issues are Public) > Components: AutoScaling >Reporter: Andrzej Bialecki >Assignee: Andrzej Bialecki >Priority: Major > Fix For: 7.4, master (8.0) > > Attachments: SOLR-11670.patch, SOLR-11670.patch, SOLR-11670.patch > > > Some high-impact cluster changes (such as split shard) leave the original > data and original state that is no longer actively used. This makes sense due > to safety reasons and to make it easier to roll-back the changes. > However, this unused data will accumulate over time, especially when actions > like split shard are invoked automatically by the autoscaling framework. We > need a periodic task that would clean up this kind of data after a certain > period. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-11670) Implement a periodic house-keeping task
[ https://issues.apache.org/jira/browse/SOLR-11670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16400214#comment-16400214 ] ASF subversion and git services commented on SOLR-11670: Commit 21e2915f0d7b64cec2e02280fb4035cf687165ec in lucene-solr's branch refs/heads/branch_7x from [~ab] [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=21e2915 ] SOLR-11670: Allow for larger diff in simulated time. > Implement a periodic house-keeping task > --- > > Key: SOLR-11670 > URL: https://issues.apache.org/jira/browse/SOLR-11670 > Project: Solr > Issue Type: Sub-task > Security Level: Public(Default Security Level. Issues are Public) > Components: AutoScaling >Reporter: Andrzej Bialecki >Assignee: Andrzej Bialecki >Priority: Major > Fix For: 7.4, master (8.0) > > Attachments: SOLR-11670.patch, SOLR-11670.patch, SOLR-11670.patch > > > Some high-impact cluster changes (such as split shard) leave the original > data and original state that is no longer actively used. This makes sense due > to safety reasons and to make it easier to roll-back the changes. > However, this unused data will accumulate over time, especially when actions > like split shard are invoked automatically by the autoscaling framework. We > need a periodic task that would clean up this kind of data after a certain > period. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-11670) Implement a periodic house-keeping task
[ https://issues.apache.org/jira/browse/SOLR-11670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16400207#comment-16400207 ] ASF subversion and git services commented on SOLR-11670: Commit 25ec5cda0b2f4ce36366c56edc20e62e33040188 in lucene-solr's branch refs/heads/master from [~ab] [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=25ec5cd ] SOLR-11670: Allow for larger diff in simulated time. > Implement a periodic house-keeping task > --- > > Key: SOLR-11670 > URL: https://issues.apache.org/jira/browse/SOLR-11670 > Project: Solr > Issue Type: Sub-task > Security Level: Public(Default Security Level. Issues are Public) > Components: AutoScaling >Reporter: Andrzej Bialecki >Assignee: Andrzej Bialecki >Priority: Major > Fix For: 7.4, master (8.0) > > Attachments: SOLR-11670.patch, SOLR-11670.patch, SOLR-11670.patch > > > Some high-impact cluster changes (such as split shard) leave the original > data and original state that is no longer actively used. This makes sense due > to safety reasons and to make it easier to roll-back the changes. > However, this unused data will accumulate over time, especially when actions > like split shard are invoked automatically by the autoscaling framework. We > need a periodic task that would clean up this kind of data after a certain > period. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-11670) Implement a periodic house-keeping task
[ https://issues.apache.org/jira/browse/SOLR-11670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16400189#comment-16400189 ] ASF subversion and git services commented on SOLR-11670: Commit e1b0f796af9363b5496cc4ca8f17755f3c10e59b in lucene-solr's branch refs/heads/branch_7x from [~ab] [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=e1b0f79 ] SOLR-11670: Use TimeSource's value of NOW consistently when parsing date math. Add a unit test for TimeSource's epochTime. > Implement a periodic house-keeping task > --- > > Key: SOLR-11670 > URL: https://issues.apache.org/jira/browse/SOLR-11670 > Project: Solr > Issue Type: Sub-task > Security Level: Public(Default Security Level. Issues are Public) > Components: AutoScaling >Reporter: Andrzej Bialecki >Assignee: Andrzej Bialecki >Priority: Major > Fix For: 7.4, master (8.0) > > Attachments: SOLR-11670.patch, SOLR-11670.patch, SOLR-11670.patch > > > Some high-impact cluster changes (such as split shard) leave the original > data and original state that is no longer actively used. This makes sense due > to safety reasons and to make it easier to roll-back the changes. > However, this unused data will accumulate over time, especially when actions > like split shard are invoked automatically by the autoscaling framework. We > need a periodic task that would clean up this kind of data after a certain > period. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-11670) Implement a periodic house-keeping task
[ https://issues.apache.org/jira/browse/SOLR-11670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16400181#comment-16400181 ] ASF subversion and git services commented on SOLR-11670: Commit 17cfd87a28275b58deeec95d1172ed8cec2d1304 in lucene-solr's branch refs/heads/branch_7x from [~ab] [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=17cfd87 ] SOLR-11670: Implement a periodic house-keeping task. > Implement a periodic house-keeping task > --- > > Key: SOLR-11670 > URL: https://issues.apache.org/jira/browse/SOLR-11670 > Project: Solr > Issue Type: Sub-task > Security Level: Public(Default Security Level. Issues are Public) > Components: AutoScaling >Reporter: Andrzej Bialecki >Assignee: Andrzej Bialecki >Priority: Major > Fix For: 7.4, master (8.0) > > Attachments: SOLR-11670.patch, SOLR-11670.patch, SOLR-11670.patch > > > Some high-impact cluster changes (such as split shard) leave the original > data and original state that is no longer actively used. This makes sense due > to safety reasons and to make it easier to roll-back the changes. > However, this unused data will accumulate over time, especially when actions > like split shard are invoked automatically by the autoscaling framework. We > need a periodic task that would clean up this kind of data after a certain > period. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-11670) Implement a periodic house-keeping task
[ https://issues.apache.org/jira/browse/SOLR-11670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16398413#comment-16398413 ] Andrzej Bialecki commented on SOLR-11670: -- I created SOLR-12091 to track the method renaming. > Implement a periodic house-keeping task > --- > > Key: SOLR-11670 > URL: https://issues.apache.org/jira/browse/SOLR-11670 > Project: Solr > Issue Type: Sub-task > Security Level: Public(Default Security Level. Issues are Public) > Components: AutoScaling >Reporter: Andrzej Bialecki >Assignee: Andrzej Bialecki >Priority: Major > Fix For: 7.4, master (8.0) > > Attachments: SOLR-11670.patch, SOLR-11670.patch, SOLR-11670.patch > > > Some high-impact cluster changes (such as split shard) leave the original > data and original state that is no longer actively used. This makes sense due > to safety reasons and to make it easier to roll-back the changes. > However, this unused data will accumulate over time, especially when actions > like split shard are invoked automatically by the autoscaling framework. We > need a periodic task that would clean up this kind of data after a certain > period. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-11670) Implement a periodic house-keeping task
[
https://issues.apache.org/jira/browse/SOLR-11670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16397106#comment-16397106
]
Andrzej Bialecki commented on SOLR-11670:
--
bq. it's confusing that TimeSource.getTime & getEpochTime return nanoseconds
{{System.nanoTime}} returns time in ns, too ... but I see your point, we can
rename these two methods to {{getTimeNs}} and {{getEpochTimeNs}}.
> Implement a periodic house-keeping task
> ---
>
> Key: SOLR-11670
> URL: https://issues.apache.org/jira/browse/SOLR-11670
> Project: Solr
> Issue Type: Sub-task
> Security Level: Public(Default Security Level. Issues are Public)
> Components: AutoScaling
>Reporter: Andrzej Bialecki
>Assignee: Andrzej Bialecki
>Priority: Major
> Attachments: SOLR-11670.patch, SOLR-11670.patch, SOLR-11670.patch
>
>
> Some high-impact cluster changes (such as split shard) leave the original
> data and original state that is no longer actively used. This makes sense due
> to safety reasons and to make it easier to roll-back the changes.
> However, this unused data will accumulate over time, especially when actions
> like split shard are invoked automatically by the autoscaling framework. We
> need a periodic task that would clean up this kind of data after a certain
> period.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-11670) Implement a periodic house-keeping task
[
https://issues.apache.org/jira/browse/SOLR-11670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16397085#comment-16397085
]
David Smiley commented on SOLR-11670:
-
[~ab] I think it's confusing that TimeSource.getTime & getEpochTime return
nanoseconds. I think the methods should be renamed so that it's clear what
unit it is, otherwise the conditions are ripe for continuing bugs in unit
conversions. Perhaps rename getTime to {{getTimeNs}}.
> Implement a periodic house-keeping task
> ---
>
> Key: SOLR-11670
> URL: https://issues.apache.org/jira/browse/SOLR-11670
> Project: Solr
> Issue Type: Sub-task
> Security Level: Public(Default Security Level. Issues are Public)
> Components: AutoScaling
>Reporter: Andrzej Bialecki
>Assignee: Andrzej Bialecki
>Priority: Major
> Attachments: SOLR-11670.patch, SOLR-11670.patch, SOLR-11670.patch
>
>
> Some high-impact cluster changes (such as split shard) leave the original
> data and original state that is no longer actively used. This makes sense due
> to safety reasons and to make it easier to roll-back the changes.
> However, this unused data will accumulate over time, especially when actions
> like split shard are invoked automatically by the autoscaling framework. We
> need a periodic task that would clean up this kind of data after a certain
> period.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-11670) Implement a periodic house-keeping task
[ https://issues.apache.org/jira/browse/SOLR-11670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16396897#comment-16396897 ] ASF subversion and git services commented on SOLR-11670: Commit ed6feded6de7f1c268986df6de6a5dc9db6a3f34 in lucene-solr's branch refs/heads/master from [~ab] [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=ed6feded ] SOLR-11670: Use TimeSource's value of NOW consistently when parsing date math. Add a unit test for TimeSource's epochTime. > Implement a periodic house-keeping task > --- > > Key: SOLR-11670 > URL: https://issues.apache.org/jira/browse/SOLR-11670 > Project: Solr > Issue Type: Sub-task > Security Level: Public(Default Security Level. Issues are Public) > Components: AutoScaling >Reporter: Andrzej Bialecki >Assignee: Andrzej Bialecki >Priority: Major > Attachments: SOLR-11670.patch, SOLR-11670.patch, SOLR-11670.patch > > > Some high-impact cluster changes (such as split shard) leave the original > data and original state that is no longer actively used. This makes sense due > to safety reasons and to make it easier to roll-back the changes. > However, this unused data will accumulate over time, especially when actions > like split shard are invoked automatically by the autoscaling framework. We > need a periodic task that would clean up this kind of data after a certain > period. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-11670) Implement a periodic house-keeping task
[ https://issues.apache.org/jira/browse/SOLR-11670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16391281#comment-16391281 ] ASF subversion and git services commented on SOLR-11670: Commit b17052e8520bb57bcfe126aa2f8e6bd0b9aa76c5 in lucene-solr's branch refs/heads/master from [~ab] [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=b17052e ] SOLR-11670: Implement a periodic house-keeping task. > Implement a periodic house-keeping task > --- > > Key: SOLR-11670 > URL: https://issues.apache.org/jira/browse/SOLR-11670 > Project: Solr > Issue Type: Sub-task > Security Level: Public(Default Security Level. Issues are Public) > Components: AutoScaling >Reporter: Andrzej Bialecki >Assignee: Andrzej Bialecki >Priority: Major > Attachments: SOLR-11670.patch, SOLR-11670.patch, SOLR-11670.patch > > > Some high-impact cluster changes (such as split shard) leave the original > data and original state that is no longer actively used. This makes sense due > to safety reasons and to make it easier to roll-back the changes. > However, this unused data will accumulate over time, especially when actions > like split shard are invoked automatically by the autoscaling framework. We > need a periodic task that would clean up this kind of data after a certain > period. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-11670) Implement a periodic house-keeping task
[ https://issues.apache.org/jira/browse/SOLR-11670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16391127#comment-16391127 ] Shalin Shekhar Mangar commented on SOLR-11670: -- Thanks for explaining! > Implement a periodic house-keeping task > --- > > Key: SOLR-11670 > URL: https://issues.apache.org/jira/browse/SOLR-11670 > Project: Solr > Issue Type: Sub-task > Security Level: Public(Default Security Level. Issues are Public) > Components: AutoScaling >Reporter: Andrzej Bialecki >Assignee: Andrzej Bialecki >Priority: Major > Attachments: SOLR-11670.patch, SOLR-11670.patch, SOLR-11670.patch > > > Some high-impact cluster changes (such as split shard) leave the original > data and original state that is no longer actively used. This makes sense due > to safety reasons and to make it easier to roll-back the changes. > However, this unused data will accumulate over time, especially when actions > like split shard are invoked automatically by the autoscaling framework. We > need a periodic task that would clean up this kind of data after a certain > period. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-11670) Implement a periodic house-keeping task
[
https://issues.apache.org/jira/browse/SOLR-11670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16391030#comment-16391030
]
Andrzej Bialecki commented on SOLR-11670:
--
bq. why do we make a copy of triggerListeners in the processor
If we change {{AutoScalingConfig}} while an event is being processed then we
may get inconsistent listener notifications. I noticed this situation in tests
where old events (from a previous test) would continue to be processed during
the next test method, but they produced only some of the notifications one
would normally expect (because the listeners have been cleared between eg.
BEFORE_ACTION and AFTER_ACTION).
Under normal circumstances this shouldn't be a big deal, but in tests this is
much more likely to happen.
> Implement a periodic house-keeping task
> ---
>
> Key: SOLR-11670
> URL: https://issues.apache.org/jira/browse/SOLR-11670
> Project: Solr
> Issue Type: Sub-task
> Security Level: Public(Default Security Level. Issues are Public)
> Components: AutoScaling
>Reporter: Andrzej Bialecki
>Assignee: Andrzej Bialecki
>Priority: Major
> Attachments: SOLR-11670.patch, SOLR-11670.patch, SOLR-11670.patch
>
>
> Some high-impact cluster changes (such as split shard) leave the original
> data and original state that is no longer actively used. This makes sense due
> to safety reasons and to make it easier to roll-back the changes.
> However, this unused data will accumulate over time, especially when actions
> like split shard are invoked automatically by the autoscaling framework. We
> need a periodic task that would clean up this kind of data after a certain
> period.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-11670) Implement a periodic house-keeping task
[ https://issues.apache.org/jira/browse/SOLR-11670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16390895#comment-16390895 ] Shalin Shekhar Mangar commented on SOLR-11670: -- Looks great Andrzej! Only thing I don't understand is why do we make a copy of triggerListeners in the processor? +1 to commit regardless of that. > Implement a periodic house-keeping task > --- > > Key: SOLR-11670 > URL: https://issues.apache.org/jira/browse/SOLR-11670 > Project: Solr > Issue Type: Sub-task > Security Level: Public(Default Security Level. Issues are Public) > Components: AutoScaling >Reporter: Andrzej Bialecki >Assignee: Andrzej Bialecki >Priority: Major > Attachments: SOLR-11670.patch, SOLR-11670.patch, SOLR-11670.patch > > > Some high-impact cluster changes (such as split shard) leave the original > data and original state that is no longer actively used. This makes sense due > to safety reasons and to make it easier to roll-back the changes. > However, this unused data will accumulate over time, especially when actions > like split shard are invoked automatically by the autoscaling framework. We > need a periodic task that would clean up this kind of data after a certain > period. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-11670) Implement a periodic house-keeping task
[
https://issues.apache.org/jira/browse/SOLR-11670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16389506#comment-16389506
]
Andrzej Bialecki commented on SOLR-11670:
--
Updated patch addressing the issues from review. Using {{ExecutePlanAction}} is
a good idea - I changed {{InactiveShardCleanupAction}} to
{{InactiveShardPlanAction}}, which only produces "operations" to be executed by
{{ExecutePlanAction}}.
> Implement a periodic house-keeping task
> ---
>
> Key: SOLR-11670
> URL: https://issues.apache.org/jira/browse/SOLR-11670
> Project: Solr
> Issue Type: Sub-task
> Security Level: Public(Default Security Level. Issues are Public)
> Components: AutoScaling
>Reporter: Andrzej Bialecki
>Assignee: Andrzej Bialecki
>Priority: Major
> Attachments: SOLR-11670.patch, SOLR-11670.patch, SOLR-11670.patch
>
>
> Some high-impact cluster changes (such as split shard) leave the original
> data and original state that is no longer actively used. This makes sense due
> to safety reasons and to make it easier to roll-back the changes.
> However, this unused data will accumulate over time, especially when actions
> like split shard are invoked automatically by the autoscaling framework. We
> need a periodic task that would clean up this kind of data after a certain
> period.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-11670) Implement a periodic house-keeping task
[ https://issues.apache.org/jira/browse/SOLR-11670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16388379#comment-16388379 ] Shalin Shekhar Mangar commented on SOLR-11670: -- Thanks Andrzej! A few comments: * Autoscaling.java -- the constant AUTO_ADD_REPLICAS_LISTENER_NAME is not used anywhere * ExecutePlanAction -- the counter appended to the asyncId has been removed. It is not useful today I agree but it will be needed again when SOLR-11605 is implemented * InactiveShardCleanupAction -- It is not safe to compare nanotime if the overseer leader changed between the shard split and the cleanup task. Perhaps we stick to currentTimeMillis() here? * Perhaps we should let InactiveShardCleanupAction only produce operations that are later executed by ExecutePlanAction? The advantage is that those operations (and others created by clean up tasks in future) can be performed in parallel once SOLR-11605 comes in. But more importantly, an exception thrown in one clean up action will not cause the action processing to be aborted. This point is moot however if you envision each future clean up action to have its own scheduled trigger (in which case we should increase the schedule interval for this one to something very large e.g. once a day). It is also possible that future clean up tasks do not produce solrj requests at all. I'm curious to know what you think about this. * Exceptions thrown from the delete shard API should also be added to context properties so that they can be sent to listeners * SimSolrCloudTestCase -- many unused imports > Implement a periodic house-keeping task > --- > > Key: SOLR-11670 > URL: https://issues.apache.org/jira/browse/SOLR-11670 > Project: Solr > Issue Type: Sub-task > Security Level: Public(Default Security Level. Issues are Public) > Components: AutoScaling >Reporter: Andrzej Bialecki >Assignee: Andrzej Bialecki >Priority: Major > Attachments: SOLR-11670.patch, SOLR-11670.patch > > > Some high-impact cluster changes (such as split shard) leave the original > data and original state that is no longer actively used. This makes sense due > to safety reasons and to make it easier to roll-back the changes. > However, this unused data will accumulate over time, especially when actions > like split shard are invoked automatically by the autoscaling framework. We > need a periodic task that would clean up this kind of data after a certain > period. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-11670) Implement a periodic house-keeping task
[
https://issues.apache.org/jira/browse/SOLR-11670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16386652#comment-16386652
]
Andrzej Bialecki commented on SOLR-11670:
--
[~shalinmangar] suggested another approach (thanks!) and this patch implements
it - instead of a separate component for managing the tasks it reuses the
{{ScheduledTrigger}} and custom trigger actions that implement maintenance
tasks.
This indeed simplifies the execution and management of these tasks, and reuses
familiar concepts. The (small) downside is that it's less convenient to
pre-register some tasks that we know the cluster should run by default, but
it's the same situation as with any default triggers (eg. autoAddReplicas
trigger).
{{ScheduledMaintenanceTriggerTest}} illustrates the registration and how the
{{InactiveShardCleanupAction}} works.
(The patch seems larger than before but that's due to some refactoring of
common utility methods for waiting on collection state.)
> Implement a periodic house-keeping task
> ---
>
> Key: SOLR-11670
> URL: https://issues.apache.org/jira/browse/SOLR-11670
> Project: Solr
> Issue Type: Sub-task
> Security Level: Public(Default Security Level. Issues are Public)
> Components: AutoScaling
>Reporter: Andrzej Bialecki
>Assignee: Andrzej Bialecki
>Priority: Major
> Attachments: SOLR-11670.patch, SOLR-11670.patch
>
>
> Some high-impact cluster changes (such as split shard) leave the original
> data and original state that is no longer actively used. This makes sense due
> to safety reasons and to make it easier to roll-back the changes.
> However, this unused data will accumulate over time, especially when actions
> like split shard are invoked automatically by the autoscaling framework. We
> need a periodic task that would clean up this kind of data after a certain
> period.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-11670) Implement a periodic house-keeping task
[
https://issues.apache.org/jira/browse/SOLR-11670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16379030#comment-16379030
]
Andrzej Bialecki commented on SOLR-11670:
--
Patch containing the following:
* a {{MaintenanceTask}} API that components may use for registering tasks with
Overseer. Tasks are initialized using key / value pairs from
{{/clusterprops.json}}
* {{MaintenanceTasks}} component, which is a registry and runner of tasks.
This component also monitors changes to {{/clusterprops.json}} and
re-initializes tasks and their schedule as needed.
* implementation of {{InactiveSliceCleanupTask}}, which deletes inactive
slices that exceeded a configured TTL time. This task is registered by
{{SplitShardCmd}}.
* changes to {{SplitShardTest}} to exercise this code.
This code is also available on branch {{jira/solr-11670}}.
> Implement a periodic house-keeping task
> ---
>
> Key: SOLR-11670
> URL: https://issues.apache.org/jira/browse/SOLR-11670
> Project: Solr
> Issue Type: Sub-task
> Security Level: Public(Default Security Level. Issues are Public)
> Components: AutoScaling
>Reporter: Andrzej Bialecki
>Assignee: Andrzej Bialecki
>Priority: Major
> Attachments: SOLR-11670.patch
>
>
> Some high-impact cluster changes (such as split shard) leave the original
> data and original state that is no longer actively used. This makes sense due
> to safety reasons and to make it easier to roll-back the changes.
> However, this unused data will accumulate over time, especially when actions
> like split shard are invoked automatically by the autoscaling framework. We
> need a periodic task that would clean up this kind of data after a certain
> period.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-11670) Implement a periodic house-keeping task
[ https://issues.apache.org/jira/browse/SOLR-11670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16310110#comment-16310110 ] Andrzej Bialecki commented on SOLR-11670: -- Another candidate data for periodic cleanups are async status ids - as of now, REQUESTSTATUS does not automatically clean up the tracking data structures, meaning the status of completed or failed tasks stays stored in ZooKeeper unless cleared manually. > Implement a periodic house-keeping task > --- > > Key: SOLR-11670 > URL: https://issues.apache.org/jira/browse/SOLR-11670 > Project: Solr > Issue Type: Sub-task > Security Level: Public(Default Security Level. Issues are Public) > Components: AutoScaling >Reporter: Andrzej Bialecki >Assignee: Andrzej Bialecki > > Some high-impact cluster changes (such as split shard) leave the original > data and original state that is no longer actively used. This makes sense due > to safety reasons and to make it easier to roll-back the changes. > However, this unused data will accumulate over time, especially when actions > like split shard are invoked automatically by the autoscaling framework. We > need a periodic task that would clean up this kind of data after a certain > period. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-11670) Implement a periodic house-keeping task
[ https://issues.apache.org/jira/browse/SOLR-11670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16265542#comment-16265542 ] Erick Erickson commented on SOLR-11670: --- Here's a way to generate dead nodes in 6x: Set up a single solr instance, starting it like this: bin/solr start -z localhost:2181 -p 8981 -s example/cloud/node1/solr Create a collection (one shard and one replica will do). Stop that instance and start another changing the port but keeping the same SOLR_HOME: bin/solr start -z localhost:2181 -p 8982 -s example/cloud/node1/solr (note, the port has changed, but the -s points to where the old core.properties file is located). legacyCloud=true needs to be true for this case. 7x doesn't exhibit the behavior at all. I think the issue is that coreNodeName gets defined in core.properties and is used to update an existing znode if things like the base_url or node_name change. > Implement a periodic house-keeping task > --- > > Key: SOLR-11670 > URL: https://issues.apache.org/jira/browse/SOLR-11670 > Project: Solr > Issue Type: Sub-task > Security Level: Public(Default Security Level. Issues are Public) > Components: AutoScaling >Reporter: Andrzej Bialecki >Assignee: Andrzej Bialecki > > Some high-impact cluster changes (such as split shard) leave the original > data and original state that is no longer actively used. This makes sense due > to safety reasons and to make it easier to roll-back the changes. > However, this unused data will accumulate over time, especially when actions > like split shard are invoked automatically by the autoscaling framework. We > need a periodic task that would clean up this kind of data after a certain > period. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
