[jira] [Commented] (CASSANDRA-12039) Add an index callback to be notified post bootstrap and before joining the ring
[ https://issues.apache.org/jira/browse/CASSANDRA-12039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15509606#comment-15509606 ] Sergio Bossa commented on CASSANDRA-12039: -- Thanks for helping and reviewing [~beobal]! > Add an index callback to be notified post bootstrap and before joining the > ring > --- > > Key: CASSANDRA-12039 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12039 > Project: Cassandra > Issue Type: New Feature >Reporter: Sergio Bossa >Assignee: Sergio Bossa > Fix For: 3.10 > > > Custom index implementations might need to be notified when the node finishes > bootstrapping in order to execute some blocking tasks before the node itself > goes into NORMAL state. > This is a proposal to add such functionality, which should roughly require > the following: > 1) Add a {{getPostBootstrapTask}} callback to the {{Index}} interface. > 2) Add an {{executePostBootstrapBlockingTasks}} method to > {{SecondaryIndexManager}} calling into the previously mentioned callback. > 3) Hook that into {{StorageService#joinTokenRing}}. > Thoughts? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12039) Add an index callback to be notified post bootstrap and before joining the ring
[ https://issues.apache.org/jira/browse/CASSANDRA-12039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15506862#comment-15506862 ] Sergio Bossa commented on CASSANDRA-12039: -- bq. The patch should only go into 3.x though. Fine for me. > Add an index callback to be notified post bootstrap and before joining the > ring > --- > > Key: CASSANDRA-12039 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12039 > Project: Cassandra > Issue Type: New Feature >Reporter: Sergio Bossa >Assignee: Sergio Bossa > Fix For: 3.x > > > Custom index implementations might need to be notified when the node finishes > bootstrapping in order to execute some blocking tasks before the node itself > goes into NORMAL state. > This is a proposal to add such functionality, which should roughly require > the following: > 1) Add a {{getPostBootstrapTask}} callback to the {{Index}} interface. > 2) Add an {{executePostBootstrapBlockingTasks}} method to > {{SecondaryIndexManager}} calling into the previously mentioned callback. > 3) Hook that into {{StorageService#joinTokenRing}}. > Thoughts? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12039) Add an index callback to be notified post bootstrap and before joining the ring
[ https://issues.apache.org/jira/browse/CASSANDRA-12039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15505238#comment-15505238 ] Aleksey Yeschenko commented on CASSANDRA-12039: --- CI results look clean to me (with one failure that I'm pretty sure is unrelated). The patch should only go into 3.x though. > Add an index callback to be notified post bootstrap and before joining the > ring > --- > > Key: CASSANDRA-12039 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12039 > Project: Cassandra > Issue Type: New Feature >Reporter: Sergio Bossa >Assignee: Sergio Bossa > Fix For: 3.x > > > Custom index implementations might need to be notified when the node finishes > bootstrapping in order to execute some blocking tasks before the node itself > goes into NORMAL state. > This is a proposal to add such functionality, which should roughly require > the following: > 1) Add a {{getPostBootstrapTask}} callback to the {{Index}} interface. > 2) Add an {{executePostBootstrapBlockingTasks}} method to > {{SecondaryIndexManager}} calling into the previously mentioned callback. > 3) Hook that into {{StorageService#joinTokenRing}}. > Thoughts? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12039) Add an index callback to be notified post bootstrap and before joining the ring
[ https://issues.apache.org/jira/browse/CASSANDRA-12039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15488110#comment-15488110 ] Sam Tunnicliffe commented on CASSANDRA-12039: - Ok, I've rebased my dtest branch as there was some restructuring of the {{tools}} module since last update. For simplicity's sake, I've opened a [new PR|https://github.com/riptano/cassandra-dtest/pull/1316] based on my branch and will commit this as soon as the dtest get a +1. > Add an index callback to be notified post bootstrap and before joining the > ring > --- > > Key: CASSANDRA-12039 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12039 > Project: Cassandra > Issue Type: New Feature >Reporter: Sergio Bossa >Assignee: Sergio Bossa > > Custom index implementations might need to be notified when the node finishes > bootstrapping in order to execute some blocking tasks before the node itself > goes into NORMAL state. > This is a proposal to add such functionality, which should roughly require > the following: > 1) Add a {{getPostBootstrapTask}} callback to the {{Index}} interface. > 2) Add an {{executePostBootstrapBlockingTasks}} method to > {{SecondaryIndexManager}} calling into the previously mentioned callback. > 3) Hook that into {{StorageService#joinTokenRing}}. > Thoughts? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12039) Add an index callback to be notified post bootstrap and before joining the ring
[ https://issues.apache.org/jira/browse/CASSANDRA-12039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15449274#comment-15449274 ] Sam Tunnicliffe commented on CASSANDRA-12039: - Already done :) > Add an index callback to be notified post bootstrap and before joining the > ring > --- > > Key: CASSANDRA-12039 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12039 > Project: Cassandra > Issue Type: New Feature >Reporter: Sergio Bossa >Assignee: Sergio Bossa > > Custom index implementations might need to be notified when the node finishes > bootstrapping in order to execute some blocking tasks before the node itself > goes into NORMAL state. > This is a proposal to add such functionality, which should roughly require > the following: > 1) Add a {{getPostBootstrapTask}} callback to the {{Index}} interface. > 2) Add an {{executePostBootstrapBlockingTasks}} method to > {{SecondaryIndexManager}} calling into the previously mentioned callback. > 3) Hook that into {{StorageService#joinTokenRing}}. > Thoughts? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12039) Add an index callback to be notified post bootstrap and before joining the ring
[ https://issues.apache.org/jira/browse/CASSANDRA-12039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15449267#comment-15449267 ] Sergio Bossa commented on CASSANDRA-12039: -- bq. I just pushed an update to my dtest branch which changes resume_test to use byteman Yay! I didn't know it was possible to use Byteman, great job. Could you just remove the {{timeout}} parameter from {{_base_test}}? > Add an index callback to be notified post bootstrap and before joining the > ring > --- > > Key: CASSANDRA-12039 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12039 > Project: Cassandra > Issue Type: New Feature >Reporter: Sergio Bossa >Assignee: Sergio Bossa > > Custom index implementations might need to be notified when the node finishes > bootstrapping in order to execute some blocking tasks before the node itself > goes into NORMAL state. > This is a proposal to add such functionality, which should roughly require > the following: > 1) Add a {{getPostBootstrapTask}} callback to the {{Index}} interface. > 2) Add an {{executePostBootstrapBlockingTasks}} method to > {{SecondaryIndexManager}} calling into the previously mentioned callback. > 3) Hook that into {{StorageService#joinTokenRing}}. > Thoughts? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12039) Add an index callback to be notified post bootstrap and before joining the ring
[ https://issues.apache.org/jira/browse/CASSANDRA-12039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15449255#comment-15449255 ] Sergio Bossa commented on CASSANDRA-12039: -- Looks like we're in an impasse here: no one can see what's wrong with the patch, or the tests, and the test infrastructure doesn't provide enough aid to properly debug the problem. The test will not magically fix itself, so we either remove it or pause this issue until the test infrastructure is improved to _at least_ allow to take thread dumps of hanging nodes. Thoughts? > Add an index callback to be notified post bootstrap and before joining the > ring > --- > > Key: CASSANDRA-12039 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12039 > Project: Cassandra > Issue Type: New Feature >Reporter: Sergio Bossa >Assignee: Sergio Bossa > > Custom index implementations might need to be notified when the node finishes > bootstrapping in order to execute some blocking tasks before the node itself > goes into NORMAL state. > This is a proposal to add such functionality, which should roughly require > the following: > 1) Add a {{getPostBootstrapTask}} callback to the {{Index}} interface. > 2) Add an {{executePostBootstrapBlockingTasks}} method to > {{SecondaryIndexManager}} calling into the previously mentioned callback. > 3) Hook that into {{StorageService#joinTokenRing}}. > Thoughts? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12039) Add an index callback to be notified post bootstrap and before joining the ring
[ https://issues.apache.org/jira/browse/CASSANDRA-12039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15449240#comment-15449240 ] Sam Tunnicliffe commented on CASSANDRA-12039: - I just pushed an update to [my dtest branch|https://github.com/beobal/cassandra-dtest/commits/CASSANDRA-12039] which changes {{resume_test}} to use byteman to inject a failure into the bootstrap process, as {{bootstrap_test:TestBootstrap.resumable_bootstrap_test}} now does. This seems to have stabilized the test (at least running locally) and also has the side effect of reducing the amount of data needed during {{_base_test}}, which speeds up the run considerably. > Add an index callback to be notified post bootstrap and before joining the > ring > --- > > Key: CASSANDRA-12039 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12039 > Project: Cassandra > Issue Type: New Feature >Reporter: Sergio Bossa >Assignee: Sergio Bossa > > Custom index implementations might need to be notified when the node finishes > bootstrapping in order to execute some blocking tasks before the node itself > goes into NORMAL state. > This is a proposal to add such functionality, which should roughly require > the following: > 1) Add a {{getPostBootstrapTask}} callback to the {{Index}} interface. > 2) Add an {{executePostBootstrapBlockingTasks}} method to > {{SecondaryIndexManager}} calling into the previously mentioned callback. > 3) Hook that into {{StorageService#joinTokenRing}}. > Thoughts? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12039) Add an index callback to be notified post bootstrap and before joining the ring
[ https://issues.apache.org/jira/browse/CASSANDRA-12039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15449224#comment-15449224 ] Philip Thompson commented on CASSANDRA-12039: - I cannot see anything in the test that would cause the issue > Add an index callback to be notified post bootstrap and before joining the > ring > --- > > Key: CASSANDRA-12039 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12039 > Project: Cassandra > Issue Type: New Feature >Reporter: Sergio Bossa >Assignee: Sergio Bossa > > Custom index implementations might need to be notified when the node finishes > bootstrapping in order to execute some blocking tasks before the node itself > goes into NORMAL state. > This is a proposal to add such functionality, which should roughly require > the following: > 1) Add a {{getPostBootstrapTask}} callback to the {{Index}} interface. > 2) Add an {{executePostBootstrapBlockingTasks}} method to > {{SecondaryIndexManager}} calling into the previously mentioned callback. > 3) Hook that into {{StorageService#joinTokenRing}}. > Thoughts? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12039) Add an index callback to be notified post bootstrap and before joining the ring
[ https://issues.apache.org/jira/browse/CASSANDRA-12039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15448850#comment-15448850 ] Sergio Bossa commented on CASSANDRA-12039: -- Ping [~beobal], [~philipthompson]. > Add an index callback to be notified post bootstrap and before joining the > ring > --- > > Key: CASSANDRA-12039 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12039 > Project: Cassandra > Issue Type: New Feature >Reporter: Sergio Bossa >Assignee: Sergio Bossa > > Custom index implementations might need to be notified when the node finishes > bootstrapping in order to execute some blocking tasks before the node itself > goes into NORMAL state. > This is a proposal to add such functionality, which should roughly require > the following: > 1) Add a {{getPostBootstrapTask}} callback to the {{Index}} interface. > 2) Add an {{executePostBootstrapBlockingTasks}} method to > {{SecondaryIndexManager}} calling into the previously mentioned callback. > 3) Hook that into {{StorageService#joinTokenRing}}. > Thoughts? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12039) Add an index callback to be notified post bootstrap and before joining the ring
[ https://issues.apache.org/jira/browse/CASSANDRA-12039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15437331#comment-15437331 ] Sergio Bossa commented on CASSANDRA-12039: -- I tried to debug it but I couldn't find anything relevant in the logs, and the test actually hangs up locally for me most of the time. The test looks correct and I have no idea why the node hangs out of the blue: I tried taking some thread dumps but I couldn't find the log they end up in. The only thing I can think of is to remove the test, thoughts? > Add an index callback to be notified post bootstrap and before joining the > ring > --- > > Key: CASSANDRA-12039 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12039 > Project: Cassandra > Issue Type: New Feature >Reporter: Sergio Bossa >Assignee: Sergio Bossa > > Custom index implementations might need to be notified when the node finishes > bootstrapping in order to execute some blocking tasks before the node itself > goes into NORMAL state. > This is a proposal to add such functionality, which should roughly require > the following: > 1) Add a {{getPostBootstrapTask}} callback to the {{Index}} interface. > 2) Add an {{executePostBootstrapBlockingTasks}} method to > {{SecondaryIndexManager}} calling into the previously mentioned callback. > 3) Hook that into {{StorageService#joinTokenRing}}. > Thoughts? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12039) Add an index callback to be notified post bootstrap and before joining the ring
[ https://issues.apache.org/jira/browse/CASSANDRA-12039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15436954#comment-15436954 ] Philip Thompson commented on CASSANDRA-12039: - Thanks for catching the bad test reports. I've looked at the jobs, and it's a problem with the new dtest, {{resume_test}}. It's hanging, or otherwise not finishing in a way that causes the machine it ran on to not report test results. > Add an index callback to be notified post bootstrap and before joining the > ring > --- > > Key: CASSANDRA-12039 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12039 > Project: Cassandra > Issue Type: New Feature >Reporter: Sergio Bossa >Assignee: Sergio Bossa > > Custom index implementations might need to be notified when the node finishes > bootstrapping in order to execute some blocking tasks before the node itself > goes into NORMAL state. > This is a proposal to add such functionality, which should roughly require > the following: > 1) Add a {{getPostBootstrapTask}} callback to the {{Index}} interface. > 2) Add an {{executePostBootstrapBlockingTasks}} method to > {{SecondaryIndexManager}} calling into the previously mentioned callback. > 3) Hook that into {{StorageService#joinTokenRing}}. > Thoughts? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12039) Add an index callback to be notified post bootstrap and before joining the ring
[ https://issues.apache.org/jira/browse/CASSANDRA-12039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15434713#comment-15434713 ] Sam Tunnicliffe commented on CASSANDRA-12039: - I've rebased both the C* and dtest branches, but I'm still seeing some issues on the CI runs. The dtest failure on the trunk branch looks unrelated and I can't repro locally, but the one on the 3.9 branch looks relevant. I've tried to repro that but it consistently times out for me when run locally. I'm also concerned about the test counts; the 3.0, 3.9 & trunk runs are reporting 692, 863 & 669 tests respectively compared to the mainline jobs reporting 858. As of last night when I kicked off the last run, the custom dtest branch was completely up to date with riptano/master so I'm not sure how to account for those discrepancies. I know we had some setup issues recently which presented like this, but I thought those were all fixed now. Any ideas [~philipthompson]? ||branch||testall||dtest|| |[12039-3.0|https://github.com/beobal/cassandra/tree/12039-3.0]|[testall|http://cassci.datastax.com/view/Dev/view/beobal/job/beobal-12039-3.0-testall]|[dtest|http://cassci.datastax.com/view/Dev/view/beobal/job/beobal-12039-3.0-dtest]| |[12039-3.9|https://github.com/beobal/cassandra/tree/12039-3.9]|[testall|http://cassci.datastax.com/view/Dev/view/beobal/job/beobal-12039-3.9-testall]|[dtest|http://cassci.datastax.com/view/Dev/view/beobal/job/beobal-12039-3.9-dtest]| |[12039-trunk|https://github.com/beobal/cassandra/tree/12039-trunk]|[testall|http://cassci.datastax.com/view/Dev/view/beobal/job/beobal-12039-trunk-testall]|[dtest|http://cassci.datastax.com/view/Dev/view/beobal/job/beobal-12039-trunk-dtest]| dtest branch: [CASSANDRA-12039|https://github.com/beobal/cassandra-dtest/tree/CASSANDRA-12039] > Add an index callback to be notified post bootstrap and before joining the > ring > --- > > Key: CASSANDRA-12039 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12039 > Project: Cassandra > Issue Type: New Feature >Reporter: Sergio Bossa >Assignee: Sergio Bossa > > Custom index implementations might need to be notified when the node finishes > bootstrapping in order to execute some blocking tasks before the node itself > goes into NORMAL state. > This is a proposal to add such functionality, which should roughly require > the following: > 1) Add a {{getPostBootstrapTask}} callback to the {{Index}} interface. > 2) Add an {{executePostBootstrapBlockingTasks}} method to > {{SecondaryIndexManager}} calling into the previously mentioned callback. > 3) Hook that into {{StorageService#joinTokenRing}}. > Thoughts? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12039) Add an index callback to be notified post bootstrap and before joining the ring
[ https://issues.apache.org/jira/browse/CASSANDRA-12039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15421549#comment-15421549 ] Sergio Bossa commented on CASSANDRA-12039: -- With [~philipthompson]'s suggestion, dtests now pass, except some apparently unrelated failures: https://cassci.datastax.com/view/Dev/view/sbtourist/job/sbtourist-CASSANDRA-12039-3.0-dtest/17/#showFailuresLink Are we good to merge? > Add an index callback to be notified post bootstrap and before joining the > ring > --- > > Key: CASSANDRA-12039 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12039 > Project: Cassandra > Issue Type: New Feature >Reporter: Sergio Bossa >Assignee: Sergio Bossa > > Custom index implementations might need to be notified when the node finishes > bootstrapping in order to execute some blocking tasks before the node itself > goes into NORMAL state. > This is a proposal to add such functionality, which should roughly require > the following: > 1) Add a {{getPostBootstrapTask}} callback to the {{Index}} interface. > 2) Add an {{executePostBootstrapBlockingTasks}} method to > {{SecondaryIndexManager}} calling into the previously mentioned callback. > 3) Hook that into {{StorageService#joinTokenRing}}. > Thoughts? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12039) Add an index callback to be notified post bootstrap and before joining the ring
[ https://issues.apache.org/jira/browse/CASSANDRA-12039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15420505#comment-15420505 ] Philip Thompson commented on CASSANDRA-12039: - I tried running it with debug output on the node that's failing to start, and all that did was make it pass every time: https://cassci.datastax.com/view/Parameterized/job/parameterized_dtest_multiplexer/246/ This makes me think bumping the start timeout on that node should be sufficient, or at the very least, that the issue is not in your C* patch > Add an index callback to be notified post bootstrap and before joining the > ring > --- > > Key: CASSANDRA-12039 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12039 > Project: Cassandra > Issue Type: New Feature >Reporter: Sergio Bossa >Assignee: Sergio Bossa > > Custom index implementations might need to be notified when the node finishes > bootstrapping in order to execute some blocking tasks before the node itself > goes into NORMAL state. > This is a proposal to add such functionality, which should roughly require > the following: > 1) Add a {{getPostBootstrapTask}} callback to the {{Index}} interface. > 2) Add an {{executePostBootstrapBlockingTasks}} method to > {{SecondaryIndexManager}} calling into the previously mentioned callback. > 3) Hook that into {{StorageService#joinTokenRing}}. > Thoughts? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12039) Add an index callback to be notified post bootstrap and before joining the ring
[ https://issues.apache.org/jira/browse/CASSANDRA-12039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15418696#comment-15418696 ] Sergio Bossa commented on CASSANDRA-12039: -- For the record, the failing dtest passes for me locally /cc [~philipthompson] > Add an index callback to be notified post bootstrap and before joining the > ring > --- > > Key: CASSANDRA-12039 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12039 > Project: Cassandra > Issue Type: New Feature >Reporter: Sergio Bossa >Assignee: Sergio Bossa > > Custom index implementations might need to be notified when the node finishes > bootstrapping in order to execute some blocking tasks before the node itself > goes into NORMAL state. > This is a proposal to add such functionality, which should roughly require > the following: > 1) Add a {{getPostBootstrapTask}} callback to the {{Index}} interface. > 2) Add an {{executePostBootstrapBlockingTasks}} method to > {{SecondaryIndexManager}} calling into the previously mentioned callback. > 3) Hook that into {{StorageService#joinTokenRing}}. > Thoughts? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12039) Add an index callback to be notified post bootstrap and before joining the ring
[ https://issues.apache.org/jira/browse/CASSANDRA-12039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15415034#comment-15415034 ] Sergio Bossa commented on CASSANDRA-12039: -- I've rebased both the C* and dtests patches, but dtests keep failing in strange ways, i.e.: https://cassci.datastax.com/view/Dev/view/sbtourist/job/sbtourist-CASSANDRA-12039-3.0-dtest/15/testReport/junit/secondary_indexes_test/TestPreJoinCallback/manual_join_test/ I believe it's some sort of instability in the test run, rather than in the test itself, any suggestions how to move forward? > Add an index callback to be notified post bootstrap and before joining the > ring > --- > > Key: CASSANDRA-12039 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12039 > Project: Cassandra > Issue Type: New Feature >Reporter: Sergio Bossa >Assignee: Sergio Bossa > > Custom index implementations might need to be notified when the node finishes > bootstrapping in order to execute some blocking tasks before the node itself > goes into NORMAL state. > This is a proposal to add such functionality, which should roughly require > the following: > 1) Add a {{getPostBootstrapTask}} callback to the {{Index}} interface. > 2) Add an {{executePostBootstrapBlockingTasks}} method to > {{SecondaryIndexManager}} calling into the previously mentioned callback. > 3) Hook that into {{StorageService#joinTokenRing}}. > Thoughts? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12039) Add an index callback to be notified post bootstrap and before joining the ring
[ https://issues.apache.org/jira/browse/CASSANDRA-12039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15407754#comment-15407754 ] Sergio Bossa commented on CASSANDRA-12039: -- bq. your new dtests look good to me, would you mind opening a PR for them please? Done: https://github.com/riptano/cassandra-dtest/pull/1170 > Add an index callback to be notified post bootstrap and before joining the > ring > --- > > Key: CASSANDRA-12039 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12039 > Project: Cassandra > Issue Type: New Feature >Reporter: Sergio Bossa >Assignee: Sergio Bossa > > Custom index implementations might need to be notified when the node finishes > bootstrapping in order to execute some blocking tasks before the node itself > goes into NORMAL state. > This is a proposal to add such functionality, which should roughly require > the following: > 1) Add a {{getPostBootstrapTask}} callback to the {{Index}} interface. > 2) Add an {{executePostBootstrapBlockingTasks}} method to > {{SecondaryIndexManager}} calling into the previously mentioned callback. > 3) Hook that into {{StorageService#joinTokenRing}}. > Thoughts? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12039) Add an index callback to be notified post bootstrap and before joining the ring
[ https://issues.apache.org/jira/browse/CASSANDRA-12039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15407702#comment-15407702 ] Sam Tunnicliffe commented on CASSANDRA-12039: - The patch LGTM, but there seem to have been a few issues with CI. It looks like environmental issues have prevented the testall job from running successfully, and there are some failures on the latest dtest run. Two of the three failures there are for bugs that have been fixed recently, so I've rebased on latest 3.0 and re-started CI. I'll commit when I get a clean run. [~sbtourist] your new dtests look good to me, would you mind opening a PR for them please? > Add an index callback to be notified post bootstrap and before joining the > ring > --- > > Key: CASSANDRA-12039 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12039 > Project: Cassandra > Issue Type: New Feature >Reporter: Sergio Bossa >Assignee: Sergio Bossa > > Custom index implementations might need to be notified when the node finishes > bootstrapping in order to execute some blocking tasks before the node itself > goes into NORMAL state. > This is a proposal to add such functionality, which should roughly require > the following: > 1) Add a {{getPostBootstrapTask}} callback to the {{Index}} interface. > 2) Add an {{executePostBootstrapBlockingTasks}} method to > {{SecondaryIndexManager}} calling into the previously mentioned callback. > 3) Hook that into {{StorageService#joinTokenRing}}. > Thoughts? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12039) Add an index callback to be notified post bootstrap and before joining the ring
[ https://issues.apache.org/jira/browse/CASSANDRA-12039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15404586#comment-15404586 ] Sergio Bossa commented on CASSANDRA-12039: -- [~beobal], I've updated the patch to make it work when joining after survey mode as described above, and added dtests for all cases (hopefully), here are the updated links: |[3.0 patch|https://github.com/apache/cassandra/compare/cassandra-3.0...sbtourist:CASSANDRA-12039-3.0?expand=1]| [dtests patch|https://github.com/riptano/cassandra-dtest/compare/master...sbtourist:CASSANDRA-12039?expand=1]| [dtests run|https://cassci.datastax.com/view/Dev/view/sbtourist/job/sbtourist-CASSANDRA-12039-3.0-dtest/]|[testall run|https://cassci.datastax.com/view/Dev/view/sbtourist/job/sbtourist-CASSANDRA-12039-3.0-testall/]| Regarding the second problem described above, moving the "ring wait" and "migrations wait" outside of the bootstrap block is probably too invasive for this narrow case, so I ended up ignoring it, assuming that if the 2i is migrated after the node joins, the callback will have nothing to do (basically treating it the same way as if a new table were created). > Add an index callback to be notified post bootstrap and before joining the > ring > --- > > Key: CASSANDRA-12039 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12039 > Project: Cassandra > Issue Type: New Feature >Reporter: Sergio Bossa >Assignee: Sergio Bossa > > Custom index implementations might need to be notified when the node finishes > bootstrapping in order to execute some blocking tasks before the node itself > goes into NORMAL state. > This is a proposal to add such functionality, which should roughly require > the following: > 1) Add a {{getPostBootstrapTask}} callback to the {{Index}} interface. > 2) Add an {{executePostBootstrapBlockingTasks}} method to > {{SecondaryIndexManager}} calling into the previously mentioned callback. > 3) Hook that into {{StorageService#joinTokenRing}}. > Thoughts? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12039) Add an index callback to be notified post bootstrap and before joining the ring
[ https://issues.apache.org/jira/browse/CASSANDRA-12039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15404570#comment-15404570 ] Sergio Bossa commented on CASSANDRA-12039: -- Thanks. > Add an index callback to be notified post bootstrap and before joining the > ring > --- > > Key: CASSANDRA-12039 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12039 > Project: Cassandra > Issue Type: New Feature >Reporter: Sergio Bossa >Assignee: Sergio Bossa > > Custom index implementations might need to be notified when the node finishes > bootstrapping in order to execute some blocking tasks before the node itself > goes into NORMAL state. > This is a proposal to add such functionality, which should roughly require > the following: > 1) Add a {{getPostBootstrapTask}} callback to the {{Index}} interface. > 2) Add an {{executePostBootstrapBlockingTasks}} method to > {{SecondaryIndexManager}} calling into the previously mentioned callback. > 3) Hook that into {{StorageService#joinTokenRing}}. > Thoughts? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12039) Add an index callback to be notified post bootstrap and before joining the ring
[ https://issues.apache.org/jira/browse/CASSANDRA-12039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15404320#comment-15404320 ] Brandon Williams commented on CASSANDRA-12039: -- Write survey mode works exactly like bootstrap, it just never sets the completed flag, so nodes continue to forward 'extra' writes to it, but then later if you decide the node should join you can do so with nodetool. Or if you decide it's bad, you can just kill it, since it never became a full member of the ring. > Add an index callback to be notified post bootstrap and before joining the > ring > --- > > Key: CASSANDRA-12039 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12039 > Project: Cassandra > Issue Type: New Feature >Reporter: Sergio Bossa >Assignee: Sergio Bossa > > Custom index implementations might need to be notified when the node finishes > bootstrapping in order to execute some blocking tasks before the node itself > goes into NORMAL state. > This is a proposal to add such functionality, which should roughly require > the following: > 1) Add a {{getPostBootstrapTask}} callback to the {{Index}} interface. > 2) Add an {{executePostBootstrapBlockingTasks}} method to > {{SecondaryIndexManager}} calling into the previously mentioned callback. > 3) Hook that into {{StorageService#joinTokenRing}}. > Thoughts? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12039) Add an index callback to be notified post bootstrap and before joining the ring
[ https://issues.apache.org/jira/browse/CASSANDRA-12039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15403873#comment-15403873 ] Sergio Bossa commented on CASSANDRA-12039: -- I'm writing dtests for all cases requiring a call to the pre-join task, and I noticed the following: 1) There's a bootstrap test ({{bootstrap_on_write_survey_test}}) that explicitly verifies the bootstrap state is left in progress after survey mode, so it seems to me we can actually rely on it to signal the bootstrap status on joining; pinging again [~brandon.williams] for feedback. 2) There's one more problem related to calling the pre-join task when joining *without* bootstrapping; in such case, the node doesn't wait for the migrations to complete (that is, {{MigrationManager#waitUntilReadyForBootstrap()}} is called only if bootstrapping), so it misses to wait for the new schema containing the column family, and no task is invoked. Should we run the {{MigrationManager}} checks regardless if the node is actually streaming or not (I'd say so)? /cc [~beobal] [~iamaleksey] > Add an index callback to be notified post bootstrap and before joining the > ring > --- > > Key: CASSANDRA-12039 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12039 > Project: Cassandra > Issue Type: New Feature >Reporter: Sergio Bossa >Assignee: Sergio Bossa > > Custom index implementations might need to be notified when the node finishes > bootstrapping in order to execute some blocking tasks before the node itself > goes into NORMAL state. > This is a proposal to add such functionality, which should roughly require > the following: > 1) Add a {{getPostBootstrapTask}} callback to the {{Index}} interface. > 2) Add an {{executePostBootstrapBlockingTasks}} method to > {{SecondaryIndexManager}} calling into the previously mentioned callback. > 3) Hook that into {{StorageService#joinTokenRing}}. > Thoughts? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12039) Add an index callback to be notified post bootstrap and before joining the ring
[ https://issues.apache.org/jira/browse/CASSANDRA-12039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15397457#comment-15397457 ] Sergio Bossa commented on CASSANDRA-12039: -- [~beobal], I've pushed one more commit addressing your latest points, and I'll look into writing any dtests. Also: bq. One thing that we should note is that when pre-join tasks are executed as a result of manually joining (i.e. start in write survey mode, then join via nodetool/JMX), the value of the bootstrap flag is always false. Obviously, this might not always be accurate as a node can perfectly well start up in survey mode but also perform bootstrap before an operator joins it. Short of exposing the overload of joinRing and extending the nodetool join command, I'm not sure there's any practical solution to this. I think we could actually leverage the bootstrap state, which is left pending at {{IN_PROGRESS}} during write survey and then set at {{COMPLETED}} when finally joining. But, I'm not really sure why the write survey mode works that way, that is why the bootstrap state is left in progress, and I don't want to rely on some unintentional behaviour: any ideas, [~brandon.williams]? > Add an index callback to be notified post bootstrap and before joining the > ring > --- > > Key: CASSANDRA-12039 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12039 > Project: Cassandra > Issue Type: New Feature >Reporter: Sergio Bossa >Assignee: Sergio Bossa > > Custom index implementations might need to be notified when the node finishes > bootstrapping in order to execute some blocking tasks before the node itself > goes into NORMAL state. > This is a proposal to add such functionality, which should roughly require > the following: > 1) Add a {{getPostBootstrapTask}} callback to the {{Index}} interface. > 2) Add an {{executePostBootstrapBlockingTasks}} method to > {{SecondaryIndexManager}} calling into the previously mentioned callback. > 3) Hook that into {{StorageService#joinTokenRing}}. > Thoughts? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12039) Add an index callback to be notified post bootstrap and before joining the ring
[ https://issues.apache.org/jira/browse/CASSANDRA-12039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15391866#comment-15391866 ] Sam Tunnicliffe commented on CASSANDRA-12039: - Thanks, the updated patch is looking good. If you amend slightly the new log statement in {{SIM}} to include the value being passed in, we can properly check that in a dtest. Otherwise, modulo the couple of nits below I think we'll be good to go once we get some dtests in place. One thing that we should note is that when pre-join tasks are executed as a result of manually joining (i.e. start in write survey mode, then join via nodetool/JMX), the value of the bootstrap flag is always false. Obviously, this might not always be accurate as a node can perfectly well start up in survey mode but also perform bootstrap before an operator joins it. Short of exposing the overload of {{joinRing}} and extending the nodetool join command, I'm not sure there's any practical solution to this. However, I guess this is quite a niche scenario, so I suggest that clearly documenting this limitation in {{NEWS.txt}} is sufficient. Would you mind adding an entry to {{NEWS.txt}} please? Nits: The filter condition in {{StorageService::executePreJoinTasks}} can be streamlined to: {code} .filter(cfs -> Schema.instance.getUserKeyspaces().contains(cfs.keyspace.getName())) {code} {{SIM::executeAllBlocking}} handles null {{Callable}} correctly, so the default impl of {{getPreJoinTask}} can just return {{null}} rather than a task which returns {{null}}. There's a couple of other minor formatting things, but I can fix those on commit. > Add an index callback to be notified post bootstrap and before joining the > ring > --- > > Key: CASSANDRA-12039 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12039 > Project: Cassandra > Issue Type: New Feature >Reporter: Sergio Bossa >Assignee: Sergio Bossa > > Custom index implementations might need to be notified when the node finishes > bootstrapping in order to execute some blocking tasks before the node itself > goes into NORMAL state. > This is a proposal to add such functionality, which should roughly require > the following: > 1) Add a {{getPostBootstrapTask}} callback to the {{Index}} interface. > 2) Add an {{executePostBootstrapBlockingTasks}} method to > {{SecondaryIndexManager}} calling into the previously mentioned callback. > 3) Hook that into {{StorageService#joinTokenRing}}. > Thoughts? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12039) Add an index callback to be notified post bootstrap and before joining the ring
[ https://issues.apache.org/jira/browse/CASSANDRA-12039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15385707#comment-15385707 ] Sergio Bossa commented on CASSANDRA-12039: -- [~beobal], new version pushed, fee free to review again. > Add an index callback to be notified post bootstrap and before joining the > ring > --- > > Key: CASSANDRA-12039 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12039 > Project: Cassandra > Issue Type: New Feature >Reporter: Sergio Bossa >Assignee: Sergio Bossa > > Custom index implementations might need to be notified when the node finishes > bootstrapping in order to execute some blocking tasks before the node itself > goes into NORMAL state. > This is a proposal to add such functionality, which should roughly require > the following: > 1) Add a {{getPostBootstrapTask}} callback to the {{Index}} interface. > 2) Add an {{executePostBootstrapBlockingTasks}} method to > {{SecondaryIndexManager}} calling into the previously mentioned callback. > 3) Hook that into {{StorageService#joinTokenRing}}. > Thoughts? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12039) Add an index callback to be notified post bootstrap and before joining the ring
[ https://issues.apache.org/jira/browse/CASSANDRA-12039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15385680#comment-15385680 ] Sergio Bossa commented on CASSANDRA-12039: -- bq. A failed bootstrap causes the node to remain in the JOINING state until successfully resumed. When it does finally succeed though, it follows the same code path as leaving survey mode, so fixing that should also take care of this. Got it, good point. bq. The impl that you added to CassandraIndex should be moved to Index, making it a default method Of course, and that's to preserve backward compatibility with other index implementations. Let me fix that. > Add an index callback to be notified post bootstrap and before joining the > ring > --- > > Key: CASSANDRA-12039 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12039 > Project: Cassandra > Issue Type: New Feature >Reporter: Sergio Bossa >Assignee: Sergio Bossa > > Custom index implementations might need to be notified when the node finishes > bootstrapping in order to execute some blocking tasks before the node itself > goes into NORMAL state. > This is a proposal to add such functionality, which should roughly require > the following: > 1) Add a {{getPostBootstrapTask}} callback to the {{Index}} interface. > 2) Add an {{executePostBootstrapBlockingTasks}} method to > {{SecondaryIndexManager}} calling into the previously mentioned callback. > 3) Hook that into {{StorageService#joinTokenRing}}. > Thoughts? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12039) Add an index callback to be notified post bootstrap and before joining the ring
[ https://issues.apache.org/jira/browse/CASSANDRA-12039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15385641#comment-15385641 ] Sam Tunnicliffe commented on CASSANDRA-12039: - bq.I think this is ok, as the node will have already joined the ring, and this is mainly a pre-join callback. A failed bootstrap causes the node to remain in the {{JOINING}} state until successfully resumed. When it does finally succeed though, it follows the same code path as leaving survey mode, so fixing that should also take care of this. bq.I provided an inline "null" implementation similar to what happens with other get*Task methods, what should I do differently? The impl that you added to {{CassandraIndex}} should be moved to {{Index}}, making it a {{default}} method on the iface so that adding that same implementation to every custom 2i is not necessary. > Add an index callback to be notified post bootstrap and before joining the > ring > --- > > Key: CASSANDRA-12039 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12039 > Project: Cassandra > Issue Type: New Feature >Reporter: Sergio Bossa >Assignee: Sergio Bossa > > Custom index implementations might need to be notified when the node finishes > bootstrapping in order to execute some blocking tasks before the node itself > goes into NORMAL state. > This is a proposal to add such functionality, which should roughly require > the following: > 1) Add a {{getPostBootstrapTask}} callback to the {{Index}} interface. > 2) Add an {{executePostBootstrapBlockingTasks}} method to > {{SecondaryIndexManager}} calling into the previously mentioned callback. > 3) Hook that into {{StorageService#joinTokenRing}}. > Thoughts? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12039) Add an index callback to be notified post bootstrap and before joining the ring
[ https://issues.apache.org/jira/browse/CASSANDRA-12039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15384752#comment-15384752 ] Sergio Bossa commented on CASSANDRA-12039: -- [~beobal], thanks for your comments, see my answers below: bq. Pre-join tasks are not executed if a node is started in write survey mode and then fully joins the ring later. Good point, I can fix this. bq. If bootstrap fails and is subsequently resumed, pre-join tasks are not executed on its completion. I think this is ok, as the node will have already joined the ring, and this is mainly a pre-join callback. bq. Index::getPreJoinTask should have a default no-op implementation I'm not sure I get this: I provided an inline "null" implementation similar to what happens with other {{get*Task}} methods, what should I do differently? bq. I think it would be sufficient to add some debug logging to StorageService::executePreJoinTasks and check for that in the node logs. Sure I can do that. > Add an index callback to be notified post bootstrap and before joining the > ring > --- > > Key: CASSANDRA-12039 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12039 > Project: Cassandra > Issue Type: New Feature >Reporter: Sergio Bossa >Assignee: Sergio Bossa > > Custom index implementations might need to be notified when the node finishes > bootstrapping in order to execute some blocking tasks before the node itself > goes into NORMAL state. > This is a proposal to add such functionality, which should roughly require > the following: > 1) Add a {{getPostBootstrapTask}} callback to the {{Index}} interface. > 2) Add an {{executePostBootstrapBlockingTasks}} method to > {{SecondaryIndexManager}} calling into the previously mentioned callback. > 3) Hook that into {{StorageService#joinTokenRing}}. > Thoughts? -- This message was sent by Atlassian JIRA (v6.3.4#6332)